David Thomson

CTO

David is the CTO of TechnologyChecker, responsible for the engineering and architecture behind the platform's crawling infrastructure. Before joining TechnologyChecker, he spent five years at Google on the Search team, where he worked on large-scale crawling and indexing systems that shaped his approach to building high-performance data infrastructure.

He oversees the detection systems that scan over 50 million domains monthly, ensuring accurate and timely identification of technology stacks across the web. His work focuses on scalable data pipelines, real-time processing, and maintaining detection accuracy across HTTP headers, JavaScript libraries, DNS records, and HTML patterns.

Based in Edinburgh, David is a devoted single malt whisky enthusiast when he's not architecting distributed systems.

Areas of Expertise

Scalable Data PipelinesReal-Time ProcessingWeb Crawling ArchitectureDistributed Systems

Credentials

  • MEng Computer Science, University of Edinburgh
  • AWS Solutions Architect Professional
  • Contributor, Open Source Crawling Frameworks

Achievements

  • Architected infrastructure processing 50M+ domain scans per month with 99.9% uptime
  • Reduced detection latency from hours to under 60 seconds for real-time alerts
  • Built a technology fingerprinting engine covering HTTP headers, JS libraries, DNS records, and HTML patterns
  • Analysed petabytes of Common Crawl data to power TechnologyChecker's historical technology adoption database
David Thomson

15+ years of experience

Articles by David Thomson