David Thomson

CTO

David is the CTO of TechnologyChecker, responsible for the engineering and architecture behind the platform's crawling infrastructure. Before joining TechnologyChecker, he spent five years at Google on the Search team, where he worked on large-scale crawling and indexing systems that shaped his approach to building high-performance data infrastructure.

He oversees the detection systems that successfully crawl 29.9M live business domains per sweep, ensuring accurate and timely identification of technology stacks across the web. His work focuses on scalable data pipelines, real-time processing, and maintaining detection accuracy across HTTP headers, JavaScript libraries, DNS records, and HTML patterns.

Based in Edinburgh, David is a devoted single malt whisky enthusiast when he's not architecting distributed systems.

Areas of Expertise

Scalable Data PipelinesReal-Time ProcessingWeb Crawling ArchitectureDistributed Systems

Credentials

  • MEng Computer Science, University of Edinburgh
  • AWS Solutions Architect Professional
  • Contributor, Open Source Crawling Frameworks

Achievements

  • Architected infrastructure processing 29.9M domain scans per sweep with 99.9% uptime
  • Reduced detection latency from hours to under 60 seconds for real-time alerts
  • Built a technology fingerprinting engine covering HTTP headers, JS libraries, DNS records, and HTML patterns
  • Analysed petabytes of Common Crawl data to power TechnologyChecker's historical technology adoption database
David Thomson

15+ years of experience

David’s Impact

Written from real expertise — every post adds first-hand know-how and original insights.

Articles
11
Total reads
2.7K
Hours of work
308h
Topics covered
10

Top articles by reads

All-time
  1. 1
    We analyzed HTTP protocol adoption in 2026: 27.8% of the web still runs HTTP/1.x (August 2026 Update)
    607 reads84 min read~42h spent creating the content
  2. 2
    Web Traffic Statistics 2026: We Analyzed Billions of Requests - Here Are the 15 Numbers That Matter (August 2026 Update)
    521 reads62 min read~41.5h spent creating the content
  3. 3
    We Analyzed robots.txt Across Cloudflare's Network: Which AI Crawlers Get Blocked Most and Why (August 2026 Update)
    498 reads67 min read~41.5h spent creating the content
  4. 4
    We Analyzed 8.7B SSL Certificates: RSA Retook the Lead From ECDSA in July 2026 (August 2026 Update)
    307 reads56 min read~33h spent creating the content
  5. 5
    Bot Traffic Statistics 2026: How Much of the Web Is Bots? (August 2026 Update)
    270 reads46 min read~27.5h spent creating the content

Content effort estimates the hours David spent researching and creating each post. Every article is written in David’s field of expertise, so it carries first-hand know-how and unique insights — not commodity information rehashed from elsewhere.

Articles by David Thomson