David Thomson

CTO

David is the CTO of TechnologyChecker, responsible for the engineering and architecture behind the platform's crawling infrastructure. Before joining TechnologyChecker, he spent five years at Google on the Search team, where he worked on large-scale crawling and indexing systems that shaped his approach to building high-performance data infrastructure.

He oversees the detection systems that successfully crawl 29.9M live business domains per sweep, ensuring accurate and timely identification of technology stacks across the web. His work focuses on scalable data pipelines, real-time processing, and maintaining detection accuracy across HTTP headers, JavaScript libraries, DNS records, and HTML patterns.

Based in Edinburgh, David is a devoted single malt whisky enthusiast when he's not architecting distributed systems.

Areas of Expertise

Scalable Data PipelinesReal-Time ProcessingWeb Crawling ArchitectureDistributed Systems

Credentials

  • MEng Computer Science, University of Edinburgh
  • AWS Solutions Architect Professional
  • Contributor, Open Source Crawling Frameworks

Achievements

  • Architected infrastructure processing 29.9M domain scans per sweep with 99.9% uptime
  • Reduced detection latency from hours to under 60 seconds for real-time alerts
  • Built a technology fingerprinting engine covering HTTP headers, JS libraries, DNS records, and HTML patterns
  • Analysed petabytes of Common Crawl data to power TechnologyChecker's historical technology adoption database
David Thomson

15+ years of experience

David’s Impact

Written from real expertise — every post adds first-hand know-how and original insights.

Deepest Research
Articles
12
Total reads
2.7K
Hours of work
401h
Topics covered
10

Top articles by reads

All-time
  1. 1
    HTTP Protocol Adoption Q3 2026: HTTP/3 Is Shrinking, HTTP/1.x Isn't
    607 reads100 min read~50h spent creating the content
  2. 2
    Web Traffic Statistics Q3 2026: The 15 Numbers That Matter
    521 reads79 min read~51h spent creating the content
  3. 3
    robots.txt AI Crawler Report Q3 2026: Who Gets Blocked
    498 reads79 min read~49h spent creating the content
  4. 4
    SSL Certificate Statistics Q3 2026: ECDSA Now Matches RSA, 50/50
    307 reads74 min read~43h spent creating the content
  5. 5
    What Percentage of Internet Traffic Is Bots? Q3 2026 Statistics
    270 reads67 min read~40.5h spent creating the content

Content effort estimates the hours David spent researching and creating each post. Every article is written in David’s field of expertise, so it carries first-hand know-how and unique insights — not commodity information rehashed from elsewhere.

Articles by David Thomson