AI Crawler List: Every Bot, What It Wants, and How to Block It
An AI crawler list of all 62 user-agent strings, reconciled against operator docs and robots.txt telemetry, plus the bots and tokens no directory covers.
David ThomsonBlog
News, insights and more from TechnologyChecker
David Thomson
TikTok Statistics 2026: Users, B2B Insights, Revenue, Downloads & Trends
TikTok hit $14.3B revenue and ~1.37B modeled users. Full 2026 TikTok statistics: users, demographics, downloads, creator pay, ad benchmarks and the U.S. ban.
Elif Arslan
HubSpot Statistics 2026: Trends, Insights and HubSpot Market Share
We analyzed 121,300 HubSpot domains and 99,700 companies to reveal HubSpot's market share, migration trends, and company adoption data for 2026.
Sophie Clarke
•Updated
Firefox Market Share August 2026: 3.6% Overall, 7.4% Desktop, 0.7% Mobile
Firefox holds 3.6% of human web traffic in August 2026, flat year over year. It's also the only browser with no weekday-weekend swing — because nobody is issued Firefox.
Emma Davies
HTTPS Adoption 2026: 98.6% of Human Traffic Is Encrypted
Live Cloudflare Radar data: 98.6% of human web traffic is HTTPS in 2026, but bot traffic is only 87.8%. Full 121-country breakdown, 2025 vs 2026.
Emma Davies
Webflow to Next.js Migration: The Complete 2026 Guide (Why, How, Costs & Services)
How to migrate from Webflow to Next.js in 2026: why teams switch, a step-by-step process, real cost ranges, and how to choose a migration partner.
David Thomson
•Updated
AI Crawler Statistics in 2026: What AI Crawlers Actually Want? (August 2026 Update)
Live Cloudflare Radar data for July 2026: training is 44.5% of AI crawl purpose, up from 35.7% a year ago but off June's peak. Includes a correction to our 52.3% figure.
David Thomson
•Updated
Bot Traffic Statistics 2026: How Much of the Web Is Bots? (August 2026 Update)
Live Cloudflare Radar data: bots are 34.9% of web traffic, Meta retook the #2 operator slot as Anthropic slid to 10.5%, and human traffic has a weekly rhythm bots can't fake.
David Thomson
•Updated
Mobile vs Desktop in the UK: We Filtered Out the Bots — Here's the Real Human Split in 2026 (August 2026 Update)
UK web traffic ran 59% desktop in July 2026 per Cloudflare Radar, but filter the bots out and real humans are still mobile-first at 54% — and 60% every weekend. See the UK device data.
Emma Davies
•Updated
Certificate Authority Market Share 2026: Top SSL CAs Ranked
Certificate authority market share in 2026: Let's Encrypt still issues 57% of web certificates, and Amazon Trust Services is the fastest riser, up five-fold in a year to challenge Sectigo for third. Year-over-year CT-log data.
Emma Davies
•Updated
Chrome Market Share August 2026: 68.7% Overall, 69% Mobile, 68% Desktop
Chrome holds 68.7% of human web traffic in August 2026, the #1 browser by 4x. It wins every platform but the iPhone, and it's the only browser immune to the weekly work-home cycle.
Emma Davies
•Updated
Technology Detection Methods: How We Scan 50M+ Domains
Technology detection explained: we match 6,675 technologies across 14 signal types on a 50M-domain database, re-crawled monthly and scored 0–100.
David Thomson
•Updated
Instagram User Statistics 2026: We Analyzed Demographics, Growth & Country Rankings
Instagram has ~3 billion users in 2026. We analyzed the demographics, country-by-country web rankings (Q1 vs Q2), and the 31,886 sites that use it.
Elif Arslan
•Updated
Data Center Market Statistics 2026: AI, $416B Market & the Power Ceiling
Data center market statistics for 2026: a $416B market in 2024, the AI segment growing 25%+ a year, Nvidia at $193.7B, and power demand up 165% by 2030.
Emma Davies
•Updated
Safari Market Share August 2026: 17.6% Overall, 26.1% Mobile, 6.7% Desktop
Safari holds 17.6% of human web traffic in August 2026, #2 but slipping. It also gains 2.2 points every weekend — the biggest weekly swing of any browser.
Emma Davies
•Updated
AI in Marketing Statistics 2026: 35 Stats on Adoption, ROI and Trust
AI in marketing by the numbers for 2026: 35 stats on market size, adoption, ROI and the consumer trust gap, plus real use cases, charts and data tables.
Emre Elbeyoglu
•Updated
Influencer Marketing Statistics: 40+ Key Data Points for 2026
40+ influencer marketing statistics for 2026: $32.55B market size, ROI benchmarks, AI creator matching, B2B adoption, and TikTok's $153B brand value. Fully sourced.
Elif Arslan
•Updated
Social Media Advertising Insights 2026: 35 Stats, Plus 2M Live Ad Pixels
We tracked 2M+ live ad pixels and mapped them against $480B in projected spend. 35 stats on social media advertising ROI, engagement, and real platform traffic in 2026.
Emma Davies
•Updated
Europe E-Commerce Statistics 2026: Germany, France & Italy Platform Mix
We track 13 e-commerce platforms in the EU's top 3 markets. PrestaShop beats Shopify 3.6:1 in France, WooCommerce leads Italy, Ecwid tops Germany.
Emma Davies
Open-Source AI Adoption 2026: 5.6M Projects vs Real Deployment
Stanford counts 5.6M open-source AI projects. We scanned 50M+ domains for what's deployed: Botpress leads at 1,558 vs 52,682 for the closed OpenAI API.
David Thomson
•Updated
AI Market Size Statistics 2026: 26 Charts From $94B to $1.67 Trillion by 2031
Global AI market: $94.81B (2020) → $1.675T by 2031. 26 charts on segments, adoption and GenAI — plus why six major forecasters project $1.7T to $4.8T.
Emma Davies
•Updated
Claude Statistics 2026: #2 in GenAI, $30B Run-Rate (August 2026 Update)
Cloudflare Radar (July 2026): Claude holds #2 in Generative AI for 191 straight days; ClaudeBot eased to 16.28% off its June peak, still 1.7x GPTBot. Plus $30B ARR.
Emre Elbeyoglu
•Updated
Turkey E-commerce Statistics 2026: The 4.57 Trillion TL Year, in 9 Charts and 7 Patterns We Saw
Turkey's e-commerce hit 4.57T TL in 2025. ETBİS plus Q2 2026 Cloudflare Radar: Trendyol #3 globally in Fast Fashion, 955 Shopify stores.
Elif Arslan
•Updated
Gen Z UK Social Media Statistics 2026: TikTok, SHEIN & AI Trends
70+ Gen Z UK social stats (TikTok 120 min/day, SHEIN displaces ASOS) plus a Cloudflare Radar year-over-year: UK mobile-OS share fell back under 50% by the Q2 2026 close, and AI answer-engine crawlers tripled to ~33% of UK AI-bot traffic.
Elif Arslan
•Updated
Technology Trends 2026: We Checked 39 Forecasts on 29.6M Sites
We checked 39 tech forecasts (Gartner, IDC, IEA) against our 29.6M-domain detection data. ChatGPT is live on 0.18% of the public web. See the gap.
Emma Davies
•Updated
Mobile Internet Usage by Country: Device Share Statistics for 2026 (August 2026 Update)
July 2026: mobile is still the human majority at 54.64%, but 61% of the desktop rise is now bot composition, not behavior. Plus the weekend swing that every country ranking hides.
Emma Davies
•Updated
Most Reliable Internet by Country: TCP Connection Quality Ranked in 2026 (August 2026 Update)
July 2026: Israel took the top spot outright at 92.4%, Iraq's rebound finally held for a full month at 79.5%, and global reliability is still structurally flat near 80%.
Emma Davies
•Updated
Search Engine Market Share 2026: Google Still Sends 87.6% of Referrals (August 2026 Update)
Google sent 88.08% of search referrals in July 2026, ChatGPT a record 0.913%: up 5.3x in a year and still under one point. Cloudflare Radar referral and crawl-ratio data.
Mehmet Suleyman
•Updated
DNSSEC Adoption in 2026: Only 0.62% of DNS Queries Are Actually Protected (August 2026 Update)
Cloudflare Radar data shows 8.2% of DNS queries reach DNSSEC-signed domains, but only 0.62% are validated end-to-end. In July 2026 that number is up 68% year over year.
Emma Davies
•Updated
Ecommerce marketplace market share in 2026: eBay held #3 through July, and three category leaders kept rank 1 for a full year (August 2026 Update)
July 2026 Cloudflare Radar rankings: Amazon, Shopee and Shein each held rank 1 for a full year, eBay kept #3, and our Shopee-versus-Temu low-cost call was wrong.
Mehmet Suleyman
•Updated
Marketing Technology Statistics: 30 Key MarTech Trends and Data Points for 2026
30 marketing technology statistics — market size, AI adoption, automation, CDP growth — cross-checked against our own detection data on 50M+ domains and live AI-crawler traffic.
Elif Arslan
•Updated
ChatGPT Statistics 2026 - 900M Users, $25B ARR, and the Cloudflare Crawl Data That Flipped Right Back (August 2026 Update)
Live Cloudflare Radar data (July 2026): GPTBot took 9.7% of AI-bot traffic to ClaudeBot's 16.3%, inverting last July's order. Plus 900M users, $25B ARR, 50M paid subs.
Emma Davies
•Updated
Web Traffic Statistics 2026: We Analyzed Billions of Requests - Here Are the 15 Numbers That Matter (August 2026 Update)
July 2026 Cloudflare Radar web traffic data: bots reached 34.7% of requests, all-traffic desktop hit 60.8%, HTTP/3 fell below 20%. 15 numbers, with July 2025 baselines.
David Thomson
•Updated
DMARC adoption statistics 2026: 89% of emails pass DMARC but 14.5% still fail SPF (August 2026 Update)
Cloudflare Radar, July 2026: DMARC pass hit 89.37% and no-policy mail fell to 5.44%. The spoof wave broke to 17.67%, but encryption in transit slid again, to 80.77%.
David Thomson
•Updated
We analyzed HTTP protocol adoption in 2026: 27.8% of the web still runs HTTP/1.x (August 2026 Update)
HTTP/3 fell below 20% in July 2026, to 19.84% from 21.64% a year earlier, while HTTP/2 took 52.27% and HTTP/1.x held 27.89%. The plateau we called became a decline.
David Thomson
•Updated
Internet Outage Tracker 2026: Power Failures Retake the Top Cause as Exam Shutdowns Fade (August 2026 Update)
July 2026 internet outage report: power failures caused 5 of 11 disruptions (45%), up from 4% in Q2. Cuba's grid collapsed twice, and Iraq's exam shutdowns ended July 8.
Emma Davies
•Updated
AI Adoption Trends in 2026: We Analyzed 4 Internet Layers of Cloudflare Radar Data to Map How AI Hype Becomes Real Adoption (August 2026 Update)
Training crawls stopped climbing in July 2026. Our 4-layer Cloudflare Radar read: training flat at 44.5%, Anthropic now the #3 bot operator, Claude-User the #2 bot.
Emma Davies
•Updated
We Analyzed robots.txt Across Cloudflare's Network: Which AI Crawlers Get Blocked Most and Why (August 2026 Update)
robots.txt analysis across Cloudflare's network, July 2026 data: Anthropic's crawl-to-refer ratio fell to 1,917:1, OpenAI passed Perplexity, and AI-bot 403s hit 9.64%.
David Thomson
•Updated
Cloud Provider Traffic Share in 2026: AWS, Google Cloud, and Azure's Share of Internet Traffic (August 2026 Update)
How much internet traffic do AWS, Google Cloud, and Azure carry? July 2026 Radar data doubles Azure's byte share year over year and cuts AWS first byte to 76 ms.
Emma Davies
•Updated
Most Popular Platforms & Technologies by Category in 2026: All 16 Category Leaders Held #1 for a Full Year, Perplexity Took #3 Back from Gemini, and Music's Top Nine Didn't Move a Seat (August 2026 Update)
Cloudflare Radar DNS rankings, July 31, 2026: all 16 category #1s are unchanged since July 2025, Perplexity retook #3 in AI from Gemini, and Music's top nine never moved.
Emma Davies
•Updated
Salesforce Statistics: Trends, Insights and Salesforce Market Share 2026
We analyzed 36,143 Salesforce domains and 27,256 companies to reveal market share, migration trends, and company adoption data for 2026.
Sophie Clarke
•Updated
We Analyzed 8.7B SSL Certificates: RSA Retook the Lead From ECDSA in July 2026 (August 2026 Update)
RSA retook the public-web lead from ECDSA in July 2026 (50.3% vs 49.7%). Correcting the record too: Cloudflare's revised data moves the crossover from May to June.
David Thomson
•Updated
Google Analytics 4 Migration: Why Millions of Sites Still Haven't Switched in 2026
We scanned 29.9M active domains in March 2026: GA4 sits at 3.69M sites while 6M still carry dead UA tags. See the migration flows and best alternatives.
Elif Arslan
•Updated
2026 Marketing Automation Report: Market Share & Trends Based on 29.9M Active Domains
Our crawlers analyzed 29.9M active domains to reveal 2026's marketing automation landscape. 17 platforms profiled: HighLevel's explosive growth, Klaviyo's e-commerce dominance, Constant Contact's dramatic decline & the agentic AI shift.
Mehmet Suleyman
•Updated
How to Find Companies Using n8n in 2026: Full Guide + Best Detection Tools
Discover how to find companies using n8n workflow automation in 2026. Learn the best tools for backend detection, including TechnologyChecker.io's real-time alerts and historical data – outperforming basic scanners for accurate n8n user lists and leads.
Elif Arslan
•Updated
BuiltWith Alternatives in 2026: 5 Tools I Actually Tested
Tired of BuiltWith's $199/mo price? I tested 5 alternatives on accuracy, price, and contact data. Here's which fits which job, plus real G2 reviews.
Mehmet Suleyman
•Updated
Shopify Analytics: Trends, Insights and Shopify Market Share 2026
We crawled 2.4M live Shopify stores and enriched 80,970 with company data. See real market share, platform migration, and the tech stack merchants run.
Mehmet Suleyman
•Updated
The Technology Lookup Software Industry: 2025–2026 Statistics & Insights (from my know-how)
The latest 2025–2026 insights, statistics, and trends in the technology lookup software industry from BuiltWith and Wappalyzer to SimilarTech and Technologychecker.io. Includes March 2026 web technology market share data, detection methods deep-dive, and how technographics drive smarter B2B sales, marketing, and growth strategies.
Mehmet Suleyman
•Updated
Why 73% of SaaS Companies Switch CRMs Within 3 Years (And What That Means for Sales Teams)
73% of SaaS companies abandon their CRM within 3 years, costing $2.5-5M in lost revenue per switch. Explore CRM failure rates, adoption data, and strategies to break the switching cycle.
Sophie Clarke
•Updated
Shopify Migration Data 2026: 31,946 Stores In, 10,271 Out
Our crawl tracked every store that switched between Shopify and 7 rival platforms. Shopify wins migration roughly 3:1 — here's exactly who moves and where.
Mehmet Suleyman