AI Crawler Directory · ByteDance
Bytespider:
what it is and how to verify it.
OperatorByteDance
PurposeAI model training
Respects robots.txtPartially
Executes JavaScriptNo
Hits verifiableNo — nothing published
TypeHTTP crawler / fetcher
ByteDance publishes no crawler documentation; site owners widely report inconsistent robots.txt compliance.
Bytespider is ByteDance’s web crawler, widely understood to collect training data for the company’s AI models, including the Doubao assistant. It is one of the highest-volume AI crawlers on the web — server-log studies have repeatedly placed it among the most aggressive fetchers by request count — yet ByteDance publishes no official documentation for it.
That lack of documentation is the story. There is no published IP list, no reverse-DNS scheme, and no stated robots.txt policy; webmasters report mixed compliance in practice. The only contact surface is the spider-feedback@bytedance.com address embedded in the UA string.
For crawler analytics this means every Bytespider line in your logs is unverifiable. If its crawl volume is a burden, robots.txt is worth trying first, but rate-limiting or blocking by UA at the CDN is what actually holds.
Bytespider User-Agent string
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)Representative format — ByteDance does not publish an exact canonical UA string; the bot token is the stable identifier.
How to verify Bytespider hits are real
The User-Agent header is plain text — any client can claim to be Bytespider, and scrapers routinely do. No published IP ranges or reverse-DNS scheme — Bytespider hits cannot be verified.
Because nothing is published, the honest label for every Bytespider log line is unverified — it may be ByteDance, or a scraper borrowing the name.
Allow or block Bytespider in robots.txt
This is a training-data control: allowing it lets your content shape what future models know; blocking it keeps your material out of training corpora without affecting live citations. A policy call, not a traffic one.
Allow
User-agent: Bytespider
Allow: /Block
User-agent: Bytespider
Disallow: /
# Compliance is inconsistently reported — add a CDN/WAF rule if volume persists.Bytespider — frequently asked questions
What is Bytespider?
Bytespider is ByteDance’s web crawler, associated with collecting content for its AI models such as Doubao. ByteDance publishes no official documentation for it, which is unusual among major AI crawlers.
Does Bytespider respect robots.txt?
Reports are mixed. Some sites see it honor disallow rules; many others document continued fetching. With no official policy statement from ByteDance, treat robots.txt as a first attempt and CDN-level blocking as the enforcement path.
Can I verify Bytespider hits?
No. ByteDance publishes no IP ranges and no reverse-DNS scheme, so a Bytespider UA cannot be authenticated. All hits are unverified by definition.
Why is Bytespider so high-volume?
Server-log analyses consistently rank it among the most request-heavy AI crawlers, likely reflecting large-scale corpus building. High volume with no verification or documented rate controls is why many sites choose to block or throttle it.
See which AI crawlers hit your site — and what the traffic earns
Attrifast tracks AI referrals as revenue lines: ChatGPT, Perplexity, Claude, and Gemini visits tied to real Stripe money. $9.99/mo flat.
- One script tag and a Stripe key — live in minutes
- Cookieless, so no consent banner for analytics
- Every AI referral matched to the payment it produced
7-day free trial · $0 due today · then $9.99/mo · cancel anytime
