Bytespider: ByteDance's Training Crawler
ByteDance · Training crawler · Last verified October 2026
Bytespider is ByteDance's crawler, gathering training data for its models. It has high crawl volume relative to its citation footprint — and widespread independent reports say it ignores robots.txt. If you want it gone, robots.txt won't do it: you need firewall or CDN (WAF) rules matching its user agent.
What is Bytespider?
ByteDance's crawler, gathering training data for its models. High crawl volume relative to its citation footprint.
The user-agent string
Mozilla/5.0 (compatible; Bytespider; +https://bytespider.com/)
Does Bytespider respect robots.txt?
Widely reported no. Multiple independent reports show Bytespider fetching pages that disallowed it, and it publishes no compliance documentation. Treat robots.txt as advisory for this bot.
How to block Bytespider in robots.txt
User-agent: Bytespider Disallow: /
Important: Bytespider is widely reported to ignore robots.txt. Write the rule, but enforce it at your firewall or CDN — see "blocking beyond robots.txt" on the overview page.
How to explicitly allow Bytespider
Useful when your robots.txt has broad Disallow rules and you want to carve out an exception:
User-agent: Bytespider Allow: /
One bot per User-agent block; specific tokens beat User-agent: *.
What blocking Bytespider does (and doesn't do)
A robots.txt block against Bytespider is mostly theater. Real blocking means WAF rules (e.g., Cloudflare) matching Bytespider in the user agent, or nginx/Apache blocks — see "blocking beyond robots.txt" on the overview page. If your logs show heavy bot traffic with little AI-visibility payoff, Bytespider is a usual suspect: a common candidate for rate-limiting rather than full blocking.
How to check if Bytespider is crawling your site
Search your access logs for the token:
grep -i "Bytespider" /var/log/nginx/access.log | wc -l
On Cloudflare, filter Firewall Events or Analytics by User Agent. If the count is zero over a month, the bot isn't visiting you — either your robots.txt blocks it, or it simply hasn't found you worth crawling.
Beyond robots.txt
robots.txt is a request, not a barrier. For server- and CDN-level blocking (nginx, Apache, Cloudflare WAF), see blocking beyond robots.txt on the overview page.
Bytespider vs. related bots
Changelog
No published compliance documentation as of October 2026. Last verified October 2026.
What does Bytespider see on your site?
Operator docs: ByteDance · All AI crawlers