Perplexity-User: The Fetcher That Ignores robots.txt
Perplexity · User-triggered fetcher · Last verified October 2026
Perplexity-User fetches a page when a Perplexity user asks about a specific URL. Perplexity documents that it generally ignores robots.txt for these fetches, treating them like a browser visit rather than a crawl. A robots.txt block does not reliably stop it — this is documented behavior, not a bug.
What is Perplexity-User?
Reported token for fetches triggered by a Perplexity user asking about a specific URL.
The user-agent string
Perplexity-User/1.0
Does Perplexity-User respect robots.txt?
No — by design. Perplexity's docs state Perplexity-User generally ignores robots.txt because a person triggered the request.
How to block Perplexity-User in robots.txt
User-agent: Perplexity-User Disallow: /
Important: Perplexity-User is documented as potentially ignoring robots.txt (see above). This rule is still worth writing — it signals intent and stops compliant variants — but don't rely on it alone.
How to explicitly allow Perplexity-User
Useful when your robots.txt has broad Disallow rules and you want to carve out an exception:
User-agent: Perplexity-User Allow: /
One bot per User-agent block; specific tokens beat User-agent: *.
What blocking Perplexity-User does (and doesn't do)
A robots.txt block against Perplexity-User is mostly theater. If you genuinely need to stop these fetches, you need server- or WAF-level blocking by user agent — and even then, weigh whether breaking the experience for users asking Perplexity about your pages is worth it. It has no effect on PerplexityBot's index crawling either way.
How to check if Perplexity-User is crawling your site
Search your access logs for the token:
grep -i "Perplexity-User" /var/log/nginx/access.log | wc -l
On Cloudflare, filter Firewall Events or Analytics by User Agent. If the count is zero over a month, the bot isn't visiting you — either your robots.txt blocks it, or it simply hasn't found you worth crawling.
Beyond robots.txt
robots.txt is a request, not a barrier. For server- and CDN-level blocking (nginx, Apache, Cloudflare WAF), see blocking beyond robots.txt on the overview page.
Perplexity-User vs. related bots
Changelog
Last verified October 2026.
What does Perplexity-User see on your site?
Operator docs: Perplexity · All AI crawlers