webfetch tool gets blocked by Anubis

[vikanezrimaya]

Description

(cont. from #2228)

Since OpenCode's webfetch identifies itself as a browser, but doesn't execute JS, it gets stuck on Anubis challenges.

Reading the source code, Cloudflare has special-casing, but not Anubis. While mimicking browser User-Agents is bad in general, I suppose it has use in adversarial environments where non-browser UAs get blocked. However, Anubis (at default setting) actually rewards honesty — honest user-agents do not get blocked. Playing by its rules would be beneficial for all parties.

Note: Anubis challenges may respond with HTTP 200 depending on configuration — intentional behavior by the developer (perhaps so dumb scraper bots don't start hammering the website through residential proxies as a bypass tactic, or alert the operator with elevated error rates on fetches). Most robust way to detect will probably be checking Set-Cookie headers in the response, since Anubis stores a challenge ID in cookies.

Plugins

No response

OpenCode version

all versions since 84e4afc are affected

Steps to reproduce

1. Find a page with an Anubis challenge (or put up Anubis on a test stand)
2. Tell your agent to fetch it
3. Observe agent get stuck on challenge. (Craftier models might think about bypassing it manually)

Screenshot and/or share link

No response

Operating System

n/a

Terminal

n/a