docs: Add guide security-of-web-scraping

[Mantisus]

Description

• Add a guide covering the security threats a crawler can encounter while scraping, how to handle each of them, and the Crawlee defaults that already mitigate some of them.

Issues

• Closes: Secure scraping guide #1872

[vdusek]

a few thoughts

[vdusek]

LGTM, @szaganek, would you like to take a look as well?

[szaganek]

This is very, very LLM-heavy 😅 I'd suggest to:

• get rid of all en dashes, break these long sentences into shorter ones
• get rid of bold for emphasis
• use contractions
• trim a lil fat, in general. What I mean by that, people won't read this doc "from cover to cover". So sections need to be a little more self-contained and easier to scan through. Starting a section with "Everything so far has been about the requests your crawler makes." doesn't make sense, because a doc is not meant to be read this way.
• adding to that, I'm not sure about the information architecture. Maybe a little extra structure to these headings would help (like threats: requests vs data).

[Mantisus]

@szaganek, I've updated the text based on some of your suggestions.

adding to that, I'm not sure about the information architecture. Maybe a little extra structure to these headings would help (like threats: requests vs data).

I'm not sure about restructuring (though I agree the current architecture isn't perfect). I worry about adding another heading layer for grouping. It could actually make this harder to read because some threats are tightly related but might naturally belong to different meta-sections. For example, queueing URLs through enqueue_links could just as well live under untrusted-content handling and SSRF attacks.

[szaganek]

Thank you, @Mantisus!

Grouping information in a clear way is always a challenge :) Not blocking at this point.