Proposal: Standardize User-Agent headers in Importers to prevent 403 blocking #2123
Kiran95021
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Hi everyone! 👋
I am a new contributor looking into the
importersmodule. While analyzing how we fetch data from upstream sources (like GitHub, GitLab, and NVD), I noticed that we don't seem to have a standardizedUser-Agentstrategy.The Problem
Many importers currently use the default library User-Agents (e.g.,
python-requests/x.x.xoraiohttp/x.x.x).From my experience, many modern WAFs (Web Application Firewalls) and APIs block these default agents, resulting in silent failures or 403 Forbidden errors. This makes our data collection less reliable.
The Proposal
I would like to propose standardizing our network identity across the project.
1. Centralize the Configuration
Add a constant in
vulnerablecode/settings.py:Beta Was this translation helpful? Give feedback.
All reactions