I run a SearXNG instance and rate limiting has never been a issue until now. so i wanted to ask what the cheapest and most privacy respecting vpn/proxys are to use.
I run a SearXNG instance and rate limiting has never been a issue until now. so i wanted to ask what the cheapest and most privacy respecting vpn/proxys are to use.
On the vpn/proxy idk, I think the bots would pass through it.
There are some nginx blocklists like the ultimate bad bot blocker, I could share my Cloudflare rules if you want it, as these are rules, it can be abstracted to anywhere.
Nginx also has support for rate limiting built in.
On the topic of blocking, I block useragents starting with
Mozilla/5.0that are using HTTP/1.X, since all modern browsers default to HTTP/2.0 and anything else is usually always bad bots. You can also return 426 with theUpgrade: h2cheader to let some older browsers know to use HTTP/2.0.