How do other RSS services handle rate limiting or bans?

question
#1

I’ve noticed in the responses to issues the fix is to contact the publisher and try and get un-banned. How do other services prevent a ban in the first place? If Newsblur is slamming the site with requests I can see how this might raise a red flag. How do other services prevent this?

#2

We have exponential backoff in the case of failing sites, so when we do get blocked, we repeatedly fetch from the site with a greater and greater delay until we’re able to fetch again. Mostly this works, but sometimes it requires manual intervention on the part of the publisher.

2 Likes
#3

Yep always good to back off :+1:

#4

Also wanted to add, that I’m noticing that sometimes Reddit returns a response code that isn’t a 429. Is Newsblur backing off in cases where the response code isn’t a 429?