i run a code which connects to the newbslur api with my credentials and retrieves page by page all my saved stories, in the past week i’ve been getting error code 429 when getting to about page 48, how can i overcome this ?
I guess you get rate limited
The HTTP 429 Too Many Requests client error response status code indicates the client has sent too many requests in a given amount of time. This mechanism of asking the client to slow down the rate of requests is commonly called “rate limiting”.
Actually, yes but , after looking at the commits on github , it looks a rate limit mechanism has been implemented
Yep, I added ratelimits to saved stories. I suggest just going under the rate limits and you’ll be fine.
As you can see from the commit, I set it to 50 requests in 5 minutes, which should be easy enough to stay under: Ratelimit starred stories · samuelclay/NewsBlur@54c60c1 · GitHub
Thanks, i understand that this is a required step to stop all sorts of network vulnerabilities and security aspects.
But this causes my export of saved stories process to become very slow, is there any way to speed it up or bulk export ?
at the moment i querying page by page and if someone has uhhm alot of saved stories then this process can be very time consuming
What are you running that you need to export so many saved stories?
Well, i have a lot of (20k) saved stories , don’t have enough time to read them all.
So i’m trying to export them and back them up , just in case.
at the moment i’m forced to visit each page and extract them from the page and save it ,
I’m on the same boat as you, have (sadly) 35k saved stories and worry that they will get lost before I get to go thru them.
If there were a way to increase the amount of items per page (or reduce the amount of data per Item, i basically would only need the url) that would help to run the backup
All you need to do is heed the rate limit, which is 50 requests per 5 minutes. That should get you everything pretty quickly.