Feed Retrieval Problems: 403 Errors

problem
21

I wrote to Samuel directly yesterday, but haven’t yet received any response. I hope he’s fighting this issue.

I’ve contacted a few of the admins of pages that cause trouble for me and they claim they don’t block news readers, even if they hammer to their sites to get the latest feed. At the same time I must admit none of them responded directly to my question to whitelist NB in their Cloudflate environment…

Since yesterday I’ve been testing other RSS readers and they don’t seem to suffer from this issue.

I’ll switch if that’s unresolved. I’ll miss the premium archive feature, but there are similar solutions available with competition (though I need to say, they are not that thorough).

22

It would be great if you could follow up with those publishers, cc’ing me, and ask if they could double check if they are enforcing a bot blocking on cloudflare, since this is a significant enough issue that I’d love to resolve it once and for all. It’s likely that the cloudflare bot blocking heuristic is blocking newsblur.

It’s possible that I could spin up another task server that only takes 403’s sites and attempts to fetch them to get around the blocked op restriction.

23

Yeah this is probably it. I’ve been doing some extensive research on Cloudflare’s heuristics, and I’ve seen cases where it blocks a reader or even a search engine, even though it’s been approved by it’s verified bots program.

It’s a shame because having RSS readers like Newsblur going to every single website owner to try to get them to add a custom WAF rule exception is just not feasible.

24

I’ll probably be adding a special 403 feed fetcher soon, since i agree, it isn’t tenable to ask each individual publisher to whitelist newsblur

25

Sure let me know if you need any help. It should probably use residential, data center, or mobile IPs because anything else cloudflare will likely block as well.

I think you’ve already applied and got Newsblur approved as a verified bot right? Out of curiosity, did they send you any notification or verification email when approved?

26

Yeah NewsBlur has been an approved bot for a long while, but we switched servers earlier this year and that’s when the trouble started. I’m going to be spinning up a couple servers on the old hosting provider and seeing if that helps. It’ll be a week before I can get to this though, but it’s my higher priority right now.

27

Open rss is cracking down on cloudflare websites that are causing problems for rss readers

28

I’m working on a fix, it’ll take a few more days I think.

29

Thanks for your efforts on this, Sam!

I see that four of the sites I reported (NewsBlur, NewsBlur, NewsBlur and NewsBlur now retrieve ok if I do the retrieval manually (which is ok by me). Will see how it goes…

NewsBlur is still giving the 403 error but that’s one I could live without if I had to… Thanks, again!

30

Good news, Open RSS publishes a post addressing this issue and a Cloudflare employee responded on the Hacker News thread: Using Cloudflare on your website could be blocking RSS users | Hacker News

So we might see a Cloudflare-wide resolution soon.

2 Likes
31

Now that is good news!!

32

I’ve just moved from Inoreader to Newsblur. I’m running into the same problem.

These are the two sites that I am currently seeing problems with here but did not see in Inoreader or Feedly. The DATES in the log errors below are mostly before I joined up, so it looks like this problem occurs regularly but periodically. Is this a clue as to what the root of the problem might be? I question if it is actually Cloudflare since it appears that the feeds are working between these error entries?

Fix a misbehaving site
MishTalk 55 subscribers

Option 1:Retry
Retry fetching and parsing

Option 2:Change RSS Feed Address
RSS/XML URL:

https://mishgea.wpcomstaging.com/feed/

Parse this RSS/XML Feed
2024-10-18 14:09:35
HTTP Error (403)
2024-10-18 04:56:15
HTTP Error (403)
2024-10-13 19:27:32
HTTP Error (403)
2024-10-09 20:39:06
HTTP Error (403)
2024-10-05 16:46:28
HTTP Error (403)
2024-10-01 15:18:56
HTTP Error (403)
2024-09-27 19:13:07
HTTP Error (403)
2024-09-23 22:33:47
HTTP Error (403)
2024-09-20 05:47:31
HTTP Error (403)
2024-09-17 05:53:56
HTTP Error (403)
2024-09-13 22:14:28
HTTP Error (403)
2024-09-11 03:49:11
HTTP Error (403)
2024-09-08 17:51:00
HTTP Error (403)
2024-09-06 09:17:12
HTTP Error (403)
2024-09-04 04:08:27
HTTP Error (403)
2024-09-01 07:46:18
HTTP Error (403)
2024-08-30 15:20:39
HTTP Error (403)
2024-08-28 18:41:15
HTTP Error (403)
2024-08-27 03:18:02
HTTP Error (403)
2024-08-25 17:44:57
HTTP Error (403)
2024-08-24 12:23:25
HTTP Error (403)
2024-08-23 15:52:22
HTTP Error (403)
2024-08-22 21:12:02
HTTP Error (403)
2024-08-22 08:57:24
HTTP Error (403)
2024-08-21 22:57:06
HTTP Error (403)

Recommended Option 3:Change Website Address
Website URL: not allowed to post this link

Fetch Feed From Website
2024-10-18 14:09:36
OK (200)
2024-10-18 04:56:15
OK (200)
2024-08-24 12:23:25
OK (200)
2024-08-21 14:45:07
OK (200)
2024-08-21 09:59:55
OK (200)

Option 4:Just Delete This Feed

Fix a misbehaving site
The Unz Review:109 subscribers

Option 1:Retry
Retry fetching and parsing

Option 2:Change RSS Feed Address
RSS/XML URL:
https://www.unz.com/xfeed/rss/all/

Parse this RSS/XML Feed
2024-10-18 14:22:48
HTTP Error (403)
2024-10-18 14:03:01
HTTP Error (403)
2024-10-18 14:02:45
HTTP Error (403)
2024-10-18 14:02:09
HTTP Error (403)
2024-10-17 08:35:04
HTTP Error (403)
2024-10-16 00:00:39
HTTP Error (403)
2024-10-14 15:46:24
HTTP Error (403)
2024-10-12 23:54:13
HTTP Error (403)
2024-10-11 12:06:20
HTTP Error (403)
2024-10-10 03:03:48
HTTP Error (403)
2024-10-08 18:52:26
HTTP Error (403)
2024-10-07 08:33:41
HTTP Error (403)
2024-10-05 17:12:32
HTTP Error (403)
2024-10-04 08:16:28
HTTP Error (403)
2024-10-02 20:28:21
HTTP Error (403)
2024-10-01 08:40:58
HTTP Error (403)
2024-09-30 14:54:47
HTTP Error (403)
2024-09-29 05:39:40
HTTP Error (403)
2024-09-27 17:03:33
HTTP Error (403)
2024-09-26 06:30:29
HTTP Error (403)
2024-09-24 20:26:27
HTTP Error (403)
2024-09-23 10:02:25
HTTP Error (403)
2024-09-22 01:30:37
HTTP Error (403)
2024-09-20 15:50:46
HTTP Error (403)
2024-09-19 06:55:50
HTTP Error (403)
Recommended

Option 3:Change Website Address
Website URL: Not allowed to post this link

Fetch Feed From Website
2024-10-18 14:22:50
OK (200)
2024-10-18 14:03:03
OK (200)
2024-10-18 14:02:11
OK (200)
2024-10-04 08:16:28
OK (200)
2024-10-01 08:41:00
OK (200)

Option 4:Just Delete This Feed

33
1 Like
34

Haven’t seen similar 403 errors at Feedly. Sorry, can’t continue with Newsblur.

35

The errors are there, too…

36

Any updates on this, SamClay?

37

Am guessing there is not…

38

I just reached out to another contact at Cloudflare. Nothing guaranteed, but I’m working on this.

1 Like
39

I’ll be transparent here - for the moment I’ve changed to another RSS client (starting with I), and… It’s not even half that good as NB, UX is far from what NB offers (or used to offer). @samuelclay, we need you to fix your product, thumbs up here!

1 Like
40

How did it end up?

I’m having 403 issue with https://www.extremetech.com/feed