Problem parsing https://airinsight.com/ feed

problem
#1

https://airinsight.com/ now redirects their feed to http://feeds.specificfeeds.com/airinsight. That seems to contain a well-formed RSS feed (http://www.feedvalidator.org/check.cgi?url=http%3A%2F%2Ffeeds.specificfeeds.com%2Fairinsight), but NewsBlur is not recognizing it as such (http://newsblur.com/site/6720830/airinsight has no articles).

The specificfeeds.com appears to be a FeedBurner like service, and they seem to do some user agent sniffing (that URL shows me RSS when used with curl, but an HTML page in Chrome). Perhaps NewsBlur’s user agent is getting sent down the HTML path for some reason. 

2 Likes
#2

Yep, that would be correct, as NewsBlur uses a slightly fake user agent to get around the inverse of this problem: when a publisher checks for a browser like user agent to show the feed. 

I anticipated something like this and have a failsafe for when a feed throws a > 400 status code and I re-check the feed with a fake user agent. 

https://github.com/samuelclay/NewsBlur/blob/master/utils/feed_fetcher.py#L126-L129 and https://github.com/samuelclay/NewsBlur/blob/master/apps/rss_feeds/models.py#L1074-L1077

But as you can see, the fake user agent also has the browser issue. In fact, we’re not even getting to this point because they’re just serving an HTML page with a 200 status code, so I don’t have the opportunity to change the user agent.

I suppose I could check if the content_type is text/html and then perhaps kill the user agent entirely, but that’s awfully long winded just to handle a site’s broken implementation. They should be made aware of this issue, ideally by updating the RSS feed to point to a unique location.