The Dreaded Ten (ie, aggressively deleting feed item)

yardena · October 6, 2018, 7:25pm

So, I went to read this feed today: https://newsblur.com/site/6885443/awol-the-ancient-world-online

The first thing I notice is, there are 10 unread items. For the past few months, this has become the “Dreaded Ten” for me. One would hope it means “there are exactly 10 unread items”, but nope, most of the time it means “Newsblur has automatically marked everything as read except for the most recent 10, and possibly deleted the older stories altogether”.

This list of “Dreaded Ten” feeds has seemed to grow for months now. At first it seemed reasonable, because it was only very busy feeds nobody could possibly keep up with, like Al Jazeera [1] or Map Porn [2] … but then it started happening to more typical feeds such as the Greenlining Institute [3], Richmond Confidential [4], Dawn News Blog [5] and Open Democracy [6].

These are not very busy feeds, so the impact is very disproportionate to whatever performance gains you’re getting from this. Take Dawn News for example: the oldest story in Newsblur is from September 24. This can’t be possible - they update every few days. If I switch to “all stories” the oldest one is STILL from September 24.

So, I thought, maybe it’s based on interaction. Maybe I don’t interact with them enough. Maybe I pressed “mark all as read” just a few too many times after only skimming the titles. I’d still object to this metric since for me, skimming titles is a valid and important way of interacting with a feed, but at least I’d understand it.

But today I find my AWOL feed has joined the Dreaded Ten, and now I’m truly baffled. I have been CONSISTENTLY opening items from this feed in new tabs, and saving them. I currently have 75 items saved in this folder. But as of today, the oldest item in this feed is from Oct 3, three days ago! How many items have I missed? Who knows.

Could you please address the algorithms involved here, perhaps even point me to the code controlling it and the git commits where you started enforcing it? Thanks for your help.

[1] https://newsblur.com/site/5418039/aljazeeraenglishs-youtube-videos
[2] https://newsblur.com/site/7071267/inline-reddit-mapporn
[3] https://newsblur.com/site/656747/the-greenlining-institute
[4] https://newsblur.com/site/799010/richmond-confidential
[5] https://newsblur.com/site/7101369/the-dawn-news-blogs
[6] https://newsblur.com/site/5612633/opendemocracy

yardena · November 6, 2018, 7:00pm

No response at all? If I’m going about this wrong, I’d like to know. Thank you.

yardena · February 13, 2019, 10:59pm

Perhaps the most obnoxious illustration of this problem is when I get errors in my android app such as “could not find story to save” - because in the time between me opening the app and me pressing save, the app has synced and learned that Newsblur has deleted the story for being … to old? Too what?

Try it yourself. Subscribe to, say, the Frontiers in Neurology feed, wait a while, then open it in your android app and try to save the first item. Do you see “could not find story to save” also? https://newsblur.com/site/6759690/frontiers-in-neurology-new-and-recent-articles

yardena · May 14, 2019, 4:44am

And now my “Feminism in India” feed has joined the Dreaded Ten! https://newsblur.com/site/6571287/feminism-in-india

I love this feed. It has wonderful content, I check it dutifully once a month to save items I’m interested in. I’ve done this for years. But this time around, there are only ten items for me to read, and the oldest one is from May 9 - four days ago!

What is going on here? If I can’t use a newsreader to read feeds once a month, what’s the point of having one at all? And why have I gotten no responses to this issue after six months?

samuelclay · May 21, 2020, 9:09pm

Hi, sorry for not writing back. You wrote a very good bug report and I try to hit everything, but I’m now back on the NewsBlur beat and want to fix these kinds of issues.

Quick question, are you a free or premium user? Premium users should have a higher number of stories saved. However, if you are premium and only 10 stories are being saved, that’s because the feed must not be updated very often. If that is the case, then I’d be happy to figure out why it’s misbehaving.

Also, anytime a feed is doing something odd, right-click it on the web and go to Statistics. That dialog should be pretty explicit about what’s going on and why it isn’t fetching as often as you expect. I realize this is about the story_cutoff (# of stories saved) being too low, but that’s related to how many people read the feed and are active on the site.

Here’s the code that handles this:

github.com

samuelclay/NewsBlur/blob/master/apps/rss_feeds/models.py#L1482-L1527


def story_cutoff(self):
    cutoff = 500
    if self.active_subscribers <= 0:
        cutoff = 25
    elif self.active_premium_subscribers < 1:
        cutoff = 100
    elif self.active_premium_subscribers <= 2:
        cutoff = 200
    elif self.active_premium_subscribers <= 5:
        cutoff = 300
    elif self.active_premium_subscribers <= 10:
        cutoff = 350
    elif self.active_premium_subscribers <= 15:
        cutoff = 400
    elif self.active_premium_subscribers <= 20:
        cutoff = 450
        
    if self.active_subscribers and self.average_stories_per_month < 5 and self.stories_last_month < 5:
        cutoff /= 2
    if self.active_premium_subscribers <= 1 and self.average_stories_per_month <= 1 and self.stories_last_month <= 1:

This file has been truncated. show original

Looking at it I see that there’s a 10 for users who haven’t read anything in the last month. So if you’re using the official Android app, you should be OK. But to double check, try reading that feed on the web and then the number should grow. Then if it goes back to the dreaded ten, you’ll know why.

I’m happy to help and if I don’t respond here, you can always email me: samuel@newsblur.com. Feel free to ping here and over email until this gets fixed.

yardena · June 13, 2020, 1:23am

Thanks Samuel for your reply! I am indeed a premium user (since 2013), although I have been relatively inactive for the past year.

Back when I wrote this post, I was very active on both the website and the official Android app, so read_stories_last_month should never have been zero for me, unless there was a bug.

samuelclay · June 16, 2020, 6:22pm

I noticed something strange. In this feed - https://newsblur.com/site/200856/opendemocracy-rss - when I took a look there were only 10 stories, but when I subscribed to it I get hundreds. How many stories do you see?

yardena · June 17, 2020, 1:16am

I see hundreds in your link, but only 10 in mine. Seems there are 2 different feed IDs with the same fetch url, exactly 185 subscribers, but different results. Very strange!

My link (5612633) is also more outdated, with the most recent of the 10 being yesterday. Whereas yours (200856) has items from today.

samuelclay · June 17, 2020, 1:58pm

Ah-ha, you have a branched feed, which is a dupe feed that differs by the feed link (not the feed url, but the link to the original website). I suggest deleting and re-subscribing to that feed to fix it.

yardena · June 20, 2020, 11:05pm

Thanks. Solved that mystery, at least