Filter out duplicates?

For reasons I won’t bore you with, I monitor a lot of local newspapers’ feeds, and I often see the same story posted several times, due to the consolidation of media ownership. Just to clarify, I don’t just mean different papers reporting on the same event, I mean the exact same report by the same reporter being run by several different papers owned by the same company. It would be great if there was a way to filter out these duplicates, to make my feed less cluttered.


I put most of my feeds through Yahoo Pipes. Yahoo pipes takes in RSS feeds, lets you manipulate them and then spits back out a new RSS feed that you can add to reader. I use this to consolidate many feeds into a single stream, sort and de-dup it. It has a “Unique” filter that could be based on the post title or on the post url.