- Unsupervised Learning
- Posts
- How I Collect and Display Interesting Content I Find Online
How I Collect and Display Interesting Content I Find Online
[ NOTE: Yahoo Pipes has been murdered, so a lot of this post is now stale. ]
Like many others in technology I spend a good amount of time reading news and marking content that I may want to come back to later or share with others. Given the number of sources I consume it’s difficult to do this in a centralized way, and for this reason I pieced together the following.
The system
The center of the system is Yahoo! Pipes, which I just think is brilliant. It allows you to collect content from all sorts of sources and aggregate it into a single feed. My feed for the project in question is my Discovered Feed. This feed consumes three sources:
My Pocket stream
My Reddit upvotes
My Hacker upvotes
As the diagram shows above, these are used in the following ways:
My main newsreader is Feedly, and when I like something there I send it to Pocket, and since my Pocket stream is pulled into the Yahoo! Pipe Discovered Feed, I’m good there.
I have customized my Reddit content fairly extensively, so it’s a great source of information for me. Anything I upvote there comes to the Yahoo! Pipe Discovered Feed as well (your upvoted items have their own feed).
For Hacker News I had to add an extra step and go through Page2RSS. So I took my voted up content page, put that into Page2RSS, grabbed that feed, and put it into my Yahoo! Pipe.
The result is that for my main three sources for content–Feedly, Reddit, and Hacker News, I can quickly mark that I like something and have it added to my one Discovered feed.
Displaying the content
On my website’s homepage (danielmiessler.com), I show two main streams–content I created myself, and content I’ve found elsewhere. As you can probably guess, the content I’ve found elsewhere is a pull from my Discovered Feed from Yahoo! Pipes.
I think this site concept is superb, as it provides a flavor of interesting content for a given person without conflating what they’re recommending from external sources and what they’ve created themselves.
Two people that get this are Dave Winer and the late Aaron Swartz. Both of their blogs have clear distinctions between link rolls, blog rolls, and their own content.
Challenges
One downside of the system, however, is that it adds complexity to your RSS feeds. Are users supposed to follow your blog feed or your discovered feed? Oh, both? Where do you get those feeds? Now you’ve just given them work.
I’ve addressed this by having a separate /syndication page that has various feeds broken down, but this could actually make it worse by giving people options that pull them from your main feed.
For the time being, however, I’ve chosen to accept that risk in hopes that the increased granularity will be worth it.
Anyway, I just thought I’d share my enthusiasm for aggregating discovered content in this way, and the idea of separating it when you share it with others.