Yahoo Pipes Podcast by Fast Wonder

Using Yahoo Pipes to Filter Results for Iterasi PositivePress

dawn@fastwonder.com (Dawn Foster) — Thu, 20 Aug 2009 14:30:27 +0000

I’ve been beta testing Iterasi’s PositivePress for a while and was excited to finally see it released to the public! It’s a great way to track and archive blog posts and articles from across the web while creating reports with links to certain articles that can be delivered to clients or friends. CNET did a great job of covering the release, so I won’t rehash it here. I encourage you to read the CNET article for more information about PositivePress.

Since PositivePress uses RSS feeds as input, I decided to do a quick video showing you how to use Yahoo Pipes to filter results from your RSS feed to make sure that you are only archiving and reporting on the most relevant results. This video shows you how to take a list of feeds from a CSV file, fetch the results, and filter them for a set of keywords. The end result is fed into PositivePress and a new report is generated from the filtered results.

If you haven’t already watched the rest of the Yahoo Pipes Video series, you might benefit from watching these other videos first. They provide a little more background on the concepts used in this Yahoo Pipe.

This Yahoo Pipes video series is also available as a podcast on iTunes, and as a separate feed.

You can watch the embedded video above, but I recommend downloading the higher resolution Quicktime file (30 MB) to watch. The quality will be much better than the flash version above.

More Details on the Pipe:

The Demo Pipe. A copy of the Using Yahoo Pipes with Iterasi PositivePress pipe click “View Source” to see the modules.
Fetch CSV Module. Enter the URL of a CSV file. Here is the example we used in the video.
Loop module with Fetch Feed. Loops through each element in the CSV file and fetches the feed associated with the item.
Filter Module (Permit). Filter by a couple of keywords to permit only the results that match at least one of your keywords
Filter Module (Block). Filter out any results that match undesired keywords.
Sort Module. Sort by date in descending order to make sure things are sorted in a logical manner.
Pipe Output. The final module in every Yahoo Pipe.

I’ve created many Yahoo Pipes, and most of them have been published on my Yahoo Pipes and RSS Hacks page where you can also learn more about my Yahoo Pipes Training courses.

Keyword CSV Files and Searching: 2 Minute Yahoo Pipes Demo

dawn@fastwonder.com (Dawn Foster) — Fri, 01 May 2009 19:08:46 +0000

It’s been a while since I’ve posted a 2 minute Yahoo Pipes video, but I had an opportunity to do a guest post on the Yahoo Developer Network blog (yay!), and I thought that a 2 minute screencast would be the best way to demonstrate my pipe.

This 2 minute demo takes a series of keywords from a CSV file and uses Yahoo Pipes to run each keyword through various searches. This technique can be very powerful when you search multiple services, but I’ve simplified the process to make it easier to understand the fundamentals of the technique by running the list of keywords only through Twitter search.

Here’s a quote about the technical details of the pipe from the Yahoo Developer Network post:

Technical Details and More Information:

The Demo Pipe. A copy of the Keyword CSV demo pipe – click “View Source” to see the modules.

Fetch CSV Module. Enter the URL of the CSV file (make sure that it already exists in a publicly accessible location) along with information about column names and separation characters as needed.

Loop module with URL Builder. Loops through each element in the CSV file and builds a search URL formatted for RSS output. In this case, we used Twitter search.

Loop module with Fetch Feed. Loops through each URL built in the previous step and fetches the feed associated with the item.

Sort Module. Sort by date in descending order to make sure things are sorted in a logical manner.

Pipe Output. The final module in every Yahoo Pipe.

I’ve created many Yahoo Pipes, and most of them have been published on my Yahoo Pipes and RSS Hacks page where you can also learn more about my Yahoo Pipes Training course on May 7th.

Related Fast Wonder Blog posts

Accepting Google Docs Input: 2 Minute Yahoo Pipes Demo

dawn@fastwonder.com (Dawn Foster) — Mon, 23 Feb 2009 20:30:57 +0000

This video shows how to use data from Google Docs as an input into Yahoo Pipes. In a previous video, I showed how to use a list of items in a CSV file. If you haven’t already watched the CSV input demo, you should start with it, since this pipe builds on the previous one. A CSV file is a great choice when you want to fetch a bunch of feeds and change them frequently without having to update your pipe, but a Google Doc is a great choice for when you want to easily update inputs to the pipe without needing to update a file on a webserver.

More Details

The Demo Pipe. A copy of the Using Google Docs Inputs in Yahoo Pipes Demo click “View Source” to see the modules.
Fetch CSV Module. Enter the URL of a Google Spreadsheet that has already been published as a web page and append &output=csv to the end of the URL. Here is the example we used in the video.
Loop module with Fetch Feed. Loops through each element in the CSV file and fetches the feed associated with the item.
Filter Module. Filter by a couple of keywords.
Sort Module. Sort by date in descending order to make sure things are sorted in a logical manner.
Pipe Output. The final module in every Yahoo Pipe.

I’ve created many Yahoo Pipes, and most of them have been published on my Yahoo Pipes and RSS Hacks page where you can also learn more about my Yahoo Pipes Training courses.

Related Fast Wonder Blog posts

Using Location Data: 2 Minute Yahoo Pipes Video Demo

dawn@fastwonder.com (Dawn Foster) — Mon, 09 Feb 2009 12:22:21 +0000

Yahoo Pipes can make it easier to work with location data (latitude / longitude) across multiple RSS feeds, even when the original feeds have different ways of specifying the location data. In this demo, I’ll show you how to use the Location Extracter module to normalize the data and then filter on lat / long data to get results only within a certain area.

More Details

The Demo Pipe. A copy of the Using Location Data: 2 Minute Yahoo Pipes Demo pipe click “View Source” to see the modules.
Fetch Feed. In this example, we will use a feed from Shizzow containing information about places and a feed from the USGS with earthquake data. Both store their lat / long data in different ways within the RSS Feed.
Location extractor. This module is atypical, since it takes no parameters and has no user configurable elements. It simply looks for lat / long data and saves what it finds as y:location.lat and y:location.lon.
Sort. Sort by date (descending) to make sure that the output from your various source feeds appears in sorted order.
Filter. We then filter the source feeds to include only the entries located in the western part of the United States (25 < latitude < 50 and -130 < longitude < -115).
Pipe Output. The final module in every Yahoo Pipe.

I’ve created many Yahoo Pipes, and most of them have been published on my Yahoo Pipes and RSS Hacks page where you can also learn more about my Yahoo Pipes Training courses.

Related Fast Wonder Blog posts

YQL (Yahoo Query Language): 2 Minute Yahoo Pipes Video Demo

dawn@fastwonder.com (Dawn Foster) — Mon, 02 Feb 2009 01:41:00 +0000

After learning that Yahoo Pipes introduced a new module called YQL (Yahoo Query Language) last week, I knew that I had to find a way to work it into one of my 2 minute videos. YQL uses an SQL-like syntax for more powerful and flexible inputs into Yahoo Pipes; however, working it into a 2 minute demo was quite challenging, since queries can be complicated. I cheated a little by glossing over the query language and focusing on how to use YQL effectively within Pipes. Honestly, I’m never going to teach someone to use an SQL syntax without extensive training, so this 2 minute demo is really geared toward people with database experience or programming backgrounds. I also did a more complicated version of this pipe with a more extensive explanation that you might be interested in reading to get more details about using YQL within Pipes.

More Details

The Demo Pipe. A copy of the YQL: 2 Minute Yahoo Pipes Demo pipe click “View Source” to see the modules.
YQL. The query:
select * from flickr.photos.info(20) where photo_id in (select id from flickr.photos.search(20) where tags = “igniteportland”).
Selects 20 photos from Flickr with the tag “igniteportland”. flickr.photos.search is used to find the images, but it doesn’t provide much information about the images, so I ran it through flickr.photos.info to pull in more data. If you omit the (20), YQL returns 10 results by default, and you can set this number to something appropriate for your needs.
Rename. Maps item.urls.url.content into the more standard item.link, which is recognized by RSS readers and the Yahoo Pipes output for clickable links back to the original content.
Pipe Output. The final module in every Yahoo Pipe.

I’ve created many Yahoo Pipes, and most of them have been published on my Yahoo Pipes and RSS Hacks page where you can also learn more about my Yahoo Pipes Training courses.

Related Fast Wonder Blog posts

Filter By Date: A 2 Minute Yahoo Pipes Demo

dawn@fastwonder.com (Dawn Foster) — Mon, 26 Jan 2009 12:16:55 +0000

This 2 minute Yahoo Pipes demos shows how to filter by date using the filter module combined with the date builder module to find all of the posts in the past 2 days from our feeds. If you haven’t already watched the 2 Minute Yahoo Pipes Introductory Demo, I strongly suggest that you watch it. This demo builds on the basic structure developed in the introductory demo with the same feeds from ReadWriteWeb and GigaOM that we used in the original demo.

More Details

The Demo Pipe. A copy of the Filter By Date Pipe click “View Source” to see the modules.
Fetch Feed. Fetches the elements from 2 feeds: ReadWriteWeb and GigaOM.
Date Builder Module. Defines the date parameters for filters. In this case, I’ve filtered on ‘2 days ago’
Filter Module. Filter by blocking any item where date (item.y:published.utime) is greater than 2 days ago.
Sort Module. Sort by date in descending order to make sure things are sorted in a logical manner.
Pipe Output. The final module in every Yahoo Pipe.

I’ve created many Yahoo Pipes, and most of them have been published on my Yahoo Pipes and RSS Hacks page where you can also learn more about my Yahoo Pipes Training courses.

Related Fast Wonder Blog posts

Pipes within Pipes: A 2 Minute Yahoo Pipes Video Demo

dawn@fastwonder.com (Dawn Foster) — Mon, 19 Jan 2009 12:18:18 +0000

We’ve talked about many different uses for Yahoo Pipes in previous 2 minute Yahoo Pipes demos, but you can also use most existing Yahoo Pipes within other pipes. In this demo, we’ll use a couple of pipes I’ve previously created as the input into a new Yahoo Pipe, and we’ll introduce the union operator as a way to pull several inputs together.

More Details

The Demo Pipe. A copy of the Pipes within Pipes demo pipe click “View Source” to see the modules.
Union Module. Pull several inputs together into a single output.
Unique Module. Use this to prevent duplicate results when multiple search queries might return the same results by filtering on item.link.
Sort Module. Sort by date in descending order to make sure things are sorted in a logical manner.
Pipe Output. The final module in every Yahoo Pipe.

I’ve created many Yahoo Pipes, and most of them have been published on my Yahoo Pipes and RSS Hacks page where you can also learn more about my Yahoo Pipes Training courses.

Related Fast Wonder Blog posts

Using PostRank: 2 Minute Yahoo Pipes Demo

dawn@fastwonder.com (Dawn Foster) — Mon, 12 Jan 2009 15:35:25 +0000

Yahoo Pipes can also be used with other services, like PostRank, for example. PostRank, originally called AideRSS, is a service that rates the posts on any blog by popularity using comments, links, bookmarks, tweets, and other measurements to give each post a rank from 1 – 10. Luckily for us, PostRank conveniently stores the rank right in the RSS feed allowing us to access it from Yahoo Pipes. In our last Yahoo Pipes demo, I showed you how to Modify RSS Feeds to work better for your purposes. I recommend watching the 2 minute demo about modifying RSS feeds if you haven’t already, since we will use those concepts again in this pipe.

More Details

The Demo Pipe. A copy of the Using PostRank demo pipe – click “View Source” to see the modules.
PostRank. Using the 2 feeds that we’ve been using for most of these demos, ReadWriteWeb and GigaOM, we analyze each of them using PostRank, filter by the best posts, and copy the RSS feed for the output.
Fetch Feed. Fetches the elements from 2 PostRank feeds for ReadWriteWeb and GigaOM from the previous step.
Sort Module. Sort by PostRank in descending order to keep the posts with the highest rank at the top.
Loop module with String Builder Module. Loops through each element in the feed and builds a string with item.postrank:postrank: item.title. This string is assigned back into item.title.
Pipe Output. The final module in every Yahoo Pipe.

I’ve created many Yahoo Pipes, and most of them have been published on my Yahoo Pipes and RSS Hacks page where you can also learn more about my Yahoo Pipes Training courses.

Related Fast Wonder Blog posts

Modifying RSS Feeds: 2 Minute Yahoo Pipes Video Demo

dawn@fastwonder.com (Dawn Foster) — Mon, 05 Jan 2009 11:12:26 +0000

While many people use Yahoo Pipes to filter RSS feeds, Pipes can also be used to modify RSS feeds to work better for your purposes. In this example, we will add the author name to the beginning of the title to make it easier to see the author without opening the item in your rss reader. I recommend watching the Introduction to Yahoo Pipes: 2 minute Yahoo Pipes Video Demo if you haven’t already, since we’re going use the basic pipe developed in that demo using fetch feed and sort modules, but without the filter module.

More Details

The Demo Pipe. A copy of the Modifying RSS Feeds demo pipe click “View Source” to see the modules.
Fetch Feed. Fetches the elements from 2 feeds: ReadWriteWeb and GigaOM.
Sort Module. Sort by date in descending order to make sure things are sorted in a logical manner.
Loop module with String Builder Module. Loops through each element in the feed and builds a string with item.author: item.title. This string is assigned back into item.title.
Pipe Output. The final module in every Yahoo Pipe.

I’ve created many Yahoo Pipes, and most of them have been published on my Yahoo Pipes and RSS Hacks page where you can also learn more about my Yahoo Pipes Training courses.

Related Fast Wonder Blog posts

CSV File Inputs: 2 Minute Yahoo Pipes Video Demo

dawn@fastwonder.com (Dawn Foster) — Mon, 29 Dec 2008 21:54:36 +0000

This video shows how to get input from a list of items in a CSV file, and it introduces the loop module. A CSV file is a great choice when you want to fetch a bunch of feeds and change them frequently without having to update your pipe. If you haven’t already watched the 2 Minute Yahoo Pipes Introductory Demo, I strongly suggest that you watch it. This demo builds on the basic structure developed in the introductory demo. The CSV file I’m using also contains the same feeds from ReadWriteWeb and GigaOM that we used in the original demo plus a few others.

More Details

The Demo Pipe. A copy of the CSV File Input in Yahoo Pipes Demo click “View Source” to see the modules.
Fetch CSV Module. Enter the URL of the CSV file (make sure that it already exists in a publicly accessible location) along with information about column names and separation characters as needed.
Loop module with Fetch Feed. Loops through each element in the CSV file and fetches the feed associated with the item.
Filter Module. Filter by a couple of keywords.
Sort Module. Sort by date in descending order to make sure things are sorted in a logical manner.
Pipe Output. The final module in every Yahoo Pipe.

I’ve created many Yahoo Pipes, and most of them have been published on my Yahoo Pipes and RSS Hacks page where you can also learn more about my Yahoo Pipes Training courses.

Related Fast Wonder Blog posts