Swiftly.org | Filtering Realtime Datastreams

Moving on from SwiftRiver and Ushahidi

Thu, 30 Jun 2011 22:12:06 -0400

Directing the SwiftRiver Initiative to where it is today has been fantastic. The project has matured a lot, in no small part to the volunteer developers and thinkers who contributed over the past two years. My focus with Swift from the beginning was to construct a platform that attempts to…

Some Reflections on Sweeper from N.E.A.T. Nigeria

Sat, 25 Jun 2011 11:55:55 -0400

In April we were contacted by a group out of Georgia Tech, M.I.T. and student on the ground in Nigeria about the, then, upcoming elections. This group of individuals, together working as N.E.A.T. (the Nigerian Election Aggregation Team) wanted to run a campaign that mashed up data from several different Ushahidi deployments, Twitter and other sources, displaying them in their own Ushahidi deployment. They ended up writing a lot of custom code but this was the first ‘stress test’ of the SwiftRiver platform and our Sweeper application to date.

The following is a review of the N.E.A.T. team’s experiences with Sweeper. It was written by Thomas Smyth from Georgia Tech just after their election project was complete on May 2:

What Sweeper Did Well

Quick setup: Jon had our instance up in running in what seemed like a heartbeat. This was much appreciated.

Reliability: Sweeper stayed up pretty reliably as long as I didn’t break it!

Auto-Tagging: This feature was pretty neat and our system used Sweeper’s tags for meta-analysis.

Support: Matt was available consistently for in-depth help and scheming. We appreciated this.

Issues With Sweeper

Bugginess: Several major bugs were encountered, e.g. the duplication service. But this is to be expected for a young project.

Twitter lag: Twitter updates weren’t showing up for many minutes. Since Twitter was our main source of timely information, this was a big problem. We ended up implementing our own scraper using Twitter’s stream API, which has worked brilliantly. Matt and I have discussed this.

Searching: Sweeper currently doesn’t allow searching of reports, and this was a desired feature which we implemented. We also implemented a ‘saved search’ feature, which turned out to be quite useful. It allows the user to specify a search string (such as “guns or bombs or knives”) to be “tracked”. The system then searches all incoming reports and maintains a time series visualization. This allows a user to see what topics are ‘spiking’. Something like this would fit nicely in the the analytics panel in Sweeper.

Analytics panel: There are a few good things here but the interface could be a lot denser, so that more useful analytics could be added. For instance, top tags could be represented with a compact table rather than a bar chart. Charts should only be used in cases where the visual representation provides a clear benefit. Pie charts are usually unnecessary, etc.

Geolocation problems: The automatic geolocation service was quite dodgy. I didn’t do any actual counting but I’d say upwards of half the results were wrong. I think it’s a difficult thing to do automatically. So much ambiguity, etc. We ended up building a custom solution for geolocation, incorporating polling booth data (120k of them!) from INEC. The system could automatically recognize a polling unit code like 03/04/12/013 in a tweet, and translate that into a geolocation.

Scanning interface: The main interface of sweeper, where users quickly scan through reports and categorize them, could be more efficient. It’s not clear why each report needs to take up so much space, and why the interface doesn’t scale to fit the whole screen. The animations were also somewhat disorienting. In our system, we tried a system where users ‘checked out’ a batch of 10 reports and quickly scanned them in a compact table format, marking relevant ones with a checkbox. This seemed to work nicely, and didn’t require (I think) as many requests to the server. In general, I think Sweeper’s interface could be tightened a lot. Users are more likely to be experienced, frequent visitors, rather than occasional ones (I think). Therefore you can make it a little more efficient and specialized than a general purpose website. I think users would appreciate this. I’d be happy to consult further here if there is interest.

Code and documentation: Much of the functionality described above could perhaps have been added to Sweeper. However, we found it hard to get started on adding plugins. The codebase could be better organized so that it is clear where code for different components should go. The code itself could also be cleaner in places. Also, documentation needs to be available. But again, we realize Sweeper is a young project and these things are surely on the TODO list!

That’s all I have for now guys. Let me know if you have any questions. Many thanks for everything. Let’s keep talking!

This is great feedback and some of it we’ve already begin working on, while the rest (both the code and the suggestions) have been added to our roadmap.

(Photo from http://www.uiowa.edu)

Knight News Challenge Grant!

Thu, 23 Jun 2011 10:41:00 -0400

It’s truly an honor to accept a $250,000 grant from the Knight Foundation for the SwiftRiver project! It’s the culmination of a long journey that began in 2008 but evolved in 2010 when I joined the project as (product designer) and later Matthew Griffiths (lead developer).

Swift is an open-source initiative who’s goal is to make the process of vetting information more efficient. The project to date has progressed well thanks in no small part to the following people: Matthew Griffiths (so important to this project I mentioned him twice), Ahmed Maawy, Charl Van Neikerk, Heather Ford, Vladimir Ermakov, The Ushahidi team, Omidyar Networks, Chris Blow, Ed Bice, Kaushal Jhalla, Neville Newey, Edmar Ferreira, Pete Warden, Patrick Meier, Anahi Ayala, Ethan Zuckerman, the TED staff, Google’s 2010 Summer of Code Participants (Mang-Git, Soe, Nishith Rastogi), the Guardian’s Activate staff, Product(RED) and many others. This project would be nowhere without you all so thanks for making it happen.

For many of us, this project represents a new way of democratizing access to the tools for understanding and vetting information which is needed by Ushahidi, journalists, and many others.

Open Source Bookmark Curation

Fri, 10 Jun 2011 20:54:00 -0400

With the latest release of Sweeper, you can roll your own bookmarking service. This is really powerful when you start activating plugins like our auto-tagger SiLCC or our our Push plugins which can output all of your bookmarked content as a feed that can be consumed by other applications.

We call this little plugin Quiver. It’s where you manually collect and store information using Sweeper. Essentially it turns Sweeper into a your free and opensource Delicious clone, with all the contextualization and aggregation features that people have come to love it for.

So how does it work? It’s simple! Just download and install any version of Sweeper following the current release of v0.3.2 which can be found here.

Once you’ve done that, go to the ‘sources panel’.

Select ‘Quiver’ from the list.

Drag the bookmarklet to your browser bar.

Done! Sweeper is a tool for the curation of real-time media. Now the things you find interesting can be mashed up with the content you’re aggregating from the web, twitter, email and other feeds! It’s particularly useful for journalists or researchers who need the real-time content, but who want to augment that with their personalized interests and findings.

Get it from Swiftly.org

Algorithms Augmenting Human Decisions

Wed, 08 Jun 2011 11:34:00 -0400

Here’s an update about the SwiftRiver platform from PDF11 which I had the pleasure of speaking at yesterday. My slides are below and here you can find video of my presentation.

Crowdsourcing 102: Mining Real-Time Data

The summation of the talk is that the Swift project has been assigned a very complex and incredibly difficult task: to verify and contextualize data from the mobile and social web. How do we do this, well this seems to be the part that confuses people. It’s not any of our apps, and it’s not any of our individual APIs that we rely upon to do this. It’s the combination of all these things together, as part of one robust algorithm that tries to digitally reconstruct the real-world context, using the features extracted from the content t prioritize and de-prioritize information relevant to that context.

I like to refer to this as folksonomic triage where layers of historic, social, temporal, geospatial and other types of information are layered one another to perform a function, and the system (through a process called active learning) then learns how to improve form the user’s interaction. What this attempts to do is allow the human to give the machine algorithms some insight into the types of content they prefer, and the types of content they dont. A statistical profile of the content features of each type is recorded, with varying degrees of nuance in-between including accounting for bias, crosstalk, irrelevancy and falsehoods.

Some of this happens on the application side, some of it happens on the logic/cloud side of things. This is because it’s very important that user understand that the platform is there to serve them, and not the other way around; algorithms augmenting human decision making. This means we’ve abstracted some elements of the system logic (the elements that everyone needs to re-use over and over again) while the things specific to the use of the platform, are defined in the UI.

Usecases

We’re really excited to have had a number of really amazing partners new and old using the platform. This includes groups like Newsti.ps who are building a ‘people’s newswire’ using the Swift products.

There are also some really big uses that are occurring. For instance this BBC article profiles the PAX system that is using our platform to power a conflict early warning system. They want to index massive amounts of data from around the world and then use that data to spot historic patterns and trends that then can be used to demonstrate confidence in future patterns.

One of our favorite uses of the Swift platform to date was Product (RED)’s use last year to mashup large quantities of social media activity to power their Turn The World (RED) campaign.

There have been many more uses that we can’t talk about yet, but hopefully those become pubic soon.

Some Numbers

There are currently eight different code repositories housing the greater Swift project. Each of these API elements is tackled as if it were a single problem. This includes code for location disambiguation, natural language processing, influence detection, reputation monitoring and duplication filtering. You can find more about them here - http://blog.swiftly.org/post/5788873594/resources-for-developers

These combined repos contain around 150,000 lines of code (not including frameworks like Kohana)
Over 7,000 downloads of Sweeper to date
Which theoretically means at least 7,000 users of our APIs
Sweeper users tend to aggregate thousands of items of content over the life of a deployment which means we’ve taken around 70,000,000 items of unstructured data and done things to it like add location, tags or filtered the duplicates. That’s a very liberal extrapolation, but if gives you a sense of the amount of data we’re dealing with.
As the project moves forward, and all our APIs are finally completed, this number will grow exponentially. With RiverID alone (which tracks the reputation of content and people online) we expect to be indexing over half a billion items of content and actions from the social web alone by the end of the year. That’s just one API, the others will also need to scale on equal terms.

Photo by Fabrice Florin

Building a people's newswire with Newsti.ps

Wed, 08 Jun 2011 10:18:34 -0400

[Guest blog post by Jenka Soderberg, a 2011 Knight Fellow at Stanford University and Evening News Director at KBOO Community Radio in Portland, Oregon. She can be reached at jenka [at] stanford [dot] edu]. This is a cross-post from the Ushahidi blog.

When I first started working on www.Indymedia.org in 2000, I was really excited about the platform it provided: a way for people who witnessed news events to immediately publish text, audio, video and photos to an OPEN newswire. This was unprecedented on the web at that time, and led to an explosion of open multimedia content-posting sites. Since its inception at the World Trade Organization protests in Seattle in 1999, the Independent Media Center expanded into over 200 local sites worldwide, all funneling featured content into the main (global) site www.indymedia.org. In many ways, this could represent the way news organizations operate in the future – but most of the major news companies haven’t caught on to this trend just yet.

I got into the world of journalism because I didn’t trust the media. Time and again, I’d read, hear or watch news stories that were grossly inaccurate, one-sided and oversimplified. So I took seriously the slogan, “Don’t hate the media, BE the media”, and helped launch a bunch of indymedia centers and microradio stations all over the world, always with the hope of giving voice to the voiceless, allowing people to tell their own stories and to share in the narrative that was developing about them without the often-damaging involvement of advertising dollars and managing editors who presume to dumb things down for audiences they believe they have to entertain as well as inform.

Now, with more and more people turning away from traditional media to get their news online (see chart), it seems those audiences, about whom so many assumptions were made by the management of media corporations, are trying to find their own way in the new media world and find the news that they think is important and valuable.

Unfortunately, this often means that people seek out only news sources that confirm and uphold their existing points of view, and may be just as full of inaccuracies, speculation and oversimplification as the news media that they were trying to escape.

How can we get through the mess of misinformation to find the real tips of breaking news events, as they’re happening, and get this information out to as broad an audience as possible?

I’ve been working with a team at Stanford this year to use Ushahidi’s Swiftriver platform, and specifically Sweeper (one of the multiple tools in the Swiftriver toolbox) to try to extract real newstips from the deluge of 140-character texts and tweets, and try to figure out which newstips are real and accurate. Our project description and current newswire is at www.newsti.ps

We’re implementing this in the Occupied Palestinian Territories, an area where many news incidents are under-reported in the US, and others are over-reported, giving US audiences a skewed perspective of the reality on the ground. We’re using the Swiftriver platform to skim the web and twitter for keywords that are then filtered by keyword, location, reputation and duplication and organized into a database. Our reporters in different parts of the Palestinian Territories (the West Bank, Gaza and Jerusalem), can follow up on the most poignant of these tips and verify their accuracy. These reporters have created the International Middle East Media Center (www.imemc.org), currently the most widely-read English-language news site based in the Palestinian Territories.

We’re also working on a way to allow people who witness news events but don’t have the luxury of a smart phone yet (only 2% of cellphone users in the Palestinian Territories have smart phones, and 3G is extremely spotty), to send texts and photos directly into our system as well. For translation of Arabic texts, we’ve solicited the help of the crowdsourced translation team of www.meedan.net.

Like with Indymedia, we think that this work can be an alternative to the mainstream media – although, as always, they are free to use these news stories, it seems unlikely that many will. When news corporations are focused on selling advertising dollars instead of providing accurate news for their audiences, they will continue to go the way of the dinosaurs, as they are doing. Unfortunately what we’re losing right now are lots of good, investigative news reporters who held politicians’ feet to the fire, reported on breaking news events and local issues, investigated wrongdoing by large companies, connected audience members with the stories of people in different circumstances far across the globe, but with whom they could relate due to the strength of the writing and storytelling. What we’re left with right now, to a large extent, are cable news channels whose focus is on entertainment and advertising, and vitriolic talk radio that exuberantly embraces speculation, rumor and misinformation over fact-checked, accurate news reports. On the local news front, AOL’s newest branchild, patch.com, threatens to replace real local reporting with half-hearted, badly-written reports that are unapologetically inaccurate.

Can we get a ‘people’s newswire’ based on eyewitness reports of newsworthy events? I believe we can – if we combine the automation of systems like Swiftriver, the data visualization possibilities of tools like Ushahidi, and the insight of trained reporters who can follow up on potential leads. Heck, if we can do it in the Palestinian Territories, then we can do it anywhere!

The video below is a short presentation about this project. Be sure to check out our website www.newsti.ps for real-time updates during the upcoming humanitarian flotilla to break the siege on the Gaza Strip.

Video of my presentation earlier today at Personal Democracy...

Tue, 07 Jun 2011 14:49:00 -0400

Video of my presentation earlier today at Personal Democracy Forum

Members Only

Wed, 01 Jun 2011 09:15:47 -0400

Introducing Push Plugins

Wed, 25 May 2011 04:24:00 -0400

Anyone pulling from the nightly repo may have noticed a cool new feature for the Swift Core that Ahmed wrote last month, our Push Plugin architecture. This, as well as a number of other features will be released with the next release of Sweeper and the Swift PHP Core.

How Push Plugins Work

Push Plugins allow SwiftRiver applications to acquire content via push (versus pull) commands. For instance, if a user needs an SMS gateway to submit to a Swift app, you no longer need to poll the server for that content, instead, the gateway can tell your app when there’s content by pushing to the application.

This plugin architecture currently supports receiving data through the standard HTTP methods GET and POST.

In addition, this architecture can be extended through Push Plugins to support the injection of any kind of data into the system. For example, the uploading of content from files, or the use of bookmarklets such as the Quiver extension.

How To develop Push Plugins

Locate the Modules/SiSPS/PushParsers folder, this is where you’ll find push parsers. To develop a push parser you will need to do the following:

Create a file named PushParser.php (the class name needs to be the same as the file name).
Needs to be in the namespace Swiftriver\Core\Modules\SiSPS\PushParsers;
Implement the following methods:

PushAndParser($raw_content = null, $post_content = null, $get_content = null)
GetDescription() - This is what gets displayed in the Sweeper UI that describes how the parser works
ReturnType() - Returns the type that describes what the push parser is all about

The second and third methods are implemented for display purposes so that your parser can be displayed correctly in Swift applications.

The first method is where you need to write code to convert the content being received by your parser into the Swift object model, this function should also return the content back to the rest of the SwiftRiver Core once its finished.

Depending on the type of resource your parser is listening out for, it will receive the content in one of the three variables $raw_content, $post_content and $get_content.

Jon’s talk from TED Global 2010 about the evolution of the...

Mon, 23 May 2011 22:39:45 -0400

Jon’s talk from TED Global 2010 about the evolution of the SwiftRiver platform and learning from mistakes in crowd-sourcing.

Resources for Developers

Mon, 23 May 2011 22:31:00 -0400

If you’re a developer looking to build on our open-source stack or simply curious about our work, check out the following links. You can follow our core team on twitter @swiftriver and code commits to our project @swiftdev:

Overview of platform - https://docs.google.com/document/d/1geoJCwS6Z3uOX2XqqdbxP4gMqEcN5W7ES6MHSIZhEUE/edit?hl=en&authkey=CIvby-8C

Dataflow in Swift - http://www.flickr.com/photos/ww4f/sets/72157625922117348/

Dataflow Video - http://vimeo.com/19271268

Sweeper User Guide - https://docs.google.com/document/pub?id=10VPGHnyvBiyr7i2cwqIpDehxOvIDlUgMid3lUBlgCwk

Screenshots of Sweeper - http://www.flickr.com/photos/ww4f/sets/72157624792893922

Info on writing plugins - http://wiki.ushahidi.com/doku.php?id=intro

Monthly Newsletter - http://eepurl.com/de3jc

Our Repos on GitHub
http://github.com/ushahidi/swiftriver
http://github.com/ushahidi/sweeper
http://github.com/ushahidi/reverberations
http://github.com/ushahidi/silcc
http://github.com/ushahidi/geodict
http://github.com/ushahidi/swiftmeme
http://github.com/ushahidi/sicds
http://github.com/ushahidi/riverid

Sweeper V0.3.1 Released - Twitter Streaming and Proxy Support

Fri, 15 Apr 2011 05:49:00 -0400

Download Sweeper V0.3.1 With Support for Twitter Streaming API here

Any of you keen Swiftriver lovers out there will remember the recent launch of our 0.3 version of the Sweeper app. The release went well and we got loads of feedback.

We were able to identify a couple of bugs in the release that were causing headaches for you guys and girls out there trying to set Sweeper up. So we fixed them …

Fixed the mysql table engine bug, thanks to everyone on the Swiftriver google group for finding and helping to fix this!

Fixed the link back to original content in the content source popup.

So why didn’t we give you these fixes a week ago … well we wanted to hold off as we were sooooo close to finished two other pieces of work that it didn’t seem right to give you the bug fixes without a little ‘we’re sorry’ present, so here it comes:

Swiftriver and Sweeper now support collecting content from the Twitter Streaming API. This is a massive win for our software and we know that there will be loads of you who want to check it out. Its still early doors on this at the moment and there are a few things that we haven’t quite got right – for example we cant tag and translate all the content coming from a stream yet – but we didn’t feel that this was reason enough to hold you dudes back. So if you LOVE data and want to see a whole lot of it, download the latest V0.3.1 release of Sweeper and get cracking.
Sweeper can be now be deployed behind a proxy. This is great news for any of you wanting to use Sweeper on your corporate networks and we know you have had issues with this in the past, well they are now truly in the past!!!

We hope that you love this latest mini release as much as we do and as always, we love your comments, questions and criticisms (yes we really do!)

Just a small note for those of you who are already running a version of Sweeper but who want to take advantage of the new features in this release … while Sweeper does not yet officially support upgrades, it is possible. Basically all you have to do is dump the new code over the existing install then run through the installer (http://[where-you-installed-sweeper]/installer) and when you get the ‘create admin user section’, just skip it (just click on ‘6:Proxy Server Setup’ at the top of the page) without entering a new admin password. Hope this helps, and we are working on an official upgrade path to make adoption of the latest versions of Sweeper easier in the future!

So that’s all for now folks, till next time,

Matthew Griffiths,

Director of Platform

SwiftRiver

Download Sweeper V0.3.1 With Support for Twitter Streaming API here

The Sweeper User Guide

Mon, 04 Apr 2011 12:15:31 -0400

Sweeper has been around for just over a year now and although the code is still changing quite rapidly. However, the core platform is now stable enough that we’re releasing this User’s Guide. It’s certainly long overdue. Sweeper is the a free and opensource multimedia curation platform that we’ve been developing over the past few months. It’s a quick and easy way to create a social media monitoring dashboard. You can find the User Manual embedded below or at http://swiftly.org/userguide

So what are some of the things one can perform with the Sweeper application?

Combine many feeds into one: then curate, filter and translate
Count and archive Twitter mentions
Mashup content from Email, Twitter, Blogs, Flickr, News sites, RSS/ATOM etc.
Automate the addition of context to data: Location, priority, influence, reputation, tags etc.
Structure unstructured data
Buffering against an excess of crowdsourced data
Set-up pipes of conditional logic for automating data processes
Collect and manage realtime data from SMS while completely offline
Translate real-time content from social media on the fly
Datamine real-time content aggregated by you or your team
Keyword monitoring from Twitter, Posterous, Blogger, Google News and Wordpress
Native integration with Ushahidi and Crowdmap

Sweeper can be downloaded today from Swiftly.org.

Sweeper User Guide v0.3 View more documents from Ushahidi

Subjectivity, Veracity and Truth

Thu, 24 Mar 2011 13:55:00 -0400

SwiftRiver is constructed from the viewpoint that there are no absolute truths and that what is considered to be factual by most is still highly subjective or biased depending upon context.

Thus, we build tools which allow users to curate their own depiction of a perspective. In the same way that there are more than one newspaper, more than one political party in most countries, more than one religion, even more than one ‘official’ source for occurrences like earthquakes or climate change. We build tools that enable people to convey their confidence in datasets. This in no way implies that data is unbiased.

Swift apps add many layers of context to data as meta-data for making processing data faster, which in turn give our own systems more ammowith which to attempt to understand and auto-mate it’s processing.

What does veracity mean?

Veracity is simply the term we use to represent the baseline of trust that our users have conveyed about content, sources and events. This baseline allows us to do things like recommend related content that is likely be relevant to their view; or in the case of an organization, the collective view.

A number of factors go into creating this profile: how content is organized, how it’s interacted with, how people have behaved in the past, how certain communities feel about it’s members and vice-versa, calculations for real-world phenomena like time and location of an event and so on.

When it comes to verifying data, our tools serve two purposes.

For a public-facing deployment of our apps (including Ushahidi), we offer tools that allow the user to make a case to the public about a particular view. For example, these are the people in the crowd whom they trust, and what those people had to say about an event.
For a non-public facing deployment, some of our apps (like Sweeper) can be used to structure data, conditionally filter and view it. This is useful for setting up automated workflows like ‘pass only approved content, taged with location and these tags, and pass that data over to Ushahdi or some other application’.

In both cases the users, the people behind the deployment, are creating their unique baseline for trust, and therefore are putting forth what they consider to be accurate, or favored, content.

Isn’t this bias?

Yes. Any system operated by a human, and I would go further to say machine created by a human, is subject to some sort of bias.

What does it mean to verify data

In the context of most Ushahidi applications ‘verified’ means corroborated or confirmed by a human. This means on the receiving end, the person ‘verifying’ the data is essentially saying “I’m taking the onus to approve this report because something, or someone, has indicated that this is true.”

Does that mean untrue information can be ‘verified’ either intentionally or accidentally? Yes. All of the terms are highly subjective and people have a number of preconceptions about what these things mean. They are mere abstractions that represent user behavior and intended use.

Verification Levels

It is a mistake to assume that because something is ‘verified’ or has a high veracity score, that it is a fact. What these indications are actually telling viewers and/or the deployer is that this is the baseline for accuracy set forth by an editing body (the deployer). Even verifying reports multiple times by independent participants will not account for human bias or fallibility.

The numbers are there because they are an additional layer of context (readable by machines and humans) allowing the deployer(s) to curate information based on the trust profile they’ve set forth through their interactions.

Likewise, viewing the same data geo-spatially simply implies that this is one community’s understanding of the collected data and what they perceive it to represent. It’s simply a faster way to view data to build up a baseline of ‘favor’ and then use the scores to filter out the content is less likely to fit that profile.

- Jon Gosier, Director of Product

Translating Realtime Social Media

Thu, 24 Mar 2011 01:32:53 -0400

One of the problems a lot of crowdsourcing projects have is that they end up pulling in massive amounts of data from the web, Twitter and other channels from around the world. This means content arrives in many different languages, often languages that the deployer doesn’t speak.

Currently in Sweeper and soon in Ushahidi, users can translate real-time content from one language into another, on the fly, as they receive it. This is done using our Google Translate plugin which currently supports 50+ languages.

For the Sweeper deployment we’re using to monitor the situation in Japan internally, we’re using this feature to monitor events, since we can’t manually translate every single message coming through. We’ve found it a significant timesaver. You can also see below that we’re showing the user what language the message was translated from, or if it’s been translated at all…

Before:

After:

It’s important to understand, that this is machine translation, so it’s far from perfect. But if you’re monitoring feeds from multiple countries across Twitter, RSS, Email or SMS it’s sometimes useful enough to get a quick sense of what’s being said, where to potentially look for more info, or perhaps where to direct human translators.

Integrating with Swiftriver – Using systems we haven’t even though of

Mon, 21 Mar 2011 03:36:00 -0400

Just the other day I was asked a couple of questions about integrating Swiftriver with data sources and third-party systems that the Swiftriver team haven’t thought of.

I though the answers may be helpful to others who are thinking of doing the same thing or just want to get a bit more insight into this side of the platform:

Question 1:

Can Swiftriver be extended to manage streams of discreet data such as geographic coordinates coming from a source like SMS messages or remote sensing platforms

My Answer:

Most definitely! Our core platform has several points of easy extension and one of these is the plug-in system we call Parsers. Each parser knows how to communicate with one type of source and how to process data coming from that source. Examples of existing parsers are; the ‘Twitter Search Parser’, the ‘Frontline SMS Parser’ and the ‘Google News Parser’. The Parser plug-in architecture is very simple to programme for, meaning that new Parsers for any new source are simple to produce and then leverage. It would therefore, be a relatively simple task for a developer to create a Parser that ‘understood’ how to use discreet data such as geo-coordinates (or in fact any other type of data) and knew how you receive that data from a source such as an SMS Gateway. Once written, the Parser can literally be dropped into the correct folder of the software install and this new Channel (combination of source and data type) would instantly become available for use.

Question 2:

Would it be possible to link the Swift platform to a data platform like Excel, SPSS or JMP?

My Answer:

Again, this is exactly the kind of use case that we had in mind when we sketched out the first architecture diagrams for Swiftriver. Our Core platform has several points of easy extension and one of these is the plug-in system we call Reactor Turbines. These Reactor Turbines react to system events and have the ability change, control or redirect the flow of content within Swiftriver. For example, we already have a ‘Ushahidi Reactor Turbine’ that is responsible for sending content items that have been collected and processed by Swiftriver (and by processed I mean that they have been passed to our Web Services for Auto NLP Tagging, Auto GEOLocation etc.) directly to the Ushahidi mapping platform. It would be a relatively simple task to write a Reactor Turbine that pushed content into any of the Statistical Analysis tools you mentioned.

In addition, because the architecture of our platform is designed to be de-coupled, the Web Applications – such as Sweeper – communicate with our core platform via a web based API. The practical upshot of this is that any software system can be extended to talk to the same API and gain access to all the functionality currently shown in our Web Applications.

So; it is very possible to link Swiftriver with any other data driven platform or in fact any other application of any kind. The type of operations the third party system would wish to enact would dictate the type of coupling chosen: If the third party system is only interested in being sent the processed data from Swiftriver then the Reactor Turbines architecture would suffice, if however greater interaction is required from the third party system then a direct connection to our API would be more suited.

I hope that these answers provide a good insight into the ways you can integrate Swiftriver with other systems.

As always, if you want to ask any questions or just start an interesting chat about how Swiftriver can be used, why not join the community via our Google Groups Chat [http://groups.google.com/group/Swiftriver].

Till next time,

Matt

Director of Platform, Swiftriver

Positions Open at Swiftly.org

Mon, 14 Mar 2011 16:27:15 -0400

Ushahidi is currently seeking to hire individuals in the following full-time and contract positions: Sr. Web Application Developer, Online Ethnographer/Behaviorist, Computational Linguistics Expert. As these positions are filled this post will be updated to reflect what’s still available.

Contact: Jon Gosier, Director of SwiftRiver at jg[at]swiftly.org

Sr. Web Application Developer (Python/PHP)

Experience Requirements: At least 4 years professional experience in PHP/XHTML/MySQL/CSS building web applications. This position is minimum full-time for 12 months. Developers with a background in Design, experience with Ruby (Rails), Python (Django) and PHP Frameworks are definitely preferred but all candidates are welcome to apply.

Location: Anywhere, Global

Salary: $60k per year, U.S. dollars. 75% full-time commitment expected although candidates are welcome to maintain side-projects so long as they don’t affect primary deliverables and deadlines.

Online Ethnographer/Behaviorist

Experience Requirements: PHD or PHD-Candidate level with a background in the qualitative study of network dynamics and ethnography of online communities. Position will require deep analysis of dynamics in online communities, and work alongside computer science teams to assist in the development of applications and algorithms based upon their research.

This position is minimum full-time for 12 months.

Location: Anywhere, Global

Salary: $60k per year, U.S. dollars. 75% full-time commitment expected although candidates are welcome to maintain side-projects so long as they don’t affect primary deliverables and deadlines.

Computational Linguistics Expert (Python)

Experience Requirements: At least 5 years professional experience in the development of computational linguistic algorithms using Python. Applicant would supervise the development of open-source semantic technologies, with an emphasis on modularity and scalability. This position is contract.

Location: Anywhere, Global

Salary: Contract. Negotiable.

Sweeper v0.3.0 Released

Tue, 01 Mar 2011 04:36:00 -0500

Download Sweeper V0.3 Now - Click Here!

Hi all you Swiftriver and Sweeper followers out there.

The Swiftriver team is over the moon to announce the launch of our latest version of the Sweeper app.

Those of you who have been following our progress will know that this release comes hot off the heels of the V0.2 that we pushed to you all a couple of months ago.

As always, the Swiftriver guys and girls have had their heads down crafting and coding some great new features that go a long way to making Sweeper bigger and better than ever before.

So, what can you expect to see out of the V0.3 release?

The Sweeper Dashboard

Users of the new release will be greeted by a lovely shiny new dashboard that makes use of our custom built analytics module and the great jQuery graphing library jqplot [http://www.jqplot.com/] (thanks Chris for this easy to use and powerful tool). You can expect to see a lot more data visualisation in upcoming releases and we intend to utilise the power of the new analytics module throughout the Swiftriver family.

Tag Based Navigation & Channel Based Navigation

Building our powerful tagging service, the latest version of Sweeper now offers users the ability to refine their view of content based on tags that are important to them. Simply click on any tag in the content list and see the list repopulated with only content that contains that tag!
Want to compare and cross check what people are saying on Twitter with what they are posting to flickr? Well now you can with Sweeper V0.3. The new channel based navigation filter makes it dead easy to view only content collected by a specific channel while still continuing to collect and process content from all over the web.

Content Clustering

This is a great new tool that allows you to see how similar other content is to a new piece of content. Make sense? Well basically, every new piece of content that comes into Sweeper now has some scores attached to it, showing you how its set of tags match up to the larger set of tags in the system. To make this really relevant, we first show how similar this content is to all other content, then we show how similar it is to content you’ve already marked as ‘accurate’. We expect this feature to grow and grow and the Swiftriver team think this will be a key metric in the battle to cut through the noise and get down to only really interesting content.

Content Refresh Message

The new version of Sweeper keeps you updated with messages about how many new pieces of content have been collected!

As always, we didn’t manage to cram everything into this release that we would have liked to – but no worries, it just makes the next release all the more juicier!

So before I go, I can give you a little taste of some of the things you can expect to see from the next release:

Revised Sweeper Panel
We have been doing some work internally on improving the user workflow around voting on content, expect the next release to have some jazzy new buttons for you all to press!
Pluggable Content Ordering Module
Building on some of the great statistical tools we have in Sweeper already, the next release will ship with a whole new plugin framework (in addition to our Parser and Turbine models) that will allow you to change the order in which content is delivered based on things like location, RiverID score, tag clustering etc. And because it’s pluggable, you can dream up your own wild and wonderful ways of sorting the quality from the noise!

Well that really is it for now. Thanks for following our project and please get in touch if you have any issues, suggestions or would like to contribute.

Thanks as always to the Swiftriver team and our brothers and sisters over at Ushahidi for making it all possible!

Matt

Director of Platform, Swiftriver

Developing Plugins for SwiftRiver Applications

Sun, 30 Jan 2011 07:30:00 -0500

Ahmed Maawy, the newest hire to the SwiftRiver project, recently compiled this great ‘how-to’ guide on writing plugins for SwiftRiver applications like Sweeper and SwiftMeme. These plugins can mostly be found at http://plugins.swiftly.org while the wishlist for things we’d like to see built can be found here.

For a great example of how Swift plugins work, check out the Ushahidi Report Push plugin, which allows content verified in Sweeper to be passed along to Ushahidi. Coupled with the Yahoo Placemaker plugin, this is really powerful as it allows all content to pass from Sweeper to Ushahidi, auto-geolocated.

You can view a fully formated version of this guide on Google Docs

Before we begin it is worth noting that all SwiftRiver applications have 3 major components:

SwiftRiver Core - the engine behind content retrieval, processing and storage.
Installer - in charge of initial setup of the SwiftRiver platform.
Sweeper - Sweeper is the application built on top of the Kohana PHP framework that acts as a web application that renders or provides a UI on behalf of the operations performed by the SwiftRiver core.

There are 3 very important elements to understand for SwiftRiver applications when developing and extending the platform for customized functionality (These 3 elements can be considered as “plugins” for SwiftRiver).

Impulse Turbines - Are elements that process and add value to content received from external sources.
Reactor Turbines - Are event handlers, and are not necessarily meant to add value to content but to react to specific events within SwiftRiver.
Sources - Are parsers for different types of content. They are responsible for retrieving content from the Internet or other relevant sources, and translating this content to SwiftRiver content items, so that content from different sources can all have a uniform format within SwiftRiver.

Its is important to note that the SwiftRiver /Modules folder contains a number of these event handlers (Reactor turbines and Impulse turbines). However, Sources (Also known as Parsers) are developed within the /Modules/SiSPS/Parsers folder.

This is a step by step approach regarding how content is received and processed within the core:

Parsers take the content from the various external sources, and convert it to the Swift object model.
Impulse Turbines may act on the SwiftRiver content items and add value to these content items.
Reactor Turbines may be used to work on the end result of the content either before they are processed by Impulse Turbines, or during their processing cycle, or anytime within the lifetime of the content after specific user actions (such as mark content as accurate).

Parsers / Sources

Parsers are located within the /Modules/SiSPS/Parsers folder and follow the following important rules:

Have to have a Parser.php file name format
The class name has to be the same as the file name
The class name must implement the IParser class
It must be within the namespace SwiftRiver\Core\SiSPS\Parsers
Must contain the following functions:

GetAndParse($channel): returns an array of Content Items

ListSubTypes(): Returns the sub types of the Parser

ReturnType(): Returns the type of the parser (Which has to have the same name as the parser you specified in

ReturnRequiredParameters(): Returns an array of the parameters required to initiate a single source entry for this parser.

You may take a look at how content items for Twitter are generated to get an example on how parsers work. Content Items are also passed back together with Source data where available. You may also need to know how the object model for a Channel, Source, and Content are structured. These classes are located within the /ObjectModel/ folder.

Impulse Turbines

Located in the /Modules/ folder. Use the following important rules:

Have a PreProcessingStep.php file name format.
The class name has to be the same as the file name.
The class must implement the \Swiftriver\Core\PreProcessing\IPreProcessingStep class.
Must be in the namespace Swiftriver\PreProcessingSteps
Contain the following functions:

Process($contentItems, $configuration, $logger): Which does processing on the content items.

Name(): Returns the impulse turbine name.

Description(): Returns the description of this pre-processing step.

ReturnRequiredParameters(): Returns an array of required parameters for the pre-processing step.

You may refer to the file GoogleLanguageServicePreProcessingStep.php in /Modules/GoogleLanguageServiceInterface/ folder for an example.

Reactor Turbines

Are located in the /Modules/ folder with the following important rules:

Have a EventHandler.php file name format.
The class name has to be the same as the file name.
The class must implement the \Swiftriver\Core\EventDistribution\IEventHandler class.
Must be in the namespace Swiftriver\EventHandlers
Contain the following functions:

HandleEvent($event, $configuration, $logger): Contains the event code.

Name(): Returns the impulse event.

Description(): Returns the description of this event.

ReturnRequiredParameters(): Returns an array of required parameters for the event.

ReturnEventNamesToHandle(): Returns an array of the event enumerations the turbine tends to handle.

You may refer to the file UshahidiAPIEventHandler.php in /Modules/UshahidiAPIInterface/ folder for an example.

Important notes to consider during the EventDistribution phase

The ReturnEventNamesToHandle() function points to an enumeration from the /EventDistribution/EventEnumeration.php file. This is where you can design your own custom enumeration.
It is most appropriate to place event handlers within the application’s workflow. Application workflows are placed within the /Workflows/ folder. For example, all workflows related to channel activities are placed within the /Workflows/ChannelServices/ folder.
The example below demonstrates how you would invoke an event within a specific place within the workflow:

$event = new \Swiftriver\Core\EventDistribution\GenericEvent(

\Swiftriver\Core\EventDistribution\EventEnumeration:: $ContentPostProcessing,

$processedContent);

$eventDistributor = new \Swiftriver\Core\EventDistribution\EventDistributor();

$eventDistributor->RaiseAndDistributeEvent($event);

Please feel free to contact the SwiftRiver team for any further assistance and help. You can contact us by emailing support@swiftly.org

U St. Brainstorming Session

Sat, 29 Jan 2011 13:26:01 -0500

Patrick Meier and some friends and users of Swift stopped by Affinity Labs in Washington a few days ago with some great suggestions and feature requests for the next release of our Sweeper application. Our work was largely centric around rethinking user interaction options. It was an exciting day and we’re really looking forward to incorporating these suggestions in our next release.

Check out the slideshow for some shots of our brainstorming session and the image below for a sneak peak at the proposed redesign.