FabFi Wireless

Wifi and the Internet of Things

noreply@blogger.com (Keith Berkoben) — Wed, 18 Jan 2012 17:24:00 +0000

Today is a day for thinking about the future of the web. By virtue of being the author of this blog, that gets back to the future of communications infrastructure, and especially wifi. Apologies in advance if I wax poetic, Wikipedia, Craigslist, and Reddit are all down today in protest of SOPA/PIPA.

In recent years there's been a lot of chatter about the nature of technological convergence and the future of global telecommunications infrastructure. Technological optimists, particularly those interested in the developing world trumpet, "mobile, mobile, mobile" and, on many levels, they're right... While we won't be ditching big screens and real keyboards any time soon, all statistics point to the fact that mobile devices account for an increasing number of transactions, tasks, and amount of user online time. Without doubt, mobile has been disruptive in the west, speeding the transition to cloud computing; a game-changer for Africa, bringing millions online; and largely responsible for an global wave of grassroots protest movements. The thing that people tend to miss, however, when talking about the future dominance of mobile is that the device and the communications infrastructure are not one in the same.

Our lust for data is killing mobile providers (you need only look as far as the near-extinction of the unlimited data plan), and it's technologically challenging to shrink mobile cell sizes enough to [economically] support traffic loads on limited spectrum licenses. Mobile carriers depend on the extremely efficient use of spectrum that comes with careful site planning and modeling. This gets exceedingly hard to do as cell sites become more numerous and demand cheaper installations.

The licensed nature of mobile spectrum/devices also makes the sort of grassroots innovation and entrepreneurship that typifies the online world very difficult. If we were only talking about making smartphones maybe we wouldn't care, but the smartphone is only the tip of the iceberg for the growing world of ubiquitous, cloud-enabled computing.

The ability to be "connected" at all times is clearly desirable (even if we choose to unplug sometimes for our own sanity). Smartphones have achieved a permanent position in our pants pockets formerly afforded only to wallets and house keys; but connectedness is only useful to the degree that we can interact with our world through being connected. Social was easy: "I'm connected. You're connected, let's share some photos and poke each other", but what about STUFF?

The natural next step for a connected person is to have a connected world to interact with. We've already started with the big ones: Thermostats, security systems, home theater. But the possibilities are endless, and it seems that the consumers are hungry. If you think I'm kidding, check out this Kickstarter project from some of my former Media Lab colleagues that garnered a half a million bucks to create a wifi-enabled rubber(?) brick, with a temperature sensor and accelerometer, that can be used for useful stuff like telling you when your laundry is done. Innovation to fill this new space of connected "stuff" will only thrive with the low entry barriers of an open internet, unlicensed spectrum and cheap hardware.

In any case, the monolithic model of wireless communication is just not going to work. A natural consequence of having EVERYTHING connected is a whole lot of unplanned and uncoordinated wireless communications generated by cheap, duty-cycled devices, flying around all over the place. This is the sort of thing that breaks traditional mobile communications.

Wifi, especially with 802.11n and soon 802.11ac boasts so much capacity that even in an environment where 40 or more uncoordinated access points are all visible to each other (such as my bedroom), you can still expect to share 15Mbps or more with all the devices in your home, while your neighbor can do the same, simultaneously. The limitations of wifi, in terms of range and penetration of obstacles, are strengths in the world where everything is a transmitter. You won't send 15Mbps simultaneously to every home in my neighborhood with 4G...

As opposed to falling off in popularity as mobile improves, wifi has continued to strengthen. Wifi will ultimately bridge the gap between wireline and mobile providers. Cable companies have already embraced wifi as a way to increase the value of their services. In Europe, mobile providers like T-Mobile have embraced wifi for years, and US mobile providers are now being forced to add wifi to offload traffic. Ubiquitous, cheap, unlicensed. It's a recipe for innovation, and a guarantee that wifi will only increase in importance as we become increasingly connected. Keep hacking...

Development Kits Live!

noreply@blogger.com (Keith Berkoben) — Wed, 18 Jan 2012 04:31:00 +0000

Thanks to Nathan from California, the first Fabfi development kit went out the door today. With the Monday holiday, I had the chance to do a little fit testing of the hardware on a spare pole I had lying around. The itelite enclosure with integrated 18dBi dual-polarization patch took some modifications to mount the RouterStation, but was very robust and had a solid rubber gasket and grooved interface around the edges to keep water out. The built-in ethernet extension plug was also a nice touch. 19" pigtails for the external antennas exit through an optional port on the itelite enclosure and screw directly into the 2.4Ghz antennas.

Below is a sample look at the current kit spec all mounted up. Mounting with the top of the pole above the tops of the antennas is a common trick to protect against lightning strikes (20cm is usually enough):

I'd prefer to see the stock antenna mounts getting the antennas farther from the pole for decreased RF shadow and better spatial diversity, but this could be easily remedied with a crossbar. For applications requiring less 5Ghz gain, a clean option might be to offer a 6-way radome. I might get a chance to test this out in the near future.

Fabfi In Domus Magazine

noreply@blogger.com (Keith Berkoben) — Sat, 31 Dec 2011 19:21:00 +0000

Yesterday Amy and I received a belated Christmas present via DHL. Though we knew it was coming, we had no idea what it might be. Upon opening it we discovered a thick, colorful book from Italy, entitled DOMUS. Domus is an Italian architectural magazine packed full of colorful images and stories about art and design. I'm always surprised how often Fabfi gets picked up in the design-world media. Perhaps designers and architects are freer to follow ambitious visions undeterred by technical challenges of implementation; or maybe they just think fluorescent orange acrylic looks snazzy. Either way, we're grateful for the publicity, and the clever re-rendering of my graphics as seen below (see how CC-BY-SA is useful!). I'll refrain from commentary on the editorial itself, which is roughly historical in nature. One geeky side point is that anyone with the magazine has all the data they need to build a Fabfi reflector.

Find the article online article HERE.

Happy New Year everyone!

My Router has a bad case of S.A.D.*

noreply@blogger.com (Keith Berkoben) — Fri, 02 Dec 2011 14:02:00 +0000

In the last post I blathered on quite a bit about sizing solar panels for routers. As it turns out, the theoretical and the empirical don't match up so well. Instead of running 24h/day, the Farm School solar nodes are currently running more on the order of 6h. I suspect this has mostly to do with the fact that the sun is so low for much of the day in this part of the world that it's behind trees, but it's also possible the new charge controllers aren't as efficient as billed. Whatever the reason, I'm about 18h/day short of the full monte (translate: waaaaay short), which has me thinking about power optimization.

In laptops, a very common way to save juice is to adjust the clock speed down when the device is idle, and it seems someone has built a package to control CPU clock for the RouterStation. It would be a great project for someone to wrap this with a script that measures load and adjusts speed accordingly. Anyone interested??

*S.A.D. stands for Seasonal Affective Disorder

EDIT: I took a few minutes to try manually switching the clock speed with the above package. A few Results:

Changing speed requires a reboot (the package is simply a command line tool for rewriting the bootloader)
The overall reduction in power going from 680Mhz to 200Mhz is only abvout .25W at idle
Running a RouterStation at 200Mhz makes it boot waaaaay slower

Sizing Solar to Routerstation Power Specs

noreply@blogger.com (Keith Berkoben) — Wed, 16 Nov 2011 17:04:00 +0000

As I mentioned in the last post, we've been having a little trouble solar powering our nodes (in the woods) through a New England winter. Turns out we get crap for sunlight. Who knew? :P. As a result I'm trying to get the most out of limited resources and redesign the nodes so they fail gracefully when they run out of juice.

I did a little experimenting with the Ubiquiti RouterStation earlier today to figure out exactly what its characteristics were. Here's a few things I learned:

Minimum reliable operating voltage 9V? (It was difficult for me to determine this number exactly due to the limits of my power supply, but it will DEFINITELY not operate reliably below 8.5V and at 9V it seems to boot reliably and send data over one radio. Using multiple radios hit the current limit of my power supply at 9V, so we just have to assume that the RouterStation itself is not current limited in the same way)
Peak current draw with three radios transmitting at capacity @12v 1.5A(est)
Average power consumption with three radios transmitting full time: 17.5W(est)
Average power consumption with three radios powered, but idle: 7.6W

The first element to consider when working with 12V batteries is that the margin between the minimum expected battery voltage and the minimum operating voltage of the device is quite small (~2V). Even in a perfect world, the voltage drop when sending the peak current over 24AWG cat5 wire (assuming worst-case single conductor, 11V battery voltage), limits the cable length to just over 20 feet. This number gets smaller if you have questionable cabling. This short analysis suggests the use of 24V solar systems instead of 12V.

An additional problem that may arise, in the event that I'm right on the edge with the amount of battery storage I'm using, is that the devices will brown-out and hang. This has already happened a couple of times during some cloudy weather, and requires someone to go out and hard-reboot the devices. An easy solution to this problem is to use a power supply with a Low-Voltage Dropout (LVD) circuit instead of powering devices direct from the batteries. This has the dual feature of protecting the batteries from over-discharge and cleanly cutting off your device before the voltage gets too low; in this manner enabling devices to turn off in the wee hours when there is insufficient light and come back in the morning without fanfare.

I've been eying these gadgets for quite some time now:

They come in 12 and 24V inputs and with direct power out or DC-DC conversion. They take input from a solar panel and passive POE, charge your batteries, and provide both passive POE and wired output with LVD. At $60 in quantity 1, they're a reasonable one-stop solution to building a single setup that will work almost anywhere.

Since I'm stuck with some big 12V panels that I can't easily re-wire, I bought the 12-24V up-converting version. According the company, this conversion is 80-85% efficient, which is why in the final spec we want to run native 24V.

Now some solar math (doing all this math for the intended system, not the one at Farm School)...

First find yourself a solar energy calculator. For the US, this one is pretty cool. For locations outside the US, try the older version. Using the map, you'll be able to find the amount of energy per m^2 at your location, and then import into the PVWATTS calculator. The PVWATTS tool calculates total available solar energy by month, as well as the total energy output of your system after derating (since we're using direct DC, you can omit the derating for any of the AC components).

Now you might be asking "what does kWh/m^2/day have to do with the 60W rating on my panel?". To make sense of all this, you must read the fine print:

PV module power ratings are for standard test conditions (STC) of 1,000 W/m² solar irradiance and 25°C PV module temperature. Caution: these are different than PVUSA (PTC) test conditions

The important take-home from above is that "standard" conditions for the rated wattage of most solar panels are based on a solar energy of 1kW/m^2. As a result the, value of kWh/m^2/day, is equivalent to the number of effective hours that the solar panel will be able to operate at its rated output (assuming the panel operates at the same efficiency over a broad range of input energy, which is generally true). for example, a 60W panel during a day with 3kWh/m^2/day of available solar energy will generate 180Wh of energy before derating.

Looking at our device, we draw 17.5W at full throttle and 7.6W at idle. If we're only using a little under 50% of the radio capacity on average, we get about 12W worst-case power draw. Over one day that amounts to 288Wh of energy. In Athol, MA, the worst months provide about 3kWh/m^2/day of energy.. Our DC power system, based on the PVWATTS numbers, probably derates to about 0.86 of the nameplate value, so we need:

288Wh / 3h / 0.86 = 111 W of solar to operate 24hours during the dead of winter.

Since mounting etc, are bound to be imperfect, you'll probably want to round this number up a bit too.

Though a 120W panel might be enough to power our device on average, any given day might have more or less solar energy. In order to size panels to the average, you need enough battery capacity to cover cloudy days. Batteries are generally rated in Amp hours (Ah), which is the number of hours they can provide the equivalent of 1 Amp at 12V, which is 12W. Conveniently, our device draws 12W, so the number of Ah in a battery is the same as the number of hours it will operate the device, except for one twist. The Ah value usually describes the capacity of the battery to full discharge. Fully discharging the battery over and over can damage it, so we want to plan for a less than full discharge. For a deep cycle battery (deep-cycle = old-fashioned lead-acid battery with thick, unsophisticated electrode plates), 80% is a drop-dead limit. For easy math, we'll say 75% is our discharge limit, and that we want to operate for 36h sun-free. In the worst case, that means a 48Ah battery, which is on the order of the battery in your midsize passenger car.

At Farm School, I'm operating two radios on the solar-powered devices at near-idle, so estimating 8W and derating a little extra for the DC-DC conversion gives me a need for about 85W of solar and 36Ah of batteries. We'll see how this goes...

Start Small. Make History (Farm School Testbed)

noreply@blogger.com (Keith Berkoben) — Tue, 15 Nov 2011 19:35:00 +0000

Originally this post was slated to be a cut-and-dried show and tell describing our live pilot at The Farm School, but the developments over the last 16h at Occupy Wall Street, NYC, and some of the awesome guerrilla reporting that's been happening, provides some perspective for us developers that refuse to let anything be done until it's "perfect".

The particular story I have in mind is that of a liveblogger named Tim Pool, of TheOther99, who has been streaming live now from lower Manhattan for about fifteen continuous hours. The part I love about Tim and his stream is how he has cobbled together an ultra-simple technical solution with free services and has been debugging the setup and learning the tech in real time with a live audience. Over the course of the stream, Tim sources replacement batteries from the viewing audience, takes technical suggestions from their text stream, and works through mini-crises like his phone number getting posted to the internet (oops!) to keep his stream live. By the time of the pivotal hearing verdict at nearly 5pm, he amasses an arsenal of equipment spares, has over 100,000 viewers, is being rebroadcast by major networks, and has a pro-level live video rig on the way (donated) from ustream.tv.

When we're sitting around (on Skype) at Fabfi repeatedly pondering the intricacies of "perfect" security and "fully-automated" setup, we would certainly do well to take inspiration for the model of, "get out there now. Sort the details later". This was, in fact, the way we ran the show for Fabfi 1-4 (there hasn't been a single version of Fabfi yet where I knew exactly how it would work when I arrived on location). With all this "planning" we've been doing, it's easy to forget that the best way to do user-centered design is to design some stuff, subject a whole bunch of people to it, and make rapid changes based on real data.

Every time we've gone live at a new site in the past we learned four dozen new things in no time at all, but in our latest iteration we've been thinking so big (1000 nodes), that we've been neglecting the value of a small live test for some time. What can five or six nodes tell you about a network that scales to a thousand? As it turns out, quite a bit...

Over the past two weeks I've been working out at The Farm School building a small FF5 testbed. Despite being little more than an hour from Boston, this site has a lot of the features we might find in a developing area: A DSL connection provides only 3Mbps of capacity, many of the residences (staff live on the farm) are entirely off-grid, and small buildings provide little opportunity for high mounting points without significant investments in mounting poles or hardware. About a dozen users reside at the site full-time, with day users potentially doubling that number.

Keeping with the theme of approximating developing areas, I went out of my way in this deploy to minimize the additional mounting hardware I brought to the table, instead mounting nodes on existing structures with materials I was able to scrounge on site.

Here's some field-test porn:

Wireless gateway node (101):

Off Grid Nodes:

Node on a (stationary) bus:

Live map:

Within moments of installing the network, we began to learn new things about our system. Here's the highlights:

The largest time investment in node installation comes from mounting solar panels. For the panels to be effective they must be mounted at a relatively accurate angle and direction. Working this out with "found" supplies is tedious and slow, even with power tools and a lot of experience.
The AP+ADHOC single radio config that worked for us in bench tests, had very strange behavior (>95% packet loss on one link) when three nodes were live with connected clients. This was resistant to all sorts of RTS/CTS configs and was remedied by either dropping out the third node or separating the AP off to a separate radio. We're unsure whether this was due to the shift to a real environment or a change on the ath9k driver between builds.
You need more solar than you think. I originally believed would be possible to sustain a routerstation with two radios on as little as 48 Watts of solar at our site and a 10Ah battery. In practice, even our setup with 80W of panel and a 14Ah battery had difficulty after a few cloudy days. (likely more battery is needed). 130W and 24Ah seems to be more than sufficient if there is no sun obstruction, but we have plenty of that...
<(expected) sad face>RS devices are not tolerant of brownouts, and require manual power-cycling after low-voltage condition
2.4Ghz speeds are more than double those of from our urban tests, and signal strengths below -75dBm provided useful speeds.

The new network has a number of useful features that were missing from FF4, namely a live map, which clicks though to network statistics. Nodes automatically add themselves to the map and stats interface. This interface will be steadily improved over the coming weeks.

At the moment, the network is operating with un-encrypted open access, but Antoine promises that his new "portalgun" access controller is but days away.

more to come...

~Keith

Wireless Performance: Gosh Darn Urban Environments!

noreply@blogger.com (Keith Berkoben) — Wed, 26 Oct 2011 19:11:00 +0000

Over the last couple weeks I've been doing some testing to determine the performance of the baseline hardware config for FF5. Some things I've been thinking about are directional vs. omni antennas; RTS/CTS vs. not; how fast is 802.11n, really?

The executive summary:

Use directional antennas whenever possible
Move traffic to 5Ghz links as quickly as possible
RTS/CTS provides a noticeable benefit in obstructed PtMP applications
Sharing 2.4Ghz radio between access and mesh networks is unlikely to scale.

Now some details:

Here's the general idea for FF5:

A 5Ghz backbone serving a local 2.4Ghz mesh clouds, which also provide 2.4Ghz access to clients. The key objective for this design is to make 2.4Ghz (Circle) nodes installable by relatively untrained users, while still allowing the system to perform at broadband speeds. 5Ghz backbone (Triangle) nodes are expected to be installed by more thoroughly trained technicians. Ideally, this network should provide every user performance equivalent to the broadband connection in a suburban US neighborhood. In technical terms, this breaks down to the following requirements:

Peak Client Speed (link-local): 4Mbps
C-Node Aggregate 2.4Ghz Throughput to node with 5Ghz Uplink: 10Mbps
C-Node 5Ghz Throughput to T-Node: 10Mbps
T-Node Aggregate 5Ghz throughput to child C-Nodes: 30Mbps

In the 5Ghz layer, I was able to achieve the desired T-C speed with a single 20Mhz channel and a single Omni antenna at the T-Node, connecting to C-Nodes with directional links. This represented the best compromise of simplicity, cost and performance. Handshaking (RTS/CTS) proved useful with the existence of hidden nodes, and directional links improved throughput. Interestingly, the 5Ghz layer with directional links achieved roughly 80% of the expected theoretical maximum, even in the nasty RF soup that is my urban residential neighborhood. Here's our test setup:

At 2.4Ghz, the most surprising result of preliminary tests was the effect of congestion control backoff (presumably) on throughput. Living in an urban neighborhood, a 2.4Ghz node can easily "see" as many as 50 APs at any given time. Consequently, it is always sharing airtime... Inside my plaster and wire-lathe walled apartment, it is possible to find locations where a node will provide 50Mbps of real throughput to a laptop. Take the same setup outside, and suddenly the performance drops to a little as 15Mbps at close range. While it's reasonable to expect a much more friendly situation in the developing world, these speeds call the feasibility of a design sharing 2.4Ghz radios between clients and mesh at our desired scale into question (sorry, Amy). It also highlights the importance of moving traffic off of the 2.4Ghz mesh as quickly as possible. Here's some graphs of upload and download speeds vs. signal strengths for various configurations. Larger (more horizontally directional) omni antennas provides marginally better results after normalizing for power:

Click here for raw Somerville Wireless Test Data

JoinAfrica Progress Report (and so much more)

noreply@blogger.com (Keith Berkoben) — Mon, 24 Oct 2011 16:11:00 +0000

Looking back at the post history, it's been real long time since we've had an update in fabfi-land. I'd say its entirely my fault, but there are a half-dozen other people who can write to this blog <hands on hips>. At any rate, it's been a long summer for Fabfi, and JoinAfrica, the first African small business venture based on the Fabfi platform. So here's to getting everybody up to date...

As you may remember, I was in Kenya much of summer 2010 working with students from the University of Nairobi to build an all-wireless community ISP with Fabfi. For the last year, the Nairobi team has been building a small business around the pilot project in Mt. View. Here's their latest update:

JoinAfrica Kenya

Progress Report, October 2011

Overview

As of October, 2011 the JoinAfrica network in Mt. View / Kangemi is operating with fourteen nodes, providing coverage to approximately eighty residences. After beta testing in the fall of 2010, the network had been in live production since February 2011, beginning with seven nodes and expanding to fourteen over the following months.

Of the eighty residences covered, thirteen have become regular subscribers to the wireless service. Most users pay KES 2000/mo. for a 512kbps peak rate, while a two of the customers pay KES 4000/mo. for a 1Mbps peak rate. In addition to the thirteen existing subscribers, eight additional subscribers are waiting to be upgraded to the paid service. At the time of this writing, the network reliably saturates the available bandwidth of 1.2-1.5Mbps at peak times, requiring that more bandwidth be added for additional users to be served. As a point of perspective, the gross monthly revenue from current user subscriptions is roughly equivalent to the installed cost of one node.

In September, FabCom, LLC was incorporated as a legal entity dedicated to operating the Mt. View network and providing other networking and ISP services on a consulting basis. Additionally, Mr. Nyakundi and Mr. Ochoti have undertaken specialized training in security, routing & switching, virtualization and storage. Mr. Gone has been developing an updated IPv6/802.1x version of the platform used in JoinAfrica with the Fabfi Wireless team. Mr. Gone has also been supporting the fledgling JoinAfrica network in Njabini, Kenya.

Key Challenges

A key concern limiting the expansion of the network is the cost of bandwidth. The current connection has been generously donated by Wananchi Online, but FabCom has been unsuccessful engaging the company in a commercial relationship or negotiating an upgrade of the existing donated bandwidth.

FabCom was quoted retail prices for dedicated bandwidth from other vendors at a rate of KES 21,000/Mbps. At this price it is difficult to provide competitive subscription prices to users. FabCom is actively searching for alternatives to the retail option that can achieve significant savings below the retail price. One potential approach to this problem is acquiring a CCK Network Facilities Provider (tier 3) license. Such licensing would require other ISPs to deal with FabCom at a wholesale level, however the financial barrier to entry for this option is significant (min KES 200,000/yr), and comes with no pricing guarantees. An alternative arrangement would be to enter into a revenue sharing arrangement with an existing ISP, which could easily be sustainable on a revenue sharing model. FabCom is exploring both approaches at this time.

Funding and Future Expansion

Once a suitable bandwidth contract can be secured, FabCom plans to expand the network to serve the eight customers on the waiting list. Increased bandwidth will also enable efforts to increase penetration in areas covered by existing nodes.

A second tier of planned expansion includes jumping across the Wyaki Way highway from Mt. View, and a variety of new sites, including Waterfront Gardens, Highrise, Langata and parts of Westlands. Expansion to these sites will require some outside investment to implement on a short timescale.

A third tier, pending achievement of a CCK license and performance of a small demonstration project, involves the provision of internet to student hostels on UoN campuses.

Fabcom is talking to potential investors (2) to provide funds to enable the first two expansion described above. This capital will in part go into acquiring CCK licence, while the remaining portion will primarily go to hardware and first-month costs for bandwidth.

Additional Opportunities

The expertise FabCom has acquired to in the construction and management of the Mt. View network has placed the company well to act as a contractor for the construction and management of wireless networks for other communities or commercial ISPs.

One specific opportunity for this model is partnership with Orange Telkom, which is in the process of rolling out wireless hotspots in various locations like campuses, airports,shopping malls and other recreation areas. We are actively soliciting them to contract with us to build out these hotspots on a revenue sharing model.

While the ISP project alone is not yet large enough to be sustainable business on its own, the FabCom team has had success leveraging their experience from operating the network into other business opportunities.

Perhaps more exciting than the Mt. View pilot itself, is the capacity addition of the Nairobi team has added to the Fabfi ecosystem. The the FabCom team has begun to support other operators starting Fabfi networks in Kenya and worldwide (Tom from FabCom just hinted at a new network in Madagascar a few days ago). Tom from FabCom has also been instrumental in the development of our new IPv6 fabfi network (more about that soon).

Relatedly, the main JoinAfrica website, which we hope will be a digital gateway to more community-scale networks in Africa as they come online, is now live. (thanks to Plaid Creative for the hot graphic design). The Kenya team has their own page here. There are plenty of photos from the Kenya project HERE on my website as well.

If you want to keep up to date on the goings-on, be sure to get wired into our social media. The more of you are listening, the more pressure I get to post useful content :).

Fabfi on Facebook
Fabfi on Twitter (new!)
Fabfi Google Group

JoinAfrica on Facebook
JoinAfrica Google Group

Mail the Kenya JoinAfrica team directly HERE.

help wanted

noreply@blogger.com (Amy) — Tue, 28 Jun 2011 23:43:00 +0000

FabFi Afghanistan is looking for dev/test help for an upcoming new deployment and upgrade. If you know Linux, C, networking, and/or PV solar power electronics, please email fabfi at fabfolk.com.

Also needed is Fab Lab make-able weather enclosures. Architects – hate how engineers keep making “boxes”? Now’s your chance to make something iconic for years to come.

FabFi on the front page of the New York Times

noreply@blogger.com (Amy) — Sun, 12 Jun 2011 16:09:00 +0000

This weekend's New York Times has several photos of FabFi Afghanistan in this piece on subversive community communication networks. (We're in the slide set). To be clear, the FabFi project in Afghanistan was not one of those secretly funded projects described in their article (see here where I itemize the bulk of the costs and how they were funded - mostly through personal savings accounts of those who participated and in part through a National Science Foundation grant. (And sure, they never actually talk about us specifically in the piece. But look at the pictures!

However, the urgency and significance of the project are the same. As long as there’s pressure from those seeking a reasonable life where they can go about their business, there’s hope we can throw a lifeline with these so called undermining capabilities.

As the director of the Jalalabad Fab Lab and lead of the Fab Fi project, I've been asked several times about how to scale the Fab Lab and Fab Fi experiences to more fully saturate a city, as well as spinning this off into more cities. While I can provide a technical, programmatic answer, Fab Lab/Fi doesn’t solve everything. It’s only one piece: the rest have to develop at the same time. Infrastructure like roads, power, water, schools, teachers, and systems maintenance as well as the user terminals (laptops and computers), people who use them, and the content they'll consume. It’s crazy to think that there was no cell phone service in the country in 2002 and now it’s pretty solidly working in every major population center (at least when the tower isn’t turned off or bombed). From roads to power to water, the task at hand (officially US or not) was to set off a program that could go from zero to servicing 30 million people in a few years. Imagine deciding to colonize Mars and sending 30 million people first, ahead of the infrastructure.

I think there are maybe three kinds of places in Afghanistan:

There are safer, quieter places that have known better times and whose residents are working to get back to those better times. There’s still crime and killing but it’s a shocking event when it occurs.
Poor, forgotten places that have never known modernization and are harder hit by economic problems (some of which we’ve unwittingly caused).
Places like Kandahar with an almost insidious infestation of crazy. Remember those boys that would pull the wings off of bugs and set ants on fire? Beliefs aside, an environment like Kandahar doesn’t provide the social pressure that prevents them from growing up into full fledge people-hurting psychopaths.

In the first group are cities like Herat, in the western sector of Afghanistan. It's plainly ready and asking for a Fab Lab and associated wealth of possibilities. You could imagine a Fab Lab and Training Center there augmenting and strengthening the communications infrastructure with a parallel or overlaid “subversive” mesh, perhaps through the school system which I hear is quite healthy and respected.

The second group of towns, like Jaghori in Ghazni province, need only to follow the good examples of the first – so much the better if there is strong municipal leadership that both welcomes business activities while keeping them in check.

The third are places like Kandahar, which is our biggest opportunity. Mel King, famous community organizer in Boston, often says that “the wheels in the back of the bus never catch up to the wheels in the front unless something extraordinary occurs”. Fighting over raisins, road tolls, heck, fighting over fighting, these are the things that they know about. “New” doesn’t always mean good on it’s own right, but in this case “new” can simply bewilder long enough for the skinny gimpy-legged kid to grab the football and run. Mixed metaphors, I know. It’s late. Another recent article from educators highlights how the labs are excuses to try something new with rewarding results.

In a recent round of catch-ups with the Afghan collaborators who helped start Fab Lab and the Fab Fi projects in Jalalabad (many of whom were university students when we met), I’m thrilled to tell you that all are gainfully employed in technically enabled positions. A (surprising?) majority have taken the plunge to starting their own technology, logistics, or consulting companies, bravely negotiating the bewilderingly paperwork intensive contracts with ISAF and providing jobs to Afghans. I believe in the need for the private sector to create jobs.

With the depressingly slow rate of new job creation at home in America, it’s hard not to be extra proud and amazed at their optimism and willingness to give it a go and make forward progress in their little corner of the world. I won’t take credit for their success – they were shaped by a long chain of parents, family, teachers, and other opportunities – but at least one was nice enough to say that it was his experiences of previously unexpected self-enabled successes in the Fab Lab that was his inspiration.

Thanks to edits by Kanani Fong of the Kitchen Dispatch.

The Hump - Bringing it all Together

noreply@blogger.com (Keith Berkoben) — Thu, 28 Apr 2011 21:20:00 +0000

May and June look like big months for Fabfi Kenya. Following last month's realization that their 5Mbps of dedicated backhaul was shared to the point where it looked a lot more like 1Mbps most of the time, the Nairobi team has been focusing on a plan to secure the future of their network. In-particular this means supplementing their connection with more bandwidth.One way or the other, this crisis has been very effective in motivating the completion of a few things that have needed doing for a while.

First, the business end of the team recently submitted the paperwork to incorporate FabCom as a legal entity in Kenya -- an essential step to dealing with other companies. (yeay, legitimacy!)

Second, the team has created relationships in Mt. View to become a preferred provider and identified a number of new sites that will expand coverage to most of the estate. (yeay, Demand!)

Finally, Tom has been hard at work building a dashboard to enable enterprise-class monitoring of network performance. It was the first efforts in this arena that originally led us to understand our bandwidth problem:

An initial look at his graphs provided no easy answer. Usually when a network is saturated one sees a flatline on the graph at peak times. Our graph showed nothing of the sort, yet users still complained. The trick to solving the problem was a pair of realizations: 1) We don't have enough users to assume average rates, and 2) it turns out we don't have a dedicated speed either. Real time testing showed speeds slowing to 200kbps at times. Obviously this is no way to run a network!

So What's next?

Tom and I have been working on a new firmware build that will happily manage another connection (hello, iproute2) as well as provide better performance data. In the meantime, Nick has been off finding the best deals on more bandwidth. The challenge is scale. Prices begin to decrease, I am told, as we begin to buy larger connections (5Mbps and more). Thus, the Nairobi team is preparing to quickly triple or quadruple the number of users. At the 40+ user level we expect to be able to pay for the bandwith with breathing room for more expansion.

Now to getting that done...

Bandwidth crisis?

noreply@blogger.com (Onsomu Ochoti) — Thu, 24 Mar 2011 10:55:00 +0000

The number of paid users is growing steadily even before we do a serious marketing campaign. This, obviously, is something that is getting us very excited at the possibility of running a successful business of selling Internet access in Mt. View estate. Residents clearly need internet service in their homes as intimated by an assistant at the Resident's association office. Here is what she had to say:

"Residents always come here to ask about who we have decided to invite to provide internet services in the estate. We have been trying to reach out to a specific ISP but there is no positive response. We hope your (phone) number works coz the ones given by the ISP don't seem to work."

We are currently serving 13 homes, 3 of which are demo users and are willing to pay after they test the service and find it worthy paying for.Most of these users were introduced to us by our earlier customers who found the service good enough that they recommended to their friends. This is the best thing that can happen to a business since, in addition to getting new customers, you are sure the existing ones are happy. This is a marketing line we are keen to encourage.

The disturbing thing, however, has been frequent network outages we have experienced in the last couple of days which have seen even our most trusted customers complain. Here is a text I received from one of them:

" I have tried again with several machines and yet again I cannot connect. What is happening? These outages are too much. "

Most conspicuous of the outages is the network being down every Sunday. Here is what another customer had to say:

" U have to come n check out this net 2mrw. It's been acting up lately. So so so slow n keeps on disconnecting. + every Sunday it does not work."

We anticipated this scenario. Only it came knocking earlier than we expected. Letting down our customers is the last thing we want to do. It hurts. We must do better than the big boys with hundreds of thousands of customers to keep happy.

On the cards is emphasis on quality of service, very high uptime and excellent customer support. We are well on our way to becoming a community ISP for real.

This calls for strict adherence to bandwidth requirements. A dedicated connection fully paid for or subsidized is not a far fetched idea. More bandwidth is most welcome at this point.

Fabfi 4.0 Progress Report

noreply@blogger.com (Keith Berkoben) — Tue, 08 Mar 2011 15:12:00 +0000

It's been two months to the day since the beginning of my follow-up visit to the Fablab in Nairobi, and high time for a little status update, first about the networks themselves, and then about what makes them tick

The Networks

As of today, there are three Fabfi networks under development in Kenya:

Mountain View Estates / Kangemi
Njabini
ARO Fablab, near Bondo

All three networks are operated by local residents, and each has a unique business model.

Mountain View / Kangemi

While the latter two networks above are still pre-production, the Mt. View network has been in full production for about two months. Mt. View is the closest to US-Style commercial model where the typical user pays a subscription fee to receive unlimited access for a month at a predetermined speed.

The network currently operates with 23 nodes across 9 distribution points, providing network coverage to between 60 and 75 households (green circles are "indoor" quality coverage while purple ones are "outdoor" quality):

After some trial-and-error with power solutions, the entire network is also now protected by 2-6 hours of expandable battery backup per node, keeping the internet available even in the event of a total blackout (which are really frequent, wouldn't ya know...)

A few weeks ago, the network made its first leap into Kangemi District, which is a considerably more "underserved" area than Mountain View. As the team begins to automate the monitoring and management of existing infrastructure over the next few weeks, they will be looking for locations to expand on the Kangemi side of the wall.

Of the 9 distribution points currently operating, 6 were added during the months of January and February, with word-of-mouth rapidly turning up residents eager to host new nodes. We now have at least one subscriber for every distribution point, and John and Nick have been out and about actively recruiting more.

An unexpected challenge in building a subscriber base has been structuring our demo program. As you may have seen on the website, the team is offering one week free to new subscribers, however the original program of "Try it for a week, then pay" was exploited almost immediately by groups of people calling for a user account, sharing it for a week, then calling for a new one the next week. As a result, the offer has been changed such that your first "month" of subscription comes with an extra week subscription bonus.

Surprisingly, we have seen very little discovery of the limited free service option. This is the option where you can sign up and have a small amount of access per day (15min or 15MB, whichever comes first). This option has not been aggressively marketed, but have had fewer people than expected asking about it given that it's mentioned only one click in from the main splash page. I expect we'll see a lot more of this use once users can sign themselves up from a web form on the splash page.

Njabini:

As you may have read in earlier posts, I spent a significant amount of time in Njabini working with local Entrepreneur, Peter Murimi, and the Flying Kites orphanage to get a network up there.

One of the biggest challenges in his area is the mobile dongles are the only connectivity method. In this network, the pricing structure of the mobile dongles informs a business model where Peter, the local operator can buy bulk packages from the mobile provider and share those packages among local users who buy smaller units at a price below the Safaricom rates. This is partially enabled by the use of an aggressive web-caching system that runs locally to save bits.

While the necessity of pay-per-bit (as opposed to pay per unit of capacity) informed a solid business model, the resource constraint immediately caused us to pause and rethink our rollout strategy. In this cost scenario system overhead equates directly to cost so Tom is busy working out some proper performance monitoring software that will help us trim the fat before we go live.

The Nairobi team will be visiting Njabini this weekend to make some modifications and teach some network management, after which I expect we'll see a lot more activity out of the sit.

ARO Fablab:

On the outskirts of Bondo-town and truly off the grid, the ARO Fablab is perhaps the most extreme location for a Fabfi deploy. Our friend Boris is out there working with them now. Once they're consistently online the global Fabfi crew can log in and provide remote support. I received mail from the lab Manager today saying:

"Tom from Nairobi came over to assist us complete our FabFi configurations. We successfully configured the headnode and its tested and working fine. We tried to flash the pico devices with the new firmware but did not succeed so far, the old versions we did while I was in Nairobi works well.

We were planning to connect the nearby high schools and also to the cyber cafes at the nearby market centre. This would assist us in paying for the internet connection and providing readily access to the internet for school students and their teachers."

The Tech

While we like to give the impression that building fabfi consists of simply pointing a couple wifi routers at each other and calling it a day, there's an enormous amount of development that goes into creating that illusion. The current fabfi system does first-order integration with dozens of open-source projects, including:

OpenWRT
Coova-chilli and its awesome-chilli wrapper
Freeradius
Daloradius
Nagios
Squid

And with every day we're adding new tweaks to improve stability and performance that require testing in the UoN Fablab development network.

Hardware

The move to version 4.0 prompted the switch to all new hardware, while all the new routers are now working very well, we're still cleaning up the prototype reflector feeds for mass consumption. Info on the process and the designs can be found in the RF section of the wiki.

System Management

Over the last couple of months, the most development has gone into ways of keeping track of nodes and users. We have a few things working nicely now, including a remotely hosted splash page for user logins, a live network map, and a web form that makes .kml for Google Earth or Maps showing all the important data for each node.

Tom is actively developing Nagios-based service alerting and bandwidth monitoring tools that we should be seeing in the next couple of weeks. I also hear rumor that he wants to build a web form to let users sign themselves up in the fablab (yes, please!).

Documentation

Today (fittingly at rev. 400) I finally completed the core wiki for Fabfi 4.0, and updated the fabfi site to point to the new content. We also finally have a complete site for JoinAfrica Kenya where users can learn about their local networks and get online.

The Connections

One of the most exciting parts of what's going on in Kenya right now is networking of the social variety. When Paul came to me last spring and said "I want to find a sustainable way to get more people online in Africa," my biggest concerns were not technical (though I had a few of those) but social. What was the use case for the internet in rural Kenya? Who knows how to operate computers competently? At least the latter is something that even the most technologically advanced societies on earth still wrestle with. Sustainable internet expansion is not simply an economic issue but one of knowledge transfer. You need to carry the knowledge of how to use the tool with the tool.

In Kenya, the UoN Fablab team is now not only expanding internet networks, but creating a network of operators and supporters that share their aims and their skills. They are actively training the operators of the two other networks, as well as working with organizations like KENET and local bandwidth providers to create meaningful partnerships.

Meanwhile operators like Peter from Njabini are setting an example of what could be the future of rural and urban-fringe networking. By combining technical knowledge from the Fabfi team with his local knowledge of community needs and supporters, guys like Peter are key to bridging the gap between the available services and their utility to local residents. Personally, I can't wait to see the results...

Also Special Thanks To...

About a million people, but this project would especially not be possible without contributions from these major supporters and virtual team members:

Paul English
Kamau Gachigi, and the UoN Science and Technology Park
Amy Sun and Sherry Lassiter of FabFolk, Inc.
Antoine VanGelder of Afrimesh
MIT Center for Bits and Atoms
Anjali Sastry, Shiba nemat-Nasser and Nipun Virmani from MIT Sloan
Kerry Lynn
Riyaz Bachani of Wananchi Online
Meoli Kashorda and KENET
Christiaan Adams from Google Earth
Pete Szolovits (for letting Keith spend so much time on this project)

You all rock, and we hope you'll stay with us as we try to grow the networks described above.

Stop Watching Me!

noreply@blogger.com (Keith Berkoben) — Sat, 05 Mar 2011 16:02:00 +0000

The wireless watchdog from the last post was short lived, as the data it sent back made a clear case for changing the wireless configuration (it also had some annoying side-effects on the link).

The link in question is firing through some foliage (if you know the guy with the trees, please ask him to cut the tops off :)), and wind seems to be having the effect of causing the signal to levels that disconnect the STA. As a more permanent fix, we switched the link to ADHOC mode, which limits some of our HT options, but seems to fix the stability problem while still giving us link speeds near the peak speed of the uplink.

It's funny, you can totally see the trees waving in the breeze in the ping times:

64 bytes from 10.100.0.218: seq=295 ttl=64 time=1.172 ms
64 bytes from 10.100.0.218: seq=296 ttl=64 time=719.076 ms
64 bytes from 10.100.0.218: seq=297 ttl=64 time=1.207 ms
64 bytes from 10.100.0.218: seq=298 ttl=64 time=1.805 ms
64 bytes from 10.100.0.218: seq=299 ttl=64 time=425.842 ms
64 bytes from 10.100.0.218: seq=300 ttl=64 time=2.308 ms
64 bytes from 10.100.0.218: seq=301 ttl=64 time=6.422 ms
64 bytes from 10.100.0.218: seq=302 ttl=64 time=1.321 ms
64 bytes from 10.100.0.218: seq=303 ttl=64 time=2.304 ms
64 bytes from 10.100.0.218: seq=304 ttl=64 time=23.272 ms
64 bytes from 10.100.0.218: seq=305 ttl=64 time=1.172 ms
64 bytes from 10.100.0.218: seq=306 ttl=64 time=7.758 ms
64 bytes from 10.100.0.218: seq=307 ttl=64 time=21.644 ms
64 bytes from 10.100.0.218: seq=308 ttl=64 time=13.355 ms
64 bytes from 10.100.0.218: seq=309 ttl=64 time=9.076 ms
64 bytes from 10.100.0.218: seq=310 ttl=64 time=281.356 ms
64 bytes from 10.100.0.218: seq=311 ttl=64 time=1.200 ms
64 bytes from 10.100.0.218: seq=312 ttl=64 time=302.242 ms

Despite the high jitter, there's no packet loss at all on the link. Hurray for greenfield deployment!

Thanks to our new clients for their patience while we sorted this one out. I expect that the quality of the link should be much improved going forward.

Watch Me!

noreply@blogger.com (Keith Berkoben) — Fri, 04 Mar 2011 05:35:00 +0000

It's been a long week of updating wiki pages and building firmware to clean up all the loose ends from January and February. A big data dump is coming soon, but to tide you over here's a little techie interlude...

As I may have mentioned before, we added infrastructure mode wireless connections to our config options to capture HT throughput (the adhoc HT driver is coming! I can practically taste it). While AP / STA can be fast, it also adds the whole complication of association. In greenfield environments, the noise floor is generally so low that we can run fast links with scary-low signal strengths (the floor on the link budget is often device sensitivity, not noise at all). In adhoc mode that works great, but in infrastructure mode, there's some setting I haven't found yet that keeps a device from reassociating with a link that's running below a certain strength (or maybe a limit on the number of reassocs, I'm not sure).

In any case, devices dropping links that are perfectly good has been an issue lately, but it seems bumping the AP's wireless connection is enough to get it back, but we can't just sit there waiting for the connection to go down? Or can we? Enter watchdog (details truncated for brevity):

#!/bin/sh
while true
    do
    resp=""
    resp=`iw dev wlan0 station dump | grep -c "[MAC you care about]"`
    if [ $resp == 0 ]
        then
        wifi
      echo `date` No assoc. bump wifi >> /tmp//log.log
      sleep 10
    fi
    sleep 5
done

It's not the ideal solution, but seems to be working so far. And I'm not one to screw with what works...

Capturing the moment

noreply@blogger.com (Keith Berkoben) — Sat, 19 Feb 2011 21:30:00 +0000

(now that I'm over the giddy part about being off-grid and on-line, time to make my actual point)

It's easy to get spoiled living in place where a 1.5Mbps connection is the bargain-basement line and become utterly unable to appreciate how utterly life-altering it is to have a simple, reliable 200K of data connection. Then you have a moment like the one captured in the screenshot below from Peter's computer (if it wasn't dark when this happened, I'd need to have one of Peter's expression that was just utter joyous disbelief)

After a day of permanently fastening cables, heatedly discussing the best pricing schemes [given our current reliance on mobile], and tromping around town to different locations looking for a backhaul link Peter and I sat down to take a look an audio issue with his laptops. As it turns out, his hardware doesn't work out of the box on linux, so we had to search some forums and install some packages to fix the issue.

Just for fun, I thought I'd try to stream grooveshark while we were waiting for some stuff to install (external speakers worked while internal ones didn't). A minute or so of buffering later, we intersected the satisfaction of diagnosing and fixing a computer for free using online resources with the realisation of new technical possibilities (streaming) and the fully dominant music delivered by those possibilities. You may use the soundtrack below for recreating the moment in the comfot of your own home:

Joe Pug - Hymn 101 from Sam Molleur on Vimeo.

It's dark, there's no power...

noreply@blogger.com (Keith Berkoben) — Sat, 19 Feb 2011 19:42:00 +0000

...and I'm on the internet. Ok, maybe this is childish, but being on wifi when there's no power for miles is friggin' cool. :)

More toys revealed soon.

Always an Angle: Njabini Online

noreply@blogger.com (Keith Berkoben) — Wed, 16 Feb 2011 11:17:00 +0000

As I mentioned in an earlier post, our site in Njabini is a prime location for creating a truly cooperative community network. The nagging problem is bandwidth. Originally we had thought that Orange would come in with a middle mile solution for us through the Digital Villages initiative, but it seems that building backhaul is not in the DV model, even when the business case might exist. Upon receiving the GPS coordinates of the Njabini site Orange promptly said, "sorry, can't help you".

After hearing the bad news, I nearly resigned myself to abandoning the Njabini site (which probably would have been prudent from a time management perspective), but I really REALLY hate being beaten and the FK / Njabini community is incredibly excited about lighting up the town with internet.

Without the promise of an "unlimited" connection from Orange, I was initially hard-pressed to find a reasonable business model. Mobile connectivity is gosh-darn expensive to resell, and the idea of paying by the bit is rather unpalatable (though realistic as a service provider).

Then I looked at the rates... Information asymmetry is a dangerous thing. If you don't have easy access to a price list (which most people don't know they do), it's easy to fall victim to the design of the mobile operator's billing scheme. In Kenya most operators sell bulk mobile internet connectivity in "bundles", which vary in price/MB based on how much you purchase at once. For instance, Safaricom charges 3.33KSH/MB if you buy 100KSH at a time and .5KSH/MB if you buy 9,999KSH at a time. The real scam, however, is that not buying bundles doesn't keep you off the internet, it just costs you 8KSH/MB. Bundles also expire in 30-90 days, depending on the bundle, so one can't buy in bulk for the future very easily.

For most internet consumers, the idea of purchasing or consuming 9,999KSH (20GB of data) in 90 days is nothing short of ridiculous. Much like all things here, people tend to buy only what they need when they need it and pay a serious premium for the flexibility. The rub is that they often think they're paying less. One of my team members, who lives in Nairobi and uses the internet daily (albeit rarely from a phone), was convinced until five minutes ago when we looked at the rates that data on mobile (with no bundle) is cheaper than using a USB modem. Our partner in Njabini believed the same. In a way they may be right because web pages designed for mobile devices are often much smaller than the complete version of the same pages, but the rate/MB is much, much higher.
Using Fabfi, a single provider with a strong connection can buy the biggest bundles to get a good rate, passing on those savings to wifi consumers who can purchase connectivity flexibly at lower rates. Local caching and services can then further increase that margin. A real HSPA connection would even be fast enough to do this at a reasonable service level.

Of course, Njabini doesn't have a reasonable HSPA connection (heck, they only got grid power three years ago), but even sharing a crappy mobile connection with a couple of people is better than browsing all by yourself, so away I hacked.

I'm not proud of the aesthetics here, but I assure you the mounting is temporary...

The first linksys connects to the Safaricom 3g dongle and shares the connection to our headnode, which runs a local webcache and all the usual mumbo-jumbo. While we're only getting 256k from the dongle, point-to-point speeds inside the network are impressive.

Over five hops (Ethernet -> 5Ghz -> Ethernet -> 2.4Ghz -> Ethernet) and 2.5km, we're pulling nearly 30Mbps real (in marketing bitrates that's about 75Mbps) to the Flying Kites house, where users connect to a wifi cloud.

Since FK is off the grid, we're using 60watts of solar and 14Ah of battery to keep the node up. With two ubiquiti devices, this setup runs 24h in the current weather, and I expect to have enough margin to keep it up during waking hours in the rainy season. Notably, users can still browse when all the power in town is out (a daily occurrence). There's something very satisfying about checking your email in a dark room on the side of a mountain at midnight...

We also have a wifi cloud in town and wired computers at a cyber cafe run by Peter, our local partner.

This weekend the fabfi team will be in Njabini running a training for Peter and FK. Wicked exciting!

NOW OMG PLEASE WILL SOMEBODY GET US SOME REASONABLE BANDWIDTH? (I'll make my own ptp up to 10km...)

Mobile, Mobile, Mobile, Harumph

noreply@blogger.com (Keith Berkoben) — Mon, 07 Feb 2011 04:53:00 +0000

So everybody says mobile is the next internet in Africa, and indeed Nairobi is chock full of teeny little stores that look like the one at right. Despite their number, they all seem to do reasonably good business, at least in 20-somethings ogling, fondling asking questions about the latest offerings. While having a shiny new phone may be "cool", it is absolutely not the solution for internet expansion in the next 5 years.

Even in the city, with 100% signal strength in the middle of the night, my brand new Orange 3G+ modem does about 256k. Respectable, and certainly as good as the other options, but not mind-blowing.

Out in the rural areas, most places with mobile service have data speeds that will make you cry. Do you know what a roundtrip time is? If you do, then you'll know that 9 SECONDS is not even sorta-kinda-almost-close to acceptable. (disclaimer: that was the worst I saw. Average was closer to 700ms, and measured on a safaricom stick). The facts are that long-range mobile wireless is hard to do well, a cell tower is only as good as it's backhaul, and the latest capacity numbers posted by wireless hardware companies are not much more than bench-test values. In the real world, mobile eats it.

Nonetheless, when you've got nothing else, you might as well plug it into your fabfi. Which Is why I'm working on this:

That's an Orange 3G+ modem in the back of a linksys. Right now it connects, but needs to be hotplugged right before you try to do so for it to work (read: I don't have the modem commands worked out yet). Nonetheless, progress...

Welcome Mt. View (for real this time)

noreply@blogger.com (Keith Berkoben) — Thu, 03 Feb 2011 12:00:00 +0000

I don't want to speak too soon, but as I write this John and Nick are diligently repairing the power supply problems we've been having at Mt. View over the last week. In all likelihood JoinAfrica should be back up for keepsies in Mt. View by the end of the day today.

If all goes well, we should be opening the network to general enrollment starting Saturday. Sweet

Off the Grid in Njabini, repairs, re-engineering and all the other stuff we did last week.

noreply@blogger.com (Keith Berkoben) — Thu, 03 Feb 2011 05:08:00 +0000

This weekend was yet another learning experience for team fabfi. Thanks to a little prodding from Paul, our most recent enabler, we made the trek out to a little agricultural town called Njabini, and we're undoubtedly better for it.

Njabini is officially in the middle of nowhere. No internet (to speak of), regular power outages, nothing but Chinese-brand electronics (and not many of them). If the path to development is clean water, clean power, email and Skype (and it is), then Njabini has a long way to go, but what it lacks in infrastructure it makes up for in community and cute kids.

With the best growing climate in Kenya, Njabini is a reasonably prosperous rural town (as in more people have jobs than not, even if they don't pay much). It also has a great, albeit somewhat competing, group of community leaders -- head of the agricultural association, head of the water board, the mayor, the director of the local orphanage -- all with a meaningful interest in finding ways to improve their little town. So why do we care?

First of all, Njabini is primed for implementing the vision of free-to-fee that I've always envisioned -- build a sustainable backbone of user/subscribers and use that backbone to support free access for surrounding residents. Thanks to the vocal support and networking from Flying Kites Global (operator of the local orphanage) we have a whole bunch of people interested being that backbone as soon as we have the bandwidth.

Second, Njabini is HARD. If you didn't bring it with you, you probably can't get it, and Murphy's law will apply in spades to everything you do bring (so bring two of everything).

Following from reason number two, one of the most important objectives for team Nairobi is getting them to be prepared for the unexpected. Since we've begun deploying "node in a box", I've insisted that the team do all the installing themselves. Making mistakes where it counts is the fastest way to learning how not to make mistakes, and during our first trip to Njabini, Nick got a hefty dose of walking the tightrope without a net.

As it turned out, our nifty CPU power supply hack had a .5A current limitation that was only labelled on some of the supplies and went unnoticed into the first four nodes. Even though we tried charging from a discharged battery in the lab, the city power was clean enough to let most of the supplies survive and the failures we did see were easily attributed to other circumstances. In the field, however, we drove the batteries to the limit powering our gear while we wired up the first node, and hooking them up to the supplies on questionable mains promptly fried two supplies in a row before we realised the issue. In any normal deploy, we'd just grab a battery charger from a local hardware store and be done, but the best Njabini had to offer were some tiny 1A universal power bricks, which Nick cleverly disassembled and wired together like so:

This nearly worked out, except that wishful thinking got ahead of hard science and only two of the 1A bricks were used to power four devices and a switch, leaving the system a few watts short of equilibrium -- a problem that wasn't noticed until the next afternoon when the batteries cut out midway through peaking of a 2.5km link to the Flying Kites orphanage. In the end, we were able to prove a link to FK from the center of town was possible, but weren't able to work out a permanent install before returning to Nairobi for some emergency maintenance.

While Nick and I were in Njabini on friday, Tom and John were in Mt. View installing a new node. All was well and good on the new install except that the long-haul to Lavington disappeared, and given that the feed elements were still prototypes we fully expected that something had simply fallen off the reflector on the Lavington end. It took until Tuesday to get an engineer to let us into the Lavington tower where we found nothing amiss, and the old feed still firmly in place. Meanwhile another power supply failure had occurred on the remote end of the link leaving Nick to scramble to find temporary power for testing that the link was A-OK.

While waiting we replaced the old feed on the tower for good measure and got the link back up with no trouble, but were left with a pile of Mr. Fix-it for this week. While this was all a huge pain in the tukkus, it did drive one point home for team Nairobi, and that's the value of testing and contingency planning. Last night Nick and John spent some hours building and itemising the contents of a failsafe field kit containing all the equipment required to adapt to almost any unforseen circumstance. We'll see how it works today when they go on a blitzkreig of node repair without me. And let's hope that this time the fixes stay fixed for good.

Serial Wiring for NanoStationM2-Loco

noreply@blogger.com (Keith Berkoben) — Wed, 26 Jan 2011 20:51:00 +0000

Long Day at the Office

noreply@blogger.com (Keith Berkoben) — Wed, 26 Jan 2011 20:47:00 +0000

Some days you win, some days you lose, some days you get 3hrs of sleep, work for twelve hours and learn all about what to do differently next time. Today was showtime for Nick and John. While Tom and I have been hacking network configuration for the last week or so, Nick and John have had the responsibility of designing and building a permanent installation solution and working out all of the logistics for meeting node hosts. Today we were scheduled to deploy three nodes, beginning at 7am. As Tom and I were glued to computers, John and Nick were left to make sure that when the tech was ready, the world would have waterproof housing, power and rigid mounting poles to accept it.

Having done this a couple of times already, I tried to be adamant about the importance of having EVERYTHING ready (and tested) in advance and thinking through contingencies in case something went wrong in the field, but I don't (yet) know the answer to every management problem and got a lot of yes, yes, yes, without the associated substance. Not surprisingly, the team got sent straight to the headmaster of experience at 6:45am.

When I walked in at ten-of-seven (hoping that my own residual debugging wouldn't be critical path) I found Nick feverishly hammering away at a mounting box and John was nowhere to be found -- great for my moral high-ground but not for today's productivity. By 9:30 I had sorted the network, but the hardware train didn't roll until after a shopping trip an hour or two of packing and a hasty lunch. We had to cancel our first appointment, got lucky that the second cancelled for us, and had to bail on the third for lack of time by day-end.

As I look at what worked and what didn't, the importance of design and planning stand out strikingly. The team did a lot of things right, but in neglecting the details undermined a lot of their own hard work. Nick spent the better part of two weeks designing, sourcing, and finishing a mounting solution for the new Fabfi

In an unexpected move, John made an on-the-fly design mod to put the pole behind the bracket and the box on the front, cutting down the materials list and solving a tension problem with the pole attachments. John also pounded the pavement to cobble together an effective power backup out of computer parts, solar charge controllers and UPS batteries. Between the two of them, they created a beautifuly integrated, reasonably priced installation:

Mounting box: 250KES
Mounting bracket + HW: 600KES
Charge controller: 2900KES
Power supply: 800KES
Battery: 1000KES
TOTAL: 5650KES (=$70USD)

With all the components mounted in the box installation involves only drilling three holes and tightening a few bolts. Power-in, ethernet/POE out, done (it's very clever) While the mounting was sturdy and simple to attach to the wall, by deciding to assemble the solution in the field, the team lost most of the benefits of a well-thought-out design. During the four hours in the field today, more than three were spent mating the box to its bracket and mounting components inside. The other four hours that could have been spent in the field were occupied by packing the field kit that could have been ready to roll the night before. In all, what could have been an 8-hour workday was strung out to a 12-hour marathon where less than half the time translated to progress.

That said, the results for the one node that the team did complete were impressive. As a mentor, I can only hope that wisdom is a better motivator of behavior than I am.

Note: the rig on the left in the picture is KDN butterfly. As far as I can tell, nobody uses it, ever....

More nodes tomorrow.

~Wrench

Power Strugglez

noreply@blogger.com (Keith Berkoben) — Fri, 21 Jan 2011 20:24:00 +0000

Continuing on the "ends in Z" theme...

One of the big questions for this trip has been how to provide backup power for the nodes. It's been in the back of my mind to DIY a power solution in the lab, and with the help of the interwebs (Google: LVD circuit) I don't think it would be that big a deal, however there's always too much to do and too little time. As a result, we did some shopping around, and here's what we found / tested.

Sparing you the long iterative process, we came upon two solutions. The first, coming in at about $55USD was a computer power supply ($9), wired with +12,-3.3 to an Apple 5 charge controller ($22 in USA, $36 here!) and a 7Ah battery ($10). We originally tried to save on the power supply by using the Ubiquiti PoE bricks to drive the charge controller, but they were nowhere near up to the task. Here's Nick hacking the power supply together (or apart, as the case may be):

The Second, coming in at $38 was a good ol' Chinese-brand UPS with the same battery capacity.

I figured the OTS UPS would perform markedly less well given the extra DC-AC-DC conversion, but the results were more striking than I imagined:

With the same load (roughly 6W), the OTS UPS lasted less than three hours (partially because the UPS doesn't overcharge the battery quite as much) while the Apple+some crap lasted longer into the night than we cared to stay. (X axis on graph is fraction of a 24-hour day) Needless to say we're going with the former solution.

A side benefit to the above experiments is some hands-on education with LVD circuits (because I'm certainly no electrical engineer...). A previous worry that had kept us from building the LVD circuit is that if the battery drained too far and became too strong of a load when the power returned, it would hang the routers. It turns out, that with LVD and a reasonably strong power supply, the internal resistance of the charging battery is sufficient to bring the line voltage right back up to 14ish volts and turn the electronics right back on when the mains kick in.

With a little fabbing, we could probably bring the cost of the UPS down to $25.

Sneak Previewz

noreply@blogger.com (Keith Berkoben) — Fri, 21 Jan 2011 19:00:00 +0000

Sooo, Clooooooose....