Holden's Blog

Whats new in Spark this week #1

2013-05-06T23:11:00.003-07:00

Whats new in Spark will look the activity in the Spark commit logs every week and attempt to summarize what new features and bug fixes have occurred. This not intended to summarize everything, mostly things that might be useful to application developers. Without further ado lets get started:

zipPartions was added to the Scala & Java APIs. zipPartions joins an up to 4 RDDs using a user supplied function. It requires that all of the RDDs have the same number of partitions with no such requirement on size within each partition. https://github.com/mesos/spark/commit/c9c4954d994c5ba824e71c1c5cd8d5de531caf78 https://github.com/mesos/spark/commit/c9c4954d994c5ba824e71c1c5cd8d5de531caf78
unPersist() was added to the Scala & Java APIs allow the removal of an RDD from persistence once it is no longer needed. https://github.com/mesos/spark/commit/93091f6936262a4006d875bf69b3f8c31c291617
Bugfix was added to validate that local directories can be created when being added. https://github.com/mesos/spark/commit/c9c4954d994c5ba824e71c1c5cd8d5de531caf78
Spark’s block UI manager had a bug with Spark Streaming blocks which was fixed https://spark-project.atlassian.net/browse/SPARK-740 https://github.com/mesos/spark/commit/538ee755b41585c638935a93ec838b635149f659
The shuffle writer now looks at spark.shuffle.file.buffer.kb to determine the buffer to use. Previously the default buffer was 8kb which could cause a lot of unnecessary disk seeks, and the new default is 100kb. https://github.com/mesos/spark/commit/1055785a836ab2361239f0937a1a22fee953e029

That is all that I found interesting in skimming this weeks commit logs, if I missed something important feel free to let me know :)

Automatic spelling corrections on Github

2011-08-28T15:07:00.001-07:00

English has never been one of my strong points (as is fairly obvious by reading my blog), so my latest side project might surprise you a bit. Inspired by the results of tarsnap’s bug bounty and the first pull request received for a new project(slashem - a type safe rogue like DSL for querying solr in scala) I decided to write a bot for github to fix spelling mistakes.

The code its self is very simple (albeit not very good, it was written after I got back from clubbing @ JWZ’s club [DNA lounge]). There is something about a lack of sleep which makes perl code and regexs seem like a good idea. If despite the previous warnings you still want to look at the code https://github.com/holdenk/holdensmagicalunicorn is the place to go. It works by doing a github search for all the README files in markdown format and then running a limited spell checker on them. Documents with a known misspelled word are flagged and output to a file. Thanks to the wonderful github api the next steps is are easy. It forks the repo and clones it locally, performs the spelling correction, commits, pushes and submits a pull request.

The spelling correction is based on Pod::Spell::CommonMistakes, it works using a very restricted set of misspelled words to corrections.

Writing a “future directions” sections always seems like such a cliche, but here it is anyways. The code as it stands is really simple. For example it only handles one repo of a given name, and the dictionary is small, etc. The next version should probably also try and only submit corrections against the conical repo. Some future plans extending the dictionary. In the longer term I think it would be awesome to attempt detect really simple bugs in actual code (things like memcpy(dest,0,0)).

You can follow the bot on twitter holdensunicorn .

Comments, suggestions, and patches always appreciated. - holdenkarau (although I’m going to be AFK at burning man for awhile, you can find me @ 6:30 & D)

I <3 Topatoco

2009-10-18T23:58:00.000-07:00

I just had a wonderful customer service experience with them, and I feel the need to share and let everyone know how awesome Topatoco is.

Topatoco sent me what has to be one of the best packages I've received in a long time. After some initial problems with my first bear-monster hoodie, they mailed a replacement, which (thanks to USPS) got mixed up, but no worries, they re-sent it. With buttons! and stickers :) And most of all a hand written note.

Translation of Pigs Can Fly Site Monitor

2009-08-06T21:37:00.000-07:00

Pigs Can Fly Site Monitor is now available in Spanish and French. The French translation is a bit more shifty than the Spanish one. If you find a translation error, drop me an e-mail , holden@pigscanfly.ca .

Pigs Can Fly Site Monitor Notification

2009-08-02T18:58:00.001-07:00

Pigs Can Fly Site Monitor Notification
Originally uploaded by dmcopernicus

I'm pleased to announce the launch of Pigs Can Fly Site Monitor for the Google Android platform, a free application. Pigs Can Fly Site Monitor (pcfsm) behaves like other regular site monitoring software, polling your website at customizable intervals to ensure it is online. Since the site monitor runs directly on your phone, you don't have to worry about haveing a second computer to act as the monitoring station, or any difficulties with slow SMS delivery.

Pigs Can Fly Site monitor is available from the Google Market for free. Users with the Google Market on their Android can download it by click here or looking under the Tools section in the Google Market on their Android phone. For users without access to the Google Market place (like NeoFreeRunner uses such as my self) you can install it from pcfsm.com.

In addition to the traditional polling, PCFSM also handles basic regular expression matching, and can optionally check if your site is linked from slashdot or reddit (as being linked to from there may cause massive spikes in traffic).

If you don't have an unlimited data plan, I'd recommend setting the polling interval to a very high number, on the other hand if you do have an unlimited data plan go wild :)

PCFSM is still pre-1.0, so there may be some bugs. If you find any please e-mail me at holden@pigscanfly.ca , make sure to include PCFSM in the subject so that I notice it.

DeviceScape now available on the OpenMoko

2009-07-24T17:44:00.001-07:00

I'm pleased to be able to post the DeviceScape ipkg's of DeviceScape for download. The binaries consist of two packages devicescape (also mirrored on the csc) and a different wpa version(or on the also mirrored on the csc). It has been tested on the ASU software image of the OpenMoko.

While I will respond to bug reports ( holden@pigscanfly.ca ), this is likely the end of the line for this software package.

It seems like the OpenMoko software stack doesn't have a lot of life left in it, and I've got an application (involving site monitoring) I'd like to write for the Android. As such I'm going to be putting Android on my FreeRunner and hopefully crank out that application that this Saturday/Sunday.

xkcd404 - the xkcd that wasn't

2009-07-15T09:32:00.001-07:00

xkcd404 - the xkcd that wasn't
Originally uploaded by dmcopernicus

I was reading the xkcd archives to switch my mind away from math (temporarily) and I was reminded of xkcd #404 . xkcd comics are of the from http://xkcd.com/[comicnumber] and #404 just gives back a 404 error.

update to web2.0collage

2009-07-08T08:03:00.000-07:00

my web2.0collage
Originally uploaded by dmcopernicus

I've updated web2.0collage, it now uses stateless web servelets (thanks to a patch in plt scheme :)), which gives the load balancer a much easier time, sine session are no longer sticky. It is covered in a few other places, besides slashdot, now (like gigaom/nyt) and being the hobo that I am, I'm trying to see if I can get the story up on digg ftw :)

If you run into any bugs with it (which is likely) or have suggestions I'd love to hear them :)

Your browser history is showing (an open source web application in scheme)

2009-07-02T10:36:00.001-07:00

my web2.0collage
Originally uploaded by dmcopernicus

Over the course of last weekend I wrote web2.0collage, a browser history sniffing collage generator in scheme. Web2.0collage is designed to graphically illustrate just how easy it is for sites to determine what your browser history is. When you visit the site it sniffs your browser history, and creates a collage of the (safe for work) sites that you visit. It is an interesting application of potentially scary technology (imagine a job application site using this to screen candidates). Ideally, given some time in my schedule, I'd like to make it a bit more user friendly and robust so that I could perhaps show it to the general public to increase awareness of privacy issues on the web.

The code, while not good since I was learning how the plt-webserver & imagemagick bindings worked at the time, is available under the agpl. Today it hit the front page of slashdot, causing some less than fortunate scaling issues to be discovered. Hatguy & myself managed to fix them (sort of) without too many interruptions.

Devicescape, OpenMoko, StarBucks & Boingo mobile

2009-06-23T14:45:00.000-07:00

I finally got a replacement battery for my FreeRunner allowing me to perform a rather important test, namely Starbucks support. Unfortunately the Canadian Starbucks use a different Wi-Fi provider than the American Starbucks, so the free wifi login support with Devicescape doesn't currently work. However, Boingo has a free 30 day trial for boingo mobile, which is a roaming partner with Bell (one of the Canadian Starbucks wireless providers) and Devicescape does support boingo hotspots.

Much to my pleasant surprise, the existing code worked with only a few minor modifications. I came across and fixed a minor bug involving not being able to stop the connection process, so you can take back over manual control if you so desire. Once again, if you are interested in testing this release give me a shout ( holden@pigscanfly.ca ), make sure to include openmoko in the subject somewhere so it gets through.

Now that my FreeRunner is working again I'm hoping to get a UI prototype up at the end of next weekend or two.

First glance at su.pr

2009-06-10T17:12:00.000-07:00

After reading about su.pr I was interested in giving it a spin (especially since I need to be concentrating on abstract algebra, I needed to take it for a spin right away). They posted beta invite code to the stumbleupon twitter feed. Of the features mentioned in su.pr's initial press release, all seem to be functional with the notable exception of "seo friendly" links (aka 301 redirects) so that search engines count the links as going to your site and shorten on your domain. I haven't had a chance to try out any of the other features, like suggested posting times, as it seems like they are tailored to each user so it requires a bit of data first. The settings panel seems a bit buggy (I haven't managed to get it to add more than one site that I'm "promoting"), but that seems like an easy fix. Overall I'm not entirely sure what all the buzz was about, it seems kind of cool but lacks sufficient compelling features to convince me that its not a bad thing to use a url shortener.

Less than fun server times

2008-12-29T13:50:00.000-08:00

So it looks like my host pulled the plug on my server on the morning of the 27nth. There appears to be some mix up with finding my account, which could be anything from a mix-up to "oops we wiped that server". Since its been awhile I've decided that the server probably isn't coming back online anytime soon, so I've got a VPS set up. Most of the stuff is backed up, sadly back in Waterloo. Fortunately I learnt from my more recent laptop failures and the most important bits (namely my delicious code) is in a variety of locations (three cheers for git :) ).
On the plus side I've lost a lot of cruft of configuration that had built up over the years, but on the downside I've got a lot of configuration and sys admin work to do for the next couple of days.

Almost done with interviews

2008-11-22T17:13:00.000-08:00

Interviewing for full-time is quite different than what my Co-Op interviews have prepared me for. For one thing, companies are much more interested in having you come on site, which is pretty cool in that I've gotten to see a lot of different work environments, but also has the downside of keeping me busy flying all over. Fortunately, I've managed to get the remaining 2 west coast companies I'm interviewing with to co-ordinate so I don't have to make separate trips out :) I was a little worried with job hunting during this economic slump, but it seems like most technology companies are still hiring (albeit maybe not as many people as before). Having the Amazon offer has made the whole process much less stress-full in some ways, but in other ways its made my schedule a lot more packed since the deadline is the end of this month.

Random beer

2008-11-21T21:29:00.000-08:00

Oddly enough a lot of people from Ottawa end up going to the University of Waterloo (or at least they seem to, in the Math/CS segment). Apparently, I am so far out of touch with Ottawa that I didn't know about the creation of a new brewery (called beaus) (complete with blog). Kevin was kind enough to bring down a big (~2L) jug of "Lug Tread" which was surprisingly good. So that this isn't a total non-sequenter with the rest of what I write, I wonder what sort of challenges they faced doing a startup and how those compare to tech startups? And now back to that free beer....

Update Yahoo! Zimbra Desktop vulneraible to Man in the Middle

2008-11-21T03:23:00.000-08:00

Once again, Yahoo! has made a slight mis-step with protecting their users' information. In my attempt to enable interoperability between pcfspam & Yahoo! Mail, I uncovered another problem with the most recent Yahoo! Zimbra Desktop. The new Zimbra Desktop (build 1344) uses the same login methodology as the web login, which is already known to be replayable. Unfortunately, unlike the web login, it doesn't notify the user in the event of an SSL certificate mismatch. This makes Yahoo! Zimbra vulnerable to a man-in-the-middle attack, exposing both usernames and passwords.

To reproduce this bug, simply download Zimbra desktop & set your host file (/etc/hosts) for login.yahoo.com to point to your local machine (127.0.0.1) by adding:

127.0.0.1 login.yahoo.com

Alternatively, you can configure bind and add the Yahoo! zone:


;
; BIND data file for the fake yahoo zone
;
$TTL 604800
yahoo.com. IN SOA localhost. root.localhost. (
;@ IN SOA localhost. root.localhost. (
         2  ; Serial
    604800  ; Refresh
     86400  ; Retry
   2419200  ; Expire
    604800 ) ; Negative Cache TTL
;
yahoo.com. IN NS ns1.yahoo.yahoo.com.
login.yahoo.com. IN A 127.0.0.1
login.yahoo.yahoo.com. IN A 127.0.0.1
ns1.yahoo.yahoo.com. IN A 127.0.0.1

. Then start an SSL webserver (I used apache) on port 443 and take a look at the access log to see the request:

127.0.0.1 - - [21/Nov/2008:00:27:39 -0500] "GET /WSLogin/V1/get_auth_token?appid=0YbgbonAkY2iNypMZQOONB8mNDSJkrfBlr3wgxc-&login=albertsanchezo&passwd=kingof HTTP/1.1" 404 401 "-" "Jakarta Commons-HttpClient/3.0"

You can clearly see the variables login & passwd contain the username and password. It should be noted that no warning message was shown to user and this was done with a self-signed cert for a localhost.localdomain. A malicious attacker would have to exploit only one of the many DNS poisoning vulnerabilities and pass the authentication information through to be able to capture the usernames & passwords of a large number of Yahoo! users. While you can see that I didn't bother passing the information through, you could also get a similar effect with squid (or another proxy) and still allow authentication to complete.

The impact of this is much lower than the previous vulnerability with Yahoo! Zimbra desktop, but is still serious as it exposes usernames & passwords with only a trivial amount of effort.

At the time of the writing Yahoo! security has been notified.
p.s.
For anyone from Yahoo! reading this, I'm still waiting for the shirt I was promised from the first time I reported a vulnerability, but its all good :)

Blog comment spam

2008-11-17T09:29:00.000-08:00

I seem to be getting a reasonable amount of blog comment spam (especially on the older posts). I've allready enabled captchas, but that apparently isn't enough. Since there aren't many comments I'm turning on comment moderation. I will let anything through which isn't spam.

Upgrading to 8.10 & random

2008-11-11T20:09:00.000-08:00

So I upgraded my main laptop to Ubuntu 8.10 (). The initial estimated upgrade was approximately 8 hours, so I headed into campus (where the main csclub & Canadian ubuntu mirror server is) and did my update in about 30 minutes instead.
This rest of this month is incredibly busy with trying to finish up interviews before the Amazon deadline.
After this weekend I'm hoping to have a working CLI port with login functionality for Device Scape on the OpenMoko.

tastey tasey wireless bits

2008-11-02T17:11:00.000-08:00

In what came as a bit of a surprise my latest device scape build seems to be working pretty well. It successfully selected the correct network, and did the automatic login at a Starbucks in NYC. I've got another round of code cleanups to do, and there is only a CLI interface at present, but I'm hoping to have some ipkgs ready for testing soon. If you're interested in taking them for a spin, send me an e-mail ( holden@pigscanfly.ca ) and I can send you the freshest bits :)

More Yahoo! funtimes, this time with the iPhone

2008-10-08T09:54:00.001-07:00

iPhone Sends Yahoo! Mail
Originally uploaded by dmcopernicus

You may remember, my previous blog post Another security diversion, Yahoo! Zimbra client exposes passwords in the clear over the wire (also yahoo IMAP access now available with some fudging). It turns out that more than just Yahoo! Zimbra Desktop is effected by this security oversight, although not to quite the same degree. After reading a post to the Zimbra forums suggesting Yahoo!'s iPhone applications use the same servers, I became worried that Yahoo!'s iPhone application might be affected by this.

I decided to enlist the help of my friend, Jerry (who has an iPhone), to confirm my suspicions. It turns out that Yahoo!'s One Connect application is secure. Sadly, it turned out that the iPhone mail application (with its pre-sets for Yahoo!) also fails to use encryption for everything but authentication. This means that, on the iPhone your username & password is secure. However, every e-mail (which is automatically downloaded) is transmitted over wireless (remember iPhone guys) in plaintext. In addition, doing a bit sleuthing reveals that Yahoo! is sending the outgoing mail over HTTP (you read that correctly, HTTP), and is in plaintext as well. This caught me by surprise, as I was expecting SMTP traffic. You can see the two captures of it sending & receiving here

Any e-mails from your bank, employer, girl/boy friend, is now visible to anybody with a laptop sitting in the same Starbucks as you. You are much safer if you only check the mail over the cell networks, but for those of us in countries without unlimited data plans, that isn't much consolation. In the opinion of this author, this is a phisher's wet dream.

Yahoo!'s security contact has been informed of these issues and I'm told there are no present plans to add encryption, however it is something which they would like to do at some point. Maybe if enough people point out that they don't like people snooping on their e-mail we could see this changing.

Another security diversion, Yahoo! Zimbra client exposes passwords in the clear over the wire (also yahoo IMAP access now available with some fudging)

2008-09-26T21:59:00.001-07:00

Capture of Yahoo! Desktop transmitting auth information
Originally uploaded by dmcopernicus

I've never really intended for this blog to be about security, but sometimes it just lands in your lap.

Taking a break from my regular coding and school work, I went to the Yahoo "hacku" day in Waterloo. I wrote a basic system to help me deal with the problem of false negatives in e-mail spam which I'm planning on improving on. Since, like the majority of students I know, I use Gmail I initially made my program work with gmail. However, since the food was being purchased by Yahoo, I figured I should try and make my system work with Yahoo as well.

At first glance, it didn't seem possible. Yahoo! doesn't presently offer IMAP support, and all the cool parts of there mail API require a pro account (which I later got, but didn't have at the time). Doing some digging, suggested that Yahoo did syncing for the new Yahoo Zimbra desktop product over IMAP, which wasn't available to others. So I downloaded the Linux binary and with a bit of help from my good friend netstat found the imap host (not surprisingly imap.mail.yahoo.com ). If it had worked, all would have ended there without digging my nose around any further. Sadly, the server didn't want to talk to my client.

I figured I would look at the difference between what my client was sending and what the Zimbra client was sending. Launching wireshark and looking at port 143 quickly lead to two important discoveries.

1) The Yahoo! imap server's require that you announce you are Zimbra (just send [ID (guid 1 os "Linux" "os-version" "2.6" "vendor" "Zimbra")]) before auth :P
2) The Yahoo! imap server's used by the Yahoo Desktop don't support SSL and the password was being transmitted in plain text

Since it was about 5am at this point, the implications of #2 didn't really hit home until after taking my pre-class nap.

What does this mean for you? If you use Zimbra to access your Yahoo mail, you almost certainly need to change your password and stop using Zimbra immediately (especially if you've ever done so over wireless).

P.S.

Sadly, my hack didn't end up placing. In retrospect it probably wasn't the best forum to bring up the security defects, but it was the most convenient. I did however get some free pizza out of it (although not enough to have leftovers :( ). The hackday brought forward a lot of interest people into writing interesting code, I certainly hope to see more of these (sponsored by Yahoo or otherwise) in the future.

I'm planning on adding a number of additional features and rolling out my anti-spam code slowly. If you're interested in hearing more about my not exactly a spam filter you can sign up for a mailing list at pcfspam.com or just subscribe to my blog since I will likely post updates here as time goes on.

I'm super excited to begin work on this project [porting DeviceScape to the OpenMoko]

2008-08-18T15:20:00.000-07:00

You may remember awhile back I wrote about hopefully being able to announce an interesting project I was working on. Well it took a bit longer to sort out all the details than I originally thought it would, but everything looks good to go :)

I'm super excited to begin work on porting Devicescape to the OpenMoko. Devicescape is one of the applications which I used heavily on my previous Windows-Mobile phones. It automatically sign on to Wi-Fi systems (such as FON,Starbucks, and more importantly for me Waterloo). Since I'm too cheap to have a big (or really any) data-plan this is how I plan to be getting my e-mail and pretty much everything on my OpenMoko. For now I'm going to be targeting the OM2008 image since it seems to provide the right mixture of bleeding edge while still being functional.

To the best of my knowledge this is the first (or one of the first) non-FIC commercial applications being ported/developed for the OpenMoko/FreeRunner stack.

Thinking of neat things to do with this, it looks like it might even be possible to trigger wake up from wi-fi so that the phone could wake up, log on, grab data intensive stuff (like say e-mail attachments or maps) and then go back to sleep. Although I'm not sure how much power would be drawn during this, it might be a bit too high to be feasible.

I'm still going to continue to work on my other side projects, including my spam filtering work, but there are only so many hours in the day (even with coffee) so they will probably slow down a bit.

Updated parallel blacklist lookup

2008-08-15T20:09:00.000-07:00

I've expanded the blacklists quiried and added a few more tests. I've also re-factored some of the code so doing matching with masks is much cleaner. You can grab the latest version of dnsrbl from hackage Sadly I lost the comments that I got on irc from an untimely combination of server reboot (with my screen session) and laptop hoboing. If you have any comments on how I can improve this drop me a line at holden@pigscanfly.ca .

For the next bit of my spam filter funtimes I'm planning on playing around with some python code, which should be a fun learning experience (although I feel I'm a bit late to the python party).

Dodgy facebook application now finished!

2008-08-10T03:45:00.000-07:00

I took a nice diversion from writing C good to write a facebook application with my room-mate Jerry. It integrates Amazon wishlists into your profile. Its un-imaginatively called wishlist. Life is incredibly busy this week so I probably won't get anything else done until next week.

Its back!

2008-08-04T12:57:00.000-07:00

I've got my computer back. The repair job was done in the back of a land rover. With two 20 something guys standing around the back of a truck the police decided to take a quick look, but once they saw the computer all was well. This is one of the odder places I've had my computer repaired, but the new component seems to be working well. Now I can get back to fun coding times :)

Hardware failures for the loose

2008-07-31T21:40:00.000-07:00

My Dell laptop (which is my primary computer while I'm in the states) has died today. I'm typing this on a Asus EEE PC I bought the last time I had to send my regular computer in for service. Hopefully all goes well, but this means I've probably lost a weeks worth of code :(