Intridea - Company Blog

Get Clearer Code With Underscore.js's Chaining Syntax

2013-05-16T00:00:00Z

Recently, I was looking for a way to do a task N times in JavaScript (like Ruby's N.times method), which I found (_.times) in the Underscore.js library. This discovery exposed me to an alternate way to call the Underscore methods, using its chaining syntax.

If you do a lot of work with Backbone, this syntax will be familiar. In Backbone, you can call many of the Underscore methods on models and collections:

bookCollection.each( function(model) {
  console.log( model.get('name') );
});
// >> 'American Gods'
// >> 'Anansai Boys'
// >> 'Coraline'

But what might not be obvious is that if you want to iterate over an object with Underscore, you can do the following:

_(["American Gods", "Anansi Boys", "Coraline"]).each( function(book) {
  console.log(book);
});
// >> 'American Gods'
// >> 'Anansai Boys'
// >> 'Coraline'

Which is the same as doing this:

_.each(["American Gods", "Anansi Boys", "Coraline"], function(book) {
  console.log(book);
}
// >> 'American Gods'
// >> 'Anansai Boys'
// >> 'Coraline'

This becomes beneficial if you're working on a Backbone project and have a non-Backbone object that you want to iterate over. Going from collection.each(function(){}) to _.each(objs, function(){}) can be jarring, while _(objs).each(function(){}) is less confusing due to its similiarity.

Google Glass Hello World

2013-05-15T00:00:00Z

You have to start somewhere. We're excited to hear what co-founder Chris Selmer learns at Google I/O, but until then let's jump right in with a basic "Hello World" example.

To deploy this example you'll need access to the Mirror API Developer Preview. If you have access, go to the Google Glass Java Quick Start page and follow the instructions for configuring your environment. If you don't have access we've included the output so you can follow along.

Timeline items, aka "cards," display information retrieved from a REST endpoint. A card is the basic unit of visual display for Glass; think of it like a simple web page. It's easy to work with timeline items because they're represented as a JSON structure.

To insert a new card, POST your JSON to the REST endpoint. You'll need to take three simple steps: create the new timeline item, set the text, and execute the command. "Hello World" in Java looks like this:

TimelineItem timelineItem = new TimelineItem();
timelineItem.setText("Hello world");
service.timeline().insert(timelineItem).execute();

Which creates the following raw HTTP:

POST /mirror/v1/timeline HTTP/1.1
Host: www.googleapis.com
Authorization: Bearer {auth token}
Content-Type: application/json
Content-Length: 26

{ "text": "Hello world" }

Glass will add the card to your timeline, and your user will see "Hello World."

Design For Mobile To Sell Local

2013-05-09T05:30:00Z

When it comes to selling products on a local basis, mobile rules the day. Whenever I travel anywhere, the first thing I do is fire up the Yelp or Google Maps apps to find someplace to eat. Next I check out the websites of those places. If I like what I see, I've found my next meal; if not, I'm moving on to their competitors.

And I'm not the only one...

According to the latest research from The Nielsen Company, 45% of consumers use their mobile device first when searching for local products and services. Ultimately, 53% of mobile users are making their purchases offline or in-store, and a whopping 74% of smartphone users complete their transactions offline.

What does this mean for you?

That means that your customers are searching for your products and services on their smartphones and tablets, and if they like what they see, they're buying. If not, they're heading to your competition.

Mobile Is Not A Choice

Providing a superior mobile experience, be it in the form of a mobile-optimized site, or a mobile application, is no longer a choice, it's a necessity. And with mobile Internet use set to surpass desktop Internet use in 2014, this is a trend that is rolling over companies.

And when I say "mobile-optimized site", I'm not talking about merely taking your entire website and making it mobile. What you need to take into account is how your customers use the mobile Internet, what devices they're using, what they expect when using those devices, and what information they need to move to a decision. This is where user experience design comes in.

To hammer this point home, we've taken the crucial findings of the research and turned into an infographic for you:

What Do You Think?

Is your experience the same as in the findings? When you're searching locally for products or services, does your mobile experience determine whether or not you visit a store?

Let's talk about it in the comments below.

Intridea Partner MyEnergy Acquired By Nest Labs

2013-05-08T10:00:00Z

Less than five years ago, two young entrepreneurs named Ben Bixby and Greg O'Keeffe came to Intridea with an ambitious goal: launching a platform for helping homeowners track and reduce their energy consumption. It was a great idea, but we needed to work fast to have it done by their Earth Day launch target. Ben and Greg knew they were asking a lot, but their enthusiasm and desire to make a positive change in the world were contagious. It was an exciting and caffeine-fueled time.

After months of hard work and close collaboration between the Intridea and MyEnergy teams, we launched the beta version on Earth Day 2009. It was just the beginning of great things for Ben and Greg. That’s why we’re just as excited today to congratulate MyEnergy on their latest success: their acquisition by Nest Labs, home of the learning thermostat.

When I look back, I always think of that beta version of the MyEnergy platform as a perfect example of rapid prototyping and agile design and development done right. Under a time crunch, both our teams had to work closely to build out the minimum viable product while also creating a robust system that wouldn’t fail under load. I’m really proud of the work we all did.

After the successful beta launch and helping MyEnergy recruit and hire an internal engineering team, we've continued to stay close to them, offering engineering advice and the occasional extra hand when a big development push was needed. It’s been great following their growth and getting to partner with MyEnergy throughout the years.

Here’s what Ben Bixby, CEO of MyEnergy, had to say back in 2009 about working with our team:

"Intridea is a great partner for any startup looking to get off the ground quickly. The Intridea team worked closely with us to build our beta solution on a highly accelerated schedule and then helped us construct an internal development team to continue on from there."

Nest Labs, with their vision of changing the way people think about energy use, is the perfect home for MyEnergy. The simple design and usability of the Nest thermostat shows that Nest values good design and quality engineering. MyEnergy and Nest Labs: the perfect pairing.

Congratulations to both the MyEnergy and Nest teams -- we look forward to continued success!

Get Mobile With MoDevUX 2013

2013-05-07T14:05:00Z

Go ahead and reach in your pocket. What's there? I'm willing to bet you've got a smartphone in your hand right now. And if I was even more of a betting man, I might wager that either you or someone at home also has a tablet of some sort.

The rise of mobile has been well documented. In fact, if the trend continues, mobile Internet use will surpass desktop Internet use in 2014. That makes it imperative for every business - your business - to have a mobile strategy. And that means providing an experience for people using smartphones and tablet devices.

But how to learn all this? Answer: MoDevUX.

MoDevUX is back with another excellent line-up of speakers, teachers, and hackathon judges for MoDevUX 2013. Starting this Thursday and lasting 3 days, attendees will hear from industry thought leaders, peers and up and coming stars.

We at Intridea are happy to sponsor MoDevUX for the second year in a row. Last year was awesome, and we're sure this year will meet the same expectations.

Here's a quick video about the conference:

Tickets are still available! Use code "GROUP5" for a 30% discount for five or more when you register today.

If you're attending come by the Intridea table and introduce yourself.

I'll see you there!

Day of Fosterly 2013 A Great Success

2013-05-06T11:45:00Z

We were very happy to sponsor Day of Fosterly 2013, which happened this past weekend. More than 600 attendees showed up, and you could feel the energy in the Artisphere. Congrats to Adam Zuckerman and the entire foster.ly team that made the weekend a great success.

A special shoutout to Dov and Brittany who helped get us everything we needed to make DOF2013 a success for Intridea as well.

Here's what went down...

Design, UX/UI & You

As part of a panel, our very own Jurgen Altziebler weighed in on current design trends and what should really drive design - the customer.

At the end of the day, everyone agreed that design is super important. Frankly I couldn't agree more :)

Here's a pic of Jurgen not on the panel, but giving Google Glass a try.

A big thanks to Stephanie Nguyen for letting us all geek out with the glass.

Workshop: How To Develop A Responsive App Quickly With DivShot And Rails

Tom Zeng and Brandon Hunter led this well-attended workshop, where they showed attendees how to quickly and efficiently prototype a 3-page responsive app built on Rails.

Check out their presentation:

If you haven't tried Divshot yet, sign up for the free beta. Build your UI using drag-and-drop, and then export to HTML and CSS.

Walt Mossberg Keynote Highlights

Walt Mossberg is the author and creator of the weekly Personal Technology column in The Wall Street Journal, a column that's appeared every week since 1991. He is also the co-creator and co-producer of the "D: All Things Digital" conference, as well as the executive editor of the All Things D website.

When Walt speaks, people listen, and listen they did - to his keynote. Here are the main lessons Walt gave us, paraphrased by me. Additional comments in italics.

Customer: focus on a single customer. The way I like to put this is that it's expensive to sell and market to everyone. Select a group of people to help, and focus on helping them.
Product: focus on your product and executing it; crappy products never win.
Curate: choose what to put in, and even more importantly,leave out.
User Experience: the business model is crucial, however the business model doesn't drive the product; the product is driven by user experience. If the experience isn't good, you're dead.
Mobile: mobile, mobile, mobile. If a site is not mobile-friendly, it won't be a very rich experience for mobile users. Build for mobile first - the world is becoming mobile centric, here in the U.S. and especially elsewhere in the world.
Ignore Computer Theology: If you spend a minute of your time in a theological debate about apps versus the web, iOS vs. android, you've wasted it. Spend your time figuring out how to reach people. People don't care what your app was written in. _At Intridea, we develop web applications in Ruby on Rails, along with other technologies including Node.js, BackBone.js, and mobile applications for both iOS and Android. So when it comes to web apps, we're a bit biased :)

See You Next Year!

If you missed DOF2013, be sure to catch us next year at Day of Foster.ly 2014. I'm sure it will be excellent and well worth your time.

Join Us For A Day Of Entrepreneurship And Learning At Day of Foster.ly

2013-05-02T09:30:00Z

Starting at 10 AM on Saturday, May 4th (2013), hundreds of entrepreneurs will gather at Artisphere for more than eight hours of panels, workshops, chats, demos, expo, job fair and more at Day of Foster.ly 2013. If you're interested in starting, growing, or exiting a company, this conference is for you. And if that's not enough to pique your interest, the keynote will be delivered by legendary tech journalist Walt Mossberg.

Join Intridea at Day of Foster.ly

We're happy to announce that three Intrideans will be presenting at DOF.

From 10:15am - 11:00am, Jürgen Altziebler, our Managing Director of UX, will be on the UX/IU panel. And from 2:15pm - 3pm, Brandon Hunter, one of our excellent UX/UI designers and Tom Zeng, our Director of Engineering, will be conducting a 45-minute workshop: How To Develop A Responsive App Quickly With DivShot And Rails.

Also, I'll be manning our table, so be sure to stop by and say hello.

See you on Saturday!

Switch Workshop: Progress Making Forces

2013-04-16T00:00:00Z

Thanks to 37signals and the Re-Wired Group for hosting the Switch Workshop in Chicago last week. Jason, Bob, and Chris introduced us to the Jobs-to-be-Done framework and explored what makes us buy the products and services we buy.

Buyers always switch from one product to another. For example, Facebook users didn’t engage in new social behavior; they simply switched from using the phone and email to using Facebook.

In a nutshell, the #JTBD framework focuses on four “progress making” forces. I’ll use Michael Bleigh’s recently funded Divshot as an example.

Two forces drive change:

Push of situation. “We need a better solution. Static wireframes aren’t getting it done.”
Magnetism of the new solution. “Divshot has drag-and-drop, and exports to HTML & CSS? Awesome!” This force is incredibly motivating for engineers.

Two countervailing forces block change:

Anxiety of the new solution. “If I move to this platform will it do all the things I need? Will it be around next year?”
Habit of the present. “I may not love Balsamiq for wireframes, but I already know it.”

Divshot has a low barrier to entry and the people who purchase it are also the people likely to actually use it (designers and engineers). If it turns out you don’t enjoy building your UI in the browser you can always go back to your previous tool. Thus, the “anxiety of the new solution” isn’t a big impediment to adoption. For bigger purchases such as custom enterprise software—especially when the purchasing manager and the user are not the same person—overcoming a manager’s anxiety of the new solution can be the greatest challenge to seeing your software adopted.

I’ve only scratched the surface of what we learned in the workshop. Next week I’ll blog about fundamental timelines and interview techniques that will fit perfectly with your existing user interview process. Jason Fried, Bob Moesta, and Chris Spiek are exceptional hosts and teachers. If you have a chance to attend a Switch workshop, it’s worth your time.

DC Council Data Set and the NEW Surfiki Developers API

2013-04-12T15:15:00Z

As I regularly do; look around for new datasets that I can explore and process with Surfiki, I came across the following:

The Open Code - Digital Copy of DC's Laws

As the author Tom MacWright mentions on his site:

"I couldn’t be happier to write that the project to bring DC’s laws into the digital era and put them in everyone’s hands made a big breakthrough: today you can download an unofficial copy of the Code (current through December 11, 2012) from the DC Council’s website. Not only that, but the licensing for this copy is officially CC0, a Creative Commons license that aims to be a globally-effective Public Domain designation."

That sounds like a GREAT invitation :) From my reading his post, it seems that this was difficult to acquire. He mentions many people, communications and time all working together to make this available to the public. He goes on to mention:

"What else is there to build? A great smartphone interface. Topic-specific bookmarks. Text analysis. Great, instant search. Mirrored archives to everywhere. Printable copies for DIY and for print-on-demand services. And lots more.

We’re not waiting till then to start though: dc-decoded is a project I started today to finish the long-awaited task of bringing DC’s laws into The State Decoded. The openlawdc organization on GitHub is dedicated to working on these problems, and it’s open: you can and should get access if you want to contribute."

As Intridea is a DC based firm, it made perfect sense for us to run this data through our own Data Intelligence Processing Engine; Surfiki. As well, it is a perfect opportunity to introduce the Surfiki Developers API. For which, we are making publicly available as of RIGHT NOW :) However, we are manually approving users and apps as we move forward. This, to assist us in future scaling and better insight in to the bandwidth required for concurrent and frequent developer operations. I encourage anyone and everyone to create an account. Eventually, all requests will be granted, and will be upon a first come first serve basis.

I think it is best that I first explain how we processed the data from The Open Code - Digital Copy of DC's Laws. Followed by a general introduction in to Surfiki and the Surfiki Developers API.

The initial distribution of The Open Code - Digital Copy of DC's Laws was in Microsoft Word Documents. This may have been updated to a more "ingestion" friendly format by now, although I am not sure. The total number of documents was 51, ranging in size from, 317K to 140MB. You may think, "Hey, that's NOT a lot of data"... Sure, that's true, however I don't think it matters much for this project. From what I gather, it was important to just get the data out there and available, regardless of size. As well, Surfiki doesn't throw fits due to small data or big data anyway.

First order of business was converting these Microsoft Word documents to text. While Surfiki can indeed read through Microsoft Word documents, it generally takes a little longer. Therefore, any preprocessing is a good thing to do. Here is a small Python script that will convert the documents.


#!/usr/bin/env python 
# -*- coding: utf-8 -*-

import glob, re, os
f = glob.glob('docs/*.doc') + glob.glob('docs/*.DOC')

outDir = 'textdocs'
if not os.path.exists(outDir):
    os.makedirs(outDir)
for i in f:
    os.system("catdoc -w '%s' > '%s'" %
              (i, outDir + '/' + re.sub(r'.*/([^.]+)\.doc', r'\1.txt', i,
                                   flags=re.IGNORECASE)))

With that completed we now have them as text files. I decided to take a peak in to the text files and noticed that there are a lot of "END OF DOCUMENT" lines. For this I assume is representative of singular documents within the larger contextual document. (I know, I know… Genius assumption :))

This "END OF DOCUMENT" looks like the following:


For legislative history of D.C. Law 4-171, see Historical and Statutory Notes following § 3-501.

DC CODE § 3-502

                Current through December 11, 2012                               

END OF DOCUMENT

And from my initial count script, there are about 19K lines that read: "END OF DOCUMENT". Therefore, I want to split these up in to individual files. The reason for this is I want Surfiki to process these as specific documents for search and trending purposes. Therefore, with the following Python script, I split them in to individual documents. As well, I cleaned out the '§' character. Note: Surfiki uses both structured storage and unstructured storage for all data. The reason behind this is both business purposes as well as redundancy. Business purposes, structured storage allows us to connect with common enterprise offerings, such as SQL Server, Oracle , etc., for data consumption and propagation. As for redundancy, since for a temporal period we persist all data concurrently between the two mediums, where process may abase, we can recover within seconds and workflow can resume unimpeded.

Note: docnames.txt is just a static list of the initial text documents converted from Microsoft Word documents. I chose that method rather than walking the path.


#!/usr/bin/env python 
# -*- coding: utf8 -*-

def replace_all(text, dic):
    for i, j in dic.iteritems():
        text = text.replace(i, j)
    return text

with open('docnames.txt') as f:
    set_count = 0
    for lines in f:
        filename = str(lines.rstrip())
        with open(filename, mode="r") as docfile:

            file_count = 0
            smallfile_prefix = "File_"
            smallfile = open(smallfile_prefix + str(file_count) + "_" + str(set_count) + '.txt', 'w')
            for line in docfile:
                reps = {'§':''}
                line = replace_all(line, reps)
                if line.startswith("END OF DOCUMENT"):
                    smallfile.close()
                    file_count += 1
                    smallfile = open(smallfile_prefix + str(file_count) + "_" + str(set_count) + '.txt', 'w')
                else:
                    smallfile.write(line)

            smallfile.close()
            set_count += 1

After the above processing (few seconds of time), I now have over 19K files, ranging from 600B to 600K, PERFECT! I am ready to push these to Surfiki.

It's important to understand Surfiki works with all types of data as well as locations of data. Web data (Pages, Feeds, Posts, Comments, Facebook, Twitter , etc.) As well it works with static data locations, such as file systems, buckets , etc. Streams and Databases… In this case, we are working with static data; text documents in a cloud storage bucket. Without getting to detailed in to the mechanism that I push these files, on a basic level, they are pushed to the bucket where an agent is watching and once they start arriving the processing begins.

Since this is textual information, the type of processing is important. In this case I want to use the standard set of NLP textual processing within Surfiki. Versus any customized algorithms, such as specific topic based classifiers, or statistical classifiers , etc. The following is what will be processed within Surfiki for this data.

Sentiment - Positive, Negative and Neutral
Perspective - Objective or Subjective
Gunning Fog Index
Reading Ease
Lexical Density
Counts including: words/sentence, syllables/sentence, syllables/word

As well, we provide the following for all data.

Keywords (Literal) - Literal extraction of keywords
Keywords (Semantic) - Semantic conceptual generation of keywords
Trends - n-grams - Uni, Bi and Tri
Trends Aggregate - n-gram weighted distribution
Graph - n-gram relationships/time (Available likely on Monday)
Time - Insert and document time extraction

These will all be available in the Surfiki Developers API

Once you go over to Surfiki Developers API and read through the documentation, you will find how simple it is to use. We are adding datasets on a regular basis so please check back often. As well, our near real-time Surface Web data is always available, as are previously processed data sets. If you have ideas or even data sets we should look at, please just let us know by submitting a comment on this post.

If you want to contact us about using Surfiki within your organization, that would be great as well. We can put it behind your firewall, or operate it in the cloud. It is available for most architectural implementations.

Contact Intridea

Ethics of Big Data - A Review

2013-04-04T13:55:00Z

Ethics; heavy :) How fitting is this quote from Ambedkar - "History shows that where ethics and economics come in conflict, victory is always with economics. Vested interests have never been known to have willingly divested themselves unless there was sufficient force to compel them."

Certainly, when discussing Big Data... We are indeed speaking of economics; therefore fitting. This might also explain why the book is less than one hundred pages. ;) I can't help but think this is "too soon". Too soon to be discussing Ethics in Big Data. Without any real hard examples in the wild to illustrate the risks, we are only guessing as to outcomes. It's difficult to establish what is "Ethical" until someone or something is hurt.

Ethics of Big Data - Balancing Risk and Innovation

Let's assume this is a primer, a primer for a larger and more in depth consideration sometime in the future. With that assumption, this book is a decent "quick" read. The book lays out the framework for Big Data risks from the eyes of the business, and should serve quite well as a catalyst for discussion.

As for Ethics, as stated above, I don't have a good perspective as to the range. The range of risk potential, the establishment of normalcy and extremes. So I am left just as I started. I understand that Information Ethics is an important topic, yet, I am unsure what it means within a data social context, nor a personal data privacy context. The book doesn't cover what I consider to be a real exploration of Ethics. Other than some rare tidbits, Example. Instagram in 2012 saying they own your photos and can do what they wish with them; we, the Big Data practitioners group and its consumers, lack causation upon this topic. It is important for you to realize that a large portion of Ethics have natural and innate constraints as defined by society. Therefore, there should only be a few, relatively speaking, areas where Ethics, or the question of Ethics is appropriate for data.

A distinction the book doesn't cover is that of innate privacy or personal privacy at all actually; very little. There is nothing un-ehtical or ethical in regards to innate privacy, as it exists with distinction by default. An example would be our Medical Records. As our records are being continually digitized, it is certainly becoming Big Data. However, what's innate about Ethics in this respect is EVERYONE feels their information is private. Therefore, there isn't a need to address the Ethics. Let's put aside conspiracy theory and potential for abuse as Ethics do not restrain those whom exploit in the first place. Medical Records is only an example. There are, indeed many more.

The question I want to ask is: What Big Data is actually subject to the standardization of Ethics? Which is contrary to the very definition of Ethics. "Ethics is the means in which we explore our personal morality". And I do mean, explore. As, our Ethics are subject to change, throughout history, Ethics is a moving target. Example. Roman Colosseum; Man vs. Beast etc. The books title is superfluous, a better title might have been: Privacy Practices for Big Data, or: Big Data, Business and You...

As you can see it is a passionate subject, as I find myself getting off track. That is a good example of what this book might illicit within yourself; questions of practice and judgement.

Using the example of Instagram and their photo debacle, was it ethical for them to post this information in their Terms of Use? Yes; it certainly was, and they did. While you may not like their decision, they were ethical in my opinion. You have a choice to use their service or not, they were honest in their intent. You may argue that the decision by them to claim ownership of your photos is un-ehtical. Is it? Is it really? They are a service that you choose to use, they have terms that they define, and you, need to abide by them, when choosing to use their service. That certainly seems ethical to myself. I can hear a few of you saying that Ethics should apply even when choice exists. That maybe so for some, but I am unconvinced. I acknowledge that Instagram wouldn't exist without users, and their decisions should be beneficial for themselves and for their users, However, it remains your choice to use their service.

Back to the book. If you want a Primer in regards to Big Data and some questions you should likely be asking yourself as a practitioner, this is a good start. I didn't seek information on the author until after I read the book. In my mind I kept saying, hopefully this author is a philosophy major or I am going to be a little agitated. Indeed the author does have a degree in philosophy. I can keep my criticism to a minimum.

This book was provided free, for purpose of review from O'Reilly Media

Practical Computer Vision with SimpleCV - Review

2013-03-29T20:20:00Z

Coming from having worked in CV for many years in varying degrees, as well, knowing some rather heavy weights working within it today, I was interested in this book from a review perspective.

Practical Computer Vision with SimpleCV

I found it encouraging that the author did dive right in with some high level informative definitions, common challenges and practical use case. Contrary to my encouragement is that it misses the mark with low level detail, theory and any real in depth explanation upon computer vision itself.

For a beginner, this is a decent title. Be aware though, if you are a beginner, you will need to embrace a quick rhythm and progression throughout. While it may lack in providing "an education in theory", it makes up for with "an education in application".

A few examples within the book did catch my attention such as the XBox Kinect material, which is quite relevant with the buzz surrounding the tech and its accessibility. A bonus here is the escape from Microsoft tools that some may feel make the Kinect undesirable. The clear examples in Python should address concerns with its use for practical training and application outside the Microsoft's Developer Tools ecosystem.

I enjoyed Chapter 7 (Drawing on Images) as this is where I spent a lot of time in the past with imaging annotations for medical applications. The ability to work with layers, objects, lines, etc. The author did a good job with describing the canvas but lacked in the actual drawing sections. The lollipop example was rather crude and SimpleCV's support for drawing is demonstrably more robust.

A lot of time was spent upon histograms which does make sense to me. I believe though, there was too much time spent on it. Which, will likely create some discontent with the quick progression I mentioned in a previous paragraph. I realize it is a vital concept/feature within CV and therefore the attention that was spent on it is good, however, not for this book.

I was disappointed to find there was very little in the book regarding SURF and SIFT, used for feature detection. (Arguably the most prominent CV industry application) Arguments regarding which algorithm's perform better or worse may have prevented their inclusion. As well, While SimpleCV doesn't have an implementation of SIFT, it does for SURF. In practice, within feature detection initiatives one of the two will typically be utilized. The following link is some open source SURF and SIFT with OpenCV work I and a friend worked on a few years back. Specific to feature detection around brand logos. LINK

Generally speaking, when you ask a beginner why they are interested in Computer Vision, most likely the answer will be something along the lines of: Facial Recognition and or Object Recognition. The author does provide examples and lesson within those more interesting facets. Therefore, it hits the mark for the intended reader.

This book was provided free, for purpose of review from O'Reilly Media

Open Analytics Summit Review

2013-03-27T10:35:00Z

I always love traveling to DC. Meeting up with fellow Intrideans is incredibly motivating and satisfying. I had this chance yet again over the last couple days for an event; Open Analytics Summit, whereby Intridea was a sponsor. As well, we had a speaking slot. A few other companies were sponsoring/speaking as well, such as Basho, Elasticsearch and 10Gen.

The morning started pretty slow, I am certain people were around the corner, just not sure which corner that was. It was an intimate setting, only a few tables for vendors/sponsors and a select group of practitioners.

Presentations were longer than what I consider normal, 45-50 minutes. Topics were focused for the most part upon Open Source software which included: Applications, Architecture, Engineering, Methods and Systems used within Data Analytics. As a sponsor it was difficult to get away and listen to the presentations, certainly a few of them looked as though they would have been quite interesting to have attended.

My presentation titled: Data Science in the NOW - It Takes an ARMY of tools! Focused mainly upon the vast array of available Open Source DB's, Indexing Engines, File Systems, Query Engines and Streaming Engines. As well, I spent a little time on the definition of "NOW" (within the context of data analytics), latency and our own human (physiological) limitations with perception. I made it a point to mention most that are available as well as their history, general feature set, strengths and weaknesses. I selected a few out of the myriad for special attention. Examples included: Storm, Cassandra, HBase, xQL's, Hadoop and a few others. The presentation is available for anyone to view on slide-share. Unfortunately without the notes attached it may seem a lot of the detail is missing. If you want to read through the notes that apply to each slide, please just let me know and I will send them to you.

My only gripe is the venue itself. While quaint, it had some real drawbacks. For example, power outlets in front of the vendor tables, rather than behind. A lounging area directly in front of the vendor tables whereby attendees backs were to us. Therefore making it rather difficult to engage in useful discussion. Finally the main presentation room is built upon tiers, much like you have experienced in large collegiate classrooms. However, the drawback was that attendee's each were behind a small barrier that hid their hands. With long presentations I noticed a lot of arm/elbow movement indicating QWERTY abuse or thumb wrestling rather than focus upon the presenter.

We met a few really cool people that we are already following up with. All in all it was a good event, glad we were part of it.

Small Multiples with Stately and jQuery

2013-03-25T00:00:00Z

Small Multiplies are a type of data visualization popularized by Edward Tufte, a pioneer in the field of data visualization and noted author in the field of information design.

Small Multiples are a series of small similar visualizations shown together that allow for a quick visual comparison.

We recently came across this great time-lapse from the Washington Post tracking flu levels by week for the 2012-2013 flu season and thought the data would make an excellent candidate for a small multiples graph using our open source font Stately and jQuery.

We start by adding the Stately assets to our project and adding the CSS and markup to the page. The data for the Washington Post time-lapse came from the CDC, so we grabbed the data and formatted it into some JSON for our needs.

First, we use jQuery to grab the JSON file using the $.getJSON method inside our Document ready function.

$(document).ready(function() { 

    $.getJSON("data.json", function(data){ 

    }); 

});

The first bit of data we need are the weeks so we know how many graphs we need. The relevant JSON looks like this:

"weeks":["10/13/2012","10/20/2012","10/27/2012","11/3/2012","11/10/2012","11/17/2012","11/24/2012","12/1/2012","12/8/2012","12/15/2012","12/22/2012","12/29/2012","1/5/2013","1/12/2013","1/19/2013","1/26/2013","2/2/2013","2/9/2013"]

With the jQuery $.each method, we can iterate over the each of the weeks and clone the map for each week week of data and also add a label.

$.getJSON("data.json", function(data){  
    $.each(data.weeks, function(i){
        $('#week').clone().attr( 'id', 'week_' + i ).appendTo('#container');
        $("#week_"+i+" h2").html(this); 
    });
});

Each state has data for every week to show it's flu level for that week, here is the JSON data for Alabama:

{"state":"Alabama","values":["minimal","minimal","minimal","minimal","minimal","minimal","low","high","high","high","high","high","high","high","high","high","high","moderate"]},

We need to look at each state and all of its values, so we use nested $.each methods to look at every state and iterate over its values for every week and add classes for every value.

$.each(data.states,function(i){
    $.each(data.states[i].values,function(x){
        $("#week_"+x+" li[data-state='"+data.states[i].state+"']").addClass(""+this+"");            
    });
});

The final jQuery looks like this

$(document).ready(function() {

$.getJSON("data.json", function(data){  
    $.each(data.weeks, function(i){
        $('#week').clone().attr( 'id', 'week_' + i ).appendTo('#container');
        $("#week_"+i+" h2").html(this); 
    });

    $.each(data.states,function(i){
        $.each(data.states[i].values,function(x){
            $("#week_"+x+" li[data-state='"+data.states[i].state+"']").addClass(""+this+"");            
        });
    }); 
});

});

You can you view the live example on the Stately GitHub page.

Rails Assets Prefix May Disable your Session

2013-03-20T15:55:00Z

I recently worked in a Rails project with Peter (@sporkd). The project is intended to be used as a sub-site, and should be served under sub-URI. After google, we ended up by setting config.assets.prefix and wrapped all routes in scope. The solution is simple and worked well. But soon, some weird bugs were found, and Peter was successfully isolated the problem to session (see demo sporkd/asset_prefix_test)

After several hours debugging, I finally found out the cause. To make a long story short, the routes configured in routes.rb should not start with config.assets.prefix, otherwise session save is skipped. The demo sporkd/asset_prefix_test can be fixed by simply setting config.assets.prefix to /one/assets. You also get a bonus by setting a unique prefix for assets, since it is easy to add expiration header for assets in web server.

If you are curious why config.assets.prefix can affect session and want to know some internals about the X-Cascade header in Rails, please read on.

X-Cascade Header in Rails

I never knew X-Cascade header in Rails before. @soulcutter has a post that described its usage.

The basic idea is this: if you return a response from a controller with the X-Cascade header set to "pass", it indicates that your controller thinks something else should handle the request. So rails (or is it rack? in any case...) will continue down your routes looking for the next rule that matches the request.

Indeed, X-Cascade is not only restricted in controller, if a mounted engine sets this header, Rails also continues down the routes searching.

It is a feature of Rails. Since 3.2, Rails has moved the routes logic to journey. The X-Cascade trick can be found in journey/router.rb#L69.

Pay attention that, the rack env object is shared when request is passed on. So if env is changed by former route, the latter one is affected. This is the root cause of the weird session issue, because session is controlled by env['rack.session.options'].

Sprockets, who skips the session

Sprockets, the gem for rails assets pipeline, mounts itself on config.assets.prefix and prepends the route to Rails. So if user accesses a page which path starts with config.assets.prefix, sprockets always processes the request first.

Maybe for performance, sprockets disables session save by changing env['rack.session.options']:

env['rack.session.options'] ||= {}
env['rack.session.options'][:defer] = true
env['rack.session.options'][:skip] = true

The options are changed even when asset is not found. If so, sprockets returns 404 and sets the header X-Cascade. Then Rails passes the request to controller, and correct page is rendered as expected. However, since the session is already disabled by sprockets, the changed session in controller is never saved.

Because env is a shared resource between routes when X-Cascade is set, it should not be changed unless it has to. When asset is not found, sprockets should just pass though without touching env, so I submit a PR for it.

How we Debug

Peter and I worked in different time zones. He first found the session issue because several features related to session did not work. He made the demo sporkd/asset_prefix_test to isolate the issue using minimum code at the end of the day in his time zone and left me the message.

When my day started, I got the message and started debugging on session based on the demo in doitian/asset_prefix_test.

Because session store class is customizable, I inherited one from default cookie store and added breakpoints using pry. Soon I found out that options[:skip] was true, but I had no idea where it was set to true. Then I did a grep (using ag) in all gems, and fortunately, only sprockets has set this option to true. The remaining work was just figuring out why sprockets is invoked before controller action.

Introducing REBIN

2013-02-27T12:00:00Z

While building Surfiki, our real-time data intelligence engine, we realized that the logic contained in any number of natural language processing (NLP) and machine learning (ML) binaries is locked up on a server, with no access to the web. What if we could pass input parameters to a binary and express the output as JSON via a RESTful endpoint?

Say hello to REBIN.

REBIN was created to facilitate a simple method of exposing CLI applications–regardless of their language–to the web. Much of our current NLP and ML research is written in C and C++ because these languages offer the advantages of speed and scale.

But REBIN isn’t limited to compiled executables: as long as you can flag a script with +x you’re good to go! This means it’s easy to expose your R, Python, and Ruby libraries to an instant web service. With REBIN, we’ve included a visual dashboard where you can define your endpoints and executables. It’s trivial to create an endpoint:.think of REBIN as an instant web service for any binary or script.

REBIN is built on node.js and redis by Anthony Nyström and Jeff Baier. Hit us up on Twitter if you have questions or ideas, and grab the source on GitHub right now!

Remote teams aren't a perk or a fad, they're the future of work

2013-02-25T09:30:00Z

With Yahoo's recent decision to disallow any and all remote work the remote work debate has reached a new fever pitch. DHH weighed in with his thoughts mostly around trust and what signal you're sending your employees. To me the more troubling idea isn't about what signal you're sending to your employees but what path you're setting for your business. Remote work isn't a fad, it's a requirement for the future of productive creative work.

The oft-used phrase "software is eating the world" to me boils down to a core concept: software and automation are gradually (and in some cases quickly) replacing all forms of non-creative endeavor. That means that over time the number of "ass in a chair" jobs will dwindle to zero. Office environments are great for these types of jobs because the primary measure of output is time. Put more time in, get more work output. When that's the case, the best scenario is to optimize for maximum time output; work as long as possible with as few of breaks as possible.

In a creative environment output has only a passing correlation with time spent. Creative work depends on being able to work at the often unexpected moment of inspiration, not on a set schedule. Sitting in an office for most of the day is hardly a guarantee of productivity and in fact can oftentimes work counter to it; creative energies are drained by monotony.

Setting a company culture of "never work from home" is exactly equivalent setting a company culture of "be less creative." If a developer at Yahoo is struck by inspiration at 10pm while at home, she will have to decide whether she should head back into the office to get some work done or "maybe I'll just work on it tomorrow." If she has been required to sit at a desk for nine hours that day already, the latter becomes even more likely. Of course, the next day the inspired moment has passed and work output is at perhaps 10-20% of the productivity our theoretical Yahoo would have the previous night.

Over the next ten years most successful companies will work hard to create a culture that can run entirely without an office, even if there is still a company headquarters and even if that headquarters has a number of people working in it every day. Those companies that cling to a traditional mandatory-office model will find themselves increasingly unable to land creative talent and increasingly outmaneuvered by the creative, flexible businesses with a truly distributed culture.

Intridea is a long-time proponent of distributed teams. We collectively save more than 9,000 hours a year in commute time and have built a creative team spanning three continents. Also, we're hiring.

Client Side MVC with Backbone, CoffeeScript, and Rails

2013-02-19T09:00:00Z

As a backend developer, I always felt it's tedious and time consuming to build frontend UI interfaces, and coding interactive behaviors in JavaScript, jQuery, HTML and CSS with Rails MVC has not been one of my favorite things - until recently.

I do a lot of R&D prototypes, which requires building new apps quickly, and often times the apps have to look very good in order to generate that 'wow factor'. I can always get the job done by tweaking the JavaScript, HTML, and CSS here and there, but it just hasn't been nearly as satisfying an experience on the frontend as I've had with Rails on the backend. There just hasn't been a single omg, wow, I can't believe it can be done so elegantly moment on the frontend. Also, the constant context switching between JavaScript and Ruby only makes things worse.

This dilemma got alleviated for me last year when I found Sencha Architect, it allowed me to build frontend UI using the GUI editor very quickly and intuitively, and code JavaScript in small fragments in the IDE. By far, Sencha Architect has been the most productive tool for me in terms of building frontend UI. But there's still this context switching problem. I just love Ruby so much to want anything to do with JavaScript (when possible).

Enter Backbone.js, CoffeeScript, and Twitter Bootstrap. (thought I'd never want to use the 'Enter ...' expression, it always annoyed the heck out of me :))

This trio has been the closest to the Holy Grail of the frontend UI development for me so far.

Firstly, Backbone now provides structure to the front end apps with MVC (call it MV* if you prefer), making it a lot easier to develop and manage large scale client side MVC apps in the browser. Compared to the traditional server side MVC apps, the front end MVC single page apps are more responsive, and behave more like desktop apps. Then there is CoffeeScript with its similarly elegant syntax to Ruby that makes the context switching problem disappear, and more importantly, it enforces JavaScript best practices transparently (feel free to toss the JavaScript best practices and Nina books you bought).

Last but not least, Twitter Bootstrap with its beautifully (in the eyes of this beholder) crafted HTML/CSS components makes building awesome looking web apps a whole lot simpler.

For Rails developers, the backend MVC experience can be mostly carried over to the front end. For single page apps, server side MVC is largely not needed, especially the View part. And Rails is mostly an API server, which is just fine.

For behavior-driven development, there is Jasmine. When used with CoffeeScript, it's almost the same as RSpec. And I also like Sinon.js for subbing and mocking server responses.

For client side MVC, Backbone is certainly not the only game in town, we have more than a few choices. Below are the brief summaries of some popular ones, feel free to check them out:

Backbone.js - Simple, Easy, Popular, Flexible, Extensible
Ember.js - Data & View Bindings, Computed Fields, Rails like
Angular.js - Minimal Boilerplates, View Binding and DOM Abstraction, Extensible Markups
Spine.js - Simple, MVC, CoffeeScript, Rails like
Sencha - Full Featured, Comprehensive UI Components, Powerful Class System

All above are great choices for client side MVC along with some others. I like Backbone for its simplicity, especially when coupled with CoffeeScript, it's a joy to use.

If you are interested in learning Backbone, here are some basics and also check out the resources listed below:

Model - business domain models, key/value properties, events, persistence, validation
Collection - list of models, provides convenience methods for sort, find, filter, iterator, CRUD, event handling, and etc
View - Rendering templates, handle app & DOM events
Router - map routes(url patterns) to handlers
Template - UI pages that get compiled and rendered to the browser

And Backbone's MV* compared to traditional MVC in the context of Rails apps

Backbone	MVC
Model	Model
View Template(eco)	View(erb)
View + Router	Controller

Simplicity is an advantage of Backbone, making it easy to learn, and flexible and easy to extend. But for large scale apps, this simplicity comes with a price - a lot of boilerplate code.

To alleviate the boilerplate code problem, and also address some common issues facing large scale apps (memory management, complex layout, and etc), several higher level frameworks are available, below are three of them, feel free to check them out:

Backbone.Marionett http://marionettejs.com
Chaplin.js http://chaplinjs.org
Thorax (by Walmart Labs) http://walmartlabs.github.com/thorax

And I recently did a presentation at the DC Ruby User Group Jan 10, 2013 meetup on developing client side MVC apps using Backbone, CoffeeScript, and Rails, the slides can be found here http://www.slideshare.net/tomzeng/intridea-backbonerailsdcrug The sample app mentioned in the slides is available on Github at https://github.com/tomz/todo-backbone

Resources

Backbone.js http://backbonejs.org
CoffeeScript http://coffeescript.org
Backbone on Rails https://github.com/meleyal/backbone-on-rails
Jasmine http://pivotal.github.com/jasmine
Twitter Bootstrap http://twitter.github.com/bootstrap
Sinon.JS http://sinonjs.org

Using Storable and Faker to Create Mock Collections

2013-02-13T10:05:00Z

As part of my work, I often create prototypes of Rails applications. My preferred tool for doing this is Serve. But as excellent as it is, it's a very thin application with no persistence layer. To be honest, I don't really want a persistence layer at this stage. But there are times when I want to be able to iterate over collections of objects the same way that I would in a Rails environment. Creating an index view of subscribers is a great example.

Now, I could just draft this all as html:

...but that's time consuming. Especially if you have more than a few. Sometimes you want to stress test a larger data set (to show pagination, for example, or to work out JavaScript performance on larger data sets) or just give a better match to real data.

To accomplish this, I use the Storable and Faker gems.

Enter Storable

Serve provides a views/view_helpers.rb file, which is where I will store my ruby code. Since Storable can read YAML files (and because I do often use fixture data in real Rails apps), I use YAML as my data store.

Let's set up the data store first, using a Subscriber model. You can go about this one of two ways:

build your models by hand (the same way you would create fixtures for RSpec), or
randomly generate a set number of models.

You should choose your approach based on what your end goal is; since I'm just trying to quickly populate a lot of data, I'll go with some random generation.

Enter Faker

The ERB loop uses the Faker gem to create a first and last name, which it then uses to create the object's key and describe the object's attributes. This setup will create 10 of these objects. I could even use this later to seed fixture data for tests; in that case, I'd define those objects manually.

Now that we have the data store in place, let's do something with it: create some ruby objects:

This creates a class that we can marshal our data in to, enabling us to create our collection. The field definitions default to being Strings, but (as you can see with :id) you can specify a type using Ruby's hash syntax.

Massage that fake data!

I've also added a full_name method to the class, since I know that I'll want to display a Subscriber's full name in the "first_name last_name" pattern. You can define any number of methods that you want here.

Next is a self.bootstrap method, which accepts a file path as a parameter. This reads in the specified YAML file, processing it through ERB (the load command with the ERB wrapper can be replaced by load_file if you're just using straight YAML), and then creates a new Subscriber instance for each object defined in the YAML file.

Finally, there's the self.all method, which similarly to the Active Record finder method of the same name, without the database call. It will search the active Ruby object space and return all of the objects of the Subscriber class.

This implementation of #all is a bit crude, but it works.

Voila

With all of that taken care of now, we can finally get back to our view and do this:

...which is much easier to maintain, especially if that bit of HTML code is rapidly changing. Storing the data in the YAML file (especially inside of a loop), means that adding/removing/modifying the object's attributes is much easier to do as well, and with a lot less typing.

Introducing Yourtime.Intridea

2013-02-12T00:00:00Z

The average commuter spends eight days every year just traveling to and from work. Not us: every Intridean works from home. This means we saved an aggregate of 9,380 hours last year for the things we love instead of sitting in a car or a bus.

We wondered how our peers spend their time, so we asked. We didn’t know what to expect: skateboarding, civil war re-enactment, extreme knitting?

As it turns out, 40% of Intrideans mentioned spending more time with family: Ben got to see his son take his first steps. Linus started playing the piano again—after a fifteen year hiatus. Kathleen, Chris, Batman, Javie, Mike, Serign, and Tom all run. We’ve shared our stories in a mini-site built using maps from Tilemill.

In the coming weeks, we’ll dig deeper into the numbers. Yourtime is the first part of a platform we’re building to measure our lives, our health, and our carbon footprint. We’ll share our experiences as we grow, and soon you’ll be able to add your own company’s team to the board.

Come see how we spend our time. And if you’d like to recapture eight to ten days of your life every single year, come tell us how you’d like to spend your time.

Speed up your Android dev with x86

2013-02-07T19:35:00Z

A significant portion of my development time is simply wasted, and as they say “Time is Money.”

My Android development environment is Eclipse; not unlike most developers. However, I also find that debugging for Android on an AVD device is excruciating. The majority of my frustration comes from its lack of performance. And even after searching the web for tips and tricks to make AVD more responsive, I’m still twiddling my thumbs while the emulator tries to keep up with my requests.

If we could put men on the Moon and return them safely to Earth with little more than a TI-86 and duct tape, then surely there is a way to make my quad-core computer execute code with speed comparable to that of a mobile phone.

That was then…

After a lot of searching (because, you know, I had a lot of time between breakpoints), I pieced together information and resources from various places to come up with a better debugging emulator.

The solution that I found is an Android-x86 based system running inside of an Oracle VM VirtualBox. However, the key to this is Intel’s Houdini binary translator (libhoudini.so), which allows nearly-native execution of ARM code.

After downloading and testing many Android-x86 ISO’s, I found one that has fully integrated support for ARM right out of the box. Another key feature is that it’s not just a live image, but an installation disk as well. And as a bonus, it supports the Android market.

While I haven’t had an issue with compatibility yet, it is important that all final testing be done on AVD or a physical device before delivery.

The current version is based on Android 4.0.4 (Ice Cream Sandwich).

This first post is quite simple and we decided to just provide the iso already setup and ready to go :) Next week, I'll post how to do it from scratch!

http://goo.gl/oZ4kO