Symbian Planet Community

iOS 7: Using NSURLSession for Asynchronous Networking

Tasneem Sayeed — Thu, 19 Mar 2015 13:50:00 +0000

In iOS 7, Apple introduced NSURLSession, which is a suite of classes that replaces NSURLConnection as the preferred method of networking. In this article, we cover the benefits of using NSURLSession, and when and how to use it.

What are the benefits of using NSURLSession?

There are a number of new advantages and benefits of using NSURLSession:

Uploads and Downloads can be done in the Background

When the NSURLSession is created, a configuration option can be selected to allow background networking. This helps to save battery life, supports multitasking and allows developers to use the same delegate model as in-process transfers.

Allows the pause and resume for network operations

When the NSURLSession API is utilized, any networking task can be paused, stopped and restarted. There is no NSOperation subclassing required.

Configurable container for storing network request configuration

Each NSURLSession provides a configurable container for storing network requests. For instance, for setting an HTTP header option, we will need to only do this once and each subsequent request will utilize the same configuration

Allows a private storage which is subclassable.

Each NSURLSession is subclassable and one can configure a session to use private storage on a per session basis. This allows one to have private storage objects outside of the global state.

Improved authentication handling

When using NSURLConnection, if an authentication challenge was issued, the challenge would come back for an arbitrary request, so one would not know exactly what request was getting the challenge. However, with NSURLSession, the delegate handles the authentication.

Rich delegate model

NSURLConnection does have some asynchronous block based methods. However, a delegate cannot be used with them.

Uploads and downloads allowed through the file system.

This encourages a separation of the data (i.e. file contents) from the metadata (i.e. URL and settings)

NSURLSession

NSURLSession is designed as a replacement API for NSURLConnection. An NSURLSession is made using an NSURLSessionConfiguration with an optional delegate. After the session is established, the network requirements are satisfied by creating NSURLSessionTask.

According to Apple's URL Loading Programming Guide, you can use the NSURLSession API in two ways: with a system-provided delegate or with your own delegate. In general, you must use your own delegate if your app does any of the following:

Uses background sessions to download or upload content while your app is not running.
Performs custom authentication.
Performs custom SSL certificate verification.
Decides whether a transfer should be downloaded to disk or displayed based on the MIME type returned by the server or other similar criteria.
Uploads data from a body stream (as opposed to an NSData object).
Limits caching programmatically.
Limits HTTP redirects programmatically.

If your app does not need to do any of these things, your app can use the system-provided delegates.

Basic sequence of method calls that your app needs to make and completion handler calls that your app receives when using NSURLSession with the system-provided delegate :

Create the NSURLSessionConfiguration
Create a session specifying a configuration object and a nil delegate.
Create task objects within a session that each represent a resource request. The task objects are subclasses of NSURLSessionTask—NSURLSessionDataTask, NSURLSessionUploadTask, or NSURLSessionDownloadTask, depending on the behavior you are trying to achieve. These objects are analogous to NSURLConnection objects, but give you more control and a unified delegate model.
We need to have our delegate implement the required methods from the NSURLSessionDownloadDelegate protocol.
When a task completes, the NSURLSession object calls the task's completion handler.
When your app no longer needs a session, invalidate it by calling either either invalidateAndCancel (to cancel outstanding tasks) or finishTasksAndInvalidate (to allow outstanding tasks to finish before invalidating the object).

Important: If you are using the NSURLSession class without providing delegates, your app must create tasks using a call that takes a completionHandler parameter, because otherwise it cannot obtain data from the class.

Note: NSURLSession does not report server errors through the error parameter. The only errors your app receives through the error parameter are client-side errors, such as being unable to resolve the hostname or connect to the host. The error codes are described in URL Loading System Error Codes. Server-side errors are reported through the HTTP status code in the NSHTTPURLResponse object. For more information, read the documentation for the NSHTTPURLResponse and NSURLResponse classes.

Step #1: Creating the NSURLSessionConfiguration

There are three ways to create an NSURLSessionConfiguration:

defaultSessionConfiguration

creates a configuration object that uses the global cache, cookie and credential storage objects. This configuration provides a session to look very much like the NSURLConnection.

ephemeralSessionConfiguration

This configuration is for "private" sessions and has no persistent storage for cache, cookie or credential storage objects

backgroundSessionConfiguration

This configuration is used for making networking calls from remote push notifications or while the app is suspended.

Once the NSURLSessionConfiguration is created, then the setting of various properties on it can be done as follows:

NSURLSessionConfiguration *sessionConfig =
[NSURLSessionConfiguration defaultSessionConfiguration];
 
// restricts network operations to Wifi
sessionConfig.allowsCellularAccess = NO;
 
// sets all requests to only accept JSON responses
[sessionConfig setHTTPAdditionalHeaders:
          @{@"Accept": @"application/json"}];
 
// configures timeouts & restricts app to only have one network connection to a host
sessionConfig.timeoutIntervalForRequest = 30.0;
sessionConfig.timeoutIntervalForResource = 60.0;

sessionConfig.HTTPMaximumConnectionsPerHost = 1;

Step #2: Creating the Session

With NSURLSession, you can create the tasks using the block based convenience methods, set up a delegate or both. For instance, if you want to download an image, then you will need to create an NSURLSessionDownloadTask.

// set the image URL
NSString *imageUrl = @"http://someImage.png";
 
// create the default session configuration
NSURLSessionConfiguration *sessionConfig =
  [NSURLSessionConfiguration defaultSessionConfiguration];
 
// create a session using the current class as a delegate
NSURLSession *session =
  [NSURLSession sessionWithConfiguration:sessionConfig
                                delegate:self
                           delegateQueue:nil];

Step #3: Download the image by creating a task with a completion handler



// the task is created from a session
NSURLSessionDownloadTask *getImageTask =
[session downloadTaskWithURL:[NSURL URLWithString:imageUrl]
 
    completionHandler:^(NSURL *location,
                        NSURLResponse *response,
                        NSError *error) {
        // the image is uploaded as NSData
        UIImage *downloadedImage =
          [UIImage imageWithData:
              [NSData dataWithContentsOfURL:location]];
      // update UIImageView image to show the new file
      dispatch_async(dispatch_get_main_queue(), ^{
        // do stuff with image
        _imageWithBlock.image = downloadedImage;
      });
}];
 
// start up the task
[getImageTask resume];


Step #4 Implementing the delegate methods
We need to have our delegate implement some methods from the NSURLSessionDownloadDelegate protocol. For instance, 
we need to get notified when the download is complete:


-(void)URLSession:(NSURLSession *)session
     downloadTask:(NSURLSessionDownloadTask *)downloadTask
didFinishDownloadingToURL:(NSURL *)location
{
  // see code above from completion handler
}
Here we are provided with the location that the file is downloaded to so you can use it to work with the image.
Tracking Download Progress.
To track the download progress for either task creation method, do the following:
-(void)URLSession:(NSURLSession *)session
     downloadTask:(NSURLSessionDownloadTask *)downloadTask
     didWriteData:(int64_t)bytesWritten
totalBytesWritten:(int64_t)totalBytesWritten
totalBytesExpectedToWrite:(int64_t)totalBytesExpectedToWrite
{
  NSLog(@"%f / %f", (double)totalBytesWritten,
    (double)totalBytesExpectedToWrite);
}
Step #5: When the task is finished, URLSession:downloadTask:didFinishDownloadingToURL: is called. 
In the case of a file download, this is when you can save the file from the temporary location to a permanent one.
When the download fails or is cancelled, you can get the data to resume the download.
NSURLSessionTask
Both NSURLSessionDataTask and NSURLSessionDownloadTask are derived from NSURLSessionTask, which is the
base class as illustrated below.
Image: Courtesy of Ray Wenderlick's NSURLSession Tutorial

NSURLSessionTask

This is the base class for creating session tasks, which are generally one of the following subclasses.

NSURLSessionDataTask

This task is used for issuing HTTP GET requests to pull down data from servers.

The data is returned in the form of NSData, which needs to be converted to the appropriate

XML, JSON, etc. format.



NSURLSessionDataTask *jsonData = [session dataTaskWithURL:yourNSURL
      completionHandler:^(NSData *data,
                          NSURLResponse *response,
                          NSError *error) {
        // handle NSData
}];
NSURLSessionUploadTask

This class is used to upload something to a web service using HTTP POST or PUT commands.

The delegate for the tasks also allows you to watch the network traffic while it is being

transmitted.

Upload an image:

NSData *imageData = UIImageJPEGRepresentation(image, 0.6);
NSURLSessionUploadTask *uploadTask =
[upLoadSession uploadTaskWithRequest:request
fromData:imageData];
The task above is created from a session and the image is uploaded as NSData. Separate methods are available for uploading using files or a stream.

NSURLSessionDownloadTask
NSURLSessionDownloadTask makes it super-easy to download files from remote service and/or pause and resume the download at will. This subclass is a little different than the other two.
This type of task writes directly to a temporary file.
During the download the session will call NSURLSessionDownload URLSession:downloadTask:didWriteData:totalBytesWritten:totalBytesExpectedToWrite: to update status information
When the task is finished, URLSession:downloadTask:didFinishDownloadingToURL: is called. This is when you can save the file from the temp location to a permanent one.
When the download fails or is cancelled you can get the data to resume the download.
This feature can be useful for downloading photos to your device's camera roll.
We have covered the essential sequence of method calls required for NSURLSession when using a system provided delegate. For further thorough documentation and detailed information on the NSURLSession and associated APIs, please refer to the Apple's URL Loading System Programming Guide or Ray Wenderlick's NSURLSession Tutorial .
If you find this post helpful or want to share your experiences, feel free to send me comments here or follow me on Twitter @tasneemsayeed. Have fun with NSURLSession Coding!

iOS: How to Send Asynchronous Requests

Tasneem Sayeed — Tue, 17 Mar 2015 20:40:00 +0000

Many of the applications we build often have to fetch some sort of data through the network. While there are a number of ways one could fetch the data in our iOS application and a number of networking libraries available, it is still useful to understand how NSURLConnection actually works.

Step #1:
Need to have your class to conform to the NSURLConnectionDelegate protocol and declare a var to store the response data

Step #2:
Implement the NSURLConnectionDelegate protocol methods

Step #3:
Create an instance of NSURLRequest and NSURLConnection to kick off the request

In Step#1, when you class conforms to the NSURLConnectionDelegate protocol, it means that your class will need to implement all the required methods, which takes you to Step#2.
The idea is that when you kick off the request in Step #3, you want to be notified of events that will happen such as when the response comes back, and need to handle them appropriately.

Step #1:
Within the class that you will be using NSURLConnection, specify in the header file that it conforms to the NSURLConnectonDelegate protocol. Also, declare an instance variable for holding the response data.

Step #2:
Next, implement the NSURLConnection protocol methods

Step #3
Now, we are finally ready to perform the asynchronous request.
Let's start by creating an instance of NSURLRequest, and assign it to the URL. Next, create an instance of a NSURLConnection and call the initWithRequest method, passing it your NSURLRequest.

After the NSURLRequest is dispatched and the request is handled, the callback handler, didReceiveResponse will be invoked to indicate that the server has responded. Then, the callback handler, connection:didReceiveData will be invoked. This is where you will append the new data to the instance variable you declared. In connectionDidFinishLoading callback is invoked when the request is complete and the data has been received, so you can parse the response data and do whatever you want to do with the data. That's all there's to it!

In the next blog post, I will cover how to send an asynchronous request using NSURLSession and Blocks!

The fastest scope in the store… or Formula one in 5 minutes with Twitter

Wed, 11 Mar 2015 11:19:13 +0000

I recently blogged about making a scopes in 5 minutes using youtube. I have seen also a fair amount of new scopes being created using RSS. By far, my favourite way to use scopecreator is Twitter.

If you want to check a few examples, I have published previously twitter-based scopes like breaking news, la liga and a few others. Today, I give you Formula One:

The interesting thing about twitter is that many brands upload minute by minute new updates, which make a really good source for scopes.

To create a Formula One scope,I started by going to twitter and creating a list under my scope account (you can use your personal account). The list contains several relevant “official” Formula One accounts. Using Twitter, I can then update the sources by adding and removing accounts from the list without the user needing to download an update for the scope.

Again, it took me about 5 min to get a working version of the scope. Here is what needed to do:

First, we followed Chris’ instructions to install the scope creator tool.
Once we had it set up on my laptop, I run:
scopecreator create twitter vtuson f1 cd f1
Next, I configured the scope. The configuration is done in a json file called manifest.json. This file describes the content of what you will publish later to the store. You need to care about: “title”, “description”, “version” and “mantainer”. The rest are values populated by the tool:
scopecreator edit config { "description": "Formula One scope", "framework": "ubuntu-sdk-14.10", "architecture": "armhf", "hooks": { "f1": { "scope": "f1", "apparmor": "scope-security.json" } }, "icon": "icon", "maintainer": "Your Name ", "name": "f1.vtuson", "title": "Formula One", "version": "0.2" }

The following step was to set up the branding: Easy! Branding is define on an .ini file. “Display name” will be the name listed on the “manage” window once installed, and also will be the title of your scope if you don’t use a “PageHeader.Logo”. the [Appearance] section describes the colours and logos to use when banding a scope.
scopecreator edit branding [ScopeConfig] ScopeRunner=./f1.vtuson_f1 --runtime %R --scope %S DisplayName=Formula One Description=This is an Ubuntu search plugin that enables information from Yelp $ Author=Canonical Ltd. Art= Icon=images/icon.png SearchHint=Search [Appearance] PageHeader.Background=color:///#D51318 PageHeader.ForegroundColor=#FFFFFF PreviewButtonColor=#D51318
The final part is to define the departments (drop down menu) for the scope. This is also a json file and it is unique the twitter scope template. You can either use “list” or “account” (or both) as departments. The id is the twitter handle for the list or the account. For lists you will need to specify in the configuration section what account holds the list. As I defined a single entry, the formula one scope will have no drop down menu.
scopecreator edit channels
{
“departments”: [
{
“name”:”Formula One”,
“type”:”list”,
“id”:”f1″
}
],
“configuration”: {
“list-account”:”canonical_scope”,
“openontwitter”:”See in Twitter”,
“openlink”:”Open”,
“retweet”:”Retweet”,
“favorite”: “Favourite”,
“reply”:”Reply”
}
}

After this, the only thing left to do is replace the placeholder icon, with a relevant logo:
~/f1/f1/images/logo.png
And build, check and publish the scope:
scopecreator build

This last command generates the click file that you need to upload to the store. If you have a device (for example a Nexus4 or an emulator ), it can also install it so you can test it. If you get any issues getting the scope to run, you might want to check your json files on http://jsonlint.com/. It is a great web tool that will help you make sure your json doc is ship shaped!

It is super simple to create a scope for a twitter list! so what are you going to create next?

Make a scope for your YouTube channel in 5 minutes

Mon, 16 Feb 2015 14:00:26 +0000

Not long back Chris Wayne published a post about a scope creator tool. Last week, I was visiting bq and we decided with Victor Gonzalez that we should have a scope for Canal bq. The folks at bq do an excellent job at creating “how to” and “first steps” videos, and they have started publishing some for the bq Aquaris E4.5 Ubuntu Edition.

Here is a few screenshots of the scope that is now available to download from the store:

The impressive thing is that it took us about 5 min to get a working version of the scope. Here is what needed to do:

First, we followed Chris’ instructions to install the scope creator tool.
Once we had it set up on my laptop, we run:
scopecreator create youtube com.ubuntu.developer.victorbq canalbq cd canalbq
Next, we configured the scope. The configuration is done in a json file called manifest.json. This file describes the content of what you will publish later to the store. You need to care about: “title”, “description”, “version” and “mantainer”. The rest are values populated by the tool:
scopecreator edit config { "name": "com.ubuntu.developer.victorbq.canalbq", "description": "Canal bq", "framework": "ubuntu-sdk-14.10", "architecture": "armhf", "title": "Canal bq", "hooks": { "canalbq": { "scope": "canalbq", "apparmor": "scope-security.json" } }, "version": "0.3", "maintainer": "Victor Gonzalez " }

The following step was to set up the branding: Easy! Branding is define on an .ini file. “Display name” will be the name listed on the “manage” window once installed, and also will be the title of your scope if you don’t use a “PageHeader.Logo”. the [Appearance] section describes the colours and logos to use when banding a scope.
scopecreator edit branding [ScopeConfig] DisplayName=Canal bq Description=Youtube custommized channel Author=Canonical Ltd. Art=images/logo.png Icon=images/logo.png SearchHint=Buscar LocationDataNeeded=true [Appearance] PageHeader.Background=color:///#000000 PageHeader.ForegroundColor=#FFFFFF PreviewButtonColor=#FFFFFF PageHeader.Logo=./images/logo.png

The final part is to define the departments (drop down menu) for the scope. This is also a json file and it is unique the youtube scope template. You can either use “playlists” or “channels” (or both) as departments. The id PLjQOV_HHlukyNGBFaSVGFVWrbj3vjtMjd corresponds to a play list from youtube, with url= https://www.youtube.com/playlist?list=PLjQOV_HHlukyNGBFaSVGFVWrbj3vjtMjd
scopecreator edit channels{
“maxResults”: “20”,
“playlists”: [
{
“id”: “PLjQOV_HHlukyNGBFaSVGFVWrbj3vjtMjd”,
“reminder”:”Aquaris E4,5 Ubuntu Edition”
},
{
“id”: “PLjQOV_HHlukzBhuG97XVYsw96F-pd9P2I”,
“reminder”: “Tecnópolis”
},
{
“id”: “PLC46C98114CA9991F”,
“reminder”: “aula bq”
},
{
“id”: “PLE7ACC7492AD7D844”,
“reminder”: “primeros pasos”
},
{
“id”: “PL551D151492F07D63”,
“reminder”: “accesorios”
},
{
“id”: “PLjQOV_HHlukyIT8Jr3aI1jtoblUTD4mn0”,
“reminder”: “3d”
}
]
}

After this, the only thing left to do is replace the placeholder icon, with the bq logo:
~/canalbq/canalbq/images/logo.png
And build, check and publish the scope:
scopecreator build

It is super simple to create a scope for a youtube channel! so what are you going to create next?

uBrick – a Lego Scope

Tue, 20 Jan 2015 17:49:17 +0000

Just a quick note to tell you that I have published a new scope called uBrick that brings you the awesomeness of Lego, as a catalogue powered by brickset.com, directly to your Ubuntu phone home screen.

I wrote the scope in Go cause I find it easier to work with for a quick scope ( took about 8 hours with interruptions over 2 days to write this scope). The scope is now available at the store, just search for uBrick.

Here are some pics:

Also I have to congratulate the folks at Brickset for a very nice API, even if it is using SOAP :)

Common Workflow Language

John Kern — Fri, 16 Jan 2015 10:14:00 +0000

Analyzing genomic data requires one to use a sequence of programs (e.g., assemblers, aligners, variant callers) called pipelines. One needs to understand the details of these programs to get them to work well together. Since these genomic data sets are large, it is desirable to distribute the work across multiple machines.

A number of well-known platforms exists to provide this infrastructure.

Arvados

website ( https://arvados.org )
source ( https://github.com/curoverse/arvados )
License: Apache License 2.0

Galaxy

website ( http://galaxyproject.org/ )
source: ( https://bitbucket.org/galaxy/galaxy-central )
License: AFL 3.0.

Seven Bridges

website: https://www.sbgenomics.com/
License: Commercial

BioDatomics

website: http://www.biodatomics.com/
License: Commerical

Taverna

website ( http://www.taverna.org.uk/ )
source code: ( https://github.com/taverna/ )
License: LGPL
As of October 2014, this project entered the apache incubator project. See http://taverna.incubator.apache.org/. I assume the license is now apache.

DNAnexus

website ( https://www.dnanexus.com/ )
License: Commercial

The BOSC 2014 codefest gave birth to the Common Workflow Language(CWL) project ( group | github ). Its goal is to define a language which would allow pipelines to run on any compliant platform. In forthcoming posts, I plan to talk about our efforts to breath life into CWL and port bcbio-nextgen to it.

bcbio-nextgen: walk through of cancel/normal example

John Kern — Tue, 19 Aug 2014 14:40:00 +0000

On this blog, I've been documenting my experiences with bcbio-nextgen. With this post, I want to walk through and extend a well known, bcbio-nextgen example. It is called Cancer Tumor Normal. "This example calls variants in a paired cancer sample with tumor/normal sequencing data using raw data from Han et al in PLoS One." The current documentation does a good job telling you how to setup things up. Here I would like to inspect the input. Describe the output and suggest a way to visual it and highlight the use of the Gemini to export the database.

Bcbio uses a yaml files for configuration. Yaml files are nice because they provide a high level abstraction which bcbio translates into specific commands. Storing these configuration files in a configuration manage system provide a great way to track changes. There are two main configuration files. One details the samples. The other provides options for system software. We will focus on the sample file. Let's look at the important bits of the sample config file for this example. You find the full yaml file here. It defines two samples: normal and tumor. Here is the first sample. [aside: I have collapsed the algorithm options for now. ]

- algorithm:
...
analysis: variant2
description: ERR256785
files:
- ../input/ERR256785_1.fastq.gz
- ../input/ERR256785_2.fastq.gz
genome_build: GRCh37
metadata:
batch: batch1
phenotype: normal

The keywords for sample configurations are defined here. The algorithm keyword is important. We will address it in the next paragraph.

Bcbio-nextgen allows us to build pipelines. At the moment, two are used most of the time. Here we use variant2 for the variant calling analysis. Another common one is RNA-seq.
The description keyword is the unique name for this sample
The data for this study is pair end reads. We use the files keyword to define the input files.
The genome_build keyword defines the reference genome.
With the metadata, we define a unique name for this batch and the phenotype.

The prefixes ERP and ERR are defined by EMBL-EBI and stand for the Study and a Run respectively. So, ERR256785 is a normal sample from the Hans et al study.

Now lets look at the algorithm's section.

- algorithm:
aligner: bwa
coverage_interval: regional
mark_duplicates: false
recalibrate: false
realign: false
platform: illumina
quality_format: standard
variant_regions: ../input/ERP002442-targeted.bed
variantcaller: freebayes

The import point here is we have defined bwa for alignment and freebayes as our variant caller. variant_regions defines the relevant regions for this analysis.

The bcbio-nextgen documentations does an excellent job talking about how to setup the directory structure with the input files and how to start the job running. However, it doesn't talk about the output. Go to the final directory. A file called batch1-freebayes.vcf.gz can be loaded into a genome view. I used Integrative Genome View (IGV) from the Board Institute. At first it is completed zoomed out. If we navigate to genes referenced in the paper, you see some results.

The astute user will also notice a file called batch1-freebayes.db. This is an sqlite3 database created by Gemini. Bcbio uses Gemini to import the VCF file and query other databases to retrieve relevant annotations. The Gemini documentations does a nice job describing this process. "Using the GEMINI framework begins by loading a VCF file into a database. Each variant is automatically annotated by comparing it to several genome annotations from source such as ENCODE tracks, UCSC tracks, OMIM, dbSNP, KEGG, and HPRD. All of this information is stored in portable SQLite database that allows one to explore and interpret both coding and non-coding variation using “off-the-shelf” tools or an enhanced SQL engine."

-jk

coverage - a case study in extending a bcbio-nextgen pipeline

John Kern — Fri, 11 Jul 2014 09:00:00 +0000

bcbio-nextgen is a python toolkit providing best-practice pipelines for fully automated high throughput sequencing analysis. It has pipelines for variant calling and RNA-seq analysis. This post talks about how to extend an bcbio-nextgen pipeline. We will delve into the internals parts of the source which are relevant to integrating chanjo, a better way to handle sequence coverage data in clinical sequencing.

What is coverage? The input to this process is next-generation sequencing (NGS) data. Due to limitations with respect to physics and chemistry, the data is a large number of tiny DNA fragments (also called reads) and associated quality scores. Once these are realigned to the reference genome, the astute researcher would like to see how they are distributed. Since there is inherit uncertainty, we want to ensure that there are many reads associated with the location of interest.

Let's use chanjo on a concrete example using the output from my last post, in which, I ran a yeast sample through the pipeline. The resulting bam file, ERR302903-ready.bam, contains the aligned reads. The researcher is almost certainly interested in a specific location. Here we'll using this bed file.

chrVII 193708 196403 RAD54
chrXII 212516 213930 RAD52
chrVII 1007672 1010766 RAD2

This specifies the locations of the genes RAD54, RAD52 and RAD2 on yeasts chromosome 7. A call to chanjo's annotate command...

$ chanjo annotate ERR302903-ready.bam yeast-repair.bed

... yields this output....

chrVII 193708 196403 RAD54 30.5808605341 1
chrXII 212516 213930 RAD52 27.8416961131 1
chrVII 1007672 1010766 RAD2 27.3085621971 1

[aside: bed files are tab separated. This output has been formatted for readability.]

This fifth column is the average coverage depth for the gene in question. So, RAD54 has an average number of reads in this region of about 30x. Nice tool! However, we do not want to have to do this manual each time. Let's integrate it with the bcbio-nextgen pipeline.

Pipelines are defined in bcbio/pipline/main.py. Running a standard pipeline takes a long time. For this integration work, the results of a running a standard pipeline run simply creates input for chanjo. Since bcbio-nextgen is a toolkit for creating pipelines, let's create one that only runs chanjo. Consider this code snippet.

class ChanjoPipeline(AbstractPipeline):
""" Test only: This pipeline only calls the coverage tool. It was designed solely
to support integration with chanjo. Assumes the bam and bed files exist.
"""
name = "chanjo"

@classmethod
def run(self, config, config_file, parallel, dirs, samples):
with prun.start(_wres(parallel, ["chanjo"]),
samples, config, dirs, "multicore") as run_parallel:
with profile.report("coverage", dirs):
samples[0][0]['work_bam'] = 'ready.bam' # Todo: update with valid input
samples = coverage.summary(samples, run_parallel)
logger.info("Timing: finished")
return samples

In this post, I'm just going to focus on the important bits for this integration. All pipelines inherit AbstractPipeline. It mainly requires the subclass to define a run method. For chanjo, we need two files. The bam file is normally generated by the pipeline. Here we will simply hardcore a path to an existing bam file. The corresponding bed file is defined in the sample config file.

upload:
dir: ../final
fc_date: '140611'
fc_name: yeast
details:
- algorithm:
aligner: bwa
recalibrate: false
variantcaller: gatk
quality_format: Standard
coverage: ../input/yeast-repair.bed
analysis: chanjo
description: ERR302903
files:
- ../input/ERR302903_1.fastq.gz
- ../input/ERR302903_2.fastq.gz
genome_build: sacCer3

The main change to the config file is to add a coverage attribute which is associated with the bed file provided by the researcher.

After this setup is done, the summary() function is simple. It cobbles together a command line for the chanjo and runs it.

Using bcbio to analyzing a Saccharomyces cerevisiae sample

John Kern — Fri, 20 Jun 2014 11:39:00 +0000

bcbio-nextgen is a python toolkit providing best-practice pipelines for fully automated high throughput sequencing analysis. While it focuses on the human genome or human-like diploid organisms, it strives to support a wide variety of genomes. This post documents how to use a much small genome: Saccharomyces cerevisiae ( or baker's yeast).

With the human genome, bcbio-nextgen starts by aligning the sample with a well known reference genome: GRCh37. This is called re-sequencing. This assumes you have have a reference genome. Programs which take the raw sample and generate a DNA sequence are called assemblers.

In this post, we will re-sequence a sample using the reference genome for Saccharomyces cerevisiae. In a future post, I plan to use an assembler.

Downloading a Reference Genome

bcbio-nextgen knows how to download some reference genomes. If you look at the help on the upgrade command you'll see the '--genomes' option which enumerates them.

$ bcbio_nextgen.py upgrade --help
usage: .....
--genomes {GRCh37, hg19, mm10, mm9, rn5, canFam3, dm3, Zv9, phix, sacCer3}
Genomes to download
.....

The token 'sacCer3' represents the latest yeast reference genome. We can download it with...

$ bcbio_nextgen.py upgrade --data --genomes sacCer3

Getting the data

After searching around the National Center for Biotechnology Information (NCBI), I selected ERX276244: Whole Genome Sequencing of fungal endophyte sp. D3-2B19-1. On the NCBI website, Next Generation Sequencing (NGS) data is found on the Sequence Read Archive (SRA) section.

The selected experiment is a paired reads run on an Illumina HiSeq 2000. Paired reads signify that the fragment is read in both directions. This implies we will need two files, one in each direction.

The Sequence Read Archive basically includes intensity, read and alignment data. All this data requires a lot of space. We only want to extract the two fastq files representing the reads. NCBI provides a collection of command line tools called the SRA toolkit to extract fastq files. We will use the fastq-dump command to retrieve the data we require. Consider:

fastq-dump --split-files ERR302903

ERR302903 is the accession identifier for the run in which we are interested. If you do not use the '--split-files' option, all the data is glommed into a single file. By applying this option, fastq-dump splits the fastq files into two, one for each direction yielding: ERR302903_1.fastq and ERR302903_2.fastq

Running the Pipeline

Now that we have a reference genome for our species of interest and a sample, let's use bcbio-nextgen to analyze our sample. Let's start by creating a working directory structure like this one:

yeast
|- config
|- work
|- input

Copy the fastq files we downloaded from NCBI into the input directory.

The main configuration file for a bcbio-nextgen pipeline run is called a sample file. Like all bcbio-nextgen configuration files it is in yaml format. It defines all the parameters we will need to analyze our sample.

upload:
dir: ../final
fc_date: '140611'
fc_name: yeast
details:
- algorithm:
aligner: bwa
recalibrate: false
variantcaller: gatk
quality_format: Standard
coverage_interval: genome
analysis: variant2
description: ERR302903
files:
- ../input/ERR302903_1.fastq.gz
- ../input/ERR302903_2.fastq.gz
genome_build: sacCer3

This yaml file should be copied to the config directory.

Finally, we change to the work directory and execute this command:

bcbio_nextgen.py ../config/ERR302903-yeast.yaml -n 4

I ran this on an AWS m3.medium instance: 3.75Gb of Ram and 1 cpu. This run took about three hours and requires about 2Gb of disk space.

Future posts will talk about using the results of this pipeline.

-jk

Quick Tutorial: Implementing Your Own Delegates in Objective-C

Tasneem Sayeed — Wed, 07 May 2014 17:44:00 +0000

The delegate pattern is another simple, yet powerful design pattern. Many UI elements in iOS (i.e. UIScrollView, UITextField, UITableView,etc) use delegates to control their behavior. A delegate is an object that acts on behalf of, or in coordination with, another object when that object encounters an event in a program. The delegating object is often a responder object—that is, an object inheriting from NSResponder in AppKit or UIResponder in UIKit—that is responding to a user event. The delegate is an object that is delegated control of the user interface for that event, or is at least asked to interpret the event in an application-specific manner. For a more thorough description on delegates, refer to the Apple documentation.

In order to implement your own custom delegate protocol, you will need to

Modify the header (.h) file for MyClass (i.e. MyClass.h)
Add the @protocol delaration
Add a delegate @property
Declare the methods that delegates can implement.

Next, make sure to check if the delegate is set and that it responds to the selector anytime you want to call the delegate within your implementation.

Next, for the classes that you want to conform to your new protocol, include MyClass.h header file and delegate protocol in the @interface

Finally, set its delegate to self somewhere and implement the delegate method

View more documents from tasneemsayeed.

How to Send Email Within Your iPhone Application

Tasneem Sayeed — Tue, 06 May 2014 21:25:00 +0000

This article provides a tutorial to help you send an email from inside your iPhone application using the iPhone SDK built-in APIs.
The iPhone SDK provides the built-in MessageUI framework, which greatly simplifies the implementation of email functionality within an iOS application.

Creating a Simple Email UI

Create a simple app with a UI View Controller and call it the name, "SimpleEmailAppViewController". Then, add a button to the UIView by dragging it from the interface builder, and rename the button title to "Contact Us".

When the user taps the "Contact Us" button, the app will display the email user interface.

Making Connections to the User Interface

In order to make the connections from the "Contact Us" button to an action, you need to select the SimpleEmailAppViewController.xib file from the Project Navigator then go to the Interface Builder. Switch to the Assistant Editor while hiding the Utility area. Once this is done, the interface and its corresponding code are displayed side by side. Next, press and hold the key and click the "Contact Us" button and
drag it towards the "SimpleEmailAppViewController.h. As you place the pointer just below @interface and before the @end, and as you release the mouse button, you will notice a prompt that allows you to insert an outlet and action.
Select "Action" for "Connection" and enter "sendEmail" for "name" as shown below.

The event can be kept as "Touch Up Inside". When the user clicks the "Contact Us" button and lifts up the finger inside the button, it will result in invoking the "sendEmail" method. Once, you click Connect to confirm the changes, Xcode automatically adds the method declaration into "SimpleEmailAppViewController.h" file.

Implementing the Email Interface

In "SimpleEmailAppViewController.h", implement the "MFMailComposeViewControllerDelegate" as shown below.

In "SimpleEmailAppViewController.m", implement the "sendEmail" method. Also, add the implementation for the delegate method, "MFMailComposeViewControllerDelegate". Note that we will also be utilizing the built-in API from the iOS SDK called "MFMailComposeViewController".
The "MFMailComposeViewController" class provides a standard interface to allow the editing, and composing of an email message. You can use this view controller to display a standard email view within your iOS application. We populate the fields of this view with initial values including the recipient email, subject and body of the message.

This view controller class also includes the following delegate method:
(void) mailComposeController:(MFMailComposeViewController *)controller
didFinishWithResult:(MFMailComposeResult)result error:(NSError *)error

This method is invoked when the user cancels the operation and the mail composition interface is dismissed. The result parameter tells you the result code when the mail composition interface is dismissed. In a real-world application, the app should display any errors that may occur if the app fails to send the email message.

Linking with the Message UI Framework

If you try to build the application, you will notice that you will get build errors. The "MFMailComposeViewController" class is built-in within the MessageUI framework.

To fix the compilation problem(s), you will need to add the MessageUI framework so that it is linked properly with your application. In the Project Navigator, select the "SimpleEmailApp" project and then select "SimpleEmailApp" target under Targets. Then, click "Build Phases" at the top of the project editor panel. Then, click on "Link Binary With Libraries" section.

Next, click the "+" button and select the "MessageUI framework". After you click the "Add" button, Xcode will include the MessageUI framework. This should fix your error(s), and you can now run your application. When you tap on the "Contact Us" button, it will display the email composition window with the pre-populated email content.

If you find this tutorial helpful, and would like to see more of these types of tutorials, then share a comment! Or if you have any other suggestions, do let me know as well. And follow me on twitter (@tasneemsayeed).

View more documents from tasneemsayeed.

Viewing raw microarray data as an image

John Kern — Tue, 29 Apr 2014 08:22:00 +0000

I am taking course on data analysis of genomic data using R. There are already good places to learn about R and Bioconductor. Here I just wanted to record a code snippet I found useful. Often when working with microarray data, we are analyzing it statically and examine the resulting graphs. Sometimes you might want to view the raw data as an image. This code shows that image.


library(affy)
dat <- ReadAffy()
image(dat[,1])

Tech tip: linux commands for manipulating large files

John Kern — Fri, 25 Apr 2014 14:56:00 +0000

Bioinformatic files can be very large and take a long time to process. A file containing a complete human genome is about 3Gb. The raw data from the sequencer is much bigger. Runtimes are best measured in hours or days.

Often we want to manipulate large, compressed bioinformatic files. Their is a set of commands which are prefixed with 'z'. They parallel standard Linux commands such as cat, grep, egrep, less, more, diff and cmp.

How can you search a large, compressed file? Normally, one uses grep to search files but you can't do that directly with a compressed. Uncompressing the file to the file system is an option but that requires a lot of disk and you need to clean up afterwards.

The zcat command (or alternatively command '-c' option on gunzip ) uncompressing the file to standard out. So, we can 'zcat foo.gz | grep pattern'.
Alternatively you can simply use zgrep directly.

How does one view a large, compress file?

zcat, zless, zmore

How does one compare a large, compress file?

zdiff, zcmp

Long running processes are common. Simply running a process on a remote machine is brittle. If the session is interrupted, it is not possible to recover the information. "Both GNU Screen and tmux are terminal multiplexers designed for Unix-like platforms". Here is a comparison. This allow us to disconnect from a session, log off the system and return at a latter time.

-John

Initial installation of bcbio - a python toolkit providing best-practice pipelines for fully automated high throughput sequencing analysis

John Kern — Fri, 18 Apr 2014 07:16:00 +0000

Around the turn of the century, the human genome was sequenced for the first time. Since the achievement of this milestone, a ton of genomic data has been generated. A lot of it is publicly available. The latest sequencing machines, called Next Generation Sequencing(NGS), are creating sequencing at an astounding rate. This generates a lot of data. The raw output to sequence a single human genome is about 22Gb.

Once the data is sequenced analysis can begin. This is a complicated process. This blog post talks about my initial experience installing bcbio-nextgen on AWS. bcbio-nextgen is a python toolkit providing best-practice pipelines for fully automated high throughput sequencing analysis. While they framework is written in python, it leverages open source software written in a variety of languages including C/C++, Java and R.

At first blush, you might consider using a t1.micro instance. This generally works well for configuring an instance even for compiling and it is inexpensive. Unfortunately, bcbio-nextgen will fail to compile certain libraries as it will run out of memory. A t1.micro instance has 0.615 Gb of RAM. So, I switch to a m1.medium instance which has about 3Gb of RAM.

The next run went much further but I ran under disk space. By default, the instance only has about 32 Gb of disk space. To get going with bcbio-nextgen, we need:

about 20Gb for the OS and tools.
GRCh37 ( a human reference genome ) requires about 20Gb
finally we need about 30Gb to support indices generated by the tools.

So, we need at least 70Gb.

Two obvious choices are instance storage and EBS volumes. The m1.medium instance has 410Gb of instance storage. Unfortunately, this is not suitable for our purposes. If you terminate or stop the instance, your data is lost. I created and attached a 100Gb

In this case, I configured a single node. Today, this is good enough for me. This analysis is computationally intensive. Some tasks can leverages multiple cores, other tasks can be dished out to multiple systems and some must be computed sequentially. In the future, no doubt I'll want to configure more sophisticated set of nodes but I'll save that for another day.

-jk

[Ubuntu Touch] Update to Logviewer

Tue, 21 Jan 2014 19:49:16 +0000

I am pleased to announced that Logviewer is now published in the Ubuntu Touch store. The app no longer runs unconfined but uses “read_path” pointing to “/var/log/” and “/home/phablet/.cache/”. If you think there is another interested log path let me know and I will try to include it.

Also, one feature that landed by popular request is submitting a selected section of a log to pastebin , thanks to Popey for the image:

[Ubuntu Touch] Logviewer

Wed, 08 Jan 2014 15:55:13 +0000

I have been recently doing some android development for Techfunder, one thing that I have found really useful when testing my app is using CatLog. CatLog allows you to check the app and system logs on the go. This is extremely useful when you have a crash while you are not close to your laptop.

This motivated me to look into writing a similar app for Ubuntu Touch. So here it is: LogViewer!

This app, like CatLog, is for developers and requires unconstrained running. You will need to install it manually:

Download click package from launchpad
transfer to your device and install:

adb push com.ubuntu.developer.vtuson.logviewer_0.1_armhf.click /home/phablet/
adb shell
su phablet
cd ~
pkcon -p install-local com.ubuntu.developer.vtuson.logviewer_0.2_armhf.click

When you launch the app, you will get a list of .log files in /home/phablet/.cache/upstart/ , if you click on an specific log, it will be displayed in a similar manner to tail -f. You can pause the autoreading, clear the screen and copy to clipboard parts of the logs from the bottom menu.

You can also access other files, change font size of the logs and the size of the text buffered from the settings page.

You can see the code and contribute :) in launchpad:

https://launchpad.net/logviewer

Developing for Android and Ubuntu – with the same phone

Mon, 23 Dec 2013 20:35:34 +0000

We announced today a new solution to dual boot Android and Ubuntu on the same device. Over the last few weeks I have recently blogged about a Contacts import app for Ubuntu and Techfunder, an Android app for crowd funding projects. What I didn’t mention before is that I have been developing and testing both in the same device!

I have been dog-fooding and developing a small part of our dual boot solution for a couple weeks now. During that time, I’ve not only been able to boot between Android and Ubuntu as a user, but also as an application developer.

Dual boot brings no compromise to the SDK experience of either operating system. I run Ubuntu SDK with QTCreator and Android’s ADT (eclipse-based) on my 12.04 LTS laptop. And while the SDK for Android is more mature and fully featured, I still find Ubuntu (an particularly QML) much faster to prototype apps.

Dual boot is also about making the application developers life easier and cheaper. Having to buy extra devices for testing new apps can be a put you off. You can now develop for Ubuntu by jusr re-using your Android device.. without having to disrupt your android projects! For example, yesterday I was working on applications on both sides, and I was easily booting back and forward and collecting logs in each side.

I hope to see more integration in between both development environments, I think it will be particularly neat to have something like Android Monitor tool (aka DDMS) working for both OSs.

Btw, I have just released version 2.0 of Techfunder! Including home screen widgets, search and more categories. Don’t forget to check it out:

[Ubuntu QML] Importing Google Contacts

Tue, 17 Dec 2013 19:27:43 +0000

I have a lot of contacts in my phone… I am sure you will have more, but syncing over 500 contacts to Ubuntu phone using the command line for Syncevolution gets tedious.

So I wrote a little QML app to do the trick for you. Unfortunately, to run a system command the application has to run unconfined, so I have not yet submitted it to the store.

But if you want to install it yourself it is pretty simple:

Download this file
push the downloaded file to your phone, like so: adb push com.ubuntu.developer.vtuson.contactsimporter_0.9_armhf.click /home/phablet
then run this to install it: adb shell “sudo -u phablet pkcon -p install-local /home/phablet/com.ubuntu.developer.vtuson.contactsimporter_0.9_armhf.click”

You should be good to use it now, the app looks like this: (and if you want to check out the code is here)

Ubuntu Touch: Building and testing a QML extension

Mon, 16 Dec 2013 16:18:01 +0000

Hi,

I found myself trying to build a QML Ubuntu Touch app project that includes a qml c++ extension and I found that I some how stumble a bit along the way. So, here are some of my notes on how I got it done. Hope that helps.

Creating the project.

Using QtCreator, create a new project and select – QML Extension Library + Tabbed Touch. I found that it was easier to change the QML side of things than start with an extension and then add the whole project structure.

Build and Run your project locally

In QtCreator click on projects. In Build, I set up the build path as my project root path. In run, the executable is “/usr/bin/qmlscene” (make sure there is no spaces trailing) and then Arguments is set to “-I ../backend/modules $@ yourapp.qml”, with a working directory of “projectroot/app”

Now if you try to run your project it should build it locally and run your app. After that you are on a roll.

Build on target device

Click Ctrl+f11 should install the platform developer tools in your device. However, I how found that this lately does not work.

Instead from the terminal:

first we will need to make the image in your device writeable:
adb shell touch /userdata/.writable_image --> and reboot the phone. then: cd /usr/share/qtcreator/ubuntu/scripts adb devices ./device_developertools_install

Now you are ready to build, so back to QtCreator:

Build>Ubuntu>Build Application on Device

This should build the application with only some test problems, but the main binaries would be created. To package your app you will need to get

/home/phablet/dev_tmp//backend/modules/lib.so

Creating a click package

create a manifest standard manifest file. Manifest.json
{ "description": "your text", "framework": "ubuntu-sdk-13.10", "hooks": { "yourappname": { "apparmor": "yourappname.json", "desktop": "yourappname.desktop" } }, "maintainer": "your name", "name": "com.ubuntu.developer.yourname.yourappname", "title": "yourappname", "version": "0.1", "architecture": "armhf" }

You will also need a yourapp.desktop file:

[Desktop Entry] Name=yourappname Comment=description Exec=qmlscene -I plugin $@ yourapp.qml Icon=icon.png Terminal=false Type=Application X-Ubuntu-Touch=true

Note that Exec= has a -I plugin part to it. This is very important, will see later.

Now yourapp.json file that contains your confinement profile:

{ "policy_groups": [ "networking" ], "policy_version": 1 }

Now time to setup a folder with all this stuff, not that the plugin folder is going to contain your lib which your are importing with -I option on the desktop file:
myproject> -./click/ --icon.png --manifest.json --yourapp.json --yourapp.desktop --./plugin/ ---./yourlib/ ----lib(yourlibname).so ----qmldir

Now you are ready to build from your project root folder:
click build ./click
This should create a .click file in your project folder.

Installing in your device

adb pull your.click /home/phablet/ adb shell su phablet cd ~ pkcon -p install-local your.click

This should be enough, but sometimes I find that you need to restart unity:
pkill unity8 (you might need sudo)

Introducing Techfunder for Kickstarter and Indiegogo

Thu, 05 Dec 2013 12:13:01 +0000

Since I lead the Ubuntu Edge campaign, I have been trying to keep up with other crowd funding projects. I am mainly interested on technology and gadgets, and I have found it hard to navigate Kickstarter, but also to have to keep hoping between Kickstarter and Indiegogo to see what is going on. Specially since now, seems like interesting projects are evently split between them. You might share my friction on this… so I give you Techfunder:

Techfunder is an Android app that provides an easy way to browse crowd funding projects launched across the main industry websites. Techfunder gathers Technology and Design projects from Kickstarter and Indiegogo.

Using Techfunder side navigation you can easily switch between:

Popular
New this week
Staff Picks
Design – Popular

Then you can browse as many projects on a compact scrollable list. Just click on the project you are interested to expand into a full screen view.From there you can tell the world about it using the share button.

If you want to contribute or found out more about it, click on the “browse” button and Techfunder will launch the project page in your web browser. When you are finished, just press the Android back button to return to Techfunder.

I am currently planning to add a favourites/watchlist functionality and looking into a way to select additional crowd sourcing platform. I hope you enjoy it!

Introducing Techfunder for Kickstarter and Indiegogo

Tue, 03 Dec 2013 10:19:24 +0000

Using Techfunder side navigation you can easily switch between:

Popular
New this week
Staff Picks
Design – Popular

I am currently planning to add a favourites/watchlist functionality and looking into a way to select additional crowd sourcing platform. I hope you enjoy it!

Source editor worth $70

Fri, 15 Nov 2013 05:11:48 +0000

I’ve been a devoted Eclipse user for more then a decade (with most of that time spent developing applications based on Eclipse) but I’ve noticed that I am becoming less and less inclined to spend my time trying to convince Eclipse to actually edit my files. I ended up using Eclipse only for Java – Xcode is great C++/C (libclang-powered editor with C++11 support) with VIM was standard for all other files. For the last couple months I’ve been slowly migrating to Sublime – now I reached the point when I even consider using it for Java files.

This is just a “disgruntled veteran Eclipse user” view on why it is worth it to pay for your coding editor these days.

It can open files! And folders…

Cmd+S, Cmd+N and Cmd+O are the first shortcuts everybody learns. Eclipse is unique in that it breaks this convention. “Open File” is an afterthought (doesn’t even have a key shortcut), with many plugins unable to work with those files – even if your editor manages not to show an error dialog (or dump an exception in log) – some features still are not available.

Sublime not just behaves as expected – it goes one step further. I can “open” any folder – showing that folder in the Sublime “Project Explorer”-like pane. What is truly surprising is that Sublime manages to deliver many Eclipse killer features – including “Open Resource…” and “Goto Symbol in Project” (think JDT/CDT “Find Type” and “Find Function”). No metadata files is created in those folders, no indexer is ran. Right now my “workspace” has a mix of Bootstrap-based HTML application with a Python server-side – and without any plugins I can immediately focus on a CSS rule or on a Python function. I’ve also shown to my colleagues that I can open mid-sized C projects and be able to open random function within seconds. No setup, no long-running background jobs.

It opens shell scripts, .ini files without spinning up Xcode or any other heavy “external editor”.

Editor is the same

Cmd+Shift+R will go to symbol in any file in any language (that I tried) – in Eclipse, CDT and JDT installed in the same workbench have confusing separate realms with Cmd+Shift+T been bound to different dialogs depending on active editor. Cmd+Shift+R and Cmd+Shift+A are yet other parallel realms, adding to the confusion.

Sublime can expand selection to enclosing scope in Dita XML, Python or in JavaScript. Sublime actually even unifies Eclipse-like “Goto line” and “Quick outline” popups into “Goto Everything” – I’ve been wondering for years why Eclipse uses separate pop-ups for that.

One prominent Sublime feature is multiple cursors. Curiously, this feature is really familiar to JDT and CDT users – you can see multiple cursors when you rename variables. In Sublime multiple cursors can be used to select search results (quite similar to Eclipse rename refactoring) or to apply edits to several separate locations at once.

No mouse

I spend most my time in Eclipse having source editor maximized (I also used to hide toolbar – until I started using Eclipse Kepler where Cmd+3 makes toolbar visible no matter what).

In Sublime I can split the window (not the editor ) using just the keyboard. I can move editors between stacks or focus on stacks without reaching the mouse. Switching between editor predictably opens a tab to left/right – instead of randomly popping from the recent editors stack.

Niceties

There is no preferences UI as such in Sublime – keybindings and settings are stored in JSON files that are editable in the text editor.

Plenty of plugins, easy discovery. “Cmd+Shift+P”, “Install”, type filter string. Plugin is found and installed even before “Marketplace client” would open.

Files can be opened from the command-line. Reliably. Even on Mac.

Missing parts?

Sublime does not integrate SCM support. Well, there are some plugins – but I haven’t figured them out yet, mostly because I am command-line Git and Perforce user (as not all my files are in the workspace) – the only thing I miss is Perforce checkout from the editor.

No debugger UI. Surprisingly, no biggie – JavaScript users are using browser anyways, C* users get to keep their GDB/LLDB primed.

No way to run shell command from within the editor. As in VIM.

Next generation

IntelliJ and Eclipse were revolutionary tools – they eschewed RAD features and visual designers that were a cornerstone of the IDE experience and focused on the core programming experience instead – providing ultimate source editor, making refactoring mainstream. Sublime seems to do the same trick – only this time it is Eclipse that is slow and outdated. Sublime does not force me to manage workspaces (in couple weeks I will have to start working in a different branch (in Perforce it is a different folder) – ain’t it fun, with my 20+ projects?), flip perspectives (the first thing I show to QA or new team members – “If you don’t understand what’s going on on the screen, do “Window > Reset perspective”) or wait till my cancel button click is honored. Again, as 10 years ago, I’m enjoying my editor instead of wrestling with it…

Indeed, times, they are a-changing…

DevFest West 2013: Lightning Talk: Learnings, Prototypes & Use Cases on Google Glass

Tasneem Sayeed — Sat, 26 Oct 2013 18:53:00 +0000

According to IMS Research, the wearables market is poised to grow from 14 million devices shipped in 2011 to as many as 171 million units shipped by 2016! According to a recent Business Insider report, "those betting big on wearable computing believe an assorted new crop of gadgets — mostly worn on the wrist or as eyewear — will become a "fifth screen," after TVs, PCs, smartphones, and tablets."

I recently was invited to talk at the DevFest West 2013 held at the Google campus in Mountain View where I presented a Lightning Talk on "Learnings, Prototypes & Use Cases on Google Glass".

The talk provides insights and lessons learned from innovative experiments for building innovative services for Google Glass for capturing financial data picture and for mobile payments. It also covers a number of Glass Use-Cases as well as Glass Prototypes that we implemented across Intuit. If you find the presentation slides below useful, then add a comment here or follow me @tasneemsayeed for future postings!

View more documents from tasneemsayeed.

Book review: Reality is Broken

John Kern — Tue, 23 Jul 2013 09:14:00 +0000

The premise of Jane McGonigal's book, Reality is Broken, is intriguing. What if we could harness game play to attack some of the challenges of our age? Here she is talking about it at a TED conference.

Want to know what is happening on the bleeding edge of computer graphics? Games are always pushing the envelop. They are also frequent early adopters of Artificial Intelligence techniques.

I'm glad I read her book because she illuminates her thesis presented in her TED talk. The first part of the book enumerates the contrasts between what game play does well with respect to reality:

more satisfying work, better hope of success, stronger social connectivity and a chance to be a part of something bigger than ourselves.

Open Questions:

Games tend to have a clear line between right and wrong. Many of the changes of our age are thorny problems. Trade offs will be made based on value judgments. Is there a way to adapt game design to help us coalesce these competing views?
Gaming is very popular. Generally, it is competitive though sometimes collaborative. The book implies gaming is the universal learning technique. Is it? Mediation. Building things. Composing music or poems.

-jk

Learning on Accessibility for the iOS Platform

Tasneem Sayeed — Wed, 10 Jul 2013 17:14:00 +0000

According to Apple's Accessibility Guide for iOS, you should make your iPhone application accessible to VoiceOver users because:

It increases your user base. You've worked hard to create a great application; don’t miss the opportunity to make it available to even more users.
It allows people to use your application without seeing the screen. Users with visual impairments can use your application with the help of VoiceOver.
It helps you address accessibility guidelines. Various governing bodies create guidelines for accessibility and making your iPhone application accessible to VoiceOver users can help you meet them.
It's the right thing to do.

As of iOS 3.0, Apple has included the UI Accessibility programming interface, which is a lightweight API that helps an application provide all the information VoiceOver needs to describe the user interface and help visually impaired people use the application.

I had recently given a presentation on my Learnings on Accessibility for the iOS Platform at an internal event at Intuit, which I wanted to share with all of you. It provides an overview on what it means to make an app accessible for the iOS platform. It also provides guidelines for making your iOS app accessible and includes an overview on the most common accessible attributes, traits and how to add Accessibility via interface builder as well as in code. It covers Accessibility Notifications, VoiceOver specific API, Accessibility Containers, and some of the best practices for Accessibility.

If you find the presentation helpful in making your iOS app accessible, feel free to send me a comment!

Enjoy making your iOS app accessible!

Learnings for Accessibility for iOS Platform from Tasneem Sayeed

View more documents from tasneemsayeed.

MOOCs - evolving college town.

John Kern — Thu, 20 Jun 2013 14:02:00 +0000

Image a school in a not too distant future where lectures are delivered online. How would this transform college towns? Historically required lower-division courses were taught in a large lecture hall packed cheek to jowl with 150 or 200 students. If the lectures are delivered online, there is no need for a lecture hall. Students could watch lectures at any time and venue which is convenient for them.

Some initial research based on the data generated from the current crop of MOOCs is becoming available. Lori Breslow et al recently published Studying Learning in the Worldwide Classroom Research into edX’s First MOOC. A key finding was...

On average, with all other predictors being equal, a student who worked offline with someone else in the class or someone who had expertise in the subject would have a predicted score almost three points higher than someone working by him or herself. This is a noteworthy finding as it reflects what we know about on-campus instruction: that collaborating with another person, whether novice or expert, strengthens learning.

While we would no longer need to gather a hundred or more students together, colleges add value by facilitating group interaction. Instructors working with small groups, study groups, project groups.

Teachers Matter! New Roles for Teachers.

Lectures: Some teachers will specialize on delivering video lectures. These instructors will need to be more collaborative. Publishing a textbook are a good point of departure. Most teachers use a textbook developed by someone else. While the author is the one who distinguishes the book, the creation of a textbook relies on a supporting cast(e.g., editors, publishers, reviews, etc). When making a MOOC, the instructor will work with a cadre of professionals.

Pedagogy: Online courses provide us with an opportunity to learn about how people learn. The current crop of MOOCs leverage the latest web technologies and apply machine learning techniques to the large volume of data unobtrusively collected by the system as the student works through the course material. This creates a rich vehicle for social scientists who want to study student learning. Working with the software development team, social science researchers can define an experiment and measure the results.

Classroom: What will happen to the classroom?

It is difficult to make predictions, especially about the future, Yogi Berra

While no one knows for sure, we are seeing some interesting experiments with 'flipping the classroom'. The
basic idea is students watch the lectures outside of class. This frees class time of other activities. One example can be found by the collaboration between MIT, edX and San Jose State University (SJSU). SJSU leverages MITx 6.002x:Circuits and Electronics in one of its courses. In class, students are divided into small teams and work on problems.

Other teachers might use this time for projects.

Peers matter: New tools, more flexibility

At the moment, online systems have discussion forums. It can be difficult for the student to filter through all the conversion and pair up with other students. However, I think this is a promising area for ongoing research and future development. There is a lot of work going on in the Connectivism MOOC movement (see Stephen Downes book Connectivism and Connective Knowledge ) and all the work on social media in general. These ideas will find their way into online learning and improve them.

Community matters: Hackerspaces.

Since lectures would no longer force us to meet at large lecture halls in the middle of the day, other learning opportunities would are possible. Hackerspaces may provide some insight for Higher Education. These are locations where people who share a common interest can meet, socialize and collaborate. Probably the most famous in my area are the Hacker Dojo which brings people together around software development, Biocurious focusing on biotechnology offering lab space for its members to share and the TechShop is a space for those who are building things.

Google I/O 2013: An In-Depth Developer's Perspective

Tasneem Sayeed — Sat, 18 May 2013 21:51:00 +0000

In comparison to last year's Google I/O featuring the spectacle augmented reality Google Glass announcement via a breathtaking sky-diving presentation, this year's Google I/O may have seemed to lack luster with no new hardware.

Nonetheless, Google I/O kicked off a record 3.5 hours Keynote with a heavy focus on software and services. The Google I/O Keynote discussed services and feature upgrades for both Android and Chrome. If you were expecting a brand new Android phone or tablet announcement, then you may have been disappointed. However, a major takeaway was a unified user experience for Chrome and Android platforms through shared services.

According to Google, Android was at 100 million activations in 2011. And, it was at 400 million in 2012. And now in 2013, Google released the numbers for the Android activations at an incredible 900 million! Now that's no small feat...

A huge applause from the audience came when Google mentioned the introduction of Android Studio IDE based on IntelliJ! The Android Studio IDE tool has more options for Android development making the process faster and more productive. A live layout was shown that renders your app while you are editing it in realtime. It allows you to see a variety of device layouts (i.e. different form factor phone devices and tablets). Refer to TechCrunch's excellent review on Google Launches Android Studio and New Features for Developer Console Including Beta Releases and Staged Layouts for further details.

There were several other significant announcements made at Google I/O including:

Google Maps gets a complete overhaul and is released as part of Google Play Services. Three new Location APIs introduced include:

Fused Location Provider API utilizes all of the communications sensors in the phone including WiFi, GPS and Cell network while significantly saving battery life. This is a new service that greatly improves any application that uses location services.
Geofencing API allows apps to inform the user entering or exiting a configured virtual fence. The API allows each app to define up to 100 geofences simultaneously. Apps utilizing this service will provide better battery life and performance.
Activity Recognition API utilizes the device capabilities of the hardware and machine learning to determine whether the user is walking, cycling or driving. This allows apps utilizing the service to adjust their behavior depending on the user's mode of transport. It is done in a very battery efficient way as no GPS is required.

Google+ Single Sign-in API is a cross-platform Sign-in API. Fancy was used to demo during the Keynote that if you find a cool website that you like, you can sign-in with Google+ and then you will automatically be asked if you would like to install it. If you indicate yes, the the app will download and log you in automatically on your current device and other Android devices on your account - now that's very cool!
Google Cloud Messaging (GCM) for Android gets an overhaul. GCM is a service that allows you to send data from your server to your users' Android-powered device, and also to receive messages from devices on the same connection. The new features to GCM include:

Faster, easier GCM setup
Upstream messaging over XMPP. GCM's Cloud Connection Service (CCS) lets you communicate with Android devices over a persistent XMPP connection. The primary advantage of CCS are speed and the ability to receive upstream messages (i.e. messages from a device to the Cloud).
New API for Synchronizing Notifications. Maps a single user to a notification key, which you can then use to send a single message to multiple devices owned by the user.

Google Play Music All Access was launched in the US at I/O. It allows you to buy a music subscription from Google and costs $9.99/month and comes with a 30 day free trial. It allows you to explore millions of music tracks so it seems great for music discovery than its competitors (i.e. Spotify, Pandora). It is a simpler on-demand-meets-radio service with various personalization features, and works on phones, tablets and web browsers. From a design and UX perspective, it makes it easy to switch between the hands-on and hands-off experiences. If you are not sure of what you want to listen, you can just hit "Listen Now" and start listening to something right away. And, when you want to search on Google's huge on-demand catalog, you have that option as well.
Google Search. Google I/O 2013 marked the "end of search as we know it". Google announced that it is looking to change the way users go about finding information by expanding its voice search capabilities and through its various services such as Google's Knowledge Graph and Google Now with a credo the company has labeled "answer", "converse" and "anticipate". In other words, the company's core product will eventually respond better to naturally phrased questions. Anticipate means that Google Search will be able to guess what information users need the most and provide it for them easily via its Google Now service. The Google Now service pulls information from across Google services to act as a personal assistant of sorts by offering information on users' commutes, appointments and news from their favorite sources in the same place. Conversational Search was announced to be coming to all desktops and laptops via Chrome.
Chrome. Google announced that today there are 750+ Million active users of Chrome and that Chrome is increasingly used on mobile. Chrome design goals are: speed, simplicity and security. Google's Sundar Pichai said, "The same capabilities that you're used to using for Chrome on a desktop are going to be coming to Chrome on Android." Thanks to WebGL and Web Audio APIs, you will start seeing quite impressive web experiences including games and rich interactive environments that were typically limited to the desktop environment.

Better Web Imaging. Google compared JPEG vs WebP images. The quality of the image was indistinguishable, but it was about two-thirds in size (i.e. 31% reduction in file size). This will significantly improve the load times on websites as well as help users to not exceed their data plan limitations. Furthermore, WebP supports animated images as well.
Better Video compression. Google compared H.264 vs VP9. It was noted that the quality of the VP9 video is the same as H.264. However, the size of VP9 comes in at less than half the size (i.e. 63% reduction in file size).

mCommerce. Google reported that when it comes to shopping on your phone, the percentage of people that come to the purchase screen and then get out from there is incredibly high in the 90%. Google made 3 key announcements around mCommerce:

Consumer launch for pay by Gmail which is rolling out slowly with initial rollout in US
Two new APIs announced for developer launch:

Google Wallet Object APIs. The vision is to digitize whole Wallet by allowing insertion of any kind of Objects into Google Wallet
Google Wallet Instant Buy for merchants selling physical goods. The goal is to allow consumers to make purchases within 2 clicks thereby improving the user experience.

Google+ Google introduced 41 new Google+ features at I/O! First of all, Google+ gets an overhaul on its design. It is a multi-column design where its width will scale depending on the device that the user is on. It includes animations, flip and fade. It can do image analysis, recognize the image and hashtag it automatically.
Google Hangouts is now a standalone app that will work on the web desktop, Android and iOS all announced starting at I/O. This will provide an on-going conversation within the hangout and doesn't end when you sign off. Also, RealTime Communication, that is, group video is available at no charge.
Google Photos: Google announced earlier that in addition to unlimited backup of all of your standard sized photos, it will now give you 15 GB of storage for full-sized images. Google also announced machine learning algorithms that help to create a highlight reel for all your photos. The service will check all your photos in the album for blurriness, smiles and several other criteria (learnt by hundreds of actual human photographers, and produces a highlight reel.) Google also introduced an "auto-enhance" feature that will instantly adjust tonal distribution, red-eye reduction, skin softening, noise reduction and several other criteria to automatically enhance the picture. And, finally, Google introduced a new "Auto Awesome" mode. That is, if you take a burst of photos, it will make an animated gif out of them. If you take a series of screenshots, and if some are dark or if someone is smiling in one, but not the other, it can make a composite image similar to what the Galaxy S4 can do without the user having to activate that setting. And, that sounds cool! It can also handle screenshots in "Motion", HDR, and panoramic photos.

And, no developer review would be complete without the mention of the Developer Sandbox, which occupied 2 floors, with dedicated ones for Android, Wallet, Chrome, standalone ChromeBook Pixel display, Google+, Photos and Google Play. And, not to forget Google Glass had an extra large sandbox which was often crowded with spectators! Google started shipping an early version of Google Glass known as Explorer mostly to a couple of thousand developers who had requested them at last year's Google I/O conference - forked over a hefty price of $1500!

I was fortunate to explore Google Glass and the experience was truly amazing! Glass is not like wearing a laptop on your face, though it does have a 12 GB of usable storage (16 GB total), synced with Google Cloud. Glass is more like augmented reality spectacles, but without lenses. Its lightweight frame rests on the ears and nose, suspending a small prism at the upper right corner of the wearer's field of vision. Glass has a tiny touch pad built into one earpiece and a microphone to pick up voice commands. Interestingly, the earpiece uses "bone conduction" to deliver sounds by vibration against the wearer's head. You can speak aloud to get information or tell Glass to "take a picture" or "take a video" (up to 10s snippets). You can ask questions like "Google find me a restaurant" and see the results in "Knowledge cards".You can also send texts and make phone calls. If you swipe toward the back, you can see Google Now cards for flight, traffic and other information. In summary, Google I/O 2013 showcased several interesting innovations across Google products.

If you found this article useful or want to share your developer perspectives on Google I/O 2013, feel free to share your comments below, follow me by clicking on the upper left hand corner and/or follow me on Twitter @tasneemsayeed.

Best Practices for Mobile App Development on Android

Tasneem Sayeed — Tue, 02 Apr 2013 12:13:00 +0000

Designing and building apps that look great and perform well on as many devices ranging from smart phones to tablets is crucial to ensure an optimal user experience.
At the recent DevFest Silicon Valley event held at Google on March 15th, I had presented a talk on Best Practices for Mobile App Development on Android. The talk focuses on the Golden Rules & Best Practices of Performance including how to keep your apps responsive, how to effectively implement Background Services, tips for improving the performance and scalability of long-running applications, and briefly on Best Practices for User Experience and concluding with Benefits of Intents and Intent Filters. For further details, please refer to the slides attached below.

Best practices for mobile app development android march 15 2013 ts from Tasneem Sayeed

If you enjoyed this article, you may want to follow me (@tasneemsayeed) on Twitter. I always announce significant new blog posts and interesting mobile topics via a tweet. You can also subscribe to this blog. Feel free to post any comments that you may have below.

View more documents from tasneemsayeed.

Implementing Singletons for the iOS platform

Tasneem Sayeed — Fri, 08 Mar 2013 12:54:00 +0000

The Singleton design pattern is one of the most frequently used design pattern when developing for the iOS platform. It is a very powerful way to share data across different parts of an iOS application without having to explicitly pass the data around manually.

Overview

Singleton classes play an important role in iOS as they exhibit an extremely useful design pattern. Within the iOS SDK, the UIApplication class has a method called sharedApplication which when called from anywhere will return the UIApplication instance that is associated with the currently running application.

How to implement the Singleton Class

You can implement the Singleton class in Objective-C as follows:

`MyManager.h`


@interface MySingletonManager : NSObject {
    NSString *someProperty;
}


@property (nonatomic, retain) NSString *someProperty;

+ (id)sharedManager;

@end

`MyManager.m`

#import "MySingletonManager.h"

@implementation MySingletonManager

@synthesize someProperty;

#pragma mark Singleton Methods

+(id) sharedManager {
    static MySingletonManager *sharedMySingletonManager = nil;
    static dispatch_once_t onceToken;
    dispatch_once(&onceToken, ^{
        sharedMySingletonManager = [[self alloc] init];

    });
    return sharedMySingletonManager;
}
                  
- (id)init {
    if (self = [super init]) {
        someProperty = @"Default Property";
    }
    return self;
}

- (void)dealloc {
    // should never be called, but included here for clarity
}

@end

The above code fragment defines a static variable called sharedMySingletonManager which is then initialized once and only once in sharedManager. The way that we ensure that it is only created once is by using the dispatch_once method from the Grand Central Dispatch (GCD). This is thread safe and handled entirely by the OS so you do not need to worry about it at all.

If you rather not use GCD, then you can the following code fragment for sharedManager:

Non-GCD Based

+ (id)sharedManager {
    @synchronized(self) {
        if (sharedMySingletonManager == nil)
            sharedMySingletonManager = [[self alloc] init];
    }
    return sharedMySingletonManager;
}

Then, you can reference the Singleton from anywhere by calling the function below:

Referencing the Singleton

MySingletonManager *sharedManager = [MySingletonManager sharedManager];
Happy Singleton'ing! If you find this post useful, then mention me in the comments.

View more documents from tasneemsayeed.

Dica técnica: Como apontar para um local específico no vídeo?

John Kern — Sat, 18 Aug 2012 11:19:00 +0000

Eu estudo português. Os vídeos em português são uma grande experiência de aprendizagem. Em algum momento, você vai querer perguntar a alguém sobre o vídeo. Como apontar para um momento específico no vídeo de modo que as pessoas assistam-no ao mesmo tempo? Youtube permite que você especificque os minutos e segundos. Basta anexar uma série como esta "#t=1m39s" à URL do vídeo do youtube. Considere este código.

var min=0;
var sec=0;
var videoURL="http://www.youtube.com/watch?v=6lPdpdWakOw";

function createPtr(aVideoId, aMin, aSec) {
return aVideoId + "#t=" + aMin + "m" + aSec + "s";
}

// videoURL é um video que fala sobre educação a distância.

// O professor começa a falar em 1 minuto e 39 segundos.
min=1; sec=39;
createPtr(videoURL, min, sec);
// "http://www.youtube.com/watch?v=6lPdpdWakOw#t=1m39s"

// O repórter em um campo começa a falar em 1 minuto e 6 segundos.
min=1; sec=6
createPtr(videoURL, min, sec);
// "http://www.youtube.com/watch?v=6lPdpdWakOw#t=1m6s"

Challenges for MOOC

John Kern — Tue, 07 Aug 2012 13:34:00 +0000

While I do believe we have reached some kind of typing point with online courses, challenges have been identified.

STEM seems to be easier to present via online courses than the humanities. The main difficulty is how to assess the work submitted by the student. Multiple choice answers are easy. Computer programs can be run against a test suite. How do you grade essays? Coursera uses a peer based method which matches up students to grade one another's work. Can algorithms be written to provide feedback? This is still an open issue.
How do you prevent students from cheating?
How can we devise experiments to determine which online techniques work and which don't?
There is some overlap between these new online courses and textbooks. Will they merge?

In this TED talk, Dr. Daphne Koller introduces Coursera, a startup based on the results of work spearheaded at Stanford. About half way through she goes into some detail with respect to the types of assessments which can be automated and discusses having students grade one another's essays.

MOOC and the 3 Ps : Professors, Pupils and Places

John Kern — Thu, 19 Jul 2012 12:48:00 +0000

This year has been impressive for Massive Open Online Course(MOOC). The Khan Academy continues to grow. Experiments at Stanford and MIT showed impressive results which lead to several efforts to productize MOOCs: Coursera, Udacity and edX. A natural first question for the University is does this cannibalize the university's brand. Many others have covered this aspect of the story (e.g., Slate, TechCrunch, etc). I want to focus on the possibilities to expand education.

Some articles in the popular press imply professors will go away or their role will be seriously diminished. I don't believe for that a moment but their roles will change. We can already see instructors using a techniques like flipping the classroom. Here pupils watch the lesson via video before coming to class. During the class, the instructor can focus on topics that are tripping up the pupils. She can challenge the advanced students and spend more time with those who need a bit more time. The student/teacher and peer-to-peer relationships matter. Where technology proves it can enhance these relationships, it will be adopted. Creating a positive feedback loop.

Another theme where the pundits are wrong is their prediction that universities will go away. There remains a huge gulf between participating in a MOOC and the experience at an elite university. Did you know that less than 7% of the worlds population has a college degree of any kind? That means there are a lot of people who want an education. This provides an opportunity to enhance the learning experience for students who are geographically distributed with technologies but place still matters. On campus, if professors are spending less time delivering lectures and the students are better prepared, it is bound to accelerate the learning process.

Now consider the hundreds of thousands of students taking the course online. While the online course is great, they'd like to meet others in the class or who share their interests. We can already see people finding themselves and meeting at locally libraries or cafes. What should these meeting places look like? What services can we offer there? Where should they be located? Geospatical analysis should be done on this topic. Perhaps, we can glean insights from hackerspaces. These are locations where people who share a common interest can meet, socialize and collaborate. Probably the most famous in my area is the Hacker Dojo which brings people together around software development. Biocurious focusing on biotechnology. Offering lab space for its members to share. The TechShop is a space for those who are building things.

Technology can enhance the learning process and help everyone become smarter, faster. It can allow education to reach an order of magnitude more students. Oh the places we'll go.

Cultivating the Imagination for a world of Constant Change

John Kern — Thu, 12 Jul 2012 16:02:00 +0000

I went to a talk by Douglas Thomas and John Seely Brown last fall in San Francisco in support of their new book A New Culture of Learning. It is grounded in a simple question: "What happens to learning when we move from the stable infrastructure of the twentieth century to the fluid infrastructure of the twenty-first century, where technology is constantly creating and responding to change?"

In this book, culture is used in the sense of maintaining an environment suitable for cultivation. Think about a scientist controlling the environment in petri dish to growing an experiment. Heating it. Adding a specific amount of a chemical substance.

A key aspect of their approach is fostering learning environments with well-defined boundaries. These environments leverage digital information networks and encourages peer-to-peer interacts. Inquiry is central. The learner needs to form questions. This can be project based.

"The new culture of learning is based on three principles:

The old ways of learning are unable to keep up with our rapidly changing world.
New media forms are making peer-to-peer learning easier and more natural.
Peer-to-peer learning is amplified by emerging technologies that shape the collective nature of participation with those new media."

The authors explore learning in informal settings. They focus on project based learning where the projects are meaningful to the learner. In this situation, questions can more important than answers because the drive the student to dig deeper. They provide a variety of case studies showing people highlighting this approach.

References

Here is a pointer to the book on amazon.
Here is a review of this book by the Chronicle of Higher Education.
Kate Rutter drew this amazing graphic which summarized the notes from the authors presentation in San Francisco.
Peter Rawsthorne reviewed this book on his blog called Critical Technology
Playing to Learn, Canadian Education Association

-jk

Notes on mongo queries

John Kern — Wed, 11 Jul 2012 16:06:00 +0000

Hello,

Today's post is just some notes on what I've learned about mongo queries.

I am working with a collection of postsecondary educational institutes. Everything from beauty schools to elite universities. First let's confirm the number of states.

> var rc = db.runCommand({"distinct" : "institutions", "key" : "us_state"} )
> rc.values.length
59

59! Say,what is this new math? Let's list out the sorted results.

> rc.values.sort()
[
"AK",
"AL",
"AR",
"AS",
"AZ",
"CA",
"CO",
"CT",
......
"TN",
"TX",
"UT",
"VA",
"VI",
"VT",
"WA",
"WI",
"WV",
"WY"
]

Ok, it includes the US territory.

Now I would like to know how many institutions are in each state. You could iterate through the list we just computed and use the count() method. For example,

> db.institutions.find({"us_state": "WI" }).count()
151

However, it would be nice if we could compute all of them at once. Mongo provides a series of aggregation functions. Here I'll use group.

var rc = db.runCommand( {"group" : {
"ns" : "institutions",
"key" : {"us_state" :1} ,
"initial" : {"count" : 0},
"$reduce": function(curDoc,accumulator) {
accumulator.count++;
}
}})

Let's look at the options.

The value for the ns key is the collection.
The key key defines which property we want to group by.
initial initializes a variable on a per unique value of the key in question(i.e., unique on a per state basis in this case).
The function associated with $reduce takes two arguments. curDoc is a read-only view of the current document. The second argument, accumulator, is where we store results.

The results is simply an array of objects.

> rc.retval.sort(function (a,b) {return a.count - b.count})
[
....
{
"us_state" : "MO",
"count" : 262
},
{
"us_state" : "MA",
"count" : 282
},
{
"us_state" : "IL",
"count" : 399
},
{
"us_state" : "OH",
"count" : 427
},
{
"us_state" : "FL",
"count" : 523
},
{
"us_state" : "PA",
"count" : 524
},
{
"us_state" : "TX",
"count" : 566
},
{
"us_state" : "NY",
"count" : 742
},
{
"us_state" : "CA",
"count" : 1240
}
]

-jk

Tech Tip: Checking a Criteria in Spring Data MongoDB

John Kern — Thu, 21 Jun 2012 13:41:00 +0000

Hello,

I like the Spring Data MongoDB APIs. I find the way they model queries and criteria intuitive but sometimes I'm left wondering if what I wrote correctly maps to MongoDB. Here is handy way to check.

Criteria c = where("us_state").is("MI").and("location").exists(false);
System.out.println("criteria: " + c.getCriteriaObject().toString());

output:

criteria: { "us_state" : "MI" , "location" : { "$exists" : false}}

This can be cut and pasted into the MongoDB shell.

-jk

Mongo query - documents with none empty array

John Kern — Tue, 19 Jun 2012 14:15:00 +0000

Hello,

I'm working with MongoDb. My dataset has an array called "campusSet".

> db.institutions.find({}, {"campusSet":1}).limit(3)
{ "_id" : 1048598, "campusSet" : [ { "_id" : null, "address" : "917 Missile Road", "city" : "Sheppard AFB", "us_state" : "TX", "zipCode" : "76311-2263" } ] }
{ "_id" : 1048613, "campusSet" : [ ] }
{ "_id" : 1048628, "campusSet" : [ ] }

One can ask for documents with an empty campus array.

> db.institutions.find({"campusSet":[]},{"campusSet":1}).limit(3)
{ "_id" : 1048613, "campusSet" : [ ] }
{ "_id" : 1048628, "campusSet" : [ ] }
{ "_id" : 1048643, "campusSet" : [ ] }

I want to know the inverse, that is, documents with values in the campusSet array. Mongo does have a not operator called naturally enough $not but strangely, you must apply it to another constraint. So, you can't say

> db.institutions.find({"campusSet": { "$not" : [] }},{"campusSet":1})
error: { "$err" : "invalid use of $not", "code" : 13041 }

Further, I tried.

> db.institutions.find({ "$not" : {"campusSet": [] }},{"campusSet":1})

It simply returns which I do not understand. Is that the empty set? Why?

Fortunately, mongo does provide a "$size" operator. Combining it with "$not", gives one the desired effect.

> db.institutions.find({"campusSet": { "$not" : {"$size" : 0}}},{"campusSet":1}).limit(3)
{ "_id" : 1048598, "campusSet" : [ { "_id" : null, "address" : "917 Missile Road", "city" : "Sheppard AFB", "us_state" : "TX", "zipCode" : "76311-2263" } ] }
{ "_id" : 1048728, "campusSet" : [ { "_id" : null, "address" : "34091 Highway 280", "city" : "Childersburg", "us_state" : "AL", "zipCode" : "35044" } ] }
{ "_id" : 1049159, "campusSet" : [ { "_id" : null, "address" : "1805 Hillyer Robinson Industrial Parkway. Suite B", "city" : "Anniston", "us_state" : "AL", "zipCode" : "36207" } ] }

-jk

Viewing Communities as Funnels

Sat, 16 Jun 2012 15:22:35 +0000

It has been ages since I have had time to write a blog post. Last week, I decided to get my act together and start again. Of course the cold and wet summer in the UK also helps a bit. I have accumulated a lot of material and experience since my last post. Today, I will start with an idea that I first published at Open World Forum 2011, which marries the idea of a Sales Funnel with software adoption in open source communities. It then extends that model to the progression of a potential user of your software, to becoming a user and possibly a developer in your project. I will also show how the funnel can be used as a health dashboard for your community and as a community management tool.

Besides having found its way into the Sales Process, the funnel analogy is commonly used in marketing, for Freemium Adoption Models as well as Social Networking and also Information Seeking.

Key Constituencies in an Open Source Community

Before looking at funnels, it is worth looking at constituencies in an open source community. It should be relatively straightforward to define constituencies for communities in general.

Industry Segment: It helps to understand what industry segments your project is a part of. For Xen.org it is virtualization and cloud, for MySQL and PostgreSQL it is databases, etc. Why should people in a project care? A common project goal is to grow the project’s user base. Being known by as many people and vendors in your industry helps achive that goal. You will be in even better shape, when seen in a positive light.

Interested People care about the industry segment you are in and follow the news for your project regulalry. In other words, they care enough about your project to follow what is going on. People in this group may not necessarily be users (there may not be an immediate need to use your software). Their main goal is to keep themselves informed.

Users: These are direct users of your software or project. I tend to also include users of commercial software that builds on top of your project, as long as they know they use your software. This is not necessarily a given: for example AWS is built on top of the Xen hypervisor, but a large number of AWS users will not know that they use Xen. It frequently makes sense to group users by the level of engagement users give to stake-holders within and ourside of your community.

Developers are individuals (and companies) that develop the software in your project. Typically these are also users.

Progression

People progress from the outer circle to the inner. At each step, there will be factors that influence progression. For example:

How well is the project known
News and social media activity about the project
How easy it is to get started
How easy it is to overcome difficulties and to get support
How vibrant and active is the developer community
Etc.

This is all very well understood by most open source practitioners. But it is sometimes difficult to explain all this in a simple model that can be easily understood by your manager or people whose support you need. In fact, having had difficulties getting funding for activities I wanted to start, led me to search for an effective tool to pitch my ideas. I found that there were no off-the-shelf solutions, but realized that communites can be described as funnels; albeit funnels with a difference to say a sales funnel.

The Community Funnel

The way I think about the funnel model is as a permeable and flexible membrane (or sieve) in the shape of a funnel. People travel through the funnel and drop out at various stages. As people travel through the funnel they assume roles which relate to typical activities and behaviours. This gives us one view of the Community Funnel.

Then there are factors which lead to people dropping out of the funnel and other factors which keep them in. In other words: different areas of the funnel may have different permeability. Usually it is possible to indirectly (or directly) measure the permeability of the funnel boundary. It is also possible to take action to reduce (or increase) permeability for sections of the funnel.

In software adoption and particularly FOSS adption, a crucial stage of recruiting users is the trial of your software. Say, it is hard to find your download page, there are few useful instructions on how to get started or the software doesn’t work, a prospective user typically will give up and won’t return. A bad experience means my funnel is more permeable in the trial stage than a good experience. By improving the trial experience more people will progress to the next stage.

The diagram above shows some example factors that affect the permeability of the funnel boundary. As an aside, the shape of the funnel (or rate of drop-outs at different stages) is controlled by the permeability of the boundary at different locations and the size of the mouth.

The goal of a community leader or manager is to reduce the permeability of the boundary and thus keep more people progressing down the funnel.

Constructing a Funnel for my Community

At this stage it is worth noting that not all communities have the same goal. Typical goals for open source communities are:

Increase user base
Build a vibrant and diverse developer base
Secure the long-term future of the project
Upsell services or software to my users

The goal of a community influences the roles people assume, what they typically do, what they care about and thus the factors that keep people in the funnel. As a community leader or manager, it should be possible to create a community funnel that reflects the goal of the community, the roles people assume as they travel down the funnel, their typical activities as well as factors that influence the permeability of the boundary. I will leave it up to the reader to test this and share ideas. I have tested this for the community goals listed above and believe that the model is flexible for most communities.

Metrics and Funnels

The Community Funnel Model is also extremely useful as a visualization tool. Open source community leaders usually measure a number of metrics, such as

Media and press coverage
Social media buzz
Website visits
Comments on a blog
Downloads and upgrades
Number of conversations (answered and unanswered) on fora, mailing lists, etc.
Contributions
Etc.

Such metrics can be mapped to a section on the funnel boundary. I usually set myself goals for these metrics and apply a simple traffic light scheme: red (bad), yellow (ok), green (good). I then map visualize select metrics on the boundary of the funnel, in effect using the funnel as a visual dashboard.

Going one step further, metrics related to the effectiveness of community activities and programs can also be displayed in the funnel using the traffic light approach.

When I started to work for Xen.org, lack of good documentation and the capability to find documentation was an issue that was frequently raised on mailing lists. To address this, we improved the infrastructure to make it easier to contribute and find documentation (by moving to MediaWiki) and started a monthly Documentation Day. This is a monthly on-line meeting where the majority of developers and many interested users get together and fix documentation. I can measure the effectiveness of Documentation Day by

Attendance on the day itself
The number of changes/additions made to the wiki on the day itself
The number changes/additions made to the wiki overall
The views and users of the wiki

Placing community health metrics as well as indicators for community activities onto the Community Funnel, gives me an excellent overview of how well my community is doing.

The Community Funnel as a Community Management Tool

Generally, I found that the funnel analogy is extremely useful in many circumstances. It has been particularly powerful in the following circumstances:

Assess the health of my community: the funnel serves as a model for my community and a health dashboard in one diagram. Combining these two is very powerful.
To communicate the importance of my community and the need to increased spending on certain activities. The funnel model is intuitively understood by most people. Due to the popularity of sales funnels in sales and marketing, a pitch for funds involving the community funnel, makes your pitch more easily understood by decision makers.

The funnel analogy is also extremely useful to help me prioritize what to focus on by providing insights which are otherwise easy to miss: typically issues on the mouth (left) of the funnel lead to more people dropping out early. Fixing issues on the left affects more of your community over time and has thus a larger impact. Because the progression of people along the funnel takes time, issues at the mouth take longer to have a positive impact. For that reason you may want to focus on the left first. Other insights can be clusters of issues in one area of the funnel. Of course these are oversimplifications: but they help put things into perspective and provide a thought framework. A concrete example where this is useful, would be the question of whether a community should first fix issues with software adoption or a developer governance issue. If both solutions take similar effort, it is more advisable to fix adoption issue first. Even though this is clear to most community leaders, the funnel analogy can help explain your thinking better to others.

Conclusion

My personal experience shows that the Community Funnel is extremely useful for …

visualizing the health of your community
explaining how you manage a community to stake-holders in your community
making pitches to get extra funding
planning changes and evolving a community

I would be very interested to hear other views, get feedback and comments. And of course I hope that some of my ideas will help your project and community.

Lesson Learned: don't use $ in R formulas

John Kern — Thu, 22 Mar 2012 12:42:00 +0000

Hello,

Today I was using Rminer for a class project. I ran into a small problem.

SV <- fit(trainingData$area~temp+rain+wind+RH,trainingData, model="svm",search="heuristic")

Error in if (transform == "scale" && B != 0) x <- (x - A)/B else if (transform == :
missing value where TRUE/FALSE needed
In addition: Warning messages:
1: mean( ) is deprecated.
Use colMeans() or sapply(*, mean) instead.
2: sd( ) is deprecated.
Use sapply(*, sd) instead.

Suboptimal. What has gone wrong? I stopped in svm.fit() and I confirmed the data frame still had 400 rows. However, the outindex variable becomes

Browse[3]> outindex
integer(0)

and thus the data frame becomes truncated.

Browse[3]> data[, outindex]
data frame with 0 columns and 400 rows

why is outindex 0?
Browse[3]> output_index

svm.fit() uses the output_index() function to determine the index for the labels based on the formula.

outindex = output_index(x, names(data))

The function assumes the formula will only contain the field name not the fully qualified obj$field name.

function (x, namesdata)
{
return(which(namesdata == (as.character(x)[2])))
}

The formula was trainingData$area~temp+rain+wind+RH. So trainingData$area
Browse[3]> as.character(x)[2]
[1] "trainingData$area"

Doesn’t match "area".

Browse[3]> names(data)
[1] "X" "Y" "FFMC" "DMC" "DC" "ISI" "temp" "RH" "wind" "rain"
[11] "area" "sun" "mon" "tue" "wed" "thu" "fri" "sat" "jan" "feb"
[21] "mar" "apr" "may" "jun" "jul" "aug" "sep" "oct" "nov" "dec"
Better error checking would be most welcome but lesson learned. Don’t use the $ operator, which means to extract or replace parts of an object, in a formula.
-jk

Debugging R functions

John Kern — Sun, 11 Mar 2012 14:52:00 +0000

Hello,

I'm using R for a introductory class in data mining. For the SVM, we are using the e1071 package.
While working on a homework problem, I learned a few things about. Since svm() is the function to create an SVM model, plot.svm() plots it, right? The example on the help page works great.

## more than two variables: fix 2 dimensions
data(iris)
m2 <- svm(Species~., data = iris)
plot(m2, iris, Petal.Width ~ Petal.Length,
slice = list(Sepal.Width = 3, Sepal.Length = 4))

# Now let’s try the wine data.
# Next treat the last column as a numeric.
modelRegression <- svm(wineData$quality~., data = wineData)

# and call plot.
plot(modelRegression, wineData, quality ~ alcohol,
slice = list(
fixed.acidity = 8.32,
volatile.acidity = 0.5278,
citric.acid = 0.271,
residual.sugar = 2.539,
chlorides = 0.08747,
free.sulfur.dioxide = 15.87,
total.sulfur.dioxide = 46.47,
density = mean(wineData$density ),
pH = mean(wineData$pH),
sulphates = 0.6581
))

No plot is drawn. No error or warning generated. The function simple returns. Suboptimal.

Turns out R has some debug functions. Let’s use them to confirm plot.svm() is called.

> debug(plot)
> plot(modelRegression, wineData, quality ~ alcohol,
+ slice = list(
+ fixed.acidity = 8.32,
+ volatile.acidity = 0.5278,
+ citric.acid = 0.271,
+ residual.sugar = 2.539,
+ chlorides = 0.08747,
+ free.sulfur.dioxide = 15.87,
+ total.sulfur.dioxide = 46.47,
+ density = mean(wineData$density ),
+ pH = mean(wineData$pH),
+ sulphates = 0.6581
+ ))
debugging in: plot(modelRegression, wineData, quality ~ alcohol, slice = list(fixed.acidity = 8.32,
volatile.acidity = 0.5278, citric.acid = 0.271, residual.sugar = 2.539,
chlorides = 0.08747, free.sulfur.dioxide = 15.87, total.sulfur.dioxide = 46.47,
density = mean(wineData$density), pH = mean(wineData$pH),
sulphates = 0.6581))
debug: UseMethod("plot")
Browse[2]> n
debugging in: plot.svm(modelRegression, wineData, quality ~ alcohol, slice = list(fixed.acidity = 8.32,
volatile.acidity = 0.5278, citric.acid = 0.271, residual.sugar = 2.539,
chlorides = 0.08747, free.sulfur.dioxide = 15.87, total.sulfur.dioxide = 46.47,
density = mean(wineData$density), pH = mean(wineData$pH),
sulphates = 0.6581))
debug: {
if (x$type < 3) {

It is called. It has a guard condition. If the type if the model is >= 3, plot.svm() will simply return.

A closer reading of the documentation for plot.svm() reveals, “Generates a scatter plot of the input data of a svm fit for classification models by highlighting the classes and support vectors”. The types are defined by svm().

type

svm can be used as a classification machine, as a regression machine, or for novelty detection. Depending of whether y is a factor or not, the default setting for type is C-classification or eps-regression, respectively, but may be overwritten by setting an explicit value.
Valid options are:

C-classification
nu-classification
one-classification (for novelty detection)
eps-regression
nu-regression

They start counting at zero. This function should really be called plot.svm.classifiction().
-hth
John

“It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so.” Mark Twain.

Gotcha: Converting factors to numerics in R

John Kern — Wed, 07 Mar 2012 10:25:00 +0000

Hello,

Over the past month or so, I have be using the R language for an introductory course on data mining and machine learning. Today I wanted to talk about a language design decision with respect to R which I found surprising. R uses factors to represent a discrete classification. A good example is eye color: "brown", "blue" and "green".

Yesterday, I was working with a k-nearest neighbor library which returned its prediction as a factor of values: "3","4,"5,"6","7" and "8".

# kNN

predictKnn <- knn(trainingAttrs,testingAttrs,trainingLabels, k=bestK)

# let's consider the first 10 elements.

zz <- predictKnn[1:10]

> zz

[1] 5 5 5 5 6 6 5 6 5 6

Levels: 3 4 5 6 7 8

> typeof(zz)

[1] "integer"

> class(zz)

[1] "factor"

To compare this with previous results, I needed to covert it to numeric. Let's apply the as.numeric() function.

> as.numeric(zz)

[1] 3 3 3 3 4 4 3 4 3 4

Strange, did R just substract two? To get a better understanding let's consider a different set of factors. The variable ff represents eye color.

> ff

[1] blue brown brown green blue brown

Levels: blue brown green

> class(ff)

[1] "factor"

There is no natural way to convert these strings into integer. What does R return?

> as.numeric(ff)

[1] 1 2 2 3 1 2

Upon closer inspect you'll notice these numbers correspond to the strings position in the vector returned by the levels() function.

> levels(ff)

[1] "blue" "brown" "green"

So what's happening to the original zz vector is R is returning the levels() index for each element in the vector. Here is how the help page for factors recommends doing this convertion.

> as.numeric(levels(zz)[zz])

[1] 5 5 5 5 6 6 5 6 5 6

-jk

Driving Mobile Application Downloads

Tasneem Sayeed — Mon, 07 Nov 2011 18:19:00 +0000

With iTunes now carrying 225,000 apps in its "curated" collection and Android up to 100,000, it is no wonder that app discovery is a major issue for application developers.

Undoubtedly, one of the key decisions in launching a successful mobile application is determining how to drive application downloads.

Based on experience launching applications working in several mobile startups, I believe that there are at least six key mobile strategies to consider for driving adoption:

Why is mobile media support needed?

Mobile media support is needed since we need to reach consumers who may already have a mindset or may already be used to certain types of applications (i.e. facebook, twitter, etc). There may also be a need to educate users on issues such as privacy, security, and so on.

What should be the average marketing spend?

As a result of analyzing download trends and media spend across various advertisers, several marketing experts have determined that the minimum mobile media spend is $200,000 over two days to truly make an impact and accelerate measurable downloads.

When should you begin mobile media support?

The day of week apparently matters. To leverage typical download activity, it is suggested to launch your application and begin supporting it toward the end of the week - on a Thursday or Friday. This is allow you to build on the momentum of the weekend downloads.

How robust should the mobile application be?

It is critical to make your application robust because "first impression" matters. If an application frequently crashes or if critical functionality does not work properly, then it is likely that your application may not be subsequently used. While multiple features such as video, pictures, gaming and maps make for a bigger experience, if consumers cannot download the application and subsequent updates in certain WiFi zones, then it will most likely inhibit downloads.

Which form of advertising is more important: creative or contextual?

Based on mobile campaigns with those of digital display, marketing experts have concluded that in mobile media efforts, the banner creative has more of an impact on performance. However, in online campaigns, contextual is typically more important.

Which mobile devices need to be supported?

If you want to maximize coverage, then it is recommended that your application must be cross platform (i.e. must support iOS devices such as iPhone 3, 3GS, 4 and Android devices such as Droid X and/or HTC Evo). However, keep in mind that the uber-smartphone market represents less than 20 percent of smartphone penetration. Therefore, by including all phones that can access the Web, you will not only build scale, but you may also experience higher conversion rates from some of the less sophisticated devices.

Driving application downloads is not the only factor for launching a successful mobile application, but it is certainly an important one.

If you found this article useful, please retweet and/or leave me a comment.

Sequence Squeeze Contest

John Kern — Wed, 26 Oct 2011 07:29:00 +0000

Hello,

Pistoia Alliance, a trade group composed of big pharma and a bunch of informatics companies, is sponsoring a contest. The SequenceSqueeze challenges developers to develop an open source compression algorithm for Next Generation Sequencing(NGS) data.

-jk

ACM hosts Data Mining Camp

John Kern — Wed, 12 Oct 2011 18:03:00 +0000

Hello,

These days massive amounts of data from a wide diversity of sources. I very interested in investigating how we can leverage data in general. The local chapter of the ACM has organized a data mining camp this Saturday. I'm planning to attend.

The morning session is an introduction to MapReduce. It looks pretty rigorous. Looking forward to it.

-jk

Learning from Extremes

John Kern — Mon, 11 Jul 2011 07:40:00 +0000

Charles Leadbeater gave a provocative TED talk called "Education innovation in the slums".

He offers many references to interesting people and projects. It can be tough catching them in the video. However, it is easy to find them in the companion white paper called Learning from Extremes.

I particularly like the simply graph comparing setting (i.e., formal vs informal) and innovation type(i.e., sustaining vs disruptive).

"Improve schools through better facilities, teachers, and leadership." (sustaining/formal)
"Supplement schools by working with families and communities." (sustaining/informal)
"Reinvent schools to create an education better fit for the times." (disruptive/formal)
"Transform learning by making it available in radically new ways." (disruptive/informal)

Many of the ideas he presents come for other countries like Sugara Mitra Hole in the Wall project in India or Rodrigo Baggio's Centre for Digitial Inclusion.

-jk

Learning environments - adapting to change

John Kern — Sun, 10 Jul 2011 08:15:00 +0000

I'm taking a sabbatical. Initially, I was trying to apply computers to environmental issues. While there is a lot happening with alternative energy, the smart gird and optimizing supply chains, I found I was specifically interested in biodiversity and conservation. Sadly, there isn't much happening here.

After struggling with this, I hit a small epiphany. I was watching a TED talk by Nic Marks on the happiness index. While I strongly agreed with it morally, I'm not convinced we can measure happiness. Anyway, he presented a chart. On the X axis is natural resource consumption and the Y axis is "happiness". As you would expect, Western nations were in the upper right. They consume a lot of natural resources and are pretty happy. Nations in the lower left were mostly failed states where life is short and brutish. He highlighted a country that is light on natural resources and is pretty happy: Costa Rica. He went on to talk about the billions of people that will be coming on the planet and wanting a Western lifestyle. Most of whom will start life in slums of large cities. Normally when I think about this line of reasoning, providing people with material goods (e.g., cars, refrigerators, toasters etc.) comes to mind. However I have recently been reading some work by John Seely Brown former head of PARC. He just wrote a book called A New Culture of Learning. He observes change is accelerating. Over the course of the book he considers how computers systems can help people develop advanced learning environments. In addition to materialistic things, this new population will want an education. Standard methodologies of building schools and staffing them with teachers and faculty will not scale. Can we create learning environments where people could learn outside of the establishment? Many people consider the population explosion in environmental disaster. Can we turn that on its head by providing learning environments which allow them to collaborate and adapt to change quickly?

-jk

Update on Carbide, EDC, Symbian, Nokia

Ken Ryall — Mon, 16 May 2011 13:01:00 +0000

Time for an update on the state of Carbide, the Eclipse Debugger for C/C++ (EDC), Symbian, and Nokia. As usual all comments and analysis are my own.

We finally shipped Carbide 3.2 to the public! This is the first publicly available 3.x version of Carbide and the first new version seen outside of Nokia in over a year. It includes support for a new build system, a brand new on-device debugger built on top of EDC, and lots of usability improvements. The delay was a result of one of the many iterations of Nokia's strategy over the past few years: for much of 2010 and until February 11th of 2011 we were telling developers that our mobile platform was the Qt app framework, not Symbian. The Qt folks were busy building their own IDE (Qt Creator) and so it was thought that developers would be confused if they had Carbide available as well. So Carbide was designated a "platform tool" only for use by Nokia's thousands of internal developers.

Then on February 11th our new CEO announced that Nokia would be using Microsoft's Windows Phone 7 software in the future and Symbian would be phased out over the next couple years. So our team has ended up in a strange position: we need to keep Carbide up to a high standard to support Nokia's continuing Symbian efforts but we know the plug will be pulled at some point, probably within the next year. Nokia has built in some flexibility for it's Symbian roadmap and arranged a soft landing by working out a deal to transfer 3000 people to Accenture at the end of this year. Accenture will then contract us back to Nokia to do the remaining Symbian work.

Our team's chief contribution to the Eclipse community over the past 24 months has been the Eclipse Debugger for C/C++ (EDC), part of the CDT project. Faced with the need for a new C/C++ debugger for Carbide and unhappy with any of the available options we designed and built one from the ground up. But the ground was already pretty rich: we leveraged all the work done in CDT's Debug Services Framework (DSF), used the CDT language parser for expression support, and relied on the Target Communications Framework (TCF) for low level debug services.

The first version of EDC shipped with Helios but it was pretty raw as we hadn't yet shipped a Carbide product using it. Indigo EDC is a big improvement in stability, performance, and feature set and we got to fix a number of things we didn't get right on the first attempt. The two reference debuggers provided with EDC (Windows & Linux) still aren't near production quality but are great for introduction and testing and provide a good example of how to build a debugger around EDC.

As we developed EDC we found other tools people interested in the idea and it reflects valuable contributions from the CDT community. As Carbide winds down I hope it continues to offer some value to anyone moving beyond their legacy debugger. I'd like to continue improving it and the rest of CDT but that awaits a volatile future.

Just Released: FREE Sudokroid (Sudoku Game Puzzle) in Android Market

Tasneem Sayeed — Fri, 06 May 2011 01:23:00 +0000

More than 10 billion apps have been downloaded since the launch of the Apple (NSDQ: AAPL) App Store—the majority of them via iPhones. As of March 2011, 37 percent of mobile consumers who owned a smartphone had a device with an Android OS. Apple’s iOS, claimed by 27 percent of consumers, is now outpacing Blackberry, which has 22 percent of the market.

It only seemed appropriate that one of the first mobile apps I decided to release was an Android app, Sudokroid, to the Android Market. Sudokroid is a logic-based number placement puzzle, which initially starts with a partially completed 9x9 grid. The game provides the player with the options to start a "New Game", "Continue" an existing game, an "About Game" option to get further details for playing the game, and finally "Exit Game" to exit from the game.

When the player selects a "New Game" , it will let the player select the difficulty level from a choice of 3 levels: "easy", "medium" and "hard". A new game grid is displayed, partially filled, while splendid soothing music is playing in the background! If "Continue" is selected, then the game continues from the previous game state.

The goal of Sudokroid is to fill the grid so that each row, each column and each of the 3x3 boxes contains the digits 1 to 9 exactly once. Hints are provided to help in the selection, and if a selection results in "no move", then the player can go back to the previous tiles and revisit some of the prior moves.

You can download the game for FREE from the Android Market from http://bit.ly/mO8qfE. Be sure to rate it and tweet or leave me a comment ! Stay tuned for more apps in the pipeline!

GSoC: first few lessons learned

Tue, 26 Apr 2011 05:28:33 +0000

For the first time this year I have been admin for a Google Summer of code mentoring organization. Now as Google has announced the students, and while my impressions are still fresh, it’s time to share some lessons.

Don’t allow old project ideas

I allowed a few project ideas from 2010 into the 2011 list. None of them were very popular with students. I think there are two reasons:

It’s too easy for mentors to just pick up an old idea. In the end there is a risk that the mentor doesn’t take GSoC and engaging with the students as seriously as he should and don’t get into the right mindset.
Applying for a GSoC project is a bit of a gamble for students too: good students will look at ideas lists from several years and will notice that an idea is old and just not apply.

So, if an old idea is used (because it is still relevant), it is probably best to totally re-evaluate and write it from scratch.

Pro-active mentors = Lots of good student applications

We had 13 mentors, who fell roughly into two camps. Mentors who communicated with the students, but could have done better. Mentors who replied to students questions within a couple of days and ensured that there was communication about the proposal, the background, how to solve the actual problem, etc. Much of this discussion happened on a dedicated IRC channel. The projects by the mentors which were pro-active, had more student applications and the applications were of higher quality. As a result, the students working with pro-active mentors were more likely to win. Not entirely surprising. The lesson is, that as an org admin, it is important to work with the mentors early on and filter out proposals early if it becomes clear that the mentors behind these are may not be fully committed.

Unsolicited proposals need discussion with a mentor to succeed

We had a number of unsolicited GSoC proposals. They fell into two categories:

The proposal just appeared with little or no discussion with the mentor; in some case no mentor could be found. Needless to say, the proposal could not succeed.
The proposal idea was raised by the student in the community, which helped find a mentor early on. We had two such proposals, one of which succeeded.

As an org admin, the challenge is to provide ways of enabling this in a structured way.

A dedicated IRC channel and IRC meetings work

We created a dedicated IRC channel where I and a few of our mentors were present at certain times of the week for students to ask questions. The channel also was used as a channel for mentors and students to have an IRC discussion at an agreed time. We also had a timed open IRC meeting which worked very well. Maybe one meeting is not enough: probably running one at the beginning and one towards the end of the application period is better.

Build gates and motivators into your program

Having a few goals to work towards, both for mentors and students helps focus minds. The GSoC timetable does this, but this is not enough. For example, some of our mentors participated in a mentor meeting early on, to discuss and formulate project ideas. The result were well formulated ideas. The meeting also helped build relationships amongst mentors: agreeing on which students should be chosen can be divisive and knowing each other helps. IRC meetings were good to stimulate engagement between students and mentors and led to draft proposals being made available by students early. And we had mentor meetings to rank student proposals (I disabled the ranking mechanism and forced mentors to attend the meetings).

Encourage good students to hedge their bets

We had a few projects, which had 5 students competing with each other. You could end up in a situation where the top 5 students apply for one project and only one can win. If there was an engaged discussion between mentor and student, the mentor will get a feeling for such situations and should encourage students to apply for another project, or even two. However, this only really works if mentors work together and co-ordinate.

A well run program will lead to repeat applications

Despite having to turn away good students (we just had too many good proposals), we had a few students thanking us for giving them support and leading them through the process. Hopefully these students will apply again in the following year and succeed.

Getting on top of Social Networks

Tue, 29 Mar 2011 15:07:36 +0000

As community manager I should know better and regularly publish on the blog. Well, it’s been a very busy beginning of the year and taking over an existing community creates its own set of challenges. Besides community managers, I’d expect that many open source project leads which take over an existing project will face similar issues. The issue at hand this time: taking control of existing social networks.

Becoming manager of existing social networks, groups, etc.

The community I work has a presence in 40+ different social networks and other on-line services. Some support the concept of a group admin/manager. Others don’t and admin rights tend to be tied to a specific user ID. When you try and take over management of some social networks you can end up in a number of situations.

The easy one is where there is the concept of an admin or manager: all you need to do is to ask another admin/manager to promote you to manager. Now this works easily if you your existing manager cares and knows their log-in details. Which for me was luckily mostly true.
The more probematic case is where managing a group is tied to a specific account and manager ownership cannot be transferred: there are quite a few sites where this is the case. What you essentially need to do is: get the password and change the profile. If the profile was personal, e.g. Joe Bloggs, make it generic (i.e. changed it from Joe Bloggs to Xen Community Manager) otherwise the next person will face the same problems. I also changed the profile such that it mentioned that Joe Blogs was community manager during a specific time period. This was necessary because some of the history in comments refers to Joe Bloggs, which may confuse users.
The worst case scenario is where the management role is tied to a specific universal ID and cannot be changed (such as a Google, Twitter or Facebook ID). I was surprised to find out that there are a quite few such services. In this case, you are stuck if your predecessor has used a personal ID which he/she will want to continue to use.

It was a little bit surprised that a seemingly simple task like taking over the management of a few social networks was actually hard, tedious and rather inconvenient. Much harder than it should have beeen.

You can also run into other interesting problems:

A previous manager may have forgotton a password
Or their account was tied to an e-mail address which does not exist any more (because he left a few months back)
In the meantime the account may have been disabled because an e-mail to said account bounces

From my experience dealing with the support teams from some of the free social networks and services was slow and painful; in some cases non-existent. Where there was really good support, these services obviously needed to ensure that you didn’t try and abuse the service and take over somebodies identity. A disabled e-mail address does not help there.

Thus, plan for the future

The only way how to avoid such problems is to plan for the future. Basically you need to assume that you are run over by a bus tomorrow and ask yourself the question how easy would it be for successor to get started.

Here are some tips how to do this:

Where there are admins, make sure there are always several admins besides you. Make sure you always keep the list of admins up-to-date (i.e. replace an inactive admin with a new one).
Make sure the list of admins is written down somewhere. Some services don’t allow you to see who their admins are: so if you don’t know you can’t contact them. Googler Analytics is an example of one of those services.
Avoid using user IDs that are tied to your person. It’s OK to use your e-mail address if it is not used to identify your account and can be changed later.
Be careful not to use your personal twitter, google or other ID to create a new group. In particular when you know it cannot be changed later: or when you do not know. In this case, you probably want to create an identity which is tied to the job rather than yourself.
Make sure passwords are stored somewhere and shared with at least another person in your organisation.

Getting on top of the social media landscape

When you start in a new community, one of the other challenges is to find out what goes on on the web. The easiest way to do this, is to use commercial tools such as Lithium or Radian6 (note that by far these are not the only ones; google for “Social Media Monitoring” to find such services). Unfortunately almost all of these services are incredibly expensive and practically out-of-reach for open source users.

There are free alternatives

There are a number of free alternatives: you can use socialmention.com, which works really well if your project or community has a unique name. Google alerts also works well in this case. You start getting into trouble when your project has a common name. Well, Xen is such a common name and you will need to use complex search expressions to exclude keywords as you come across false positives. In the case of Xen, I need to exclude an ever growing list of terms such as guru, travel, incense, art, yoga, … you name it. In this case, brush up your skills on creating complex search expressions (I thought I saved a link for you, but didn’t … sorry).

However I found that even with complex search expressions, you still will get a lot of noise. So the only way how to eventually get on top of these is to use Google Reader, Yahoo Pipes or tarpipe to create yourself an aggregated stream of trustworthy sources as you come across them using search, socialmention, etc.

You should also be able to use Google Custom Search, but I have not tried this.

How do others cope?

This is it for today: my bit of recent wisdom. I would really be interested to hear how others in open source or other community managers grapple with these problems.

And, I promise that I will at least write one article per month from now on.

Copying Memory

Ken Ryall — Thu, 17 Mar 2011 16:04:00 +0000

I've been starting to clear some usability issues out of our backlog and ran into one that seemed simple enough but eventually required a little more thought and attention.

When debugging native code it's common to want to inspect memory so the Eclipse debug platform offers a standard Memory view. The view can host a number of memory renderings, each offering a different way of viewing a chunk of memory.

The "traditional" memory rendering is popular with C/C++ developers because it works much like the memory views in other debuggers and shows a range of memory in both binary and text formats.

"Traditional" memory rendering from CDT

Memory View in Xcode 4

Memory view from Visual Studio 2008

Memory window from CodeWarrior in 2001

Our developer (I'll call him Tom because that's his real name) wanted to do something that seemed pretty simple: select a hex value, copy it to the clipboard, and then paste it into a variable in the Variables view. But the global "Copy" command wasn't hooked up to the memory rendering so it did nothing. Tom then discovered "Copy to Clipboard" in the context menu.

But the "Copy to Clipboard" command copied all of the selected data from the rendering: addresses, binary, and text to the clipboard so it looked like this:

0x7FFF5FBFED20 74736574 00676E69 996E019D 3CBE8497 5FBFED40 testing...n...¾<@í¿_

Tom had to paste this into a text editor, then select just the binary/hex part, and finally had something to paste to set the value of his variable. This sort of thing delights some people because they find a work-around and then have another story to inflict on their colleagues, but Tom had real work to do, expected better, and logged a bug. Our team worked on other stuff that seemed more important until last week when I went back to take another look.

First I looked to see if the rendering's support for "Copy" could just do the right thing by grabbing the address, the binary data, or the text based on which part had been clicked in last. While that would work in the narrow case it wouldn't leave any indication of what it might do if you have forgotten which area you clicked in last, and the ability to copy all three (addresses, binary, text) would have been lost.

There needed to be a more explicit way to say which portion of the data you wanted to copy. So I changed the context menu to provide all four options.

This would make life better for Tom but still wouldn't fix the "select the hex data and hit control/command - c" scenario. So it also keeps track of one choice as the default option that is used when fired by the global "Copy" command, which is now hooked up to the rendering. Tom will have to select "Binary" once and then Copy will work the way he expected.