Continued – http://softwaredev.quora.com/Offering-Choices-Considered-Harmful

Continued - http://softwaredev.quora.com/When-in-Doubt-Iterate-Faster

Of late, I have been thinking of posting more often – mostly about my learnings in building software products. But you know, if you write you want people to read. Now coincidentally, Quora has just recently launched a blogging platform. I have been in love with Quora since a while. It has a highly knowledgeable audience, which is very engaged. And it is viral – as people comment and upvote your content, more people get to discover it.

And therefore I have decided to switch over my writing to Quora. The tech content would go on http://softwaredev.quora.com/ and the non-tech ones on http://www.quora.com/Vineet-Gupta/Posts. Or at least that’s the plan.

This was surprising since I most certainly did not cancel the subscription. Note that the mail provides no context as to how this cancellation took place – no tracking id, order-id, issue-id, etc.  It just says that it is confirmed that the subscription is cancelled.

I was worried that this may compromise our ability to publish updates to Talk.To for Windows Phone. In fact, just two weeks back I had renewed subscription to our Windows Developer account:

So I quickly logged into the marketplace to check what’s wrong and saw that there are two subscription options there:

The Windows Store Developer account was valid, but the App Hub Annual Membership was expired. Do I need one or both? If one, which one? Note that there is not much to distinguish between the two:

App Hub Annual Membership:

Now apparently Microsoft had announced some changes in Aug 2012 of App Hub transitioning over to Windows Phone Dev Center. I vaguely recall the announcement, but it seemed more like a branding change – I did not realize that it would lead to mails regarding account cancellation and such.

What does not help is:

1) The mail is a confirmation of subscription cancellation with no reference to how the cancellation took place, leave alone mentioning that this is part of an automated process and in what scenarios should a customer take some action.

2) On the Dashboard, there is a message that says that Subscription Status is Approaching renewal

3) If you click on that link, it takes you to Account Summary where you are prompted to Renew your subscription.

4) However, you have just renewed the windows developer account subscription and it shows correctly under https://commerce.microsoft.com/PaymentHub/Subscription?accountid=<your id> as being 11 months away!

5) So you think that perhaps you need both subscriptions since in any case the descriptions are similar – you cannot tell which one you do not need.

Could have been much better handled.

Try ssh git@github.com and you get:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: UNPROTECTED PRIVATE KEY FILE! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0660 for '/home/Vineet/.ssh/id_rsa' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
bad permissions: ignore key: /home/Vineet/.ssh/id_rsa
Permission denied (publickey).


You check permissions and find out:
-rw-rw---- 1 Vineet None 1704 Dec 23 22:57 id_rsa

So you run:
$ chmod 600 ~/.ssh/id_rsa

You expect the permission on the file to change, but it does not.  The reason is that the group in cygwin is set to ‘None’ whereas it should be set to ‘Users’

The quick fix is to change the group on all existing entries:
$ chgrp -R Users ~/.ssh.

Now run:
$ chmod 600 ~/.ssh/id_rsa,

This time the permission does change:
-rw------- 1 Vineet Users 1704 Dec 23 22:57 id_rsa

You can now ssh successfully. However, every time you create a new file, the group is set to None and you keep running into the issue of wrong permissions. Here’s how you can fix the root cause:

1) Check the group Id of the ‘Users’ group in /etc/group – it is the third field on the line starting with ‘Users’:
$ cat /etc/group | egrep '^Users:' | cut -f3 -d':'
In my case the value was 545

2) Check the primary group for your user in /etc/passwd – it is the fourth field on the line starting with <your username>
$ cat /etc/passwd | egrep '^Vineet:' | cut -f4 -d':'.
In my case the value was 513 (which was the value for the ‘None’ group in /etc/group).

3) Change this value to the one found in the previous step for ‘Users’ (545 in this example)

4) Exit cygwin and restart

Create a new file and the group should now be set to ‘Users’ by default. You should no longer get any permission errors.

Figuring out all this and fixing it took me well over 2 hours and all because of a silly bug in cygwin. I hope they fix it soon.

Most people would know of Dennis Ritchie as the person who invented C and Unix. What most folks do not realize is how profound C and Unix were when they were created and what amazing impact they have had on the industry.

Consider this – before C, you had to program directly to the instruction set of the machine you were programming on. Today when most hardware happens to be x86 or ARM, this seems simple, but back then, the variety of hardware was far more. And this hardware was different not just in terms of instruction sets, but also in terms of information representation, memory addressing, etc. Sure enough there were languages like Fortran and COBOL which could do math and data, but there was no general purpose language which could produce programs that could compete in terms of performance with custom code written for the hardware. Writing a portable program, much less a portable operating system was un-imaginable.

And yet, Dennis Ritchie imagined just such a world and invented a highly performant, high level programming language and then used that to write an operating system which could be ported to any hardware. This one change meant that programmers did not have to create a new OS for every new piece of hardware that got thrown at them – they could take for granted a set of tools, calls and programs, as being available and could spend time going up the value chain and doing more interesting things than just re-inventing the wheel. The result was a spurt of innovation and growth which allowed the industry to grow exponentially into a myriad different directions leading to the world we are in.

If you are a programmer, you owe a lot to Ritchie, whether you have ever programmed in C or worked on Unix or not.

[Edit: Bjarne Stroustrup expresses the same sentiment way more eloquently: http://herbsutter.com/2011/10/12/dennis-ritchie/

Most of us are happy to settle for incremental change. People like Ritchie don’t, and that’s what sets them apart.]

I never thought I would be so affected by the passing away of someone I never knew personally. But if you are in the world of tech and developing consumer products, it is not possible to have not thought about life as Steve Jobs – how his brain functions, how he figures out what should be built, how he sets the bar, how he hires and build teams, how he delves upon every aspect of the product, … you wish you had 0.1% of his ability, you wish you could be more like him, since being him means creating profound impact, and that is what you are really here for.

And this I believe is his true legacy. The products he helped create would have a shelf life of maybe a decade, Apple as a company would be perhaps super successful for another 2-3 decades, but what would inspire generations of current and future entrepreneurs, programmers and designers is the way he set about building products:

• leapfrogging the current state of the art, not incremental change
• relentless focus on the user
• not settling for less than perfection

This approach to building products is his true legacy, and for that, the world would be a better place.

PS: Could not cross link to references since there is no authoritative version of the C-99 standard in an HTML form – the standard docs are all in PDF.

C Declaration Grammar

Understanding a C declaration requires that we understand what elements can a C declaration have. While I will describe the overall syntax over here, I will not go into semantics:

declaration:
	declaration-specifiers init-declarator-listopt ;
declaration-specifiers:
	storage-class-specifier declaration-specifiersopt
	type-specifier declaration-specifiersopt
	type-qualifier declaration-specifiersopt
	function-specifier declaration-specifiersopt
init-declarator-list:
	init-declarator
	init-declarator-list , init-declarator
init-declarator:
	declarator
	declarator = initializer

This is what the above grammar is saying:

• A declaration is:
  • declaration-specifiers followed by an optional list of init-declarators
• The declaration-specifiers are:
  • A storage-class specifier, followed by other optional declaration specifiers
  • A type-specifier, followed by other optional declaration specifiers
  • A type-qualifier, followed by other optional declaration specifiers
  • A function-specifier, followed by other optional declaration specifiers
• List of init-declarators is:
  • An init-declarator
  • An init-declarator list, followed by a comma, followed by an init-declarator
• An init-declarator is:
  • A declarator
  • A declarator followed by an assignment operator (=),  followed by an initializer

Declaration Specifiers

As per the grammar defined above, a declaration starts with a declaration specifier. A declaration-specifier can be:

• Storage-class specifier: These are: auto, static, extern, register and typedef. Default is auto, so if nothing is specified, auto is assumed.
• Type-specifier: A type-specifier is one of the following. Default is int
  • void
  • char
  • signed char
  • unsigned char
  • short, signed short, short int, or signed short int
  • unsigned short or unsigned short int
  • int, signed or signed int
  • unsigned or unsigned int
  • long, signed long, long int or signed long int
  • unsigned long or unsigned long int
  • long long, signed long long, long long int, or signed long long int
  • unsigned long long or unsigned long long int
  • float
  • double
  • long double
  • _Bool
  • float _Complex
  • double _Complex
  • long double _Complex
  • struct-or-union-specifier
  • enum-specifier
  • typedef-name
• Type-Qualifer: Applicable only to l-values: const, volatile, restrict
• Function-specifier: There is only one function specifier: inline

Declarators

What follows a declaration-specifier is a init-declarator-list – a comma separated sequence of declarators that may optionally be initialized. A declarator [2] is defined as:

declarator:
	pointeropt direct-declarator
direct-declarator:
	identifier
	( declarator )
	direct-declarator [ type-qualifier-listopt assignment-expressionopt ]
	direct-declarator [ static type-qualifier-listopt assignment-expression ]
	direct-declarator [ type-qualifier-list static assignment-expression ]
	direct-declarator [ type-qualifier-listopt * ]
	direct-declarator ( parameter-type-list )
	direct-declarator ( identifier-listopt )
pointer:
	* type-qualifier-listopt
	* type-qualifier-listopt pointer
type-qualifier-list:
	type-qualifier
	type-qualifier-list type-qualifier
parameter-type-list:
	parameter-list
	parameter-list , ...
parameter-list:
	parameter-declaration
	parameter-list , parameter-declaration
parameter-declaration:
	declaration-specifiers declarator
	declaration-specifiers abstract-declaratoropt
identifier-list:
	identifier
	identifier-list , identifier

We can break down the above grammar the same way we did for the grammar of a declaration, but I’m gonna skip that and give some examples of declarators, just to give an idea of what declarators are supposed to be:

• i – An identifier (and hence a direct-declarator)
• i, j – A list of identifiers
• i = 10 – An Identifer followed by an assignment expression
• *p – A pointer declarator
• * const p – A pointer declarator with a type-qualifer (const)
• a[10] – An array declarator with a constant size
• a[*] – An array declarator with a variable size
• a[] – An array declarator with an unspecified size – the size will need to be defined somewhere else
• f() – A function declarator
• f(void) – A function declarator with no parameters
• f(int i) – A function declarator with a parameter
• f(int i, int j) – A function declarator with a parameter-list

Structure of a Declaration

Now that we have developed an idea of declaration-specifiers and declarators, we can see that the overall structure of a declaration is of the following form:

1. One or more declaration-specifiers followed by
2. One or more declarators (separated by commas)

However, not all combinations are valid. The following are not allowed:

f()[] // function can't return an array
f()() // function can't return a function
a[]() // array can't hold a function

The following, however, are allowed:

(* f())[] // function returning pointer to an array
(* f())() // function returning pointer to a function
(* a[])() // array holding pointers to functions

If you are flummoxed by the last three examples of combining pointers, functions and arrays, you are not alone. Such declarations can look scary till you develop a technique to start deciphering them. In order to do that however, we need to know the basic order of precedence

Operator Precedence

1. Parentheses grouping together parts of a declaration
2. Postfix operators: parentheses (for a function), square brackets (for an array)
3. Prefix operator: Asterisk (for a pointer)

Here are some examples of how this order applies:

Now the way to use these rules is as follows:

1. Read from left to right
2. For the first identifier you encounter (or if there is no identifier, then look for the inner-most construct), look to the immediate right
   • If there is nothing, or if you have a closing parenthesis, go to 3
   • Otherwise you have a function declarator indicated by () or an array declarator indicated by [] to the right of the identifier
   • Read left to right – you will have a “function returning” or “array of”
   • This would typically end with a right parenthesis, or the end of the declarator (semi-colon or assignment)
3. Look to the left
   • If you find nothing on the left, or if you find an opening parenthesis, go to 4
   • Otherwise you have a pointer declarator indicated by an asterisk to the left of the identifier. Read right to left – you will get a “pointer to”
   • This would end with a left parenthesis, or start of declarator
4. At this point you have either
   • A complete declarator – then you are done
   • Or an expression in parenthesis – go back to step 2.

If you did not realize this on reading the above description, what we are really doing in steps 2 and 3 is to convert the C declaration into a postfix expression by taking into account the lower predence of the asterisk. This same approach is used in the cdecl program given in K&R.

This will become clear with a few examples:

int * f();

• Start with identifer – f
• Parenthesis to the right – “function returning”
• Asterisk to the left – “pointer to”
• int

The postfix expression is:

f () * int // "f is" "a function returning" "pointer to int"

Here are a few more examples:

int * a[10];
// postfix expression: a [10] * int
// "a is" "array of 10" "pointers to int"

int (* a)[10]
// postfix: a * [10] int
// "a is" "pointer to" "array of 10" "int"

int **p;
// postfix: p * * int
// "p is" "pointer to" "pointer to" int

int **p[10];
// postfix: p [10] * * int
// "p is" "array of 10" "pointer to" "pointer to" int

int *f();
// postfix: f () * int
// "f is" "function returning" "pointer to" int

int (*f)();
// postfix: f * () int
// "f is" "a pointer to" "function returning" int

int (* vtable[])();
// postfix: vtable [] * () int
// vtable is an array of pointer to function returning int

int (* a[])(int, int);
// postfix: a [] * (int, int) int
// a is an array of pointer to function returning int
// and taking (int, int) as parameters

Applying Type-Qualifiers

The way you apply a type-qualifier (const, volatile) is that:

1. If next to a type-specifer, it applies to the type-specifier
2. Otherwise applies to the asterisk (pointer) on its immediate left

Examples:

const int * p;
// postfix: p * const int
// p is a pointer to a const int

int const * p;
// postfix: p * const int
// p is a pointer to a const int

int * const p;
// postfix: p const * int
// p is a const pointer to int

int * const * p;
// postfix: p * const * int
// p is a pointer to a const pointer to int

int * const * (* p)();
// postfix: p * () * const * int
// p is a pointer to a function returning a
// pointer to const-pointer-to-int

Abstract Declarators

An abstract declarator is a declarator without an identifer. However, lack of an identifier does not mean that we can’t interpret such declarators, since the missing identifier’s position can be determined by the placement of (), [] and * Some examples will help clarify this:

int *
// pointer to int

int * [10]
// array of 10 pointers to int

int * (*)
// function returing a pointer to int and taking no arguments

int (*) [10]
// pointer to array of 10 int

int (*) (*)
// pointer to a function returning an int and taking no arguments

With this background, lets tackle this gem from Andrew Koenig’s C Traps and Pitfalls (PDF):

(*(void(*)())0)();

Clearly, we are casting 0 to some type here. The type being:

void(*)()

We know this type:

// postifx: * () void
// pointer to function returning void

In the next step, this pointer is de-referenced and then called. So what is happening here is that zero is being cast to a pointer to a function returning void, then dereferenced and called!

After having gone thru the Android documentation and videos from Google IO, I found the documentation on Blackberry to be somewhat disorganized – it is not easy to get the big picture and grasp key concepts. You need to go thru a lot of documentation to figure out how things works together as a system. I also tried to access the materials from http://www.blackberrydevcon.com/2010-on-demand but it turned out that it would require a registration for $ 20. That was a bit of a WTF moment … I mean, you want to make money by selling conference materials? Who does that?? Anyway I proceeded to register on https://www.eventreg.com/pls/pg_rim/reg_rim.reg_login?icpid=&ipr_code=ONDNON and after filling two lengthy forms, was unable to make a payment despite getting a registration id. I wrote to BlackBerryDEVCON@eventreg.com with my registration id (this was on 3rd Mar), but am yet to hear from them. Rather discourgaing for somene starting off on a new platform.

The Blackberry Platform

The Blackberry platform consists of the following key components:

• Phone: Blackberry phones support two kind of applications:
  • Java Apps: These are native Blackberry apps which can take full advantages of all the features available in a Blackberry phone, for example the BBM API.
  • WebWorks Apps: Browser based apps built using HTML, CSS, JS and capable of accessing native APIs thru the WebWorks SDK.
• Tablet: One can build playbook tablet applications in two ways:
  • Adobe AIR apps: Based on Action Script 3 with a bunch of Blackberry specific APIs. Use this to build apps with a native look and feel.
  • WebWorks apps: Similar to WebWorks apps for the Blackberry phone.
• Platform Services: Bunch of services provided by RIM including Push, Location, Advertising and Payment services
• Connectivity Services: RIM also provides a couple of connectivity services:
  • Blackberry Enterprise Server (BES): Enterprises can make their mail infrastructure available over the Blackberry network using BES which acts as a relay between the mail server and Blackberry’s NOC co-located with a carrier providing Blackberry services. One of the components of BES called the Mobile Data System (MDS) provides  HTTP and TCP proxies for a Blackberry application. This allows a Blackberry device to communicate with app / web-servers behind a corporate firewall without using VPN.
  • Blackberry Internet Services (BIS): For individuals, the alternative to BES is the BIS. This provides the ability to access mail over POP and IMAP email without going thru BES. The service is provided in partnership with a carrier.
• AppWorld: The blackberry application store front, similar to the iPhone App Store

This post is specific to Java development for the Blackberry phones. Moreover, it is specific to Blackberry 5 since 75-85% of app downloads from the App World are from Blackberry 5 devices.

Java ME

Native apps on the Blackberry phone are Java apps that run on RIM’s JVM implementation. This Java API is based on Java ME with Blackberry adding its own extensions. If you are new to Java ME like I am, here is a quick review:

Java ME (Micro Edition) is Java targeted for small devices. It is a collection of specs and technologies to address the needs of providing a programming interface to small devices. Since there is a large variety in the types of devices and their capabilities, it is quite likely that a device manufacturer would implement some specs and not others, leading to API fragmentation and hence non-portable code. To address this, Java ME defines capabilities for devices which are organized into Configurations, Profiles and Optional APIs:

(Organization of the Java ME technologies – source: http://developers.sun.com/mobility/getstart/articles/survey/)

• Configurations: A configuration is a specification for a virtual machine + some base APIs designed for a specific kind of device, based on memory constraints and processor power.
  • Connected Limited  Device Configuration (CLDC): The CLDC is meant for small wireless devices with intermittent network connections, like pagers, mobile phones and PDAs
  • Connected Device Configuration (CDC):  The CDC is a superset of the CLDC meant for larger devices with robust connections, like Set-top boxes and internet appliances.
• Profiles: A profile is more specific than a configuration. A profile is based on a configuration and adds APIs for UI, Storage, or whatever else is required to develop apps. For CLDC, a profile called Mobile Information Device Profile (MIDP) is specified. For CDC, there is a hierarchy of profiles:
  • Foundation Profile
  • Personal Basis Profile
  • Personal Profile
  • Information Module Profile
• Optional APIs: A device implementing Java ME may define additional optional APIs besides the ones that are defined as a part of a Java ME profile.

Based on the configurations, profiles and optional APIs, Java ME is able to address a wide variety of small devices including Handsets, Smart-cards and embedded devices:

(Map of the Java ME Universe – source: http://developers.sun.com/mobility/getstart/articles/survey/)

A device implements a stack consisting of a Configuration, Profile and Optional APIs. For wireless devices like mobile phones, the stack is based on CLDC, MIDP and some optional APIs. An application targeting the MIDP profile is called a MIDlet.

(Stack for wireless devices. Source: http://www.oracle.com/technetwork/java/javame/tech/technology-139316.html)

As one can imagine, despite the CLDC and MIDP being defined, a lot of commonly required functionality and the way it is provided remains un-defined, thus introducing scope for API fragmentation and un-portable code. This was addressed thru JSR 185 – Java Technology for the Wireless Industry – a specification that addresses fragmentation by defining key APIs and code portability issues by clarifying spces and providing a suite of compliance tests. JSR 185 consists of CLDC 1.0 or 1.1, MIDP 2.0 and WMA (Windows Messaging API – JSR 120 and JSR 205), with support for MMAPI (Mobile Media API – JSR 135) being optional:

(JSR 185 – JTWI Stack. Source: http://developers.sun.com/mobility/getstart/)

This was superseded by the MSA (Mobile Services Architecture) specification which defined two stacks: a full stack of 16 JSRs and a subset of 8 JSRs:

(MSA Stacks. Source: http://developers.sun.com/mobility/midp/articles/msaintro/)

Blackberry 5 is based on CLDC 1.1, MIDP 2.0 and is JSR-185 compliant. It also supports the full MSA stack except for JSR 184 (3-D graphics), JSR 180 (SIP) and JSR 229 (Payment). For other JSRs supported by Blackberry 5, see http://docs.blackberry.com/en/developers/deliverables/9095/Support_for_standard_Java_APIs_446981_11.jsp

Blackberry Applications

Blackberry applications are of two types:

• MIDlet Application: A MIDlet is an application targeting MIDP (javax.microedition namespace). Such an app can run on other MIDP 2.0 compliant phones from other vendors
• Blackberry Java Application: A Blackberry Java Application (also called a RIMlet) targets the CLDC, but uses Blackberry specific APIs (from the net.rim namespace). Such an app can run only on a Blackberry device.

Unless your code is targeting mulitple Java ME platforms, it makes sense to build Blackberry Java applications since they tend to be a lot richer than MIDlets. Specifically, if you want the look and feel of applications that RIM ships with a Blackberry (also called “native apps”), then you should build a Blackberry Java application by using net.rim.device.api.ui and not use javax.microedition.lcdui. More differences on the UI capabilities of both frameworks are given on http://www.blackberry.com/developers/docs/5.0.0api/UI-summary.html.

Note that all APIs are always available to all apps. So if your app is using the Blackberry UI, you can always invoke MIDP APIs. If your app is using the MIDP UI, you can always invoke Blackberry APIs, but of course doing so would render the application non-portable. The only exception to this is on mixing UI. The MIDP UI model allows concurrent access while the Blackberry UI model does not and hence mixing UI library calls is unsupported.

In the rest of this post, I would be focussing mostly on the Blackberry Java Application.

Startup

A Blackberry application can start in one of the following ways:

1. Use clicking an icon on the Home screen
2. By another app
3. OS starting an app automatically when the device starts
4. OS starting an app at a scheduled time

This startup is managed by an entity called the Application Manager. An running application can obtain a reference to the Application Manager by calling getApplicationManager(), and then, can use this to:

• Run or Launch another application
• Schedule an application to run at a specific time
• Post a global event to the OS
• Lock and Unlock the system

To start an application, the App Manager creates a new process and a thread within that process and then starts the main() function on that thread with a set of input parameters. The parameters are specified in the application descriptor. It is possible to create multiple descriptors for the same app and associate them with different icons on the screen, This allows the creation of multiple entry points for the same app.

Event Processing

Once started, an application can choose to do a series of operations and then quit, never showing a UI. However, most applications want to stay around and respond to system events, including UI events. For this, one of the threads of the application needs to acquire what is called the Event Lock by calling Application.getEventLock(). Only one thread in the application can acquire this lock – this thread is then called the Event Thread. Typically, the first thread acquires this lock by calling enterEventDispatcher(). Once this method is called, the application starts receiving system events. Note that this does not mean that the application is showing a UI – just that is it receiving all system events, including UI events. So as of now, the application is still running in the background. An application can control whether it wants to continue processing systems events by toggling setAcceptEvents().

Showing a UI

All the visible elements in a Blackberry Java Application are of three types:

• Fields: A building block element that is used to work with a particular kind of data for which it draws the UI and if required handles user-input. It typically corresponds to a control like a button / text-field, etc. You can create custom Fields by sub-classing the Field class, implementing the layout() and paint() abstract methods and optionally overriding getPreferredHeight() and getPreferredWidth() methods.
• Managers: Responsible for arranging fields. Each field must belong to only one manager. Since a manager is also a field, managers are composable, making it possible to build complex, rich, UI layouts. There are several types of managers provided: Horizontal, Vertical, Grid, Flow, Absolute, etc., and you can also build your own by sub-classing the Manager class and implementing sublayout(), getPreferredHeight() and getPreferredWidth() methods and optionally overriding subpaint() method.
• Screens: A screen is a window that is active at any given time. There are two types of screens: PopupScreen and a FullScreen.

(Blackberry UI Framework. Source: http://www.blackberry.com/developers/docs/5.0.0api/UI-summary.html)

The way the system works is that the UI Engine maintains a stack of Screen objects. As new screens are created, they are pushed on the display stack. Only the screen at the top receives input events. When a screen is popped from the stack, the underlying screens are drawn as necessary. Note that the Blackberry display stack is not the same as the Android back-stack. The Android back-stack is a system-managed structure, scoped to a task with acitivities from different processes getting interleaved. The Blackberry display stack is a per-process application-managed stack of Screens.

To show a UI, the application extends the net.rim.device.api.ui.UiApplication class (which implements UiEngine and extends the Application class), creates an instance of MainScreen (which is of type FullScreen) and then pushes this instance on the display stack.

class MyApp extends UiApplication {
	MyApp() {
		MyAppScreen myAppScreen = new MyAppScreen();
		pushScreen(myAppScreen);
	}

	public static void main(String[] args) {
		MyApp app = new MyApp();
		app.enterEventDispatcher();
	}
}

class MyAppScreen extends MainScreen {
	MyAppScreen() {
		// add fields etc.
	}
}

Termination

An application terminates when System.exit() is called. This method causes the Blackberry JVM to terminate the process of the caller. For applications that show a UI, the typical way of quitting is that when the close() method is called on a screen, it pops the screen from the display stack, and if the display stack is empty, calls System.exit(). So if the user keeps pressing back and hits the first screen of your application, pressing back once again would by default exit the app. To avoid this, one needs to override the close() method.

Doing Long Running Work

As described in the Event-Processing section above, the Blackberry UI model is a single threaded one. That means that you cannot do long-running work on it, and instead need to launch a worker thread. This worker thread can update the UI in two ways:

1. Acquire and synchronize on the event lock: To do this, the worker thread invokes Application.getEventLock() and then synchronizes this object to access the UI. While this is being done the Event Thread is paused, so the lock should be held for a really short period of time.

   // on the background thread
   synchronized(Application.getEventLock()) {
       // update the UI
   }

2. Inject an event in the UI message queue: To inject an event, you model it as a class that implements Runnable and inject it into the Event Thread message queue by calling invokeAndWait() or invokeLater(). The event thread would then call the runnable’s run() method at the next available opportunity. While for invokeAndWait(), the worker thread would block till run() returns, invokeLater does not block and returns immediately.

   // on the background thread:
   UiApplication.getUiApplication().invokeLater (new  Runnable() {
       public void run()
       {
           // update the UI - runs on the Event thread
       }
   });

Interprocess Communication

There are two ways in which Blackberry applications can communicate with each other:

1. Using the RuntimeStore: The RuntimeStore is a system-wide volatile storage where applications can place data to share with other applications.
2. Using Global Events: Use Application.addGlobalEventListener() to add a GlobalEventListener to listen to global events. Post a Global Event by calling ApplicationManager.postGlobalEvent().

Security Model

Blackberry has a complex security model, owing to its roots in enterprise software. When businesses issue their employees Blackberry devices, they also expect to able to exercise control over what the user is able to do on the device. This is made possible thru a policy which is pushed to the device using BES or the Blackberry Desktop Manager. Such policies can limit the resources available to an application, and to handle that gracefully, applications should:

1. Find out what permissions they have: By calling ApplicationPermissionsManager.getApplicationPermissions(). This returns an instance of ApplicationPermissions class which gives all the permissions that are set.
2. Ask for permissions they need: By calling ApplicationPermissionsManager.invokePermissionsRequest() and passing it an instance of ApplicationPermissions with the requested permissions set.

Besides this, RIM has marked a number of APIs as controlled APIs. To access these APIs on the device, you must sign your application using a key or signature obtained from RIM. For details, see http://us.blackberry.com/developers/javaappdev/codekeys.jsp. On the simulators, these APIs are accessible without code-signing.

Packaging

When you build your app, you create a JAR, alongwith a manifest. Java ME defines certain mandatory and optional attributes that applications should include in the JAR manifest, besides any other app-specific info that maybe present.

However, a Blackberry device cannot directly execute a JAR. It first needs to be converted into a proprietary format file called a .COD file which is essentially RIM’s own version of the JAR. So if you want to run a MIDlet on a Blackberry device, you first need to convert the JAR into a COD file which can be done using the Eclipse plugin or by using the RAPC compiler on the command line. Some more details on using the RAPC compiler are given here.

However, the max size of a COD file is 128 kb with a 64 kb limit on compiled code and 64-kb limit on resource data. In case the compiled code produced is larger than these limits, the compiler breaks the code into mulitple COD files, naming them in increasing numerical order like this:

• HelloWorld.cod
• HelloWorld-1.cod
• HelloWorld-2.cod
• HelloWorld-3.cod
• …

The compiler then takes the numbered files (called sibling COD files), zips them and names the zip as a COD file (HelloWorld.cod above) – this is called the main COD file. There is a max limit to the main COD file too – it can contain up to 127 sibling COD files, thus imposing a theoretical size limit. For more details, see http://supportforums.blackberry.com/t5/Java-Development/The-maximum-size-of-a-BlackBerry-smartphone-application/ta-p/502534.

Packaging the application for Wireless or Over-the-Air (OTA) distribution requires more work. For this, Java ME defines an additional manifest called a JAD (Java Application Descriptor) that Java ME applications should provide for OTA distribution. The main purpose of a JAD is for locating a JAR and figuring out its size. Besides the JAR URL and JAR size, the JAD must also contain certain other attributes that are also required in a JAR manifest and may contain any attribute that is in a JAR manifest. For details, see here. Blackberry specific attributes for a JAD are given here. Also, for OTA distribution, you need to extract the sibling COD files from the main COD file. For details see here.

Distribution

Blackberry applications can be distributed in two ways:

1. Direct Connection: Connect the device to your computer and install the app. There are two possibilities here:
   • Using the Blackberry Desktop Manager: The application needs to have a COD file alongwith an additional manifest file called the ALX file.
   • Using a Web Page: For this, you need to put up an ActiveX control called a  Web Loader on the web-page from where you want the installation to happen. The user browses to a web-page on his desktop using IE and the control starts an installer. In case the user visits the page using a Blackberry device, the user is prompted to connect the device to a desktop.
2. OTA: There are two ways in which OTA distribution may happen:
   • BES Push: An organization with a Blackberry Enterprise server deployment can push notifications to Blackberry devices for applications based on policies defined by an administrator.
   • Blackberry App-World: The process is described here.

Android Architecture

The following diagram (from http://developer.android.com/guide/basics/what-is-android.html) gives the overall architecture of the Android OS:

The Android OS is basically a Linux 2.6.4 kernel fork. The Linux kernel is used for memory management, process management, security, driver model, shared lib model. Google has added Android specific enhancements: Power management, Kernel debugger, Logger, Low Memory killer, Shared Memory Driver, IPC Driver, etc.

Above the kernel, in the user space, sits the Hardware abstraction layer – this deals with stuff like Radio, Wifi, GPS, Graphics, Audio, Camera, Bluetooth, etc. Basically this defines a set of interfaces that drivers need to implement. These libaries are loaded by the system at runtime on a need basis.

Above the HAL, sit a bunch of libraries – the most important of course is libc – Google has written a port called Bionic libc which is optimized for embedded use – in terms of being small in size (since it is loaded in every process) and fast (since CPU is limited), but also because libc is GPL and Google wanted to protect apps from being GPLed. Besides there is webkit of course and SQLite. Google also provides a Media Framework, OpenGL library, and libraries for UI rendering and audio.

The Android programming environment is based on Java. For this, the OS ships with Dalvik – a custom implementation of the JVM optimized for the embedded environment. Dalivk does not run the Java bytecode, but its own custom bytecode called Dex – Dalvix Executable. It uses runtime memory very efficiently, the bytecode interpreter is highly CPU optimized, and because of this low footprint, the VM can be run in multiple processes – basically each app gets its own Linux process with its own instance of a Dalvik VM. The VM is exposed by a set of core Java APIs which provide the familiar Java library.

All of the above is wrapped in a set of platform services called the Android Application Framework – these services work behind the scenes to provide abstractions like Activities, Packages, Windows, Resources, Content Providers, View System, hardware access, etc.

Startup

Like a typical Linux system, in Android, the bootstrapper loads the Linux kernel and starts the init process, which in turn starts various daemons like Android Debug Bridge (adbd), Radio Interface Layer Daemon (rild), etc. After this, init starts what is called the Zygote process. The purpose of this process is to kick off the rest of the Android system and later help in the instantiation of apps. The Zygote process initializes a Dalvik VM instance and loads a bunch of libraries and starts listening on a socket for requests to spawn more processes (with VM instances). As requests come in, it forks to create new processes with VM instances. Copy-on-write is used to maximize reuse and minimize footprint.

(Android Startup Sequence – source: http://sites.google.com/site/io/anatomy–physiology-of-an-android)

Once Zygote is in place, the init process starts what is called the Runtime process. This basically does two things:

1. Start the Service Manager – this is responsible for managing IPC and all services are required to register with it. It acts like a local DNS providing a way to bind to a service given its name.
2. Ask Zygote to fork what is called the System Server. This is the first process (besides Zygote) that has a running VM instance. The System server starts services for display (Surface Flinger) and audio (Audio Flinger). These services register with the service manager like all services are expected to. Now other apps can start using display and audio. After this, the System server starts up all the core platform services – Window manager, Telephony manager, Power manager, Activity manager, etc. Each one of these services also registers with the Service Manager, so that apps can use these services.

At this stage, we have the following processes in place:

1. Init – the original init process started by the bootstrapper
2. Daemons – started by init
3. Runtime – also started by init
4. Zygote – the original zygote which will continually get forked as new apps get launched
5. System Server – the first managed process which contains all the core services and platform components.

After this, the Home Screen or the Idle screen is launched – basically the Activity Manager (which is inside the System Server) sends a message to Zygote to start the “Home” Activity, which causes Zygote to fork into a new process with a Dalvik VM and the Home activity. Now depending on the action carried out by the user, the appropriate app would be launched with the Zygote forlking each time to create a new VM instance inside a new process. So for example, if the user starts the Contacts app, the apt process would be setup:

(Startup processes in Android -  source: http://sites.google.com/site/io/anatomy–physiology-of-an-android)

Each new app gets a unique user Id which is opaque to the app, and the system sets permissions for all files in the app so that only the user id assigned to the app can access those files.This makes the system secure since the app only has access to the components that it needs to do its work and nothing else. However,  apps may need to communicate with each other and the system services, which are all running in separate processes. This is accomplished thru Inter Process Communication

IPC

With its emphasis on isolating apps and services in process boundaries, it is clear that Android requries a lightweight IPC. The IPC mechanism in Android is called Binder and is based on shared memory. Recall that when a process starts it registers itself with the Service Manager. This happens behind the scenes, and on registeration, each process gets what is called a Context object – a reference to the Service manager. Now lets say app A needs to communicate with service B, and the two are running in two separate processes. To do this, app A asks the Context for the Service B by passing the well known name of the service. The Context returns a reference to the service to A, on which A can call a method – say foo. This method call is intercepted by the Binder driver. The driver marshals the object and passes a reference of it to the receiver – B. Note that this is passing by reference, not passing by value, in which the object is serialized. On the side of the service B, the Binder maintains a thread-pool (transparent to the service). One of the threads in this pool receives the incoming call, locates the actual object in the service B and make the call. The return value is then similarly passed back to the caller. The following diagram (from http://sites.google.com/site/io/anatomy–physiology-of-an-android) illustrates this:

The entire operation is synchronous, and the objects are reference counted so that when they are no longer in use, they can be deleted. Since there is no serialization / de-serialization, there is no overhead and therefore this model can also be used to communicate with services that run in your own process without any penalty.

Applications

An application in Android is a collection of components. There are four types of components:

• Activity: An Activity is a UI component corresponding to one screen with which a user interacts in order to do something.
• Service: A Service is an application component without a UI used to perform long running operations in the background.
• Broadcast Receiver: A broadcast receiver is a component that responds to system-wide broadcasts (for example screen turned off, battery low, etc.)
• Content Provider: A content provider is a component that stores and retrieves data and makes it available to all applications. There are different types of content providers: audio, video, contacts, etc. and you can create your own custom provider also.

As mentioned earlier, each application runs in its own process. By default, all components used by that app also run in the same process, and on the main thread. However, it is possible to make a component of your app run in a separate process thru the manifest file. Thus, an application in Android may span multiple processes.

Application Startup

Android follows a fairly unique model in that there is no single entry point for an application – there is no main() function. Instead, a component in one application can start another application’s component, thus bringing the application to life. This communication across apps happens thru the IPC mechanism descibed above. So while an Activity is owned by an application, it is possible for another application to start it (if the owning application allows it). An example is clicking on a hyperlink in an app opening up the browser. This applies for not just Activities, but other types of components also. In order for this to happen, there are two steps required:

1. In case the application is not already running, the Android system would bring the application to life in a new process forked from Zygote.
2. The desired component inside the app would need to be activated.

Note that in case the application is already running, the new component would be by default instantiated in the same process.

As mentioned above, IPC happens thru a Context object. So when a component A inside one app needs to activate another component B in a different app, or give it something new to do, it basically uses the Context object to send a message to the other component. In the case of an Activity, Service or a BroadcastReceiver, this takes the form of a what is called an Intent – a passive data structure that defines the operation to be performed for an Activity and a Service, and for a Broadcast Receiver, a defintion of the announcement being broadcast.

Content Providers however are not activated thru Intents. Instead, activation happens on a request from a ContentResolver, which acts as a mediator between the requesting component and the Content Provider.

The Activity Back-Stack

Consider the following scenario:

1. You are on the Home Screen. This is Activity 1
2. You click on the Mail app icon, and that activates the main activity in the Mail app which comes to the foreground. This is Activity 2
3. You now click on Compose and that activates the Compose activity in the Mail app. This is Activity 3
4. You decide to cancel out of composing a new message and press the back button. You come back to Activity 2

Here is what happens in the background:

• When one activity starts another activity, it stops and its state is saved. So when Activity 1 starts activity 2, activity 1 is stopped, its state saved and so on
• The system maintains a stack (called a back-stack) with the latest activity on top and the oldest activity at the bottom.
• When the user presses the back-key, Activity 3 is popped and Activity 2 is started from its saved state

(The Activity Back Stack – source: http://developer.android.com/guide/topics/fundamentals/tasks-and-back-stack.html)

This approach allows Android to seamlessly transition from one app to the other in a consistent way.

Tasks

In the scenario described above, assume that when the user was in Compose mail (Activity 3), he decided to call someone and pressed the Home key. This would not unwind the back-stack, but start a new stack. In order to do this, the collection of Activities in the first stack needs to go into the background. This is achieved thru the notion of a Task – a cohesive unit of Acitivities. When a task moves to the background, all activities in the Task are stopped but the back-stack for the task remains intact, so that when the user returns back to the Task, she can resume where she left off. However, to save memory, the back-stack for a background Task is not retained for a very long period and if the user does not go back to the task, the back-stack is cleared except for the root activity.

(Background and Foreground Tasks – source: http://developer.android.com/guide/topics/fundamentals/tasks-and-back-stack.html)

Activity Life Cycle

The lifecycle of an activity is affected by its association with other activities, its task and its back-stack. There are three states in which an activity can exist:

• Resumed / Running: Activity is in the foreground and has user focus. Such an activity is never killed by the system,
• Paused: Activity is visible, but another activity is in the foreground and has user focus. This could happen if the other activity is at the top, but is translucent, or because it does not cover the entire screen. In the paused state, the Activity object is retained in memory, maintains all state and member info and remains attached to the Window manager. However it can be killed in extremely low memory conditions.
• Stopped: The Activity is not visible to the user.While the Activity object is retained in memory, maintains all state and member info, it does not remain attached to the Window manager. The activity can be killed by the system to reclaim memory if required.

(Activity Lifecycle – source: http://developer.android.com/guide/topics/fundamentals/activities.html#Lifecycle)

Saving Activity State

Note that it is entirely possible that the system, in order to reclaim memory, may destroy an Activity, or even the process in which the Activity was running. However, when the user comes back to the activity (thru the back-stack), you still want to resume at the point the user left. To do this, an activity must save its state. This happens thru the Activity.onSaveInstanceState() method.

(Saving and restoring Activity State – source: http://developer.android.com/guide/topics/fundamentals/activities.html#SavingActivityState)

Process Lifecycle

The Android system may need to kill a process in order to reclaim memory. To ensure that this creates minimal impact on the user expereince, Android ranks processes in a priority order:

• Foreground Process: A process that is required for what the user is currently doing. Such a process is killed only as a last resort
• Visible Process: This process is not in the foreground but can affect what the user is seeing on the screen. For example it may host a paused Activity. Such a process is not killed unless doing so is required to keep all foreground processes running
• Service Procress: A process that is running a service and is not one of the two types above. For example the service may be playing music or downloading something. The system would keep such a process running unless there is not enough memory to keep Foreground and Visible processes running.
• Background Process: A process that is holding an activity not currently visible to the user (the activity is stopped). Such a process has no impact on the user experience (if the activity lifecycle is correctly implemented and the activity state is being properly saved and restored) The system can kill such a process any time. Typically there are mulitple background processes running and the system maintains a LRU list which is used to kill such processes.
• Empty Process: An empty process does not hold any active component. The only reason such a process is kept alive in the first place is for caching and to improve startup time. The system often kills such processes to maintain the balance between process caches and underlying kernel caches.

Of course it can so happen that a higher priority process is dependent on a lower priority process. In such a case the ranking of the serving process is increased to the same level as that of the dependent process.

A situation where this ranking has a direct impact is this:

• Your app needs to download something big which may take time and the user is likely to move out of the app’s activity to something else.
• If you spawn a worker thread for this task and the user moves out, the process would become a background process.
• However, if you instead spawn a service, the process would be a service process and be less likely to be killed

Main / UI Thread

When an application is launched and a new process created to host it, the application gets a single thread called the main thread or the UI thread since this thread is responsible for servicing the UI. The way this works is mostly the same for almost any UI implementation – be it desktop operating systems or mobile OS. Basically, the UI thread runs an infinite loop which checks a queue to see if there are any pending UI events. In the case of Android, this concept is formalized in the form of a Looper.  The looper loops over a MessageQueue which contains the messages to be dispatched. The actual task of managing the queue is done by a Handler which is responsible for handling (adding, removing, dispatching) messages in the message queue.

(Android Threading – source: http://sites.google.com/site/io/inside-the-android-application-framework)

For the main thread, a Looper is setup by the system. However, you can also associate a Looper with your own thread. Note that the Looper can be associated only with one thread and that association cannot be changed. The way Android ensures this is by putting the Looper on the thread local storage of the thread and not exposing a constructor for the Looper. You get a Looper by calling the static prepare method which first checks the TLS to see if there is already a Looper, and if not creates one. You then call the loop method on it to start pumping messages. Unlike a Looper, mulitple handlers can be associated with a message queue.

So quite obviously, one should not run a blocking operation on the UI thread. If you do, messages would stop getting processed from the Message Queue and your app would become unresponsive. This is what leads to the famous Application Not Responding (ANR) dialog. One solution is decsribed here: create a handler, spawn a worker thread, post results back from the worker thread to the handler, update the UI. But this is a bit cumbersome to do. A simpler solution is to use a AysncTask which takes care of the underlying plumbing.

Service

As mentioned earlier, a Service is a type of component that is used to carry out a background operation, and does not provide a UI. An app component can start a service, and even if the user switches out of the application, the service would keep running. Typical use cases include playing music and downloading a file. Android ensures that a process running a Service is not killed as far as possible (see Process lifecycle above).

Note that a service by default runs on the main UI thread in the same process. This means that if it carries out a blocking operation, your UI would become unresponsive since the thread would not be available to run the Activity. To circumvent this, you should launch a worker thread inside the service.