Sony has a real shot at leading this new video game consoles generation, if some basic assumptions hold and recently leaked(made public voluntarily?) information is correct.

The Playstation 3

I was extremely excited about the Cell processor when information about it came out, long before the PS3 shipped. I would look for new information/resources related to programming that beast, low-level architecture insights, anything really. I (still) think it's a really elegant processor (this won't be a technical post so I won't go there), but it was a departure from the norm for game developers; partitioning work into tasks, message queues, local-isolated memory with DMA based transfers between PUs(ok, it's a bit technical), and that's just some of the new paradigms developers had to become familiar with if they wanted to ship a game that was taking advantage of the system. To make things worse, the GPU of the PS3 is very weak, especially compared to the 360's, and if that wasn't enough for them poor developers and publishers, the PS3 was super expensive and the network experience was sub-par compared to what was available on the Xbox with their Live service. All in all, Sony didn't make it easy for developers, gamers or retailers. This of course made it possible for Microsoft to leap-frog past Sony's collective market share(PS2+PS3) - even though they had their own problems (red rings) - soon thereafter.

If the rumors are to be believed, the PS4 is more powerful than the Xbox720(or whatever they will call it), very easy to program (conventional architecture, no exotic systems on there) and, apparently, provides a more efficient pathway to bare-metal operations/programming which should provide an even better advantage to developers, in the long turn, for (presumably) the new Xbox games and services performance will be subject to more/thicker layers of indirection(drivers, APIs, etc).

The PSN outage

The PSN intrusion which led to some 80 million or so accounts being stolen in April 2011 was a huge blow to Sony. It took them a long, long time to figure things out(well over a month), cost them hundreds of millions of USD and ruined their relationships with their customers(not just gamers), partners and the media. Sony lost big, but I think it's gaining even more in the long run as a result of that incident and its long term consequences.

Sony rethought everything. They rebuilt parts of their infrastructure, hired many talented people to work on improving existing services, and what's even more important, all that probably led to a revision to their current PS4 plans, at least in terms of services, infrastructure, security and overall experience, that wouldn't have happened, I think, if Sony didn't have to face that huge issue, fix it and understand the cause and effects chain on there.

The New Sony

Under the leadership of Kaz Hirai, Sony is transforming itself and now stands a chance of regaining it's former glory; it's dropping unprofitable products and units, streamlining it's services and products, improving the experience. They really have no choice. Apple, Microsoft and the South Korean electronics giants (Samsung, LG) are beating them hard and they must hold their ground least they become irrelevant. I think they are doing a remarkable job at it, certainly compared to what old Sony was doing and the PS4 is crucial to their bounce-back efforts.

The Living Room

Everyone will tell you 2013 is the year of the living room. New video game consoles from Sony and Microsoft, new 'microconsoles' (Ooya, Gamestick, the Steambox) and more importantly, the Apple TV(set), and more systems we haven't heard of yet, are coming out soon. It's all about the TV and all things entertainment; video games, movies, music, everything. Sony has a good chance to be a key player in this new battlefront. They cannot afford not to be.

Apple TV (maybe the current-gen appliance, but most likely that'd be the full-blown TV set, thing) will enable the implementation of a service I have long wanted to use.

I want to be able to watch a movie or TV series episode with my brother, or a group of friends the way I would if I was in the same room with them. That is, having the chance to comment about the video or other topics, pause playback, just like it'd work if a group of friends were watching an episode of the Big Bang Theory in the same room.

In addition to synchronised viewing (if I or another virtual group participant was to pause playback, it should pause for everyone else, too), context specific information could be overlaid on top of the frames (e.g IMDB information, whatever), and more.

Apple will probably make this possible soon; it should become possible with iPads, iPhones and Apple TVs, and it will be great. I can't wait to watch movies with my brother and other friends, voice chat about the events taking place and have a genuinely good time, together.

I suppose one could put come up with a crude approximation of that experience even now, but that's about taking too many leaps of faith and even so, a single glitch would ruin it anyway.

If Apple doesn't provide this experience itself out of the box, I am sure other fellow developers will jump to it. If not, I will.

If I wasn't who I am, which is to say, if I wasn't into computers, software development, and all things 'technology' (this term is overused and abused by the way ), I think I would really enjoying being either a fisherman, a tavern owner, a shepherd, a lawyer, a scientist (physics, math, astronomy, ..) or any combination thereof.

Diablo 3 is such a monumental achievement in video games. Going through it felt every bit as refreshing as it was for me when I played Diablo 1 on my brother's recently bought PC some 15 years ago. Perfectly balanced, though I could do without the endurance and patience challenge that is the boss fights, long, magical and perhaps resting above all such qualities, the wonderful drop in and out coop support and its implementation. I can't think of any other game that supports it, or one that does it so well anyway. Always-on connectivity is not an issue for me insofar it doesn't prevent me from actually playing the game. Blizzard should have been better prepared, whatever that means, but as it is with most things, issues will be ironed out soon enough and none will remember them soon, just like it was with Valve's Steam (it was even more frustrating at the time, too).

I am deeply troubled by the ongoing developments in this country. My friend Steve warned me about the severity of this recession long before it was even acknowledged by most. I was hoping he would be wrong, that perhaps things would just get better soon. Alas, he was right. Its going to get even worse, especially if other countries in the EU wind up collapsing too. This brought to light the kind of corruption and decay that has been eroding our society since forever. It all comes down to people, of course, and, even more so, the people tasked to create laws, uphold them and run the country. Political parties and unions may be a corner stone of democracy but they are really just a means for individuals to gain power, control through them and define the fate of others driven by their own desires. Meritocracy is a solution, as far as I am concerned, but that is not happening anytime soon. If political parties and unions were to dissolve and people would vote and support individuals who would be judged by their individual qualities not by the party or union they stand for, I am confident things would have been better. Much better. "Much that once was, is now lost"

Dear Steve,

I cannot thank you enough for making the world a better place for me, my family and friends. Thank you for the hardware, the software, the services and the movies your values and charisma made possible to deliver in such high, unmatched, quality, and for being successful at safeguarding those values from naysayers and conventional 'wisdom'.

Rest in peace Steve Jobs

Amazon's Jeff Bezos unveiled the new breed of Kindle e-readers and their new flagship consumption device, the Kindle Fire tablet. Prior to the announcement most folks posited that Amazon was the one company that could challenge Apple's dominance in the space.

Amazon is the king of the content game. They are also the largest e-tailer and their brand is stronger than most. They have a bazillion loyal customers. It doesn't take much to come to that conclusion.

And so, everyone was seemingly correct to assume this would be the case. The Fire, specifically, has been met with overwhelming praise and even lust by the majority of press and people at large.

Amazon didn't jump into the me-too game, like others always do, usually attempting to emulate Apple's strategy(failing miserably has been a common conclusion). Amazon plays on its strengths. They are practically unchallenged in the content space. They have the largest library of digital books(by far), millions of movies available for instant access, music from their 'MP3 store'(I really don't like that name), they even have Android apps on their very own app store(Google execs are not happy). They also have 'infinite' compute and storage capacity (AWS) that they put to a great use with the Fire. What do others, other than Apple, have to offer? They _still_ believe (or perhaps, hope) they can provide viable alternatives to the iPad by winning in the hardware specs battle. It was naive years ago, its even more naive nowadays. Its all about the experience, the content, and the apps.

We should all be pleased that Amazon delivered. Competition drives evolution. Apple needs it. The Fire will be the ideal tablet for many. To me, the Kindle (either model) is the perfect gift to give out to friends and family.

It will be interesting to see how Google and Microsoft will react to that. Microsoft may have lost the tablet war before it even had a chance to compete - their OS and the hardware that runs it will not be out for a long while. By then, the iPad and Amazon's Fire will be so far ahead, Microsoft will have no chance. Slick as Metro may be, that along won't be enough to make the difference. And then there is Google, their Motorola business, the many different OEMs that build Android phones and tablets where you can't tell one from the other. Maybe Google did do the right thing with Motorola. Maybe they didn't just buy the company just for their patent portfolio. Maybe they figured out that they need to control not just the OS and core applications stack, but so much more, to compete with Apple(and now, Amazon).

Exciting times.

The enemy of free is indeed 'easy'. This is more of the case when you grow up and, when time is more important than money and getting frustrated about getting something to work or trying to spin in ways it is not meant to spun is just plain not worth it.

Amazon unveiled its Mac applications store. I was expecting them to further expand their digital downloads services with a Windows centric marketplace(higher demand, no competition) but maybe this will be their next step. Amazon is apparently run by very smart people - some of those probably looked at the numbers on their Kindle business (they recently acknowledged Kindle ebooks sales surpassed the physical books) and, of course, the success of the Apple operated app stores as well as game download services (Steam mostly, for it owns 70% of that market space).

You didn't have to be able to see far into the future to accept the rise (and eventual dominance) of the new digital distribution service. Besides, there was a precedent there already; music sales shifted from physical to digital(iTunes Store, music subscription services, ..). Multimedia content(books, magazines) and software (applications, games) will undoubtedly continue be available in physical form for sometime to come, but eventually they will become either hard to come by or expensive to own and maintain(the vinyl analogy).

I have stopped buying physical copies of books, games and software(disks) for some time now, save for video games for my game consoles - although the vendors there are aggressively expanding their online offerings, making it possible to download and buy full games. It is widely expected their forthcoming iterations will be optimized for offering access and delivery to full games directly from the 'Cloud'. Even if I don't get to play a game I just bought (which is subject to automatic updates, is available for re-download whenever I need it), I may decide to do so in a year or two - it will be there waiting for me, no need to track it down or worry about misplacing the disk. Same is true for books and magazines( plus, I can pick up a book right where I stopped reading on another device than the one I was reading it before).

Sometimes resistance is futile and pointless.

CloudFFS

So we built and deployed this new service not long ago at work, and @stelabouras suggested we document some parts of it for internal consumption. Given that I haven't blogged for months, I thought I 'd just pour those words here instead.

CloudFFS (yes, it is a funny name) is a file-system (but not in the traditional sense, it doesn't hook into the kernel VFS layer or anything ) that provides storage for unbound number of files and very fast access to them over HTTP. It can manage PB scale volumes and up to 2^64 files per namespace(see below).

We have hundreds of millions of static files(images, video, text files, you name it) stored across our storage devices; having to deal with those many files is not a pleasant task, for our sys.operators and developers alike. We wanted a solution that frees our developers from having to worry about storage and provides a very simple way to store and retrieve files, and at the same time help our systems guys deal with backups and management of those files efficiently.

There are many problems associated with the use of multiple files. Wasted inodes/disk blocks, slower access time (iterating a path components is not free, looking up a directory entity within a directory is not free either), difficulty in making backups, need and use of elaborate directory naming schemes in order to deal with large directories, and more. In addition that, accessing those files over a network filesystem (e.g NFS) is not efficient by any means. Developers need to be aware of those limitations and of the rules that are in place in order to deal with said limitations, which places an unnecessary burden on them.

None of the solutions we looked into really seemed all that great for us, so we went ahead and build our own. Though, to be fair, we almost always end up building our own anyway. This practice has worked great for us all those years and given that we are a technology company, it makes sense for us to disregard the 'not invented here' approach.

Data Model

Files are uniquely identified by a 64bit number. They belong in namespaces, for example 'blogs', or 'images, or 'mails'. A file can hold up to 1GB of data. Files can also be either public, or private. Public files can be accessed directly (e.g http://x.pstatic.gr/me/305896160755729.jpg), whereas private files require HTTP authentication. This makes it possible to, say, make everything accessible over the public Web, except files that should not be accessible in that fashion (e.g log files, archived content, emails, etc ).

On disk, each namespace is represented as a directory in the 'root' directory. Within each namespace directory, there are 2 types of files. Data files and indices. They are further subdivided into 'live' and 'immutable' files. Immutable files, once they are created, are never updated again. In practice, there is almost always a single live datafile/live index associated with a namespace (there are cases where there can be 2 for a few seconds, whenever a live datafile is converted to an immutable one) and that live datafile holds incoming updates. Whenever a SET or DEL operation is executed, a new record is appended into the live data file and its respective live index. Once the live data file size exceeds a threshold, it turns into an immutable one and a new live datafile is created and used instead. Whenever it is necessary, a compaction task kicks in that will merge immutable files, discard deleted files and duplicates and create a new set of immutable files out of them and delete the old ones. This happens fairly infrequently though ( depends on the volume of data, but usually once every few month ). A live or immutable data file can hold thousands, millions, or even billions files, packed one after the other. Maybe I will get to write more about the structure of those files and how they are used in a future blog post.

Operations

There are 4 operations that are all mapped to HTTP methods/verbs. Get(GET), Set(PUT), Stat(HEAD) and Delete(DEL). A few things that may be worth noting; Whenever you set a file you can specify if it is a public, or a private file. Whenever you Set or Delete a file, or Get or Stat a file that has been stored as private, you need to specify authentication credentials(username/password). Otherwise, the operation fails (authentication failed). The list of users is defined in the configuration file.

Internals

The service is implemented as a multi-threaded application. A single thread handles network I/O (asynchronous I/O multiplexing w/ vector I/O). There are also some threads for processing requests and 2 threads for processing system tasks (compactions and live to immutable data conversions). The network I/O thread also accepts incoming connections and parses in HTTP requests. Whenever such request is parsed, it is placed in the 'mailbox' queue of the first idle thread for processing. The network I/O thread also accepts RPC connections/requests (for management) and also talks to our message queue service ; whenever a file is stored or deleted a new event is published into the queue service so that we can replicate data or whatever else we choose to do whenever those events are created. There are 2 types of caches in place. One for immutable datafile keys ranges(a keys range holds a series of file ids and their (offset, size) into their respective immutable datafile), and another one for compressed content (whenever the agent/browser supports gzip/deflate encoding and the file requests can be provided back to the client in compressed form, we compress and cache the contents into that cache for later use). They both operate in an LRU fashion are protected by spinlocks; the cached objects are ref.counted.

Whenever a Get request is executed, we iterate through the list of the live datafiles for the namespace(usually 1). If not found in there, we walk the list of the immutable datafiles(they are always sorted by creation time) until we get a match, or not. A Set operation appends the data into the active live data file for the namespace, syncs on disk and then lazily appends on the index.

Files on disk (both on live and immutable files) are stored as {header} data {footer}. The header holds the key/id, last modified timestamp, size and flags(private, public, etc). The footer currently holds a crc32 checksum which we consult whenever we pull data from disk for integrity checks. If for whatever reason any file (live or immutable, datafile or index file) is corrupt in any way, the system tries to rebuild it, if possible, or salvage as much as possible from it. XFS is our filesystem of choice for the locally attached storage that holds the CloudFFS datafiles. Each of those datafiles can hold millions of files ( each namespace has its own capacity / datafile threshold ); typically each immutable data file is around 1GB in size, but can be TB in size - there is no hard limit there. All those datafiles make up the namespace.

Accessing data

A file/object is accessible at http://domain/namespace/id. e.g http://x.pstatic.gr/me/305896160755729.jpg. The ID is a 64bit integer that identifies the file to be retrieved. An alternative way to access a file is by accessing http://domain/namespace/n/string. In this case 'string' is used to generate an 64bit identifier. e.g http://x.pstatic.gr/avatars/n/M/96/5/22/markp.jpg. In addition, alphanumerical characters can succeed the 64bit identifier - those are mostly ignored, though a filename extension, if present in that string of characters, is used for identifying any rules specified in the configuration file for special treatment of with said extension. For example, you can specify that 'css' files content type will be text/css, and that they expire within 1day since they were created and that they can compressed if the client supports compressed content. Those kind of options can be set on a per namespace basis and on a per namespace extension basis. The HTTP service will look for Last-Modified and ETag headers and will respond with HTTP 304 Not Modified if needed.

We are migrating existing static files into CloudFFS but so far it has worked great for us; very fast access to data (0.005 seconds for a typical file), a few dozen files to manage instead of millions, easy and efficient access to the files for our developers. Our forthcoming CloudFS project will provide more features (TB scale files, random read/write access to files, distributed storage and fault tolerance, etc) but this service is far more suitable for the kind of static files we have and keep creating every second. Maybe someday we will get to talk more about the services that we built and run in house.

Stelios tagged me so here is my iPhone 4 home screen. I am supposed to pass the torch to someone else, so my brother will take it from here.

I spent a few (no more than 2) hours on developing for Android, and the better half of a day building for Windows Phone 7 / Silverlight, mostly because I wanted to learn enough to understand the development model of both platforms, and compare said models to iOS's.

I went through Android source code when it was released ( spending most time on the 'Skia' 2d drawing component ) and figured out through the basic concepts. Its been a long time since then.

I was drawn to Windows Phone 7 because I really like the UI. Its clean, simple and elegant. Its also fresh and unlike iPhone's UI(everyone is copying Apple, left and right, as it has always been the case). When Silverlight was announced, I looked into it but was put off by the use of XAML files and some weird naming decisions in the classes tree. Other than building a trivial 'lets see what this is all about' application, I didn't spend more time on it.

As expected, Android and WP7/Silverlight also adopt the familiar views/controls paradigm. On Android, you got tasks (processes) and each holds a stack of Activities; an activity is more or less a page, that holds a content view. That content view is usually a container view that contains other views. Activities do not need to come from the task that owns the activities stack. They are popped out in a LIFO fashion, and its alls simple and nice. You also get services ( really, tasks with no front-facing UI, which are cool ), intents ( effectively, messages with action and payloads ) and other niceties. It all more or less make sense - the one thing I don't like about Android is the UI of the controls. Everything is ugly. The emulator is also slow and, well, ugly, which makes things even worse than they probably are. Google is no Apple, sure, but they should have done something about it. It all reminds me of those Java Swing components (or even worse, AWT components used in Java Applets when applets were cool - which is a long, long time ago ). You get to use Java to build the applications. I am not fond of Java, but I don't really mind it ( a bit too high level for my taste, among other things). You also get to use Eclipse ( which makes it really easy to build stuff, with intellisense, on-the-fly compilation and all those nice things people expect nowadays from IDEs ), or use any other IDE or just use the tools on the terminal , if you don't like IDEs or for whatever other reason. That's what I did. The tools are easy to use and it takes very little amount of time to feel comfortable enough with the environment.

I spent an hour or so trying to find my way around Windows Phone 7 Silverlight concepts and paradigms. Those XAML files, that I hated on SL back in the day, were still here and I just didn't want to deal with them. It turns out, that unlike what the documentation may make you believe, you don't need to use them. You can delete them and do everything programmatically ( though its not straight forward, but it make sense once you do it once or twice ). You need to decide if you want to build a Silverlight application, or an XNA application. If you want to build 'high performance' games, you need to build an XNA app. Otherwise you will want to build a Silverlight app. They are concepts and classes that are unique to each approach and it just doesn't feel right, having to restrict yourself to either of those as opposed to building an application that can access all facilities offered by the device. You are going to use C# to build Windows Phone 7 applications. C# is nice. Java on steroids. Visual Studio 2010 Express for Windows Phone is a free download from Microsoft, providing everything you need to build your applications. So, you got your Application instance ( every WP7 application needs a class that derive from System.Windows.Application ). Every application instance has a RootVisual property. Its the main application UI ( a System.Windows.UIElement derived class instance ). The convention/requirement on Windows Phone is to have a special class instance set to RootView(PhoneApplicationFrame) and that should hold a PhoneApplicationPage derived class instance ). That page in turn holds a Content - which is the content view, etc. Similar concept to Android's activities and their content view, and iOS's Windows, Navigation controllers and their views. Again, simple stuff - just make sure you stay away from XAML documents. By the way, there is no support for multitasking on Windows Phone 7, unlike Android and iOS. There is no supports for sockets, either, which is weird and rather sad ( Access over HTTP and 'web services' is not good enough ). Hopefully, this will change soon.

I am going to build a 'real' application for Windows Phone 7 in whatever spare time I have this week and submit it on their Market Place. It should be fun; if nothing else I will should learn enough to help our Mobile Unit folks at work with upcoming Windows Phone 7 projects.

Our large-scale, high-performance and highly available ( those were the goals anyway, I hope we attained them ) data store has been more or less ready for production for a few weeks now.

We have yet to actually deploy it, although two forthcoming (in-development) projects will be built on top of it. There are a few things here and there we could, and will, change, cleanup the client library API and all that, but as far as I can tell, there are no real issues left to resolve. During testing, we got unto 40K GET(value by key) and over 50K PUT(value by key) operations/second on a 3 nodes system (quorum arrangement). Adding nodes increased capacity and throughput which was one of the design goals.

We got a few more similar projects in the pipeline; more building blocks for our services stack. We are going to build two different file system (one will be optimized for very high performance access to files, another for availability and storage of files not limited in size), a MapReduce framework/infrastructure and a new distributed lock manager which will also replace ad-hoc solutions we currently rely on.

I am very proud of our team; they are smart and inventive, passionate and hard working. They let me toy around with ideas that do not always make sense and they always find ways to make me feel great about what we do. Good times ahead.

Matt chat 56: Ocarina of Time

I love Matt Barton's work - his books and his video series on Youtube on great games and game makers - and this latest one is about what perhaps is the greatest game ever made, one that I have yet to play.

I got the Nintendo Wii mostly - if not entirely - for Zelda:Twilight Princess. A great game I had to give up playing for I just couldn't stand the silly wiimote. I hoped I would purchase a gamecube controller to play it the way it is meant to be played, but never got around to do so. Related links: The Legend of Zelda Retrospective on GameTrailers.com. If you like Zelda, or video games in general, you need to should this 6 parts retrospective.

iPad Launch

The iPod, the iMac, the iPhone, the iPad. They were all dismissed as just 'too different' to matter. Apple proved them wrong on all accounts in the past. If this turns out to be true with the iPad ( it will be ) it is going to change everything. It has began.

I have been reading about cryptography, digital certificates and signatures, symmetric and asymmetric keys based encryption lately(well, yesterday). Developing iPhone applications (and even more so deploying them to devices or the AppStore) involves dealing with certificate signing requests(CSRs), digital certificates, provision profiles, and more.

Apple went into great lengths to simplify the processes involved - for the most part its trivial and they guide you through every step. (alas, it wasn't always so; back in the day, code signing and other related processes were the cause of much pain among iPhone Developers ).

Nevertheless, whenever I come across something I haven't been familiar enough with, I can't help not 'wasting' time to learn it(how it works, why it works, etc). This practice has deprived me of many hours I could have spent elsewhere (working on other stuff, spending time with loved ones, whatever) but I just can't help it; I "need" to learn how things work.

So here is simplified, basics mostly, overview of those concepts, in case someone needs to get started with them.

The fundamental problems encryption solve are those of data integrity validation and identity authentication. That is to say, to verify that data(messages, anything) sent from Alice to Bob (Alice and Bob) can be verified by Bob that indeed came from Alice and that the data were not tempered with/modified in any way while they were in 'transit'. That's about it.

Encryption is the process of generating new data from the original data. The new data is usually unintelligible to anyone but the intended recipient. Decryption is the process of transforming that data back to the original data.

Those processes are facilitated by a cryptographic algorithm(a cipher). Mostly, this involves the use of a key(a number) which is used with the algorithm to perform the encryption and decryption. The same key is used for both encryption and decryption. With symmetric-key encryption, the encryption key has to be kept secret by both parties. If someone else gains access to the key, he can not only decrypt the data but also encrypt new data and send them to Bob and Bob would assume they came from Alice. That clearly is not desirable. Enter Public-Key Encryption.

Public Key Encryption(asymmetric encryption) involves a pair of keys. The public and the private key. The public key is published and is freely available. The private key is kept secret. Alice never reveals the private key. Ever.

The fundamental idea is that data encrypted using the public key can only be decrypted using the private key. Bob, who has access to the public key much like everyone else, can encrypt his message using Alice's public key, send it over to Alice and only Alice can decrypt it, for it is only one who has the private key.

PKE is computational expensive though and not always suitable for large amounts of data. Often enough, a hybrid approach is employed. PKE is used to send a symmetric key, which then can be used(since both parties will know that secret key) to encrypt additional data using symmetric encryption. Using a symmetric key to encrypt and decrypt data is far less computational expensive. SSL and other protocols rely on this hybrid approach.

It is also possible to encrypt a piece of data using a private key which can only be decrypted using the public key. Given that Alice shared her public key with anyone interested, that wouldn't make much sense if Alice was to send data to Bob encrypted with her private key. Anyone could read it

Well, it does make sense, thanks to digital signatures. A digital signature can be used to verify that data sent from Alice - encrypted or not - were not modified in any way by the time Bob received them. In other words, it validates the authenticity of the data. It deals with tampering and impersonation.

Alice will use a hashing algorithm to generate a signature out of the message she wishes to send to Bob. That signature will then be encrypted with her secret private key. Then, she will send both the data she wishes to send to bob and the encrypted signature she generated from that data. It will also send Bob information about the hashing algorithm Alice used to generate the signature of her message.

Bob will use Alice's public key to decrypt the signature. Then, he will use the same hashing algorithm to generate a signature from the message he received. If the signature matches the signature Alice provided him with, it means the message is authentic. That is so because only Alice knows the private key and only Alice could have encrypted the signature like that. Unless Alice lost her private key, it is 'impossible' for Alice to deny that she signed the data she sent to Bob, or for anyone to 'sign' anything, send it over to Bob and claim she is Alice.

A hashing function converts data into a single value (often a big integer). Hash functions are fundamental in the design and implementation of some of the most important data structures.

There is one last issue that needs to be addressed. Confirming identities. Digital Certificates solve this problem.

A certificate is an electronic document that is used to identify an entity(individual, company, anything) and associate that entity with a unique public key. Your passport identifies you and associates bits of information with you (your name, etc). Public Key cryptography uses certificates to address impersonation problems.

Much like one would go about obtaining a driving license, by providing the authorities with whatever information and credentials required, so that they can verify the identity of the applicant and then issue her the driving license, Certificate Authorities(CAs) serve a similar purpose. They will get Alice's application for a certificate (which includes her public key and information about her), they will validate the information she provided is correct and indeed represent her, and then issue her a Digital Certificate. In essence, the Digital Certificate binds a public key to an entity. They help prevent the use of fake public keys. So, the digital certificate contains the public key of the entity, its name and other information (key/value pairs, e.g name=Steve, organization=Apple, Inc.) It also includes the digital signature of the CA. It is that digital signature that allows the certificate to function as a verified and trusted certificate, by users who know and trust CA (in other words, have the CA's public key and know that that public key belongs to that CA) and trust the CA but do not know the entity identified by the certificate.

Apple is a Certificate Authority. Before deploying your iPhone application to a device, you need to obtain a certificate and a provision profile. So you prepare a Certificate Signing Request(CSR). This contains information about you. It is the information you want Apple to verify. When you create the CSR, the public and private keys pair is also created. The public keys is included in the CSR. The private key is never sent to Apple. The private key is used for signing your binary. Apple will get your CSR, create a digital certificate based on it. Then, you need to create a provisioning profile. The provisioning profile holds application IDs, device IDs and certificates.

You will need to submit CSRs and install Digital Certificates if you need to deploy and distribute iPhone applications, use Apple iPhone Notifications and In-App purchases. Hopefully this helps understanding why those are needed and what they are about.

Here is a progress update to my current main project (we call it 'CloudDS' which stands for cloud data store which is a silly name but it will have to do until we can find a replacement ). I have been working on the data store component of the service. It has taken at least x4 as much time and effort as I thought it would. A prime reason for underestimating the time requirements is that the initial features list I wanted to implement doubled in size. In addition to that, testing for most of the possible logic paths that could result to failure also took a long time - even if some of that testing was automated, not all of it was and validating results is harder than setting up the test environment.

In such a service, it matters little if most of underlying components fail (I/O and tasks scheduler, garbage collector, cache subsystem, etc) as long as the data management component is not affected. Suffering from a service outage is bad, suffering data corruption and/or data loss is something that has to be prevented by any means necessary.

As it stands, that said component now deals fine with reads and writes, self-healing, caching and performs faster than I hoped it would. The data model is based on BigTable, Dynamo, Cassandra and some earlier prototypes/projects we toyed with in the past. It borrows Cassandra's ColumnFamily/SuperColumn/Column key value representation model. Data are pushed into MemTables and an append only commit log, memtables are flushed into SSTables to disk.

The GCollector merges SSTables whenever required to reclaim space, resolve conflicts and extract a single value out of multiple versions, etc. All operations supported by Cassandra are implemented (query by path, predicate, column names, key ranges, etc ) and CloudDS clients/users will also be able to use a scripting language to describe explicitly down to bytes what they need(i.e give me the first couple of bytes for those values, or gimme a concatenation of those values, etc etc).

Now that that component is out of the way, I can move on to the rest; those are relatively straight forward to implement ( the tasks scheduler and the network I/O subsystems are mostly done ).

Monolithic RDBMs are losing ground to key-value data stores, particularly persistent distributed in nature. mySQL mounting problems was perhaps the key reason (pun intended) people looked elsewhere. Google's brilliant engineers realized that a key/value data model can satisfy the needs of almost every class of application that needs a datastore backend.

Key/value datastores are simple to build, easy to understand, easy to optimize, easy to scale. The, now famous, CAP theorem states that it is not practically possible to guarantee consistency, availability and partitioning resilience/tolerance all at the same time; one of those traits has to be sacrificed. Again, most applications really do not require all three to function. The CAP theorem is most likely derived from the Project Triangle mode.

Most web-based applications are built on simple data models. Most web-applications eventually suffer from service capacity and availability issues(i.e scalability woes). It is trivial to scale out(vertically) application logic processing(application servers), HTTP requests processing(web servers, load balancers). It is not easy to scale out an RDBMS. Some expensive systems(Oracle, etc) provide ways to address those issues (e.g Oracle RAC) but its expensive to deploy them, and most of them rely on a shared everything setup which just doesn't work in the long run. (Shared nothing is really the way to go).

Google released a bunch of papers ( actually, a bazillion of papers ), many of them defining and shaping the development of future related technologies. Namely, the papers describing GFS, BigTable, MapReduce (and of course, the paper the changed everything, "The Anatomy of a Large-Scale Hypertextual Web Search Engine" ) steered everyone to the right direction.

In the datastores domain, Hadoop/HBase, Radix, Cassandra and others, based on BigTable and Amazon's Dynamo papers, all relying on the simple key/value datastore model, are gaining market share - rightly so. Coupled with Memmache and similar services(in-memory key/value stores) they are solving the problems of service capacity and availability. This is a paradigm shift. Its a downhill for heavy-footprint, complex and inflexible datastore systems. They wont go away but will not be such a valuable(pun intended) component in tomorrow's technology landscape.

We are going to gradually migrate from RDBMs - though, we are not relying that much on them nowadays - to a key/value datastore (we are currently building one, also based on BigTable and Dynamo ). If nothing else, those simple systems are both simple and beautiful (for the most part).

The iPad was launched yesterday. What was, perhaps, the most hyped and anticipated product of this eon, finally became known as Steve Jobs held it proudly in his hands. These kind of products usually wind up being the recipients of all kinds of crazy expectations. Everyone wants the next-big-thing to do everything, better, in new radical ways and, if that is not enough, they expect even more in the end (in the case of Apple, that comes after Job's "one more thing" statement). All in all, most people are not pleased with the iPad. I am not one of those people. Here is my take on (some of) the 'issues' raised so far.

The name 'iPad' indeed feels wrong. I was expecting it to be called 'tablet' or 'Applet tablet' or something along those lines. Of course, I failed to consider the fact that this would not be a tablet, according to Apple at least. This is not a netbook either. In fact, this is a product that carves a new niche and defines it. So Apple didn't use tablet in the title. I remember when the first Macbook came out. People didn't like the name. They don't mind about it now - in fact they may even come to like it.

Multitasking The iPad does not allow multiple apps to run at the same time. This made much sense on the iPhone, but would make little to no sense to a laptop or netbook. Well, again, this is neither of those. My gut feeling tells me the reason Apple chose to go with this is threefold.

Cameras I would have loved it if iPad had a front-facing camera so that I could video chat with my brother. Well, actually, I never use video chat, but that need is valid for enough folks to make it an important omission. Apple has been keen on adding an iSight camera on just about every Mac product it has released. iMacs, Macbooks, the cinema displays, you name it. They love it when people video chat over iChat, use PhotoBooth to go silly and whatnot. What became apparent with the iPhone and even more apparent with the iPad is that Apple, presumably, has a very good set of reasons that led it to (at least for now) make it impossible (one way or another) to do so on this class of devices. It is likely that AT&T and other carriers are to blame here. Apple is giving up some features in exchange for others(better deals with the carriers?). Win some, lose some.

Adobe Flash support Apple doesn't like Flash. It can't be more obvious than that. They could list a number of technical and semi-valid reasons as to why this is a bad thing, but none of that would matter. I personally couldn't care less about Flash support on Safari, but the vast majority of potential users would, especially the folks in the US where, I hear, Hulu has become the go-to site for all things entertainment there. Unless things go way south for Apple, I don't see it changing its stance on the subject.

On screen keyboard When the iPhone came out, naysayers and pessimists sure had lots to say about the onscreen virtual keyboard. Nowadays though, people seem to actually prefer those kind of keyboards to the traditional physical ones, me included. Why waste device physical space, weight and looks for a 'real' keyboard, which is there even if you don't want it, when this new virtual keyboard works for you? I have been trying to type using both hands for the past few days. I can type now at least x2 many words/minute than I was able to do so in the past, when I was using just my thumb and I hope, expect, to get better at it. On the iPad, which features even bigger keys, things should be even better for me.

Books and magazines I wanted an ebooks reader for quite some time now. I was hoping for an ebooks store tightly integrated to iTunes and the iTunes store, the ability to subscribe to magazines (Wired, Edge, ..) and have them delivered to my 'subscriptions inbox' ( with a nice badge indicating new subscriptions count; me and Stelios would sure love that ) and a reader that would provide all benefits PDF readers come with, but with even more thrown in. Well, it won't really work like that on the iPad, at least for now, but this iBooks application sure looks sweet and well done. Apparently, each magazine, newspaper or other content provider will come up with its own solution to the digital content challenge ( NYTimes app demoed was pretty impressive ), which is one way to do it. Apple is playing safe there, again. There is no subscriptions aggregation place/app, or anything like that. Wired will need to build its own app and same will be true for everyone else. I can't wait to see what they will come up with.

All in all, this is v1.0 of a new product that, again, occupies and defines a new category on its own. Recall v1.0 of OS X, the iMac, the iPod, or any other product, produced by any company. Most v1.0 products are there to establish a baseline. Evolution bless them with more feature in later releases. This will certainly be the case with the iPad too. I never needed a tablet device ( my MBP 17" is everything I would 'ever' need ), but I am so buying one for me and Dora ( and for my brother and Dimitris if they themselves wont get one ). Exciting times (ahead).

I thought of two ideas for iPhone applications yesterday and I thought I would share it with anyone interested in pursuing the tasks. I may do it myself if time and motivation allows it, but do feel free to try your luck with them. So here goes nothing.

Life-tracker

The main principle idea is that you could use your iphone to keep track of your life through time, specifically where you have been, what have you been doing and what your thoughts on any given date. The said application would be really simple to use. You launch it, two buttons will make it possible to record your existing location (geolocation coordinates) and/or record thoughts or, say, what it is that you are doing at the moment. You can do that as many times as you wish, whenever you wish. Sometime later, you can sync all that with a web-based service. You could access your life activities through that service ( what have been thinking a year ago this very day? where have I been last week when I was on vacation in London? you want that on a google map - there you go) and so on, so forth. A more or less trivial application to build.

Javascript driven native iPhone applications

This is a no brainer, in fact I wonder why someone ( Apple even ) hasn't thought of it yet. One can expose the iPhone functionality(framework facilities) through Javascript (Javascript objects), have a simple runtime application that 'all' it does is act as the VM/runtime for javascript code. 'Everyone' knows Javascript, everyone(?) likes Javascript, why not make it possible to build real (i.e not hosted on Safari, web-apps ), native applications using the language? A developer would still submit an iPhone application ( the Javascript VM/runtime, with the javascript files and resources in the bundle ) to Apple, Apple, nor the users, would be able to tell the difference. Hey presto, a gazillion apps flood the App store - most will be crappy ( the nature of things ) some will turn out to be gems. If I could make it possible for my brother and my fellow javascript gurus at work to build any app they want as easy as they build our web-apps, that would be kinda cool. Here is what it could look like:

var myButton = new UButton();
myButton.text = "Hello World";
myButton.addListener('click', function(event)
{
alert('Your geolocation is:"+(new ULocation()).toString();
}));
thisWindow.containerView.addView(myButton);

Terry Pratchett on the right to die : Pratchett is one my favorite authors. His books ooze hilarity. He seems an all around awesome person, all things considered, too. Recently, he was diagnosed with Alzheimer's and he seems brave(?) enough to wish to die before this treacherous disease disrupts his mental abilities.

Every single time I need to use Windows, it feels wrong in a profound way. Every single time.

Paypal now available for US Xbox Live Accounts : A great idea, I hope it wont be long before this option becomes available for everyone else, too. Dealing with Microsoft points is an often laborious, not to mention futile, process. You would think MSFT would make it easy for us to give them money.

Check out the water crisis article on wikipedia and HowStuffWorks's why can't manufacture water? topic.

I got to spend some more time on the SGL revision, mentioned in previous post. The Garbage Collector (initially a mark and sweep based facility with a single heap for objects allocation, but will experiment with tri-colour marking and generation based segmentation (i.e generational GC) later on ) is almost in place, the main runtime component seems to be operating as expected(values are stored in 'registers', no stack manipulation, will use thread dispatching using GCC's goto *pointer extension ). I went through the WebKit's JavascriptCore source code for ideas. Came up with lots of notes and concepts that I want to toy with, if time and motivation permits. I am looking forward to diving into Google's V8's source code next week. I am afraid I will have to freeze the project in 2 weeks though and switch gears to work on our new file system (PFS, for lack of a better name), based on design decisions applied on Gooogle's GFS and Amazon.com's Domino.

I am somewhat let down by myself; over 5 projects are left in a semi-complete state. I need to focus on one at a time, wrap it up and move to the next one. Its just that August is mostly about trying out new stuff and research. Its the least demanding month of the year for us, most of the folks at work are away on their much deserved vacations. Little to no sound is produced ( I can't deal with sound, laughter, yelling, phone calls, you name it ) and that does wonders for my (currently degraded, due to Summer) productivity.

If you still have doubts about Javascript: becomingbeing the most popular programming languge, its probably because you are not exposed enough to the web-based Applications paradigm shift efffects.

Not only there seems to be more javascript code (in terms of sheer volume) out there, its also about the number of users using applications that are driven by it, most of them not really knowing, or wanting to know, what it is, but thats an entirely different story for someone to tell, again.

We are relying on 4 primary programming languages. C/C++ for backend 'stuff', PHP, Javascript and SGL for frontend/light-weight 'stuff'. Well, we do use bash scripting for _so_ much systems and operations 'stuff', some python and perl here and there, as well as some java and Flash/AS3 for more frontend 'stuff'.

SGL is our home-grown programming language, it stands for Switch Glue Language, Switch being the main framework/library everything - all services, tools, other libraries, etc - are based on. The idea is that we can use this language anywhere we want to script operations and 'glue' things(services, resources, operations, etc) together. Currently, its used for two major services.

Our frontend developers eventually have to learn, or at least get familiar with, all those three main frontend languages, PHP, Javascript, SGL. Interestingly enough, Javascript code output surpassed PHP output, in terms of volume, mostly because our apps got more functional, fancy, whatever cool bang you get from client-side logic on the browser -- I wouldn't know really, I don't know much about frontend development, our main frontend team do though and that's all that matters (partial unordered list: phaistonian, hatdi, sug, stelabouras).

Given that SGL has been long due for a rewrite ( the currenty language syntax and semantics ), I thought I put aside some time to rwrite it, this time around using Javascript language syntax and semantics so that, when its ready, we could replace PHP with SGL thus, effectively, switching from 3 frontend languages to just one. Our developers, current and future ones, would only need to learn a single language, which may be the greatest benefit to this shift, but it sure is not the only one.

This will be my third attempt to writing a programming language ( SGL being the second, PASTE was the first.. those were the days) and thanks to Javascript being a standard, its a 'simple' enough matter of writing an efficient enough VM that will run the emitted bytecode ( I am toying with the idea of being able to target PHP and other languages, eventually, generating - say, PHP code from SGL code and so on ).

So far the lexer, most of the parser and some parts of the VM are in place. Hopefully, there will be enough time and sustained motivation to keep this going (its a side project, so it can't really preempt current major Phaistos projects ) until its ready, perhaps by the end of the month.

Simplicity is the ultimate sophistication- Leonardo da Vinci

My (by far) most awaited movie release is Tarantino's new film 'Inglourious Basterds', coming out in August. Check the latest trailers. Pitt's talent in bringing to life the badass character Tarantino penned, coupled with the overall theme makes for an exploding, action-packed funny movie, the way I see it. I can't wait to watch it.

Google Says Mobile App Stores Have No Future : Unless web-based applications offer the same kind of functionality native applications can and do offer, I don't see how everything will be moved off to the Web. It may make sense for a class of applications, but, realistically speaking - there are a few hoops that are just too difficult to jump over for this to come to pass. Palm is betting big on this concept, now, with Pre and its Mojo SDK - which is really not working out for them. Maybe in a few years when the underlying hardware and operating system services will be, somehow, become available/exposed to remotely executed apps it will make more sense to expect this to be the case.

Serious Doubts : Marco Arment argues it is hard or next to impossible to run a serious business of writing iPhone applications. I still haven't developed a 'real' iPhone app, though I wouldn't really do it for the money, not that I would mind getting any in the long run. As far as I am concerned, those digital distribution systems solve, well, the physical distribution problems(which are many) and facilitate access to content that has traditionally been hard to access. Everything else is a byproduct of competition and demand.

Now that Monkey Island Special Edition and the first episode of Tales of Monkey Island are out, my new most awaited game is Bioware's Dragon Age : Origins. I have high hopes for the spiritual successor to Baldur's Gate, not the least concerned with the somewhat negative previews so far( people seem to dislike the 'adult' content in the game). It will ve possible ( much like it was the case with Baldur's Gate and other Infinity Engine games ) to issue orders to party members in real time, or pause and queue up actions that will be carried once upon unpausing the game. As far as I am concerned, this is the best control scheme for CRPGs ever devised. Who knows, maybe Minsc will make a surprise cameo appearance. "Go for the eyes, Boo! Go for the eyes!"

I downloaded and played The Secret of Monkey Island : Special Edition on Xbox Live Arcade. It took less than few minutes to download the 500+ MB demo and no more than 10 seconds after I launched it to be excited. The classic intro screen was on display with the classic - much beloved - monkey island music theme(youtube video) and then it faded out to the new and improved screen, spotlighting the most exciting enhancement to the remake.

The folks at Lucasarts pulled it off; it is the same wonderful game we 'all' know and love, outfitted with gorgeous hand drawn graphics, dialogue voice-overs and a user interface that actually works on the consoles.

I wish I could purchase the full game, alas, the whole Microsoft Points deal is messy due to the fact that Xbox Live is not 'officially' supported in Greece and the implications related to credit cards processing. Links: Review on Gamespy, Guybrush Threepwood

I turned on my dekstop PC tonight in order to look for Trine on Steam. I came across a review on Giantbomb and was really impressed by the premise and the visuals of the game. Some friends have played it and had nothing but words of praise for it. Hopefully, I will play for a while this weekend. Speaking of my PC, in retrospect, it was one of the least meaninful purchases I ever made. My intentions were to install Linux so that I could work on the Linux Kernel and prepare for Larrabee's arrival so that I would eventually get to develop for it ( I am very excited about Larrabee if previous posts haven't made it clear by now ). Turns out, I only use my beloved MBP anyway for I have so little time to spare lately. Oh, well.

Being Amiga users back in the day was a lot like choosing a platform with a soul over PCs. I feel the same away about using Apple products (especially OS X). It feels great, it feels right, it feels much like it felt when I was using our Amiga. Whenever I have to use Windows ( thankfully, not often nowadays ) it feels wrong in so many ways. Come to think of it, the only thing I like about Windows is the Win32 API

Just like every summer, I am going through a huge productivity slump. I can't wait for Autumn. This time around I intent to try to deal with it though.