A. Nene

Web Services Architecture – When to Use SOAP vs REST

2014-10-13T21:18:00.000+01:00

SOAP (Simple Object Access Protocol) and REST (Representation State Transfer) are popular with developers working on system integration based projects. Software architects will design the application from various perspectives and also decides, based on various reasons, which approach to take to expose new API to third party applications. As a software architect, it is good practice to involve your development team lead during system architecture process.

This article, based on my experience, will discuss when to use SOAP or REST web services to expose your API to third party clients.

Web Services Demystified

Web services are part of the Services Oriented Architecture. Web services are used as the model for process decomposition and assembly. I have been involved in discussion where there were some misconception between web services and web API.

The W3C defines a Web Service generally as:

A software system designed to support interoperable machine-to-machine interaction over a network.

Web API also known as Server-Side Web API is a programmatic interface to a defined request-response message system, typically expressed in JSON or XML, which is exposed via the web – most commonly by means of an HTTP-based web server. (extracted from Wikipedia)

Based on the above definition, one can insinuate when SOAP should be used instead of REST and vice-versa but it is not as simple as it looks. We can agree that Web Services are not the same as Web API. Accessing an image over the web is not calling a web service but retrieving a web resources using is Universal Resource Identifier. HTML has a well-defined standard approach to serving resources to clients and does not require the use of web service in order to fulfill their request.

Why Use REST over SOAP

Developers are passionate people. Let's briefly analyze some of the reasons they mentioned when considering REST over SOAP:

REST is easier than SOAP

I'm not sure what developers refer to when they argue that REST is easier than SOAP. Based on my experience, depending on the requirement, developing REST services can quickly become very complex just as any other SOA projects. What is your service abstracting from the client? What is the level of security required? Is your service a long running asynchronous process? And many other requirements will increase the level of complexity. Testability: apparently it easier to test RESTFul web services than their SOAP counter parts. This is only partially true; for simple REST services, developers only have to point their browser to the service endpoints and a result would be returned in the response. But what happens once you need to add the HTTP headers and passing of tokens, parameters validation… This is still testable but chances are you will require a plugin for your browser in order to test those features. If a plugin is required then the ease of testing is exactly the same as using SOAPUI for testing SOAP based services.

RESTFul Web Services serves JSON that is faster to parse than XML

This so called "benefit" is related to consuming web services in a browser. RESTFul web services can also serve XML and any MIME type that you desire. This article is not focused on discussing JSON vs XML; and I wouldn't write any separate article on the topic. JSON relates to JavaScript and as JS is very closed to the web, as in providing interaction on the web with HTML and CSS, most developers automatically assumes that it also linked to interacting with RESTFul web services. If you didn't know before, I'm sure that you can guess that RESTFul web services are language agnostic.
Regarding the speed in processing the XML markup as opposed to JSON, a performance test conducted by David Lead, Lead Engineer at MarkLogic Inc, find out to be a myth.

REST is built for the Web

Well this is true according to Roy Fielding dissertation; after all he is credited with the creation of REST style architecture. REST, unlike SOAP, uses the underlying technology for transport and communication between clients and servers. The architecture style is optimized for the modern web architecture. The web has outgrown is initial requirements and this can be seen through HTML5 and web sockets standardization. The web has become a platform on its own right, maybe WebOS. Some applications will require server-side state saving such as financial applications to e-commerce.

Caching

When using REST over HTTP, it will utilize the features available in HTTP such as caching, security in terms of TLS and authentication. Architects know that dynamic resources should not be cached. Let's discuss this with an example; we have a RESTFul web service to serve us some stock quotes when provided with a stock ticker. Stock quotes changes per milliseconds, if we make a request for BARC (Barclays Bank), there is a chance that the quote that we have receive a minute ago would be different in two minutes. This shows that we cannot always use the caching features implemented in the protocol. HTTP Caching be useful in client requests of static content but if the caching feature of HTTP is not enough for your requirements, then you should also evaluate SOAP as you will be building your own cache either way not relying on the protocol.

HTTP Verb Binding

HTTP verb binding is supposedly a feature worth discussing when comparing REST vs SOAP. Much of public facing API referred to as RESTFul are more REST-like and do not implement all HTTP verb in the manner they are supposed to. For example; when creating new resources, most developers use POST instead of PUT. Even deleting resources are sent through POST request instead of DELETE.

SOAP also defines a binding to the HTTP protocol. When binding to HTTP, all SOAP requests are sent through POST request.

Security

Security is never mentioned when discussing the benefits of REST over SOAP. Two simples security is provided on the HTTP protocol layer such as basic authentication and communication encryption through TLS. SOAP security is well standardized through WS-SECURITY. HTTP is not secured, as seen in the news all the time, therefore web services relying on the protocol needs to implement their own rigorous security. Security goes beyond simple authentication and confidentiality, and also includes authorization and integrity. When it comes to ease of implementation, I believe that SOAP is that at the forefront.

Conclusion

This was meant to be a short blog post but it seems we got to passionate about the subject.

I accept that there are many other factors to consider when choosing SOAP vs REST but I will over simplify it here. For machine-to-machine communications such as business processing with BPEL, transaction security and integrity, I suggest using SOAP. SOAP binding to HTTP is possible and XML parsing is not noticeably slower than JSON on the browser. For building public facing API, REST is not the undisputed champion. Consider the actual application requirements and evaluate the benefits. People would say that REST protocol agnostic and work on anything that has URI is beside the point. According to its creator, REST was conceived for the evolution of the web. Most so-called RESTFul web services available on the internet are more truly REST-like as they do not follow the principle of the architectural style. One good thing about working with REST is that application do not need a service contract a la SOAP (WSDL). WADL was never standardized and I do not believe that developers would implement it. I remember looking for Twitter WADL to integrate it.

I will leave you to make your own conclusion. There is so much I can write in a blog post. Feel free to leave any comments to keep the discussion going.

Liferay and AngularJS Made Simple: Connecting AngularJS to a Backend with REST and JSON

2014-10-01T08:55:00.000+01:00

Introduction

Liferay is the leading Open Source Enterprise Portal. One may asked what an enterprise portal is and this question is very valid as it has been asked on every single Liferay project that I have worked. This blog post is not about defining what an enterprise portal is but it wouldn't be a crime if we provided a brief definition:

An enterprise portal is a web application which provides services required by an enterprise such as: user management, authentication and authorisation services, ability to connect to third party applications and provide a single point of access to multiple applications, hence the "portal".

The above is my own definition and it could be extended to encompass web content management, content management system (CMS) and single sign on (SSO). This post is about Liferay and the use of its web content management system (WCMS) to create single page applications using AngularJS. The motivation to create portlets using AngularJS instead of Java is as:

Portlet development using Java is very expensive
Not many Java developers with portlet experience
Java portlets development requires heavy duty tools such as build tools, IDE and JVM
Portlet developers need to be familiar with the Portlet API, lifecycle and framework

We will focus on Liferay available RESTful web services API but do remember that you can create your own custom web services using Liferay service builder SDK.

Liferay RESTFul API and Security

Liferay ReSTFul and SOAP API implement the same security as the core library:

API can be secured so that only authenticated users can access them (AUTHENTICATION)
API can be secured so that only users with the right roles can executed certain API calls (AUTHORIZATION)

When creating your own custom API, Liferay Services Builder will create the necessary permission for the web services API.
For a list of API available in Liferay, point your browser to the following

http://<your-server-address>:<your-server-port>/api/jsonws

Liferay will provide a means of testing the services calls when the above URL is loaded. Most services execution will require authentication or a secured token to be passed on with the calls. This level of a security is required in an enterprise environment. It is possible to stop Liferay from checking for the secured token in portal-ext.properties as

Auth.token.check.enabled=false

Software developments should promote code reuse, therefore by separating the business logic from the portlet code, developers can share the business logic with third party applications.

Why Use AngularJS to Create Web Applications (Not Portlets)?

This is not a tutorial on AngularJS. Developers should use the same approach for developing any AngularJS application to developing Liferay web applications.
AngularJS is a popular JavaScript framework promoting Object Oriented Development (OOD) and Model View Controller (MVC) to the JavaScript community. Java developers are already custom with the methodology through the use of Spring MVC and JSF for front end developments. Developers familiar with Google Web toolkit (GWT) should find themselves in familiar territory. Now to answer the question of why use AngularJS to create web applications on Liferay?
AngularJS is JavaScript and therefore can be executed in the browser without recompilation and redeployment. Liferay Web Content Management System (WCMS) provides an HTML editor and content versioning. Liferay JSONWS API runs on the same server and can be accessed through the JavaScript written in the WCMS. AngularJS modules can be written in a third party editor such Notepad++ and uploaded to Liferay Content Management System (CMS). The Liferay CMS provides a link to the latest version of the file which can be referenced in the HTML/ JavaScript code. By creating the web services in Java through Liferay Services Builder, the java developer can focus on the business logic – including testing. The front end developer can utilise his skills in HTML and JavaScript to develop the user interfaces and any necessary interactions with the backend through the ReSTFul services. There is a clear separation of work and accountability. The learning curve for the Java developers to create the services will be minimal. To preview the live code, the frontend developer only has to save the content (WCMS) and refresh the page to see the latest changes.
Here is a quick example:

 <div ng-app="" ng-controller="companiesController">  
   <ul>  
    <li ng-repeat="x in data">{{'title: ' +x.title + ', group Id: ' + x.groupId }}</li>  
   </ul>  
 </div>  
 <script>  
   function companiesController($scope,$http) {  
    $http.post("http://localhost:8080/api/jsonws/assetentry/get-company-entries/company-id/10157/start/0/end/5?p_auth=cbSXanJ2")  
    .success(function(response) {$scope.data = response;});  
   }  
   companiesController.$inject = ['$scope', '$http'];  
 </script><script src="http://ajax.googleapis.com/ajax/libs/angularjs/1.2.15/angular.min.js"></script>

You can copy the above in a new web content article and see the result displayed on the page. Make sure to change red bold values to your system specific:

Company id: for ease of testing, you can retrieve that value from the control panel
P_auth: this value can be looked up programmatically as it will change every time the user logs into the portal

Traditionally, the simple code above would require a JavaServer page or JSF application which is slow for rapid prototyping. In software testing, portlets controller are one of the most complex components to test. By creating a clear separation between components, testers and automated tools can test each components individually. The sample code pulls information from Liferay REST web services and displays a list of registered companies on the page.

Conclusion

Liferay has a rich set of features which allows developers to create enterprise components and applications. When working with ReSTFul services, Liferay Web Content Management editor can act as an Integrated Development Environment in the browser. AngularJS is approaching maturity and it is very popular with web developers. Liferay 7 (next release as of writing ) will be introducing single page portlets but this is already possible with AngularJS and ReSTFul web services API. Needless to say that you can use any web browser to create content in Liferay CMS and debug your code in real time using tools such as Firebug.

Big Data Architecture Best Practices

2014-09-25T13:19:00.000+01:00

The marketing department of software vendors have done a good job making Big Data go mainstream, whatever that means. The promise of we can achieve anything if we make use of Big Data; business insight and beating our competitions to submission. Yet, there is no well-publicised Big Data successful implementation. The question is: why not? Clearly this silver bullet where businesses have seen billions of dollars invested in but no return on investment! Who is to blame? After all, businesses do not have to publicise their internal processes or projects. I have a different view to that and the cause is on the IT department. Most Big Data projects are driven by the technologist not the business there is create lack of understanding in aligning the architecture with the business vision for the future.

The Preliminary Phase

Big Data projects are not different to any other IT projects. All projects spur out of business needs / requirements. This is not The Matrix; we cannot answer questions which have not been asked yet. Before any work begin or discussion around which technology to use, all stakeholders need to have an understanding of:

The organisational context
The key drivers and elements of the organisation
The requirements for architecture work
The architecture principles
The framework to be used
The relationships between management frameworks
The enterprise architecture maturity

In the majority of cases, Big Data projects involves knowing the current business technology landscape; in terms of current and future applications and services:

Strategies and business plans
Business principles, goals, and drivers
Major framework currently implemented in the business
Governance and legal frameworks
IT strategy
Pre-existing Architecture Framework, Organisational Model, and Architecture repository

The Big Data Continuum

Big Data projects are not and should never been executed in isolation. The simple fact that Big Data need to feed from other system means there should a channel of communication open across teams. In order to have a successful architecture, I came up with five simple layers/ stacks to Big Data implementation. To the more technically inclined architect, this would seem obvious:

Data sources
Big Data ETL
Data Services API
Application
User Interface Services

Big Data Protocol Stack

Data Sources

Current and future applications will produce more and more data which will need to be process in order to gain any competitive advantages from them. Data comes in all sorts but we can categorise them into two:

Structured data – usually stored following a predefined formats such as using known and proven database techniques. Not all structured data are stored in database as there are many businesses using flat files such as Microsoft Excel or Tab Delimited files for storing data
Unstructured data – businesses generates great amount of unstructured data such emails, instant messaging, video conferencing, internet, flat files such documents and images, and the list is endless. We call the data "unstructured" as they do not follow a format which will make facilitate a user to query its content.

I have spent a large part of my career working on Enterprise Search technology before even "Big Data" was coined. Understanding where the data is coming from and in what shape is valuable to a successful implementation of a Big Data ETL project. Before a single a line of programming code is written, architects will have to try and normalise the data to common format.

Big Data ETL

This is the part that excites technologists and especially the development teams. There are so many blogs and articles published every day about Big Data tools that this creates confusions among non-tech people. Everybody is excited about processing petabytes of data using the coolest kid on the block: Hadoop and its ecosystem. Before we get carried away, we first need to put some baseline in place:

Real-time processing
Batch processing

Big Data - Data Consolidation

The purpose of Extract Transform Load projects, regardless of using Hadoop or not, is to consolidate the data into a single view Master Data Management for querying on demand. Hadoop and its ecosystem deals with the ETL aspect of Big Data not the querying part. The tools used will heavily depends of processing need of the project: either Real-time or batch; i.e. Hadoop is a batch processing framework for large volume of data. Once the data has been processed, the Master Data Management system (MDM) can be stored in a data repository such as NoSQL based or RDBMS – this will only depends on the querying requirements.

Data Services API

As most of the limelight goes to the tools for ETL, a very important area is usually overlooked until later almost as a secondary thought. MDM will need to be stored in a repository in order for the information to be retrieve when needed. In a true Service Oriented Architecture spirit, the data repository should be able to expose some interfaces to external third party applications for data retrieval and manipulation. In the past, MDM were mostly created in RDBMS and retrieval and manipulation were carried out through the use of the Structured Query Language. Well this does not have to change but architects should be aware of other forms of database such NoSQL types. The following questions should be asked when choosing a database solution:

Is there are standard query language
How do we connect to the database; DB drivers or available web services
Will the database scale when the data grows
What security mechanism are in place for protecting some or whole data

Other questions specific to the project should also be included in the checklist.

Business Applications

So far, we have extracted the data, transformed and loaded it into a Master Data Management system. The normalised data is now exposed through web services (or DB drivers) to be used by third party applications. Business applications are the reason why to undertake Big Data projects in the first place. Some will argue that we should hire Data Scientists (?). According many blogs, Data Scientist roles is to understand the data, explore the data, prototype (new answers to unknown questions) and evaluate their findings. This is interesting as it reminds me the motion picture The Matrix, where the Architect knew the answers to the questions before Neo has even asked them yet and decides which one are relevant or not. Now this is not how businesses are run. It will be extremely valuable if the data scientist may suggest subconsciously (Inception) a new way to do something but most of the time the questions will come from business to be answered by the Data Scientist or whoever knows the data. The business applications will be the answer to those questions.

User Interfaces Services

User interfaces are the make or break of the project; a badly designed UI will affect adoption regardless of the data behind it, an intuitive design will increase adoption and maybe user will start questioning the quality of the data. Users will access the data differently; mobile, TV and web as an example. Users will usually focus on a certain aspect of the data and therefore they will require the data to be presented in a customised way. Some other users will want the data to be available through their current dashboard and match their current look and feel. As always, security will also be a concern. Enterprise portal have been around for a long time and they are usually used for data integration projects. Nevertheless, standards such as Web Services for Remote Portlets (WSRP) make it possible for User Interfaces to be served through Web Service calls.

Conclusion

This article show the importance of architecting a Big Data project before embarking on the project. The project needs to be in line with the business vision and have a good understanding of the current and future technology landscape. The data needs to bring value to the business and therefore business needs to be involved from the outset. Understanding how the data will be used is key to its success and taking a service oriented architecture approach will ensure that the data can serve many business needs.

4 Reasons Football Teams Should Use Big Data and IoT

2014-09-17T13:14:00.001+01:00

This is a case why football teams (or soccer for my transatlantic friends) should make greater use of the Internet of Things and big data.

Here are some 5 simple reasons:

1. Player health tracking

In 2012, then only 24 years of age, Fabrice Muamba a footballer from Bolton Wanderers suffered an heart attack on live TV during an FA Cup game. This was not the first such incident on live TV; Marc-Vivien Foe, the Cameroonian player, died on June 26 2003 during a confederation cup game against France. Not to mentioned what happens off-pitch, football club have a duty to look after their players and that's including their health.

Let's clear, I'm not saying that current health wearable devices could have saved them but they could have played a great role. Read the signs early to avoid the disaster as prevention will always be better than the cure.

So how can be big data helps us improve the athletes health? You may ask. I will simplify it for illustration purposes. Teams need to put in a place an IoT and Big Data strategy where athlete will wear various devices during training and games. Think about a heart monitoring bracelets that players can wear during training and games for the physiotherapy team to analyse later or in real time. The devices will record the time, number of foot steps (within a period - in order to differentiate between walking and running) and heart rate. Let's say that player John Smith health is changing during the game, the physios will get alerted of the change of condition and can take appropriate actions. Maybe we can add devices to shin pads so that players do not have to wear uncomfortable devices.

2. Player positioning and technique

Great players do this naturally but not all players are born equal. I remember when I used to play football in my early teen for USO back in France. In training, we would learn how to shoot, head the ball and basic skills. The trainer would then let us know what we did wrong and how to ameliorate our techniques. Now professional players have a large team working with them daily; strikers will have coaches to help them for attacking positioning, goal keeper will have coaches too and etc... The current tech used by the teams are heavy duty machine which are employed behind close doors. How can IoT make improve on this? Again through wearable technology, we can track players movement. GPS integrated devices will track player movement over a map (football field), chin pad enabled devices will help visualise the player shooting techniques. The list of devices are only limited to your own imagination but the possibilities are endless. Players can then analyse their own data or with the help of their coach, see how they can improve. No more killing pigeons with those bad shots over the bar!

3. Team metrics

Now what is the point of all that data generated if not put to use? We have health data and game metrics (positioning and technique) about the whole team. The players health will give us an indication of who is fit for game. Being a big fan of the beautiful game, we have observed how various conditions can affect a player performance such as weather and pitch condition. The data scientists would analyse who can perform better under the current conditions. Don't get it wrong; top players, if fit for the game, are indispensable to the team but all advices should be welcomed. You will have what we call "the best team on paper". Gamblers use this technique in various events to pick their winners/ losers.

4. Opponents analysis

Usually we can find a wealth of information about teams from various sources such as sport sites, videos and blogs. The more you know about your opponents the better you will perform (Sun Tzu - Art of War). Team changes twice a year (transfer window) for those who can afford it. Is a team really as good as their last game? This question can be answered with big data analytics. Taking the conditions into consideration and our own team metrics, we can pick a strategy (team formation, defensive or attacking...). The big data gives (all) the information that we need in order to make decisions.

Conclusion

The Internet of Things (IoT) and Big Data provides an opportunity, even if not new, to gather information and make executive decisions much faster than before. There should not be any reasons not to have the full picture before hand. Football, as any other sports, is very competitive and everyone wants to create advantage. IoT devices are getting cheaper and more data can be collected for analysis. Big Data has the tools but not a silver bullet. We can prevent tragedies by tracking our players health and help them improve by analysing their techniques. Let's realistic that winning a game is probably beyond Big Data and IoT alone but they will play a big part in shaping the future of the game.

Technology news about start-ups big data cloud IoT Liferay

2014-09-08T16:34:00.000+01:00

After over two years of break, I'm back at it again – this time with a redefined purpose. As before, I will be sharing my technical views and experience through articles, whitepaper, tutorials. I will include interviews with tech start-ups from across the web; the system architecture and methodology. The start-ups will include videos and best practices learn from experience. Previously, I was approached by many to do some reviews of tech gadgets and sites, therefore I will try to accommodate that crowd whenever I can.

Over the past two years, I have been involved in some very exciting projects such as launching a new public facing web portal for a leading German car manufacturer to launch the fleet of electric cars. Working as a principal architect on the project and managing team across geographical boundaries. Then moved on as Principal Portal Architecture at Accenture on project for Europe leading grocery retailer to develop business to business portal and integrating it with various third party applications. Sure – this meant that I couldn't blog anymore or even attend some of the tech events around me. I missed the blogosphere and the sharing of knowledge but I'm back now.

Why not make some requests on the sort of tutorials you would like me to do or site (start-ups) you would like to review. As always, my primary focus will be Big Data, Cloud, Business Intelligence, Liferay and Java Development. The Internet of Things will also be covered as Java is playing a major role in its enablement.

Hope you enjoy my new content as you enjoyed the old ones and see you soon.

I Got hacked today and never use GoDaddy.com for hosting

2012-12-31T16:41:00.001+00:00

I'm going to get straight to the point; the last 24hrs one of my domain got hacked. Initially I thought it was one my developer messing stuff on the server. Today, 31st December 2012, I tried accessing the domain through the web browser to see if they were any updates from the development team. When tried to browse to the web site, my antivirus started to go crazy: Threat detected - Exploit Blackhole Exploit Kit (type 2371). First thing I did was to get the team on the phone and you can imagine how conversation went. These are the steps have taken so far:

Conversation with dev team: in summary; make sure that you didn't use any code that you copied and pasted from the internet and change all user names and passwords.

Conversation with GoDaddy Support:

Godaddy: Hi, How can I help you today?
Me: My domain xxxx got hacked today...
G: Ok?! What can we do for you?
M: I would like to have a copy of the FTP log for my domains, please.
G: Oh...we can't do that
M: Why not?
G: It's for security reason that we cannot provide you with the logs.
M: very strange as my account got hacked and this means that you have an intruder on your network, therefore you system is potentially vunerable to more attacks...
G: Sorry sir but our systems are secured...
M: clearly not, so what do you suggest I do?
G: You can delete all files and folders in your domain...
M: We are alreay reverting the system to a backup state but that does not garantee to our system will be safe from any hidden script put in place by the hacker...
G: Sorry but that's all we can do...
M: Do you have a virus scanner that you can run on our domain?
G: Yes...Let me check the cost for that...It will cost about £45+VAT
M: Is there any default protection that comes with my package, I am on the Ultimate package?
G: No, there isn't any default antivirus provided for any packages...
M: So there is nothing I can do check for viruses and malware unless I pay the £45?
G: No, you have to pay for the service...
M: No problem, I'll just take my business to a more secure hosting provider
G: Ok.
M: Good bye!

Now, if you navigate to Godaddy hosting page you will see that there is a malware scanner provide but no antivirus therefore leaving my systen open to threats. It is shame because I have been with them for quite awhile but you can only the strength in a relationship only when it is put to the test. On this final note, I do not recommend GoDaddy hosting for any businesses.

Feel free to share your views in the comments.

PaaS Cloud - The Rise of The Developer Entrepreneur

2012-05-22T16:27:00.001+01:00

In bad times such as recession, there is the inevitable fact that some people will be laid off by their employer. In these circumstances, when software developer are made redundant they usually set-up shop. The majority goes into providing software development services for small company or friends. They set-up websites to promote their services and few focuses on building their own product.

In London, services based company took a massive hit during the downturn and it is still felt now; well the UK and now in a double-dip recession. So what is my point here? It is much harder to sell services and even harder to export services than products. When providing services, your market has geographical constraint; just think what would take for your to sell your services to a potential client 100 miles from your local?

It used to be hard to build products in the past due to high cost of hardware; to build a java-based product you had to find a suitable host and then pay for a dedicated server or VPS. As you are already tightening your belt, it is likely that you do not want to spend £60 + VAT a month on services to host your product and all the headache of troubleshooting it in case of problem.

Enter the PaaS, Platform as a Service. For this posting I am going to look at PaaS from a Java perspective but it should be applicable to any other programming language.

The competition in the PaaS market is fierce there is no single winner yet. Amazon is the leader for IaaS but not much to be deployed on Beanstalk as yet. If you look around the market, you can see that major technology vendor are jumping on the bandwagon:

Some of those PaaS services are free such as OpenShift and CloudBees. OpenShift runs on JBossAS7 and if you are using Eclipse for it becomes a breeze to develop and deploy, but the only downside is that it's still in beta stage.

Utilising the current ecosystem of PaaS, you can go from:

inception (idea) -> development -> product

for almost nothing. The prototyping is now free for example you can use Heroku to build Facebook apps making all the Facebook API directly available to your app and hosted for free (until you out grow your free account of course). I will suggest that developers familiarises themselves with Lean Startup and Lean Software Development.

If you are a developer and out-of work, you have no excuse of no building your cloud apps. Even if you do not want to start a business and you can do it as a learning practice and use that to your advantage next time you are interviewing for a new role.

With PaaS Cloud, every developer is a potential entrepreneur and the barriers to entry are almost non-existent. If you are looking for a job, it is no longer acceptable just to have some sample code on GitHub but you should also have a sample project hosted on a free cloud provider to showcase.

This is blog was written from Cloudstock 2012

Developers should not believe the HTML5 hype

2012-05-06T01:20:00.001+01:00

As an Architect and Lead developer, I have to sit down with the non-techie and convince them why I think a technology or framework is the right one for a project and then I build my team to deliver the solution. The past two years or so, the marketing engine of technology companies have been spinning at full throttle. HTML5 is a the future, everything will run in the browser and more blah blah blah.
Don't get me wrong, HTML5 brings some good technologies to the web application developer. Nevertheless, I think there is a lot of hype as HTML5 doesn't really bring anything new to the table.
First of all, there are multiple type of applications that we are all well-aware of;

Consumer
Business
Enterprise

Consumer Application

If you are building a consumer oriented application which doesn't use any of the native functionality of OS, then I would recommend the application to be built in HTML5 and all the RIA fanfare that comes with it. You can best view this in the mobile space where developers build either native or web apps depending on the application requirement. Remember that web application (HTML5 ) do not have direct access to printers, USB port or any other hardware such as Bluetooth and network services. Simple word processing application that can be developed in HTML5 such as Google Doc (where this blog is typed from). I haven't come across any serious application written in HTML5 or the likes yet (JavaFX, Flash and Silverlight are not HTML5).

Business Application

Critical business application such as POS which requires access to barcode scanner as an example can't be written using HTML5. You can have a native application delivered through the browser such as Java Applets ( or JavaFX) using webstart. Java applets can access OS features and hardware and provides another layer of security. Business application are delivered in controlled environments, for example, the application can be deployed on Linux desktop in company "a" environment only. For as much "fanfare" one might create around HTML5, these type of application will not cease to exist.

Enterprise Application

Enterprise applications come in various forms, from desktop to servers. As this is a comparison to HTML5, I will only focus on desktop application. First, let take a financial company such as a stock brokerage firm. There is a reason why stock trading application run as closely to the OS as possible ( and also to the exchange), in one word, PERFORMANCE. Web browsers performance sucks regardless of which one you are using, JavaScript is just too slow to implement some of the logic. In the trading business, a millisecond is all that is required to lose millions of £. Can you for one second imagine building a Bloomberg trading platform using web technologies, that's laughable. There is a reason why the finance industry are still using Java Swing as their desktop technology of choice.

Conclusion

HTML5 is a promising step in the right direction to building scalable robust web application but it will not obliterate desktop applications, not today, not tomorrow, not ever (really!?). We can already see it in the mobile space where developers rather write native application so that they can utilize OS features and hardware. Web applications cannot access you local files directory (I am not talking uploading a file to a site) to read or write. A simple operation such as reading available space in a directory or writing a log to a local directory is not possible (again I am not talking about downloading or saving a page). Therefore, developers should not believe the hype. HTML5 is not the silver bullet and it is a shame when companies such as Adobe sends mix-messages by discontinuing their Flex offering. Anyway, Adobe has never been a big player in the enterprise desktop application market.

If you disagree with my points, feel free to share your thoughts.

5 Things all Java developer should know when developing for the cloud

2012-04-30T12:56:00.000+01:00

The last couple of years, "Cloud Computing" replaced Web 2.0 as the new buzzword. You can read, hear and see everywhere the cloud is coming. To most developer, this is still the same old sh*t. If you have experience in developing distributed system then you should be fine, you say. Well not entirely true, the IT department wants to deploy on cheap cloud and therefore some restrictions now applies. I will list 5 things that I think all developers should know when working with cloud Platform as a Service provider such as Amazon Beanstalk or Google App Engine. This list also applies to IaaS architecture. Some of the points might be obvious to the more experienced, nevertheless, they need to be mentioned.

Static objects

We all know the difference between instance variable (non-static) and class variable (static variable). We use static to tell the JVM that they should only be one instance of this variable (singleton). If the static variable is declared with the "final" keyword, this will not cause a problem in a distributed environment as the value will never change. The problem is when we expect the value of the variable to change. As in a cluster environment, GAE and Beanstalk run your application in multiple JVM. If a the value of your static variable has changed in JVM, it will not be propagated to the cluster therefore leading to inconsistencies. I recommend that you avoid static variable unless that set as "final" and their values are hard-coded so there is no way to change their values are runtime.

Caching Objects

This one is related to performance in order to avoid expensive operations such as running database queries and others. Sometimes we need to cache objects in memory and therefore we implement our own caching strategy through the use of simple HashMap or some other caching solutions available outthere. Caching has many benefits but implementing a caching strategy should be approached with care. This is because caching has the same problem as static objects. Your cache will be in the local JVM therefore not it will not be visible in the cluster. There are some solutions, for example, GAE uses Memcached and Beanstalk can make use of Amazon ElastiCache which is compliant with Memcached. When developing for a PaaS environment, make sure to not implement your own caching system but look for one that is supported by the vendor. I know this can lead to vendor lock-ins.

Server-side Session

Something we do take for granted in single environment is storing application session data on the server. Based on experiences, mainly using GAE, I encountered multiple issues with session management. Since then, Google has fixed alot of the issues with the way GAE handle sessions for Java application. To minimize writing session to a datastore, we store application state in memory. Most application are written without any vendor approach in mind; so we use JEE as-is. This approach would work in you deploy in any self hosted clustered environment but Google PaaS. Google implements their own session management which is off by default therefore you need to enable it in appengine-web.xml and make sure that all your objects implements the java.io.Serializable interface.

Note: Note, session data is always written synchronously to memcache. If a request tries to read the session data when memcache is not available (or the session data has been flushed), it will fail over to the datastore, which may not yet have the most recent session data. This means that asynchronous session persistence may cause your application to see stale session data. However, for most applications the latency benefit far outweighs the risk.

Event-driven Execution

This is more about running a process at a given time such as Scheduling task. Again, in a managed environment, it is straightforward to implement a timer or scheduler service. But this is a clustered environment which is not managed by yourself and their stack his different to yours. I personally use Quartz Scheduler when working in a single server environment. In a clustered environment such as Beanstalk or GAE, it is difficult to know which instance will be triggered and execute the task only once. The folks at Google have provided another solution with their own implementation of Cron for Java which can be used. At the time of writing, Amazon Beanstalk didn't have a solution yet. Therefore, consider before-hand when designing your system, which approach to take in order to create scheduled tasks for your application.

JRE white list

I believe this related to GAE J only. Google App Engine for Java doesn't allow the use for all available API in Java, especially if they do require access to the file system. The fact that there is a such a restriction impose by the Google has led us to look elsewhere for some of our projects. The cost of re-developing our application to please them is much higher than deploying them elsewhere. Also, another downside of GAE J is doesn't fully support JEE servlet specification. You cannot implement custom security for your application through your web.xml therefore pushing you to use Google own security mechanism. I would recommedn using GAE J when developing a greenfield project which can be built with these restrictions here and here in mind. If you want to be locked-in using GAE J for your application, then I recommend it as a cost efficient way to testing your application otherwise, look somewhere else.

I hope this was helpful and if there's mistake, feel free to get back to me and I make any corrections. Also, I am sure that I am missing some other points, add them to the comments sections.

P.S. here is a nice comparison from IBM

Cheers and Happy Coding.

The traveling robot problem

2012-04-25T17:54:00.000+01:00

I am currently between contracts and for first time in many years, I am requested to write some complex code from home so I took the liberties to share my result on my blog.

A startup based in London sent me this problem:

Problem:
A robot can be programmed to run 1, 2, 3, 5 and 10 kilometers and it takes 10, 5, 3, 2 and 1 minutes, respectively. Once it runs to programmed kilometers, it must be turned off for 2 minutes. After 2 minutes it can again be programmed to run for a further 1, 2, 3, 5 or 10 kilometers. How would you program this robot to go exactly 43 kilometers in minimum amount of time.

Please write a java program that solves the problem by finding the minimum time required to travel 43 kilometers. Your solution should print out the correct sequence of distances to be programmed into the robot: for example (10,5,10 and so on...)
When coding your solution please pay special attention to design, data structures, dynamic programming, faster execution time and creating generic solutions. (it should be fairly easy to change the parameters) Compute and provide Big O notation for main piece of your solution too.

END-OF-QUESTION

Before you look at my answers, it will be cool for you to try the exercise too so that we can share feedback in the comments section.

The question brought back memory of my Computer Sciences days, especially Data Structure and Algorithm lectures. At first glance, this looks like a simple problem but it goes deeper than that. Like artists, there are multiple ways to write code but once you look at someone else code, then it becomes difficult to be original ( IP patent law suit anyone?). Any experience developer can write code that would work perfectly for this problem, looking closely to what is being requested and you will notice that there is an extra requirement; dynamic programming. For those who needs more information about what dynamic programming is, I suggest the following wikipedia article. I had to spend sometimes reading the article to refresh my memories and off-I-went-hacking-away. Bear in mind that you could solve this problem by rewriting a recursive application but that is not a dynamic programming approach.

Anyway, here are my results ( the algorithm was inspired by a similar problem set).

I hope this helps someone in the future in any ways.

Happy Coding

You can download the source code this github https://github.com/armelnene/the-traveling-robot-problem

Primefaces Mobile - Weather App Example

2012-04-20T21:48:00.000+01:00

As a Java developer, I usually get requests of building mobile apps. I like building Java application; enterprise, web and mobile. The latest projects that I have been involved with make heavy use of JSF and Primefaces in particular.

I am quite confortable with JSF and therefore I decided to build a mobile application using JSF and Primefaces Mobile. Primefaces Mobile wraps JQuery as a JSF component so that you do not have to write any Javascript. This approach has huge benefits: JQuery is a well tested framework used by large companies such as Google and Microsoft.

I will make this post quite brief. I wanted to know how easy it would be to recreate the demo from the Primefaces Mobile labs page.

Here is a screenshot of the final application running in Firefox 11 (Windows 7 64 Bit).

This was a simple example taken directly from the website therefore I was expecting to a short exercise to make build and make it run. Well not so fast.

I created a Maven based Java EE 6 project using Netbeans IDE 7.1.1. After browsing Google for a bit, I spent a few minutes getting the right repository and dependency in place.

Once the dependency where in the place, I had to create the beans required for the actual JSF page to work. I found the Primefaces Mobile backing beans on Google Code. So now I had everything setup and running. At first glance, the application seemed to running fine and working. Then I tried the application on my iPad and the Android Emulator, and nothing was working. The user interface was displayed but the "get forecast" button was not making any Ajax calls.

So I started to debug the application everytime I had some spare time. I also noticed that, while running on a desktop browser, the application would be able to an Ajax called and updated the screen with the values (see screenshot above) but you wanted to make another to find out the temparature of let's say London, the nothing would actually unless you refresh the page and try again. OK, so it's not working as expected but the example on Primefaces Mobile labs worked fine on my iPad, emulator and desktop browser (IE 9 excluded).

So I ran the application using the NetBeans debugger and decided to look through Firebug. The first call goes through and stops at the breakpoint but subsquent calls do not even get to the managed bean. Firebug shows that the other code are being retrieve from the cache. I manually set all the HTTP headers so that it does not cache any content but this is still the same result NOT WORKING!

I have uploaded the code to GitHub click on the link to download it.

In conclusion, it's not plug-and-play to make the examples on Primefaces.org labs work. The documentation for the examples are quite poor and I hope that the good folks from Primefaces can take look at my code and tell us what I am doing wrong.

A part from that, Primefaces is a good JSF framework that I use on a daily basis on multiple projects therefore I cannot really put them down but I wished the mobile examples work and tell us what is needed to make it work.

Please share your experience in the comments below or advice on how to fix it. Feel free to download the code from GitHub and have a look to.

Happy coding :)

Parachutist Dilemma - a parallel programming problem

2012-04-19T16:42:00.000+01:00

Here is a little mental problem solving exercise for you to try to do in 30 seconds; that was the time I was allowed. So I had a telephone interview from a company today and this guy asked me this question utilizing a parrallel programming algorithm. Here are the details:

Two parachutist (A and B: A is in front of B) land on single line; one behind the other. Their parachutes also lands behind them respectively.
We have a set of instructions which are as follow (all instructions are executed, meaning A and B will executed instructions at the same time).

move forward 1 step
move back 1 step
if standing on a parachute go to (1 | 2)

Question:

We need our parachutist to somehow meet each other, what are the sets of instructions that can make that happen?

Please try to answer before reading my answer below

My answer:

execute 2
execute 3

Explanation:

When both parachutist move back 1 step, they will both be on their parachutes

when parachutist A stands on his parachute, he will move back 1 step therefore go to 1
when parachutist B lands on his parachute, he will move forward 1 step

Therefore our two parachutist will meet in B original position.

NB I had to change the original questions as the interviewer explanations led to the parachutist to never meet.

Cloud computing vs Open Source - Most CIO cannot tell the difference

2011-02-20T23:23:00.000+00:00

Few weeks ago, I posted a discussion here in a CIO forum on LinkedIn and frankly some of the comments are very interesting to say the least. Now I decided to bring some technical guys to the discussion table. My argument was and still is as follow

“I have been reading many threads about Cloud computing and I came to conclude that most CIO (maybe because of their business background) do not know the difference or real benefits of Cloud computing or Open Source software.

Hi, I am not expecting everyone to agree with me but here is my argument. Most CIOs are interested in Cloud computing because of cost. COST!!? That's very interesting because to my humble opinion, this only applies to IaaS (Infrastructure as a Service). If you do not want to spend money on hardware and be left with redundant system then IaaS is a good approach but that's about it. I work with Cloud computing (specifically SaaS) and this next one might hurt my business but it has to be said; SaaS is not cost saving when you comparing it Open Source alternatives that you can host on your IaaS system…. Read more here”

I am not sure that they understood the argument. I was not expecting anybody too agree with me and I wanted CIOs in the group to share their insight not sound like a copy of a Gartner report. Before making any conclusion, keep in mind that I am talking here about Software as a Service (SaaS), as defined here. In network architecture design, the external network is represented by the “cloud” symbol therefore I would have to agree Wikipedia’s definition of cloud. It is a software served over an external network regardless of a private network (private cloud) or public network (public cloud). YES – the internet was built on cloud technology. If you use any software over the internet; be it email, search, video or social network, the system is hosted by an external party which you do not have control over. The name cloud is just a marketing gimmick because hosted applications have been around for more than 20 years since Compuserve and Internet Relay Chat (IRC & BBS). So you see, “cloud” is definitely not new. Therefore software delivered over the internet such as Google apps and Salesforce can be also deployed using Open Source alternative. Cloud promises the following: scalability, peace of mind as it maintained by the providers and lower cost. CIO, I will tell you that your software, that it is deployed in the cloud or not will not be able to scale beyond the hardware and network capacity. I have another question for CIOs:

Do CIOs care or know where all their applications are running?

That question is usually left to the technical team to decide within the budgets of the project and any other company politics. The biggest failures of IT projects is due to the fact that the CIO (or “the business” as it is usually referred as) do not know what they are trying to achieve (lack or misunderstandings of requirements) therefore it goes past the delivery and budget.

So where does Open Source fits in, one might ask? Open Source software allows a business to take best of breed software and customised them to their own needs for free (as in free beer). If you do not want to support the Open Source software you can get support services but the software is hosted on your own system including your private cloud (Infrastructure as a Service). Let’s put it in to context; if a company requires a CRM system - they can download vTiger, customise it and either support it themselves or buy support. The same goes for an ERP system like Apache Ofbiz and MySQL for database. I am in favour of SaaS offering, after all I run a SaaS-based company. Nevertheless, CIOs do not confuse SaaS as being cheap, if you want something that has a low cost of ownership, then I suggest you go for an Open Source alternative if it exists. So why do I say that most CIO cannot tell the difference; SaaS is not cheap and does not have a lower cost of ownership when comparing it to Open Source alternatives. And by the way, you do not need to have an army of developers to build and customise your Open Source software. Do you really want the SaaS provider on a shared system to have access to your LDAP system? What happens when you want to integrate or port your existing data to your new fancy cloud based system? CIOs should not try to lie to me. I attended many meetings and conferences to know that most of them just want to please the board. What are the differences between cloud and virtualisation? What happens to your existing hardware once you moved your applications to cloud, have you thought of using them for virtualisation? So please do not make uneducated comments as “I don't know a single CIO who can't tell the difference between Cloud Computing & Open Source - maybe I move in different circles.” I am not talking about dictionary or Gartner definitions.

CIOs, after reading this and before replying you should go and sit down with your technical team and ask them the difference, only after that you can provide a good educated comment. If you want to be on the “cloud” train then you are free to do so but if SaaS is not part of cloud offerings, please feel free to enlighten me as I am always willing to learn something new.

Who should make the decisions to go cloud or not? You are the CIOs, you tell me. Is it the developer, the architect, the project manager or the almighty CIOs from his ivory tower? There is no trickery here, every fool can define a word – just do this; think of a word then Google using the following “what is ‘the word’” then voila your definition. If you need someone to school you up then drop me an email. Based on the comments on LinkedIn, all I can say is WOW. Leave the technical aspect to the technical guys and you can focus on playing golf.

Apple vs Geeks - is this the end of a love affair?

2010-04-12T00:20:00.001+01:00

This isn't news anymore but I wanted to wait for more light on the subject before I made any comments. Apple vs Adobe is actually an understatement. Actually, this is no surprise as Apple has always been a control freak. I mean, just take a look around you. How many PCs do you see running Apple MacOS which has not been branded by Apple Inc? Anybody trying to build a PC which runs their OS gets sued. Seriously, Microsoft may be evil but they have created an economy which in Steve Jobs world would have never existed. History is about to repeat itself; Apple Inc introduces a great idea/ products but would eventually become a niche player (well, only time will tell!!!).

I don't actually believe that real geeks use MacOS; Linux or if you work for a company this would MS Windows. Let's face it, Apple PCs (or Book, G5 or whatever) look very good on the receptionist desk and that's about it. Steve doesn't want you to innovate, he wants you to follow him to whatever **** he introduces (ipad anybody?).

Let me get back to the subject. Being software developer and I have invested time and money to build knowledge and skills. I cannot possibly learn every single programming language and therefore would welcome any tools that help me to cross-compile my code. I am not even a flash developer and seriously I don't care less about what they do. But the new Apple mobile SDK TOS affects more than just Adobe, take a look at the Open Source PhoneGap where you develop native mobile applications by programming using (the most popular language???) HTML + JavaScript. Thanks to the control freaks at Apple Inc, now this would not be possible. This is the same as saying that when you go on holiday to another country, you cannot use a translator to communicate and that you better go learn whatever language they speak in that country or STAY OUT!!!

First they didn't want Flash and Java to run on the iPhone. To be fair to them, those virtual apps kills the phone performance (so they say.). Therefore if you wanted to run any other program on their platform, you needed to translate it into a native app. EVEN THAT'S NOT LEGALLY POSSIBLE ANYMORE!!!! This is a big **** YOU to all the other programming languages. Can you imagine if Oracle said that Groovy, JRuby or Jython were not allowed to run on the JVM anymore? Or Microsoft saying Java app are not allowed to run on any MS platform through an intermediary (JVM)? Well, this is exactly what Apple Inc. is doing?

This is the best part; THERE IS NOTHING YOU OR I CAN DO ABOUT IT!!! Right now, the iPhone is the market leader for new generation of smartphones ( there are not the market leader in smartphone, that crown goes to Nokia) supported by its ecosystem of developer providing more than 150k apps in the App Store.

Look there is no way I am going to learn Objective-C unless I am getting paid for it. All we can do that is hope for Android, RIM and Windows Phone 7 to gain a large enough market share and we would see history repeat itself again. I am definitely not saying that you should boycott their products but if you are looking to build a company and make profit developing software for mobile devices, maybe you should think twice about venturing on the iPhone (off course, unless you know Objective-C).

Feel free to let me know your thoughts and I am sure that the Apple fanatics would not hold their thoughts on this one.

Facebook Patents RSS-like Technology - Is That a Joke?

2010-02-26T23:59:00.001+00:00

First thing first, Facebook did not patent RSS but their news feed on their site!!! I have to admit, this is a quite neat feature and what would be the point of a social (networking) site if you did not know what your entourage were upto. So Facebook included a friends activity log (news stream) to make it easy for people to find out. Thats a good feature but it's by no means an innovative technology. DZone, a developer social "networking" site provides a log about news that users have posted, commented on and rated. This is to show that there's nothing innovative about that feature but a convenience.

I have been very busy lately and did not have time to check my Google Reader but I did today. And voila! I see the news flash from Mashable - Facebook Secures Patent for News Feed. My first reaction, based on the title before I even read the article, was WOW!!! Facebook managed to patent RSS or ATOM or anything else on that scale. Then came the joke when I started to read the article specially the abstract of the patent:

“A method for displaying a news feed in a social network environment is described. The method includes generating news items regarding activities associated with a user of a social network environment and attaching an informational link associated with at least one of the activities, to at least one of the news items, as well as limiting access to the news items to a predetermined set of viewers and assigning an order to the news items. The method further may further include displaying the news items in the assigned order to at least one viewing user of the predetermined set of viewers and dynamically limiting the number of news items displayed.”

I don't know about you genius guys but I could not make much sense of the abstract but I did pickup a few keywords such as:

News Feed
Social Networking

That's it! Everything else is just pure jibberish. I can summarized their abstract as follow:

We know about RSS/ ATOM which are real news feed. RSS provides real-time (depending on how often you poll) updates; can that be classified as a site master activity update. It also includes links and media about the site or users generating the feeds. That's a cool feature and we copied the idea and implemented in our site, now we are the biggest community site in the western hemisphere so we are telling not to use any RSS or derivated technology when building a community site (social networking).

Let's face it, I feel sorry for the smaller community sites but I do not think that patent will affect Twitter or Google but watch out MySpace. I did not even know you could patent someone else already patented technology. This is not a new way of doing something but more like patenting the environment where the technology is applied. To be very geeky about this, Facebook News Feed are just a bunch of SQL queries displaying latest records on a User Interface. I did not know you could even dare trying to patent that. To be frank, I am not sure that they could have patented that in Europe. I am looking forward to seeing how to will now exercise their new POWER. What would happen to Ning, BuddyPress and Elg, those are software and platform used to create social site but what about Youtube and UStream. Anyway site operating within the US and implementing the patent features of Facebook will have to pay them. But outside the US territoty, those company would be free to operate. How to you control access to the Internet without censoring like China and Iran?

Anyway, this is just my opinion but you are welcomed to share yours with the rest of us. For now, why not read about RSS here and see exatly what I am talking about.

Will Oracle really make NetBeans the BEST IDE for Java

2010-01-31T11:58:00.001+00:00

Not yet!!! But according to their webcast they want to make NetBeans the best Java IDE. This is a statement not to be taken lightly. Oracle are now investing in three IDEs which are all established in different ways. JDeveloper is Oracle's IDE of choice, I am currently using it in my current project as I am developing for Oracle WebCenter. Oracle is also a strategic developer and board member of the Eclipse foundation. I do not understand what exactly is a strategic developer but I know that JDeveloper is their strategic IDE. Now, introducing NetBeans, Oracle has inherited NetBeans as part of Sun acquisitions (and a ton of products and services). Until now, Oracle has never made any plug-ins for the NetBeans platform but now they promise to make it the best IDE for Java, how will they make that happen?

I am a NetBeans evangelist and as the rest of the community, it was good news to actually know where it stands in the "Oracle" vision. So Oracle will never drop JDeveloper therefore the real fight is between Eclipse and NetBeans (as it always has been). In the first step of making NetBeans the best IDE, Oracle shold immediately start porting all their Eclipse plug-ins to NetBeans. Eclipse has strength in its alliance which also includes IBM (fierce rival to Oracle). Something has to give and Eclipse will not never close shop because Oracle has left them which is not as worse as lack of investment in the NetBeans community. It would also be a foolish decision to give NetBeans to the Apache Foundation (no disrespect here but it seems that public funds do not grow on trees).

There is something funny about Oracle "choice" statement. Did anybody tried to develop for WebCenter using NetBeans or Eclipse? That's a nightmare!!! There is no choice given, they literally forced us to use JDeveloper (Do you require any Oracle support?) in our company which is the largest in its field in Europe (I am avoiding to mention the company name but you should get it from my twitter stream).

Eclipse and NetBeans are direct competitors with Eclipse being the de facto IDE for Java, dropping Eclipse support would close doors to millions of developers and dropping NetBeans will disaster for many paying customers (Sun's partners). I am not sure what is going to happen in the NetBeans/ Eclipse and JDeveloper saga but something has to give.

Let me know what you think.

Google Analytics vs Woopra

2010-01-05T18:48:00.001+00:00

As you can see by reading my blog that I am not the type to compare and endorse products as I think the products owners surely must have a marketing department for that. Anyway, as a blogger, I like to be aware about how well my entries are doing. For awhile, I have been using Google Analytics (so as the top 100+ websites) as it is free and provides me with enough information on the state of my blog, corporate and client's sites. The problem with Google Analytics is that it gives me late news (same as BBC, CNN, TF1 and Sky News and every other news @ 10 channels). I have been working in Business Intelligence for past three years and let me tell you that old news is what caused the current economy downturn. You should not based future goals on outdated data (information), you need to have information at your fingertips on demand and this is where Woopra fits in.

Woopra is not free but they do provide a free version (basic) which is enough for most website. I believe if you need more than what they provide, I am sure that you could afford the fee. Anyway, what I really like with this software is its real-time ability and I think that it is something that the folks at Google can easily implement. So I decided to give it a try (last night) and see what was the "big" deal:

This is what we call real-time data, the fact that I could see where my visitors (to my blog) where coming from, what they were currently looking at (which article) and then where they went next in real-time. This is not a breathrough but the fact that it was available for free made it even better.
Visitors tagging; they are so many sites which is used to tag content on the internet (Digg, Delicious, DZone, Reddit and etc...) so that you can share and easily found what you were looking at before. Ok, Woopra tagging cannot be shared but you can tag a visitor so that you can "spy" on his activity: how many times did it visit the site, which articles is he more interested in, track the comments and more. This is an important feature in profiling your audiences.
Real-time chat; this is a good feature and I think that is indispensable for those running an E-Commerce site. Users are not forced to install any software, no plugins, nothing in order to chat with the site's owner (Any security concern, contact Woopra). I tried it on my blog and to be frank, I haven't been successful initiating a chat with visitors on my blog (they all refused the request :( ) but I can see it being successful on E-commerce site. Another aspect of this feature is the ability for the site owner to start the chat. So you can see what the user doing and which page he's currently looking at and therefore offer your help. I call it the virtual shop assistant. Users can also start a conversation so it's two-way.

I know I did not go into details about Google Analytics but you can register for free and see for yourself. Google Analytics seems out of date, Twitter is real-time, Google and Bing are implementing real-time features; don't you think that you need to be able to track your website performance in realtime too.

Another thing, how come Woopra shows some countries as EU when UK, France, Italy, Greece, Germany, Ireland, Spain, Belgium and Poland are shown as their respective countries. What does the European Union means in that case? Also, it is a shame that Woopra is not a web based application (the current web version is so weak you might aswell use Google Analytics ;).

Anyway, if you have use it before or have a different view, feel free to leave your comments.

N265Q9AKMDFZ

Happy New Year Techies!!!

2009-12-31T15:00:00.001+00:00

I just wanted to say happy new year to the community, without you guys, we would still be living in the dark ages. Here are a few that I am looking forward to in the new year:

Oracle Sun merger:- The untold future of NetBeans, MySQL, Swing and every open source that Sun has been working on in the past. What would happen to Sun Open Source (SOS!!!) Movement.
Looking forward to JavaFX1.3 release, Authoring tool and improved Composer plugin.
Android uprise against iPhone (not because I can easily write Android based application) and hopefully with Visual XML builder to build Android UI. JavaFX running on Android, anybody????
Looking forward to seeing real-world JavaFX application and possibly a showcase linked to the JavaFX.com site.
Java EE 6 support in Google App Engine
Google Wave open to the public and how well it does against Twitter and Facebook
Java Store open to Europe (well this is where I am based and want to sell applications not just provide free).
My new video blog ( well that's me trying to be more Armel 2.0 - the sequel) to better engage with the community.
looking forward to networking with fellow developers and techies.
looking forward to the new buzzwords (old technology, new name)

Well, 2009 is was good year for Java and tech scene. I hope that you all enjoyed the year as much as I did. Why not share what wish list with me (comment box). Subscribe to follow my blog as I will try to bring more interesting articles in the new year.

Happy 2010 New Year, wish you all; success, properity, fame (yup) and fortune. Don't forget, if you need a server-side, UI, Android or JavaFx developer, just mail me.

Dummies Guide to GWT and OpenID - with example code

2009-12-23T00:50:00.001+00:00

Once (?) in awhile, we developers browse the web for some quick and dirty solutions to some of our coding problems (sometimes not even problems, right?). This is another of those days, everything was going fine on this project I am currently working on. This is not a social networking sites but a site for sharing and comunicating (how I missed the days of the chatrooms) similar in many aspect to Digg and DZone but which also allows users to communicate in realtime and leave comments. So the back-end was sort of completed but so now time to focus on the user registration.

The sites needed to be open and avoid repeatitive tasks such registering users. The way forward was to implement some authentication to allow users to register with some sort of universally accessible ID (or sort of). Facebook Connect, OpenSocial and Google Accounts have their advantages but to me personally; the disadvantages outweight the pros. These are some of the disadvantages of those platform:

Facebook is a popular site hence is its platform, Facebook Connect, in mostly Europe and America but not everybody in Europe and America have a Facebook account.
OpenSocial, when it comes to single logon, has more advantages than Facebook. I supposed that if we do take into consideration all the OpenSocial sites, we might have apossible larger user based than supporting Facebook alone. Even that was too limited.
Google Accounts, one thing Google does not advertise its user base. I could be wrong but do anybody actually knows how many people uses the various Google services (excluding search). Google has the same disadvantages as Facebook.

Enter OpenID. OpenID has been around longer than most Internet-based decentralised authentication platform. The beauty of this platform is that it also supported by most (if not all) large site on the Internet. From AOL, BBC, CNN to YAHOO and ZIMBRA, as I said most sites (based on the alphabet) support OpenID, check the OpenID Directory. It was recently announced that OpenID has reached 1Billion enabled accounts (read here). Based on those figures, I decided that OpenID for now was the authentication choice for this application. I will not be discussing any security issues in this entry, there are plenty of resources available on the Internet for that.

OK, so I am a Java developer and I am developing the front-end using GWT and Java on the server-side. I searched on the Internet for solutions on how to implement the authentication as GWT RPC are different to normal servlet call. I spent more time reading about the OpenID specification and implementations examples on the Internet. I have to admit that some of tutorials that I found on the Internet were somehow confusing and not helpful at all. Therefore I decided to write my own Dummies Guide to GWT and OpenID.

Dummies Guide to GWT and OpenID

First of all, it is important to know how OpenID works (please check the OpenID site for more info).

In a nutshell, OpenID allows you to authenticate with website (supporting the standard) with just an URL and voila. Your URL has to be hosted by an OpenID provider in order for it to work. For example, my blog is hosted by Google on blogspot.com. Google supports OpenID authentication therefore if a reader wanted to leave a comment on my blog, he does not have to have a Google Account as he can log-in with his Yahoo or AOL or Facebook or Wordpress or any other OpenID provider sites, and then leave a comment, that simple.

GWT to OpenID and back

There are two ways to authenticate a user with an OpenID provider and GWT supports both. When authenticating a user, the relaying site (the site the user is trying to access) redirects to user to the OpenID provider (i.e. Google) login page.

The problem will be with the GWT RPC mechanism. GWT RPC calls are asynchronous and you cannot make any redirections. Therefore we need a way to execute the redirection from the client side, here is the code (I use OpenID for Java to handle the openID discovery from the RPC servlet), I then used USER object (just a simple POJO which only stores the redirection URL and the parameters) to be sent back and forth between the front-end and back-end.

#######################################################

public User authenticateOpenId(String provider_url) {

    try {

        ConsumerManager manager = new ConsumerManager();

        // This is the URL where the OpenID provider will redirect the user

        // once logged in.

        String returnToUrl = "http://localhost:8084/GwtOpenId";

       

        // the provider URL will be used to identify the user OpenID provider

        List discoveries = manager.discover(provider_url);

        DiscoveryInformation discovered = manager.associate(discoveries);

        // OpenID 4 Java needs to  have an HttpServletRequest object, GWT sevlet

have

        // convenient methods to retrieve the HttpServletRequest object

and manipulate its

        // parameters

        this.getThreadLocalRequest().setAttribute("openid-disc", discovered);

        this.getThreadLocalRequest().setAttribute("openid.mode", "popup");

       

        AuthRequest authReq = manager.authenticate(discovered, returnToUrl);

        FetchRequest fetch = FetchRequest.createFetchRequest();

        // I want to exchange the following attributes from the OpenID provider

        // remember that teh attribute will be returned only if it exits

        fetch.addAttribute("email","http://schema.openid.net/contact/email",true);

        authReq.addExtension(fetch);

        // Simple POJO to persist the data

        User user = new User();

        // In a normal servlet development, the following statement would make

        // a redirect call but this would not work when using GWT RPC

        if(!discovered.isVersion2()){

            user.setRedirect(authReq.getDestinationUrl(true));

            // fakes the redirect by sending the POJO with the required parameters

            // to make a client-side redirect

            return user;

            } else{

            user.setParams(authReq.getParameterMap());

            user.setRedirect(authReq.getDestinationUrl(false));

            // fakes the redirect by sending the POJO with the required parameters

            // to make a client-side redirect

            return user;

        }

        } catch (MessageException ex) {

        Logger.getLogger(GWTServiceImpl.class.getName()).log(Level.SEVERE, null, ex);

        return null;

        } catch (DiscoveryException ex) {

        Logger.getLogger(GWTServiceImpl.class.getName()).log(Level.SEVERE, null, ex);

        return null;

        } catch (ConsumerException ex) {

        Logger.getLogger(GWTServiceImpl.class.getName()).log(Level.SEVERE, null, ex);

        return null;

    }

}

#############################################################

The above codes will format the request in order for the front-end to execute a redirect and allow the user to authenticate with his OpenID provider. Now here is the front-end code which executes the authentication and reads the data back.

#################################################################

 // This is an extract from the openidEntryPoint.java



// This is where the magic happens - This code is only usefull when the OpenID

provider

// redirects the user back to your site - please visit openid.net for valid

parameters.

// The "if" statement checks to make sure that it is a valid response from

the OpenID

// provider - You can do anything you want with the results here such

as verifying the

// response with the server-side code

if(Window.Location.getParameter("openid.rpnonce") != null){

    String s = Window.Location.getParameter("openid.mode");

    // executes this if the user cancels the authentication process and the OpenID

    // providers returns to the your site

    if(s.equals("cancel")){

        sign.setText("Server returned openid.mode=cancel");

        openIdUrl.setText("You need to Accept not Cancel");

    }

    // Here, I am assuming that everything is fine and that the user has

been sucessfully logged in

    else{

        sign.setText("You have successfully signed in");

        vp.setVisible(false);

    }

   

}

RootPanel.get().add(contentPanel);



}



private class UserAsyncCallback implements AsyncCallback<User> {

   

    public void onFailure(Throwable caught) {

        Window.alert(caught.getLocalizedMessage());

    }

   

    public void onSuccess(User result) {

        if (result != null) {

            //                    Window.open(result.getRedirect(), "_blank",

 "height=200,width=400,left=100," +

            //                            "top=100,resizable=no,scrollbars=no,

toolbar=no,status=yes");

            // this the most important line in order to make the authentication.

 Here, I am redirecting the user

            // from the client side to the OpenID provider URL with the discovery

 data generated from the

            // RPC call to the servlet.

            Window.Location.assign(result.getRedirect());

            } else {

            Window.alert("Ooops!!! Couldn't find your provider");

        }

    }

}

#######################################################################################################

I have attached the full NetBeans project with dependencies. The code is provided as-is and use at your own risk ;). Here is a screenshot of the working application:

Step1: Authenticate on the site (enter the URL)

Step2: Redirect to OpenID provider (Google is my provider :) ), authenticate with your provider

Step3: Allow the application to access your OpenID details and redirect back to the original site

Step4: final step, check the parameters from the provider and proceeds accordingly

Take a look at the URL in each of the above step to see the OpenID data. OK, so my code actually works (yuppie), now you know that it is possible form GWT to OpenID and it's not as complicated as many other sites are trying to show. The code is just for authentication but once authenticated, you can retrieve any parameters that you need. In this example, the query is sent through the browser URL (GET) but you can change it to be encoded in a form submit action. Iwrote some of the code to allow the user to authenticate via a popup window, the code is not complete and maybe someone else might want to have a go. My only problem is getting the redirect back to the original window but apart from that it works.

Download the source code

I hope you guys enjoyed it and Happy Coding.

Firefox IE8 Chrome Opera Safari - best browser for Java?

2009-12-17T22:54:00.001+00:00

Initially, I was not going to write up about browsers until I had to download all the known web browsers to test a project front-end. I already had Firefox, IE8 and Chrome installed therefore all I had to do was to download and install Opera and Safari. There might be other browsers outthere but this what I am working with for the moment. So what's the problem? You might ask. Well, I decided to see how simple it would be for a normal user without much computer experience to view a Java-enabled application embedded in a web app or simple HTML page.

Before I go on about what I think, first let me give you an idea of the PC I am currently using:

Dell PC with a dual 22" screen (customised version of Inspiron 530)
4Gb of RAM
Intel Pentium Dual 1.80GHz (looking to upgrade this after xmas)
Windows Vista Ultimate (how rubbish is that?)
Java RE version 1.6.0_17
and a few other hardware and software.

So as you can see, my personal PC is quite simple in settings. Ok, for some reasons, I decided to launch all the browsers at once and load the JavaFx homepage (http://www.javafx.com) and all I wanted to do was to compare which one loaded the fastest. Here are my findings, in brief:

Firefox
This is probably the best browser outthere but lately it has been crashing on me. And btw, it does not cope well with Google Wave. Anyway, here is what Firefox gave (see screenshot):

As you can see on the above screenshot, the JavaFx homepage loads in its entirety. To be quite honest, the page was loaded without any fuss but bear in mind, I already had Java installed. The "Demos & Samples" box in blue requires Java to be installed as it a JavaFx applet. As I already have the JRE installed, the browser did not ask me for any plug-ins ( I could be wrong feel free to share your experience).

Internet Explorer 8

I am not a great fan of IE in general, but for this simple test; IE8 loaded the page faster than Firefox. Also, it did not ask me to install a specific plugin in order to load the JavaFx portion of the page. IE handles Google Wave much better than Firefox, so this is the browser I use to interact on the Wave.

Google Chrome

Google Chrome, like Firefox and IE, did not cause any problem and it was the fastest at loading the JavaFx page including the Java applet. So far, it is the best performing browser but it is not my personal choice so I rarely use it. Also, I like the fact that Chrome scales the page to fit nicely in its window.

Opera

As you can see in the above screenshot, it seems as if the applet was not fully loaded. Indeed, Opera has loaded some portions of the JavaFx applet but I do not know what is happening here now. This page has been the same since I loaded the browser until now. Again, Opera did not ask me for any plugins to be installed nor did it ask for anything else. I suppose this is not an issue from Sun ;) but something that the Opera team should look at. I know Opera can find the JRE on my system (see screenshot below) but it seems not to be a big fan of Java. In the screenshot below, you can see the JavaFx splash screen but this is all you will see and nothing else. Off-subject, I really like the user interface and I might start using it for a bit longer just to see if it can convert me.

Safari

Ha! what the hell happened here. I have my JRE installed, the previous four browsers did not ask me to install Java (again!!!). The demo applet did not even load, this is Safari running on Windows machine not Mac. So I did what it has asked me for the thousandth time. I clicked on the provided link, takes me to the Java RE download page and I did all the required and look!!!!

Java!!!!! This is the confirmation that I already have the software installed so what's the big idea? The Apple team are not doing a good job to support Java but I can browse to Youtube and watch video without installing Flash player (it recognises that it's already installed). Well I do not think I will be using this browser again anytime soon.

Final words
When it comes to browser to support, you have to test your application on multiple browser ( and OS platform too) to make sure it will not affect the expected user experience that we are so accustomed to.

I do not know why Opera and Safari have issues with the JavaFx site, it could be anything from the site designers to the respective browsers' team. I believe that Java is probably the best Cloud computing platform currently outthere (when it comes to high performance applications) and that browser providers should make sure that at least they do support the platform.

I have a question:

Who should be to blame if Java or any other plugins such Flash is not supported in the browser? The plug-ins developer or the browser developer?

I believe the plug-ins developer should be to blame as they can be a million plug-ins on the net and you cannot possibly cater for all of them. But on the other hand, as in Java 's case, the plug-in is supported without any major issues by the top 3 which accounts for 93.9% percent of the market (based on November 2009), should you as the plugin developer care? I say yes at least for the Mac users' sake. We can overlook opera for now but I am sure that all the Apple fans still use Safari as their prefered browser. Do not try to leave them out, they brought us the iPhone.

Anyway, based on my simple test, I think the best browser for Java applet is (in order of best performance):

Google Chrome
Internet Explorer (make sure you do support this browser as a priority)
Mozilla Firefox (this should also be supported right after IE)

The other two browsers are not included as they did not even successfully launch the applet. I am not saying that you guys are going to have the exact same issue as me and therefore looking forward to your comments.

JavaFX Composer RAD Tool - First Review

2009-12-16T12:12:00.001+00:00

Just over a week ago, Sun announced a RAD tool for JavaFx built on the Matisse framework, I believe. I was very critical of JavaFx for its lack of tool for building UI and I think this is a step in the right direction. The tool was made available through the NetBeans Update Center on the 14/12/2009. OK, so I have installed the plugin and here are my views; not just on the tools but also on other stuff I think would benefit the JavaFx community:

When comparing something, it only makes sense when we use a benchmark; here my benchmark will be Adobe Flash Builder (formerly known as Flex Builder). Over the years, Adobe had made it easy for the designer to build impressive user interface with minimal coding. Sun, in the other hand, made it easier for developers to build application, yes I am aware of some nice UI in Java but they still do not compare to the eye candy of Flash/ Flex.

I am going to look at the tools; Flash Builder and JavaFx Composer plugin, from a developer perspective.

Components:
A key feature of RAD tools is the amount of components they make available to developers without having to write too many codes. I understand the plugin is at a "preview" stage, whatever that mean (alpha?), but there are alot of missing components; as an example, this release version was meant to be a "preview" of what to look forward to but I cannot drag a "combox" from the components palette into my form, no data grid, no chart, no menu bar, no date components and can't even draw a rectangle which is possible but only through coding. I hope the JavaFx team add all the components available in JavaFx plugin to the Composer.

NetBeans JavaFx UI Composer

Adobe Flex/ Flash Builder

Round-trip code update
One thing I dislike with Java IDE's (or most of them) is the inability to change the generated code without requiring you to write more code. The Adobe team actually made a good job in giving more freedom to the developer. In Flash Builder, you can design your UI through drag-n-drop but also customise it directly through the XML file (MXML). This feature was not available in NetBeans Matisse, I could be wrong, but and again Matisse was not really used in large project (no comments, thanks), at least not were I worked. Why all the fuss, you might ask? Try to create a simple interface and add a "rectangle" object to it which you will use as a toolbar and tell me how simple that was.

The coding style in JavaFx is very similar to Flex/ Flash ActionScript (so why the "V"oid instead of void, but that's another issue) and very easy to learn. So far, I found it easier to actually code the canvas then using the Composer plugin. Another thing, when inspecting object properties; not all properties are available, for example: the gradient properties are not available, which will require you to write more lines of code.

UI Preview Panel
Again, a feature which Adobe Flex/ Flash builder excels at (I feel like I am starting to sound like an Adobe salesman) is the synchronize preview of codes. This is not a due to JavaFx Composer plugin but this seems as a bug as sometimes, you might have to restart the IDE in order for the UI Preview panel to start working again. Hopefully the introduction of the RAD (or not so much RAD) tool, will fix this issue.

Conclusion
You might feel that I was on a JavaFx bashing quest but this is not the case. JavaFx might not have a large components set (well what about all the SWING components available which you can use? You might not be able to "skin" them to your application look and feel but they still availabe to you) but I still think it has a good future. If you take a look at the screenshot below which was built with Adobe Flex, it took me less than five (5) minutes to build. Now time to synchronize your watches and tell me how fast it will take you to build the same interface using JavaFx Composer plugin. It will probably take me less than five (5) minutes if I was designing it with Matisse. My point is; a RAD tool is supposed to promote productivity and YES!!! I have realized this is a "PREVIEW" release but can you actually use it? I know I will still be coding JavaFx for the foreseable future and I would love it to succeed. If you are going to call a tool "a RAD tool for building Form-based JavaFx UI" then I suggest that you provide most of the form components.

Should JavaFx UI RAD tools be based on XML like Android and Adobe Flash Builder?

5 components I like to have on JavaFx

Menu and tool bar (come on guys, this was there in previous release)
Grids (even just a simple table will do. For now I use JTable)
Date picker, Rich Text and Navigator components
HTML panel (something that display HTML and can also be used as iFrames)
Panels similar to JInternalFrame (this will be useful in portlet-like applications)

I know it's not really 5 things but they will make a difference. Thanks for reading and tell me what you think about JavaFx and its UI Composer.

JavaFX vs GWT - A developer dilemma in building a RIA application

2009-12-02T11:59:00.001+00:00

For the past few days, I have been working on a simple proof of concept; a web based stock streaming application with a community theme to it. I set myself to building the backend first. As you can guess, I am developing in Java and here are some of the requirements that I have to meet, at least:

The application shall allow the user to query a stock (quotes) either using the stock ticker or the company name and display the following: stream stock price with basic info, stock historical price; tabular view and financial chart analysis.
The ability to rate, comments and share stock information.
provide a real-time chat features for users of the application to interact together.
single sign-on integration with OpenSocial, OpenID and Facebook connect.

The above are some basic features in order for people to actually use the site. As you can see, they are mostly front-end requirements. Here is another important requirement:

The application will be hosted on a Cloud-based service, the application shall keep to a minimum the load on the server as CPU and memory use will be charged per request.

I believe that last requirement was the key decision factor. I needed something that will make use of the user resources instead of stressing the server. Something that would execute on the browser, client-side. Being a Java developer, it is only normal for me to look for something that I can grasp easily and freely available on the market. I looked at a few RIA frameworks and tools but I needed something that can integrate within my development environment without the need switch between IDEs.

I had a look at JavaFX and GWT. Having worked on GWT in the past, I found that I could implement most of the requirements. GWT has a rich api and components sets (widgets). I could use the Google search, map, visualization... to build the front-end and implement the client side code.

It took me about two days to implement the basic front-end and back-end features; mostly concentrating requirements 1 and the last. But then I hit a brick wall, something that all Javascript developers already knew but I just came across:

The application is a mashup of different services from around the net. Some of those services were available for free and others were not available at all but I will still be connecting and parsing the relevant data to displayed on the screen. As requirement "1" was working fine, there was a disaster about to happen. Here is the basic architecture:

when a user queries a stock, the data is sent to the back-end services.
the back-end services queries a third-party web service which I do not have control over. ZERO.
when the third-party web service returns, the data is then parsed into a format understood by the front-end and sent to the front-end services to display.

The services was working fine until I had about 12 stocks displayed on my screen for testing purpose. The data was being streamed with a 1 minute interval. After about 30minutes later (and 360 requests = 12 *1 pm * 36m), I have noticed that the stock data was not being updated anymore and some screens were blank. No errors were logged and some other queries were still working such as the historical data. As I could not see the errrors from my debugging session, I therefore decided to query the third-party service manually through Firefox and IE and VOILA! The service provider had put a temporary ban on my IP address for making too many requests, it says. This was a major blocker as it seems that I could not access my primary data anymore. This left me questioning the feasibility of the project.

If a single user gets banned for tracking 12 stocks, what would happen when 100 or more users start watching their stocks?

OK, so I did not give up, at least not without a fight!

I looked at the problem from another perspective, the third-party view. How do they track my request as I do not have to register or sign-in? The obvious answer is my IP. So I thought, instead of making the request to their services from the back-end, I should send the request directly from the front-end. If each users make the same request, the third-party services would register different IP addresses for each one of them. Correct indeed but one big problem:

The front-end code is written in GWT which is then compiled to Javascript. Being a Java developer, I was not aware of the limitation of using AJAX in a mashup such as mine. After making the necessary changes, I then run the app just to get hit with another blocker - SOP (Javascript Same Origin Policy). What's in a monkey name was this all about? So after researching it - Google and Wikipedia, I knew what I was dealing with and I am sure all Javascript developers already knew that but not me :(. I never give up, it is not part of my nature, therefore I decided to look for ways around it and came across a tutorial on GWT site that shows you how to implement a workaround. Did the tutorial, implemented some basic methods and that worked fine but implementing the third-party service, did not work :(. The problem was simple, the workaround needs to be in JSON format which I coded but the third-party service needs to be able to send the request back to a "callback" method. This is it!!! Remember that I mentioned that I had ZERO control over the thrid-party service and therefore I could not implement this feature.

Remember, I never give up; when there's a will there's a way. So I hit a major blocker due to the technology, in this case it was Javascript. I know that if I was working on a pure java front-end GUI, this would have never occurred and Flex allows you to query other third-party services but I have no knowledge of ActionScript. I decided to go for JavaFX just because it is Java and allows me to connect and get my data (actually it's not mine). My options were Webstart, Applet and JavaFX. First of all, it has to be part of the site so my real options were Applet and JavaFX. Applet do not have a rich set of tools such as Map, Chart and everything else that JavaFX provides and I do not want to waste time coding them.

So same adventure but different episode; I am currently looking to implement the JavaFX UI so stay tune for the sequel and screenshot. I could have never anticipated those problems with the third-party and then Javascript but believe me it was a good experience and my Javascript just went +1.

GWT Visualization Example - Annotated Time Line Chart Tutorial

2009-11-22T12:29:00.001+00:00

If you are a server-side Java developer such as myself, I am sure you want an easy way to create AJAX based application but pure JavaScript (ECMAScript) is not your strong skills. The beauty of JavaScript is that it runs on any web server, no need for a servlet container or anything of that sort.

So, I have been writing AJAX based UI using the GWT framework for the past year and half. I am still not a web developer but I understand enough CSS to spice up my site. Anyway, I am writing this tutorial because I think that the GWT Visualization team, even tough they did a good job, over complicates the creation of charts in their tutorial. I was trying to create an Annotated Time Line chart so I looked up my previous code I wrote for the GWT Portlet JSR-168, see here. I also decided to run a search (don't be evil, Google is your friend) to try to find different examples and hopefully some nice custom charts. First of all, based on the search result; I thought that some of examples, including Google's own, were confusing and over complex. Most examples uses AJAXLoader to load the Visualization API, but you shouldn't have to use this. This is the simplest way to create a GWT Annotated Time Line Chart.

1. Using your favorite development tool with support for GWT framework, create a new Java web application, you can export the compiled JavaScript later to a non-based web application, if you use an IDEW you should code-completion support.

2. Make you sure you have included the GWT Visualization API module in your classpath.

3. In your project source code root directory, you should <AppNamexxxx>.gwt.xml file. make sure to add teh following line to it in order to make the module available to your application.

<inherits name="com.google.gwt.visualization.Visualization"/>

4. In order to keep this tutorial short and straight to point, I have included the chart creation code in my <AppNamexxxx>EntryPoint.java class


public class MainEntryPoint implements EntryPoint
{
    
    
    /**
    * Creates a new instance of MainEntryPoint
    */
    public MainEntryPoint()
    {
        
        
    }
    
    
    
    /**
    * The entry point method, called automatically by loading a module
    * that declares an implementing class as an entry-point
    */
    public void onModuleLoad()
    {
        
        Runnable onLoadCallback = new Runnable()
        {
            
            
            public void run()
            {
                
                DataTable data = createHistoryTable(result);
                AnnotatedTimeLine.Options options = AnnotatedTimeLine.Options.create();
                options.setDisplayAnnotations(true);
                options.setDisplayZoomButtons(true);
                options.setScaleType(AnnotatedTimeLine.ScaleType.ALLFIXED);
                options.setLegendPosition(AnnotatedTimeLine.AnnotatedLegendPosition.SAME_ROW);
                AnnotatedTimeLine atl = new AnnotatedTimeLine(data, options, "600px", "200px");
                RootPanel.get().add(atl);
                
            }
            
            
            
        }       
        ;
        VisualizationUtils.loadVisualizationApi(onLoadCallback, AnnotatedTimeLine.PACKAGE);
        
    }
    
    
    
    // This method will create the Data used by the chart
    private DataTable createHistoryTable()
    {
        
        DataTable data = DataTable.create();
        data.addColumn(AbstractDataTable.ColumnType.DATE, "Date");
        data.addColumn(AbstractDataTable.ColumnType.NUMBER, "Price");
        data.addColumn(AbstractDataTable.ColumnType.NUMBER, "Low");
        data.addColumn(AbstractDataTable.ColumnType.NUMBER, "High");
        data.addRows(5);
                   
            DateTimeFormat dtf = DateTimeFormat.getFormat("yyyy-MM-dd");
            data.setValue(0, 0, dtf.parse("2009-11-21"));
            data.setValue(0, 1, 100);
            data.setValue(0, 2, 120);
            data.setValue(0, 3, 90);


            data.setValue(1, 0, dtf.parse("2009-11-22"));
            data.setValue(1, 1, 90);
            data.setValue(1, 2, 110);
            data.setValue(1, 3, 100);


            data.setValue(2, 0, dtf.parse("2009-11-23"));
            data.setValue(2, 1, 100);
            data.setValue(2, 2, 180);
            data.setValue(2, 3, 80);


            data.setValue(3, 0, dtf.parse("2009-11-20"));
            data.setValue(3, 1, 130);
            data.setValue(3, 2, 100);
            data.setValue(3, 3, 130);


            data.setValue(4, 0, dtf.parse("2009-11-19"));
            data.setValue(4, 1, 130);
            data.setValue(4, 2, 170);
            data.setValue(4, 3, 150);

            
        }
        return data;   
    }   
}

Don't forget to import all the necessary classes and voila!

I hope this will save you some time in creating your charts.

Develop High Transaction Web Applications with Java MySQL & NetBeans

2009-11-12T20:40:00.000+00:00

This entry is a brief tutorial on how to develop high transactional web application. I will be looking at how to develop a high transactional application while maintaining data integrityFor the purpose of keeping this entry simple, I will be using some RAD tools, the NetBeans IDE, to generate most of the code.

In order to follow the tutorial, you will need the following:

NetBeans IDE 6.7+
Java JDK 1.6+ (my version is 1.6.0_17)
MySQL 5.1+
MySQL Sakila database
Apache Tomcat 6.x
An understanding of JPA transactions and ReSTful web services

This is my definition of High Transaction Applications:

A high transaction application is one that can serve multiple simultaneous requests from clients and keep them secure from each other. The application has only two purposes: read or write data to/from a repository either JMS or DB. The transactions have to meet the ACID criteria in order to be deployed in the real world.

Ok, the above is my definition and you are free to redefine it. I am going to build a web application that will be an n-tier application:

Database back-end (MySQL)
Entity classes
ReSTful Web Services to allow other developers to integrate the application with theirs
A web based front-end

For the simplicity of the article, I will not implement any security such as user or application level security (authentication and database table privileges). This tutorial is mostly geared toward the newbies but I am sure that more advanced developers would benefit too.

Let's get coding.

Make sure that you have loaded the MySQL Sakila database into your MySQL database. You can download the Sakila database from the NetBeans plugin centre (see here).
Create a new web application and name whatever you like. I have named mine "WebApplication". I will now refer the application as WebApplication. Make sure to choose Tomcat as your deployment server

Add the MySQL driver "mysql-connector-java-5.0.7-bin" to the WebApplication libraries. NetBeans will work and connect to the DB even without the driver but once you deploy your application to a server, the application will not be able to connect to the DB and throw a ClassNotFoundException com.mysql.jdbc.driver.
We are going to develop the back-end first. NetBeans makes it very easy for us to create Entity classes from database. I would recommend newbies to learn how to manually create Entity classes and configure the persistence.xml file. This tutorial makes use of JPA but one should ignore the drawbacks of JPA/ ORM frameworks.
Right click on the project name and choose Entity Classes from Database... On the next screen choose the "filmActor" table and click on the Add button. Make sure that "Include Related Tables" box is checked underneath the Selected Tables panel. The screenshot does not show the "filmActor" table as I have previously generated the Entity class but I am sure you get the idea.
If you are required to create a Persistence Unit, click on the persistence button -> you can accept the default name -> choose your Persistence Library -> choose your database connection -> choose "None" for Table Generation Strategy -> click Finish

After choosing the table to generate the Entity from click next -> fill Package name -> tick Generate Name Query... -> click Finish

Right click on the project name "WebApplication" -> RESTful Web Services from Entity Classes... -> Choose the Entity Classes that you would like to generate the WS for and click add or add all -> click next -> on the following screen, accept the default values and click Finish

You can go ahead and test your ReSTFul Web Services by right clicking on the name of the application and click on Test RESTful Web Services

The previous step will launch your web browser within which you can test your web services (click on the node on the left and see the queries on the right)

Back in the NetBeans IDE; right click on the project name "WebApplication" -> click on JSF Pages from Entity Classes... (The JSF pages will not use the web services as there are packaged together with the Entity Classes. This will improve performances and still allow external applications to integrate). Choose the Entity Classes that you would like to generate the pages for -> click next. On the final screen, fill in the package name for the JPA Controller and JSF Classes -> click finish

NetBeans will generate the necessary files to create a CRUD application with a user interface. I suggest that you familiarize yourself with the generated code.

Expand the Configuration folder under your project name "WebApplication" and the web.xml file. At around line 38, change the content of the welcome-file-list to look as follow

<welcome-file-list>

<welcome-file>faces/welcomeJSF.jsp</welcome-file>

</welcome-file-list>

This will make the generate JSF page to be the landing page for the application when requested. Make sure that you do have a "welcomeJSF.jsp" file before making the change.

Right click on the application name -> Run. The application should load in your web browser. Now go on, play around with the application. And why not create a client to send request to the web services? (not today)

You can load test your application by using Apache JMeter. It is easy to run and configure. If you want to take a look at how JPA implements the ACID features, browse to your controller classes. Here is a short introduction to JPA transaction. Hope you enjoyed and if you need any clarification, just leave me a comment and try to get back to you ASAP (if time permits).

Develop Your Own Google with Apache Lucene (Java Nutch Solr)

2009-11-09T19:05:00.000+00:00

Apache Lucene is Open Source API that allows a Java developer (.Net libraries available) to write indexing and full-text search capable applications. I have been writing applications based on Lucene for the last 3 years and some of the applications have been deployed at large corporations. I know there are other libraries available to developers who wish to write indexing engine but this blog will solely focus on Apache Lucene. I will not compare it to other API.

Lucene is a very mature API and can be found in NetBeans IDE, Liferay, JackRabbit among others. IBM has written a very good document about the Lucene architecture, therefore I will not delve into it here.

Lucene, alone, is pretty much useless as any other API. Let's now introduce Nutch. Nutch is a web crawler built-on top of Lucene to provide file crawling capabilities. Nutch was designed to handle large amount of data from the internet (http). Due to its plugin architecture, it was later extended to provide local network crawling such as FTP, databases and Microsoft Windows Shares (I am the author of the protocol-smb plugin and co-author of the index-extra plugin found on the Nutch site). We had extended Nutch and turned it into an Enterprise Search application but most of the source codes were locked behind closed doors (company politics).Anyway, Nutch has evolved to become but still very complex in its inner working. The initial Nutch was developed to process data in a batch but there are ways to turn it real-time but that's for another day. Ok, so Nutch is good for crawling and indexing of data but it does not handle search directly. There is a web application available with Nutch but it is quite poor so let's now introduce the Solr.

Solr is a powerful web-based search server built-on top of Lucene. The application was developed by CNET Networks and donated to the Apache Foundation. I believe, not too sure so I might need some references here, Solr was powering the search feature on their site but it is definitely used internally by the company. Late 2009, Lucid Imaginations receives $7.5 million in funding to provide commercial services built around Solr (and Lucene possibly?). Here is a very good presentation about Solr. Solr is a very good indexing engine. The keyword here is "indexing engine". It does not have any support for crawling data therefore requiring the developer to create applications that will feed it the data to index. I do believe that it is a good feature of the application as it gives the ability to integrate with various systems as long as they can post data over HTTP.

Nutch is a good crawler but it does not provide an enterprise-grade search interface to its data. Solr, in the other hand, is powerful indexer and has an enterprise grade search interface but it does not know how to gather data in its own. I am sure by now it has become obvious how we can integrate them both together.

We want Nutch to gather the data, by-pass its indexing cycle and feed the data directly to Solr. Lucid Imagination has a good tutorial about it here.

After reading the tutorial from Lucid Imagination, you will notice that Nutch is run by executing some bash files. This is something I strongly disagree with. If Nutch is based on Java (an OS independent language), why do we need to execute some UNIX/LINUX shell script. Also, the fact we need to install CygWin on MS Windows platform to be able to run is a big negative for me. I wrote a simple Java application that will launch Nutch and send the indexing to Solr but as you can see in the source code, you still need a UNIX like environment to run successfully. You can write a platform independent version by looking up Nutch API and calling the methods directly.

Well, I hope that this entry help you understand how to use Nutch and Solr built-on top of Apache Lucene. If you need any clarification, leave comments and I will try to gave ASAP if time permits.

package com.etapix.nutchsolr;

import java.io.BufferedReader;
import java.io.File;
import java.io.FileFilter;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.logging.Level;
import java.util.logging.Logger;

/**
 *
 * @author Armel Nene
 */
public class Indexer {

    public static void main(String args[]) {
        if (args.length < 3) {
            System.out.println("Usage:" +
                    "\ncrawlName        -   This will be used to store crawler files in CrawlName directory" +
                    "\nurlFolder        -   The path to the folder containing the URL to crawl" +
                    "\nsolrUrl          -   The URL to the Solr server");
            return;
        }
        String crawlerName = args[0];
        String urlFolder = args[1];
        String solrUrl = args[2];
        String inject = "bash bin/nutch2.sh inject " + crawlerName + "/crawldb " + urlFolder;
        String generate = "bash bin/nutch2.sh generate " + crawlerName + "/crawldb " + crawlerName + "/segments -topN 10 -numFetchers 5";
        String export = crawlerName + "/segments/";

        String invertLinks = "bash bin/nutch2.sh invertlinks " + crawlerName + "/linkdb -dir " + crawlerName + "/segments";
        String indexSolr = "bash bin/nutch2.sh solrindex " + solrUrl + " " + crawlerName + "/crawldb " + crawlerName + "/linkdb " + crawlerName + "/segments/*";
        try {
            System.out.println("Injecting URLs in crawldb");
//            int state = 0;
            InputStream in = Runtime.getRuntime().exec(inject).getInputStream();
            System.out.println(convertStreamToString(in));


//            state = Runtime.getRuntime().exec(inject).waitFor();
//            System.out.println("process completed: " + state);

            for (int i = 0; i < 3; i++) {
                System.out.println("Generating segments");
                in = Runtime.getRuntime().exec(generate).getInputStream();
                System.out.println(convertStreamToString(in));

                System.out.println("Setting environment variable $SEGMENT");
//            String segs = convertStreamToString(Runtime.getRuntime().exec("ls -tr " + crawlerName + "/segments|tail -1").getInputStream());

                String segments = export + lastFileModified(export).getName();
                System.out.println("$segments: " + segments);
//            in = Runtime.getRuntime().exec(export + segs).getInputStream();
            System.out.println(convertStreamToString(in));

                String fetch = "bash bin/nutch2.sh fetch " + segments + " -noParsing";
                String parse = "bash bin/nutch2.sh parse " + segments;
                String update = "bash bin/nutch2.sh updatedb " + crawlerName + "/crawldb " + segments + " -filter -normalize";

                System.out.println("fetch segments");
                in = Runtime.getRuntime().exec(fetch).getInputStream();
                System.out.println(convertStreamToString(in));

                System.out.println("Parse segments");
                in = Runtime.getRuntime().exec(parse).getInputStream();
                System.out.println(convertStreamToString(in));

                System.out.println("Update crawldb");
                in = Runtime.getRuntime().exec(update).getInputStream();
                System.out.println(convertStreamToString(in));
            }
            System.out.println("Inverting links");
            in = Runtime.getRuntime().exec(invertLinks).getInputStream();
            System.out.println(convertStreamToString(in));

            System.out.println("Indexing contents to Solr " + solrUrl);
            in = Runtime.getRuntime().exec(indexSolr).getInputStream();
            System.out.println(convertStreamToString(in));

        } catch (Exception ex) {
            Logger.getLogger(Indexer.class.getName()).log(Level.SEVERE, null, ex);
        }
    }

    public static String convertStreamToString(InputStream is) {
        /*
         * To convert the InputStream to String we use the BufferedReader.readLine()
         * method. We iterate until the BufferedReader return null which means
         * there's no more data to read. Each line will appended to a StringBuilder
         * and returned as String.
         */
        BufferedReader reader = new BufferedReader(new InputStreamReader(is));
        StringBuilder sb = new StringBuilder();

        String line = null;
        System.out.println("Now converting inputstream to text");
        try {
            while ((line = reader.readLine()) != null) {
                sb.append(line + "\n");
            }
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            try {
                is.close();
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
        System.out.println("Finish converting to text");
        return sb.toString();
    }

    public static File lastFileModified(String dir) {
        File fl = new File(dir);
        File[] files = fl.listFiles(new FileFilter() {

            public boolean accept(File file) {
                return file.isDirectory();
            }
        });
        long lastMod = Long.MIN_VALUE;
        File choice = null;
        for (File file : files) {
            if (file.lastModified() > lastMod) {
                choice = file;
                lastMod = file.lastModified();
            }
        }
        return choice;
    }
}