Real World Software Development with Openshift

OpenShift is Red Hat’s free, auto-scaling platform-as-a-service for Java, Ruby, PHP, Perl and Python applications.

In this practical talk, Jeremy Brown will demonstrate how to use Red Hat’s OpenShift PaaS within a team of developers to rapidly develop software.

The talk will describe how a team of developers can develop, test and release code in parallel without losing development velocity and will draw on real world examples and insights.

Jeremy will demonstrate how to use and configure a Jenkins Continuous Integration server with a Java project hosted on github.com, while multiple developers commit code and features in parallel, with real time selenium testing as code is released. This talk will include code and configuration samples as well as a hands on demonstration of the end-to-end project workflow.

You will come away with a greater understanding of how you can use OpenShift to create a highly effective cloud development environment with zero server configuration.

On Tuesday at 4:30pm I found out that we needed someone to speak at a conference in Prague the next day… slightly foolishly I volunteered and 24 hours later I found myself presenting!

The conference was the “International Conference ITTE 2011 — Cyber Security and Defense” - http://www.afcea.cz/ My topic was “Open Source and Security: Engineering Security by Design’ and I had to brush up a bit on the content as it’s not something I had ever presented on before.

Still it went OK I think and I enjoyed my brief 24 hours in Prague — though not to the 24 hours before sweating and prepping for the talk and the panel discussion that followed afterwards.

Here are my slides: Open Source and Security: Engineering Security by Design

StackScripts™ provide a flexible way to customize our distribution templates. They’re very easy to use — find a StackScript, answer its questions, and click deploy. When the deployment is first booted, the script is executed and does its thing. You can even watch its progress by viewing the console.

I’ve used these for a while and decided to open source these scripts to make them publicly available on github - https://github.com/tenfourty/LinodeEnterpriseLinuxStackScripts feel free to fork and update these or let me know of your variations on these on Linode’s StackScripts library (where my code is also available).

I think I can summarise my reasons based on this email:

• Faster updates
• Professional
• Security and bugfix updates separate
• Security updates also for older versions of SL
• More reliable than CentOS

The CentOS guys seem more focused on infighting rather than focussing on getting CentOS 6 out the door, considering how long RHEL 6 has been available this has turned into a bit of a joke!

When I googled around I couldn’t find a concise article that explained how to do this, so here is a my simple guide to switching from CentOS 5.6 to Scientific Linux (SL) 5.6, it was relatively straightforward and my rock solid VPS has stayed up and reliable throughout the process. These instructions are based on the following articles:

• CentOS Migration Guide
• This thread on the SL user mailing list.

If you want to tell what version of CentOS you are currently running you can use the following command:

cat /etc/redhat-release

Now to “upgrade” to SL. All commands should obviously be run as root.

You might also want to back up your release file, you can do that with the following command:

cp /etc/redhat-release /etc/redhat-release-saved

Then just use the following commands to remove CentOS and install SL:

yum clean all

rpm -e --nodeps centos-release-notes centos-release yum-3.2.22-33.el5.centos.noarch redhat-logos redhat-artwork redhat-menus

rpm -e --nodeps yum-3.2.22-33.el5.centos.noarch

rpm -ivh http://ftp.scientificlinux.org/linux/scientific/56/i386/SL/sl-release-5.6-1.i386.rpm http://ftp.scientificlinux.org/linux/scientific/56/i386/SL/sl-release-notes-5.6-1.noarch.rpm http://ftp.scientificlinux.org/linux/scientific/56/i386/SL/redhat-artwork-5.0.9-2.SL.4.i386.rpm http://ftp.scientificlinux.org/linux/scientific/56/i386/SL/redhat-logos-4.9.16-2.sl5.6.noarch.rpm http://ftp.scientificlinux.org/linux/scientific/56/i386/SL/redhat-menus-6.7.8-3.el5.noarch.rpm http://ftp.scientificlinux.org/linux/scientific/56/i386/SL/yum-3.2.22-33.sl.noarch.rpm http://ftp.scientificlinux.org/linux/scientific/56/i386/SL/yum-autoupdate-1.1-1.SL.noarch.rpm http://ftp.scientificlinux.org/linux/scientific/56/i386/SL/yum-conf-56-1.SL.noarch.rpm

yum update all

If you now check what release version you are on you should see something like the following:

[root@wahala ~]# cat /etc/redhat-release
Scientific Linux SL release 5.6 (Boron)
[root@wahala ~]#

Next I think I’ll attempt to upgrade or rebuild my VPS to run Scientific Linux 6… the subject of a future blog post!

As we were doing performance testing we wanted to run EDS within a cluster of JBoss EAP nodes, now clustering EAP nodes is fairly straightforward and you can then setup a front end load balancer with Apache httpd, in my case I used the Red Hat product based on the Apache web server called Enterprise Web Server (EWS) and mod_cluster to cluster and load balance my application servers. Now this sort of clustering is fine if you want to do replication of your applications and use distributed cache replication within the cluster, however the question was how do you do load balancing on the ODBC and JDBC connections that EDS provides for the application tier? As these types of connections can’t be load balanced by the Apache Web Server we had to come up with another way to do this.

As we were at a very large enterprise client their initial suggestion was that we use a hardware TCP load balancer to do this, however I was pretty sure this was a straightforward problem that must have been solved already and that a software based solution must exist… and low and behold there is one and it is called HAProxy.

HAProxy is a really cool load balancer that is very powerful and flexible and also really easy to use and it seems that not too many people are aware that it can be used to load balance ANY TCP connection, this blog post lead me in the right direction and with the ethos of trying to help all of you out there on the web here is my very short how to and sample configuration for how to load balance ODBC and JDBC connections — specifically for Teiid embedded in the JBoss Enterprise Data Services product.

First you need to get HAProxy, in my case I was running everything on RHEL so the easiest thing is to add the Fedora EPEL repository to your machine and then just do a “yum install haproxy”, you should now have haproxy installed on your machine — now you just need to configure and launch it!

So first configuration, I’ve included a sample configuration file for 4 nodes, as I had two physical machines with 2 nodes running on each of them. The first node on each machine was running the default JBoss ports and the second was using ports 100 greater than the default. Here is my file:

You’ll notice that I also create a listener for stats on on port 9090 — this means that you can have a nifty web stats page that you can view on http://<haproxyIP>:8080/haproxy?stats (username/password = admin/admin).
Now that we have HAProxy configured we just need to run it, in this case it would be “haproxy –f <config file> –V”, I used the verbose flag so that I could see what was going on but you may not necessarily want this for production.

Now just launch the stats page to see your nodes connected and you are up and running — don’t forget to configure the ODBC and JDBC listener sections to put your own ports to listen to depending on the setup you are going to use.

Oh and by the way the performance testing we did showed that Teiid is amazingly performant!

22 GB of memory
33.5 EC2 Compute Units (2 x Intel Xeon X5570, quad-core “Nehalem” architecture)
2 x NVIDIA Tesla “Fermi” M2050 GPUs
1690 GB of instance storage
64-bit platform
I/O Performance: Very High (10 Gigabit Ethernet)

The really interesting part that has got a lot of people interested is the fact that it has two high-powered graphics cards which can be used to do massively powerful parallel computing. Now there are many potential applications for these GPUs — image and video processing, computational biology and chemistry, fluid dynamics simulation, CT image reconstruction, seismic analysis, ray tracing and so on. But what really interests me is the possibility of using these GPU instances for password cracking.

It’s now well-known that using a GPU to crack passwords can reduce the time required from something like 2 months to 3 days, but what happens when you throw one of these new Amazon instances at the problem? And what if it’s not just one instance but a cluster of them that is used to do the massively parallel computation? The sheer computing power that even a small cluster of these machines has available would make short work of cracking all sorts of passwords. Some that come to mind are:

• System Password files which use the MD4, MD5, NTLM or SHA1 algorithms.
• WPA-PSK or WPA2-PSK network passwords (WEP is already trivial to crack).
• Password protected RAR or ZIP files.
• Password protected Microsoft Office or Open Office files.
• Password protected PDFs.
• Encrypted disks

Some people have already tested these GPU instances to crack password hashes and Pyrit has been tested on it (could be used to crack WPA/WPA2). The performance of a single instance is impressive, the cost is equally impressive ($2.10 for an hour). Just a few years ago this kind of computing power was only available to organisations that had a large amount of resources such as governments, large corporations and a few universities and research organisations. Now anyone with a bit of technical knowledge and a credit card has access to it.

It’s only a matter of time before someone uses a cluster of these instances in anger to start cracking passwords, in fact I’m sure someone already is. How long will it be before someone releases a commercial service based on this platform?

The only commercial service for password cracking that I’ve found so far is WPA Cracker who claim to have a 400 CPU cluster, however a service that uses a few EC2 GPU instances could blow away the performance of WPA Cracker. We could soon start to see passwords being cracked in just a few minutes with a large enough cluster. I wouldn’t be surprised if it wasn’t long before someone sets up a service like this which integrates nicely into backtrack or some other wi-fi sniffing software that grabs the required wi-fi packets and uploads it to an EC2 cluster that cracks the password in a few minutes.

All of this is a very strong argument for using longer and more complex passwords that are less vulnerable to dictionary and brute force attacks, and one more reason not to assume that your wi-fi network is secure because you are using WPA or WPA2 instead of WEP.

In addition I hope that this post will highlight the fact that you must optimise your site for low-bandwidth users, especially if you operate in countries with poor Internet connectivity. In fact it’s not too hard to do these days if you take a few simple steps, but first you need to be aware of the issue.

Having lived in Cameroon for the last two years I’m more than familiar with surfing on a poor unreliable connection and it staggers me that so few Cameroonian companies have made an effort to improve the user experience for the majority of people who visit their websites over these slow connections. Internet users in Cameroon either have their own 128 kbps “broadband” connection or they are at an Internet café that shares a slow connection between their users (who are all on Facebook and YouTube competing for the bandwidth available). Some are using USB dongles to access mobile data networks, but as none of the operators have a 3G license users are stuck at slower GPRS speeds with patchy coverage.

The ISPs

The major ISPs in Cameroon today are:

• Camtel — the state-owned incumbent who has a monopoly on all the fibre in the country and the SAT3 cable, all other ISPs must buy bandwidth from Camtel. Camtel provides Internet access via ADSL or over Wireless using CDMA.
• MTN — the mobile operator with the largest market share, providing Internet through GPRS, wireless hotspots and WiMax.
• Orange — the second largest mobile operator, providing Internet through GPRS and WiMax.
• Ringo — they claim to have more bandwidth available than their competitors (apart from Camtel). Ringo is using proprietary SCDMA based McWill technology from XinWei in China and have also started to provide wireless hotspots in the major cities.
• Matrix Telecoms — ISP using wireless technology to provide access.
• Creolink — uses cable to provide access to their customers.

I’ve tried to summarise the cheapest options for consumers from each of these ISPs in the table below.
[table “2” seems to be empty /]

As you can see 128 kbps is pretty typical, even if some providers claim to have faster speeds the actual speed is closer to 128 kbps in real world conditions. Price wise they are all pretty similar, though it’s quite a difference from the prices we are used to paying in Europe! For example, from Virgin in the UK you can pay €15 a month for a claimed 10 Mbps.

Analysis and Results

So given all of this I thought I would do a comparative analysis of the major Cameroonian ISP’s websites to see how they actually fared on a typical 128 kbps connection. I used a combination of pipes and the ipfw command on my mac (for bandwidth simulation), Page Speed/YSlow plugins for Firefox and the developer tools in Google Chrome to measure the page load results. The tools used and approach wasn’t totally scientific but they give a pretty good indication of real performance. If you are interested in running this test yourself Aptivate have a great blog post of how to do this. The metrics I measured were:

• Page load time, empty cache and primed cache
• Page size, empty cache and primed cache
• Number of HTTP Requests, empty cache and primed cache
• Page Speed Score
• YSlow Grade

The table below details the results.

On the whole the results are pretty disappointing and show just how out of touch with their users the ISPs really are, Matrix does best with a page load time of 15 seconds, not too bad but they still have room for improvement, in comparison it takes 10 seconds to load www.google.cm’s 174K page when the cache is empty and 1 second when primed. The results for the other websites go from bad to worse to incredible!

What is really frightening about these results is that the slowest loading sites took nearly a minute or more, surely none of their customers spend that long waiting for the sites to load or much time browsing on their pages. The big culprit in all of this is of course Flash! As far as I’m concerned it just isn’t a technology that should be used on websites, especially in low-bandwidth situations. Usually countries where bandwidth is very low the PCs being used to the access the Internet are old and slow and Flash causes even more problems as it hogs CPU, a double whammy of user pain.

Flash used on these sites to promote the ISP’s products, however because they take so long to display the user has usually moved on by the time they load so they don’t even achieve the aim that justified Flash in the fist place.

One interesting point to note is that though MTN’s site has a relatively small size in comparison to some of the other sites, with very few HTTP requests, they still take a long time to load because they redirect the browser several times, this significantly increases the total time it takes to load the page.

All the sites tested could improve their page load times by taking the recommended steps in the Page Speed and YSlow plugins, removing errors and broken HTML and mostly by removing Flash.

Conclusion

In conclusion we can see that most Internet Service Providers in Cameroon seem to be out of touch with their customers and pay little attention to the user experience on their websites AND Flash is very bad in low-bandwidth situations!

Does a website reflect a company’s treatment of their customers in real life? I’m curious to hear what your experience is, please leave a comment below.

You may have heard that here at Google we’re obsessed with speed, in our products and on the web. As part of that effort, today we’re including a new signal in our search ranking algorithms: website speed. Site speed reflects how quickly a website responds to web requests.

Speeding up websites is important — not just to site owners, but to all Internet users. Faster sites create happy users and we’ve seen in our internal studies that when a site responds slowly, visitors spend less time there. But faster sites don’t just improve user experience; recent data shows that improving site speed also reduces operating costs. Like us, our users place a lot of value in speed — that’s why we’ve decided to take site speed into account in our search rankings. We use a variety of sources to determine the speed of a site relative to other sites.

Here are more good reasons why speed matters:

Speed is among the most significant success factors web sites face. In fact, your site’s speed directly affects your income (revenue) — it’s a fact. Some high traffic sites conducted research and uncovered the following:

• Google.com: +500 ms (speed decrease) -> –20% traffic loss [1]
• Yahoo.com: +400 ms (speed decrease) -> –5–9% full-page traffic loss(visitor left before the page finished loading) [2]
• Amazon.com: +100 ms (speed decrease) -> –1% sales loss [1]

A thousandth of a second is not a long time, yet the impact is quite significant. Even if you’re not a large company (or just hope to become one), a loss is still a loss.

So how do you speed up your WordPress website to get that extra edge in search rankings and give a better experience to your users? Well you install a caching plugin of course! However not all caching plugins are created equal. For years WP Super Cache has been my weapon of choice because it is simple to install, very fast and is being constantly developed and improved.

One thing always annoyed me though, when I ran any of the performance analysis plugins like Page Speed and YSlow in Firefox or the built in Webkit developer tools in Chrome and Safari, my sites still weren’t scoring top marks even though caching was fully on. This really bugged me as I’m such a perfectionist! I just hated to see those red results in the Page Speed report.

Some of the things that always came up when I looked at the generated reports included:

• Make fewer HTTP requests
• Add Expires headers
• Compress components with gzip
• Make JavaScript and CSS external
• Use a Content Delivery Network (CDN)
• Configure entity tags (ETags)
• Use cookie-free domains

When I finally came across W3 Total Cache I knew I’d finally found the solution to all of this. After making the switch from WP Super Cache to W3 Total Cache I know I’ll be doing this for all WordPress implementations I do in the future. Don’t get me wrong though, for a simple low traffic site WP Super Cache is probably the way to go every time for it’s simplicity and the lack of technical skills required to install and get it up and running. However if your site has a lot of traffic or you want to improve your websites performance by an order of magnitude then I would recommend switching to W3 Total Cache. It requires a little bit more technical knowledge, but it is well worth it.

We were running WP Super Cache (fully optimised) on our site limbelabssolutions.com before switching to W3 Total Cache, the stats below speak for themselves.

This could make all the difference to your server if you get a lot of traffic or want to be prepared for a sudden spike in traffic and of course improve your search ranking at the same time.

Here are the steps I recommend you take before installing W3 Total Cache, including some gotchas to watch out for.

• Benchmark your site before, during and after to understand the impact of your changes. There are many tools out there that you can use. I would recommend a combination of the following:
  • Use Google Webmaster Tools, they have some nice stats on crawling your site, page load times and page sizes.
  • Use the Page Speed and YSlow plugins for Firefox to profile your site.
  • Safari and Chrome have a great Webkit profiler built into the developer menu.
  • There are some online tools that you can use, some I like are www.showslow.com, www.webpagetest.org, tools.pingdom.com
• Remove all existing caching plugins AND delete them. I didn’t do this and it cause me endless problems until I realised what was going wrong.
• Install the W3 Total Cache Plugin, comprehensive instructions are here. Read them before you start as there are a few extra essential steps that differ from the norm and they will throw you if you don’t RTFM. Here’s a good tutorial on how to do it.
• I used our own simple Content Delivery Network, which was very easy to setup (see the the tutorial link above). One comment I would have on the CDN is that I wouldn’t host your minified CSS/JS on the CDN as they aren’t gzip compressed when served up, if you keep them on your main site then W3 Total Cache will serve them gziped. I’m running all this in a shared hosting environment, if I had a dedicated server I would have more control. You will also need to set your WordPress cookie domain in your wp-config.php file if you use this setup.

So there you have it, definitely use W3 Total Cache over WP Super Cache if you want to get that extra edge. However it is a bit more complicated to install and keep running.

It seems that the 29th of June 2006 was the last time there was content on the site. I’m sure I have some backups of all those old posts — should I try and resurrect them?

I probably won’t update the site too often so please don’t be disappointed if I don’t blog too much, however it is here and I guess I will occasionally write stuff or update the site.

You can read more about me, or view my CV (dynamically updated from my Linked In account).