I decided to write a second part to my “4 reasons why Appium is not ready for prime time” article. There was an interesting comment posted by Satyajit Malugu which I thought required a post by itself because it raises some important questions. Hopefully, some additional comments will keep coming as I’d like to inform as much as possible testers and developers about the current state of Appium.

Here are the 4 comments made by Satyajit

1. No implicit waits – Appium has good support for implicit waits. Ruby_lib is using them and my tests rarely have to wait on something. Let me know I can provide my logs of this works.

I’m not a big fan of implicit waits as they don’t offer flexibility for specific elements you want to wait on. You basically define a global wait-for-element for the entire duration of the test. Typically I woud want to use different type of waits wether I’m dealing with screen-flow for my app or if I need to wait for the return of a back-end call. Implicit waits don’t give me this flexibility. Additionally, I can’t specify the element I want to wait on. Finally, I didn’t see any capabilities to wait for other element’s charactertics such as count, color, enable/disable, etc. Explicit waits are much more powerful and flexible. That’s why we spent a lot of time implementing them in our product and we add new ones almost every week based on the feedback we receive from our community.

2. No parallel runs – Appium/sauce labs philosophy is to use emulators in their cloud and the support parallel tests, its in their road map but not there yet

I have 2 problems with this comment:

• I can’t recommend an open source mobile test automation that rely on a commercial vendor to run tests. I like sauce labs, I like what they’ve done for web browser testing but if I want to use an open source solution, I want the benefit of open-source all the way. So if I’m not using SauceLabs, I’m left with buying 500 Mac Mini if I want to run 500 iOS tests in parallel. Not too sure this is a good investment.
• Sauce Labs executes tests on simulators. So if my tests pass on a simulator, am I good to go? Can I ignore testing on real-device? Of course not and I still don’t know why testers still run their regression tests on simulator. I understand why developers use them: It’s cheap, it’s integrated within their IDE, it’s great to debug their apps. But testers? Let’s get serious. Simulators are not the right platform to run tests in my opinion: Way too many false-positive (and negative!), they come with stock OS which is a problem on Android where there are plenty of OEM customizations, network connectivity comes from the client (PC or MAC) and is very different from what you get on a real devices and finally the CPU/Memory they simulate is not fit to run any serious performance testing.

3. Gestures – there is support for gestures, not in the most easiest of the ways but I am able to flick, swipe, long tap etc. Now tell me which other framework provides you even that?

Mobile apps rely heavily on gestures and support for ALL OF THEM in a mobile test automation product is critical. It shouldn’t require any coding and should be easy to use. Appium is not there yet and somehow I find it perfectly normal. I’ve seen the SOASTA engineering team working endless hours to support ALL OF THEM across native, hybrid and Mobile Web apps on both iOS and Android. It’s a LOT OF WORK but this is the price to pay to offer ease of use and full support to testers and developers on all platforms. We today support 60+ standard gestures on both iOS and Android, so yes it is possible.

4. Limited support for android < 4.1. Now is it appium’s fault that google decided to come up with new framework 17+? Also there are couple of projects underway to merge selendroid into appium, or using espresso. If you are motivated enough, you could create implement a page object design pattern and use selendroid and appium in the same test suite.

When you build an automation framework, you look at your market and you’re trying to support the majority of your users. Today, Appium supports 18.2% of the market. If I’m a developer or a tester, is this sufficient? Can I wait for Appium to come up with support for < Android 4.1? Or should they not start their automation project until carriers and manufacturers decide to upgrade the phones of their users? Should they mess with Selendroid and read pages of support forums to make their solution work for Android < 4.1? Appium decided to invest in the future and that’s fine. But it means that testers will use the solution … in the future.

Keep the comments coming!

You can take the survey here.

Note: I have no affiliation with the survey.

As someone responsible for a mobile automation product, I spend a reasonable amount of time looking at the competition. I particularly like when I see some innovation, something that forces us to do even better and stay ahead of everyone else. Usually, this innovation comes from small company or from the open source community. Appium has been on my radar ever since it came out, but I waited for Appium to get some maturity before trying it out. My conclusion is very simple: Appium is not ready for prime time. It reminds me of Selenium and webDriver in its infancy. Not something you want to invest in for serious work. Maybe fine for individual developers to build some smoke tests, but if you have an enterprise app for example, with complex gestures and a lot of tests to build and maintain, Appium is not a good candidate, yet.

These are the main shortcomings I’ve noted. Some are really showstoppers for me.

#1 – No support for intelligent waits. 

This is the biggest shortcoming and a big red flag. You can code some time delays (polling loops) and that’s it. A time delay is the WORST mechanism to manage your page flow or address back-end response time variability. Reasons being:

• How long should I wait after a back-end call? Yeah, let’s put 10 seconds, it should be way enough. It might work for 90% of the time and you end up spending time investigating errors for 10% of your test cases. The goal of automation is to find regression issues in your app! Not problems with your environment.
• How long should I wait to account for device performance? Transition between pages depends a lot on the performance of your device. Might be almost instantaneous on the latest ipad Air for example with their new CPU and could be crawling on older iOS devices. So should you add 2 secs each time you transition between screens in your app? Should be enough right? Sure, it should be fine. Multiply these 2 seconds by your number of transitions, number of tests, number of devices and you end up with a lot of hours spent WAITING! When you have 10000 tests to run during the night for example, that’s not something you can afford. Especially when developers need feedback FAST!

Without being able to wait on a UI element characteristics (visibility, value, etc.) to manage page flow or back-end call, an automation framework for mobile is pretty much worthless in my book.

#2 – On iOS, you can only execute one test at a time per Mac.

This is a limitation coming from Apple Instruments that Appium uses for execution. I’m working with customers running 5000+ tests per night, on 30+ devices, all in parallel. It would take them many days to run all their tests sequentially. This is a SERIOUS limitation and I don’t know how Appium is going to solve this since they have no control over Apple Instruments. You can always buy 1000 Mac Minis! Yeah right.

#3 – Limited support for gestures.

If your app only use simple gestures, like tap, you could be fine. But this is 2013 and gestures are the name of the game to improve the user experience (double-tap and pan for Google map for example. It allows you to zoom with your thumb for example. Pretty neat!). Appium allows you to write Javascript wrappers to support these gestures. But c’mon .. Do you really want to spend time writing support for gestures or would you rather write new tests for your app?

#4 – Limited support for Android < 4.1

Appium only supports API Level >=17. If you look at the number of devices running the different versions of Android (via the Android Dashboard) you realize that Appium can only support a very limited share of the market. Do you want to invest seriously in a framework that only allows you to support 18.2% of your market? I don’t think so. I thought it was pretty crazy to have such limited support and I dug a bit. Apparently, you can support Android < 4.1 by using a Selendroid library but it’s not for the faint of heart by judging by the number of people struggling to make it work.

These are my 4 reasons why I think Appium is not ready for serious work. They’re showstoppers in my book. I’m also not a big fan of the approach Appium is taking, which is fairly similar to Selenium/Webdriver. If you like to write tons of code for your functional tests, be my guest. Of all the WebDriver implementations I’ve seen in my career, a lot required a big investment, especially on the maintenance side. But I haven’t seen all projects in the world.

Appium is young and it might grow to something fit for mobile. And honestly, I hope it will! I’m a big fan of open source and the innovation it brings. But mobile is a big business and the product you pick to test your app is an important choice and investment, and you want to make sure it covers all your requirements. I just don’t think Appium is there yet.

Spring is right around the corner and it’s going to be fairly busy for me! I’m very exciting to take my talk arond the United States. If you’re getting started with mobile test automation, this is a great talk to attend. I’ve worked with dozens of customers to help them avoid typical pitfalls and get them on the right track. Mobile test automation is an exciting journey that will make a BIG difference for the quality of your mobile apps. But it is a true software project by itself that needs careful planning, dedication and some best practices to follow. This is what my talk is all about!

If you attend “Top Best Practices for Successful Mobile Test Automation“, you will learn:

• How to set the right goals and expectations for your automation project
• How to track success and ROI
• How to automate your mobile automation
• When to start automating mobile tests
• What mobile tests to automate
• Where to run mobile tests
• How to build testability in your mobile app
• How to quickly increase your mobile test coverage
• How to best manage mobile flows in your test
• How to leverage your functional tests in your performance tests

This is my current engagements for the begining of the year. If you hear of any other great software testing conferences to attend, let me know!

• TestKit – Mar 3-5 – Arlington 
• Quest – April 7-11 – Baltimore
• StpCon – Apr 14-17 – New Orleans
• StarEast – May 4-9 – Orlando

There are a lot of discussions in the software testing community discussing the benefits and drawbacks to use an instrumented browser approach to run mobile web automation. As I’m working with a number of customers on this exact subject, I thought I would share my opinion on the subject.

If you want support for precision, complex gestures (multi-touch) while being able to access DOM elements as objects, there are no best way today than to use an instrumented browser. On mobile devices, at least on iOS and Android, applications are sandboxed, meaning that they’re not allowed to access the environment of other apps. This is why, there are no good options today to retrieve objects from native browsers (Mobile Safari, Chrome, Opera, etc.). This is why most solutions on the market today have developed their own instrumented browser which comes in the form of a UIWebView. On iOS, there are hard restrictions from Apple that prevent anyone to come up with a different type of browser. Apple imposes the use of the built-in UIWebView as well as a single-process model. It means that rendering on Chrome, Opera and any other browsers will be similar. But any other third-party browsers can’t use their own JavaScript engines or use Apple’s Nitro engine. A huge advantage for Apple for performance!

I’ve seen solutions that drives Apple UIAutomation (or Android UIAutomator) to perform mobile web automation. Unfortunately, at least for now, they all fall short of expectations. UIAutomation doesn’t provide any precision, so good luck automating mobile web apps such as Google Map. It is also tight to Instruments that can only run one instance per Mac OS. This is not realistic in a Continuous Integration scenario. These solutions require also a lot of coding and personally I’m not a big fan of coding for functional testing. First it takes forever to write the tests but you end up quickly in a maintenance nightmare. I love these type of tests for unit tests, not for functional testing. Finally, these solutions have problem supporting the latest OS and on iOS it’s a big problem since adoption is very, very fast: In less than a week, 50% of people had upgraded to iOS7. We’re at close to 80% today. If the automation framework you’re relying on doesn’t support the new OS from day 1, you’re in big trouble!

Yes, it would be awesome to be able to record tests on native mobile browsers on real devices, be able to perform complex gestures, be able to replay them on other mobile native browsers and add meaningful object based validations. It’s just not possible today due to too many restrictions. Your best bet is still today to use an instrumented browser. I personally didn’t find one customer having problems with this approach.

It’s been some VERY busy 6 months for me getting back to the United States but things are settling down nicely! So expect more updates from me on this blog! Aren’t you some lucky readers?

At SOASTA, we’ve always been big fan of CloudBees and this is why last year we had partnered with them to offer mobile developers and testers a way to continuously build and test mobile apps.

One of our goal is to compress the build-test-feedback process to a minimum. We want to provide developers a continuous feedback loop so they can assess the quality of their code as soon as they’re done writing it. Traditionally this feedback is given to them within days by traditional testing organization using outdated products. With Jenkins and TouchTest, we want them to give them this feedback in minutes while extending the scope of their tests.

Our plugin makes it easier for them to setup this feedback loop and adds new functionalities to build a more robust regression test environment.

• Ability to play a composition, make the App Touchtestable automatically, Install App on device and capture screenshots.
• Reboot iOS devices
• This is a testing best practice that allow to run a set of tests under a “clean” state. After a while, a mobile device can become bloated and unstable. Being able to automatically reboot devices reduce the number of false-negatives and developers can spend time fixing real problems. This capability is unique on the market today for iOS (This capability comes free on Android)
• Wake up iOS device
• This functionality can be used to make the test environment more robust by waking up any iOS devices and put them in a “ready-to-test” state. A more robust test environment allows developers and testers to focus on more important activities ie. Building new tests!

You can learn how to get started with the plugin here.

Next week, I will be in Palo Alto at the Jenkins User Conference where I hope we can meet you so I can show you the great stuffs we’re cooking for developers. I especially want to demonstrate some end-to-end scenario I’ve been working on allowing to combine, via Jenkins, performance tests and functional tests of mobile apps. A lot of customers I’m working with want to understand the impact of back-end load on the performance of their mobile app front-end. Being able to do these type of tests continuously in order to find performance problems early in the development cycle, really help flush out the type of bottlenecks you don’t want to find when you run your 1M+ load test later on. The sooner you find these problems, the easier and cheaper it is to fix them! 

See you there!

A quick personal note to let you know that after 6 great years spent in Europe, my family and I are moving back to the Bay Area! This is the third time I’m packing my bags for the West Coast! Back in 1989, I spent one year in Seattle (Go Seahawks!!) at the university. I’ve spent 8 years between 1998 and 2006, working for IBM. I’m now heading back for god knows how long to work more closely with my colleague at SOASTA. I guess there is something special about this country …

I can’t resist to show you the latest video we’ve sent to our customers on Valentine’s day. This is real footage of our latest product mPulse, born soon after the logNormal acquisition only a few months ago. I’m so amazed by the work done by our engineering team in such a short time! In a nutshell, mPulse redefines the Real User Measurement space, allowing customers to understand in real-time how their web and mobile apps are behaving in the field. To help deal with bad user experience of mobile and web apps, mPulse comes to the rescue with Big Data, Real-time Processing, Actionable Intelligence and amazing 3D visualization!!

Get plenty of info and FREE RUM!

Enjoy the show!

I’m having more and more discussions with customers looking at investing into a mobile test automation solution. It’s a great way for me to listen to their problems and discuss potential solutions. One recurring requirement I’m hearing is the capability for the automation solution to integrate with HP QTP. And I have a big problem with the rationale that makes them request such integration.

I have nothing against HP products (or any testing products) as long as they’re used for what they’re designed for. As an example, Loadrunner is probably best fit for load and performance testing of client-server applications rather than large scale testing of web or mobile applications. Same goes with QTP in my opinion. It’s probably king for functional testing of desktop applications in a waterfall environment but is completely unfit for mobile applications which require dev and test high velocity. Do we still need a rudimentary script language (vbscript! Come on, this is 2013!) for fast test creation? Do we want to write code or rather record directly on an actual mobile device? Do we want to use a product that was built for mobile or a product built for desktop apps, developed 12 years ago? Do we want developers to use … QTP? Really?

I understand the reasons why a mobile test automation company would want to integrate with QTP. This product has a large user base as it’s been used quite extensively in the past to test desktop apps. A huge market opportunity! A lot of testers are (too) comfortable with QTP and as they transition to mobile testing, their learning curve would seem minimal. Makes sense right? This is unfortunately a very shortsighted view that would leave a lot of testers behind in a very tough job market.

Mobile apps are very different animals than desktop apps. Remember those 18 months releases for your typical desktop apps? It was quite comfortable wasn’t it? Well things changed and today release cycle for a typical mobile app can be measured in days (and sometime hours for the most popular one!) and the automation product you’re using needs to have this new paradigm as part of its DNA. QTP has a desktop DNA relying on a prehistoric scripting language requiring heavy volume of maintenance each time your mobile app is built (like every hours!). Additionally, have you ever tried to have a developers write tests using QTP? Mobile apps are all developed in an agile environment where everyone is testing and a lot more developers are writing functional tests (Could they be the future of Software Testing?). I have never seen QTP installed on a developers machine as it is a big fat memory hog and I can’t imagine a developers go back to … VBScript to write … anything.

I might get some traditional testers a bit angry at me for saying that they’re part of the problem. I understand that after working a long time with QTP, you master the tool and as you transition to mobile testing, you might get tempted to continue using it. Changes can be seen as scary, pushing us to get out of our comfort zone without guarantee that we’ll be successful. So quite a number of testers want to stay in the past and are dragging innovation down for the rest of us. Some vendors would do anything to grab that share of the market, knowing quite well that this is not the best technical solution, only driven by revenue. I feel software testing would benefit tremendously if we could eradicate this behavior and maybe adopt a more open and forward thinking mentality similar to what we find in the development world. I don’t forget that there is a new wave of testers riding the innovation wave (you know who you are) but throughout my discussions with customers I’m also a bit shocked to realize that a lot of them are still in the dark ages and are at risk in the long term.

Full disclaimer: Yes, I work for SOASTA and you might find my opinion a bit biased. More than 2 years ago, I’ve joined this incredible team because I was convinced that they wanted to change the status-quo and build the future of software testing. I’ve not looked back ever since and each and every day brings new reasons to think that this was the best decision of my career. Yes I’m responsible for our automated mobile solution and you bet I’m gonna evangelise it a whole lot! But those who know me would tell you that I’m able to keep my impartiality when it comes to software testing. I’m just too passionate to be able to BS on a subject so close to my heart. I would acknowledge innovation when I see it and there are other testing companies on the right path (they push us to be better!) but I feel it is my responsibility to point dysfunction in our industry when I see one.

QTP, leave them mobile testers alone.

I’ve been a hobbyist photographer for the past 10 years. Pretty much synched with the birth of my daughter who is today happy to enjoy 15,000 pictures of her first week in life The time was 2002 with the first decent digital camera out there. My first one was a Kodac DC3200 and was a great tool for me to learn the basics of photography. Then I moved to a Canon G2 before switching to a Canon SLR as Digital SLRs were costing an arm, a leg and everything in between. I made the mistake to rent a Leica M6 (At Keeble and Suchat in Palo Alto) so I could understand all the fuss about these cameras and feel confident that it was for snob. Huge mistake … I had to have one the following week, and another one, and another one … Oh and how about a Rolleiflex, oh and a Hasselblad, oh and look at those nice Russian Camera. At some point I probably had more than 20 working camera at home … And a few lenses. I was processing my own black and white and really, really enjoyed everything about photography. I was carrying a huge bag with many camera inside at all time. Then life catched up. Lots of traveling, different priorities. Sold most of the camera (Kept the Leica. Come on!), still have 300 rolls of film in my basement, and settled on a mid-range Canon digital camera for family events.

Beginning of the year, I’ve purchased an iPhone 4S to replace my Android HTC Desire (which takes crappy pictures) and started playing with it. I “discovered” Instagram back in May during a photo trip to Seattle with one of my good friend. We decided to shoot only with our iPhone for 2 days. I fell in love with the simplicity of the combo iPhone+Instagram. Reminded me of the time of my Leica + 50mm lens. No zoom (zoom with your feet!) and I’ve always liked the square format (Makes you think differently). Instagram allows me to give some “pop” to my photos (mainly contrast) and I’m able to share my pictures with a very private group of friends around the world. When I travel, it’s also a great way to keep in touch with the family as well (FaceTime does help too).

You can see some of my favorites pictures of 2012 on Flickr

I wasn’t surprised by yesterday Instagram’s announcement. This is a business, publicly traded with revenue expectation. There is no free lunch in this world and Andrew Lewis hit it right on the nail yesterday: “If you’re not paying for something, you’re not the customer; you’re the product being sold.”. A business cannot survive without a revenue stream and it is clear that users are not willing to pay for popular social networks (not yet anyway). So they will see ads, a lot of them! Facebook is expected to quadruple its $339 million on mobile ads by 2014!

Would I rather see no ads? Sure! I hate ads but if they are subtle enough, I can deal with them. And I believe 90% of Instagram’s users can probably deal with them. Do I care if they use my own pictures as part of these ads? Not really, especially with their latest clarification.

The real issue is probably more for professional photographers rather than people shooting cappuccinos and cats. Instagram allows them to build up an audience which leads to commissioned work while getting instant feedback and interaction. They’ve been hooked to a powerful drug and now the pusher man has decided to take his cut. New terms revealed yesterday, suggest that these photographers will help Facebook to become the biggest stock imagery agency out there. They will basically feed their new and powerful competitor. A lot of photographers are concerned and I can understand why. But I can bet a lot of them will remain on Instagram after January 16th when the new TOS take effect. Including National Geography who put their account on hold yesterday.

As Tim Barber  said: “if you want a free, fast, beautifully designed, complex digital image platform on your telephone that enables you to communicate in a revolutionary new way with an ever expanding global network with no strings attached, build one yourself or quit your whining.”

For private users, like me, it sounds like a non-issue, especially with the revised statement by Instagram released today. I was indeed a bit concerned that my pictures could be used in ads outside of Instagram. This is not the case as of today and this is a good thing. Second concern was about who would see my pictures in ads. I would have been eventually concerned if pictures of my family would be used in ads outside my private network. The revised statement is clear: Only people following me would see ads that could include my own pictures. Well, I don’t care because they’ve already seen these pictures! So go ahead Instagram, takes my pictures and create ads and make a few bucks! I’m happy to help the platform which has rejuvenated my passion for photography!

My biggest disappointment with the whole thing is around communication. It would have been nice, for a change, to see one of these powerful social network, to reach out to its community and open a communication channel to discuss its monetization plan. Facebook has a long track record of communication blunder and they’re not getting any better …

Call me naive, but for now: Long live Instagram!