Figure 1 illustrates some of our test devices during testing on all mobile platforms. Due to the fact that the mobile market is growing, the key question for a mobile Quality Assurance person is:

Which devices are right for testing? It’s simply not possible to do testing on every device!

Figure 1: Our mobile Apps on different platforms

Fragmentation only on Android?

I bet every time people hear the word mobile fragmentation, they always think this only happens in the Android world! Yes, the Android world has a really big challenge due to fragmentation of different vendors with different hardware and software installations as well as customized user interfaces. However, if you look at iOS there is some fragmentation as well. Sure, you have just one vendor, but also different software versions and devices. The range of touch devices starts with the iPod touch, the iPhone classic, the iPhone 3G/ 3GS/ 4/ 4S, the iPad/ iPad2 and the new iPad. All of the devices also have differing hardware specifications and can run different iOS versions. So as you can see, the fragmentation issue also exists in the Apple iOS world.

If you think about mobile websites there is another challenge that can be really painful: the various mobile browsers!

There are browsers likes Safari on iOS, Opera Mini for iOS and Android, Dolphin Browser for iOS and Android, the native Android browsers, Google Chrome for Android, Firefox for Android, IE 9 mobile browser on Windows Phones, the BlackBerry browsers and some other feature phone browsers.

So what is the right choice for testing? Just use the latest devices and browser versions? Buy every device on the market?

No, this can’t be the solution because not every user will upgrade their device or software every time a new version is available. The “typical” user normally buys a mobile device with a 2-year contract and will stay with that smartphone for at least that period of time.

So the best way to choose the right devices, operating systems and browser version is to know your customers! If you know which devices and software versions they use, you can downsize the development and testing effort required on mobile devices. Otherwise it’s very hard (impossible, actually) to test your apps on nearly every device on the market! Sure you can use external testing services like utest or apkudo, but maybe you’ll end up spending a lot of money to test devices your customers don’t use!

User statistics – You’d better know your mobile users

We at the XING mobile team know which devices and software version our users have. A good starting point when it comes to getting information about your customers is the app store of the specific vendor (Apple App Store, Google Play Store, Windows Marketplace).

If you already have an app in one of the stores, you will receive information about the devices, software versions, languages and carriers used by your customers. As an example, see Figure 2 below from the Google Play Store which shows statistics from our XING Android App.

Figure 2: Statistics from Google Play

For mobile web pages there is no app store to provide user data. Besides the market statistics, we also have information about the ‘user_agents’ (devices, browsers) used within the mobile website. With this data from the apps and the mobile web app we’re now able to setup a mobile test infrastructure for developing and testing purposes.

Test devices

We split our test devices running Android, iPhone and touch.xing.com into three different groups. Group 1 represents older phones with a less powerful CPU, memory (RAM), and a low screen resolution. The fact that we only have a few users with that kind of device group 1 is a low priority: C. Group 2 covers average devices with a good single-core CPU (1 GHz) and with memory of around 512MB. These devices mostly have a screen resolution of 480×800 pixels (HDPI). This group is graded with medium priority: B. The third group represents high-end devices with at least a dual-core CPU and memory >512MB. Most of these devices have also a big screen with a really good screen resolution (XHDPI). This group has the highest priority: A.

Besides the hardware specifications the groups also cover the software versions (operating systems, browser versions).

Overview of the group categories:

• Group 1, priority C: Small devices with less CPU, RAM and low resolution. Older software versions
• Group 2, priority B: Mid-range devices with avg. CPU, RAM, good screen size and resolution. The software is not the latest.
• Group 3, priority A: High-end devices with a dual-core CPU, RAM (>512MB) and a high screen resolution. Latest software versions.

These three groups cover most of our users and represent other phones on the market that fit into the same group.

Android

For the 1st group running Android we found out that the HTC Legend with Android 2.2 fits into this category. It only has a small CPU (600MHz), just 384MB of memory, a small screen and low screen resolution (MDPI). The HTC Legend runs Android 2.2 which is the lowest Android version we support.

For the 2nd group we chose:

• HTC Desire with Android 2.2
• Samsung Galaxy S with Android 2.3
• Sony Ericsson Arc with Android 2.3

For the 3rd group we chose:

• Samsung Google Nexus with Android 4.0.4
• HTC Sensation with Android 4.0.3
• Samsung Galaxy S2 with Android 2.3.4
• Motorola Razr Droid with Android 2.3.5
• Samsung Galaxy Note with Android 2.3.6
• LG P990 with Android 2.2

Figure 3: Some of our Android test devices

iPhone

We also performed the same grouping for iPhone devices and came up with the following three groups:

Group 1, slow iPhones with iOS versions from 4.0:

• iPhone 3G iOS 4.0
• iPhone 3G iOS 4.2.1

Group 2, iPhone 3GS with different iOS versions:

• iPhone 3GS iOS 4.3
• iPhone 3GS iOS 5.0

Group 3, latest iPhones with different iOS versions:

• iPhone 4 iOS 4.3
• iPhone 4 iOS 5.0.1
• iPhone 4 iOS 5.1
• iPhone 4S iOS 5.0.1
• iPhone 4S iOS 5.1

Figure 4: Some iPhone test devices

We have around 10 devices we use for manual and automated testing of the native iPhone and Android apps. Each new feature we develop is tested with at least one device from group 1 with priority C. When it comes to groups 2 (B) and 3 (A), we used a lot more devices for testing. For manual testing we avoid using mobile simulators in order to have a real testing environment.

touch.xing.com

Based on the data we know from touch.xing.com, we also have three groups of web browsers that should be used for testing on real devices. Most of our touch.xing.com users are iPhone users, while the second most-used platform is Android.
With that knowledge we created the following groups for mobile browsers:

Group 1, non-iPhone and Android Devices:

• BlackBerry Bold with Native Browser (Webkit)
• Nokia N8 with Native Browser (Webkit)

Group 2, a mixture of iPhone and Android Devices with different browsers:

• iPhone 3GS with Safari (Webkit)
• Samsung Galaxy S with Native Browser (Webkit)
• HTC HD with Dolphin Browser (Webkit)

Group 3, latest iPhone and Android devices with latest browser versions:

• iPhone 4S with Safari (Webkit)
• iPhone 4S with Dolphin Browser (Webkit)
• iPhone 4 with Safari (Webkit)
• Samsung Galaxy S2 with Native Browser (Webkit)
• Samsung Galaxy Nexus with Google Chrome (Webkit)
• HTC Sensation with Firefox (Gecko)

Figure 5: touch.xing.com on different devices

By creating priority groups for our test devices and browsers, we have managed to downsize the effort required during developing and testing to make our daily work more efficient. All of the groups are updated from time to time with new devices, software and browser versions.

An overview of our mobile apps can be found here.

If you want more information about our mobile test automation please read the following blog posts.

• Robotium @ XING. Automated regression tests on mobile Android devices
• Keep It Functional – iPhone Test Automation

How it happened

So how do we migrate an entire platform without any trouble? We thought about running both libraries in parallel or maybe moving everything over at once. In the end we chose the second option in order to avoid an increase in asset file size and retain good testability.

The migration

Since we have > 2,500 unit tests we were able to port many core components to jQuery code without running into any problems. After the basic re-usable modules had been changed we worked on the section-specific code. All this happened alongside our normal work, meaning that we had to account for bug fixes and upcoming features as well.

During the process of code adjustment we removed the Scriptaculous effect library and changed to jQuery animate calls. But this also led to a need for a drag-and-drop solution so we built our own lightweight solution. This is only an example as we encountered many more stumbling blocks along the way.

Luckily we were able to get rid of many browser-specific fixes that had already been implemented within the new library.

The devil is in the detail

As you can imagine, things often occurred that we hadn’t thought about in advance. One thing I’d like to mention in particular was where Prototype performed AJAX requests inside of a try catch statement. There we had a bug that no one had noticed for a long time, but fortunately it had no impact on functionality. After first performing integrational tests based on jQuery code we thought the problem was related to changed code so we looked at the JavaScript code for the responsible module. It took some time until we realized it was just a missing property in the JSON response.

The home stretch

A library replacement requires a lot of testing. Three test sessions with the whole QA department had been planned. After each session there was a time when the developers worked on the archived bugs and closely communicated the work state and updated code states for re-tests to the QA employees.

50/50 opportunity

The release date was set for February 29th (yep, 2012 is a leap year). A special plan was made to release the new code in an encapsulated environment. The actual code merge and upload went pretty smoothly and, after some further checks, the changes made over more than four months went live. I can tell you that it was a great feeling and, surprisingly, I slept very well that night. ;)

What happened afterwards

Of course not everything went to plan so we spent the next few weeks investigating the bug reports. I expected more to be honestly. :) But seriously, no major bug took more than a working day to fix.

Answering the main question “from scratch?”

No. The modules already had their own functionality. But in any case, some of the features may not exist anymore or have been solved some other way. Sometimes I felt more like a detective than a frontend developer.

Conclusions

1. If you have modularized code and unit tests you can do a lot of work in advance but there is nothing comparable to a real manual test using different browsers
2. Pending developments need to be considered at an early stage so as not to be surprised by a 5,000-line project coming “out of nowhere”. A version control system like Git helped a lot with this
3. Testing cannot be done by a single person
4. Communication is imperative, e.g. with QA when performing tests and bug fixes, and with Product for freeing up resources and planning support and re-tests, and of course between all of the developers involved in the process
5. Patience and calmness shall be with you

I’ve been working at XING for more than a year now and held this opinion until just recently. My goal as someone who works in QA is, above all, to maintain an overview of everything that goes on in my team and make sure that nothing is released without me having tested it. I had no trouble with this as long as I worked in a Scrum team with four developers. But in the middle of last year we ran into trouble as a new developer came on board, meaning that we started to develop stuff much faster. That’s when we decided that we needed someone else to help out with QA tasks. The reason for this decision is easy to understand as more developers creating more source code means that I have far more to test. Four developers were already doing a good job of keeping me busy, so the hiring of a new developer meant that a new QA team member was required. For the time being we had to wait for a sidekick to join me, during which time the developers got faster and faster and I had more and more to do. But somehow it still worked.

At the end of last year we received the good news that someone was going to join the team to help out with QA. Unfortunately this person didn’t have any QA experience as they were moving from another department in the company and wanted to learn the QA job. So I found myself in a completely new situation as we had enough work for two QA people in the team, but there was actually just me charged with the task of training someone on top of her normal work. At the end of the day the team had just 70% of the QA resources they actually needed.

After a while the team got nervous because QA had become a major bottleneck, so I had to do something. After thinking about the problem for a while I created the following new goal for myself: “I want to know everything, but I don’t want to test everything.”

How can I reach this goal?

I talked to some colleagues about the problem and discovered two major improvements which might help me. First of all I should attend the triage meeting so I’m informed about things much earlier, and secondly I should implement risk-based management.

Triage?! What does that mean? Before a sprint starts, the product owner, technical lead and user experience designer sit down together and talk about any upcoming issues and their technical implementation. A lot of other QA people in my department attend this meeting in their teams to find things out much earlier and to talk about edge cases at an early stage. This helps to improve the quality right from the beginning. Whenever I thought about attending such meetings, I came to the conclusion that it wasn’t the right place for me to think about issues and contribute real input. The results of this meeting are so unspecific that I can’t add much more than the other three people, which is why I decided not to attend the meeting. But what could I do instead? My main task was to think about people who could let me know about important things and are most interested in my point of view. I consider these people to be the product owner and the user experience guy because they design the functionality and naturally want to know what they might have forgotten to think about. So after a short talk with my product owner and user experience guy we decided to create such a meeting, a kind of second triage meeting. This would take place shortly before the estimation meeting, where we talk about upcoming backlog-items from a QA perspective. At this point the backlog items were specified in much more detail, meaning that I could come up with a lot of questions and comments. From then on I was had far more information about upcoming items and my product owner and user experience designer were happy as well as they got a much better idea of backlog item edge cases early in the process and therefore had enough time to properly prepare the backlog items for the estimation.

And what about risk-based management? Of course I already performed risk-based management when I thought about my tasks in the sprint and to what extent I wanted to go through them. As things stood this only involved me, but now I wanted to implement risk-based management for the entire team. The goal was to create risk-based management which determines whether backlog items entail no risk, or a low, medium or high risk. So we needed to ask ourselves if an item with a high risk can break and, if so, what is the worst-case scenario for the user and/or the company. At the end the no-risk and low-risk backlog items can then be tested by the developers while I test all of the remaining items. We initially tried to discuss the risk level with the whole team during the estimation meeting, but we soon realized that the group was too big to discuss it sensibly. So I decided that my product owner and I would discuss the risk level within the new second triage meeting and then present the results in the estimation meeting after that. As a result, every team member is given an opportunity to correct the risk level if we miss something out.

So far, so good

I now received far more information now and had enough time to do my work. The developers were able to work on no-risk or low-risk backlog items while I tackled the medium-risk and high-risk items. That way everyone had work to do until the end of the sprint, which leads me back to my initial question. Today I can happily tell you that I found a way in which developers and QA people are able to finish their work at the same time. And that’s a great improvement for the team, isn’t it?

Having said that, I didn’t feel entirely comfortable with the process so far. What would the quality be like if the developers tested some backlog items without any QA assistance? Would it have any negative impact? I thought about this question and decided to help the developers become good testers. So right now you’re probably thinking that I gave big training sessions to provide the developers with the required testing skills? Sorry, but I’m going to have to disappoint you as I prefer learning by doing. I therefore created a kind of test hint for every backlog item which could be a decision table, users which they should use to test something, or just some hints as to what they should think about while testing their own source code. The feedback I’ve received so far from my developers has been very positive and I also think it’s helped them to get a feel for the work I do.

That’s it! Today I can proudly say that I’m no longer the team’s bottleneck and the new process works very well for my Scrum team. Our part of the product has still the same top quality as before, and my work balance has improved vastly. It’s great to see how these two small changes I made helped to boost the team’s efficiency process quality. How cool is that?

So if someone tells you that you’re the bottleneck of the team, think about it … and try to change it!

This is what our old solution looked like:

HTML

<div class="card-shadow">
  New elements now can be as width or narrow as they want.
</div>

CSS

.two-col-content:after,
.card-shadow:after {
  background: url(data:image/png;base64, foobar) no-repeat;
  bottom: -6px;
  content: ".";
  display: block;
  height: 6px;
  left: 0;
  position: absolute;
  text-indent: -9999px;
  width: 100%;
}
 
.fullsize-grid .card-shadow:after {
  background: url(data:image/png;base64, foobar) no-repeat center top;
}
 
.fullsize-grid .two-col-content:after {
  background: url(data:image/png;base64, foobar) no-repeat;
}

We only had two images in the beginning, both with a white background but with different widths. We implemented each as a background image of the pseudo-element using base64 encoded images (see also: base64 and Data URI scheme). One reason for this approach was that we simply didn’t have a sprite wide enough to host the images.

Over time, our visual designers increasingly expanded the use of the smaller shadow to function as a visual separator. More and more base64 encoded images had to be integrated: full-size, full-size-marketing, half-width, third-width. And with both, white and grey backgrounds. We knew it was time put our foot down and save our CSS from blowing up.

This simple CSS3-based solution came to mind:

HTML

<div class="shadow">
  Wonderful CSS3-card-shadow. Fully flexible. But no IE ...
</div>
 
<div class="shadow-gray">
  Same here, but this box is placed on a gray background, so the horizontal gradient should be gray as well.
</div>

CSS

.shadow:after,
.shadow-gray:after {
  bottom: -7px;
  content: ".";
  height: 6px;
  position: absolute;
  text-indent: -9999px;
  width: 100%;
 
  background-image: -webkit-linear-gradient(left, rgba(255,255,255,1), rgba(255,255,255,0), rgba(255,255,255,1));
  background-image:    -moz-linear-gradient(left, rgba(255,255,255,1), rgba(255,255,255,0), rgba(255,255,255,1));
  background-image:     -ms-linear-gradient(left, rgba(255,255,255,1), rgba(255,255,255,0), rgba(255,255,255,1));
  background-image:      -o-linear-gradient(left, rgba(255,255,255,1), rgba(255,255,255,0), rgba(255,255,255,1));
}
 
.shadow-gray:after {
  background-image: -webkit-linear-gradient(left, rgba(205,205,205,1), rgba(205,205,205,0), rgba(205,205,205,1));
  background-image:    -moz-linear-gradient(left, rgba(205,205,205,1), rgba(205,205,205,0), rgba(205,205,205,1));
  background-image:     -ms-linear-gradient(left, rgba(205,205,205,1), rgba(205,205,205,0), rgba(205,205,205,1));
  background-image:      -o-linear-gradient(left, rgba(205,205,205,1), rgba(205,205,205,0), rgba(205,205,205,1));
}
 
.shadow:before,
.shadow-gray:before {
  bottom: -6px;
  content: ".";
  height: 6px;
  position: absolute;
  text-indent: -9999px;
  width: 100%;
 
  background-image: -webkit-linear-gradient(top, rgba(150,150,150,1), rgba(150,150,150,0));
  background-image:    -moz-linear-gradient(top, rgba(150,150,150,1), rgba(150,150,150,0));
  background-image:     -ms-linear-gradient(top, rgba(150,150,150,1), rgba(150,150,150,0));
  background-image:      -o-linear-gradient(top, rgba(150,150,150,1), rgba(150,150,150,0));
}

Fig. 1: CSS3 shadows, using :before and :after. Two classes are necessary if you want to have the shadow on an background, that isn’t white.

Browser issues: The usual suspect

The above solution is beautiful and elegant – but it doesn’t work on any version of the Internet Explorer. Even the IE9 can’t apply a background gradient to a pseudo-element. To deal with that, we offered our visual designers to put a filter on the parent element. However, that would only have solved part of the problem, the vertical gradient. Just like a box-shadow. So why not a box-shadow? Because we don’t have an explicit IE9.css – and we don’t want one, either. One IE filter should do.

Our UX guys wanted more than just a simple gradient below the parent element. That meant: back to base64. And since we could live with the fact that IE6 and IE7 don’t support :after, we decided to take a more universal base64 approach. This time we chose a png-24 because that got us half transparency. And we need that in order to be able to display the same image on grey and on white backgrounds.

So all we had to do now was to tell the CSS that we want the background image to inherit the width of its parent element.

HTML

<div class="shadow">
  New elements now can be as wide or narrow as they want.
</div>

CSS

.shadow:after,
.two-col-content:after {
  background: url(data:image/png;base64, longer-foobar) no-repeat;
  -webkit-background-size: 100% 6px;
     -moz-background-size: 100% 6px;
       -o-background-size: 100% 6px;
          background-size: 100% 6px;
 
  bottom: -6px;
  content: " ";
  display: block;
  height: 6px;
  left: 0;
  position: absolute;
  width: 100%;
}

IE9 displays everything correctly. IE8 can apply the :after, but not the background-size. We talked with our visual designers again who reassured us that this is a purely decorative element that doesn’t need to be supported by every single browser. Therefore, IE7 and IE8 will simply show a dark line below the parent element (in an separate ie.css).

Our way to finding a solution for a flexible shadow showed us again that it is very well possible to achieve satisfactory results if all teams involved (frontend engineers and visual designers in this case) discuss pros and cons from both a technical and a design perspective. What we got out of it is the use of state-of-the-art technologies as well as a nifty shadow element. True, it’s not supported by all browsers, but that was a price both teams were more than willing to pay. In short: It really helps if you talk to one another.

It all started with the vision

We sorted the points down, analyzed them, studied them and within days we locked ourselves in an onsite location, dedicating all of our time to the Open API. We set the vision for our developer portal site “A scalable and sophisticated developer site with a strong focus on transparency, convenience and innovation that will empower external developers to create compelling applications“.

The team sat together and wrote user stories fitting this vision for almost one day. We narrowed down the whole theme and focused on delivering a Minimum Viable Product by setting a deadline. Then began the task of prioritizing the user stories: We were determined to deliver the minimum viable product as fast as possible to the outside world.

Once the user stories had been prioritized, we defined the technical specifications and narrowed down all the technical feasibilities, including interdependencies within the other standing teams which we depended on. Hence, we were able to stay focused on our target and didn’t have to worry about eventual non-feasibilities that are often detected at the last stage.

Getting it done

Our team consisted of only 5 people the day we started with the brainstorming a few months back. According to our roadmap, we needed for people to speed up the development and also to be prepared for the time after the launch. As we work with Kanban and consider development and testing as one phase, the development phase was very exciting. Most of the time it is QA and developers pairing and giving feedback for the user stories which have been developed then and there. As far as the transparency in this process is concerned, our kanban board says it all.

Since everyone was busy with a different module, we introduced a weekly review meeting for the whole team, so that everybody would stay in the loop as to where we’d stand. We went through the finished modules of the  developer portal, collected suggestions, added more user stories, removed some user stories which we found out could never be an MVP, looked into usability and constantly sought for things that could be improved. Thus, it was easy from the project management side to make a clear determination as to how long we would need to finish the rest of the stories. Immediately after the review meeting we did the backlog grooming to define the size of all the cards and bring them to the wall.

A different view

Once we realized that we were getting close to the final stage, we asked the QA experts working in the different project teams to do an exploratory testing session for our developer portal. The main focus of this testing was to explore the developer portal for bugs and usability issues. Fortunately, nothing critical was found during this session.

The plan had been conceived in such a way that a session based testing would be done within the API team, just to explore the API Explorer. Since all these API calls had been tested, we were almost sure that nothing critical would pop up. So there was a 45 minutes session to test all the calls and a 15 minutes debriefing immediately after the session, just to discuss our findings. As mentioned before, there was nothing critical, but we did have to fix some bugs real quick, since we only had another 2 days left before the release.

And in the end

Once we were sure that everything was working as expected, we released the code and tested the whole developer portal in production. Thanks to a lightning test session carried out by my team, we were fortunate enough to quickly fix some misbehavior we noticed. Now it’s open for about 1,200 developers who registered with us at the first stage via the coming soon page.

So now we are collecting feedback from various channels such as the XING developer group, Twitter, email etc. and we’re further enhancing our developer portal. It’s not too late yet, so if you are a non-beta user and would like to get notified, please register at https://dev.xing.com and follow us on Twitter to get the latest news. To all the beta users out there, please keep sending us your feedback; it is greatly appreciated.