Tester's Opine

How risky is the testing of mobile application only on simulator / emulator

2013-11-25T05:05:00.002-08:00

Launching the application based on the testing on emulators / simulators could be very risky for companies / developers and may affect the brand value of the application and company. Developing a mobile application is very easy and quick these days and requires very minimum investment. Now a days with the existence of so many application stores, it is very easy to launch the application and reach to wide user base but if application has issues then it'll get negative feedback even if concept of the application is very good and has the potential of becoming top rated app. This happens mostly because of absence of proper testing of the application from the technical as well as user's point.

Many companies especially start-ups, do not invest much on testing of their mobile application that affects them adversely. Some companies do perform some testing but only on emulators / simulators instead of actual mobile handsets. It is very difficult to test the application on each and every available mobile handset but app should be tested on some candidate handsets. First few iterations of testing should be done on emulators / simulators but final testing should be done on a set of actual handsets. Following are some cons of testing application only on emulators / simulators:

Network issue: Emulators emulate the network speed but emulation is not 100% correct and most of the emulators use the network speed of the PC so performance of AUT on emulator / simulator may not be very reliable.
Resolution issue: Screen size and resolution of a PC also affects the app UI especially in case of gaming apps e.g. user experience on 4 inch screen having 1136x640 resolution is different from that on 4 inch screen having 800x480 resolution i.e user experience changes based on pixel density. Emulating exact screen size and screen resolution is very difficult on emulators as emulators use resolution of the PC.
PC resources: Emulator uses PC resources because of which actual user experience may be very different from that on emulators.
Apps using device camera or GPS cannot be tested properly on emulators.
Carrier specific testing cannot be done on emulators.
It is difficult to do interrupt testing (e.g. Call / SMS etc.) on emulators.

Testers are no more road blocker

2013-04-04T04:46:00.004-07:00

Few days back I attended a conference on "Verification Futures". It was primarily for hardware verification engineers. There were speakers from major EDA vendors like Synopsys, Mentor and Cadence. They talked about current challenges faced by verification engineers and innovations happening in the industry.

A key trend which is observed, is change of focus from design oriented VIPs to verification oriented VIPs which means if anything cannot be tested and then it should not be designed which is very true in software industry also.

As in software industry, focus is moving from first develop and then test to first write test and then develop code for it i.e. TDD (Test Driven Development). As complexity of the softwares is increasing day by day so proper testing of the software is becoming very-very important and now both clients and senior management have realized the importance of testing and hence are more concerned about proper and thorough testing and testing artifacts. Now a days, testers are not just black box tester they dig into code to find probable issues and suggest solutions. Instead of performing only random testing, they are now more systematic and focused, they use both scientific methods as well as their experience. They are expected to have more technical as well as domain knowledge.

In essence now a tester is no more considered as someone who blocks the release of the product but as one who is integral part of the team and who provides confidence in the product.

Tester surprises everyone

2012-12-25T04:30:00.001-08:00

"Oh this is also the functionality, I didn't know", have you ever heard this line from developers, probably yes. Many times we find a scenario which even developers are not aware of. This happens because each developer works on his/her module and may not be aware of the big picture of the product. Many times even product owner gets surprised with the issues which testers find in testing, because even he/she might not have thought from that point.

Product owner usually defines high level requirements so, it is the responsibility of the requirement gathering team to ask correct questions to get details. Many requirements are very obvious for PO since he/she is very familiar with the product domain, but those small and obvious requirements might play very important role in development and testing of the product.

PO looks at the product from user's point of view and a developer looks at the product from technical point of view. But a tester actually looks at the product from both user's point of view as well as technical point of view, so it is very important for him to be part of requirement gathering. Moreover everyone (developer / TL / PM / PO) look at the tester as a domain expert. So it is very important for tester to get some domain knowledge before actually starting testing. He has to think out of the box to find out some very important and severe issues in the product.

How to fire event from AUT to other app using robotium / instrumentation

2012-11-22T02:07:00.000-08:00

IMPORTANT: Code will work only on rooted device or emulator.

Many times we are required to test an andorid app which talks to any other app like (messaging, web browser etc.) As robotium or android intrumention does not allow us to access other apps from within test code we end up testing that feature manually which is a big obstacle in automating AUT.

While trying to automate one of the android app which uses messaging, email and other features of device, we also got stuck with this problem. We googled a lot and thankfully we found some solution to our problem which we think of worth sharing with all.

First approach:
The first and simplest approach is using adb shell command to perform required operation on device/emulator. Here is the sample command where running adb command to come back from third party app.
Runtime.getRuntime().exec("adb shell input keyevent 4");

Here 4 is the keycode for Back key.
This approach is very useful when you have only apk file and you are not able to modify manifest file of AUT.

Second approach:
Second approach is using INJECT_EVENTS permission in the manifest file of AUT. But problem with this approach is you need to put your AUT in sys app folder of emulator/device and for which you need system certificate. If you do not have system certificate then you need to root your device or run this test on emulator. Steps are mentioned below:

Add INJECT_EVENTS permission tag in the manifest file of AUT.
Build AUT with android debug certificate and get apk file.
Download RootExplorer.apk or similar app from internet and install it on desired emulator/device.
Copy apk file of AUT to the emulator/device (DO NOT install yet), preferrably on sdcard. e.g. adb push AUT.apk /sdcard/
Open RootExplorer and go to /sdcard and copy apk file of AUT.
Make sure current mode is R/W. To switch to R/W mode there is a button provided at the top left corner on RootExplorer screen.
Navigate to /system/apps folder and paste apk file of AUT.
Install AUT by tapping on it
Now from your code you can fire any event when AUT interacts with other external activity e.g. If AUT has a button and clicking on it launches web browser then to navigate back from web browser to AUT you can use

Instrumentation inst = getInstrumentation();

inst.sendKeyDownUpSync(KeyEvent.KEYCODE_BACK);

This approach gives us more flexibility and control over other app and is very useful if we have code of AUT.

Testing checklist for Android app

2012-10-25T02:38:00.000-07:00

1. Core functionality of the app is working as expected.

2. UI of the app is as expected like all elements are aligned properly; there is no truncation of element or text etc. UI should be consistent across all screens of AUT

3. Touch area of all touchable elements is big enough so that normal user does not feel any problem in touching them.

4. Font size of all the text should be optimum i.e. neither too small nor too big so that normal user is able to read it with ease.

5. UI remains consistent on rotating the device. If user is reading some text then on rotating device text/list should not scroll to top i.e. it should maintain current position.

6. If on rotating device there is some extra space then elements should be automatically adjusted accordingly i.e. either size of the elements should be increased or other elements (for which user was required to scroll earlier) should be displayed on the screen.

7. Smart phones automatically handle most of the suspend resume functionality automatically but testers should also test it especially in case of audio & video play.

8. If AUT is accessing any third party / inbuilt app like contacts, messaging, email, facebook etc. then it should transfer required data properly and on coming back to AUT should be smooth and should come back to the same screen from where user left.

9. If AUT needs to download very large data then it is recommended to be downloaded asynchronously i.e. initial chunks of data should be downloaded first and user should be able to read text or scroll list while rest of the data is being downloaded.

10. If AUT has some authentication screen like login screen then keep an eye on network log to make sure that AUT does not pass username & password in plain text. Also after logout such authentication tokens / session id should be deleted.

11. If AUT is storing some personal information like authentication token, password, credit card then it should be saved in encrypted form.

12. DDMS should be used to check memory leaks and memory consumed by AUT during run.

13. App should support all the orientation of the device. If not, then both variations of the orientation (in case of portrait mode - upside-down).

14. On uninstallation, all the files related to the AUT should get deleted and other apps should remain unaffected.

15. All navigation should be self-explanatory like Back button should take user to logically previous page, Cancel button should cancel current operation etc.

16. If AUT has scrollable form like screen then currently focused text field should be visible to user i.e. keyboard should not hide it. And on hiding keyboard screen should be adjusted accordingly.

17. Navigation from one screen to another or screen rotation should not be abrupt. AUT should animate components smoothly.

18. It is recommended that AUT should save all large data files on sd card not in phone memory.

19. If app is storing data on SD card then scenarios like SD not available, SD card removed in the middle of data transfer etc. should be tested properly.

20. All the required permissions should be defined in manifest file and it should not have permissions which are not required by AUT.

21. If AUT is a service which gets activated on certain events then those firing of those events should be tested properly. Also if AUT is required to be running in background continuously then scenarios like device reboot, user stops service explicitly should be tested.

22. Switching between Wi-Fi and mobile network should not affect functionality of AUT. Also if device’s Wi-Fi and network are disabled by user and AUT tries to make any network call then AUT should be display appropriate error message to user, asking him to enable Wi-Fi. In such scenario AUT should not display loading bar before should error message.

23. AUT should handle network failure scenario gracefully e.g. device loses network while AUT is downloading some content. In such scenario a proper error message should be displayed after timeout i.e. AUT should not keep displaying loading screen indefinitely.

24. Battery consumption should be under control when AUT is running.

25. Testing AUT using different network connectivity like Wi-Fi, 3G, 2G etc.

26. Special testing should be done to verify how AUT handles foreign characters and special characters.

27. If new version of AUT is available then user should be notified about the update and AUT should be updated smoothly without deleting existing user’s data and preferences.

28. One of the most important aspects is performance of the AUT. AUT should not be very slow otherwise user will move to other similar apps. AUT should keep user engaged and should display results as quickly as possible.

A very good android testing tree could be found here

Running android automation script using batch file

2012-10-24T04:38:00.000-07:00

Run android automation script and copy test result files on local machine (using batch file):

Here, I am taking example of retrieving screenshots taken by robotium. Robotium test script, in this example, runs on emulator/device and takes screenshots at regular intervals. Screenshots taken by robotium are stored in sdcard/Robotium-Screenshots/ folder. For demo purpose, I am running script on emulator but same script can be run on device also.

Note: You can retrieve any file like test results etc. in the same manner. You can even first push test data files on device/emulator and after execution pull the result files.

To run on actual device:

Comment/Remove commands which launches emulator and other emulator related commands.
Make sure device is attached to system and adb devices show the device.

Create a batch file and copy paste following code

@ echo OFF

REM Path of android sdk

SET SDK_DIR=c:\android-sdk

REM Path of folder where screenshots will be copied

SET SCREENSHOT_DIR=c:\temp\screenshot

SET TIMESTAMP=%date:~-4,4%%date:~-10,2%%date:~-7,2%%time:~-11,2%%time:~-8,2%

SET APK_PACKAGE=%1

SET APK_FILE=%2

SET EMULATOR=%3

SET EMULATOR_PORT=%4

IF [%1]==[] GOTO ErrLbl

IF [%2]==[] SET APK_FILE=*.apk

IF [%2]==[] SET EMULATOR=testEmu

IF [%3]==[] SET EMULATOR_PORT=5554

echo Setting path environment variable

SET PATH=%PATH%;%SDK_DIR%\tools;%SDK_DIR%\platform-tools

SET SCREENSHOT_DIR=%SCREENSHOT_DIR%\%EMULATOR%_%EMULATOR_PORT%_%TIMESTAMP%

echo Creating the screenshot folder %SCREENSHOT_DIR%

mkdir %SCREENSHOT_DIR%

echo Launching emulator %EMULATOR%

emulator @%EMULATOR% -port %EMULATOR_PORT% -scale 0.8 -no-boot-anim –no-window

wait 120

echo Waiting for device to come online

adb wait-for-device

echo Press any key only when emulator is completely online.

pause

adb shell input keyevent 82

echo Installing apk file: %APK_FILE%

adb install %APK_FILE%

echo Deleting existing screenshots from device

adb shell rm sdcard/Robotium-Screenshots/*

pause

echo Running test: %APK_PACKAGE%

adb shell am instrument -w

%APK_PACKAGE%/android.test.InstrumentationTestRunner

echo Copying Screeshots from sdcard to local Dir - %SCREENSHOT_DIR%

adb  pull

sdcard/Robotium-Screenshots/ %SCREENSHOT_DIR%\

echo Deleting screenshots from device

adb shell  rm

sdcard/Robotium-Screenshots/*

pause

echo Killing emulator

REM This command does not work on Windows OS but should work on other OS

REM adb  -s emulator-%EMULATOR_PORT% emu kill

REM For Windows here is the workaround which is not good solution but does desired task

taskkill /IM emulator-arm.exe

REM Below command is required to avoid execution of ErrLbl content

exit /B

:ErrLbl

echo Package name is MISSING.

Testing checklist for iOS app

2012-10-24T02:14:00.000-07:00

1. Core functionality of the app is working as expected.

2. UI of the app is as expected like all elements are aligned properly; there is no truncation of element or text etc. UI should be consistent across all screens of AUT.

3. Touch area of all touchable elements is big enough so that normal user does not feel any problem in touching them.

4. Font size of all the text should be optimum i.e. neither too small nor too big so that normal user is able to read it with ease.

5. UI remains consistent on rotating the device. If user is reading some text then on rotating device text/list should not scroll to top i.e. it should maintain current position.

7. Smart phones automatically handle most of the suspend resume functionality automatically but testers should also test it especially in case of audio & video play.

11. If AUT is saving some personal information like authentication token, password, credit card then it should be saved in encrypted form.

12. Instrumentation should be used to check memory leaks and memory consumed by AUT during run.

13. App should support all the orientation of the device. If not, then both variations of the orientation (in case of portrait mode - upside-down).

14. On uninstalling, all the files related to the AUT should get deleted and other apps should remain unaffected.

15. If AUT displays badge on the app icon then it should display correct count and on every update it should be updated accordingly.

16. If AUT has popover then popover should not have any explicit close button. Instead it should disappear when user taps on any item on it and corresponding action should take place or user tab anywhere else on screen.

17. All navigation should be self-explanatory like Back button should take user to logically previous page, Cancel button should cancel current operation etc.

18. If AUT has scroll-able form like screen then currently focused text field should be visible to user i.e. keyboard should not hide it. And on hiding keyboard screen should be adjusted accordingly.

19. Navigation from one screen to another or screen rotation should not be abrupt. AUT should animate components smoothly.

20. Switching between Wi-Fi and mobile network should not affect functionality of AUT. Also if device’s Wi-Fi and network are disabled by user (i.e. Airplane mode) and AUT tries to make any network call then AUT should be display appropriate error message to user, asking him to enable Wi-Fi. In such scenario AUT should not display loading bar before should error message.

21. Battery consumption should be under control when AUT is running.

22. One of the most important aspects is performance of the AUT. AUT should not be very slow otherwise user will move to other similar apps. AUT should keep user engaged and should display results as quickly as possible.

23. Testing AUT using different network connectivity like Wi-Fi, 3G, 2G etc.

24. AUT should handle network failure scenario gracefully e.g. device loses network while AUT is downloading some content. In such scenario a proper error message should be displayed after timeout i.e. AUT should not keep displaying loading screen indefinitely.

25. Special testing should be done to verify how AUT handles foreign characters and special characters.

26. If new version of AUT is available then user should be notified about the update and AUT should be updated smoothly without deleting existing user’s data and preferences.

For more details refer to Apple’s Human Interface Guidelines.

A very good iOS testing tree could be found here

Test WebView used within android native app

2012-10-22T05:19:00.001-07:00

Unfortunately currently robotium does not support testing of webview even if it is used with native app but there is a way to test webview using Android WebDriver. So the solution is using combination of robotium + android webdriver. Here is the sample code to test contents of webview.
Little background about AUT: Native app has a button (assume ‘Google’) clicking on which opens webview with google.com.

Sample Code:
Activity mActivity = getActivity();
WebView web;
ArrayList<View> views = solo.getCurrentViews();
for(View view : views) {
if(view instanceof WebView) {
web = (WebView) view;
break;
}
}
WebDriver driver = new AndroidWebDriver(mActivity);
driver.get(web.getOriginalUrl());
WebElement element = driver.findElement(By.name("q"));
element.sendKeys("Test");
element = driver.findElement(By.xpath("//span[text()='Google Search']"));
element.click();

Testing strategy for mobile apps

2012-10-20T11:25:00.001-07:00

The biggest challenge most of the mobile app development companies face is maintaining huge inventory of mobile devices which requires good amount of investment in procurement of the devices and then maintenance cost and operational cost like data usage etc. Also as new mobile devices are coming in the market every now and then (especially in case of android) so it’s practically not feasible to test app on each and every devices. Because of all these constraints mobile app testers should take balanced approach in testing.

Testing on emulators: Initial functional testing of the AUT should be done on emulators. Today most of emulators and simulators are capable of simulating the functionality of real device so first approach testers should follow is test AUT on emulator with target OS version to uncover most of the functional issues. AUT should be tested on each supported OS version.

Incase of android you can fully configure you emulator like SD card support, keyboard support etc.

Testing on real device: Once you feel that build is stable enough and most of the application specific functional and UI issues have been uncovered and fixed then you should start testing of AUT on real devices to uncover device specific issues. If you have all the devices available then you should perform testing on each device otherwise you should select required devices intelligently like one device of each target screen resolution and one device of each manufacturer, one device of each OS version. You should talk to developers to know if developers are using any OS version specific APIs etc.

Note: One important point to note here is you should select devices based on targeted mobile operator like AUT might work differently on two iPhones, one on Verizon and other on AT&T. This happens because of different technologies on both. And also sometimes because of customization done for mobile operator like one operator might block some APIs etc.

Testing on virtual devices: Today we have virtual environments like DeviceAnyWhere, PerfectoMobile etc. They have very good device inventory which you can use to test your AUT. They are very helpful as you need not to maintain huge device inventory in your organization. But as they charge on hourly basis, you need to make sure that you use them intelligently. When you have tested on app on different combination of screen resolution, os versions etc. then you could DAW etc. to sanity test your app on other devices with same configuration just to verify AUT works perfectly on them also. Before using them, make sure you have very good network connectivity otherwise you might observe some issues which might not occur on real devices.

Test case writing techniques:

2012-09-27T09:28:00.000-07:00

Most of the time we starting writing test cases based on our experience and intuition and sometimes we end up writing some useless or repetitive test cases (in the sense that specific scenario might have already been covered in some other test case). I am not saying that we should not use our past experience but experience should be used along with some scientific techniques.

There are many techniques which can be used (and should be used) to write test cases (especially test data). These are mainly test data generation techniques which should be used to write effective and optimized test cases. Some of the techniques are like Boundary value analysis, equivalence partitioning, orthogonal arrays etc.

BVA and EP should be used to generate valid and invalid data for each field. Orthogonal array is used when we have multiple fields on the screen and we want test data with different combinations.

Boundary Value Analysis: In BVA, we generate test data at the boundary of the field i.e. if any field accepts numeric value between 18 and 60 then test data for that field should be 17,18,60,61.

Equivalence Partitioning: In EP, first we determine valid range of the field for example in our example valid range is 18 to 60 so test data should be selected as:

Valid range: 18-60: any one value

Invalid range1: -ve no. to 17: any one value

Invalid range2: 61 to +ve no.: any one value

Invalid range3: any alphanumeric value

Orthogonal Arrays: Above two techniques are used only for single field. For different combination of multiple fields orthogonal arrays are used. For example if there are 5 fields on a screen then orthogonal arrays should be used to come up with minimal test data required to cover maximum test functionality. There are various tools like AllPairs which generate test data for combination of fields. AllPairs is a very simple and free to use tool which reads data for each field from file and generate test data file.

Finally some test cases should be created based on experience. Some of the examples are:

1. If field accepts alphanumeric values then some test cases for:

a. Special characters

b. Escape characters

c. Specific special characters combination like ';-- (database comment)

d. leading/ trailing blank spaces

e. foreign characters

and some other project specific special requirements.