Mobile Application Test Automation

Mobile Application Test Automation

Author: Mrinal Nandi

Abstract:

This document is to give a brief overview of Mobile application automation testing and to identify the different type of mobile application testing tools available in the market and to analyse the feasibility of these tools to fulfill the business needs.

Introduction:

The market for mobile applications increases every day and is becoming more and more demanding as technology grows. Testing of mobile application is also getting more importance beside the development of application. As with conventional applications, there is an increased need to automate mobile applications testing to improve scale and efficiency. A well-orchestrated automation strategy enables software developers to reduce effort and accelerate time to market.

Few Challenges in Mobile Application Test Automation:

•  Device Diversity:

1.       Multiple platforms and browsers application testing.

2.       Rendering differences.

    ·         Security Concern:

1.       For IOS devices, few automation tools require Jail Breaking to access application property which might change actual device behavior.

2.       For Android device, some tools require root or super user privilege to run application for automation testing.

  •        Hardware Challenges:

1.       Limitations in processing speed.

2.    Limitations of mobile memory size.

.

Type of Mobile Application:

Native Mobile Apps- Built using the native programming language for the platform like iPhone or iPad apps built using Objective-C, and Android application built with Java. Native apps are fast, provide better user experience and interface and have access to all device features for which it is built. On the down side, a native app can be used only for its specific platform thereby restricting the reach. For e.g., an android app cannot be run on an iPhone and viceversa. If you want to cover a larger audience across all platforms, you will need to have separate native apps for them.

Web Apps - A website built using HTML5, CSS3 etc., which resembles an application and can be accessed through a mobile browser is called a web app. The biggest advantage of web apps is that it can be used across all platforms and devices. However, web apps are not accepted in any of the native app stores thereby cutting off an important distribution channel for the app developers. Also, web apps cannot access or use the native APIs or device specific hardware features.

Hybrid Apps –A hybrid app is basically an app developed in combination with HTML 5 and native technology. For cross platform reach, developers would need to code the native part separately for each platform but they can use the same HTML5 part across all of them.

Automation Solution:

Object Identification Technique:

Imaged Based object identification: Record each test object as image and matched these object with run time images available in GUI.

OCR Object identification: The OCR feature is used to obtain the text of a control on the screen. To determine the text, it uses specific algorithms that “read” the text character by character.

Real Object or native object identification: Identify object on unique object attribute like: “id”, “name”, “class”.

DOM based identification: Identify object for web application by using DOM property. 

Automated testing with emulators VS Actual Devices:

 Mobile emulators are powerful tools for developing mobile applications and are widely used for both manual and automated mobile application testing. On the other hand, since mobile applications are used on real handsets and not emulators, testing on real devices during the QA process is required to ensure the highest level of application quality.

The pros and cons of each of these approaches must be carefully considered by enterprises when formulating their mobile testing strategy. Here are some of the main advantages and disadvantages of each approach:

Mobile Emulators: Pros

Price - Mobile emulators are free and provided as part of the SDK with each new OS release.

Simple - Just download the software, install on your PC and you're ready to go. Multiple emulators can be run in a simple and straightforward manner.

Fast – Since emulators are simple client software that runs locally on your PC, they have less latency than real devices connected to the local network or in the cloud.

Mobile Emulators: Cons

Increased Risk - With emulators, by definition, you are not testing on the same platform and network used by your users. This means that even if all goes well, you cannot be 100% sure it will actually work on a real device.

Hardware and Software Differences – Emulators are typically a "plain vanilla" version of the OS and often do not reflect the specific hardware and software features of each supported device. In addition, as time passes following an OS version release, the emulators are not updated to reflect new devices available in the market.

Differences in Computing Resources – Depending on the processing power of the PC running the emulator and the type of handset or smartphone being used for testing, performance on the emulator may be unrealistically good or bad.

Real Devices: Pros

Reliability - Testing on real handsets always gives you accurate results (no false positives or false negatives).

Interoperability testing - Real device testing is typically performed in a live network.

True user experience - Testing on real devices is the only way to truly understand the user experience, taking into account the CPU, memory, screen size, etc. for a given device.

Performance testing - Easier to expose performance defects with real handsets, as well as defects which are the result of the handset itself or its environment.

Real Devices: Cons

Logistics and costs - Best practices call for testing on up to 30-40 devices in your market and replacing ~30% of those each quarter to stay up to date. The efforts involved in procuring and managing these devices are significant.

Type of automation tools:

Device screen sharing using VNC server:Mobile Device use remote display tool (ex: VNC server) to share the device screen with Computer in which the tools is loaded. Tools captured the Image and text available in Mobile device screen available in PC by using tools like VNC Viewer.

Example: EggPlant, Sikuli,TestQuest 10

Advantage:

Easy to create the test script.

Disadvantage:
 Maintenance of the scripts could be very poor due change in screen resolution, device orientation and etc.

Cannot work in complex graphical application.

Jail breaking or rooting might be required to install VNC Server on device.


2.    Application integrated with tool Library:Library or code provided by Test Tool provider integrated or complied with the application to get access of native object.

Example: SeeTest, CloudTest Lite, M Eux(Jamo Solution).

Advantage:

Object identification based on native object.

Create stable script as object identification does not depend on screen resolution or device orientation.

Same script can work on multiple versions of devices as long devices run on same application version.

Disadvantage:

 Library or code file need to compile with actual application to get the access of native objects.

Automation Tools for Mobile application:

Conclusion:

Implementing automation testing in mobile application helps organization to reduce costs with maintaining quality and time-to-market delivery. Several tools are available in market for automating testing of mobile application. All it requires is to choose right tool depending upon various factor of mobile application and business needs.

Taking into account the pros and cons of mobile emulators and real devices, it would appear that the optimal mobile testing solution for enterprises is not "either-or" but rather "a combination of both."

If jail breaking/ rooting is not an issue for our application testing then there are multiple tools are available (Ex: eggplant (Paid), TestQuest 10(Paid), Sikuli(Freeware)). These tools use advance image and text identification technique to identify each object in device screen. 

But jail breaking/ rooting might change actual device behavior  so then we can chose another approach where we need to compile application code with tool’s library. This approach will provide access to application level object and create much more stable scripts than earlier approach. Several good tools available in market which provide application level object as well as Image and Text object identification (Ex: SeeTest, M Eux(Jamo Solution)).

 

References:

www.sqaforum.com

http://experitest.com/

http://www.jamosolutions.com/

http://www.bsquare.com/testquest-automated-testing-platform.aspx

http://www.testplant.com/

http://www.soasta.com/

www.hp.com

 * Disclaimer: This is a personal weblog. The opinions expressed here represent my own and not those of my employer.