This whitepaper speaks about how Mobile Apps play a key role in daily business functions and the changing trends of consumer web-usage along with a brief summary of mobile application performance drivers...

1. Mobile Performance Testing
Approaches and Challenges
INFOSYSTEMS
LEVERAGING INTELLECT
NOUS

2. Mobile devices are playing a key role in daily business functions as
mobile devices are adopted by most enterprises as a business tool. A
smooth functioning mobile apps are more critical today. Issues such as
performance or downtime are a greater concern, as they are linked with
revenue or brand image. This has made Mobile Performance Testing an
important stage during development
In the fast changing world, it is essential to address mobile performance
testing approach and challenges; this is especially true when it comes to
mobile where end users now expect the same level of performance from
mobile apps as they do from web apps.
In the real world, applications will be accessed by many simultaneous
users with different devices and various network conditions. In order to
understand the complete mobile user experience for an application, tes-
ters must be able to replicate a realistic load on the back end of an appli-
cation while measuring the performance of the application front end on
real devices.
In this paper we addressing, mobile performance testing approach and
challenges, which helps to performance testing team to drilldown the
performance issues which quick turnaround time
INTRODUCTION
In this upwardly mobile and fast paced world, it has become essential to be con-
stantly connected with the customers even as they go about their daily lives. It is ob-
vious that mobile usage has drastically increased in the past few years, as people
are using their mobiles to browse websites and ultimately make a product purchase.
According to a recent research report, internet usage on mobile phones in July 2013,
accounted for 17% of global internet usage. Another study estimates that the total
number of mobile phone users will reach 5 billion by 2017. In light of these statistics,
businesses need to connect with the user on their mobile.
ABSTRACT
Mobile Performance Testing Approaches and Challenges | 2

3. The below figure shows how mobile users affect a mobile app as compared to web
users:
The Internet usage on mobile phones has increased drastically over the last few
years. According to a recent research report, internet usage on mobile phones in
July 2013, accounted for 17% of global internet usage. Another study estimates
that the total number of mobile phone users will reach 5 billion by 2017.
CHANGING MOBILE SCENARIO AND TYPES OF APPS
Figure 1: How mobile users affect our application
However, mobile phones have their limitations, some of these include the varying
display size amongst mobiles; market is shared across multiple operating sys-
tems; RAM and CPU have constraints and network infrastructure offers much
lower bandwidth. The main challenge businesses encounter is delivering a fast
functioning, high performing and engaging mobile app across all devices, includ-
ing mobiles with lower hardware specifications; while retaining their end-custom-
er’s attention. This makes performance testing for the mobile a critical activity as
compared to performance testing for the desktop. The performance testing for
the mobile should be in-depth focused on back-end performance but the
front-end performance as well.
With growing number of mobile internet users, the expectation for seamless
mobile web experience has gone up as well. There are few such studies that indi-
cate this trend;
• Mobile users will wait for a maximum of five seconds for a website or
app to load
• 74% will leave a mobile website if it doesn’t load in five seconds
• 50% will exit an app if it doesn’t load within five seconds
• 71% of global mobile web users expect websites to load almost as
quickly or faster on their mobile compared to desktop
• 1/3rd of the mobile users will go to a competitor’s site if the website or
app is slow
• 80% of mobile web users don’t want to try a website or application more
than two times
Mobile Performance Testing Approaches and Challenges | 3
Given this scenario, mobile performance testing needs to be customized based
on the type of application. We need to effectively target the processes and com-
ponents involved in these different types of applications.

4. Mobile Performance Testing Approaches and Challenges | 4
The performance test of the mobile app should target every component in the app
architecture. The key components for a generic mobile application are
represented in the figure below.
The performance of a typical mobile application depends on three key factors:
MOBILE PERFORMANCE TESTING DRIVERS
Mobile
(Mobile Network
(2G/3G/4G etc...) Server
Figure 2: Key Components Governing Mobile App Performance
Back-end processing time:
This is the total time taken by the request to reach the server, response generation
time by server and the time to deliver the response back to the client. This time is af-
fected by the total load on the server, network bandwidth, data transferred and the
external component (if any).
Network Emulation:
Networks have a wide range of variability when it comes to mobile internet usage
and this makes network emulation important for mobile performance testing. We
need to effectively emulate the network condition to replicate real-time traffic
during performance testing.
Predominantly we have three types of apps:
Types of Apps
Browser Based Apps Native Apps Hybrid Apps
Due to the differences in
bandwidth, network data
plan limitations, device
configurations a desktop
browser-based mobile
applications might perform
differently. Most of the
mobile browser based
application looks like, m.
example.com. While, doing
load test for a website, it is
equally important to test the
mobile browser apps also.
By installing the app directly
on to mobile devices, native
mobile applications allow
users to access the
applications quickly. Native
apps are platform dependent
and due to this the
development cost is also
high. This app can be used
with or without Internet.
Hybrid applications
can be called as a combi-
nation of both browser
as well as native apps.
Hybrid apps are devel-
oped using web
technology and wrapped
with platform specific
shell, allows the app to
be installed like Native
app. Due to this, the de-
velopment cost is higher
than others.
Content rendering time at the device:
The time it takes to display the response delivered to the client on the device is
measured. The platform type, configuration of the device and other hardware
configurations affects this time. This time should be handled using an on-device
monitoring tool.

5. Mobile Performance Testing Approaches and Challenges | 5
Load Testing Tool
Dashboard
Real DevicesVirtual Mobile
User on cloud
Virtual Mobile
User on cloud
Firewalls
Figure 3: Recommended approach to Mobile Performance Testing
PHASES FOR MOBILE PERFORMANCE TESTING:
Planning, Develop Load Test Scripts, Execution Reporting
Planning
A well planned performance test methodology/strategy drives the success of a
performance testing project and its execution. While designing a performance
test plan, engineers need to forecast for the end user trends, apply right patterns
else the load simulations will not generate results.
There are a few key parameters we need keep in mind when the mobile perfor-
mance testing is conducted:
• Type of application (Hybrid, native, browser based)
• Type of Network (broadband, LAN, mobile, wireless)
• Browser simulation (Android, Blackberry, Symbian, Windows, iPhone, iPad,
Tablets-Android)
• Bandwidth simulation
• Users load/patterns
Few challenges that impacts the performance testing process in mobile world are
as follows
CHALLENGES FACED DURING PERFORMANCE TESTING
THE RECOMMENDED MOBILE PERFORMANCE
TESTING APPROACH
The recommended approach to testing the performance of a mobile application
is as follows. The recommended approach provided here, utilizes Neotys Neo-
Load, HP LoadRunner as examples.
• Load the application servers using traditional load testing solutions
• In parallel, the mobile performance will be checked with a minimal feasible
number of users leveraging real devices or mobile cloud solutions.

6. Mobile Performance Testing Approaches and Challenges | 6
Edit > Preferences > General settings > HTTP Recorder > Tunnel Mode Recorder set-
tings
Figure 6: Neotys NeoLoad Settings for Tunnel Mode
Source: Neotys NeoLoad
c. Recording using Native Recorder
Costly performance testing tool has launched its own mobile testing application
using LoadRunner Mobile Recorder, to record directly on to the mobile. For this, the
device must be rooted to record. Though this option we can measure the end user
experience and also create a load on the servers.
d. Recoding with emulators
During performance testing of mobile applications, we need to simulate different
networks (e.g. 2G, 3G, Edge, WiFi etc.) to reach the real-time traffic. We can enter
the real time bandwidth, latency and packet loss values using WAN Emulation.
Mobile User Activites
Recording in tunnel mode
WebServer
Figure 5: Recording in Neotys NeoLoad Tunnel Mode
To make use of Tunnel Mode it is necessary to activate the Tunnel Mode parame-
ters. This can be done by applying the below settings. Also Neotys NeoLoad has an
option to record desktop browser as a browser on a specific mobile devices.
a. Proxy Mode Most load testing tools support recording the user actions
though proxies. We need to set up a proxy into the device or on an external ma-
chine and it records all the communication done on the device afterwards.
b. Recording in Tunnel Mode
If your application cannot be used via proxy or the mobile device doesn’t support
proxy, Neotys NeoLoad has recording capabilities using tunnel mode. Tunnel
mode makes it possible to record any application, either web or native, standard
or customized, for any platform in any version.
i. Recording directly from the device itself
a. Using the proxy mode as available with Neotys NeoLoad and other tools
b. Using the tunnel mode as available with Neotys NeoLoad
c. Using a Native app (recording application) like HP LoadRunner VuGen
mobile app
ii. Recoding with emulators
iii. Recording on a desktop web browser (for pure HTML applications)
Mobile User Activites
Proxy Server
WebServer
Figure 4: Recording in Proxy Mode
Develop Load Test Scripts
With the increasing importance of mobile as a platform, several leading perfor-
mance tools have started supporting mobile performance testing. Mobile appli-
cations can be recorded in many ways:

7. Mobile Performance Testing Approaches and Challenges | 7
Figure 8: NeoLoad integration with Perfecto integration using Custom Action
Figure 9: NeoLoad Runtime dashboard
Execution
Once performance testing team complete building an automation perfecto test
script and NeoLoad load creation scripts, you can apply realistic traffic conditions
such as browser, load mix, bandwidth, and networks conditions to generate a realis-
tic load on the server.
While application under load performance testing team can monitor server’s re-
source utilizations and real devices metrics on a single dashboard, which helps to
analyze the performance bottlenecks in quick turnaround time. Performance testing
team will able to see how the DEVICES +SERVERS utilizations at various load levels
on various devices (Android, iOS, Windows Phone, BlackBerry etc.).
Figure 7: Recording using Desktop Browser
Source: Neotys NeoLoad
e. Recording using Desktop Browser
Neotys NeoLoad gives you an option to emulate desktop browser as a browser
on a specific mobile device.
2. Perfecto Test Script
Perfecto mobile is a cloud service provider, it is an Independent from hardware
platforms, OS and browser, Perfecto Mobile enables creating and executing
scripts on a wide range of physical and online virtual hardware. Pick most fre-
quently used device which you intended to measure transaction response times.
Which helps performance testing team to measure /monitor real time mobile end
user experience by running Perfecto Test Scripts along with NeoLoad test
scripts.
Possible test profiles
• Running one script on several mobiles
• Running several scripts on one mobile
• Running several scripts on several mobiles

8. • Use HTML5 web storage for browser caching, which is an alternative of
browser caching.
• By in-lining resources instead of using them as linked references it can greatly
increase the page loading time.
• Bandwidth consumption can be lowered by compressing and reducing page
resources to make it faster.
• By resizing and optimizing images for the mobile screen, we can not only
reduce the page size but also improve the processing power.
Conclusion
The popularity of mobile applications has grown rapidly and is increasing continu-
ously therefore there is the growing need for performance testing, to remain compet-
itive in the market and have engaging mobile applications. This will help deliver a
better user experience, which remains most important factor to attract and retain
mobile users.
Load testing on Mobile apps is similar to the traditional desktop apps. However, Per-
formance Engineers need to know the all types of architecture and the fundamental
differences between Mobile and desktop application to test all parameters success-
fully.
AUTHORS
Sahadevaiah Kola, Senior Test Lead
Sachin Goyal, Senior Test Engineer
Key Points – A Quick Recap
• Mobile performance testing approach and challenges
• Phases of mobile performance testing approach
• Methods to build load test scripts and perfecto test script
• It is essential to consider the mobile end users real time conditions to
simulate a load
• The mobile performance with single user should be monitored in parallel with
the load on the server.
• Leveraging one script solution for both purposes – loading the server
and single user mobile rendering could be time saving.
• Mobile performance optimization techniques for developers
Figure 10: Sample test report with different load levels vs mobile devices
Reporting
Unlike traditional performance test metrics few parameters specific to mobile
need to be understood clearly when results are presented. This includes devices
level statistics namely:
1. Memory Free (KB) 2. Memory Cache (KB)
3. Memory Used (KB) 4. Bytes In-Mobile
5. Battery Level (%) 6. Bytes Out -Mobile
7. CPU User (%) 8. Bytes Out -WLAN
9. Bytes In –WLAN
The below graph show the average response times for the key business operation
with different load levels vs real devices
Hopefully, you now understand a lot more about the challenges associated with
mobile testing of native and web applications. Our recommendations for opti-
mizing mobile performance are listed below.
• Use less number of HTTP requests between the client and server; this will
greatly improve the page loading speed.
• Maintain a constant HTML code for all devices; to change the user interface
for different devices. Use CSS media queries to specify which CSS rules should
apply for the browser displaying the page.
MOBILE APPS PERFORMANCE OPTIMIZATION
TECHNIQUE FOR DEVELOPERS
Testree is the Independent Verification and Validation (IV&V) division of Nous Infosystems. Testree has strong expertise in functional, non-functional testing, migration and consulting and advisory services on testing. Our deep
domain knowledge lies in Banking & Finance, Insurance, Healthcare & Life Sciences and Retail. We work with industry proven proprietary frameworks and commercial tools to provide precise testing solutions, applying some
of the emerging technologies. Testree’s working models include onsite testing, offshore testing, combination model and through dedicated test centers or through domain focused test labs. With over 18+ years of testing
experience and success of over 100 testing engagements, Testree offers all-inclusive testing solutions with critical focus in automation, performance and mobility testing.
For more information, please visit www.testree.com and www.nousinfosystems.com
© 2015 Nous Infosystems Pvt. Ltd. All rights reserved.