Mobile application testing

Share of computers in the global use of the Internet (web browsing) has started to decline regularly in favor of mobile devices since 2009. In 2015, these shares rose and remain at a similar level (52% for mobile). The upward trend in the popularity of mobile devices has slowed down somewhat due to the lifestyle change caused by COVID. About 4.5 million applications are available in official stores on mobile devices alone. In Poland, on average, we install 25 apps a year, and we use about 10 of them every day (30 per month).

This argument is basis to use the Mobile First approach. When analyzing your business, you should first focus on addressing User Experience based on mobile solutions. In this model, we build the most functional and simple interface to address the basic business goals, and then expand the desktop interface.

Sample test scope:

The test scope is a result of the goal to be achieved in the entire process. It may depend on the nature of the application, the quality expectations of stakeholders or the user. In each project, it must be analyzed and adapted to a given product / solution.

An example scope might look like:

1. Application performance tests:
• memory usage
• battery usage
• network connectivity
• running in the background
• switching between applications
• memory leaks
2. Application stability testing
   Checking how the application reacts to being interrupted and how it returns to the previous state (e.g. answering a phone call, messages, notifications, low battery, many hours of operation, etc.)
3. Usability testing
   Carried out to verify that the application is easy to use and understand for the end user.
4. Installation and testing of launching
   Installation testing checks if there are any problems with the installation, uninstallation, and update processes. After installing the application, the startup process is also checked.
5. Functional tests
   All functions and features of the application will be tested to verify that they are operating as specified.
6. Security testing
   Security testing to find application vulnerabilities and prevent data breaches.
7. Compatibility tests
   Carried out to ensure that the application works as expected on mobile devices that are of a different size, resolution, screen, version and hardware.
8. Confirmation Tests
   Performing tests confirming that the given functionality is working properly, and which on the previous version showed problems
9. Regression Tests
   Retry tests that were run before the code changed. Their purpose is to verify that the new or improved functionality has not adversely affected on the operation of the software.

Is it possible to check everything?

Testing mobile applications to a similar scope and with a huge variety of devices entails the need to check the tested product in many variants of individual components affecting the behavior of the application, e.g.:

• Type:
  • mobile internet applications - a website launched from the browser level of a mobile device
  • native applications - applications created specifically for a given type of operating system, using the specific features of these systems
  • hybrid applications - a combination of both of the above applications
• Modes of operation:
  • online - Internet connection is required, e.g. viewing photos, chat with users
  • offline - no Internet connection is required for operation, e.g. a mobile game, which during installation downloads all the necessary activity data files
  • hybrid - an internet connection is not required for proper operation, but from time to time you need to send data from the application to the server

Each component will cause a multiplication of test cases. Testing 100% of the cases is not possible in practice, so the best approach is to define the key personas of the users and define the most common variants in their profile. A good approach (and often enough) is a statistical approach and narrowing down to the most popular variants:

• operating systems in many versions: iOS (<9% market share with version 14.7 and 15.0), Android (91% market share with version 10 and 11)
• differences of the graphic interface of the equipment supplier (UX overlays): Samsung 32%, Xiaomi 22%, Huawai 17%, etc.
• screen sizes – The 5 most popular resolutions address half of the market

Data from 2021 (source: https://gs.statcounter.com/)

Professional solutions and testing also vary deeper parameters, such as 32 and 64bits hardware configurations, or in the case of online applications, network capabilities: WiFi 2.4GHz, 5GHz and mobile networks.

How many times to test?

The amount of test runs is not predefined and depends not only on the test plan, but also on the analysis of the results during and after completion. Testing can also be part of the software development sprint backlog and the scope can be broken down or divided into stages. However, it is worth setting the moment at which the complete run will be performed (according to the test plan) on the software version treated as the first production release (not necessarily commercial). Each subsequent release of new functionalities or patches may be tested with a course adapted to errors previously identified or the scope of changes in the software (confirmation, regression testing).