Create your own app – Part 4: App and Usability Testing

As soon as an app is completed, the next step is app testing. The app is tested for func­tion­al­ity and usability as well as its com­pat­ib­il­ity with different devices and operating systems. Most de­velopers want of course want to publish their ap­plic­a­tion as soon as possible after de­vel­op­ing it and setting the final app design – but before the software is offered in the app store, it is important to test its suit­ab­il­ity across various cat­egor­ies. Common methods for de­term­in­ing this are per­form­ance, com­pat­ib­il­ity, and usability tests. These and other testing methods are grouped together under the term “mobile ap­plic­a­tion testing” (or “mobile app testing” for short). Different app testing tools are used through­out the various stages of this process.

In order for an app to be suc­cess­ful, it not only needs to offer im­press­ive content and be well-marketed, but above all must work flaw­lessly and be user-friendly. Therefore, fun­da­ment­al testing before the release of an ap­plic­a­tion is vital – whether it is a mobile app or a desktop ap­plic­a­tion.

Testing apps is, however, becoming more difficult because of one in­creas­ingly complex factor: ad­apt­a­tion to different user devices is con­sid­er­ably more elaborate for apps than for other software. In the early years of app de­vel­op­ment this ad­apt­a­tion was easier to manage due to the re­l­at­ively small selection of smart­phones on the market. Since then, a huge selection of mobile devices has emerged with a variety of different features. Most smart­phone and tablet models function on roughly the same quality of hardware, but installed com­pon­ents such as the processor (CPU), memory (RAM), screen size, and screen res­ol­u­tion are the main factors de­term­in­ing the rendition of an ap­plic­a­tion.

In addition to its hardware, the software of a device also plays a role. Software mainly contains the different operating systems of a platform, which influence app rendition. Older operating systems can make app usage more difficult due to missing updates, and the pre­val­ence of old Android and iOS operating systems should not be un­der­es­tim­ated. The variety of operating systems in use is par­tic­u­larly evident when looking at Android systems: On the whole, the older versions of Google’s operating system are used more commonly than the current version. This has to do with the fact that many Android man­u­fac­tur­ers either deliver updates late, or don’t deliver them at all. With iOS the new version of the operating system is generally delivered more quickly, but some of the oldest devices no longer receive updates from Apple, therefore old iOS versions can still be found.

The variety of hardware and software for mobile devices is called frag­ment­a­tion and it presents many chal­lenges when it comes to app quality assurance. This is where mobile app testing comes into play: This checks whether an ap­plic­a­tion is even capable of running under different con­di­tions, and if so, whether the con­di­tions present any dif­fi­culties for playback of the app. Detecting and cor­rect­ing such issues is necessary to ensure an optimal user ex­per­i­ence for as many users as possible. Only when an app offers good per­form­ance and ease of use for as many devices as possible can it reach its maximum potential.

App testing is not all done the same way. There are many different ap­proaches to quality assurance. The two main questions to address are who should check the quality of the ap­plic­a­tion (either manual or automated tests) and how the tests should be conducted (tests directly on the re­spect­ive devices or tests via a simulator or emulator).

Manual app tests (carried out by users) The manual test is the classic approach to mobile app testing – in this case, real people test the ap­plic­a­tion in question. Since the testing of ap­plic­a­tions can be quite extensive (es­pe­cially with Android apps), the de­vel­op­ment team usually cannot do this by them­selves. Con­sequently, test subjects are used. A group is assembled (rep­res­ent­at­ive of a larger group, in the best case scenario) which then uses the app according to pre­de­ter­mined test scenarios and sub­sequently evaluates it. Outside test subjects have the advantage that they do not have any prior knowledge of how the ap­plic­a­tion should work, and therefore can evaluate with judgment close to that of a potential user. In addition to the manual mobile app testing, crowd testing is another classic approach. Crowd testing services offer a pool of testers, which allows de­velopers to choose targeted test persons to test the app on their own personal devices. Crowd testing does not take place in a special testing en­vir­on­ment, but rather in the natural en­vir­on­ment of the users, which can increase the validity of the test results. For iOS apps (as well as watchOS and tvOS apps) de­velopers can use the Test­F­light Beta Testing service to invite up to 2,000 users to test the ap­plic­a­tion and organise their own crowd tests. Google also offers software de­velopers various options for private and public beta testing of Android apps. Automated app tests (carried out by app testing tools) In contrast to manual tests, automated controls are used in areas where certain processes are con­tinu­ously tested under different con­di­tions. For automated app tests, de­velopers need to use a special software. The main advantage is how much time it saves – test pro­ced­ures can carry out pro­grammes in a fraction of the time that a user would need. However, auto­ma­tion is not suitable for all test fields.

Tests on physical mobile devices

Testing apps on in­di­vidu­al mobile devices is the ideal solution for being able to un­der­stand the app in com­bin­a­tion with specific software and hardware. However, this is quite expensive and time-consuming, because a large number of smart­phones and tablets are required for com­pre­hens­ive tests, and the behavior of the app has to be con­stantly be checked in detail on each device. The easiest way to test using this method is by using various mobile app testing vendors, which provide access to a variety of mobile devices (often called “device lab” or “test lab”) that can be used to test the app.

Tests via Simulator/Emulator

Sim­u­lat­ors and Emulators are pro­grammes that imitate par­tic­u­lar devices and allow ap­plic­a­tions to be tested virtually. However, the results obtained from these tests are not always 100% reliable, as not all user in­ter­ac­tions can be au­then­tic­ally simulated. In addition, some tests should be carried out on physical devices as the sim­u­la­tion can reach its limits in certain areas.

Sim­u­lat­ors and emulators can ad­mit­tedly not replace tests on physical devices, though they are a good addition and par­tic­u­larly suitable as an automated testing tool. With their help, it is possible to quickly test the func­tion­al­ity of an app with different hardware and software com­bin­a­tions. Many sim­u­lat­ors and emulators are also available for free – for example, those contained within the Android Studio and Apple’s Xcode pro­grammes.

In addition to the various methods for testing an app, there are also various areas of testing. Apart from the tests mentioned in this section, there is also safety testing, security testing, and loc­al­iz­a­tion testing, as well as other test types and areas.

By using func­tion­al­ity tests, a developer can check that the app is really working the way they want it to. Ul­ti­mately, all elements of the app will be checked here for correct operation – every function, screen, and button will be inspected. Func­tion­al­ity testing focuses on the following central questions:

• Is the app easily installed and un­in­stalled?

• Do all app elements function as intended?

• Can the app continue to run smoothly when minimised?

• Does the app crash?

• Does the user receive an ap­pro­pri­ate error message when an error occurs?

In addition to the app itself, tests should check the behavior of the ap­plic­a­tion in the certain following scenarios:

• Can other ap­plic­a­tions and functions of the device be carried out when the app is minimized?

• Does the app auto­mat­ic­ally minimise when a call is received?

• Are SMS and other no­ti­fic­a­tions received and stored during app usage?

App usability testing checks how user-friendly the operation of the ap­plic­a­tion is. Automated tests are not an option here – after all, only users can assess whether the how well they think an app works. The use of a simulator or emulator is also generally not helpful for usability testing, as app operation can only truly be evaluated by usage on a mobile device; only when using the actual ap­plic­a­tion can control pro­ced­ures such as swiping, scrolling, and pressing buttons be felt and their reaction times and behaviors be evaluated. Other aspects that are examined in usability testing are as follows: 

• Is operation of the app largely self-ex­plan­at­ory? Are ex­plan­a­tions provided for more com­plic­ated functions?

• Are app elements well placed, and important functions easily located by the user?

• Is the placement of elements and layout of the user interface organised in a way that is intuitive and easy to un­der­stand?

• Are the buttons optimally po­si­tioned and large enough to ac­com­mod­ate users with wider fingers?

• Are texts clearly written and formatted in a way that’s easily readable and ap­pro­pri­ately sized?

Usability testing should discover any problems with the operation of the app or short­com­ings that detract from its ease of use. In order for the results to be con­clus­ive, the usability test must be carried out by a suf­fi­cient number of test users. It is also ad­vant­age­ous if the test users (at least a majority) are people from the target group – that way the operation and structure of the app can be optimised for expected users. The resulting im­prove­ments allow the developer to offer the best possible user ex­per­i­ence.

The com­pat­ib­il­ity of an app in regard to the hardware and software of mobile devices should always be tested. In terms of hardware, one important example is testing the rendering of the ap­plic­a­tion on different displays. Because there are different screen sizes and res­ol­u­tions, display and cus­tom­isa­tion errors can occur re­l­at­ively quickly, es­pe­cially if several layouts of the ap­plic­a­tion for different screen formats have not already been created. In addition to the display format, CPU and RAM vary from device to device. Keep in mind that older mobile devices are not a rarity. Apps that are designed not only for current models, but also for older smart­phones and tablets, auto­mat­ic­ally increase their market range.

In regard to device software, the smooth playback of an ap­plic­a­tion is largely con­trolled by the different versions of the operating system – Android app testing on various versions of the operating system is es­pe­cially important, as this platform has sig­ni­fic­antly more versions with more variation of use than Apple’s iOS platform.

Com­pat­ib­il­ity testing allows de­velopers to check which hardware or software processes cause immediate problems with rendition. If the re­quire­ments for the device com­pon­ents are very high, it may be helpful to reduce certain system re­quire­ments in order to avoid shrinking the target group.

Per­form­ance tests show how many system resources an ap­plic­a­tion consumes during intense use. Does the app continue to run smoothly, or does it ex­per­i­ence lapses? Beyond that, per­form­ance tests can determine if a device’s battery will become over­loaded or if the device itself will overheat after a certain period of use. As with com­pat­ib­il­ity testing, tests on the per­form­ance of the ap­plic­a­tions are con­trolled under certain con­di­tions – for example, how the playback of the app changes if there is hardly any space left on the device’s hard drive or if the battery is almost empty. Per­form­ance testing is intended to reveal the extent to which certain cir­cum­stances neg­at­ively impact the app’s per­form­ance.

If one or more com­pon­ents are going to be subject to high util­isa­tion, then a load test (or stress test) is used. One example of this is the server stress test, where the app’s behavior is tested in reaction high server demand. For ap­plic­a­tions that require an internet con­nec­tion, a network load test is conducted to see how a weak internet con­nec­tion affects app func­tion­al­ity. Stress tests are designed to show how much stress certain com­pon­ents can take before their per­form­ance becomes re­stric­ted.

The above tests don’t ne­ces­sar­ily have to be carried out manually. Diverse testing tools can make the work con­sid­er­ably easier in many areas of app testing. Seeing as this article series discusses mobile ap­plic­a­tion de­vel­op­ment for both Android and iOS, the following app testing tools are those which are con­sist­ently available across both platforms.

Appium is a cross-platform open source programme for test auto­ma­tion that is available for download free of charge. This tool doesn't only test native Android and iOS apps, but also supports test auto­ma­tion of hybrid apps and web apps. Appium uses the same pro­gram­ming interface for both iOS and Android, so it is easy to reuse pre­vi­ously written code. The created tests can be carried out on physical devices, sim­u­lat­ors, or emulators. Appium is in­cred­ibly helpful software for the execution of automated app tests, however, the programme does require the user to have a certain level of pre-existing know-how.

On TestFairy’s website, de­velopers can upload their app and invite people to par­ti­cip­ate in app testing. These people can then download and use the ap­plic­a­tion – es­sen­tially, TestFairy allows for self-organised crowd testing. The tool has an error reporting feature and can record video of app test usage. TestFairy is available as a free download and also a paid en­ter­prise version.

Uber­test­ers is a platform that helps with beta testing for apps. De­velopers have complete control over the tests and receive detailed reports. On the platform it is easy to work together as a team, detect pro­gram­ming errors using various tools, create and test different versions of the app, and perform crowd tests (for example, in the form of func­tion­al­ity or usability tests). Uber­test­ers is offered in various packages, each with different features – the package with the fewest features is available for free download.

Obviously, the reach of an app can be extended by designing it to run on many older devices. The ad­apt­a­tion of an app to the largest possible number of devices, though, has its limits. CPU-intensive ap­plic­a­tions, for example, whose functions require a stronger processor and/or more memory, can sometimes not be prepared optimally for older hardware. Another example: if an app requires a powerful internet con­nec­tion, it should ab­so­lutely not be adapted to any usage situation in which there is only a weak internet con­nec­tion.

Some ap­plic­a­tions simply require certain technical pre­requis­ites. For example, some ap­plic­a­tions require a par­tic­u­lar screen size in order to run. Another example is the YouTube app, which can only function with access to a suf­fi­ciently fast data transfer rate; yet it is still one of the most down­loaded ap­plic­a­tions ever. Ad­just­ments and re­duc­tions of func­tion­al­ity in favor of a large app range should only be made if the app func­tion­al­ity is not reduced more than desired.