How to Create a Device Matrix for Mobile Application Testing

In this article, we will go over the key parameters to choosing the devices for mobile application testing and creating a device matrix. We will go over market share, OS versions, manufacturers, and screen size and resolution. Also, we will explain the differences between real devices, emulators, and simulators. But before we start, let us share our definition of a mobile application:

A mobile application is a written source code, in a programming language, which is designed for smartphones and operates on an OS, such as Android, iOS, and most recently developed HarmonyOS by Huawei. Mobile applications can be seen in many different areas like banking, communication, social media, games and entertainment, business, shopping and education. The purpose of mobile applications is to connect, educate, entertain, and make our lives easier.

Ensuring that your app works flawlessly on various mobile devices, OS versions and configurations is a complex and challenging task. This is because mobile technology is growing at an exponential rate. Every day, we are witness to new releases, new updates, and new developments. Not to mention, we have older — sometimes even outdated — devices or OS versions that are still being widely used by people. This means choosing the right mobile devices for manual testing can be a little tricky, to say the least.

Testing is a process where we execute a program or application with the intent of finding errors.

To really be sure you are testing your mobile application on the right devices, you need a device matrix. What do we mean by device matrix? Briefly, a device matrix is a collection of devices and specifications that are most relevant to your target users. By creating a device matrix you will be able to better plan your mobile application testing process and obtain accurate, reliable results.

Let’s get started.

Market share

The first step is to identify the devices that are used by your target market. To do this, you should look at the mobile device market share and focus on your demographic area. For example, if your target demographic area is the US, then you should look at the market share there.

Source: Counterpoint

According to Counterpoint’s US smartphone quarterly market data (2020Q1 – 2021Q2), we can see that Apple and Samsung were the most popular smartphone brands during this period.

You should also look at the operating system market share when developing your device matrix. In the images below, we can see what the operating system market share (07.2021 – 10.2021) looks like.

Source: Netmarketshare

Source: Netmarketshare

OS & OS versions

Your app should support different OS and their versions. Having an application that works on different platforms will allow you to expand and branch out into new markets easier. If at any point of the development life cycle, you focus only on one platform you are actually making a terrible mistake. Not only will you narrow down the market share but you will also slow down the expansion to newer markets.

Looking at the OS market share above, we can see that Android is at the top with over 76%, while iOS has a 21% market share. We can then delve deeper by looking at the operating system version market share for Android and iOS, respectively:

Source: Statcounter

Source: Statcounter

As we can see, the Android releases 11, 10, 9, and 8.0 have the greatest market share. In the iOS market share, however, iOS 14.7, iOS 15.0, and iOS 14.8 have the largest version market share. Collecting this type of data is extremely useful when considering devices for mobile application testing.

It’s also worth mentioning other non-standard interfaces like Xiaomi MIUI and Huawei EMUI. Take these OS versions into account as well before making the final decision about your device matrix.

Manufacturers

We all know Apple is the only manufacturer that produces iOS devices. But what about Android? Let’s see below.

Source: AppBrain

Using this data you can pick out a combination of devices that will give you the broadest coverage in terms of OS, version, and brand. Once again, it’s useful to look at your target demographics and prioritize the devices based on this data.

Screen size and resolution

Screen size and resolution is used to verify the UI compatibility and design. As screen sizes and resolutions vary across different mobile devices, it is mandatory that you consider this when choosing the right devices for mobile application testing. You should also decide on the supporting device families, such as Android tablets, iPads, and iPods to name a few.

In the following image, we can see the most commonly used screen resolutions:

Source: Statcounter

According to the data above, the most used mobile screen resolutions are:

• 360×640 – 8.12%
• 414×896 – 7.88%
• 375×667 – 7.87%
• 360×780 – 5.42%
• 360×760 – 5.16%
• 375×812 – 4.71%

Real device vs. Emulator vs. Simulator

Real devices are the various models of mobile handsets that are used to run the website or app in order to test its functioning and behavioral patterns. These are actual handsets that the end-user would use.

An emulator is a simulation that duplicates all the hardware and software features of a real device, while a simulator creates an environment that mimics the behavior and configuration of a real device.

Best practices would indicate that actual development should use emulators and simulators (and a few reference real handsets) in order to speed up the debugging of the application during the coding phase. However, sanity, performance, smoke, interoperability, network feasibility, and regression testing should be done on real handsets.

The traditional approach to meet device coverage involves the use of in-house devices and lab-based emulators and simulators. The cost of acquiring all the mobile devices in the market is extremely high and non-viable. Furthermore, testing your app on a handful of in-house devices will not help you in getting an accurate picture of your app’s health. And depending only on emulators and simulators implies more hassles in configuring them and the risk of getting inaccurate results. So what is the best solution, you ask?

After weighing the pros and cons of mobile emulators, simulators, and real devices, we can say that the optimal mobile testing solution for enterprises is an optimum combination of all three.

Emulators and simulators are suitable for testing in the early stages of application development, while real devices are ideal for seeing how an application works in real user conditions. So our advice is to not put all your eggs in one basket. Instead, you should find the perfect combination to use all three.

In the following table, we can see the difference and importance between all of them.

Conclusion

Always pick real devices which belong to the large screen size category, the medium screen size category, and the small screen size category, with different OS versions.

For all other cases, feel free to cover with emulators IF a real device isn’t an option. This way, all possible existing and future users are covered.

The mobile landscape is highly fragmented, covering many devices across OS, manufacturers, models, and screen sizes. To ensure your mobile application works seamlessly, it’s crucial that you perform mobile application testing on as many relevant devices as possible. Consider the different parameters we mentioned above and create a device matrix that will establish a strong foundation for testing.

As a QA Engineer at TestDevLab, I can proudly say that software testing is our love and passion. We always dare to do more for our clients. With more than 3,500 test devices, we can make sure your app works on all the devices from your device matrix. Get in touch and let’s discuss your project.