Software Testing on Real Devices vs. Emulators/Simulators

Here at TestDevLab, we have been testing on real devices since the company was founded 9 years ago. Now, as of 2023, we have over 3,500 real devices to test software on, including smartphones, tablets, smartwatches, PCs, TVs, drones, and other devices. We believe that the best way to do testing is working with real devices – just like the users would see and use the products.

We use devices running a wide variety of OS versions and resolutions, thus making sure that the tested product runs perfectly on every device. We also use real devices for battery/data usage testing, audio/video quality testing, security testing, performance testing, UI test automation, and more.

Software testing can be carried out using various approaches. Sometimes it is best to use emulators & simulators, but sometimes testing on real devices can yield better results. In this blog post, we will go through the pros and cons of using both approaches.

Testing on Emulators & Simulators

Emulators and simulators are software programs that allow one device to imitate another device. Emulators allow us to mimic the hardware and software of the chosen devices, which is better for testing calculations, interfaces, and compatibility with different firmware versions. Emulators are also more suitable for debugging. Simulators, on the other hand, do not mimic hardware and are more useful when testing interaction with external applications and environments.

This is also a cheap (usually free) way to replicate the environment of real devices – the only thing needed is a computer that runs the corresponding software (or can access a cloud service providing emulation services) which allows emulation and simulation of a wide range of hardware devices. However, there are also a few drawbacks to take into consideration. But first, let’s start with the advantages of such an approach.

Advantages:

• Low Cost: Emulators are mostly freely available and can emulate most of the devices you would need to test. In some cases they allow you to simulate various real-life scenarios like low battery, bad network coverage, GPS coordinates, different OS versions, etc.
• Ease of use: It’s easier and faster to switch device parameters in an emulator compared to managing a wide variety of actual devices. Testing on real devices usually means a lot of hardware is required, which has to be bought, charged, maintained, and managed – that requires more resources. Additionally, these devices degrade over time and might need replacements or service.

Disadvantages:

• Reliability: You can never be completely sure that the testing product will behave the same on a real device as it does in an emulator, which means that in the end, it would still be optimal to retest on the actual devices. You can’t really rely on this simulation to be as accurate as it would be on real devices. This is because the behavior of the testing product might be different in a real-life scenario due to the emulation software missing a few conditions, which could only happen on a real device. These conditions may include:
• The application is not running on the same hardware as in a real scenario.
• The content displayed in an emulator might not look the same.
• The emulated app might react differently to simulated network loss.
• Battery life could affect the application in a way a simulator/emulator can’t.
• Pop-ups, notifications, calls might break your app which is not always the case in an emulator.
• Software and hardware differences: Emulators usually come with an unmodified OS version and don’t run on the actual hardware. Usually, smartphone manufacturers ship devices with their own modified versions of the base OS, which are not covered when using an emulator. Hardware modules such as CPU, network module, camera, GPS, and others might also work differently and may not offer the same functionality you would expect. This is more relevant to Android devices because of the huge amount of differences between various devices made by different manufacturers.
• Computing power: A typical modern computer emulating the device is pretty fast and might not let you notice performance issues. There are cases where real devices would slow down significantly or even hang up due to comparably very limited processing power and memory. However, some emulators allow setting up hardware specs of testing devices in which case this point is not an issue.
• Connectivity issues: The emulator is not connected to the actual carrier network and does not reproduce the same conditions accurately. It also does not cover situations where during the test, the device changes the Wi-Fi network, has a weak Bluetooth connection, receives a call or notification, and other interruptions.

Testing on Real devices

It’s obvious that your customers won’t run the test product on an emulator. This is where real devices come in by allowing testers to test software in a way that it is meant to be used – running on the actual hardware and software. This is also the reason why we consider testing on real hardware a more reliable solution.

Advantages:

• Reliability: It’s always possible to count on results coming from actual devices because the intended product is running in the actual environment.
• Connected to the actual networks: The test product is connected to the actual network (carrier, Wi-Fi, Bluetooth, etc.), which can possibly produce different outcomes to what we would get in an emulator. This includes various levels of network coverages and real devices using different network connectivity technologies. It’s also possible to see what happens when a device receives calls and messages and interrupts the testing process in other ways not possible in a simulated environment. Here, at TestDevLab, we do our own network conditioning and we can test apps up to 5G networks!
• Testing user experience: On the actual device, you can get an insight into how usable is the test product, which is one of the reasons why manual testing is still important. Read more about the importance of manual testing. This way it’s much easier to notice performance issues that can be invisible during testing using an emulator.
• Testing usage parameters: Battery performance and usage can’t be tested on an emulator, and other parameters like data usage may or may not be testable depending on the used software. We have built our own data and battery usage testing solutions that allow us to understand how the app behaves in this regard release by release and against the competition.
• Accurate audio/video testing: Not every emulator supports video and audio inputs, which also has to be considered when deciding what to test on. We also have built our own hardware and software solutions for testing audio/video quality.

Disadvantages:

• Cost: Testing devices have to be bought and managed which can be significantly more expensive than using emulators/simulators. Imagine that the software needs to be tested on the 5 most popular devices of each mainstream manufacturer, on different resolutions, software versions, screen sizes, and various hardware. Assuming the average cost of the device is around 300$ (flagship devices can cost up to 1000$ and even more), budget requirements increase significantly, especially if devices are not going to be used in the future. Fortunately, in the case of short-term testing, the cost can be reduced by using cloud-based services for testing on real devices, however, this is not the best idea for long-term testing as costs can quickly surpass using your own devices.
• Set-up time: It can take a little longer time to set up devices for testing compared to testing on an emulator.
• Managing devices: Each device costs money, has to be charged, updated, tracked. Additionally, its battery will degrade over time and will have to be serviced, and sometimes devices become slow over a lifetime which may make performance tests inaccurate.

Overall it’s great to have and use emulators and simulators due to cost efficiency, however, there are a good amount of risks to consider, which might have a larger impact than expected. That’s why it’s a good idea to use both – emulators only in initial testing phases to uncover basic bugs and afterward switch to the actual devices to ensure great quality of the test product.

We believe that the best approach to testing is to consider both real and virtual devices, and to analyze risks for each testing phase and product, and also take time and budget limitations into consideration to provide the most reliable and cost-effective testing results.

Let’s schedule a call and talk through your situation and options that would be best for your specific project.