Hardware and IoT Device Testing

People can’t imagine living in a world without the internet. It governs many aspects of our lives and is present all around us — even modifying the way we use everyday objects, like cars, phones, watches, home appliances, and furniture, to name a few. Using the internet and advancements in technology, we can add life to static and dynamic objects by making them do something for us.

Most recently, IoT has not only been the driving force behind the urban transformation happening in the spaces we live, work, and play in, but it has also made a great impact in various industries. For example, utility companies and service providers for gas, water, heating and electricity are showing great interest in IoT technology that can provide reliable mobile communication over the 4G NB-IoT and Cat1 narrowband connections.

The IoT market is growing rapidly. According to IoT Analytics, the global number of connected IoT devices in 2021 is 12.3 billion and this number is expected to increase to more than 27 billion connected devices in 2025.

To ensure these connected devices work as intended, performing hardware and IoT device testing is essential. However, not everyone is up to the task. You need to find a quality assurance service provider with the expertise and resources to do it right. Let’s look at TestDevLab’s approach to hardware and IoT device testing.

Overview of TestDevLab’s IoT projects

There are various IoT and wearable hardware devices and services that we have tested over the years. We have tested smart devices that can improve the quality of life in all respects. Let’s take a look at some of the IoT testing projects we have worked on.

Smart sensors for e-bikes and e-scooters

Another project saw us testing smart sensors for e-bikes and e-scooters. For this particular task, we tested the mobile software and the hardware module attached to the e-bike and e-scooter. This module is capable of acquiring the metrics during the ride. The tracker collects the metrics from a special sensor module designed to be connected to the e-bike or e- scooter. The module communicates via Bluetooth and GSM and is able to get the GPS coordinates and send all the information to the mobile companion application. Our engineers have adapted to the client’s development methodology. Based on the requirements, the devices were tested for defects and against the use-case requirements.

LMT IoT

One of the greatest IoT trends in the utilities industry is smart metering solutions. Utility services and other companies in the city have turned to IoT technology to simplify data acquisition, improve customer relations, and receive direct usage feedback.

We tested this type of IoT solution for LMT, the largest telecommunication company in Latvia. LMT’s smart meters collect electricity, heat, water and gas readings using an IoT platform for testing pilot solutions.

For large telecommunication companies like LMT, TestDevLab can help with hardware and software testing of e2e pilot solutions. We help our customers research and develop a proof of concept that integrates third party hardware with the IoT platform. We also advise customers about their choice of devices, harmonize the requirements, and — according to the requirements — build and test the hardware integration setup.

In the smart metering world, TestDevLab engineers have gained practical knowledge of industrial meter protocols like M-BUS, WM-BUS, MODBUS, and others. Also, in line with IoT standards, we can help with the integration of network technologies like Bluetooth, LoRa, NFC. On top of that, we ensure data is transmitted over standards like MQTT, LwM2M, CoAP, OPC-UA, SNMP and similar widely used IoT communication standards in the 4G and 5G bands.

IoT architecture and how we test it

In this section we would like to generalize the architectural aspects of IoT products and describe some tools and approaches we use to test them. In the previous section we introduced several products we tested that share this common IoT architecture.

The field devices and sensors communicate with the smartphone or data collectors that collect the data and send it to a cloud platform. The platform then integrates data from the different devices and shares the most valuable analytics with applications built to address specific needs. These IoT platforms deliver the information to the end user and also allows them to interact with the device and control the environment. This information can be used to detect patterns, monitor the real-time status of the device, and detect possible problems before/after they occur by displaying user notifications or sending alerts from a remote IoT device such as an RTU (remote terminal unit) which is able to communicate with the platform.

Often, problems arise in the communication between the devices and their services. This is why IoT device testing is important. Let’s take a look at the typical components of any IoT solution.

Sensors

Sensors are designed to respond to specific conditions in the physical world which generate a signal that can represent the magnitude of the condition being tracked from physical to digital. Sensors can be attached to any physical device and also to the human or animal body. Physical devices use GPS trackers, environmental data logging devices, or process controllers. In IoT, these devices are interconnected and often stream data back to the application or service. The common IoT sensors that we use include temperature, gas, pressure, level, proximity, motion, and electricity.

In the picture below, we have three electricity sensors (counters). All of these electricity sensors count used electricity with physical pulses which communicate with their own communication protocol. Our goal is to find/test compatible parts to upgrade the already deployed ones to IoT and test the integration and data collection with SCADA or another control environment.

Sensor communication

In the table below we cover three different smart meters which are designed to have a data interface to collect readings remotely. The three meters use different protocols like Pulse, M-BUS and MODBUS. The Pulse protocol is the simplest one and its cost is the lowest as it only carries information about the kilowatts consumed. On the other hand, protocols like M-BUS and Wireless M-BUS allow sending information in a standardized format from a variety of utility meters and connect a number of sensors to the same data bus and gateway to centralize the meter reading data acquisition. The MODBUS protocol also allows control of the device when the gateway acts as a master device to control the connected meters or process controllers.

Some devices only send information, while others both send and receive. While some communications with peer devices are direct, remote communications will often need to pass through a gateway to reach their destination. In order to test the capabilities of the sensors without building an actual sensor grid, we use our knowledge of the protocol to build a simulated sensor network that generates the test data. The sensors often implement a vendor-specific data format and they need to be in line to work with the platform.

In the table below you can see some of the useful tools we use to debug smart meters. We collect the sample data format and these tools replay the data to imitate the expensive equipment and simulate the load before the actual devices are purchased. This helps customers to evaluate the integration aspects and build confidence about the selected solution.

Here are a couple of examples of what is needed for sensor communication testing. For the WM-BUS variant, USB dongle rtl-sdr and Linux wmbus meters are the simplest software/hardware tools to use to test its functionality. We test signal strength (RSSI) based on various conditions — like the distance from meter to gateway — and describe how to decode captured headers from WM meters.

For communication interfaces the best method of testing and investigating M-BUS is via terminal-session based applications — one for testing on a two-wire M-BUS and a second one for testing M-BUS devices on a TCP/RTU network. We have a mini M-BUS master at our premises that can imitate the master/slave connection and individually investigate M-BUS devices using data obtained from the terminal.

Device hardware

In general, IoT devices have several main hardware components which function together but can also be tested separately in-depth. When testing IoT devices we focus on the following components:

• Security
• Communication interfaces
• Features of firmware
• Device configuration
• Power/processing capabilities
• Interoperability
• Radio module capabilities

Let’s look at the radio module, main processor, and power management a bit closer.

A radio module is a device that sends and receives data to things connected to a wireless network. It is used in IoT communication which often uses mobile sim cards or other wireless communication protocols like Bluetooth, ZigBee, LoRa, NBIoT or WiFi to exchange data between electronic devices. IoT module manufacturers like Quectel, SIMCom, u-blox, and Telit are popular IoT radio modules for narrowband and LTe-M communication for which we can separately evaluate the IoT module and data transmission quality. Typical network metrics include the PSM activation time for the module to connect to the cell tower from wake-up, as well as the reaction time for the data to be sent over the network. In an IoT environment, the transmission and wake-up times can be measured in seconds.

Every IoT device has a main processor that runs the device firmware. In our experience, most devices use an ESP32 or similar processor, which is interfaced by Bluetooth, NFC, or USB-Serial terminal commands, to set up the configuration and commission of the device. The firmware variants can be tested separately and device configuration can be validated. We also have experience with the creating and debugging ESP32 Arduino firmware.

Power management on ultra-low power devices and IoT hardware is a crucial part of IoT device testing. The main logic is for the IoT device to be inactive when in sleep or deep sleep modes — when it consumes nano amperes of current from the battery — which allows the battery to last for at least 5 years. Power management is often a part of main processor logic. The processor wakes up the radio IoT module and the field device interfaces during the active data collection and transmission of an alert broadcast. To test power capabilities, we can measure the current consumption (mA) during the run modes and read transmission modes and ultra-low-power (uA) consumption during sleep modes. Our battery test setup is described in our article about battery usage which is also applicable for IoT devices. Using this test setup, it is possible to calculate the estimated battery life of the device in the current use-case.

Testing network and cloud applications

Since transmission requires very little power, this limits the range of communication in network and cloud applications. Because of the limited range, the devices have to operate cooperatively. As a result, the packet size is limited to 127 bytes, while communication is limited to 250kbps. Many alternate protocols have been developed for IoT environments. One such protocol is CoAP, which is a client-server IoT protocol that uses UDP as a network protocol with a server that sends back a response. Another protocol is MQTT, which is a publish/subscribe messaging protocol that consists of broker, client, topic, publisher, subscriber and QoS. For security matters, we have a TestDevLab MQTT broker set up and it is possible to establish an SSL (Secure Sockets Layer) connection and test it with your gateway device. LWM2M is a protocol designed for remote management of M2M devices. It features an architectural design based on REST, defines an extensible resource and data model, and builds on an efficient secure data transfer standard.

After the connection is made, we can also test packet loss and data transmission success. For a more in-depth analysis we are able to do USB debugging by soldering it directly to chip pins and obtaining data as a detailed log from the modem’s radio path. We simulate mobile or wireless network problems with the help of a Faraday box and various thin and large walls between the devices to capture the signal strength based on these conditions, monitor them, and give feedback to clients to help them understand what their devices/meters are capable of withstanding before packet loss occurs. You can read about our Faraday box simulators to understand this process better.

There are a variety of cloud platforms today that allow you to connect IoT devices either directly or through a broker service (edge). Some examples are Google Cloud, AWS, and Cumulocity IoT which are designed for the industrial internet of things (IIoT), consumer, and commercial solutions. For modelling the proof of concept solutions, most of the platforms have a free tier option like ThingsBoard, The Things Network, ThingSpeak where we can help with integrating IoT, Lora, and MQTT based devices. The IoT platform has several crucial functions like device management, data collection, device control, and also the data representation and application layer. From our experience, when the cloud platform needs to bridge third party solutions with their supported protocols, a data translation layer is also required. For cases like these, we can provide test automation of the software services and also mock the real device layer to conduct a simulated load and stress test to ensure the platform is ready to handle the device data.

The bottom line

Hardware and IoT device testing is essential if you want your IoT products to work seamlessly. It ensures all functionalities work according to plan, confirms connectivity between networks, and prevents unexpected glitches from disrupting the user experience. Hardware and IoT device testing can be quite challenging, so the smartest solution is to get help from someone who knows what they’re doing. Get in touch and let’s discuss your project.