How We Simulate Real-World Network Conditions For Testing

In most developed countries, with the appearance of 5G, optical fiber connections and advanced infrastructure networks reach speeds of 100Mbps on mobile and even higher on Wi-Fi and cable connections. However, these speeds are not the case for a significant portion of the world population. Rate limited plans as low as 1Mbps, crowded airports, cities with network outages - this is still a reality in the modern world. And we still want to make sure that our software can accommodate adequate user experience under those conditions. At TestDevLab we use 2000+ real devices for testing and simulate various network conditions. In this blog article, we will discuss our approach to network condition simulation.

Cisco report states that 73% of mobile connections in the Middle East and Africa are 3G and below.

That would mean that typical speeds will be somewhere from 0.1Mbps to 20Mbps on average for 75% of Africa and the Middle East (countries like Egypt, Turkey, Iran, and this region in total accommodates around 1.7 billion people, which is about 25% of the world's population, a massive number.

Five prevalent network impairments and reasons behind them

There are many different variables in global network infrastructure that can affect performance and stability. Let’s go through some of the most widespread network issues, which can put network performance at risk.

1. Limited bandwidth - a situation when the current maximum network speed is decreased
   
   1. Caused mainly by service plan limitations,
   2. It may be caused by network uplink overload (Mostly, the “last mile” can be a bottleneck).
2. Packet loss (random or sudden) - a situation where pieces of network information do not reach a destination
   
   1. It mainly occurs on wireless networks and is associated with radio signal interference,
   2. Less often can occur on highly overloaded networks, when routers' queues are overloaded, and packets simply drop instead of queueing.
3. High RTT delay (ping) - when the time it takes for a packet to travel to a destination and back to its source is too high, it lowers the overall service quality. Occurs over long distances and overloaded links on the “last mile.”
4. Jitter - variations in RTT delay (ping). It occurs when the connection to the internet is not stable.
5. Disconnections - a situation when a device disconnects from the network for some amount of time
   
   1. Can occur when the packet loss is very high, and the application can’t maintain the connection,
   2. A weak Wi-Fi signal can cause a device to disconnect from an access point.

Examples of poor network situations

• Public Wi-Fi hotspots: Slow network occurrences are widespread in throttled/limited networks and are usually encountered when using public Wi-Fi access points - in cafes, airport lounges, hotels, and elsewhere.
• Crowded Wi-Fi networks: This happens when Wi-Fi networks use the same channels as several other Wi-Fi networks nearby.

• Limited ISP data plan: Common low bandwidth limitations occur when a low-cost data plan has been exceeded.
• Switching between Wi-Fi access points: In different parts of the building, there might be seamless switching between Wi-Fi access points in one network, but during switches between networks, bandwidth decreases.
• Underdeveloped new infrastructure in emerging markets: Additionally, slow networks are common across locations where either the infrastructure is just being built or the opposite; network communication infrastructure is outdated.
• Buildings or infrastructure with outdated network communications
• Poorly maintained or configured local network

Setup and tools for network simulation

Building a stable network infrastructure takes effort and time, and there are some challenges that need to be overcome. To have data that you can trust when you start testing software that is heavily dependent on the network connection, the network itself needs to be stable and act predictably.

For testing, we use dedicated Wi-FI networks that do not interfere with each other.

Network testing setup:

• Dedicated High-Speed Internet uplink,
• Fast, stable, and not overloaded local network,
• Independent Wireless Access Point with dedicated radio frequency,
• 2x Dedicated virtual routers for each network testing setup,
• Powerful hypervisor hosting virtual routers,
• Dedicated computers for network traffic capture.

Let’s talk about tools

For traffic limitation at TestDevLab, we use a set of tools called “Linux Traffic Control”, which are installed on virtual routers based on Ubuntu 18.04 Linux and allow us to simulate any network conditions up to 250 Mbit/s.

This set gives us a couple of benefits:

It is the most sophisticated tool, that works in the Linux kernel for network impairment simulation needs and has a lot of flexibility,
Gives us the possibility to limit the network before the device, and makes the simulated network identical to a real-world network example.

Network Traffic limitation is done by an algorithm HTB (Hierarchical token bucket) discipline of Linux traffic control utility is used to simulate bandwidth.

NetEm (Network Emulator) is an enhancement of the Linux traffic control facilities that allow us to add delay, packet loss, duplication, and other characteristics to packets leaving the network interface.

Here is an example of the full script which is running on the virtual router:

#!/bin/bash
sudo tcdel eth1 --all
echo "\e[1;32m===SETTING UP Limitations===\e[0m"
/sbin/tc qdisc del dev eth1 root
/sbin/tc qdisc del dev eth1 ingress
/sbin/tc qdisc del dev ifb5873 root
/bin/ip link set dev ifb5873 down
/bin/ip link delete ifb5873 type ifb
modprobe ifb
/bin/ip link add ifb5873 type ifb
/bin/ip link set dev ifb5873 up
/sbin/tc qdisc add dev eth1 ingress
/sbin/tc filter add dev eth1 parent ffff: protocol ip u32 match u32 0 0 flowid 16f1: action mirred egress redirect dev ifb5873
/sbin/tc qdisc add dev ifb5873 root handle 16f1: htb default 1
/sbin/tc class add dev ifb5873 parent 16f1: classid 16f1:1 htb rate 32000000.0kbit
/sbin/tc class add dev ifb5873 parent 16f1: classid 16f1:129 htb rate $1.0Kbit ceil $1.0Kbit

if ([ $2 -eq 0 ] && [ $4 -ne 0 ])
        then echo "CAN'T SET JITTER WITHOUT DELAY"
        tcdel eth1 --all
  exit 2

        elif ([ $3 -eq 0 ] && [ $4 -eq 0 ])
        then
                /sbin/tc qdisc add dev ifb5873 parent 16f1:129 handle 2a2b: netem delay $2.0ms limit $5
        elif ([ $3 -eq 0 ] && [ $4 -ne 0 ])
        then
                /sbin/tc qdisc add dev ifb5873 parent 16f1:129 handle 2a2b: netem delay $2.0ms $4.0ms distribution normal limit $5
        elif ([ $3 -ne 0 ] && [ $4 -eq 0 ])
        then
                echo $3 $4
                /sbin/tc qdisc add dev ifb5873 parent 16f1:129 handle 2a2b: netem loss $3.0% delay $2.0ms limit $5
        elif ([ $3 -ne 0 ] && [ $4 -ne 0 ])
        then
                /sbin/tc qdisc add dev ifb5873 parent 16f1:129 handle 2a2b: netem loss $3.0% delay $2.0ms $4.0ms distribution normal limit $5
        else echo ----====ERROR====----
fi
/sbin/tc filter add dev ifb5873 protocol ip parent 16f1: prio 2 u32 match ip dst 0.0.0.0/0 match ip src $11/32 flowid 16f1:129


/sbin/tc qdisc add dev eth1 root handle 16f2: htb default 1
/sbin/tc class add dev eth1 parent 16f2: classid 16f2:1 htb rate 10000000.0kbit
/sbin/tc class add dev eth1 parent 16f2: classid 16f2:83 htb rate $6.0Kbit ceil $6.0Kbit

if ([ $7 -eq 0 ] &&  [ $9 -ne 0 ])
        then echo "CAN'T SET JITTER WITHOUT DELAY"
        tcdel eth1 --all
  exit 2

        elif ([ $8 -eq 0 ] &&  [ $9 -eq 0 ])
        then
                /sbin/tc qdisc add dev eth1 parent 16f2:83 handle 2a2b: netem delay $7.0ms limit $10
        elif ([ $8 -eq 0 ] && [ $9 -ne 0 ])
        then
               /sbin/tc qdisc add dev eth1 parent 16f2:83 handle 2a2b: netem delay $7.0ms $9.0ms distribution normal limit $10
        elif ([ $8 -ne 0 ] && [ $9 -eq 0 ])
        then
                echo $3 $4
                 /sbin/tc qdisc add dev eth1 parent 16f2:83 handle 2a2b: netem loss $8.0% delay $7.0ms limit $10
        elif ([ $8 -ne 0 ] && [ $9 -ne 0 ])
        then
                /sbin/tc qdisc add dev eth1 parent 16f2:83 handle 2a2b: netem loss $8.0% delay $7.0ms $9.0ms distribution normal limit $10
        else echo ----====ERROR====----
fi

/sbin/tc filter add dev eth1 protocol ip parent 16f2: prio 2 u32 match ip dst $11/32 match ip src 0.0.0.0/0 flowid 16f2:83
echo  "\e[1;32m===LIMITATION PARAMETERS===\e[0m"
tcshow eth1
tc qdisc

Script Usage Example:

Command “sudo limit.sh 2000 0 0 0 1000 3000 0 0 0 1000 172.16.200.5”
Sets 11 variables:

Download Speed limitation 2mbit/s
Download delay 0ms
Download Packet loss 0%
Download jitter +/- 0ms
Download Queue size 1000
Upload Speed Limitation 3 mbit/s
Upload delay 0ms
Upload Packet loss 0%
Upload jitter +/- 0ms
Upload Queue size 1000
Host 172.16.200.5

Network condition examples from around the world

For this part, we have taken Speedtest reports from people around the globe as well data from airports and ISP plans. These reports are different in the way that some of them contain high RTT delay, some of them contain packet loss, and some have low available network bandwidth. These kinds of situations can occur in any place on the planet.

China, Shanghai, 33 Mbps down/10 Mbps upload, jitter - 11.4ms, ping - 18ms.

Script:

sudo limit.sh 33000 8 0 3 1000 11000 8 0 3 1000 172.16.200.5
Adds: 	33 Mbit/s limitation 8 msdelay with +/- 3 delay jitter for incoming traffic 
11 Mbit/s limitation 8 msdelay with +/- 3 delay jitter for outgoing traffic 

Morocco, Fez: 3.5Mbps down/0.62 up, jitter - 87ms, RTT - 57ms, PL - 0.67%

Script:

sudo limit.sh 3500 20 0.5 10 1000 600 20 0.5 10 1000 172.16.200.5
Adds: 	3.5 Mbit/s limitation, 0.5% packet loss and 20 ms delay with +/- 5 delay jitter for incoming traffic 
600 Kbit/s limitation, 0.5% packet loss and  20 ms delay with +/- 5 delay jitter for outgoing traffic

Louis Armstrong New Orleans International Airport - 1Mbps down/1Mbps up (data from other airports)

Script:

sudo limit.sh 1000 0 0 0 1000 1000 0 0 0 1000 172.16.200.5
Adds: 	1 Mbit/s limitation for incoming traffic 
1 Mbit/s limitation for outgoing traffic

ISP plans in India: https://gadgets.ndtv.com/mobile-recharge-plans/jio-prepaid

Jio, Airtel, and Vodafone Idea (Vi) are the three largest telecom providers in India, taking up about 90% of the total market share when it comes to wireless subscribers, as per data from the Telecom Regulatory Authority of India (TRAI). All three operators offer a host of prepaid and postpaid plans that come in all price points, depending on the services you require. The most popular plans are the cheapest plans, so let's take a look at an example.

What is noticeable is that the speed drops to 64kbps after the data plan was exceeded.

An interesting use case can be to check how the app’s behavior will change during a test. This can be especially useful for VOIP and Video conferencing applications, where users will still expect to have acceptable audio quality.

Script to simulate change from 10Mbps to 64Kbps and back with 60 seconds step in one test:

#!/bin/sh
sudo tcdel eth1 --all
echo -e "\e[1;32m===SETTING UP Limitations===\e[0m"
/sbin/tc qdisc del dev eth1 root
/sbin/tc qdisc del dev eth1 ingress
/sbin/tc qdisc del dev ifb5873 root
/bin/ip link set dev ifb5873 down
/bin/ip link delete ifb5873 type ifb
modprobe ifb
/bin/ip link add ifb5873 type ifb
/bin/ip link set dev ifb5873 up
/sbin/tc qdisc add dev eth1 ingress
/sbin/tc filter add dev eth1 parent ffff: protocol ip u32 match u32 0 0 flowid 16f1: action mirred egress redirect dev ifb5873
/sbin/tc qdisc add dev ifb5873 root handle 16f1: htb default 1
/sbin/tc class add dev ifb5873 parent 16f1: classid 16f1:1 htb rate 32000000.0kbit
/sbin/tc class add dev ifb5873 parent 16f1: classid 16f1:129 htb rate 10000.0Kbit ceil 10000.0Kbit
/sbin/tc filter add dev ifb5873 protocol ip parent 16f1: prio 2 u32 match ip dst 0.0.0.0/0 match ip src $1/32 flowid 16f1:129
/sbin/tc qdisc add dev eth1 root handle 16f2: htb default 1
/sbin/tc class add dev eth1 parent 16f2: classid 16f2:1 htb rate 10000000.0kbit
/sbin/tc class add dev eth1 parent 16f2: classid 16f2:83 htb rate 10000.0Kbit ceil 10000.0Kbit
/sbin/tc filter add dev eth1 protocol ip parent 16f2: prio 2 u32 match ip dst $1/32 match ip src 0.0.0.0/0 flowid 16f2:83
echo -e "\e[1;32m===LIMITATION PARAMETERS===\e[0m"
tcshow eth1
sleep 60
/sbin/tc class change dev ifb5873 parent 16f1: classid 16f1:129 htb rate 64.0Kbit ceil 64.0Kbit
/sbin/tc class change dev eth1 parent 16f2: classid 16f2:83 htb rate 64.0Kbit ceil 64.0Kbit
echo -e "\e[1;32m===LIMITATION PARAMETERS===\e[0m"
tcshow eth1
sleep 60
/sbin/tc class change dev ifb5873 parent 16f1: classid 16f1:129 htb rate 10000.0Kbit ceil 10000.0Kbit
/sbin/tc class change dev eth1 parent 16f2: classid 16f2:83 htb rate 10000.0Kbit ceil 10000.0Kbit
echo -e "\e[1;32m===LIMITATION PARAMETERS===\e[0m"
tcshow eth1
echo -e "\e[1;32m===CLEARING LIMITATIONS===\e[0m"
#tcdel eth1 --all

Conclusions

Network infrastructure is progressing worldwide with fiber optics and 5G spreading widely, but that doesn't mean that most of the users have excellent network connections. Low network speed is still common in APAC countries yet users across the world can experience network impairments as well. During the COVID pandemic, the whole family might be using bandwidth-demanding video services simultaneously over the whole day that might push network usage to the limit. All of the above means that users across the world will experience different network Quality of Service and issues, such as low bandwidth, packet loss, and jitter. In order to understand the user experience - we need to test those conditions and those conditions can be simulated.

It is important to check the app’s behavior under various network conditions including low bandwidth, packet loss, jitter, and high RTT and we can achieve this in our dedicated labs. This way you will ensure that the user gets the most even under the worst network conditions.