Video Call Reliability and Quality Review—Introducing Our New Service

At TestDevLab, we understand that navigating the complexities of video testing can be overwhelming. Determining which aspects to focus on isn't always straightforward, especially when aiming to deliver seamless and reliable video calling experiences. With our extensive experience in video testing, we've seen these challenges firsthand.

To simplify the video testing process, we're excited to introduce our Video Call Reliability and Quality Review service—a set of tests designed to simulate a wide range of real-world network conditions, such as packet loss, limited bandwidth, and network interruptions. This service thoroughly assesses how your application performs under various scenarios. 

By bundling multiple tests into a single package, we offer extensive coverage at a lower cost compared to conducting each test separately. This integrated approach saves you time and resources, while providing a holistic view of your application's performance.

This initial offering is specifically designed for two-user desktop video calls, marking the first in a series of services we will be providing.

See the detailed offer here.

Contact us to learn more about this service.

In the next section, we will showcase what kind of data you can expect from the review.

Result demo: Bandwidth throttling on Google Meet

To give you a clear idea of what to expect from our Video Call Reliability and Quality Review, we conducted tests using Google Meet and created a detailed report based on two specific network scenarios. In each scenario, we executed the tests four times, collecting a range of metrics that reflect the overall call quality experience. The report illustrates how limited bandwidth impacts performance on this widely used platform, highlighting key factors affecting the user experience.

Testing scope information

Service: https://meet.google.com/

Call type: Video call

Call view: Side by side

Device tested: MacBook Pro M1 2020, MacOS 15.0

Browser: Chrome 130.0.6723.117

Network applied to measured receiver device:

• Bitrate throttling test: 3Mbps to 500kbps
  • Call duration: 8 minutes
  • Throttling configuration: Incoming and outgoing bitrate throttled for one of the users each minute:
    •  Unlimited, 3Mbps, 2.5Mbps, 2Mbps, 1.5Mbps, 1Mbps, 750Kbps, 500Kbps
• Bitrate throttling test: 500Kbps to 100Kbps
  • Call duration: 8 minutes
  • Throttling configuration: Incoming and outgoing bitrate throttled for one of the users each minute:
    • 500Kbps, 400Kbps, 300Kbps, 250Kbps, 200Kbps, 150Kbps, 125Kbps, 100Kbps

Test date: November 4–6, 2024

Video used in test: video link

Metrics gathered:

• VMAF—full-reference quality score
• Frame rate—frames per second
• Freezes—average frozen time of over 0.2 seconds
• POLQA—full-reference quality score
• Video and audio latency—end-to-end latency for incoming media on limited device 

Scoring and result overview

To evaluate the quality of the call, we have created a set of criteria and rules tailored to different network conditions.

These rules define quality rankings—Excellent, Good, Fair, Bad, and Critical—based on thresholds for different metrics: video quality, frame rate, freeze times, latency, and audio quality. Percentile-based thresholds are used to determine the consistency and reliability of these metrics throughout the call duration. 

The table below provides a detailed summary of the thresholds and quality categories for various network conditions.

And here are results from the test executed—color coded to show which conditions Google Meet meets for evaluation of different factors of the call quality.

See quality comparison below for each scenario. 

Result summary: Bitrate throttling test: 3Mbps to 500kbps

Requirements not met for Excellent ranking:

Video quality:

75% quality over 75 VMAF.  Current: 72.7

95% quality over 65 VMAF. Current: 62.53

Video fluidity:

99% frame rate over 20. Current: 18

Audio quality:

99% POLQA over 4. Current: 3.51

Latency:

98% Video latency under 350ms. Current: 466ms

98% Audiolatency under 350ms. Current: 527ms

Overtime data

Reading graphs

The graphs below show second-by-second data for each of the four tests that were performed for specific limitations. Light lines show individual test data, while the black line indicates the average across all tests.

The Y-axis represents the metric being analyzed, and the X-axis shows the seconds elapsed into the test. The labels at the top of the graph describe the limitations applied during each specific section of the test. For audio data, the X-axis shows a sample number instead. There are five audio samples evaluated each minute.

Network data

The network data shows that in this configuration, Google Meet uses around 1.25Mbit for video and audio transfer. This indicates that bandwidth limitations start affecting video quality from the 6th minute onward, as the available bandwidth to the device is limited from 1.5Mbit to 1Mbit.

Video data

Given the network behavior the impact on video quality becomes noticeable only when the bandwidth is limited to 1Mbit. At this limitation we see  some minor freezes, frame rate drops and an increase in delay, however, the app is quick to adjust to the limitation.

As the bandwidth is further reduced, the frame rate becomes slightly unstable, but still high. There is a delay increase from ~200ms delay at the start of the test to 350ms by the end.

The average VMAF goes from 76 at start of the test to 65 at the 500kbps limitation.

Audio data

For audio quality, the greatest drop happened between the 750Kbit and 500Kbit limitation.

There is also a minor delay spike at the beginning of the limitation, starting from 1Mbps.

Result summary: Bitrate throttling test: 500Kbps to 100Kbps

In this scenario, we dive into more details, as it involves really low bitrates that some apps can't even handle. This way, we can see what happens to the call and at what point. Will it disconnect, become unusable, maintain good audio, or continue to run perfectly?

For Google Meet, the breaking point occurs between 150Kbps and 125Kbps bitrate at which point video in the call freezes. However, the call continues and audio quality remains fair for the remainder of the test. That’s a good reason to give scoring for limitations between 125Kbps–100Kbps based on audio quality only.

In the table below, you can see Google Meet scores throughout the test and requirements for achieving the next level of quality.

Overtime data

Network data

In all scenarios, network usage is close to the limit, which explains some minor freezes from 500Kbps to 200Kbps limitations.

Starting from 150 Kbps, we see that network usage goes over the limit more frequently and the sender switches to sending only audio data (predictable network bitrate matching audio sample waveform).

Video data

Frame rate, freezes and delay metrics show that the call becomes increasingly unstable. The breaking point is between 150Kbps and 125Kbps, when in all calls the video either freezes or turns off. With lower available bitrate, VMAF scores also progressively go down.

Audio data

In terms of audio quality,  the data is quite stable except for a few drops at stricter limitations and a major issue for one call at the 100Kbps limit.

The takeaway

Google Meet demonstrates decent performance in handling different bandwidth levels during 2-person video calls down to 150Kbps at which point the video freezes but the call is still able to continue in an audio format.

Main app improvements could be done by having less aggressive bitrate usage for lower bitrate scenarios to maintain more stable call experience.

If you have a service that could benefit from this kind of in-depth testing, contact us to learn more about our Video Call Reliability and Quality Review.