Performance Testing of Short Video Apps

Quick-to-consume media is in high demand on the internet. This is not much of a revelation to the world, yet the market is starting to be filled by more and more newcomers, as it unites all generations at our device screens. The short video format connects us faster than before. Some may choose to send an email instead of a snail mail message to send holiday wishes, which is great, yet the younger generation prefers something more lively – a performance.

The new breakout apps are not here to test our patience; rather they aim to deliver the most outstanding content in the short form. The quick and easy way to reach media is perfect for the viewer that only intends to watch a few videos. The apps that are available today walk on a narrow path between being original and being a copy-cat; nonetheless, given the end product is similar, the approach to developing and testing these apps is very different.

We could first truly reach our most beloved creators in 2012 when the small American-made app called Vine blew up in popularity. As confusing to some as it was at first, it paved the way for the future. Near the end of its life, when Twitter had decided to kill it off, another small app surfaced – TikTok. Although now a giant in the industry facing off against the likes of Youtube, Facebook, and even Netflix, initially it wasn’t like that.

For Tiktok, the big breakthrough happened when it merged with another Chinese-made app: Musical.ly. Through combined forces, it amassed a user base growth like never before, quickly becoming one of the most downloaded apps in the world and leaving the competition behind. It continued the journey that Vine had started. Short videos, interesting, engaging content, and a community that connects and relates to each other – and all of that available in a matter of seconds, at a touch of a button. In the three years from its launch, it grew to an international user base of over 800 million. However, the speedy growth brought challenges, both technical and business. Issues like content moderation, scalability, and security riddle the app to this day.

Nowadays, the unclaimed African, Indian and Latin American markets are prime examples of the possibilities for platform growth. For India, the active online user base has doubled in just a few years. The cheap broadband connection is available for vast amounts of people, but it is often very slow. Billions of people are patiently waiting for their content, and with it, the right app to use.

Each app on the market has its own way of being unique. Instagram assumes an identity in which you see only the perfect view of a thing. Flawless image quality is expected. YouTube is more about viewability and how to grab the attention of your audience for 10 minutes so the advertisers can be happy. The whole platform has grown around the 10-minute video for years. Perfect fluidity and originality are expected. Snapchat wants us to share something more personal – it’s not about the replayability itself, more of the emotion at the moment. Neither the quality nor fluidity may be as important as long as you feel connected and safe. Then there’s Dubsmash. The app which got bought out by Reddit and kept its core idea. Small, niche communities that can be matched to the needs of nearly every individual.

Each short video app has its own benefits and drawbacks. How do you secure a stable platform while adding new features, yet at the same time preserving the existing ones without disturbing the user base? What is the best approach towards video streaming? What about the upload? Speed is important, but the ease of use is a part of that. One does not work without the other. Responsiveness, throughput, reliability, and further scalability of the system are considered in every product decision being made. Let’s take a look and compare the basic features that a user might expect.

Now we get to think, if Triller supports HD content and many of the features that TikTok offers, why is it so unpopular compared to the giants? Neither Reels nor TikTok have 1080p support, yet they have a way bigger following. There is a valid reason. Given the small device screens, rarely can anyone quickly identify between 1080p, 720p or 576p. Have you ever noticed how video aspect ratios change, or how TikTok dims the image? They also reduce the color count and downgrade the resolution. All of those are the same video shown in different apps. Although TikTok suggests that their supported resolution is 720p, all of the videos are actually in 576×1024 resolution. With clever compromises they can attain superb fluidity within the app.

All of the apps bring a unique take on the short video format. If they disappeared one day, it would feel so empty without them. Developers bring us amazing products, but more often than not they do need a little help from testers. An independent team not only can, but will surely bring a better view of the flaws. A tester can serve the roles of both, a scrutinous judge and an everyday user. One role may find the tiniest imperfections and anomalies in audio or video compression from the original to the uploaded content and the other may just validate that the user will have the expected product. Either way the goal is to prevent massive loss of business by the release or upkeep of a broken product. If a bad login experience can drive away up to 35% of the user base, then imagine the disappointment of using a difficult UI or not being able to watch a video. How about creating a video a couple of times and realizing that the app has become sluggish? Whoops, your RAM never got released. What if the app works well? Great, except the CPU usage is so high that the users need to constantly charge the device to use the app. Here is where TestDevLab comes into play.

We check whether developers conform to their own goals. Our experience and research show that there is room for improvement, and the constant strive to better their product is valid. Benchmark establishment is of utmost importance to further develop the application. Complacency leads to downfall.

Bearing all of this in mind, there are several key metrics we need to look out for:

1. Network usage and how the app reacts to changes;
2. Visual quality, artifacts, possible detail loss, and fluidity;
3. Audio quality, a fundamental part for visual entertainment;
4. Content, UI, UX, and other things that make the experience fulfilling;
5. Latency.

Network usage

The most valuable thing for the developer would be to see the app in a way that the user sees it in. We, at TestDevLab, have several possible solutions for that. We can simulate the effect of critically limited bandwidth, high or low packet-loss, jitter or switches of the network, or all of the above at once. We can simulate the precise conditions in which the user will use the device and the app. Our network laboratory testing conditions are best to test the app behaviour under network stress and continually provide stable results that can be compared from version to version. Stability, when doing these tests, is the aspect that will guide the teams towards improvement and user satisfaction.

Visual Quality

Usually, network conditions are tightly connected to the visual impression. When the app is under tight restrictions, the video may go two ways – either the bitrate of the video drops significantly to promote a video playing at all, resulting in a blocky, pixelated image in which important details such as faces, items, or the idea itself can be missed. Alternatively the video may become a slideshow in order to maintain the quality of the image, resulting in simple frustration. This can not only be detrimental to the creator whose videos do not get as many minutes or seconds of viewing, but also can quickly lead the client, in this case, the user, away from the app, given another app performs the same action in a better way. When simulating an environment that can test both video fluidity and quality at the same time, we can pair it with the assumed ideal user experience, and then process that with several video evaluation metrics that can automatically give quality scores either with or without reference material. To compare those and encapsulate the complete picture – VMAF, PSNR, SSIM, and BRISQUE algorithms help us with that. Additionally we can conduct subjective testing to acknowledge the viewpoints of real people. The recommended way is to do both subjective and objective testing, since they complement each other very well, and that is exactly what is being done at TestDevLab.

Audio Quality

This is a metric that most people would not immediately associate with a high quality video or a short video app. Most just glance over it as a given. We use 5 basic senses all of the time. The more of these we utilize, the more of a lasting impression the users receive. The goal is to satisfy our sensitive hearing sense. Given that the apps focus on visual first and foremost we expect the audio to correspond with visual feedback that we get. We see something and set an expectation, and if that expectation is broken, it can lead to a quick turn off. We can detect even the subtlest of differences from the original by testing. Of course most people won’t care that a story-telling video is played back with mono sound instead of stereo, but if it’s a dance video, the lack of stereo can diminish the depth and emotion of a performance. Audio delays, stutters or complete stalls are also likely to ruin the experience of users. Our instinct is to see, yet our strength is to listen and POLQA helps us with that.

Content and fulfillment

It will not matter how good, and how many features one can squeeze into an app if the app is difficult, frustrating, and unintuitive to use. People want to master things instantly and be accustomed to the basic functionalities as early as possible. The app has to be very flexible to accommodate the users’ needs in content that it offers. It does not matter whether you like cats, dogs, pineapples, or cars, algorithms will be hard at work to provide as long as you tap the like button. The app must change along with the user. That includes the people who have not been as fortunate in their lives. Additions, like AI generated captions or higher contrast options for interfaces can help with navigation and accessibility.

Latency

Time is a crucial factor when it comes to usability. Not only does it matter how a video acts during viewing, it also matters how quickly it loads. What is the response time from the tap to an appropriate reaction? No matter whether it’s prime time or graveyard slot, how would you ensure the same result? TestDevLab can provide the answers.

Creating a short video app was never a simple idea, it is riddled with complex issues and risks. The user base as a whole won’t miss discrepancies and will quickly pinpoint them, and be swift to switch to another app if those discrepancies are not fixed. From creation to viewing, these apps are under the highest levels of scrutiny. No time to rest in the development within this field because the market is so saturated. Apps need to be efficient. There are metrics that need to be tested which include but are not limited to: RAM and CPU usage, specific device alignment to the app, network usage, battery usage and many more.

We, TestDevLab,can be the best partners to test these and other metrics to achieve the best that one can offer. We’ve gathered a lot of confidence in our work throughout the years in this field and we’re sure that we’re the testing partner for you. Contact us for more info on how we can help improve the quality of your app.