You could almost call testing media platforms a separate discipline. You are always dealing with a really big peak load that requires simulating many thousands of potential users in the test. This was true of the performance testing we carried out for Ericsson for the streaming platform they provide for a Dutch media organisation.
For the past year, Ericsson has been supplying the ‘Ericsson Managed Player’. This allows media companies to offer streaming content over the internet. For example, one of Ericsson's customers broadcasts live matches from an international football league. We observe that consuming in a non-linear way, as for example with Netflix, is increasing rapidly. And the success of these services stands or falls by a good user experience.
Large-scale testing
Ericsson was looking for a party that could test the system and could also simulate how consumers experience the services in a life-like way. Because we are virtually the only party able to perform this kind of large-scale testing properly, Ericsson turned to Computest. For every football match broadcast by Ericsson's customer, there are about 50,000 consumers using the streaming platform. This is important information as a starting point for your test. Regarding the performance of the Ericsson platform, two factors played an important role: is the origin (or switching point) at Ericsson working properly and can the providers' CDNs (Content Delivery Networks) handle the load? We looked into both issues in detail.
Bottleneck
Testing the origin is important because this is the source of what you ultimately see at home. So it is the first thing you test. The origin needs to be tested in isolation in order to determine what it can handle, what the possible bottlenecks are and how you can fine-tune it even further if necessary. Although the CDN acts as a large caching proxy, which means that not every user has to collect the video files directly from the origin servers, because many different streams are offered and there are many CDN servers, a significant load from CDN to the origin servers is nevertheless expected. So only installing a CDN for your environment is not enough if you expect a serious number of users.
Simulating 50,000 users
During the testing, we really pushed the limits of what was possible together with Ericsson. For example, we simulated up to 50,000 viewers in a stress test in order to test the actual user experience. When doing so, we can also see anything that goes wrong at the CDN and in the chain. For instance, one of the things we found was that certain files were not being ideally cached by the CDN due to a unique time stamp in the URL. A CDN edge server passed every request to the origin, so placing an unnecessary load on the origin. We also noticed that errors sometimes arose because files supposedly didn't exist. This was due to timing issues affecting only a few percent of all users. This is something you only discover by simulating the large volume directly on the CDN. Together with Ericsson, we were able to test this properly and find areas for improvement and resolve them. We also achieved a record 50 Gbps bandwidth!
Load farm
In order to be able to do the testing for Ericsson, we used our own load farm. This can perform tests involving more than 250,000 concurrent users and millions of unique users per hour. With 50 Gbps network capacity and 2,000 unique IP addresses, we can simulate users very realistically. When testing, there must be no difference whatsoever with practice in technical terms; the website tested must therefore see no difference between real users and our simulated users. As such, we pay attention not only to what we affect, but also how.
Experience
Practice always demonstrates that testing is essential for services such as those supplied by Ericsson. My experience is that no single large set-up works properly in practice first time. So before you introduce an environment like this, you must consider the need to allocate time to test whether it actually does what it is supposed to do. What is the bandwidth you want to achieve and how many users do you want to test for? Even when using the CDN because you are expecting a lot of users on your platform, things never go smoothly straight away. Of course we will be happy to advise you on this. We are certainly not lacking in experience :-).