We continue to review variants of load tests. In this article we will go over the testing methodology and conduct a load test that we will use to try and determine the number of users that could watch and stream at the same time, meaning the users will simultaneously publish and view the streams.
One of the most common modes of operation for WebRTC servers is "one-to-many." It involves a big number of viewers connecting to the server; viewers that are all watching one stream. This is the way sports broadcasts, for example, are set up: the content is being broadcast and the viewers watch.
But sometimes a different mode is required: "one-to-one," when the streamer is simultaneously a viewer of another stream. This one could be used, for example, for video chats with no mixers or MCUs.
Diagram showing the stages of testing
Publish a stream of 1280x720, 30 fps, bitrate 2,500 kbps, on the testing WCS#1 server;
Publish the source stream using the REST API /pull/push request in the required number of instances to the tested WCS#2 server;
Using the second REST API /pull/push request, take each instance of a previously published stream from the tested WCS#2 server to the WCS#1 server;
Play a random stream on the tested WCS#2 server and visually evaluate its quality. Estimate the server load using metrics received from the Prometheus monitoring system.
Test rig
For testing, we need
two WCS servers;
standard example "Media Devices" for publishing a stream with specified parameters;
test script;
We will assume that you already have your WCS servers installed and configured. If not, install them according to this instruction, make tweaks to boost performance and connect to monitoring.
We used two servers for testing:
WCS#1 (testing) with characteristics:
24 cores, 48 streams
128 Gb RAM
WCS 5.2.946
OpenJDK version 14.0.1
and
WCS#2 (tested) with characteristics:
12 cores, 24 streams
84 Gb RAM
WCS 5.2.946
OpenJDK version 14.0.1
In one of the previous tests, we have already found out empirically that the load on the testing server is quite large, therefore, for the tests to succeed, the testing server must be at least twice as powerful as the tested one, or several servers must act as the testing server.
Our server settings:
File flashphoner.properties (w/o default settings):
#webrtc ports range
media_port_from = 20001
media_port_to = 40000
wcs_activity_timer_timeout=86400000
global_bandwidth_check_enabled=true
zgc_log_parser_enable=true
zgc_log_time_format=yyyy-MM-dd'T'HH:mm:ss.SSSZ
Here we increased the range of ports and the time for passing the test, as well as specified the settings for collecting additional metrics for network congestion and pauses in ZGC.
File wcs-core.properties (w/o default settings):
-XX:ErrorFile=/usr/local/FlashphonerWebCallServer/logs/error%p.log
# ZGC
-XX:+UnlockExperimentalVMOptions -XX:+UseZGC -Xms24g -Xmx24g -XX:+UseLargePages -XX:ZPath=/hugepages
-Xlog:gc*:/usr/local/FlashphonerWebCallServer/logs/gc-core-:time
Here we changed the log file naming settings, enabled the use of ZGC, adjusted the Heap size, and enabled the use of memory pages.
Full versions of the configuration files can be downloaded from the Useful Files at the bottom of the article. There you will also find a panel for Grafana and a script file to start testing.
Testing Script
#!/bin/bash
sourceStreamName=$1
sourceServer=$2
serverToTest=$3
testStreamsCount=$4
if [[ -z $sourceStreamName ]] || [[ -z $sourceServer ]] || [[ -z $serverToTest ]]; then
echo "Usage: $(basename $0) stream_name source_server server_to_test test_streams_count"
echo -e " stream_name\t\tstream published on source server name"
echo -e " source_server\t\tsource server ip address"
echo -e " server_to_test\t\tserver to test ip address"
echo -e " test_streams_count\t\tstreams count to push (1 by default)"
exit 1
fi
[[ -z $testStreamsCount ]] && testStreamsCount=1
# Check if source stream is published
result=`curl -s -i -o /dev/null -w '%{http_code}' -H "Content-Type: application/json" -X POST http://$sourceServer:8081/rest-api/stream/find --data "{\"name\":\"$sourceStreamName\",\"published\":true}" | grep 200`
if [[ $result != "200" ]]; then
echo "Stream $sourceStreamName is not published on server $sourceServer!"
exit 1
fi
# Push streams to the server to test
echo "Publishing $testStreamsCount copies of stream $sourceStreamName to the server $serverToTest"
for ((i = 1; i <= $testStreamsCount; i++)); do
curl -s -H "Content-Type: application/json" -X POST http://$sourceServer:8081/rest-api/pull/push --data "{\"uri\":\"ws://$serverToTest:8080/websocket\",\"localStreamName\":\"$sourceStreamName\",\"remoteStreamName\":\"$sourceStreamName$i\"}"
done
# Wait for 5 seconds
echo "Waiting for 5 seconds..."
sleep 5
# Push streams back to source server
echo "Playing $testStreamsCount copies of stream $sourceStreamName from the server $serverToTest"
for ((i = 1; i <= $testStreamsCount; i++)); do
curl -s -H "Content-Type: application/json" -X POST http://$serverToTest:8081/rest-api/pull/push --data "{\"uri\":\"ws://$sourceServer:8080/websocket\",\"localStreamName\":\"$sourceStreamName$i\",\"remoteStreamName\":\"$sourceStreamName$i\"}"
done
echo "To stop the test, just stop publishing stream $sourceStreamName"
where:
sourceStreamName — name of the source stream published on the testing WCS#1 server;
sourceServer — name or IP address of the testing WCS#1 server;
serverToTest — name or IP address of the tested WCS#2 server;
testStreamsCount — number of streams for the test.
Testing
On the testing WCS#1 server, using the example "Media Devices" publish a stream under the name "test" with parameters of 1280x720, 30 fps, bit rate 2,500 kbps.
Then, in the testing server console, run the test script with the following parameters:
./pull_push_test.sh test 172.16.40.3 172.16.40.2 200
where
test — name of the stream published on WCS#1
172.16.40.3 — IP address of the testing WCS#1 server
172.16.40.2 — IP address of the tested WCS#2 server
200 — number of streams
The result for testing 200 streams is shown in the screenshot below:
The test was successful. No stream degradation was recorded and the control stream was played with acceptable quality - without friezes, artifacts, or a decrease in sound quality:
The test was completed successfully on our hardware. Please note that the test results on your hardware can be better or worse depending on the environment, geographical location of servers and users - both subscribers and streamers.
Therefore, we recommend that, before making the final decision on the choice of servers, you conduct all possible tests that may be relevant to your task, and make a decision based on their results.
Have a good streaming!
Useful Files
Panel for Grafana:
WCS settings:
Test script: