hls

HLS load testing

peerasan buranasanti

• 3 min read
HLS load testing

HTTP Live Streaming (also known as HLS) is an HTTP-based adaptive bitrate streaming communications protocol developed by Apple Inc. and released in 2009. Support for the protocol is widespread in media players, web browsers, mobile devices, and streaming media servers. As of 2019, an annual video industry survey has consistently found it to be the most popular streaming format. (ref: wikipedia)

To stress test the http streaming, you need to request a media playlist called m3u8 which leads to a media file list (.ts). You can use load test tool as siege, wrk to call URL. Below is m3u8 example with adaptive bitrate and use siege to load test.

curl https://media.smeplanet.com/stream/patrickz/ngrp:live_all/playlist.m3u8
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-STREAM-INF:BANDWIDTH=4227072,CODECS="avc1.100.31,mp4a.40.2",RESOLUTION=1280x720
chunklist_b4227072.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1063000,CODECS="avc1.66.31,mp4a.40.2",RESOLUTION=640x360
chunklist_b978000.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=481000,CODECS="avc1.66.30,mp4a.40.2",RESOLUTION=426x240
chunklist_b446000.m3u8


curl https://media.smeplanet.com/stream/patrickz/ngrp:live_all/chunklist_b4227072.m3u8
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:10
#EXT-X-MEDIA-SEQUENCE:806
#EXTINF:10.0,
media-uth9ngn0v_b4227072_806.ts
#EXTINF:10.0,
media-uth9ngn0v_b4227072_807.ts
#EXTINF:10.0,
media-uth9ngn0v_b4227072_808.ts


siege -t 15s https://media.smeplanet.com/stream/patrickz/live/media-uth9ngn0v_b4227072_806.ts
** SIEGE 4.0.4
** Preparing 25 concurrent users for battle.
The server is now under siege...
Lifting the server siege...
Transactions:                     25 hits
Availability:                 100.00 %
Elapsed time:                  14.13 secs
Data transferred:             116.24 MB
Response time:                 12.04 secs
Transaction rate:               1.77 trans/sec
Throughput:                     8.23 MB/sec
Concurrency:                   21.30
Successful transactions:          25
Failed transactions:               0
Longest transaction:           13.89
Shortest transaction:           1.68

Actually it is quite easy, but unfortunately for live-streaming, some server give the chunk list or .ts for a moment. These file will expire and you will get 404 (file not found). So this is quite annoyed me.

Well, I would like to suggest a HLS load test tool name "Artillery".  Artillery build by NodeJS for load testing and functional testing. Ship scalable backends, APIs & services that stay performant & resilient under high load. It supported HTTP and WebSocket.

Let's try! You need to install node.js then module artillery.

npm install -g artillery

#try
artillery quick --count 10 -n 20 https://media.smeplanet.com/stream/patrickz/live/media-uvvn90vyy_1370.ts

Started phase 0, duration: 1s @ 13:33:24(+0700) 2020-08-21
Report @ 13:33:34(+0700) 2020-08-21
Elapsed time: 10 seconds
  Scenarios launched:  10
  Scenarios completed: 10
  Requests completed:  200
  Mean response/sec: 20.3
  Response time (msec):
    min: 3
    max: 535.9
    median: 5.3
    p95: 23.5
    p99: 532.6
  Codes:
    200: 200

Report @ 13:33:34(+0700) 2020-08-21
Elapsed time: 10 seconds
  Scenarios launched:  0
  Scenarios completed: 0
  Requests completed:  0
  Mean response/sec: NaN
  Response time (msec):
    min: NaN
    max: NaN
    median: NaN
    p95: NaN
    p99: NaN

All virtual users finished
Summary report @ 13:33:34(+0700) 2020-08-21
  Scenarios launched:  10
  Scenarios completed: 10
  Requests completed:  200
  Mean response/sec: 20.24
  Response time (msec):
    min: 3
    max: 535.9
    median: 5.3
    p95: 23.5
    p99: 532.6
  Scenario counts:
    0: 10 (100%)
  Codes:
    200: 200

Install plugin artillery-plugin-hls

npm install -g artillery-plugin-hls

Prepare and run a test script

hls.yml

config:
  target: 'https://media.smeplanet.com'
  phases:
      - duration: 300
        arrivalRate: 250
  plugins:
    hls: {}
      
scenarios:
  - flow:
    - get:
        url: "/stream/patrickz/live/playlist.m3u8"
        hls:
          concurrency: 2
          streamSelector:
            resolution:
              width: 1280
              height: 720


artillery run hls.yml

Report @ 13:41:48(+0700) 2020-08-21
Elapsed time: 2 minutes, 2 seconds
  Scenarios launched:  41
  Scenarios completed: 37
  Requests completed:  40
  Mean response/sec: 4.08
  Response time (msec):
    min: 1024
    max: 1561.6
    median: 1285.4
    p95: 1526.2
    p99: 1561.6
  Codes:
    200: 40
  HLS: segment download time:
    min: 5558
    max: 7781
    median: 6607
    p95: 7071.6
    p99: 7482.9
  HLS: stream download time:
    min: 12116
    max: 14208
    median: 12896
    p95: 13836.8
    p99: 14208
  HLS: segment download completed: 117
  HLS: segment download started: 120
  HLS: stream download started: 40
  HLS: stream download completed: 37

That's it. However, I notice the application quite  high CPU usage.

Reference