Monitoring IPTV via Multicast
Whenever you offer IPTV streaming to customers, you need to monitor it. Unfortunately it becomes complicated, if you can not control all components. We were urged to measure the quality of the IPTV signal, but only have access to a single switch in the distribution network.
Short Outages
The main problem of IPTV is, that customers will notice every outage, even if it lasts a second. And every such outage will result in a serious complaint to the service desk.
Therefore the most important issue was to inform the service desk as soon as possible about missing multicast packets. A few seconds after the outage an alarm has to be raised. This way the service desk is informed before the customer complains. Due to the customer reaction time, the alarm needs to be delivered within five seconds.
What we did was simple: Take an old, low performing system, join all available streams (radio as well as TV) because it does not saturate the Gigabit, and collect per stream when the last packet was seen. If a stream does not received a packet for more than five seconds, report the stream and the corresponding parameters. Those reports are injected into the alarm system, which in turn sends SMS, email or open a popup window. Finally the data can be plotted: Using logarithmic axes to enhance the short outages.
As secondary result the reporting also tells which streamer is affected and how many streams are still available from this device. If the number of streams drops to zero, urgent replacement of broken hardware can be scheduled. The alarm system can inform a different set of people about this type of events.
Details
Customers do not only complain if the movie freezes or the TV goes black, they even call if the picture or audio is garbled.
That's why our software was extended to examine MPEG-TS packets itself. There is a lot of interesting information inside:
Datenstrom liegt von a.b.c.d an. Statistik über 3 Sekunden: Empfangene UDP Pakete 3446 Paket zu kurz 0 Verarbeitete MPEG Pakete 24122 Fehlende Sync-Markierung 0 Statistiken pro Datenstrom (MPEG-Programm) Programmkennung (PID) 6630 6610 6622 6620 6621 6600 0 32 19e6 19d2 19de 19dc 19dd 19c8 0000 0020 Empfangene Pakete 56721738 865 494 377 29 28 24 verschluesselte Daten 021366 829 448 331 0 0 0 Ansetzpunkte, FullFrames 0 5 18 23 23 0 0 0 priorisierte Daten 0 5 0 0 0 0 0 0 Signalstoerung am Streamer 0 0 0 0 0 0 0 0 Taktstoerung am Streamer 0 0 0 0 0 0 0 0 Sequenzfehler(CC) 0 0 0 0 0 0 0 0 Abweichungen vom Synchronisationstakt 19d2 (absoluter Jitter) 26% || 20% || 14% |||| 8% |||||| 2% |||||| +---------------------------------------------------------------------- ms 0123456789012345678901234567890123456789012345678901234567890123456789 0 1 2 3 4 5 6 Abweichungen der Paketankunftszeiten (relativer Jitter) 66% | 52% | 37% | 22% | 7% || |||| +---------------------------------------------------------------------- ms 0123456789012345678901234567890123456789012345678901234567890123456789 0 1 2 3 4 5 6 Kanal ist durch Kunden im Moment abonniert.
Such a HDTV channel has high data rate and a low jitter. A radio stream is quite different:
Datenstrom liegt von a.b.c.e an. Statistik über 3 Sekunden: Empfangene UDP Pakete 43 Paket zu kurz 0 Verarbeitete MPEG Pakete 301 Fehlende Sync-Markierung 0 Statistiken pro Datenstrom (MPEG-Programm) Programmkennung (PID) 851 0 850 0353 0000 0352 Empfangene Pakete 282 13 6 verschluesselte Daten 0 0 0 Ansetzpunkte, FullFrames 0 0 0 priorisierte Daten 0 0 0 Signalstoerung am Streamer 0 0 0 Taktstoerung am Streamer 0 0 0 Sequenzfehler(CC) 4 0 4 Abweichungen vom Synchronisationstakt 0353 (absoluter Jitter) 59% | 46% | 33% | 19% | 6% | | | | | || || || | || | | +---------------------------------------------------------------------- ms 0123456789012345678901234567890123456789012345678901234567890123456789 0 1 2 3 4 5 6 Abweichungen der Paketankunftszeiten (relativer Jitter) 15% | 11% | 8% | | | | 5% | | | | | | || | 1% | || ||| |||||||||||| || | | | +---------------------------------------------------------------------- ms 0123456789012345678901234567890123456789012345678901234567890123456789 0 1 2 3 4 5 6 Kanal ist durch Kunden im Moment nicht abonniert.
MPEG-Transport-Streams (representing a single TV channel) consist of several programs (sub streams), which are reported separately. The program 0000 contains the index, the information which program is used for which purpose. But even without decoding the index, it's easy to derive that the highest bandwidth program will represent the picture stream. Other programs are teletext, subtitles, EPG, on or more audio channels. ZDF-HD is very rich in respect of those programs.
Statistics
Plotting those data is an obvious wish, simply to investigate customer complains post mortem. In order to find out, what was going wrong last evening, historical data is needed.
Aggregating data over a minute is use to limit the amount of recorded data: The database has to store a single group of measurements per program and minute. And those data can be viewed.
For the first example that the bandwidth as (packets * size / time). Please notice how the bandwidth of an HD-channel is dominated by the movie/picture stream.
Between 17:30 and 20:00, the bandwidth dropped, and recurred with half the nominal bandwidth. A reasonable explanation is, that the broadcasting organization switched to SDTV material, i.e. coming from an different source.
Some broadcasting organizations, especially the third programs in the German TV market, do copy the whole stream from the ARD for common broadcasting (i.e. Tagesschau). The switch does show even the different program id.
But let's go back to the bandwidth drop. The number of full frames (restart points for the decoder) may reveal, if there is a problem in the local distribution or at the broadcasting organization. The network itself is usually unable transcode the stream and inject full frames by itself.
During this period, the broadcasting organization did repeatedly retry to sync their own signal. The level of resync-points is obviously lower than the full HD stream, but the other programs contains much more resync information than normal.
It's possible, that the streamer lost the (analog) signal. In this case a special bit is set in the MPEG-TS to indicate such a loss of signal, which allows the decoder to restart from scratch instead of waiting for additional information.
There was such as loss of (analog) signal at the very beginning of the disruption. But the following effects are solely caused by the received signal itself.
There is an other source of trouble which can occur at the broadcasting organization: In order to synchronize the independent programs of the MPEG-TS, a common 33 MHz clock is used. Using this clock, audio and video streams do not diverge. Such a high clock may (according to the standard) be also used to regenerate or calibrate the clock in cheap decoder hardware. But even this clock might be changed. In this case a special bit is set by the broadcasting organization to inform the decoders.
Even this bit clearly shows, that the disruption was caused by the broadcasting organization. An error in the local distribution infrastructure can be denied.
A still open question is, how to detect packet loss and packet reorder, the most common problems with IP networks. Those are detected by monitoring the sequence numbers (CC) of the MPEG-TS. Unfortunately the sequence number space runs cyclic from 0 to 15, while up to seven MPEG-TS packets are packed into a single UDP packet. This renders this number nearly useless.
The most interesting point on this graph is the small amount of CC errors during the disruption. Several data packets are lost or reordered. A possible reason might be, that the programs are resynced over and over again from the broadcasting organization itself. The sequence counter can be reset in this case.
Finally let's take a look to the jitter:
No more questions.
Summary
It's possible to monitor several hundred of IPTV streams without spending lot's of money. Just do it.
Freundliche Grüsse
BeoService
thank you for the document published. I am curious to know the name of your software and the current status of it.
We have a new IPTV service deployed, and we are searching ways to monitor it.
Best
Total 3 comments