Overview

An automated monitor runs on all our streams to monitor for:

• network outage, when the contribution input from the studio goes offline
• dead air, when the stream remains silent for an extended duration

Alerts can be sent to one or more email addresses or Slack web-hooks.

Offline detection

Our monitor tools detect when the incoming audio connection is dropped and opens an outage. Once the connection resumes the outage is closed.

Automated alerts can be configured to trigger after an outage has been open for either 10, 20 or 30 minutes.

Volume measurement

Stream input volume is read once per minute. This information is available as a timeline graph under the stream admin dashboard.

Providers can use the timeline graph to ensure consistent volume over time for their customers that is not too soft or too loud.

Silence (dead air) detection

Volume measurement is also used to detect extended silences in the stream input.

Stream volume that remains below a threshold also opens an outage, which in turn can trigger an alert after being open for either 10, 20 or 30 minutes.

Alerts

Alerts can be sent to one or more email addresses or Slack web-hooks. An alert will be sent once an outage has been open for a configurable duration and a follow-up alert is sent once the outage is closed.

Note that email delivery times can be variable and our service cannot account for late delivery of email.

Slack web-hooks are called immediately and not (usually) subject to delivery times.

Alert intervals

Alerts should be actionable items. Receiving too many false positives can quickly lead to alert fatigue. More simply put, receiving a large amount of alerts that doesn't really require human intervention causes people to ignore important ones.

Streaming 24x7 audio over the internet is exposed to a huge amount of failure points, and downtime is inevitable.Most ISP's focus on providing large download through-put and not always reliable uploads.

The majority of these failures are temporary (frequently only seconds) and outside the control of the studio, so it helps to focus on longer lasting issues which may be caused by studio hardware or software.

Note on short outages

Short outages won't trigger alerts and can thus be "invisible" to providers but still cause significant user dissatisfaction. We already work with providers when we detect frequent short outages, and a future release will surface this directly to customers.