Introduction

Intermittent audio can be complex to troubleshoot especially on larger system that may contain more than one network audio transport and may also involve and IT network. In this guide, we will cover the basics of how audio synchronizes between devices and cover ways to determine a root cause and correct intermittent audio.

General Concepts for Digital Networks

Clocking

Running digital audio between devices requires that both devices have their sample clocks synchronized or phase locked so that the data in each sample lines up between devices. Errors in timing affect the measurement of the signal amplitude and are hear as noise. On larger systems this can even involve multiple audio formats connected to a common clock each with their own sub clock. For more information on clocking see Clocking in Soundweb London.

JItter

As Audio travels over a network some of the packets may not arrive at regular intervals because of network processing load or networks ability to handle multicast clock. We call this jitter. As some packets arrive late the distance of time between the packets changes.

If there is enough jitter in the system then the clock signal will not be able to maintain a phase lock and the audio will mute as the device tries to reestablish sync to prevent digital noise.

Device Sync

Each DSP will sync to either its internal clock, BNC, BLU link, Dante, CobraNet, or AVB depending on its features. Opening the default control panel on a device is good place to start when checking its sync. Master indicates the device is set to internal clock.

Changes in Clock-Device Sync

Changes in clocking can usually be seen in the event log and also the default device control panels. Usually they will result in a sync change from one source to another and back. Sync changes can result a new device becoming the system clock because of changes to clocking scheme, device startup/rebooting or changes in the network that affect the distribution of the clock signal.

This can usually be seen in the event log as a sync change.

Whenever a sync change occurs the system will mute momentarily. If the case of too much Jitter the system will may mute and unmute again as the device tries to recover from the timing offset.

Clocking Conflicts-Multiple Clocks

In some cases there may be a conflict between two clocks. For example you might have a device trying to clock to both which will display a multiple conductors. In this case the DSP may display sync error and it will mute the audio. You may also see multiple conductors in the event log meaning there are two or more system clocks competing. The Audio Architect event log may also display multiple conductors.

Troubleshooting BLU Link

Direction of Audio

BLU link runs between devices without any switches or other network equipment. The output of each devices feeds the input of the next device. When the loop is broken the last and first device reverse the direction of audio. When this happens audio is back channeled on redundant channels this allows the loop to have some fault tolerance. While this fault tolerance is a good thing it can make troubleshooting difficult based on audio only especially if you don't understand the actual audio flow.

Direction of Clock-Sync Change

Each device in the BLU Link ring must be set to same clock source. When a break occurs in the BLU link ring the clock will reverse direction where needed to complete the clocking loop. If the reverse in audio results in a device changing its sync (BLU link, BLU link Out, Dante) then it will log that in the event log as a sync change. In the example below if we break the cable between the first two units the units two and three will report a Sync Change. If we break the connection between only the second and third unit then only the third unit will change its direction of clock. As general rule the devices that are fall after the break will sync to their BLU link output.

Event log Example for Changing Sync to BLU Link Out/In

In the case on an intermittent issue we could look at the event log in this case to determine which units were changing sync and which unit was master. In some cases this might tell us exactly where the problem was. 

Valid and Link

The default control panel also has link and valid indicators for each BLU link port. Link indicates the port has gigabit link to another device. You will get this even if you plug it into any gigabit device even if that device doesn't work with BLU link. Valid indicates BLU link audio is present on the port and the connection is correct. The valid indicator is a good way to see the status of the audio between devices. If a device is not wired correctly such as input to an input the link indicator will light but the valid will not. Also if the device is wired incorrectly the sync indicator will show sync error.

If a BLU link port is showing a status other than green you can try the reset port button. In some cases if this port is disabled because of errors this will restore the functionality.

BLU Link Network Link Problems

If devices are not showing link on a BLU link port between two units you can perform a loop back test. Simply connect the BLU link output to the BLU link input. If both ports light up then the problem is either the adjacent device or a problem with cabling. You can perform the same test on the next device not showing link. If a device doesn't show link on either of its ports when looped to itself you can try the BLU Link port reset on the default control panel. If that doesn't work then that indicates a problem with the ports on the device and it should be serviced. BLU-BIB and BLU BOB devices don't have a reset since they don't talk directly to the software.

Divide and Conquer a Strategy for Troubleshooting BLU Link

When troubleshooting a BLU link issue, a good strategy is to break things down in a way that makes it simpler to see what is happening.

• Simplify Clocking
  • If the BLU link ring is clocked off another source such as Dante, CobraNet or BNC try disconnecting that source and see if there is any change.
• Break the Ring
  • Breaking the ring so that it is not a complete loop will allow you to see where the sync change is taking place. Note: Devices such as BIB's and BOB's don't report to the event log and may need to be removed from the ring during the initial testing and then added back in one at a time until the culprit is identified.
• Use the event log
  • With the loop broken down use the event log to see where errors occur and if they occur. Amplifiers do not store their own logs so it is important to have audio architect connected and online with them when logging errors.
• Test Individual units
  • The BLU link ports on each unit can be tested by looping the BLU link in to the BLU link out.
• Use logic for debug
  • You can link logic sources to BLU link and valid LEDS. These logic source can be tied to logic ends with custom event log entries mark when a connection is dropped or changed. See Custom Entries Application Guide.

BLU Link Port Reset and Fiber Converters

In cases with fiber converters and intermittent connections it is possible see sync change happen very rapidly as the converter loses and regains sync which can result in loss of audio. The BLU link circuit has the option of shutting off a port if too many errors occur within 1 second. There is also a threshold that can be setup for auto disable. You find these settings under the general section of the venue explorer and assign them to a custom panel.

The BLU link priority determines which device in the BLU link ring becomes master of the ring. Devices clocked to Dante/CobraNet will use those as higher priority and attempt to clock other devices in the ring to their clock.

Dropout versus Sync Change

Sync change will result in a device muting all of its inputs and outputs. This can happen as a result of re-clocking or losing/gaining a BLU Link. Dante audio dropout because of latency will generally only affect the flow of audio that is not arriving in time.

Dante

Device Sync

Troubleshooting Dante has many similarities to BLU link since they are both digital protocols. Like BLU link devices, Dante devices have to maintain sync. If a device doesn't maintance sync it will report a red sync LED in Dante control and you will see device lost sync in the Dante Controller event log.

Precision Time Protocol

Dante uses PTP (Precision Time Protocol) to distribute clock over an IT network via Multicast signal. All devices on the Dante network must sync to the same clock, otherwise devices will have their clock drift and they will mute. Anything that interferes with the PTP clock will affect performance. Asymmetrical Networks such as MPLS and GPON using non-multicast technologies can fall into this category as well as networks with a misconfigured ACL. Symptoms can include multiple PTP master showing as well as devices losing Sync and muting.

You can check the sync/mute of a Dante Network on the Dante Controller Clock Status Tab.

You can check to see how the network is handling the clocking. Opening the clock status monitor and selecting a device on the history tab will show you an overall status of its clocking including a history of corrections. As the correction become larger you will see the color change to yellow and then red showing the severity.

In cases where there is a lot of jitter you try enabling unicast delay requests. In cases where the network is not handling multicast very well this can sometimes stabilize things.

Audio Drop Out from Network Latency

You can verify the latency in Dante controller under the Latency tab on the receiving devices. If the network takes longer to deliver the audio than the latency setting provides for you will get dropout. If this happens try increasing the latency in the device configuration. When setting latency keep in mind Dante will use the latency of the highest device when sending audio between two devices.

Secondary Networks

The Dante Secondary Network must always be on a separate VLAN or preferably separate physical network. If these two networks become bridged, audio will stop working and you may also experience other trouble even talking to the device. Dante controller may or may not notify you if the network becomes bridged, so it is a good idea to try and disconnect the secondary network if you are not sure where the problem is.