Greetings everyone! Once again I bring you a gripping tale of daring and exciting adventure! Okay, so it’s not really all that daring and — according to my wife — less exciting than watching paint dry, but it’s relevant nonetheless. If you’re just looking for the important bits because you’re having issues that appear to be from DTMF, feel free to jump past the blah blah blah part and start at the resolution. You won’t hurt my feelings. I promise. But, let’s start out at how I got here.
In The Beginning …
Our story begins a couple months ago.We had a turn up at a client who was moving from old analog lines to SIP being provided by one of the local telecoms. On the client’s side, we’ve got Skype for Business Server connected to an AudioCodes Mediant 800B. From the Mediant, there is a single network cable that plugs directly into an Inno Media ESBC 9378 which is then connected to the telecom.
The initial turn up on a Friday afternoon seemed to go well enough. We used test DIDs to make inbound and outbound calls, transferred calls, put them on hold, resumed the call, you know, all those things that you do when you’re getting ready to port numbers. Everything worked, so we ported the numbers, did our testing again, things seemed good to go and so we accepted service on behalf of the client. Monday is when our troubles began.
The Plot Thickens …
We received word from the client that all was not well with the world. Syllables were being dropped, voices sounded like robots, just all around poor audio quality. Multiple phone calls to the telecom, escalation through a couple levels of support and four support engineers spanning several days led to the discovery that the DTMF Transcoding engine on the ESBC was malfunctioning. Their engineer disabled it, and hey, we’ve got great call quality! He re-enabled it and all appeared right with the world. I guess it had just lost its mind for a bit and restarting it fixed things. We danced a celebratory jig, did a few virtual fist bumps and went on our merry way.
Just When You Think It’s Safe …
Time passes, the client becomes accustomed to the new phone system, I move forward with some new projects. Then we get a call. Strangely, some people aren’t able to call the client from their cell phones. The initial discovery was that callers from Republic Telecom were unable to call. All other calls from other carriers were clear, these just wouldn’t connect. Further troubleshooting revealed that this included T-Mobile and Sprint. Pulling some logs, we found that SFB was refusing the calls and it appeared to be due to malformed SDP coming from those carriers.
So, here’s our Invite from a call that works:
And here’s an invite from one that doesn’t:
Pretty similar, but there’s a couple differences. In the one that works, we’ve got this:
In the one that doesn’t work, we’ve got this:
The immediately obvious thing is we’re missing a=fmtp:101 0-15 , but we’re also missing the 101 option in our media type descriptor. We’ll get back to this in a moment.
After passing these logs onto the telecom, they passed them further up the chain and someone somewhere decided that a hardware replacement was in order as they found an “issue”. I’m not sure what that issue was, but long story short, they replaced the eSBC and testing began. Amazingly, we’re able to call from Sprint based carriers now, but, unfortunately, that crystal clear audio that we used to have had now gone away. We’re back to the original issue of dropped syllables and robotic voices. This time, however, we are able to hear an almost clockwork frequency to the audio issues, the dropped syllables no longer sound random.
At this point I want to pause and give some props to James at the telecom. He was extremely knowledgeable and infinitely patient, he worked with me hand in hand to pull packet captures, sip traces, troubleshoot and finally find root cause for what was going on. He also taught me quite a bit more about reading SDP in the course of our adventures. I can only hope that all the engineers I get to work with are as dedicated as James.
Now, back to the story! In the course of troubleshooting, we pulled packet captures, made so many test calls that I lost count, configured SFB for TCP and connected it straight to the telecom’s eSBC, and when it was all said and done, we identified that the problem was in fact with the newly replaced eSBC. Time to dispatch another replacement.
Here We Go Again …
Replacing the eSBC didn’t fix it last time, and I’m sure you’ve already guessed, it didn’t fix it this time. Strangely enough, we found out that this new one was only five units off from the one it replaced, so they rolled off the same assembly line on the same day, five devices apart from each other. Coincidence? Maybe.
We began re-examining the SDP from our calls. I just knew that things were missing from the SDP. James explained what I was looking at and how the missing parts were significant.
A quick breakdown for you on this: “m=” is the field for media type, format, and transport address. “m= media port transport format list”. So here we have audio, port 62142 for RTP (RTCP will be the next odd port, so 62143), RTP/AVP defines RTP protocol with profile for “Audio and Video Conferences with Minimal Control”. 0 is for G.711 uLaw. It’s the 101, however, that is missing link here. The “101” means that AVP can be introduced on a dynamic basis. As the Sprint-based carriers present DTMF in G.711 uLaw and Skype for Business requires RFC 2833, this is what the DTMF Transcoder in the eSBC is supposed to be taking care of for us.
Resolution!
Disabling DTMF Transcoding on their eSBC immediately fixed the audio quality issues, but introduced new ones. Sprint-based carriers were again unable to call in, and calls out or in from other carriers were unable to use DTMF options. Enter AudioCodes. The Mediant 800B has the ability to handle that transcoding part for us, so we decided to configure it as such and here’s how I did it.
Under VOIP > Coders and Profiles > IP Settings, you’ve got your IP profiles. In our case, one for the connection to Skype for Business and one for the ITSP. We selected each of these and clicked Edit. Bear in mind, we’re running firmware 6.8 on our Mediant 800B, so my screenshots may vary from what you see on yours. First, we select the IP Profile for the ITSP and choose the SBC tab. There’s three items that I set: Transcoding mode = Only if Required, RFC 2833 Behavior = As Is, Alternative DTMF Method = In Band.
Next we choose the IP Profile for the connection to SFB and we set it the same with one exception, we set RFC 2833 Behavior = Extend.
The last place we need to make a change is under VOIP > Media > Voice Settings, and we need to set DTMF Transport Type to RFC 2833 Relay DTMF.
Last setting is in place, we make some test calls and like magic, great audio quality! Inbound calls from Sprint-based carriers work, DTMF options are working, all being handled by the Mediant.
Epilogue
After further discussion, it appears that there may be a flaw in the DTMF Transcoder on the Inno Media ESBC 9378 as this is the third one of these that we’ve had issues with. The telecom is sending the faulty devices to their engineering department for further testing and to work with the manufacturer to find how widespread this issue might be. Ultimately, though, the client is happy and their phones work as expected.
Again, a big thanks to James. It’s always a pleasure to work with people that know what they’re talking about.