This post was inspired by Turgon from the techexam board about change control. A few weeks ago I had the pleasure of bringing down a few offices sharing a connection across a SIP trunk. So how could a guy who likes to brag about being a network ninja be so dumb? hmmm. So what went wrong? I changed the duplex setting on a router without even checking to make sure the guy on the other side was even making sense (we'll come back to this later). I had a customer who had incoming calls over a SIP trunk that would have random errors. Sometimes the call would just drop. Other times it wouldn’t take DTMF, and other times if the call was transferring to an outbound call, the VOIP gateway wouldn’t make it. Add in a furious customer yelling at you over the past few days and we have a recipe for trouble. So lets get started!!!!!!
SIP appeared to be working correctly. The dial-peers where correct, SIP settings were correct and the numbers were registered. Debugs showed nothing out of the ordinary. "Stretching my head". Lets setup a syslog server and see if anything weird pops up over the next few hours in the debugs. BAM!!!!!! Here it is. I see late collisions popping up then they go away. I'm speaking to a Tier 1 NOC operator and he suggests changing the duplex settings. I ask him what is he set at. He's at auto and I'm at auto. He suggests going to 100 full. Now we are talking about Ethernet right? No time during this evolution did I think to question him on this SIP trunk since they give us 2 hand off's(once for voice and one for data). I assumed he was talking about the side for voice. We go to a 100 full I'm locked out, and I go to "OH SHIT". I'm not getting any response from the router. A reload in 10 would have saved me, but I didn't do it. MESSAGE on Playing with connectivity interfaces!!!! He still had access to his gear, I asked him to change the settings back to where they were (still haven't gotten to this story yet), and I had to call someone at the building to go reboot the gear, since they still had PHONES. Nobody answered cause they weren't in yet, so I'm assuming I may be driving out there. I finally get someone next door at the other company who goes and reboots the device for me. Back to normal and in time we make the correct changes.
Now where did I go wrong? Why didn't I ask these questions before hand? The young engineer changed the port settings on his Edge Juniper router. This wasn't the device I was connected to. I was actually connected to his Cisco IAD device so they could hand me a voice and data line. Why was he in his Juniper router instead of the IAD??? Never got around to asking him that, maybe he thought it was our equipment? The world will never know. In the end don't move to fast it can be costly. I got lucky this time. I had a buddy's bring down 10,000 call center agents, another black hole a entire section of a network, and another Reset BGP, and so on. I haven't seen anyone loose there jobs over these incidents, but we all have heard the stories. Happy Networking.
Update. I wrote this post a while back and never posted it This week I was working with a TAC engineer, we brought down a call center. Hence I'm up at 11pm doing troubleshooting at low call volume time. Once again this could have been solved with some pre planning, but I guess you learn by making mistakes.
Monday, December 20, 2010
Wednesday, December 15, 2010
I can't stop dialing 911
911. Its a number we often need in times of emergency as I have dialed this year when a man had a seizure in a office I was working at, at the time. But what about those other times when people dial it. What do they usually do? Most hang up right away then the boys in blue show up, sometimes they can be a little unhappy. Well today a customer of mines made that mistake and they got a fine as they have made the mistake several times recently. I guess nobody told them not to hang up when they call. We went over a few options on how they couldn't dial 911 and the best option was for them to change there outside dial to a 8. This was a CME system so it was done on the command line, but we can make some route patterns on a CUCM if there is interest. First thing we do is edit the dial peers. I have a few sample's below.
dial-peer voice 2004 pots
corlist outgoing call-local
description ** local dialing dial peer **
destination-pattern 8[2-9]..[2-9]......
port 0/3/0:23
forward-digits 10
!
dial-peer voice 2005 pots
corlist outgoing call-national
description ** long distance dialing dial peer **
destination-pattern 81[2-9]..[2-9]......
port 0/3/0:23
forward-digits 11
!
dial-peer voice 2006 pots
corlist outgoing call-international
description ** international dialing dial peer **
destination-pattern 8011T
port 0/3/0:23
forward-digits 10
prefix 011
Where the 8's where at were previously 9's. That was easy. We made a few outbound test calls they worked perfectly.
Now onto some telephony-service changes
call-forward pattern 8..........
call-forward pattern 81..........
transfer-pattern 8..........
transfer-pattern 81..........
secondary-dialtone 8
Yeah I realize this wasn't exciting as I thought it would be for a blog post. This is pretty straight forward stuff. In the end we made updated around 7-8 dial peers. We strip off the 8 with the correct forwarding digits and we sent calls to the PTSN. Give it a try one day!!!! Happy Networking
dial-peer voice 2004 pots
corlist outgoing call-local
description ** local dialing dial peer **
destination-pattern 8[2-9]..[2-9]......
port 0/3/0:23
forward-digits 10
!
dial-peer voice 2005 pots
corlist outgoing call-national
description ** long distance dialing dial peer **
destination-pattern 81[2-9]..[2-9]......
port 0/3/0:23
forward-digits 11
!
dial-peer voice 2006 pots
corlist outgoing call-international
description ** international dialing dial peer **
destination-pattern 8011T
port 0/3/0:23
forward-digits 10
prefix 011
Where the 8's where at were previously 9's. That was easy. We made a few outbound test calls they worked perfectly.
Now onto some telephony-service changes
call-forward pattern 8..........
call-forward pattern 81..........
transfer-pattern 8..........
transfer-pattern 81..........
secondary-dialtone 8
Yeah I realize this wasn't exciting as I thought it would be for a blog post. This is pretty straight forward stuff. In the end we made updated around 7-8 dial peers. We strip off the 8 with the correct forwarding digits and we sent calls to the PTSN. Give it a try one day!!!! Happy Networking
Sunday, December 5, 2010
Voice Translation Madness
Voice Translations
Ahhh the world of VOIP has finally come back to me, I'm adjusting to the new life, but I love being back in the technology. A few days back I received an interesting ticket on some voice translations that needed to be done for a customer. This was on a CME so using called and calling number translations weren't available to me as they would be in CUCM. There is an interesting story about that, that will be saved for another post later. So pretty much this customer wanted all inbound numbers to have a 9(off net call local) or a 91(off net long distance) added to them, so when a customer went into their directory, they don't have to edit the number to make a call, they can just dial it. Also when certain extensions dial out they wanted a few of the users DID to show instead of the main line number. Easy day, Well obviously if I'm writing a post this wasn't an easy ticket. First a quote that I follow. "Before we design a solution, it's often useful to define the problem to be solved" That is from the famous Radia Perlman. I had the problem in right? Well yes I needed to punch some code down and move on to the next ticket, but my approach was a little shaky. What I actually did is below and experienced guys will see the mistake.
1st thing I did was jump straight to the translations. I added my forward slashed and added the numbers to catch. I added my forward slashed put in the extensions to have their DID and then in the next set added the first numbers and \0/ at the end it to go back to set 0. EASY. Hell I even tested it out
router#test voice translation-rule 1 4127
Matched with rule 1
Original number: 4127 Translated number: 2025304127
Ok onto the next one
Rule was a carryover from the previous config so I left it in place. Next one are the area codes that need to get a 9 added. We live in a large area so we have several local area codes
/^302/ That means begins with so I can have a 302 later in the set, but it won't mach cause it has to begin with 302.
/\(.*\)9302\1/ This gets a little funky we have the first forward slash to indicate start of a match, but then we add a back slash "\" and it indicates a number slice the "(" indicates match numbers to keep, the "." indicates any digit, and the "*" indicates match none more numbers, which pretty much means everything we add in a ")". We add the translation to prepend a "9302" and we close it all then we add a "\1/" to indicate match set 1 which is ".*". So with all that anyone that dials in with a 302 number the number will show as a 9302 number in the directory of inbound calls. Even ran a test on this
router#test voice translation-rule 2 3025128425
Matched with rule 1
Original number: 3025128425 Translated number: 93025128425
The one for long distance numbers is pretty much the same. We are doing a catch all in the 1st match adding a "91" and matching the inbound number with set 0. So what did I do wrong? I tested it right. Well I tested it on the command line and not with a inbound call or having the customer make a outbound call based on me calling in. That's the clue I'll give. The config and where I applied it is all below.
voice translation-profile inbound
translate called 2
!
voice translation-profile outbound
translate calling 1
!
voice translation-rule 1
rule 1 /4127/ /202530\0/
rule 2 /4153/ /202530\0/
rule 3 /4151/ /202530\0/
rule 4 /4156/ /202530\0/
rule 5 /4160/ /202530\0/
rule 6 /4161/ /202530\0/
rule 7 /^.*/ /2025303180/
voice translation-rule 2
rule 1 /5858/ /5002/
rule 2 /^302/ /\(.*\)9302\1/
rule 3 /^401\(.*\)/ /9401\1/
rule 4 /^671\(.*\)/ /9671\1/
rule 5 /^803\(.*\)/ /9803\1/
rule 6 /^340\(.*\)/ /9340\1/
rule 7 /^.*/ /91\0/
Ahhh the world of VOIP has finally come back to me, I'm adjusting to the new life, but I love being back in the technology. A few days back I received an interesting ticket on some voice translations that needed to be done for a customer. This was on a CME so using called and calling number translations weren't available to me as they would be in CUCM. There is an interesting story about that, that will be saved for another post later. So pretty much this customer wanted all inbound numbers to have a 9(off net call local) or a 91(off net long distance) added to them, so when a customer went into their directory, they don't have to edit the number to make a call, they can just dial it. Also when certain extensions dial out they wanted a few of the users DID to show instead of the main line number. Easy day, Well obviously if I'm writing a post this wasn't an easy ticket. First a quote that I follow. "Before we design a solution, it's often useful to define the problem to be solved" That is from the famous Radia Perlman. I had the problem in right? Well yes I needed to punch some code down and move on to the next ticket, but my approach was a little shaky. What I actually did is below and experienced guys will see the mistake.
1st thing I did was jump straight to the translations. I added my forward slashed and added the numbers to catch. I added my forward slashed put in the extensions to have their DID and then in the next set added the first numbers and \0/ at the end it to go back to set 0. EASY. Hell I even tested it out
router#test voice translation-rule 1 4127
Matched with rule 1
Original number: 4127 Translated number: 2025304127
Ok onto the next one
Rule was a carryover from the previous config so I left it in place. Next one are the area codes that need to get a 9 added. We live in a large area so we have several local area codes
/^302/ That means begins with so I can have a 302 later in the set, but it won't mach cause it has to begin with 302.
/\(.*\)9302\1/ This gets a little funky we have the first forward slash to indicate start of a match, but then we add a back slash "\" and it indicates a number slice the "(" indicates match numbers to keep, the "." indicates any digit, and the "*" indicates match none more numbers, which pretty much means everything we add in a ")". We add the translation to prepend a "9302" and we close it all then we add a "\1/" to indicate match set 1 which is ".*". So with all that anyone that dials in with a 302 number the number will show as a 9302 number in the directory of inbound calls. Even ran a test on this
router#test voice translation-rule 2 3025128425
Matched with rule 1
Original number: 3025128425 Translated number: 93025128425
The one for long distance numbers is pretty much the same. We are doing a catch all in the 1st match adding a "91" and matching the inbound number with set 0. So what did I do wrong? I tested it right. Well I tested it on the command line and not with a inbound call or having the customer make a outbound call based on me calling in. That's the clue I'll give. The config and where I applied it is all below.
voice translation-profile inbound
translate called 2
!
voice translation-profile outbound
translate calling 1
!
voice translation-rule 1
rule 1 /4127/ /202530\0/
rule 2 /4153/ /202530\0/
rule 3 /4151/ /202530\0/
rule 4 /4156/ /202530\0/
rule 5 /4160/ /202530\0/
rule 6 /4161/ /202530\0/
rule 7 /^.*/ /2025303180/
voice translation-rule 2
rule 1 /5858/ /5002/
rule 2 /^302/ /\(.*\)9302\1/
rule 3 /^401\(.*\)/ /9401\1/
rule 4 /^671\(.*\)/ /9671\1/
rule 5 /^803\(.*\)/ /9803\1/
rule 6 /^340\(.*\)/ /9340\1/
rule 7 /^.*/ /91\0/
Subscribe to:
Posts (Atom)