Revisiting customer service. (Old post Series pt 1).

This past week has been a real difficult one. The usual outages, added on with a car breaking down, and life. The particular incident in question revolved around a medical center with a sip trunk. Pretty simple setup

2 sites with UC500 series routers.
2 Sonicwall Firewalls
1 SIP trunk to a provider.


We get a call saying they can't dial out or inbound calls aren't working. I logged in and noticed right away the numbers aren't registered to the sip server. I placed a ping to the sip server from my desk top(IP's have been changed to protect the innocent)



C:\Users\urbanciscoengineer>ping 107.141.31.21 -t

Pinging 107.141.31.21 with 32 bytes of data:
Reply from 107.141.31.21: bytes=32 time=28ms TTL=244
Reply from 107.141.31.21: bytes=32 time=28ms TTL=244
Reply from 107.141.31.21: bytes=32 time=28ms TTL=244
Reply from 107.141.31.21: bytes=32 time=28ms TTL=244
Reply from 107.141.31.21: bytes=32 time=28ms TTL=244
Reply from 107.141.31.21: bytes=32 time=28ms TTL=244
Reply from 107.141.31.21: bytes=32 time=28ms TTL=244

Looks like Its up and reachable lets try from the UC500 device.



Traceroute from UC540
traceroute
Type escape sequence to abort.
Tracing the route to 107.141.31.21.ptr.us.xo.net (107.141.31.21)

1 ip65-44-248-57.z248-44-65.customer.algx.net (65.44.248.57) 0 msec 4 msec 0 msec
2 ip65-46-107-181.z107-46-65.customer.algx.net (65.46.107.181) 4 msec 8 msec 8 msec
3 * * *
4 * * *
5 ae0d0.mcr1.nyc-ny.us.provider.net (216.153.0.18) 8 msec 12 msec 12 msec
6 216.156.7.234.ptr.us.provider.net (216.256.7.134) 8 msec
216.156.7.182.ptr.us.provider.net (216.154.7.182) 8 msec
216.156.7.234.ptr.us.provider.net (216.136.7.244) 8 msec
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *


As you can see we get lost in the network. I also traced it out from my desktop. I sent all the captured info to the user, and inform him he needs to speak to the provider. I even said he could call me back when the provider was on the line which he took me up on. So the provider calls me back and I tell him what we found. He's still saying this is our problem. This isn't making any since to me. I asked for the ticket to be escalated to someone from their IP team as this is not a SIP issue at this point. I asked the customer to stay on this as we can't be responsible for babysitting there providers. Now the customer service comes into play. After realizing that this guy isn't going to do what we need from his demeanor, I decided to stay in contact with the provider through the evening. They still couldn't get anything solved. The next morning I inform my manager that this issue is going to heat up as they have been down over 24 hours. We get the provider on the phone, sales guy and me and the customer. We say the same thing over again "WE DON"T HAVE IP REACH-ABILITY TO YOUR EQUIPMENT!!!!!!". Thats plain as day there is something wrong in your cloud. They put me back in contact with the same guy again. We ask for a escalation which they finally got us and now its 12pm. The Tier 3 guy didn't get back in contact with us until 3pm and this was over chat with the first guy we spoken with. I provide my traceroutes again for the 3rd time and inform them we aren't changing anything until we have a clear path through there network to the SIP server.


As the day went on the finally found the access-list that was updated improperly and not allowing our IP to touch the SIP registar. Things like this make you want to cringe and wonder how are people getting these jobs at these places and why does the escalation take so long. This blog was about customers service, and we stayed with the customer till the end, but man what a waste of time that I dont get to spend with my other customer.