Going for CCIE Voice

I have been working on Voice for quite a while. I have been around since the call manager 4 days. I took about 15 month break to work in the normal route/switch world and that has helped me tremendously, but I have a love for voice and that's where I"m back too. I recently worked with a TAC engineer who I work with often and we discussed my daily job and it looked like I could go straight to the IE voice with some back filling of topics I'm weak in. I have decided to go for this. Stay Tuned As I post my way through tackling the written and the lab.

Don't let people SIP from your cup

Toll Fraud has been around for ages. Weather it was the Phreakers with there whistles or people using social engineering to get outside dial tone by calling a trusting receptionist its been easy to steal long distance or Internatonal calling if you were creative. In the new world of SIP, toll fraud has followed the new technology. I got a call from a customer about there 800 number being busy. A quick look at the router reviled that all 23 lines of there PRI was being used. The IT manager said thats not true. So I figured he may be getting hit by toll fraud. The IOS output also showed me that all the calls were coming in then being sent to the PSTN overseas. We shut down the SIP dial-peers and bounced the PRI since we didn't know what calls were legit. A quick google search of SIP toll fruad showed us how easy it was to do. Newer versions of IOS 15 have features in to block it. You can view them on the link below.




In the end this has been posted a few times on a lot of blogs. Protect yourself and keep your customers happy.

Hosted VOIP vs On site

As the world moves into the "cloud", telephony has made its way there. There are several vendors selling hosted Voip solutions to customers; while others are saying buy your own gear keep it on site and you control your own destiny. I've had the opportunity to work both, and while both have there Pro's and Con's companies need to understand what they are getting into and how to think about there future needs. IP telephony is not a one size fits all service and you have to closely think about the what you will be paying for.

Hosted VOIP Pro's
You can get a feature rich system, and all you pay for is the telephones, switches and a monthly fee for services you use. Some of the providers offering messaging to email, contact center, video, faxing services. You can get large corporation features in your small business that can give you an edge. You won't need an on site telephony person to take care of your needs, and you can even provision your own phones to add new users.


Your own equipment pro's:
You own your own gear. This means you are free to find the cheapest service you can get from a provider. Also if you don't like the service contract you purchased, you can go find someone else. All you need to do is change passwords and cut access. If you have an on site Engineer, you can get changes done quickly without putting in tickets to a service desk. You can work on crazy customizations without having to pay a fee from your provider.


Hosted VOIP Cons:

You are at the mercy of the provider. They own the call processing component. If you want a feature they don't offer you are out of luck. Engineers you work with can vary in skill. If they have a outage in the data center you are stuck like chuck. One I worked with had a broadcast storm. Calls for 250 clients didn't go to the PSTN. The Hosted call center went down. We also had several voicemail outages during my time there. If the company goes out of business guess what? Who are you going to be getting your call processing from. Costs can sneak up on you. A company with high turnover MACs can add up quickly. Companies with expected high turn over like call centers should be very careful when choosing a hosted solution.


On Site equipment Cons:
Cost, Cost, Cost. There are huge upfront costs associated with getting your own telephony on site. Cisco, Avaya, Nortel, can be pricey. You will have to pay for installation, and depending on the size of your company you will have to pay for on site employees, or you have to contract it out to a provider. What I have also seen is that some other network folks inherit it, and they just want to stay as far away from the phones as they can. Something we take for granted like phones are a nightmare to people who keep the VOIP packets flowing.


At the end of the day you have to decide what your company can afford and your needs. I like both solutions, and I've seen both of them fail when IT managers didn't ask the right questions about what they were purchasing and leverage there needs against them.

When poop hits the fan!!!!!

So we talk about this a lot on the boards and in real life. So how do we handle this sort of event. You get a call that XXXX is down and nobody has internet or phone access across several sites? Where do you begin? Do you hit the sweat button? Is this your time to take a coffee or smoke break to get mentally ready to get in the game? Some people like this part of the job more than others. I just happen to be a guy that likes to deal with outages. I have several thought the day ranging from 1 user has no access to my entire company can’t make calls. I’m going to shed some insight on a method to the madness of how you can handle the call, excel, and be the envy of your peers. I also want to hear from you, you might know something better and we can all learn something new.





To me I approach all outages the same. Before I jump into gear, I ask preliminary questions?



1. What is the problem?

2. When did it happen?

3. How many users are affected?

4. Any changes to your system recently?





Ok let’s be honest, on some of these questions nobody is going to be completely honest with you most of the time. I’m usually talking to another IT guy that was probably messing around and doesn’t want to come clean, and there are several stories with that regarding call centers and auto attendants that I will explain one day, But this gives us a place to start. Next I get access to the network in whatever place is left. Sometimes it’s a terminal server if it’s a complete outage, sometimes I can VPN into another site. Once I’m in I get my tools ready and fire most of them up. Usually SecureCRT, Notepad++ Kiwi Cat Tools, Wireshark, Kiwi Syslog, and a command prompt for pings. Next I get access to whatever devices seem to be the problem or the closest I can get to it. Now comes my initial play book. SHOW Commands regarding the particular technology and checking the log (I’m also checking for last login and changes made by). 75-95 percent of the time the problem is right there if I go up the OSI model. Other times debugs will be needed. That’s where the syslog sever and other tools come in. This Is now the time when people are starting to ask questions. The usual answer of I’m running debugs looking for any errors is enough to back up most people I have dealt with. Other times the VP of technology or whatever are on the phone and they start asking questions, usally they dont know what the hell they are talking about, One enterprise architect told me that he could reach a 192.168 network from his connection at home:) I usually tell them give me a min I’m going to put you on hold, while I gather more info. If they become too problematic I can get someone else in to run damage control cause there asking for updates every 5 min taking time away to solve the problem. If I still don’t have it up after 30-45 min and it’s a global outage I will reboot whatever device it is. That has more success rate than we like to give it credit to for smart IT guys the old raytheon reset is below us. Then after a hour or so it’s time to escalate to someone smarter than me. Sometimes it’s to cisco, other times its to another engineer in the office who has worked with that technology. In the end we solve some, others kick out arse, but it’s how we all learn.

My CCIE rack and some company in DC

Over the past few weeks things have been chaotic for me. New job, and other personal endeavors have taken up most of my time (I'm about to get engaged). I just had to post this last encounter as it was too odd. Earlier this week I got a call around 4:45 pm about a network that keeps going down. We had some systems people in there working on Exchange, citrix and netapp. They had been rebooting their switches and the network would come back up and the switches would go haywire again. Anyways it gets so bad that someone had to go. I figured it was on my way home and I'll just unplug some cables break the loop and get back to normal life. Well 7 hours later I was still there with no resolution. First off we didn't have management to the switches and they just appeared to have some weird behaviors. So I'm getting hell from the switches, the owner of the company, and my girlfriend so how do we solve this problem. Back at the job there were no switches that met the needs of the customer, but wait...I have a couple of switches in my rack back home. I spoke about the switches I had and management agreed to put them in place the next day. Came home saved my configs from them, erased them, and got them ready. Plugged them in and the network is pumping out packets like a champ. What have we learned from this? Site surveys mean everything, make sure you have access to the entire environment your about to change.

Tier 1 NOC operator+ Shodown not paying attention = FAIL

This post was inspired by Turgon from the techexam board about change control. A few weeks ago I had the pleasure of bringing down a few offices sharing a connection across a SIP trunk. So how could a guy who likes to brag about being a network ninja be so dumb? hmmm. So what went wrong? I changed the duplex setting on a router without even checking to make sure the guy on the other side was even making sense (we'll come back to this later). I had a customer who had incoming calls over a SIP trunk that would have random errors. Sometimes the call would just drop. Other times it wouldn’t take DTMF, and other times if the call was transferring to an outbound call, the VOIP gateway wouldn’t make it. Add in a furious customer yelling at you over the past few days and we have a recipe for trouble. So lets get started!!!!!!


SIP appeared to be working correctly. The dial-peers where correct, SIP settings were correct and the numbers were registered. Debugs showed nothing out of the ordinary. "Stretching my head". Lets setup a syslog server and see if anything weird pops up over the next few hours in the debugs. BAM!!!!!! Here it is. I see late collisions popping up then they go away. I'm speaking to a Tier 1 NOC operator and he suggests changing the duplex settings. I ask him what is he set at. He's at auto and I'm at auto. He suggests going to 100 full. Now we are talking about Ethernet right? No time during this evolution did I think to question him on this SIP trunk since they give us 2 hand off's(once for voice and one for data). I assumed he was talking about the side for voice. We go to a 100 full I'm locked out, and I go to "OH SHIT". I'm not getting any response from the router. A reload in 10 would have saved me, but I didn't do it. MESSAGE on Playing with connectivity interfaces!!!! He still had access to his gear, I asked him to change the settings back to where they were (still haven't gotten to this story yet), and I had to call someone at the building to go reboot the gear, since they still had PHONES. Nobody answered cause they weren't in yet, so I'm assuming I may be driving out there. I finally get someone next door at the other company who goes and reboots the device for me. Back to normal and in time we make the correct changes.

Now where did I go wrong? Why didn't I ask these questions before hand? The young engineer changed the port settings on his Edge Juniper router. This wasn't the device I was connected to. I was actually connected to his Cisco IAD device so they could hand me a voice and data line. Why was he in his Juniper router instead of the IAD??? Never got around to asking him that, maybe he thought it was our equipment? The world will never know. In the end don't move to fast it can be costly. I got lucky this time. I had a buddy's bring down 10,000 call center agents, another black hole a entire section of a network, and another Reset BGP, and so on. I haven't seen anyone loose there jobs over these incidents, but we all have heard the stories. Happy Networking.




Update. I wrote this post a while back and never posted it This week I was working with a TAC engineer, we brought down a call center. Hence I'm up at 11pm doing troubleshooting at low call volume time. Once again this could have been solved with some pre planning, but I guess you learn by making mistakes.

I can't stop dialing 911

911. Its a number we often need in times of emergency as I have dialed this year when a man had a seizure in a office I was working at, at the time. But what about those other times when people dial it. What do they usually do? Most hang up right away then the boys in blue show up, sometimes they can be a little unhappy. Well today a customer of mines made that mistake and they got a fine as they have made the mistake several times recently. I guess nobody told them not to hang up when they call. We went over a few options on how they couldn't dial 911 and the best option was for them to change there outside dial to a 8. This was a CME system so it was done on the command line, but we can make some route patterns on a CUCM if there is interest. First thing we do is edit the dial peers. I have a few sample's below.


dial-peer voice 2004 pots
corlist outgoing call-local
description ** local dialing dial peer **
destination-pattern 8[2-9]..[2-9]......
port 0/3/0:23
forward-digits 10
!
dial-peer voice 2005 pots
corlist outgoing call-national
description ** long distance dialing dial peer **
destination-pattern 81[2-9]..[2-9]......
port 0/3/0:23
forward-digits 11
!
dial-peer voice 2006 pots
corlist outgoing call-international
description ** international dialing dial peer **
destination-pattern 8011T
port 0/3/0:23
forward-digits 10
prefix 011


Where the 8's where at were previously 9's. That was easy. We made a few outbound test calls they worked perfectly.

Now onto some telephony-service changes

call-forward pattern 8..........
call-forward pattern 81..........

transfer-pattern 8..........
transfer-pattern 81..........
secondary-dialtone 8


Yeah I realize this wasn't exciting as I thought it would be for a blog post. This is pretty straight forward stuff. In the end we made updated around 7-8 dial peers. We strip off the 8 with the correct forwarding digits and we sent calls to the PTSN. Give it a try one day!!!! Happy Networking