CCIE Written scheduled

We have a date Dec 7th.  I have been studying over quite a bit of topics over the last few months.  The last 2+ months I will have directed study in my weak areas (QOS,Gatekeeper) and will work on getting stronger in areas's I'm already confident in.  I've reached a pretty good level of voice experience and I'm pretty confident with my work.  Time to push onto that next level.

Video

So I've been gone again.  Great things going on and not so great.  I thought it would be important for me to make a video, but writing and editing take a while, so a quick video would be good enough with everyone's fav lab videos.

New Job

I've been going for a while.  I've seriously been trying to secure a new job.  I had a lot of post I made, but didn't want to really talk about them until the deal was signed and I showed up the 1st day.  I've worked with a few companies who promised offers only to have things change up on me.  Well Monday I started for a global manufacturing firm.  I'm the Senior Unified Communications Engineer.  I really don't have any underlings, but still the main stop in taking care of VOIP problems.  So far I've already been making impact, Fixing a few issues that they had lingering on my 1st day.  Some of them were insanely easy I'm not sure what the last guy was doing.  None the less it feels good to be in a decent size enterprise vs working for a partner.  More than likely in a few years I'll end back up at a partner due to the challenge of the work, but its good to get some rest inbetween.

Cisco does care

Recently I had a customer who went down hard on there Db replication.  It has been off prob for some time, but since they had multiple servers and no changes needed to be made the phone system just kept chugging away as usual.  Then when it had enough it just stopped working.  This customer also ended there support contract with us and called in a Sev 1 request willing to pay top dollar for a engineer to assist them.

In the 1st stages of troubleshooting I noticed that the DB replication was off.  I tried to resynch  through the CLI and no luck.  We rebooted a few boxes no luck.  Next I was thinking we may have some deeper rooted problems we should call TAC.  This feels like one of the times they may have to root into the system to get it fixed, or we can get cisco to possibly send you a new box or 2 if you have good backups. There last backup was last year in april, but thats another story.  So there only option was to get the boxes back up.


I called Tac with the serial numbers and they were no longer on support, but cisco does allow you to pay by credit card.  I spoke with the entitlement guy and he said since they were 100 percent down cisco would give you a 1 time free TAC call but urged support.  We got a tac engineer and just like I expected he had to root in and we got the boxes back up, but thats not the story.  I though cisco was going to hold their system for ransom for those credit card digits.   Just another odd day in the life of a engineer.


We will be talking about backups next!!!!!!!!!!!!

Network Plats PT3

After thinking about this, I didn't come up with a good solution.   In other words people are going to be people and its best as us who provide support should just take the higher road, and let things go.   As we have learned customers are going to be upset which they should, but maybe with some coaching we can get them to deliver us what we need and keep the focus on problem resolution and let management deal with the attitudes.  While they are going to continue to say the silly things, I don't feel they can go away.  The part that I didn't mention earlier on is that I usually focus on the Tier2/3 level of troubleshooting, so I actually don't deal with non technical people.  I'm usually dealing with IT staff which is a reason for me beef as they know how things work.  Good luck out there.

Network platitudes part 2

Today I was chatting with a friend and I was discussing yesterdays post, and I was told that my post makes since from the eyes of a engineer, but what about the customer?  As a customer when I call support I feel there is no urgency to fix my problem, or that you want to keep me tied to paying for service.  When we loose our means to communicate with our customers or do our work, customers loose confidence, and will look to spend there hard earned dollars elsewhere.

I agree with this 100 percent, but how do we get to a situation when we can collaborate to solve the problem.  I had a post a while about about the 5W's some time ago.  Those should be the main thing that a support representatives ask, and the answers that a customer should provide.  That way we can get the ball rolling and get your problem solved so we can move on to the next.  We have several more we have to get to, so our day never ends.  One another note is it possible that all of us that work in support are just being a baby about this and we should just let it rub off?  Or when we speak to support personal we feel they are below us?  These are other thoughts that crossed my mind as I worked through talking about working support, dealing with customer service, and still feeling good at the end of the day for all your hard work.  I'm going to keep digging further and this may take a life of its own into something else...

Network Platitudes

So what am I talking about???

By definition platitude means "a flat, dull, or trite remark, especially one uttered as if it were fresh or profound"

We hear these all the time like "this can't go down again" "when is this going to be back up?" These statements are part of my usual day, but I want to take some time to explore them. On one end I have a angry customer who is experiencing some sort of disruption to there day, and me on the other end trying to quickly solve there problem as I have many more waiting. This is my everyday life 99 percent of the time, but I hear these statements so much that I'm wondering why do they even say them.

Now if I was going into there network and firing off commands then something like the comments above make perfect since, but when they just called and thats the 1st thing I hear out there mouth and the last time I touched there network was months ago I get a little puzzled by it. This is something I have no answer for and I actually plan on getting some answers soon. One of my customers is a good friend of mine and he's even said these things before. Next time we are drinking beer I plan on us getting a good laugh out of this and bringing back some answers for everyone.

Passed CCNP V and I don't know ....

.... Fill in the appropriate word you think goes there. I have been studying for this cert for quite some time and I'm on the tail end and I don't know as much as I felt I would know at the other side of the tunnel. Before I embark on IE I plan on getting stronger in a lot of areas of VOIP and networking in general. The certs are cool to have and I can pass pretty much any technical interview thrown infront of me, but I would really like to be a expert. 10,000 is the magic number of hours needed to be considered one, so I'm a long ways off.

CCNP V certified

Passed my last CCNP Voice exam today. I started a while ago in the old 4x track with gateways/gatekeeper exam. This was a long journey. Glad its finally over now on to bigger and better things.

Old Series

I have a series of blog post that were never posted, I'm thinking of going through a lot of them and posting them if I feel they are good material

Revisiting customer service. (Old post Series pt 1).

This past week has been a real difficult one. The usual outages, added on with a car breaking down, and life. The particular incident in question revolved around a medical center with a sip trunk. Pretty simple setup

2 sites with UC500 series routers.
2 Sonicwall Firewalls
1 SIP trunk to a provider.


We get a call saying they can't dial out or inbound calls aren't working. I logged in and noticed right away the numbers aren't registered to the sip server. I placed a ping to the sip server from my desk top(IP's have been changed to protect the innocent)



C:\Users\urbanciscoengineer>ping 107.141.31.21 -t

Pinging 107.141.31.21 with 32 bytes of data:
Reply from 107.141.31.21: bytes=32 time=28ms TTL=244
Reply from 107.141.31.21: bytes=32 time=28ms TTL=244
Reply from 107.141.31.21: bytes=32 time=28ms TTL=244
Reply from 107.141.31.21: bytes=32 time=28ms TTL=244
Reply from 107.141.31.21: bytes=32 time=28ms TTL=244
Reply from 107.141.31.21: bytes=32 time=28ms TTL=244
Reply from 107.141.31.21: bytes=32 time=28ms TTL=244

Looks like Its up and reachable lets try from the UC500 device.



Traceroute from UC540
traceroute
Type escape sequence to abort.
Tracing the route to 107.141.31.21.ptr.us.xo.net (107.141.31.21)

1 ip65-44-248-57.z248-44-65.customer.algx.net (65.44.248.57) 0 msec 4 msec 0 msec
2 ip65-46-107-181.z107-46-65.customer.algx.net (65.46.107.181) 4 msec 8 msec 8 msec
3 * * *
4 * * *
5 ae0d0.mcr1.nyc-ny.us.provider.net (216.153.0.18) 8 msec 12 msec 12 msec
6 216.156.7.234.ptr.us.provider.net (216.256.7.134) 8 msec
216.156.7.182.ptr.us.provider.net (216.154.7.182) 8 msec
216.156.7.234.ptr.us.provider.net (216.136.7.244) 8 msec
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *


As you can see we get lost in the network. I also traced it out from my desktop. I sent all the captured info to the user, and inform him he needs to speak to the provider. I even said he could call me back when the provider was on the line which he took me up on. So the provider calls me back and I tell him what we found. He's still saying this is our problem. This isn't making any since to me. I asked for the ticket to be escalated to someone from their IP team as this is not a SIP issue at this point. I asked the customer to stay on this as we can't be responsible for babysitting there providers. Now the customer service comes into play. After realizing that this guy isn't going to do what we need from his demeanor, I decided to stay in contact with the provider through the evening. They still couldn't get anything solved. The next morning I inform my manager that this issue is going to heat up as they have been down over 24 hours. We get the provider on the phone, sales guy and me and the customer. We say the same thing over again "WE DON"T HAVE IP REACH-ABILITY TO YOUR EQUIPMENT!!!!!!". Thats plain as day there is something wrong in your cloud. They put me back in contact with the same guy again. We ask for a escalation which they finally got us and now its 12pm. The Tier 3 guy didn't get back in contact with us until 3pm and this was over chat with the first guy we spoken with. I provide my traceroutes again for the 3rd time and inform them we aren't changing anything until we have a clear path through there network to the SIP server.


As the day went on the finally found the access-list that was updated improperly and not allowing our IP to touch the SIP registar. Things like this make you want to cringe and wonder how are people getting these jobs at these places and why does the escalation take so long. This blog was about customers service, and we stayed with the customer till the end, but man what a waste of time that I dont get to spend with my other customer.

QOS part 2.

To get QOS to work the way we want it to we have to do a few things to identify the packets going across the network, then we can push them into the slow lanes and the fast lanes or even kick them off the lie. So how do we do this? We have a few tool things that can help us along the way.



Classifying and marking
Scheduling
Traffic Shaping, and link specific mechanisms


Classifying and marking

This allows us to find out who is who and how we will treat them. We can do this at the layer 2 or layer 3, 4 or even 7(cisco specific tools for this one)

At at layer 2 we can mark the Cos)class of service bit with a marking to let us know what type of traffic it is

Layer 3 we can use IP Precedence(old way) or Dirrerentiated Services Code Points(newer way)

Layer 4 we can use the TCP/UPD source and destination ports

Layer 7 we can use NBAR(Network based application Recognition)


When we are at the layer 2(CoS) level we can set the user priority bit from on the 802.1Q header. We have a max of 3 bits so that leaves us with 0-7. We usually use 5 for voice in case you are wondering.


At layer 3 the (ToS) Level we have 2 features the IPP bit I spoke about before which are the same as the CoS, but we add the next 3 bits and we come up with DSCP which is the way we prefer.

As you can see in the drawing above DSCP takes up all of the 1st 6 bits, and the IPP is the 1st 3 bits. From there we can break down PHB(per hop behaviors) for DSCP, but we'll save that for another blog.


Layer 4 is a little different so I will come back to that later, and finally we have NBAR at layer 7


NBAR is Cisco proprietary that matches traffic against Protocol Description Language Module which tells us what what type of traffic is coming across. You need to have CEF running and of course this needs to be on cisco. As of this writing Juniper didn't offer this feature, but people got around it using access-list in some method to get something simular.


Thats it for now, I will brea these down further and further as we go on. QOS is a pretty hard topic and there is no way I can cover this in few blog post. It would prob take close to 30, but this should get everyones appetite we to go out and find other resources to use to fill in the gaps.


Thanks

QOS and the need for it in the Enterprise PT1.

More than likely this will be a series of post about QOS. Around a year ago I worked on a Enterprise wide QOS WAN design for a DOD agency that went pretty well. Some guys senior to me worked them out and it was a good learning experience for all of us. One of them recently got his CCIE and the other is working on another attempt. Since then I have left the WAN world and become focused on UC. QOS has reared its ugly head again. While working with a credit union we had some quality problem going across the WAN which was a big deal as there call center was on the other side. Turns out no QOS on the wan which was MPLS VPN service from a provider , and instead they trusted there fate with a packet shaping device which turned out to be painful for all us involved. Ok enough back story lets break down this QOS thing with the Who, What, When, Where and Why?


WHO? QOS


WHAT? From Cisco Enterprise QOS SRND nov 2005(oldie, but goodie)


QOS is the mesure of transmission quality and service ability of a network or internetworks.

When? When do we need QOS. This is the big debate that happens all the time. "If I have Phat links I dont need QOS" is the argument I hear all the time. While this is true most of the time those times when your link gets congested, the time that worm attacks your network, or when your users get around your security and light off peer to peer apps, or legit traffic gets in front of your voice traffic you will run into trouble.


Where? You can put QOS on your LAN/WAN and even wireless networks.


Why? QOS will allow your to classify, Mark, Policing, and Schedule your traffic to allow you to get the best use out of your link. Now with that said QOS will not make your T1 be 2 T1's, but you will decide who gets to use the wire 1st, which according to newtons 3rd law there is a consequence of you will also decide who goes to the bit bucket.

Ok back for real this time.

Life always seems to catch up to you at one point or another. I wrote back in November that I was back, then Xmas and family came, then the new year, Knocked out a few cisco test and didn't' even update my blog. A few weeks ago I passed the CIPT2 Test(very hard) and the TVOICE test(easy). Only 1 more test for my CCNP Voice which I will be taking friday. I have some other developments coming forward, but I'll keep that for another update.