Passed CCIE Written

Once again I have forgotten about the blog and went on with normal life.  Why do all of our favorite bloggers disappear on us?  Well onto the good news I passed my CCIE V written.  I have been in contact with Mark Snow at INE and I will be attending a boot camp in the 4th quarter of this year if the timing is right.  I have a few large projects that will be going into implementation phase at that point, so 2 weeks gone can be kinda taxing to my co workers.  As for the exam is all over the place and I wish Cisco would find a way to slim down the topics,  as some of them won't even be covered(FXO, FXS).  Anyway I look to start writing here about my new experiences as I prep for the lab. 

CCIE Written scheduled

We have a date Dec 7th.  I have been studying over quite a bit of topics over the last few months.  The last 2+ months I will have directed study in my weak areas (QOS,Gatekeeper) and will work on getting stronger in areas's I'm already confident in.  I've reached a pretty good level of voice experience and I'm pretty confident with my work.  Time to push onto that next level.

Video

So I've been gone again.  Great things going on and not so great.  I thought it would be important for me to make a video, but writing and editing take a while, so a quick video would be good enough with everyone's fav lab videos.

New Job

I've been going for a while.  I've seriously been trying to secure a new job.  I had a lot of post I made, but didn't want to really talk about them until the deal was signed and I showed up the 1st day.  I've worked with a few companies who promised offers only to have things change up on me.  Well Monday I started for a global manufacturing firm.  I'm the Senior Unified Communications Engineer.  I really don't have any underlings, but still the main stop in taking care of VOIP problems.  So far I've already been making impact, Fixing a few issues that they had lingering on my 1st day.  Some of them were insanely easy I'm not sure what the last guy was doing.  None the less it feels good to be in a decent size enterprise vs working for a partner.  More than likely in a few years I'll end back up at a partner due to the challenge of the work, but its good to get some rest inbetween.

Cisco does care

Recently I had a customer who went down hard on there Db replication.  It has been off prob for some time, but since they had multiple servers and no changes needed to be made the phone system just kept chugging away as usual.  Then when it had enough it just stopped working.  This customer also ended there support contract with us and called in a Sev 1 request willing to pay top dollar for a engineer to assist them.

In the 1st stages of troubleshooting I noticed that the DB replication was off.  I tried to resynch  through the CLI and no luck.  We rebooted a few boxes no luck.  Next I was thinking we may have some deeper rooted problems we should call TAC.  This feels like one of the times they may have to root into the system to get it fixed, or we can get cisco to possibly send you a new box or 2 if you have good backups. There last backup was last year in april, but thats another story.  So there only option was to get the boxes back up.


I called Tac with the serial numbers and they were no longer on support, but cisco does allow you to pay by credit card.  I spoke with the entitlement guy and he said since they were 100 percent down cisco would give you a 1 time free TAC call but urged support.  We got a tac engineer and just like I expected he had to root in and we got the boxes back up, but thats not the story.  I though cisco was going to hold their system for ransom for those credit card digits.   Just another odd day in the life of a engineer.


We will be talking about backups next!!!!!!!!!!!!

Network Plats PT3

After thinking about this, I didn't come up with a good solution.   In other words people are going to be people and its best as us who provide support should just take the higher road, and let things go.   As we have learned customers are going to be upset which they should, but maybe with some coaching we can get them to deliver us what we need and keep the focus on problem resolution and let management deal with the attitudes.  While they are going to continue to say the silly things, I don't feel they can go away.  The part that I didn't mention earlier on is that I usually focus on the Tier2/3 level of troubleshooting, so I actually don't deal with non technical people.  I'm usually dealing with IT staff which is a reason for me beef as they know how things work.  Good luck out there.

Network platitudes part 2

Today I was chatting with a friend and I was discussing yesterdays post, and I was told that my post makes since from the eyes of a engineer, but what about the customer?  As a customer when I call support I feel there is no urgency to fix my problem, or that you want to keep me tied to paying for service.  When we loose our means to communicate with our customers or do our work, customers loose confidence, and will look to spend there hard earned dollars elsewhere.

I agree with this 100 percent, but how do we get to a situation when we can collaborate to solve the problem.  I had a post a while about about the 5W's some time ago.  Those should be the main thing that a support representatives ask, and the answers that a customer should provide.  That way we can get the ball rolling and get your problem solved so we can move on to the next.  We have several more we have to get to, so our day never ends.  One another note is it possible that all of us that work in support are just being a baby about this and we should just let it rub off?  Or when we speak to support personal we feel they are below us?  These are other thoughts that crossed my mind as I worked through talking about working support, dealing with customer service, and still feeling good at the end of the day for all your hard work.  I'm going to keep digging further and this may take a life of its own into something else...

Network Platitudes

So what am I talking about???

By definition platitude means "a flat, dull, or trite remark, especially one uttered as if it were fresh or profound"

We hear these all the time like "this can't go down again" "when is this going to be back up?" These statements are part of my usual day, but I want to take some time to explore them. On one end I have a angry customer who is experiencing some sort of disruption to there day, and me on the other end trying to quickly solve there problem as I have many more waiting. This is my everyday life 99 percent of the time, but I hear these statements so much that I'm wondering why do they even say them.

Now if I was going into there network and firing off commands then something like the comments above make perfect since, but when they just called and thats the 1st thing I hear out there mouth and the last time I touched there network was months ago I get a little puzzled by it. This is something I have no answer for and I actually plan on getting some answers soon. One of my customers is a good friend of mine and he's even said these things before. Next time we are drinking beer I plan on us getting a good laugh out of this and bringing back some answers for everyone.

Passed CCNP V and I don't know ....

.... Fill in the appropriate word you think goes there. I have been studying for this cert for quite some time and I'm on the tail end and I don't know as much as I felt I would know at the other side of the tunnel. Before I embark on IE I plan on getting stronger in a lot of areas of VOIP and networking in general. The certs are cool to have and I can pass pretty much any technical interview thrown infront of me, but I would really like to be a expert. 10,000 is the magic number of hours needed to be considered one, so I'm a long ways off.

CCNP V certified

Passed my last CCNP Voice exam today. I started a while ago in the old 4x track with gateways/gatekeeper exam. This was a long journey. Glad its finally over now on to bigger and better things.

Old Series

I have a series of blog post that were never posted, I'm thinking of going through a lot of them and posting them if I feel they are good material

Revisiting customer service. (Old post Series pt 1).

This past week has been a real difficult one. The usual outages, added on with a car breaking down, and life. The particular incident in question revolved around a medical center with a sip trunk. Pretty simple setup

2 sites with UC500 series routers.
2 Sonicwall Firewalls
1 SIP trunk to a provider.


We get a call saying they can't dial out or inbound calls aren't working. I logged in and noticed right away the numbers aren't registered to the sip server. I placed a ping to the sip server from my desk top(IP's have been changed to protect the innocent)



C:\Users\urbanciscoengineer>ping 107.141.31.21 -t

Pinging 107.141.31.21 with 32 bytes of data:
Reply from 107.141.31.21: bytes=32 time=28ms TTL=244
Reply from 107.141.31.21: bytes=32 time=28ms TTL=244
Reply from 107.141.31.21: bytes=32 time=28ms TTL=244
Reply from 107.141.31.21: bytes=32 time=28ms TTL=244
Reply from 107.141.31.21: bytes=32 time=28ms TTL=244
Reply from 107.141.31.21: bytes=32 time=28ms TTL=244
Reply from 107.141.31.21: bytes=32 time=28ms TTL=244

Looks like Its up and reachable lets try from the UC500 device.



Traceroute from UC540
traceroute
Type escape sequence to abort.
Tracing the route to 107.141.31.21.ptr.us.xo.net (107.141.31.21)

1 ip65-44-248-57.z248-44-65.customer.algx.net (65.44.248.57) 0 msec 4 msec 0 msec
2 ip65-46-107-181.z107-46-65.customer.algx.net (65.46.107.181) 4 msec 8 msec 8 msec
3 * * *
4 * * *
5 ae0d0.mcr1.nyc-ny.us.provider.net (216.153.0.18) 8 msec 12 msec 12 msec
6 216.156.7.234.ptr.us.provider.net (216.256.7.134) 8 msec
216.156.7.182.ptr.us.provider.net (216.154.7.182) 8 msec
216.156.7.234.ptr.us.provider.net (216.136.7.244) 8 msec
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *


As you can see we get lost in the network. I also traced it out from my desktop. I sent all the captured info to the user, and inform him he needs to speak to the provider. I even said he could call me back when the provider was on the line which he took me up on. So the provider calls me back and I tell him what we found. He's still saying this is our problem. This isn't making any since to me. I asked for the ticket to be escalated to someone from their IP team as this is not a SIP issue at this point. I asked the customer to stay on this as we can't be responsible for babysitting there providers. Now the customer service comes into play. After realizing that this guy isn't going to do what we need from his demeanor, I decided to stay in contact with the provider through the evening. They still couldn't get anything solved. The next morning I inform my manager that this issue is going to heat up as they have been down over 24 hours. We get the provider on the phone, sales guy and me and the customer. We say the same thing over again "WE DON"T HAVE IP REACH-ABILITY TO YOUR EQUIPMENT!!!!!!". Thats plain as day there is something wrong in your cloud. They put me back in contact with the same guy again. We ask for a escalation which they finally got us and now its 12pm. The Tier 3 guy didn't get back in contact with us until 3pm and this was over chat with the first guy we spoken with. I provide my traceroutes again for the 3rd time and inform them we aren't changing anything until we have a clear path through there network to the SIP server.


As the day went on the finally found the access-list that was updated improperly and not allowing our IP to touch the SIP registar. Things like this make you want to cringe and wonder how are people getting these jobs at these places and why does the escalation take so long. This blog was about customers service, and we stayed with the customer till the end, but man what a waste of time that I dont get to spend with my other customer.

QOS part 2.

To get QOS to work the way we want it to we have to do a few things to identify the packets going across the network, then we can push them into the slow lanes and the fast lanes or even kick them off the lie. So how do we do this? We have a few tool things that can help us along the way.



Classifying and marking
Scheduling
Traffic Shaping, and link specific mechanisms


Classifying and marking

This allows us to find out who is who and how we will treat them. We can do this at the layer 2 or layer 3, 4 or even 7(cisco specific tools for this one)

At at layer 2 we can mark the Cos)class of service bit with a marking to let us know what type of traffic it is

Layer 3 we can use IP Precedence(old way) or Dirrerentiated Services Code Points(newer way)

Layer 4 we can use the TCP/UPD source and destination ports

Layer 7 we can use NBAR(Network based application Recognition)


When we are at the layer 2(CoS) level we can set the user priority bit from on the 802.1Q header. We have a max of 3 bits so that leaves us with 0-7. We usually use 5 for voice in case you are wondering.


At layer 3 the (ToS) Level we have 2 features the IPP bit I spoke about before which are the same as the CoS, but we add the next 3 bits and we come up with DSCP which is the way we prefer.

As you can see in the drawing above DSCP takes up all of the 1st 6 bits, and the IPP is the 1st 3 bits. From there we can break down PHB(per hop behaviors) for DSCP, but we'll save that for another blog.


Layer 4 is a little different so I will come back to that later, and finally we have NBAR at layer 7


NBAR is Cisco proprietary that matches traffic against Protocol Description Language Module which tells us what what type of traffic is coming across. You need to have CEF running and of course this needs to be on cisco. As of this writing Juniper didn't offer this feature, but people got around it using access-list in some method to get something simular.


Thats it for now, I will brea these down further and further as we go on. QOS is a pretty hard topic and there is no way I can cover this in few blog post. It would prob take close to 30, but this should get everyones appetite we to go out and find other resources to use to fill in the gaps.


Thanks

QOS and the need for it in the Enterprise PT1.

More than likely this will be a series of post about QOS. Around a year ago I worked on a Enterprise wide QOS WAN design for a DOD agency that went pretty well. Some guys senior to me worked them out and it was a good learning experience for all of us. One of them recently got his CCIE and the other is working on another attempt. Since then I have left the WAN world and become focused on UC. QOS has reared its ugly head again. While working with a credit union we had some quality problem going across the WAN which was a big deal as there call center was on the other side. Turns out no QOS on the wan which was MPLS VPN service from a provider , and instead they trusted there fate with a packet shaping device which turned out to be painful for all us involved. Ok enough back story lets break down this QOS thing with the Who, What, When, Where and Why?


WHO? QOS


WHAT? From Cisco Enterprise QOS SRND nov 2005(oldie, but goodie)


QOS is the mesure of transmission quality and service ability of a network or internetworks.

When? When do we need QOS. This is the big debate that happens all the time. "If I have Phat links I dont need QOS" is the argument I hear all the time. While this is true most of the time those times when your link gets congested, the time that worm attacks your network, or when your users get around your security and light off peer to peer apps, or legit traffic gets in front of your voice traffic you will run into trouble.


Where? You can put QOS on your LAN/WAN and even wireless networks.


Why? QOS will allow your to classify, Mark, Policing, and Schedule your traffic to allow you to get the best use out of your link. Now with that said QOS will not make your T1 be 2 T1's, but you will decide who gets to use the wire 1st, which according to newtons 3rd law there is a consequence of you will also decide who goes to the bit bucket.

Ok back for real this time.

Life always seems to catch up to you at one point or another. I wrote back in November that I was back, then Xmas and family came, then the new year, Knocked out a few cisco test and didn't' even update my blog. A few weeks ago I passed the CIPT2 Test(very hard) and the TVOICE test(easy). Only 1 more test for my CCNP Voice which I will be taking friday. I have some other developments coming forward, but I'll keep that for another update.

And I'm back and Updates coming soon.

Sorry I've been away for a while. Had a wedding, moved to a new location, and had to do some specialization certs. I had to knock out the Cisco Advanced routing and switching for the job. I found the topics rather boring so I decided not to post. So in the next incoming weeks:

I have a nice CCNP VOICE, CCIE VOICE lab, not 100 percent of the gear as I feel the lab is going to change soon, but I have enough for the entire NRF kit, 2 sites, and a PSTN/FRAME/GK section. Stay turned as I plan on kicking this thing off with some things outta CIPT 2 in the next few weeks. It will slow down again during the holidays, but pick back up again after they are over.


Thanks for following.

VTP

VTP(VLAN trunking protocol)

Used to dynamically add, remove and update VLANs(2-1000) Not really used much anymore. Its a Cisco proprietary protocol. I had a jr engineer that I worked with that was pretty close to implementing VTP at one time. I guess if his configs were valid we should have let him go through with it.


One other thing to add, is that this is not a trunking protocol such as ISL or 802.1Q. What we are doing with VTP is creating vlan's in once place and letting them replicate to other devices.


So now onto VTP

VTP packets are sent in the frames of ISL, and 802.1Q message, and it contains the following information.

- VTP version 1,2 or 3
Summary Advertisement
Subset Advertisement
Advertisement Request
VTP Join Message

Management Domain Length
Management Domain Name

VTP also has a configuration revision number which let shows the which revision
VTP is currently on, this is important as we will soon discover. When a switch receives a revision that is higher than its own it will act on it.





3 modes
-Server
-Client
-Default


Server
The default mode. This is the one that can change VLANS, Sends and receives VTP updates.(2-1000) are configurable

Client
Takes orders from the server.
Sends and receive updates

Transparent(recommended mode)
Can change Vlan info
Just passes through Vlan configs to other switches
Does not act on VLAN advertisements
always at revision 0
The switches need to be in transparent mode to configure extended Vlans (1006-4096)


Authentication
Domain names and passwords can be setup to allow for protection throughout your network. Make sure you have a separate name for Lab and Production switches, as this will cause some pain later on.


VTP pruning.
Allows you to get rid of vlans that arent' needed. You can only do this with vlan 2-1000, and done from a VTP server

One think I forgot to mention is that devices need to be setup as trunks for VTP work to work, I just wanted to make sure I mentioned that before you get carried away.

How many Static routes are too many

Recently in my new position I have started to work in the small medium business arena. One of the biggest challenges in coming from the enterprise environment is the lack of routing protocols. Some of it is due to knowledge of the internal IT staff, some of it is due to the VAR's that provide service, and other times its politics. This shouldn't seem like a big deal, but I get a few calls every week of something that isn't reachable in some way shape or form. 99 percent of the time when the routing was updated someone forgot to save the changes then "power" happens and routes are lost. This customer in particular has 3 sites over MPLS VPN, and a few more sites that come in site to site VPN's over ASA's. When I pull the configs for the devices I see pages of static routes to get to locations. I have asked them several times why don't they run OSPF or EIGRP, and they always say "we like things with static routes". Well if you like them you shouldn't call me ever few weeks cause you can't get somewhere. So how do we fix these problems. We usually just go right up the OSI model. Start with the device that can't talk to the far end, trace route to where it gets suck and check the control and data plane's on that device. All and all this doesn't seem to be a big deal, but at times it can take hours when firewalls, and DMZ's and undocumented configs are into play. This brings me back to my point OSPF, EIGRP will detect link failure quickly and the packets will either find a way or if syslog is setup correctly we will get alerts of link failure. This is just another rant of the day in the life of a network engineer.

VLANS

VLAN's

What are VLAN's

Vlan's are virtual Local Area Networks. The idea here is to break up BCST domains into smaller chunks. Host in the same vlan can communicate with each other, but will have to be routed to get outside the vlan due to the ARP protocol being a BCST which is stopped at the layer 3 boundary.


Vlans should be defined by physical location. Like a switch closet should contain different vlans from other switching closets.

Vlans are a number that the switch uses with a port. 12 Bits 0-4095

Normal VLAN's 1-1005

1 Default Vlan Ports default here. Should move them to another non routed vlan for security reasons.

1002/4 Default FDDI vlans
1003/5 Default Token Rings

Extended Vlans 1006-4094


Once a vlan is created the switch or device will make a Cam table for that individual vlan. Also a spanning tree instance is created. Ports can be one of 3 things when working at the layer 2 level.

- Access
- Trunk
- Dynamic

Access - Where PC, Phones and printers plug in.

Trunks - Where Multiple Vlans can come through these ports usually used when leaving the device and going to the next devices

Dynamic - Automatically chooses port type


More on trunks:

Trunk Links are used when we want to have our vlans go between switches or routers. To do this we have to TAG the packets when they leave the device. We will add ISL(cisco prop) or 802.1Q(IEEE).


802.1Q has a 4 byte tag between the source and destination MAC and the Lenth/ethertype fields(check the pic below). The frame is modified for .1Q except for the native vlan. we don't usually see to much with ISL. Both sides of the link have to be the same, but they are local to the device.

FCS is also modified since we had changed the frame.


Native Vlan

Normal traffic will be sent untagged across the native Vlan. This is usually Vlan 1 in most devices.

QinQ support

we can have vlans stacked on top of each other on each frame. Sometimes known as Metro Tagging when used with Metro E environments.



Stay tuned for more later on...

Campus Network Design

Most campus networks have a 3 Tier setup with 3 distinct layers


Access Layer

Where user connect and we have ports that belong to certain VLAN's. Not much routing done here. So here we can have PC's, Printers, VOIP phones. This is where we see our 2960 switches located. We have multiple uplinks to distro switches. QOS is applied here, we can have Port Security here.

Distribution layer

This is where the layer 3 switches are located. The Access switches will meet up here and there are uplinks to the Core Switches. We also Run HSRP/VRRP/GLBP here. Etherchannel is also ran here. This is also where we would summarize addresses to make the routing table smaller.

Core Layer

The most powerful devices and we usually like to have little policy here we like to forward packets as fast as possible. Usually no access list or QOS.


Stay Tuned for more.


We use those building blocks cause its easy to replicate and expand on if we need changes. Blocks can be added or removed with Ease. T/S is also easier as we can isolate faults in individual blocks.