And I'm back and Updates coming soon.

Sorry I've been away for a while. Had a wedding, moved to a new location, and had to do some specialization certs. I had to knock out the Cisco Advanced routing and switching for the job. I found the topics rather boring so I decided not to post. So in the next incoming weeks:

I have a nice CCNP VOICE, CCIE VOICE lab, not 100 percent of the gear as I feel the lab is going to change soon, but I have enough for the entire NRF kit, 2 sites, and a PSTN/FRAME/GK section. Stay turned as I plan on kicking this thing off with some things outta CIPT 2 in the next few weeks. It will slow down again during the holidays, but pick back up again after they are over.


Thanks for following.

VTP

VTP(VLAN trunking protocol)

Used to dynamically add, remove and update VLANs(2-1000) Not really used much anymore. Its a Cisco proprietary protocol. I had a jr engineer that I worked with that was pretty close to implementing VTP at one time. I guess if his configs were valid we should have let him go through with it.


One other thing to add, is that this is not a trunking protocol such as ISL or 802.1Q. What we are doing with VTP is creating vlan's in once place and letting them replicate to other devices.


So now onto VTP

VTP packets are sent in the frames of ISL, and 802.1Q message, and it contains the following information.

- VTP version 1,2 or 3
Summary Advertisement
Subset Advertisement
Advertisement Request
VTP Join Message

Management Domain Length
Management Domain Name

VTP also has a configuration revision number which let shows the which revision
VTP is currently on, this is important as we will soon discover. When a switch receives a revision that is higher than its own it will act on it.





3 modes
-Server
-Client
-Default


Server
The default mode. This is the one that can change VLANS, Sends and receives VTP updates.(2-1000) are configurable

Client
Takes orders from the server.
Sends and receive updates

Transparent(recommended mode)
Can change Vlan info
Just passes through Vlan configs to other switches
Does not act on VLAN advertisements
always at revision 0
The switches need to be in transparent mode to configure extended Vlans (1006-4096)


Authentication
Domain names and passwords can be setup to allow for protection throughout your network. Make sure you have a separate name for Lab and Production switches, as this will cause some pain later on.


VTP pruning.
Allows you to get rid of vlans that arent' needed. You can only do this with vlan 2-1000, and done from a VTP server

One think I forgot to mention is that devices need to be setup as trunks for VTP work to work, I just wanted to make sure I mentioned that before you get carried away.

How many Static routes are too many

Recently in my new position I have started to work in the small medium business arena. One of the biggest challenges in coming from the enterprise environment is the lack of routing protocols. Some of it is due to knowledge of the internal IT staff, some of it is due to the VAR's that provide service, and other times its politics. This shouldn't seem like a big deal, but I get a few calls every week of something that isn't reachable in some way shape or form. 99 percent of the time when the routing was updated someone forgot to save the changes then "power" happens and routes are lost. This customer in particular has 3 sites over MPLS VPN, and a few more sites that come in site to site VPN's over ASA's. When I pull the configs for the devices I see pages of static routes to get to locations. I have asked them several times why don't they run OSPF or EIGRP, and they always say "we like things with static routes". Well if you like them you shouldn't call me ever few weeks cause you can't get somewhere. So how do we fix these problems. We usually just go right up the OSI model. Start with the device that can't talk to the far end, trace route to where it gets suck and check the control and data plane's on that device. All and all this doesn't seem to be a big deal, but at times it can take hours when firewalls, and DMZ's and undocumented configs are into play. This brings me back to my point OSPF, EIGRP will detect link failure quickly and the packets will either find a way or if syslog is setup correctly we will get alerts of link failure. This is just another rant of the day in the life of a network engineer.

VLANS

VLAN's

What are VLAN's

Vlan's are virtual Local Area Networks. The idea here is to break up BCST domains into smaller chunks. Host in the same vlan can communicate with each other, but will have to be routed to get outside the vlan due to the ARP protocol being a BCST which is stopped at the layer 3 boundary.


Vlans should be defined by physical location. Like a switch closet should contain different vlans from other switching closets.

Vlans are a number that the switch uses with a port. 12 Bits 0-4095

Normal VLAN's 1-1005

1 Default Vlan Ports default here. Should move them to another non routed vlan for security reasons.

1002/4 Default FDDI vlans
1003/5 Default Token Rings

Extended Vlans 1006-4094


Once a vlan is created the switch or device will make a Cam table for that individual vlan. Also a spanning tree instance is created. Ports can be one of 3 things when working at the layer 2 level.

- Access
- Trunk
- Dynamic

Access - Where PC, Phones and printers plug in.

Trunks - Where Multiple Vlans can come through these ports usually used when leaving the device and going to the next devices

Dynamic - Automatically chooses port type


More on trunks:

Trunk Links are used when we want to have our vlans go between switches or routers. To do this we have to TAG the packets when they leave the device. We will add ISL(cisco prop) or 802.1Q(IEEE).


802.1Q has a 4 byte tag between the source and destination MAC and the Lenth/ethertype fields(check the pic below). The frame is modified for .1Q except for the native vlan. we don't usually see to much with ISL. Both sides of the link have to be the same, but they are local to the device.

FCS is also modified since we had changed the frame.


Native Vlan

Normal traffic will be sent untagged across the native Vlan. This is usually Vlan 1 in most devices.

QinQ support

we can have vlans stacked on top of each other on each frame. Sometimes known as Metro Tagging when used with Metro E environments.



Stay tuned for more later on...

Campus Network Design

Most campus networks have a 3 Tier setup with 3 distinct layers


Access Layer

Where user connect and we have ports that belong to certain VLAN's. Not much routing done here. So here we can have PC's, Printers, VOIP phones. This is where we see our 2960 switches located. We have multiple uplinks to distro switches. QOS is applied here, we can have Port Security here.

Distribution layer

This is where the layer 3 switches are located. The Access switches will meet up here and there are uplinks to the Core Switches. We also Run HSRP/VRRP/GLBP here. Etherchannel is also ran here. This is also where we would summarize addresses to make the routing table smaller.

Core Layer

The most powerful devices and we usually like to have little policy here we like to forward packets as fast as possible. Usually no access list or QOS.


Stay Tuned for more.


We use those building blocks cause its easy to replicate and expand on if we need changes. Blocks can be added or removed with Ease. T/S is also easier as we can isolate faults in individual blocks.

Change in Direction

I have to get a few certs for the job, so I'm going to be not doing as much VOIP unless I get some crazy issue that comes in that just needs addressing. I will be getting some specialist certs for the job that focus on route/switch, so I will be posting about those topics. I do have my CCNP, but I'm starting fresh from ground zero and working my way back up to fill in holes and see what I learned new when going over basic material. From there we will go into Layer 2 topics such as STP, etherchannels, and Vlans, and then move onto layer 3 topics such as OSPF, EIGRP, BGP,MPLS, and then add on some trimmings such as IPV6, Multicast and WAAS. Stay Tuned.

Dial-Peers

Its that time. dial-Peers are one of the most important parts of your VOIP network.

2 types of Dial Peers

POTS- THey usually point to the PSTN or a FXO/FXS port.

Voice- These can point to a call manager, Gatekeeper, another voice gateway.



In most networks your going to have at least a few call legs and you are going to have dial peers on each of those legs. As an example.

Now the key part is that there has to be something that matches when it comes to inbound and outbound dial-peers and it can get a little tricky with the logic on how inbound and outbound dial-peers work.

Inbound dial-peer:

When you get a call from the PSTN from a ISDN line or a FXO port there is going to be a dial-peer that it matches. Its usually going to look something like


dial-peer voice 1 pots
incoming called-number .
direct-inward-dial
port 0/0/0:23

We have the dial-peer the number(tag) can be anything and we have it as a pots dial peer. The next part we have incoming call number with a "." to match anything. We have direct-inward-dial so we don't get another dial tone(i'll get to that in part 2) and we have the port that it comes in on. So when the call comes we are matching anything so it doesn't have to go down the list of options. If we didn't it would match:


1. Called Number/DNIS Incoming called-number(command)which we are using

2. Calling number/ANI Answer-Address(command) we can use this kinda like a route map where we can have "policy based calling"

3. Calling number with Destination-Pattern command

4. Voice port (bad choice)

5. Default dial-peer 0 (even worse choice)


So once thats taken care of we have matched the 1st call leg. Now we need a outbound dial-peer to send the call on its way.


So an outbound dial-peer will look something like:

dial-peer voice 2 voip
destination-pattern [347]..
session target ipv4:192.168.100.10

So the outbound portion is not as complex as the inbound. As you can see we have a voip dial-peer a destination-pattern where we are looking for a 3,4, or 7 and any other 3 digits and we the session target ipv4 command pointing to another IP address which can be another router, call manager and so on. If we are sending the call to call manager, call manager will start its process of routing the call, to the correct endpoint and if we are going to another router we are starting this process all over again until we reach our destination. This is a pretty high level overview. I didn't touch of overlapping digits, DID, and a few other commands you can add onto dial-peers to get more control out of them. Look for that in an upcoming topic.

Gatekeepers PT 1

Its been a while since I updated. We completed some work on MGCP, H323 and SIP, but I felt they were incomplete for use on the blog. I will get those up as soon as I get the rest of the info. I may also do it in a part 2, but for now lets talk about gatekeepers.


What are they?

In MMA they are refered to guys who can beat any other fighter in there weight class on any given night, but for some reason or another they can't muster up enough win's to get the title shot. O


Any other definition.

Well in the world of Voice/Video they can be used for a few task.

Address resolution
Admission Control
Bandwidth management
Zone Management
Call Authorization


Address Resolution

the number you dial has to be turned into a IP address so we can reach the distant endpoint, so it does "Resolution" kinda similar to DNS.

Admission Control.

You can't just make a call when a gatekeeper is in town. Gateways have to go out to the gatekeeper and request for a call to be placed

Bandwidth Management.

When we are using multiple clusters the location function doesn't work as we have different call managers around so the gatekeeper can provide CAC, by keeping track of the calls being made. This gets pretty funky with how G711 under uses the bandwidth and G729 over uses the bandwidth and we will cover this later on.

Zone Management.

Gatekeepers keep track of all the gateways in there zone through a registration process. Gateways just can't come online with talking to the gatekeeper 1st.

Call Authorization

Gatekeepers can control access to certain gateways. You can use a AAA server with Radius to get this to work.



Gatekeeper signalling

H323 uses a H225 setup messages before it send the H245 messages for capabilities. When using gatekeepers for signalling we still use H225, but we use a form called Ras. So H225 Ras is used for the signalling between gateways and gatekeepers. and there are several sub messages when using them. Stay tuned as we break down the gatekeeper signalling and call flow in the next post.

Crappy side of the new IOS Voice SIP secuirty

As we all know, SIP has become a target of hackers as they are able to break into networks and steal resources. I've had the pleasure of working a few Toll Fraud cases in my years. Today I got a call that someone's inbound dialing had stopped working. Turned on some debugs and it showed my cell phone number was getting rejected. Thats weird. I made a call to the SIP provider and asked them to watch some inbound calls for me. They saw the same thing. I asked them if they had changed there IP recently and they said on Tuesday!!!!!!!!!!!!!!!. WTF. They sent out notifications, but my customer doesn't read the mail. This was a pretty quick and easy fix. The problem is with the new VOIP security in IOS it only accepts IP address at this time. Hopefully cisco will implement putting in domain names.

DHCP

Dynamic host Control Protocol

So in the typical world we use DHCP to get a IP address from a server. This allows us to have a server give out IP address to devices instead of them having staticlly assisgned. This also allows for easier troubelshooting.


For CUCM there is a DHCP server on the server. It just needs to be turned on. The helper address on routers needed to be configured so the Phones can get to the server. The other option is to configure DHCP on a IOS device. Key things to remember is to not give out sever address or router interfaces. The easy way to do this is to do a exclusion rule. Then the DHCP pool and key for VOIP is Option 150. Option 150 has the TFTP server that will be used give the phones there configs and other items from the call manager.

So for the config There is a order of operations according to cisco

1. Exclude the IP's you don't want to have given out.

IP dhcp excluded-address x.x.x.x(we can do a range if need be)

2. Create Pool

IP DHCP POOL VOICE site 1

3. Define network

Network 192.168.1.0/24(we can define the network with slash notation)

4. Default gateway

Default-Router 192.168.1.1

5. Option 150

Option 150 ip X.X.X.X (address of TFTP server)


I can't do screen caps of a call manager here, but once you configured the device on call manager we would need to add helper address

on the voice vlan add

ip helper-address X.X.X.X address of the call manager.

Network Time Protocol

How do we keep time? and why is it so important


When Troubleshooting having the correct time on the phones is very important to gather accurate data. Some of us who also work in Large networks or for service providers need a way to gather billing.


We can use a router or some other device on the network to get time. Routers can be a server or a client.

The routers can send out either a Unicast of BCST messages to give out the time. Authentication can also be performed on the routers.

Call Manger will use a NTP reference which points to the devices. In the newer versions of call manager the PUB will synch all the other CUCM servers.

Configuration on IOS based Devices is pretty easy


ntp server 192.168.2.2

Voice Vlans

Voice Vlans

Why do we have them?

We use them to Separate Voice traffic from the data traffic. When you can add QOS and monitor the Traffic separately. Its signaled to the IP PHONE from CDP. There are a few options to get the VLAN's going


Combined Voice and Data Vlan. We do this with Soft phones. The Switch port is set to Access Mode and the Data and Voice share that Vlan

Separate Voice and Data Vlans

2 Vlans on 1 switch port. Data frames are left untagged, The voice frames are Tagged. This allows us to identify which packets are which is will need QOS and so on.

Trunks with Voice Vlan.

Set the port as a trunk instead of a access port. Data will go as the Native Vlan and the Voice will go as the Voice Vlan. We still get QOS features with the separate Vlans.


With IP precedence and COS(layer 2 QOS) set, by default this is 5, but it55 can be changed and will be changed as we move forward into setting up VLAN.


The IP phone itself is a swich and on the Switch Port we can setup the Voice and Data Vlan on how to tag traffic.

The IP phone can

Tag voice VLAN TRAFFIC with Layer 2(COS)
Untaged with no COS


With the Data we can send it through the IP phone trusted where its not checked. Untrusted where we tag it with COS of our choice.


Config examples

Single Vlan

interface FastEthernet0/1/0
Switchport mode access
switchport access vlan 101
switchport voice vlan dot1p
spanning-tree portfast


Multi Vlan
Switchport mode access
switchport access vlan 101
switchport voice vlan 100
spanning-tree portfast


Remember to have your voice vlans in lower vlan numbers if possible so spanning tree can re converge.

What is H323

H323 is a suite of protocols which are used to control Voice and Video over IP based networks. Under H323 we have the following

H225- Set's up the call setup and Teardown between Dvices.
H225(RAS) Protocol between the endpoints and the gatekeer
H235- Specified Security for messages between the gateway and gatekeeper
H245- Controls Traffic flow, Performs DTMF, limits the media, Capability exchange.
H261/263- Video Conferencing
H.450 Supplementary services between H323 devices.

H323 components
Gateways, Gatekeepers, Terminals, and MCU's(multipoint control unit)


Gateways

These contain the connection between a IP based network, and the TDM network(PSTN) We configuring our dial-peers here, these can register with the Gatekeepers, These are usually are routers in most deployments. They can also handle phones being registered with them VS the call manager.

Gatekeepers.

Centralized point in a VOIP deployment. Gatekeepers can perform CAC(call admission control), and perform address Translation. H323 Can get messy with Dial-Peers configured to point to all devices. With a Gatekeeper(s) you can configure your entire dial-plan here and save on T/S hours as gateways will all have the same info.

Terminals

IP phones, Video phones and so on. They perform 2 way communication in the IP network. They have to support H245 for channel and capability control. They sent RTP streams from there devices to the other terminal.

MCU

Allow several participants in a conference.


How does H323 Work

The gateways exchange H225 Call setup Massages bewteen each other, then a H245 message is sent with Capability exchange. The IP address of the devices are sent and if it all works RTP packets start to flow.


H323 Fast Start/Fast Connect

When you are using this one of the setup messages will have fast start encluded which has the list of capabilities and list of codec options. The other end will reply with a fast start of its own with the same info. This allows the call to start right away without the delay that we hear sometimes when a call is setting up.

So how do we configure all of this.

Stay tuned its coming in future Post as we get closer to the written and need to start getting ready for labbing.

Can you hear me? One way Audio!!!!!!

When you make a call you expect to be able to hear the person on the other side? Thats a pretty simple request. In the world of communications most calls are actually not just one stream that is shared. Its 2 streams going in the opposite directions. In the IP world sometimes the packets take different directions, other times firewalls get in the way and numerous other issues that can make you have a less than pleasant call. So I had a customer that I have been dealing with who was experiencing this issue on international calls. Real quick troubleshooting pointed to the SIP provider as all other calls were working which took PRI circuits to the PSTN. Now this is here this gets interesting. We have a provider that can only troubleshoot through packet capture. They could take the debugs showing that we were packets only coming one way. This router was in a situation where we can't get packet captures so we can do traffic export. This feature is new in IOS 12.3 So its been around for quite so time.


Config t
ip traffic-export profile audio mode capture
bidirectional
exit

!apply to an interface
interface g0/0<------interface you want to capture from ip traffic-export apply audio [size ]


Then you can use these exec (enable) level commands:

traffic-export interface g0/0 clear
traffic-export interface g0/0 start



traffic-export interface g0/0 stop
traffic-export interface g0/0 copy ftp://username:password@x.x.x.x/capture.pcap


Pretty easy to do. Then we sent them off to the provider. They adjusted some routes and my problem was solved. In the end of day make sure you understand all the tools available so you can effectively troubleshoot.

SIP Basics

What is SIP

Sip is a peer to Peer App layer(7) protocol used for signaling and setting up sessions. It's independent of the underlying transport so it can be TCP,UDP, MPLS, ATM and so on. Sip is also text based so its easier for us to read. Sip runs usually on port 5060 for connection to SIP severs and endpoints. We use SIP to setup and Tear Down Voice, Video, or any application that requires a session. The actually communication is done over a different transport which is usually RTP in the case for VOIP.

Why do we use SIP?

SIP is here for us to provide PSTN like features in a IP based world. Sip provides the call setup and signaling for this to happen. So when you pick up your phone to call someone, when you dial a number, when you hear ring, back and busy signals.


SIP components

SIP has 5 parts that play a role in getting it up and running

USER Agent

VOIP phone, Cell Phone, Video Terminal. These devices can setup and Terminate sessions when they exchange the correct message. Subsets of these include User agent Clients which can send SIP request, and User agent Servers which will act as a "proxy" and acts on the behalf of the user


Proxy Server

This acts as a server and client for making request. The request can be send on to external devices or acted upon internally.

Redirect Server

They take SIP request, Give it a address and returns them to the client.

Registrar.

Server takes request for updated the database.



B2BUA

Back to Back User agent.

Operate both endpoints of a phone call, Acts as a user agent server and client and generates request and responds.



Other things to know:

Sip functions just like HTTP in how it was made. As I said above its Human Readable text.



SIP Messages


2 types that we have

Request and Responses

Request are sent from the client to the server, and responses are sent in the opposite direction.

Sip request messages

"INVITE" which tells you someone is trying to setup a call with you.
"ACK" A final Response to a "INVITE"
"BYE" Ends a call
"CANCEL" Just like it says cancels a search
"OPTIONS" Queries capabilities
"REGISTER" Registers the User Agent with the Registrar

SIP also Has response messages which are like the usual HTTP errors we see like the 404 that we get when browsing


SIP address

Fully Qualified Domain names, E164 Address or mixed address

So how would we configure such a think on our gateway and Call Manager?

Stayed Tuned

There is a lot of flexibility so in the future we are going to cover setting up SIP to a ITSP, Setting up SIP for functionality in CUPS, CUE and possibly some SIP 3rd party phones.

Do any Pro's use Hyper Terminal?????

Most of the time if we see a guy using hyper terminal its the sign of a a amateur in the world of who use the CLI for day to day task. Once a guy at work said "well if that's there's all to use/and its free" still told me in my head "amateur". We didn't' even use hyper terminal in Cisco Academy, and that was some years ago. So the question I ask today is "who still uses Hyper Terminal" is it even included with the current version of windows? The answer to that is a big fat no.

http://windows.microsoft.com/en-US/windows-vista/What-happened-to-HyperTerminal

CCIE Voice Writen Break down

Over the next few weeks I will be working my way though the CCIE Voice Written exam. I have purchased CCbootcamps Voice Book. I will also be using SRND for CUCM and UCCX and some other documentation that I have pulled from cisco's website regarding the Voice written exam. The goal of the post are to find a place to document my notes as I work my way through the exam topics. I have a few hurdles at work coming up so this won't be a ready aim fire at the exam. STAY TUNED.

Going for CCIE Voice

I have been working on Voice for quite a while. I have been around since the call manager 4 days. I took about 15 month break to work in the normal route/switch world and that has helped me tremendously, but I have a love for voice and that's where I"m back too. I recently worked with a TAC engineer who I work with often and we discussed my daily job and it looked like I could go straight to the IE voice with some back filling of topics I'm weak in. I have decided to go for this. Stay Tuned As I post my way through tackling the written and the lab.

Don't let people SIP from your cup

Toll Fraud has been around for ages. Weather it was the Phreakers with there whistles or people using social engineering to get outside dial tone by calling a trusting receptionist its been easy to steal long distance or Internatonal calling if you were creative. In the new world of SIP, toll fraud has followed the new technology. I got a call from a customer about there 800 number being busy. A quick look at the router reviled that all 23 lines of there PRI was being used. The IT manager said thats not true. So I figured he may be getting hit by toll fraud. The IOS output also showed me that all the calls were coming in then being sent to the PSTN overseas. We shut down the SIP dial-peers and bounced the PRI since we didn't know what calls were legit. A quick google search of SIP toll fruad showed us how easy it was to do. Newer versions of IOS 15 have features in to block it. You can view them on the link below.




In the end this has been posted a few times on a lot of blogs. Protect yourself and keep your customers happy.

Hosted VOIP vs On site

As the world moves into the "cloud", telephony has made its way there. There are several vendors selling hosted Voip solutions to customers; while others are saying buy your own gear keep it on site and you control your own destiny. I've had the opportunity to work both, and while both have there Pro's and Con's companies need to understand what they are getting into and how to think about there future needs. IP telephony is not a one size fits all service and you have to closely think about the what you will be paying for.

Hosted VOIP Pro's
You can get a feature rich system, and all you pay for is the telephones, switches and a monthly fee for services you use. Some of the providers offering messaging to email, contact center, video, faxing services. You can get large corporation features in your small business that can give you an edge. You won't need an on site telephony person to take care of your needs, and you can even provision your own phones to add new users.


Your own equipment pro's:
You own your own gear. This means you are free to find the cheapest service you can get from a provider. Also if you don't like the service contract you purchased, you can go find someone else. All you need to do is change passwords and cut access. If you have an on site Engineer, you can get changes done quickly without putting in tickets to a service desk. You can work on crazy customizations without having to pay a fee from your provider.


Hosted VOIP Cons:

You are at the mercy of the provider. They own the call processing component. If you want a feature they don't offer you are out of luck. Engineers you work with can vary in skill. If they have a outage in the data center you are stuck like chuck. One I worked with had a broadcast storm. Calls for 250 clients didn't go to the PSTN. The Hosted call center went down. We also had several voicemail outages during my time there. If the company goes out of business guess what? Who are you going to be getting your call processing from. Costs can sneak up on you. A company with high turnover MACs can add up quickly. Companies with expected high turn over like call centers should be very careful when choosing a hosted solution.


On Site equipment Cons:
Cost, Cost, Cost. There are huge upfront costs associated with getting your own telephony on site. Cisco, Avaya, Nortel, can be pricey. You will have to pay for installation, and depending on the size of your company you will have to pay for on site employees, or you have to contract it out to a provider. What I have also seen is that some other network folks inherit it, and they just want to stay as far away from the phones as they can. Something we take for granted like phones are a nightmare to people who keep the VOIP packets flowing.


At the end of the day you have to decide what your company can afford and your needs. I like both solutions, and I've seen both of them fail when IT managers didn't ask the right questions about what they were purchasing and leverage there needs against them.

When poop hits the fan!!!!!

So we talk about this a lot on the boards and in real life. So how do we handle this sort of event. You get a call that XXXX is down and nobody has internet or phone access across several sites? Where do you begin? Do you hit the sweat button? Is this your time to take a coffee or smoke break to get mentally ready to get in the game? Some people like this part of the job more than others. I just happen to be a guy that likes to deal with outages. I have several thought the day ranging from 1 user has no access to my entire company can’t make calls. I’m going to shed some insight on a method to the madness of how you can handle the call, excel, and be the envy of your peers. I also want to hear from you, you might know something better and we can all learn something new.





To me I approach all outages the same. Before I jump into gear, I ask preliminary questions?



1. What is the problem?

2. When did it happen?

3. How many users are affected?

4. Any changes to your system recently?





Ok let’s be honest, on some of these questions nobody is going to be completely honest with you most of the time. I’m usually talking to another IT guy that was probably messing around and doesn’t want to come clean, and there are several stories with that regarding call centers and auto attendants that I will explain one day, But this gives us a place to start. Next I get access to the network in whatever place is left. Sometimes it’s a terminal server if it’s a complete outage, sometimes I can VPN into another site. Once I’m in I get my tools ready and fire most of them up. Usually SecureCRT, Notepad++ Kiwi Cat Tools, Wireshark, Kiwi Syslog, and a command prompt for pings. Next I get access to whatever devices seem to be the problem or the closest I can get to it. Now comes my initial play book. SHOW Commands regarding the particular technology and checking the log (I’m also checking for last login and changes made by). 75-95 percent of the time the problem is right there if I go up the OSI model. Other times debugs will be needed. That’s where the syslog sever and other tools come in. This Is now the time when people are starting to ask questions. The usual answer of I’m running debugs looking for any errors is enough to back up most people I have dealt with. Other times the VP of technology or whatever are on the phone and they start asking questions, usally they dont know what the hell they are talking about, One enterprise architect told me that he could reach a 192.168 network from his connection at home:) I usually tell them give me a min I’m going to put you on hold, while I gather more info. If they become too problematic I can get someone else in to run damage control cause there asking for updates every 5 min taking time away to solve the problem. If I still don’t have it up after 30-45 min and it’s a global outage I will reboot whatever device it is. That has more success rate than we like to give it credit to for smart IT guys the old raytheon reset is below us. Then after a hour or so it’s time to escalate to someone smarter than me. Sometimes it’s to cisco, other times its to another engineer in the office who has worked with that technology. In the end we solve some, others kick out arse, but it’s how we all learn.

My CCIE rack and some company in DC

Over the past few weeks things have been chaotic for me. New job, and other personal endeavors have taken up most of my time (I'm about to get engaged). I just had to post this last encounter as it was too odd. Earlier this week I got a call around 4:45 pm about a network that keeps going down. We had some systems people in there working on Exchange, citrix and netapp. They had been rebooting their switches and the network would come back up and the switches would go haywire again. Anyways it gets so bad that someone had to go. I figured it was on my way home and I'll just unplug some cables break the loop and get back to normal life. Well 7 hours later I was still there with no resolution. First off we didn't have management to the switches and they just appeared to have some weird behaviors. So I'm getting hell from the switches, the owner of the company, and my girlfriend so how do we solve this problem. Back at the job there were no switches that met the needs of the customer, but wait...I have a couple of switches in my rack back home. I spoke about the switches I had and management agreed to put them in place the next day. Came home saved my configs from them, erased them, and got them ready. Plugged them in and the network is pumping out packets like a champ. What have we learned from this? Site surveys mean everything, make sure you have access to the entire environment your about to change.