It’s amazing how some companies manage to function, or even survive.

I worked with a company who operated a large IBM Mainframe network.

Now back in the day (late 1980’s), essentially there were two products that could be used to monitor an IBM SNA Network.  The NetMaster product, and the somewhat inferior IBM Netview Product.

You could use either product to detect network problems, but the (free) NetView product required more skill to interpret the information.

The type of errors we’d generally aim to spot were latency and transmission errors.

Latency is a measure of how long it takes for a packet to travel along a network.  With IBM SNA, you would get a broken network connection with a latency of more than 2 seconds.  So you start to worry about traffic taking 1.2+ seconds.

Transmission errors. Much like how static on a phone line makes it hard to talk, high transmission error rates makes it hard for computers to talk on networks.  Eventually, with growing transmission errors, the computers would stop talking.

So in a well managed network, you’d monitor both, with the aim of keeping up times, up.

“We don’t do that here.”, was the comment I heard in the first week I was there.  I was amazed the company managed to function at all.

When we talk about networking, sometimes we’ll talk about Maximum Transmission Unit (MTU) packet sizing.  I briefly referred to MTUs in the SSL errors, and how to diagnose them post.

Why should you care?  Well if you want a fast internet connection, you should.

Think of the MTU as a 4 pint jug.  Your 4 pint jug can ever only hold 4 pints.  But say you are trying to fill it from an 8 pint (gallon) bucket.  You need two jugs.  This, in networking, we call packet fragmentation.  Packet fragmentation was the root cause of the issue in the SSL errors post.

Packet fragmentation is bad.  To stretch the bucket example, you’d need to make two trips as your 4 pint is too small.  It’s faster to make one trip.

So how do you set the maximum MTU for your network connection?
(Note: if you are running Windows Vista or Windows 7, you should not need to.  But if you want/need to, there is a handy guide here.)

Well there are two ways to adjust your MTU.

The hard way – MTU Ping Test

We do a series of ping tests using the ping command like this:
ping www.google.com -f -l xxxx,
where (more…)

Reading a Microsoft blog post recently, I was reminded of a customer request from late 2002.  The emailed request was something like this:

Had to use the generic Win NT4 install on a Toshiba laptop yesterday.  There were some issues.

No network adapter is installed at the time of the Win NT4 installation.* This causes an error with the installation of eTrust and the CA UniCenter software.  It also means that networking component needs to be manually installed AND Error 7001 is also written to the system log.
To fix the Error 7001, we need to reinstall Service Pack 6a.

My suggestion is that you install the Microsoft Loopback Adapter during the installation, as it will solve the build issues we’re finding.

This was a brilliant suggestion as it turns out.  We would occasionally see build failures due to our NT4 build not having the network card drivers for the newer network cards.  Older versions of SQL (SQL 2000?) also needed a network card to be installed, so SQL would install properly.
The loopback adapter was a suitable work around for those issues.

These days, I would only use a loopback adapter with virtual machines (think VMware/VPC) IF the host machine didn’t have a physical network connection.
A loopback adapter will give you a working TCP/IP stack.

On Windows 7, it’s not obvious or easy to add a loopback adapter.  But Cesar de la Torre tells you how, on his MSDN Blog.

“ In any case, if you want to run the Wizard where you can manually add hardware, you need to start it from the COMMAND PROMPT:

1. Run cmd, but do it like: “Run as Administrator”
2. From the command prompt, write down “hdwwiz.exe” and execute it. Then, the “Add Hardware Wizard” will be launched.
3. Select: Install hardware manually –> Network Adapters –> Microsoft –> Microsoft Loopback Adapter.

You can read more info about it (step by step) in the following URL:
How to install a Loopback Adapter in Windows 7 (Windows Reference site) “

~~~

* – Network adapters WERE installed for supported desktops & laptops.  The customer had an unsupported laptop.

Frankly, I don’t know, but here’s what I learnt.

It started with a customer reporting a problem

When we press the publish button on the website, we get a 403 error.

A co-worker of mine picked up the call.  After trying many different things, he asks our Network team for help.

If it works on a standard Windows PC, we’re not interested in even having a look.

“What next?”, asked my colleague.

Never, ever volunteer; but the words sprang from my mouth,

“Perhaps I can be of assistance.”

Some network tracing was done, and the problem was SSL related.  An SSL error was being thrown.  The application server was throwing a SSL Malformed Packet back at us.

As SSL traffic is encrypted, you can’t tell much more than that, unless you turn off encryption.  Not going to happen on a production system.

My first guess at the solution was wrong.  An schannel.dll update didn’t fix the problem.  I spent a bit of time analysing what the JavaScript code was doing as well.  Couldn’t find any issues with it.

So what else do we see in the network trace?  Ah… we’re getting packet fragmentation.

Hmmm.

Told my colleague to look at the PMTU Discovery setting and to turn it on, to eliminate the packet fragmentation.  That was wrong too, as it was already on.
But turning it OFF fixed the issue.

So what are the important “take-aways”?

• Sometimes your first, second, and even third guess, can be wrong.
• Sometimes you need to cut your losses with an issue, but in this particular case, we did not get to that stage.
  IMHO, I thought we were close.
• Know what is “normal” behaviour, and what is “abnormal” behaviour.  Spot the difference.
  In this case, we had two network traces; a working one, and one which captured the problem.

Update 4th Oct 2009: Eric Law’s MSDN Blog post is worth a read: Internet Explorer Cannot Download https://something
Update 6th Dec 2009: Deb Shinder’s article is worth a read: SSL Acceleration and Offloading: What Are the Security Implications?

Non routable IP addresses are IP addresses which are not usable on the internet, but can be used on your home/corporate network.

10.0.0.0 – 10.255.255.255
172.16.0.0 – 172.31.255.255
192.168.0.0 – 192.168.255.255

… as defined by  RFC1597 – http://www.faqs.org/rfcs/rfc1597.html

A senior project manager reminded me of the power of Freedom of Information requests to stop things.

Yes, we can do that for the Minister, but the Minister should be aware that his request will be FOI discoverable.

Back in the day, let’s say 1998, a request came though to the IT group.
”Configure ISDN line and Cisco router to the home of senior public service executive.”

Said senior executive thought that using a dialup modem and VPN token, which everyone else was capably using, was far too difficult for her.  So, $20,000 later, the public servant had a connection to the government network, funded by our taxpayer dollars.

This would not happen these days, as some smarty would say

… that request would be FOI discoverable …

The Cisco router was called the Henderson router.

IF you wanted to analyse some network traffic, to see where things were going wrong, you would use a network hub.  A network hub is one of the devices you can use to listen to network traffic, as it’s happening.  In other words, you can listen in on a conversation between two or more computers.*

No more.

They don’t make network hubs anymore, network switches have become cheap.  The last 4 port network switch I brought was < $30.

So what can you do to tap network traffic?

• buy a secondhand hub from someone on eBay
  (this is what I did)
• some switches can be configured to act like a hub, but those sorts of switches tend to be expensive.
• NetOptics make the 10/100 Teeny Tap, which would be my pick if I was working in the field.
• look at Cisco NetFlow capable software
  (even more expensive)

 * "Why would you want to eavesdrop on network traffic?"

A real world example:
Customer reports network-enabled document scanner fails intermittently with a "network error".

So we replace the hardware, upgrade the firmware, and generally stuff around a bit, before we decide to strap on a network analyser.

Do things change when you observe them?  I think they do, no failures observed.

We disconnect the network analyser.  The network document scanner fails.  "Quick", I yell.  "We’re got a failure!".

We capture the error, and it looks like this:

…
MESSAGE: 220 Service Ready for New User
MESSAGE: (username sent)
MESSAGE: 331 Password required for DOCUMENTSCANNER01
MESSAGE: 221 Server is closing command connection
…

The cause is that the FTP server is cancelling the connection before the document scanner replies with the password.

Reference: Analyzing FTP Communications (from Novell)

I, of course, researched everything beforehand and had it all planned out so that the implementation of the new policy went without incident. Well, maybe there was a little trial and error. Now that I think about it, I guess there could have been a few issues on the back-end. OK, so I really broke everything, but it was a good learning experience anyway.

- “Restrict Anonymous: Enumeration and the Null User” – by Timothy M. Mullen, 2001

So true, been there, and gotten that t-shirt.  If we learn from our mistakes, then I know whole volumes.

… then it’s unmanaged and open to exploit.  As happened with a customer in mid-2004.

The “batphone” rings and it’s one of our desktop support guys.

“We’re detected a wireless router plugged into the customers network, what do you want me to do?”

Wait 5 while I see if it’s authorised… (it’s not)
‘Unplug that sucker!’

Customer complaint rolls in, with justification as follows:

1. It’s an executive PA who’s using the link
2. We’re using strong encryption (they were not, it was WEP)
3. We keep the encryption key closely guarded.

In the rush to fix an Executive PA’s LAN link, the customer broke their own security model.

With the backing of the customer’s security group, we said “It’s not being connected again.”

And the original problem was fixed next day.

Further reading:
WEP: Wireless security’s broken skeleton in the closet
The Batphone
How To Build Your Own Batphone

Some things I learnt include:

• SIP
• 1st level helpdesk folk will avoid your call if the device is not sold by them
• Whirlpool has an awful lot of crap, but the back-channel chat can be invaluable.
• You only truly find out how good service is, when you have a problem.

Session Initiation Protocol (SIP), or that “ruddy SIP” when it’s not working, is how Voice Over IP (VOIP) works*.  Like most protocols, it is a one-to-one conversation.  And because it’s a conversation, you can work out what is being said:

02-11-2008 20:18:24 Local7.Debug 192.121.0.44 ==>231.23.45.23: Received SIP packet from: 203.2.134.1:5060<013><010>SIP/2.0 183 Session Progress<013><010>Via: SIP/2.0/UDP 231.23.45.23:5060;branch=4574395743957<013><010>Call-ID: lkjfldsajf98r098rwfdskj@231.23.45.23<013><010>From: FredPhone <sip:0312345678@sip.fredphone.net:5060>;tag=lj5l45sf34r<013><010>To: 0412122222 <sip:0412122222@sip.fredphone.net:5060>;tag=jflfjldsjflsd-fjdlfjdlj4<013><010>CSeq: 1019589250 INVITE<013><010>Require: 100rel<013><010>RSeq: 749414039<013><010>Allow: ACK,BYE,CANCEL,INFO,INVITE,OPTIONS,PRACK,REFER,UPDATE,NOTIFY<013><010>Supported: 100rel<013><010>Contact: <SNIP>

… sometimes, with the right tools & hardware.  Without those, you can make inspired guesses, based on experience.  The snippet above is from broken SIP session.  I ring my cell phone from my VOIP service, and it rings once and hangs up.

Contacted my ISP’s helpdesk.  I have a problem with my Maestro ADSL modem and your VOIPfone VOIP service.
“No can help, we don’t sell the Maestro brand”. Darn.  The Maestro helpdesk were no better.  Remember that 1st level helpdesks aren’t much help.

So I whinge on Whingepool Whirlpool, the Australian Broadband forum.  Not a great deal of help to be found from my fellow posters, but I had a 3rd level support guy from Maestro contact me, and the VOIP tech for the VOIPfone service, both via email! 
I only got this support by being courteous and not bagging either product.
^—— important point.

The cause?
After many SIP protocol captures, it turns out to be a problem with the new Maestro firmware.  Maestro have supplied me with a test firmware which fixes the problem, but causes a problem with another feature
At least we’re getting there.

Nitpicker’s Corner
Not technically accurate, but close enough to the truth that you can get for free.  The true truth can be found by a) giving me money or b) reading here.
The Nitpicker’s Corner is inspired by Raymond Chen.

Maestro and VOIPfone.  The names have been changed to protect the guilty.
Incidentally, Maestro is a real brand.  They make rock-solid modems.