The Case of the Windows FTP.EXE not working from the corporate network.

The heading alone should tell you the answer, particularly if I add ACTIVE FTP to it.

You see, most Corporate IT people consider ACTIVE FTP a bad thing, and block it at the internet gateway.  Sure FTP works within the company network, but as soon as you try to FTP something from outside the company network, it will fail.

The FTP clients that ship with Windows do not support passive mode. Therefore, they always need to negotiate a data port when issuing a command that returns data.
Windows FTP Client Receives Error Message 425 (MS KB271078)

But here’s the strange thing, it DID work on our network until two months ago.  I suspect our corporate IT security people have finally gotten around to locking down ACTIVE FTP, which was identified as a vulnerability back in 2000.

After much searching around, I settled on MOVEit Freely, as it’s a “drop-in” alternative to Microsoft’s FTP.EXE.

Some of the alternatives I looked at:

Send quote PASV within the FTP client.
Does not work.  It only sets the server to PASSIVE mode, not the client.

FTP_FOR_WIN32 from the GNU Project
Works, except it’s missing a couple of features, such as –s batch command(s).

NcFTP Client
Works, except it’s missing a couple of features, such as –s batch command(s).

MOVEit Freely
This is the one I decided to use, first saw it discussed here:
FTP Clients – Part 5: MOVEit Freely Command-Line Secure FTP Client

Passive FTP
Would definitely work, except that it’s a commercial product (at $29.95USD).  We’d prefer free.

TransSoft FTP Performer
Commercial product 🙁

GNU Wget
Yes it works, but the customer would have to modify their VBA scripts quite a bit to work with it.

Some of the nPing functionally is built into the Notes Client, but not much.

In February 2008, I wrote all about a Lotus Notes diagnostics utility called nPing.  It’s no more.  Go to the web link I provided then and you’ll see this message:

image

Thanks for almost nothing IBM.

You can find SOME of the nPing functionally in the Lotus Notes 8.5.x client, but not much.  Under the Files / Preferences / Notes Port menu option, there is a “Trace” button.

This is what it returns:
Lotus Notes Trace results

Or to dump it out in raw text:

Determining path to server xXXXAPPS02/Server/YYY
Available Ports:  TCPIP LAN0
Checking normal priority connection documents only...
Allowing wild card connection documents...
Enabling name service requests and probes...
Requesting address of xXXXAPPS02/Server/YYY from G0123/Server/NODDY1 on TCPIP
  Using address '10.1.2.32' for G0123/Server/NODDY1 on TCPIP
G0123/Server/NODDY1 has no address for xXXXAPPS02/Server/YYY on TCPIP
Requesting address of xXXXAPPS02/Server/YYY from G0123/Server/NODDY1 on LAN0
  Using address 'G0123' for G0123/Server/NODDY1 on LAN0
Unable to connect to G0123/Server/NODDY1: The server is not responding. The server may be down or you may be experiencing network problems. Contact your system administrator if this problem persists.
Checking for xXXXAPPS02/Server/YYY on TCPIP using address 'xXXXAPPS02'
Connected to server xXXXAPPS02/Server/YYY
Attempting Authenticated Connection
Determining path to server xXXXAPPS02/Server/YYY
Available Ports:  TCPIP LAN0
Checking normal priority connection documents only...
Allowing wild card connection documents...
Checking for xXXXAPPS02/Server/YYY at last known address 'xXXXAPPS02' on TCPIP...
    Using address '176.17.18.62' for xXXXAPPS02/Server/YYY on TCPIP
Connected to server xXXXAPPS02/Server/YYY
Connecting to xXXXAPPS02/Server/YYY over TCPIP
  Using address '176.17.18.62' for xXXXAPPS02/Server/YYY on TCPIP
Connected to server xXXXAPPS02/Server/YYY
Compression is Disabled
Encryption is Disabled

Our DNS server lies to me again!

By accident, I found out that my DNS server was lying to me again.  Last time I caught it out was when it was serving up the wrong IP addresses to me (see this post for details).

The problem occurs when I want to do something to a remote PC, say copy a file to it, or delete it from SMS.

First I try to “Ping” it to see if it’s alive on the network.  If the Ping fails, I move onto the next victim candidate.  How the Ping command fails for me, is something like this:

  1. The Ping command asks the DNS server for the IP address of the computer I want to Ping.
  2. The DNS server replies “No IP address for that computer”.
  3. I move onto the next computer in my Ping list.

Except the DNS server is lying at point 2.  If I do a second Ping, the DNS replies “Oh, the IP address is ww.xx.yy.zzz”.

Is it a “bug” or a “feature”?

Well I don’t know.  I first saw this behaviour with a Digital MicroVAX based DNS, many years ago.  The solution then was to adjust some thingymebob.  I was discussing the pros and cons of doing this with our MicroVAX guru at the time, and her comment was

‘Yeah, we can change the thingymebob setting, but it will slow down dohickey requests”.

You can tell by the precise technical terms thingymebob/dohickey, that I’ve forgotten what the settings were.  It’ll be some sort of DNS timeout value.

My workaround?

I’ll going with the “if the first ping fails, do another one” fix.  Mainly for the reason that having to explain the DNS issue to our current DNS guru, is not worth the energy involved.

And I’ve done some tests.  The DNS only lies to me once.  If the second Ping fails, the PC really isn’t on the network.

“We don’t do that here.”

IBM_3705_Front_Panel_Asmall It’s amazing how some companies manage to function, or even survive.

I worked with a company who operated a large IBM Mainframe network.

Now back in the day (late 1980’s), essentially there were two products that could be used to monitor an IBM SNA Network.  The NetMaster product, and the somewhat inferior IBM Netview Product.

You could use either product to detect network problems, but the (free) NetView product required more skill to interpret the information.

The type of errors we’d generally aim to spot were latency and transmission errors.

Latency is a measure of how long it takes for a packet to travel along a network.  With IBM SNA, you would get a broken network connection with a latency of more than 2 seconds.  So you start to worry about traffic taking 1.2+ seconds.

Transmission errors. Much like how static on a phone line makes it hard to talk, high transmission error rates makes it hard for computers to talk on networks.  Eventually, with growing transmission errors, the computers would stop talking.

So in a well managed network, you’d monitor both, with the aim of keeping up times, up.

“We don’t do that here.”, was the comment I heard in the first week I was there.  I was amazed the company managed to function at all.

Bookmark and Share

Determining your MTU, and why you should care.

When we talk about networking, sometimes we’ll talk about Maximum Transmission Unit (MTU) packet sizing.  I briefly referred to MTUs in the SSL errors, and how to diagnose them post.

Why should you care?  Well if you want a fast internet connection, you should.

Think of the MTU as a 4 pint jug.  Your 4 pint jug can ever only hold 4 pints.  But say you are trying to fill it from an 8 pint (gallon) bucket.  You need two jugs.  This, in networking, we call packet fragmentation.  Packet fragmentation was the root cause of the issue in the SSL errors post.

Packet fragmentation is bad.  To stretch the bucket example, you’d need to make two trips as your 4 pint is too small.  It’s faster to make one trip.

So how do you set the maximum MTU for your network connection?
(Note: if you are running Windows Vista or Windows 7, you should not need to.  But if you want/need to, there is a handy guide here.)

Well there are two ways to adjust your MTU.

The hard way – MTU Ping Test

We do a series of ping tests using the ping command like this:
ping www.google.com -f -l xxxx,
where Continue reading

The Microsoft Loopback Adapter, NT4 & Windows 7.

Reading a Microsoft blog post recently, I was reminded of a customer request from late 2002.  The emailed request was something like this:

Had to use the generic Win NT4 install on a Toshiba laptop yesterday.  There were some issues.

No network adapter is installed at the time of the Win NT4 installation.* This causes an error with the installation of eTrust and the CA UniCenter software.  It also means that networking component needs to be manually installed AND Error 7001 is also written to the system log.
To fix the Error 7001, we need to reinstall Service Pack 6a.

My suggestion is that you install the Microsoft Loopback Adapter during the installation, as it will solve the build issues we’re finding.

This was a brilliant suggestion as it turns out.  We would occasionally see build failures due to our NT4 build not having the network card drivers for the newer network cards.  Older versions of SQL (SQL 2000?) also needed a network card to be installed, so SQL would install properly.
The loopback adapter was a suitable work around for those issues.

These days, I would only use a loopback adapter with virtual machines (think VMware/VPC) IF the host machine didn’t have a physical network connection.
NT4 MS Loopback Adapter A loopback adapter will give you a working TCP/IP stack.

On Windows 7, it’s not obvious or easy to add a loopback adapter.  But Cesar de la Torre tells you how, on his MSDN Blog.

“ In any case, if you want to run the Wizard where you can manually add hardware, you need to start it from the COMMAND PROMPT:

  1. Run cmd, but do it like: “Run as Administrator”
  2. From the command prompt, write down “hdwwiz.exe” and execute it. Then, the “Add Hardware Wizard” will be launched.
  3. Select: Install hardware manually –> Network Adapters –> Microsoft –> Microsoft Loopback Adapter.

* – Network adapters WERE installed for supported desktops & laptops.  The customer had an unsupported laptop.

SSL errors, and how to diagnose them.

Frankly, I don’t know, but here’s what I learnt.

It started with a customer reporting a problem

When we press the publish button on the website, we get a 403 error.

A co-worker of mine picked up the call.  After trying many different things, he asks our Network team for help.

If it works on a standard Windows PC, we’re not interested in even having a look.

“What next?”, asked my colleague.

Never, ever volunteer; but the words sprang from my mouth,

“Perhaps I can be of assistance.”

SSL Malformed Packet Some network tracing was done, and the problem was SSL related.  An SSL error was being thrown.  The application server was throwing a SSL Malformed Packet back at us.

As SSL traffic is encrypted, you can’t tell much more than that, unless you turn off encryption.  Not going to happen on a production system.

My first guess at the solution was wrong.  An schannel.dll update didn’t fix the problem.  I spent a bit of time analysing what the JavaScript code was doing as well.  Couldn’t find any issues with it.

So what else do we see in the network trace?  Ah… we’re getting packet fragmentation.

Hmmm.

Told my colleague to look at the PMTU Discovery setting and to turn it on, to eliminate the packet fragmentation.  That was wrong too, as it was already on.
But turning it OFF fixed the issue.

So what are the important “take-aways”?

  • Sometimes your first, second, and even third guess, can be wrong.
  • Sometimes you need to cut your losses with an issue, but in this particular case, we did not get to that stage.
    IMHO, I thought we were close.
  • Know what is “normal” behaviour, and what is “abnormal” behaviour.  Spot the difference.
    In this case, we had two network traces; a working one, and one which captured the problem.

Update 4th Oct 2009: Eric Law’s MSDN Blog post is worth a read: Internet Explorer Cannot Download https://something
Update 6th Dec 2009: Deb Shinder’s article is worth a read: SSL Acceleration and Offloading: What Are the Security Implications?
Update 17th Dec 2009:
I had to revisit SSL sniffing in Internet Explorer has issues with session cookies, fancy that.

Bookmark and Share

$20,000 because the user was too dumb

A senior project manager reminded me of the power of Freedom of Information requests to stop things.

Yes, we can do that for the Minister, but the Minister should be aware that his request will be FOI discoverable.

Back in the day, let’s say 1998, a request came though to the IT group.
”Configure ISDN line and Cisco router to the home of senior public service executive.”

Said senior executive thought that using a dialup modem and VPN token, which everyone else was capably using, was far too difficult for her.  So, $20,000 later, the public servant had a connection to the government network, funded by our taxpayer dollars.

This would not happen these days, as some smarty would say

… that request would be FOI discoverable …

The Cisco router was called the Henderson router.

Bookmark and Share