The Weakest Link

A couple of weeks back I came across a network that was under the spell of a

major attack. The first problem which stared at our face, was pretty mild. The

Net connection was not working. We tried to figure out what had gone wrong. For

that we connected a machine between the Firewall and the main network switch

from where all other switches had been cascaded. We did so to capture all the

data going out from the network. Then we ran some tools to determine the kind of

data that was going out from the network. We found that a huge amount of SMTP

traffic was going out from the network.

We were not so concerned about it at that time because we knew we had a

firewall and content management appliance running just before the gateway to

protect the network from any kind of attack or block any mass mailing. Plus,

each and every machine on the network had a licensed version of renowned anti

virus software. So, we thought that the SMTP traffic must be authentic and not

SMAM.

Anindya Roy Issue Editor for this month

But within the next 1 hour some horrifying things happened that took the

lights out of us. Some one from the admin team logged in remotely to the mail

server, which is hosted by some service provider abroad; and saw that there are

around three lakh mails queued to be sent out. This huge chunk of mail had

caused the server to choke. The admin guy tried to freeze the queue so that no

legitimate mail gets lost, and restarted the machine thereafter.

It was that very moment when our worst fears came true. The mail server

didn't turn up. We called up the service provider to check what had bugged the

server. They retorted back saying that the server was horrifically slow and

couldn't come on the network.

So, there was no way by which we could get into the server and do some

troubleshooting. We asked the service provider's IT teams to check the problems

locally and try to get those fixed. They told that the account type, which we

had with them, was an economy account and they don't provide troubleshooting

support for such an account. All they offered us was to re-install the server in

next 2 to 24 hrs.

It took them a good 24 hrs to reinstall the server. After that we had to

install Exchange and create user accounts with all individual settings as they

didn't have any backup of the settings even. In the end we faced a total of 30

hrs of mail server downtime, several thousands of mail getting lost and around

50 machines on the network infected with viruses.

All this took another three days to get cleaned completely. Now all this

happened because most of these 50 machines were attacked by a mass mailing

Trojan. And the Trojan was using those machines to send out spams using the mail

server, which actually made it unusably slow and caused it to crash.

But the point here is that in spite of having a firewall with content

filtering and anti viruses installed on all machines how did this happen? The

answer is even more shocking. The license for the firewall appliance had expired

and nobody had been alerted about it due to misconfigured alert systems. As a

result the firewall stopped working and was blindly passing traffic going

through it. Additionally, we found that users of those 50 machines, that had got

infected, had intentionally disabled the antiviruses because they were affecting

their machine performance. So, the Trojan had attacked these machines.

Now I am not sure if this happened because of lack of knowledge on the part

of the users or because of negligence by the IT staff and users. But the most

important lesson here is that spending money on security devices and software

doesn't makes any sense unless one actually enforces the user and usage level

policies in the organization.

You actually have to find and strengthen the weakest security link in your

LAN rather than cribbing about the shortfalls in the technology deployed. The

company in question learnt this lesson in the harshest possible manner. So, all

you IT managers out there, better watch out!