by September 5, 2007 0 comments

We have tested a lot of enterprise network equipment in our Labs. From
Servers to NAS boxes to Workgroup printers and so on. However, testing a UTM
device is amongst one of the most time consuming tasks. The reason for this is
very simple.

In just one device or box we have to test at least five or ten different
components. So testing a single UTM device will include testing an anti-virus,
spam filter, content filter, firewall, IDS/IPS, DMZ, etc.

For our shootout we created a test bed for testing four of the most important
components-anti-virus, anti-spam, IDP and firewall. In the following sections we
unfold our experiences with these UTMs and describe in detail how we went about
testing them. You can use a similar setup to test not only UTMs but also other
security devices. However, a word of caution! These tests are potentially
damaging for your network as they involve attack these devices with real life
viruses, spam and hacking tools. So, never carry them out on a production

Anti-virus tests
Testing for anti-virus capability is the easiest amongst all tests. We
simply need to create a Web, FTP and SMB server, and a set of different types of
viruses on top of it.

We used a Linux machine to host these viruses so that the hosting machine
itself doesn’t get affected by them. The viruses that we used had old 16-bit
viruses to the latest Trojans and malware. We used a set of viruses with around
1000 virus files. This set was kept constant for all UTM devices.
Once the host machine was ready with all viruses hosted on top of it, we
connected it to the public port of the UTM devices one after the other and tried
downloading all viruses from the private network. Once done, we counted the
number of viruses which bypassed the UTM and got downloaded on the public

Anti-spam tests
These tests are pretty much similar to the anti-virus tests, but more
categorized. We setup a machine with a POP3 Mail server running on it and dumped
around 2000 different spam mails on it. However, before dumping the spam, we
categorized it into text, image and PDF. Then we connected the machine to the
Internet and gave it a public IP address which is mapped with the MX record of a
domain. We took the UTM devices one by one and connected their WAN port to the

We then connected a few machines to its private network and started
downloading the spam using Outlook Express. Once done we checked how many spam
the devices had missed; to either tag or block, and counted the number for all
devices. Again, to compare the performance of all devices we kept the set of
spam identical for all devices.

Firewall tests
To test the firewall, we ran the industry standard vulnerability assessment
tool Nessus; and a standard DOS attack. For running the DOS attack, we used
ettercap’s Nice DOS plugin.

The test was pretty simple. We connected the WAN port of the UTM device to
the Internet with a public IP, ran Nessus and then the DOS attack, sitting on a
machine connected to the Internet from a different gateway.

To interpret the results, we counted the numbers of warnings, and issues
discovered by Nessus. And for the DOS attack we checked whether the device was
able to log and drop the attack or not.

IDS/IPS tests
To test the IDS/IPS functionality, we focused on the capability of the
device to detect internal attacks, or attacks that are generated from a
trusted/private network.

To test this we ran an ARP spoofing tool on the IP address of the public port
of the device (the IP which is used as the gateway address for the network), and
we tried to check what exactly the device does to prevent such kind of attacks.
ARP spoofing is a mechanism by which one can compromise the ARP cache of
switches, and divert all traffic intended for some other IP, to one’s own IP.
This technique is also known as ‘Man in the Middle Attack’ or ‘ARP flip-flop
attack’ or ‘ARP Poisoning Attack’.

We ran the tests in two modes. First, we spoofed the gateway IP and then
explicitly forwarded the data coming to the hacking machine, to the destination
gateway. And in the second mode we stopped forwarding all the data to the actual

Surprisingly, very few UTMs were able to detect and log this attack in the IP
forwarding mode. And none of them were able to prevent or take a precautionary

At the same time, access to a UTM’s private or gateway IP completely stopped
when we ran the test in a ‘non-IP forwarding’ mode. This shows that even now, a
‘Man in the Middle Attack’ is one of the most dangerous attacks from inside the
network and one of the stealthiest as well.

After all it comes to performance in the end. So to test and compare the
throughput from all boxes, we used a tool called QCheck. This was done by
connecting one machine to the public port of the device and another to the
private port. Then we installed QCheck on both the machines and ran it with a
packet size of 1Mb. The result was recorded in the form of throughput in Mbps.

No Comments so far

Jump into a conversation

No Comments Yet!

You can be the one to start a conversation.