Louwrentius

The most secure server system is a system that is not connected to a network and turned off. However, little work seems to be getting done this way. So we want to turn systems on and connect them to a network, or even (God forbid) the internet.

The thing is this. A system connected to a network without any running services is almost as as secure as a system that is turned off. They also share a common property: they are useless. A system starts to get useful if you start running services on them. And make these services accessible from the network for clients.

Services

Security on a technical level is all about securing those services. Every service that you enable is an opportunity for an attacker to compromise your system. If a service is not installed or running on your system, it cannot be used to compromise your server.

If a service is enabled and accessible through the network, it is logically of vital importance that you know:

1. what does this service do?
2. what can it be used for?
3. what steps needs to be taken to properly secure it?

If you know what a service does, you can understand the potential security risks. If you understand the product you are using, you can secure it properly. Security is all about understanding. If you don't understand what you are running, then it can't be secure.

Firewalls

So if you only run required services, why do you need to run a firewall? You don't. Yes that's right. Think about it. A firewall protects services that should not be accessible and allows access to services that should be accessible. If you just disable those services that should not be accessible from the outside, why use a local firewall? You don't want the Internet to access the SNMP-service on your system, you say? But then why not bind it only to the management interface instead of the production interface? You have a separate management network, right?

Of course, firewalls are a good thing. They are an ADDITIONAL line of defense. They mostly protect you against yourself. If you make a mistake and, by accident, enable some vulnerable service on a system, a properly configured firewall will prevent access to it and save your behind. That is the purpose of a firewall.

People often wrongly see the firewall as the first line of defense. If you do, you are wrong. The first line of defense is to secure your services.

The whole point is that there are holes in your firewalls. Those holes allow access to services. Those services may be necessary, like a web server, but nevertheless holes. You are exposing services to the Internet.

Web applications (or web-based back doors?)

We are now mostly running web-based applications on the services that we make accessible for the network or the internet. Those applications run on application servers. Yes, these application servers, like Apache Tomcat or IIS ASP.NET need to be secured, but nowadays, they are almost secure by default.

All security depends on the level of security of the application you are running on your application server. Is your application written well, with security principles in mind? Does it protect against SQL-injection or cross- site scripting? Are sessions predictable? Can a user access data of another user?

Firewalls don't protect against vulnerabilities in your web applications. You need to do it right at the core level: the application itself. Just like how you harden a system. You must run secure code.

And be aware that if you run third-party code, watch out for security news. There have been many worms exploiting vulnerable commodity software such as phpBB, Wordpress or similar products.

This is the really hard part. Deploying secure software and keeping it secure during the development life cycle.

Patches

The last fundamental principle of keeping systems secure is keeping up with security patches. Many security vulnerabilities are often only exploitable under specific conditions and may not be that important. But the most important thing is to be aware of vulnerabilities and available patches. Then you can decide for yourself how to act.

There is always a risk that a security patch breaks functionality. But that's not a real problem, because you have this test environment so you can check first, right?

Keep up with security patches and non-security patches. If you first have to install 100+ patches to be able to install the latest high-risk security patch, something might break. So then it's choosing between staying vulnerable or going off-line until you have fixed everything.

Conclusion

So what are the most basic ingredients for secure systems?

1. only run required services
2. harden those required services
3. deploy a firewall as an additional defense layer
4. deploy secure application code
5. keep up-to-date with security patches
6. Audit and review your systems and application code on a regular basis.

With this small number of steps, you will be able to protect against a lot of security threats. I don't say this is everything that is necessary. But it is a good foundation to build on. You still have to identify risks that may apply to your particular situation. These risks may require you to take (additional) measures not discussed here.

My router decided to change the device name for some USB storage devices. So /dev/sdc was swapped for /dev/sdd and vice versa. The result was some file system corruption on /dev/sdc, because it was used on a remote system through iSCSI, using a different file system from /dev/sdd.

With regular internal disks, attached with PATA, SATA or SAS, the chances are very small that such an event will occur, but it is possible, especially if you start adding/subtracting disks. With USB devices the risk is substantially bigger.

To prevent your system from mixing up drives because there device names change, use file system labels. All information that follows have been stolen from this location. Since this blog is also my personal notepad, the relevant bits are reproduced here.

There are three steps involved, the third being optional:

1. add a label to the file system
2. add the label to /etc/fstab
3. update grub boot manager (optional)

Add a label to the file system

Setting a label when the file system is created:

mkfs.ext3 -L ROOT /dev/sda1
mkfs.xfs -L BIGRAID /dev/sde

Set label for existing file system

EXT3:

e2label /dev/sda1 PRIMARY_ROOT
e2label /dev/sda1

XFS:

xfs_admin -L DATA1 /dev/sdf
xfs_admin /dev/sdf

Set label for swap partition

mkswap -L SWAP0 /dev/sdb5

add the label to fstab

Example of contents of fstab:

LABEL=ROOT          /         ext3    defaults        1 1
LABEL=BOOT          /boot     ext3    defaults        1 2
LABEL=SWAP          swap      swap    defaults        0 0
LABEL=HOME          /home     ext3    nosuid,auto     1 2

Update the grub boot manager

title server
root (hd0,0)
  kernel (hd0,0)/vmlinuz ro root=LABEL=SERVER_ROOT0 rhgb quiet
  initrd (hd0,0)/initrd.img

Most web applications work like this:

The application uses a single database account to perform all actions. Users are just some records in a table. Account privileges and roles are part of this table, or separate tables.

This implies that all security must be designed and build by the application developer. I think this is entirely wrong. There is a big risk:

In such applications, SQL-injection will allow full control of the entire database.

This is something that is often overlooked. And the solution is simple. The application should not use a general account with full privileges. The application should use the database account of the user accessing the application. All actions performed by this user are thus limited by the privileges of this database account. The impact of SQL-injection would be significantly reduced.

The public part of a website is still using an application account, but the privileges of this account can be significantly reduced. To obtain elevated privileges, a user must first authenticate against the application and thus the database.

Please understand another benefit: it is not required to store username/password combinations of privileged accounts on the application server. The configuration file will only contain the credentials of the unprivileged account. An attacker compromising the application server with limited privileges, won't have access to the database with elevated privileges.

I understand that this solution requires a bit more work to setup at the start, but once implemented, it reduces complexity and improves security so much.

Of course, the security of your data is as good as the hardening of your database server. But that's another story.

Hardware RAID controllers are considered 'the best' solution for high performance and high availability. However, this is not entirely true. Using a hardware RAID controller might even endanger your precious data.

For enterprise environments, where performance is critical, it is more important that the arrays keeps on delivering data at a high speed. Professional RAID controllers use TLER with TLER-enabled disks to limit the time spend on recovering bad sectors. If a disk encounters a bad sector, there is no time to pause and try to fix it. The disk is just dropped out of the RAID array after just a couple of seconds. At that moment, the array still performes relatively well, but there is no redundancy. If another disk fails (another bad sector?) the array is lost, with all its data.

More people are building NAS boxes for centralized storage of data, for private home use. Since disks are cheap, it is possible to create lots of storage capacity for little money. Creating backups of terabytes of data is however not cheap. Or you have to create two NAS boxes. But that is very expensive and not worth the effort.

People seem to spend lots of money on expensive enterprise level hardware RAID cards, not understanding that the whole TLER-mechanism causes an increased risk for their data. In enterprise environments, budgets are relatively big, and data is always backed up. They can afford to take the risk of losing a RAID array due to these backups. But consumers often don't have the money to spend on creating backups of terabytes of data. They just go for RAID 5 or RAID 6 and hope for the best.

For consumers, if the RAID array goes, all data is lost.

So consumers should choose a RAID solution that will do its best to recover from hardware failure. Performance is not so much an issue. Reliability is. So consumers do want disks to spend 'ages' on recovering bad sectors if this means that the RAID array will survive.

Linux software RAID and ZFS do not use TLER and therefore are a safer choice for your data then regular hardware RAID controllers. You may still use such controllers (but please test them properly) but only to provide SATA ports with individual disks, the RAID part should be handled by Linux.

So in my opinion, hardware RAID controllers are more expensive, require more expensive (enterprise) disks and are less safe for your data.

This article discusses Linux bonding and how to achieve 2 Gb/s transfer speeds with a single TCP/UDP connection.

UPDATE July 2011

Due to hardware problems, I was not able to achieve transfer speeds beyond 150 MB/s. By replacing a network card with one from another vendor (HP Broadcom) I managed to obtain 220 MB/s which is about 110 MB/s per network interface.

So I am now able to copy a single file with the 'cp' command over an NFS share with 220 MB/s.

I had problems with a intel e1000e PCIe card in an intel DH67BL. I tested with different e1000e PCIe models but to no avial. RX was 110 MB/s. TX was always no faster than 80 MB/s. A HP Broadcom gave no problems and also provided 110 MB/s for RX traffic. LSCPI output:

Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI Express

The on-board e1000e NIC performed normal, all PCIe e1000e cards with different chipsets never got above 80 MB/s.

A gigabit network card provides about 110 MB/s (megabytes) of bandwidth. If you want to go faster, the options are:

1. buy infiniband stuff: I have no experience with it, may be smart thing to do but seems expensive.
2. buy 10Gigabit network cards: very very expensive compared to other solutions.
3. strap multiple network interfaces together to get 2 Gb/s or more with more cards.

This article is discussing the third option. Teaming or bonding two network cards to a single virtual card that provides twice the bandwidth will provide you with that extra performance that you where looking for. But the 64000 dollar question is:

How to obtain 2 Gb/s with a single transfer? Thus with a single TCP connection?

Answer: The trick is to use Linux network bonding.

Most bonding options only provide an accumulated performance of 2 Gb/s, by balancing different network connections over different interfaces. Individual transfers will never reach beyond 1 Gbit/s but it is possible to have two 1 Gb/s transfers going on at the same time.

That is not what I was looking for. I want to copy a file using NFS and just get more than just 120 MB/s.

The only bonding mode that supports single TCP or UDP connections to go beyond 1 Gb/s is mode 0: Round Robin. This bonding mode is kinda like RAID 0 over two or more network interfaces.

However, you cannot use Round Robin with a standard switch. You need an advanced switch that is capable of creating "trunks". A trunk is a virtual network interface, that consists of individual ports that are grouped together". So you cannot use Round Robin mode with an average unmanaged switch. The only other option is to use direct cables between two hosts, although I didn't tested this.

Results

UPDATE July 2011 : Read the update at the top.

Now the results: I was able to obtain a transferspeed (read) of 155 MB/s with a file copy using NFS. Normal transfers capped at 109 MB/s. To be honest: I had hoped to achieve way more, like 180MB/s. However, the actual transfer speeds that will be obtained will depend on the hardware used. I recommend using Intel or Broadcom hardware for this purpose.

Also, I was not able to obtain write speed that surpasses the 1 Gb/s. Since I used a fast RAID array to write the data to, the underlying storage subsystem was not the bottleneck.

So the bottom line is that it is possible to get more than 1 Gb/s but the performance gain is not as high as you may want to.

Configuration:

Client:

modprobe bonding mode=0
ifconfig bond0 up
ifenslave bond0 eth0 eth1
ifconfig bond0 10.0.0.1 netmask 255.255.255.0

Server:

modprobe bonding mode=4 lacp_rate=0 xmit_hash_policy=layer3+4
ifconfig bond0 up
ifenslave bond0 eth0 eth1
ifconfig bond0 10.0.0.2 netmask 255.255.255.0

Bonding status:

cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.3.0 (June 10, 2008)
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Active Aggregator Info:
Aggregator ID: 2
Number of ports: 2
Actor Key: 9
Partner Key: 26
Partner Mac Address: 00:de:ad:be:ef:90

Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:co:ff:ee:aa:00
Aggregator ID: 2

Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:de:ca:fe:b1:7d
Aggregator ID: 2