The "fio_generate_plots" didn't make me very happy as it didn't generate the kind of graphs I wanted. Furthermore, the script just contains some copy/pastes of the same blocks of code, slightly altered for the different benchmark types. I understand that the focus lies on FIO itself not some script to generate some fancy graphs, so don't get me wrong, but the script could be improved.

I used this script as the basis for a significantly reworked version, putting the code in a function that can be called with different parameters for the different benchmark types.

The result of this new script is something like this:

You can download this new script here. This script requires GNUplot 4.4 or higher.

The Linode Manager is the environment where you control your virtual private servers and where you pay for services. This is the environment that got compromised.

Linode uses Adobe's ColdFusion as a platform for their Linode Manager application. It seems that the ColdFusion software was affected by two significant, previously unknown vulnerabilities that allowed attackers to compromise the entire Linode VPS management environment.

As the attackers had control over the virtual private servers hosted on the platform, they decided to compromise the VPS used by Nmap. Yes, the famous port scanner.

Fyodor's remark about the incident:

I guess we've seen the dark side of cloud hosting.

That's the thing. Cloud hosting is just an extra layer, an extra attack surface, that may provide an attacker with the opportunity to compromise your server and thus your data.

Even the author of Nmap, a person fairly conscious about security and aware of the risk of cloud-hosting, still took the risk to save a few bucks and some time setting something up himself.

If you are a Linode customer and consider becoming a former customer by fleeing to another cheap cloud VPS provider, are you really sure you are solving your problems?

When using cloud services, you pay less and you outsource the chores that come with hosting on a dedicated private server.

You also lose control over security.

Cloud hosting is just storing your data on 'Other People's Hard Drives. So the security of your stuff depends on those 'other people'. But did you ask those 'other people' for any information about how they tink to address risks like zero-days or other security threats? Or did you just consider their pricing, gave them your credit card and got on with your life?

If you left Linode for another cloud VPS provider, what assures you that they will do better? How do you know that they aren't compromised already right now? At this moment? You feel paranoid already?

We all want cheap hosting, but are you also willing to pay the price when the cloud platform is compromised?

Resetting passwords is not something you do when under a simple attack such as a DDOS attack. Such a measure is only taken if you suspect or have proof of a serious security breach. I should have known.

There are strong rumours that Linode has actually been hacked. Although I signed up for a Linode VPS after the attack, I still checked my creditcard for any suspicious withdrawals.

Linode is as of this writing very silent about the topic, which only fuels my, and every other's suspicion that something bad has happened.

Whatever happened, even it isn't as bad as it seems, such an incident as this should make you evaluate your choices about hosting your apps and data on cloud services.

I don't care that much about rumours that creditcard information may have been compromised. Although in itself quite damning, what I do care is about the security of the data stored in the virtual private servers hosted on their platform.

I like this phase: "There is no cloud, only Other People's Hard Drives".

Everybody uses cloud services, so we all put our data in the hands of some other third party and we just hope that they properly secured their environment.

The cynical truth is that even so, a case can be made that for many companies, data stored in the cloud or on a VPS is a lot safer than within their own company IT environment. But an incident like this may prove otherwise.

And if you believe that data on a VPS is more secure than within your own IT environment, I believe that you have more pressing problems. The thing is that it doesn't tell you anything about the security of those cloud solutions. It only tells you something about the perceived security of your own IT environment.

The cloud infrastructure is just another layer between the metal and your services, and it can thus be attacked. It increases the attack surface. It increases the risk of a compromise. The cloud doesn't make your environment more secure, on the contrary.

So anyway, who performs regular security audits of Linode or (insert your current cloud hosting provider?) and what is the quality of the processes that should assure security at all times?

Questions. Questions.

This incident again shows that you should clearly think about what kind of security your company or customer data warrants. Is outsourcing security of your data acceptable?

Maybe, if security is an important factor, those cheap VPS hosts aren't that cheap after all. You may be better off creating your own private cloud on (rented or owned) dedicated servers and put a little bit more effort in it.

Building your own environment on your own equipment is more expensive than just a simple VPS, but you are much more in control regarding security.

About writes

So what does this mean? Well, if you write data to disk, the I/O subsystem only has to acknowledge that it has written the data to the actual medium. Basically, the application says "please write this data to disk" and the I/O subsystem answers "done, feel free to give me another block of data!".

But the application cannot be sure that the I/O subsystem actually wrote that data to disk. More likely, the application can be sure the I/O subsystem lied.

Compared to RAM, non-volatile storage like hard-drives are slow. Orders of magnitudes slower. And the worst-case scenario, which is often also the real-life scenario, is that both read and write patterns are random as perceived from the storage I/O subsystem.

So you have this mechanical device with rotating platters and a moving arm, governed by Newtons rules of physics, trying to compete with CPUs and memory that are so small that they are affected by quantum mechanical effects. No way that device is going to be able to keep up with that.

So the I/O subsystem cheats. Hard drives are relatively great at reading and writing blocks of data sequentially, it's the random access patterns that wreaks havoc on performance. So the trick is to lie to the application and collect a bunch of writes in a cache, in memory.

So, meanwhile, the I/O subsystem looks at the data to be written to disk, and reorders the write operations, so that it becomes as 'serialised' as possible. It tries to take into account all the latencies involved in moving the arm, timing that with the rotation of the platter and that kind of stuff.

A 7200 RPM hard drive can do only 75 IOPS with random access patterns, but that is a worst-case of worst-case scenario's. Real-life usage scenario's often allow for some optimalisation.

I used FIO to perform some random-IO performance benchmarks on different hard drive types and RAID configurations. It turns out that read performance was conform the 75 IOPS, but writes where in the thousands of IOPS, not a realistic figure. The operating system (Linux) employed heavy caching of the writes, lying to FIO about the actual IOPS being written to disk.

Thousands of IOPS sounds great, but you can only lie until your write cache is full. There comes a time when you have to actually deliver and write this data to disk. This is where you see large drops in performance, to almost zero IOPS.

Most of the time, this behaviour is overall beneficial to application performance, as long as the application usage patterns are often short bursts of data, that need to be written to disk. With more steady streams of data being written to disk in a random order, this might influence application responsiveness. The application might become periodically unresponsive as data is flushed from the cache to disk.

This write-caching behaviour is often desired, because by reordering and optimising the order of the write requests, the actual overall obtained random I/O write performance is often significantly higher than could be achieved by the disk subsystem itself.

If the disk subsystem is not just a single disk, but a RAID array, comprised of multiple drives, write-caching is often even more important to keep performance acceptable, especially for RAID arrays with parity, such as RAID 5 and RAID 6.

Write-back caching may help increase performance significantly, but it may come at a cost. As the I/O subsystem lies about data being written to disk, that data may get lost if the system crashes or loses power. There is a risk of data loss or data corruption. Only use write-back caching on equipment that is supported by battery backup units and a UPS. Due to the risks associated with write-back caching, there might be use cases where it might be advised not to enable it to retain data consistency.

About reads

The I/O subsystem can't lie to the application about writes. If the application asks the I/O subsystem "can I have the contents of file X", the I/O subsystem can't just say "well, yes, sure". It actually has to deliver that data. So any arbitrary write can be easily cached and written to disk in a more optimised way, reads may be harder. There is no easy way out, the I/O subsystem must deliver.

Where any arbitrary write can be cached, only a limited number of reads can be cached. Cache memory is relatively small compared to the storage of the disk subsystem. The I/O subsystem must be smart about which data needs to be cached.

More complex storage solutions keep track of 'hot spots' and keep that data cached. As a side note, such caching constructions can now also be found in consumer grade equipment: Apple's fusion drive uses the SSD as a cache and stores the data that is less frequently accessed on the HDD.

But in the end, regarding reads, chances are higher that data must be retrieved that is not stored in cache (cache miss) and thus the drives must do actual work. Fortunately, that work is not as 'expensive' as writes for RAID 5 or RAID 6 arrays.

Furthermore, reads can also be 'grouped' and serialised (increased queue depth) at the cost of latency to optimise them (setup a more sequential read access pattern for the disk subsystem) and achieve better performance. But again, at the cost of latency, thus responsiveness. That may or may not be a problem depending of the type of application.

Some remarks

If possible, it's better to try and avoid having to access the storage subsystem in the first place, if possible. Try and trow RAM memory at the problem. Buy systems with sufficient RAM memory, so that the entire database fits in RAM memory. A few years ago this was unthinkable, but 128 GB of RAM memory can be had for less than two thousand dollars.

If RAM isn't an option (dataset is too large) still try and put in as much RAM as possible. Also, try and see if server grade Solid State Drives (SSDs) are an option (always RAID 1 at least for redundancy!), although their cost may be an obstacle.

The gateway of last resort is the old trusted hard drive. If random I/O is really an issue, take a look at 15000 RPM or at least 10000 RPM SAS drives and a good RAID controller with loads of cache memory. In general, more drives or more 'spindles' equals more I/O performance.

You might encounter a situation where you want to add drives to increase I/O performance, not for the storage. More important: you may choose not to use that extra storage as it may decrease performance. Because if you put more data on a disk, the head must cover larger areas of the disk platter, increasing latency.

There are usecases where drives are intentionally under-partitioned to (artificially) increase the performance of the drives.

I have performed some benchmarks to determine how different RAID levels perform when handling a 100% random workload of 4K requests. This is a worst-case scenario for almost every storage subsystem. Normal day-to-day workloads may not be that harsh in a real-life environment, but worst-case, these tests show what kind of performance you might expect when you face such a workload.

To create a worst-case worst-case solution, I even disabled write-caching for all write-related tests.

At the moment, I only have access to some consumer-level test hardware. In the future, I'd like to rerun these tests on some 10K RPM SAN storage drives to see how this turns out.

RAID levels tested

I have tested the following RAID levels:

• RAID 0
• RAID 10
• RAID 5
• RAID 6

Test setup

• CPU: Intel Core i5 2400s @ 2.5 GHz
• RAM: 4 GB
• Drives: 6 x 500 GB, 7200 RPM drives (SATA).
• Operating system: Ubuntu Linux
• RAID: Build-in Linux software RAID (MDADM)
• File system: XFS
• Test file size: 10 GB
• Test software: FIO read-config & write-config
• Queue depth: 4
• Test script that generates RAID arrays, file systems and runs the tests.
• Cache: all write caching was disabled during testing (see script)

Test results

I also tested various chunk sizes for each RAID level. These are the results for RAID 10.

If you don't see any images, you are not using Internet Explorer 9, or a recent version of Google Chrome, Mozilla Firefox or Apple Safari.

Analysis

With this kind of testing, there are so many variables that it will be difficult to make any solid observations. But these results are interesting.

Results are in line with reality

First of all, the results do not seem unexpected. Six drives at 7200 RPM should each provide about 75 IOPS. This should result in a total of 450 IOPS for the entire array. The read performance does show exactly this kind of performance.

With all caching disabled, write performance is worse. And especially the RAID levels with parity (RAID 5 and RAID 6) show a significant drop in performance when it comes to random writes. RAID 6 write performance got so low and erratic that I wonder if there is something wrong with the driver or the setup. Especially the I/O latency is off-the-charts with RAID 6, so there must be something wrong.

Read performance is equal for all RAID levels

However, the most interesting graphs are about IOPS and latency. Read performance of all different RAID arrays is almost equal. RAID 10 seems to have the upper hand in all read benchmarks. I'm not sure why this is. Both bandwidth and latency are better than the other RAID levels. I'm really curious about a good technical explanation about why this should be expected.

RAID 0 is not something that should be used in a production environment, but it is included to provide a comparison for the other RAID levels. The IOPS graph regarding write performance is most telling. With RAID 10 using 6 drives, you only get the effective IOPS of 3 drives, thus about 225 IOPS. This is exactly what the graph is showing us.

Raid with parity suffers regarding write performance

RAID 5 needs four write I/Os for every application-level write request. So with 6 x 75 = 450 IOPS divided by 4, we get 112,5 IOPS. This is also on par with the graph. This is still ok, but notice the latency: it is clearly around 40 milliseconds, whereas 20 milliseconds is the rule of thumb where performance will start to significantly degrade.

RAID 6 needs six write I/Os for every application-level write request. So with 450 IOPS total, divided by 6, we only have single-disk performance of 75 IOPS. If we average the line, we do approximately get this performance, but the latency is so erratic that it would not be usable.

RAID chunk size and performance

So I was wondering if the RAID array chunk size does impact random I/O performance. It seems not.

Conclusion

Overall, the results seem to indicate that the actual testing itself is realistic. We do get figures that are in tune with theoretical results.

The erratic RAID 6 write performance would need a thorougher explanation, one that I can't give.

Based on the test results, it seems that random I/O performance for a single test file is not affected by the chunk size or stripe size of an RAID array.

The results show to me that my benchmarking method provides a nice basis for further testing.

I have installed a caching proxy server based on Squid, which is now used within my company. It also does content scanning using squidclamav and Clamav. I wrote an article about how to setup such a content scanning proxy.

The thing is that I didn't much care for the actual caching functionality of Squid, I deemed the content-scanning part more interesting. But I'm quite pleased with the actual caching hit ratio.

It seems that we have a hit ratio between 20% to 25% and that is more than I expected. Most content is dynamic in nature, so I would expect that most content is not cached but it seems that there is still quite some data that can be cached. This must also improve the end-user surfing experience as latency for downloading content should be reduced.

Of course, this is just a sample for the last hour. However, multiple measurements at different moments yield similar results.

I think this result proves that a caching proxy server is still relevant, especially if you don't have a fast internet connection. If you do, you can still improve the overall browsing experience due to the fact that data is cached.

There is a caveat: the proxy server itself also introduces latency. I haven't performed a side-by-side comparison and measured actual responsiveness of browsing with or without a proxy.

For instance, properties like how fast your car accelerates and how well the car handles bends and corners are often more important than its top speed. And this example also holds for storage performance.

Most people know that SSDs are often way faster than regular mechanical hard drives. But it's not about the throughput of these devices. Its all about Input/Output operations per second (IOPS). If you can handle a high number of IOPS, that is great for real life application performance. But IOPS does not tell you the whole story. To be more precise: IOPS is a meaningless figure unless tied to an average latency and a certain request size (how much data is processed with the I/O). Let's first focus on IOPS and Latency and talk about the request size later.

Latency is how fast a single I/O-request is handled. This is very important, because a storage subsystem that can handle 1000 IOPS with an average latency of 10ms may get better application performance than a subsystem that can handle 5000 IOPS with an average latency of 50ms. Especially if the application is sensitive to latency, such as a database service.

This is a very important thing to understand: how IOPS and latency relate to each other. Here, the car analogy probably breaks down. We need a different one to better understand what is going on. So picture you are in a super market. This is a special supermarket, where customers (I/Os) are served by cashiers (disk) at an average speed of 10ms. If you divide one second with 10ms, we understand that this cashier can handle 100 customers per second. But only one at a time, in succession.

It is clear that although the cashier can handle 100 customers per second, he cannot handle them at the same time! So when a customer arrives at the register, and within those 10ms handling time, a second customer arrives, that customer must wait. Once the waiting customer is handled by the cashier, handling of that customer still takes just 10ms, but the overal processing time was maybe 15ms or worst case (two customers arriving at the same time) even 20ms.

So it is very important to understand that although a disk may handle individual I/Os with an average latency of 10ms, the actual latency as perceived by the application may be higher as some I/Os must wait in line.

This example also illustrates that waiting in line increases the latency for a particular I/O to be handled. So if you increase the Read I/O queue, you will notice that the average latency will increase. Longer queues will mean higher latency, but also more IOPS!!!

How is that possible? How can a disk drive suddenly do more random IOPs at the cost of latency? The trick lies in that the storage subsystem can be smart and look at the queue and then order the I/Os in such a way that the actual access pattern to disk will be more serialised. So a disk can serve more IOPS/s at the cost of an increase in average latency. Depending on the achieved latency and the performance requirements of the application layer, this can be acceptable or not.

In future blog posts I will show some performance benchmarks of a single drive to illustrate these examples.

Disk /dev/md0: 58.0 GB
Disk /dev/md1: 2015 MB
Disk /dev/md5: 18002.2 GB
Disk /dev/sda: 60.0 GB
Disk /dev/sdb: 60.0 GB
Disk /dev/sdc: 1000.1 GB
Disk /dev/sdd: 1000.1 GB
Disk /dev/sde: 1000.1 GB
Disk /dev/sdf: 1000.1 GB
Disk /dev/sdg: 1000.1 GB
Disk /dev/sdh: 1000.1 GB
Disk /dev/sdi: 1000.1 GB
Disk /dev/sdj: 1000.1 GB
Disk /dev/sdk: 1000.1 GB
Disk /dev/sdl: 1000.1 GB
Disk /dev/sdm: 1000.1 GB
Disk /dev/sdn: 1000.1 GB
Disk /dev/sdo: 1000.1 GB
Disk /dev/sdp: 1000.1 GB
Disk /dev/sdq: 1000.1 GB
Disk /dev/sdr: 1000.1 GB
Disk /dev/sds: 1000.2 GB
Disk /dev/sdt: 1000.2 GB
Disk /dev/sdu: 1000.2 GB
Disk /dev/sdv: 1000.2 GB

You can use the following command:

#!/bin/bash
for x in `cat /proc/diskstats | grep -o 'sd.\|hd.\|md.' | sort -u`
do 
    fdisk -l /dev/$x 2>/dev/nul| grep 'Disk /' | cut -d "," -f 1 
done