'Gzip With Parallel Compression Support: Pigz'

November 22, 2009 Category: Uncategorized

The speed at which files are compressed with gzip is currently almost always determined by the speed of the CPU. However, standard unix gzip is single- threaded and only uses a single CPU (core).

However, the maintainer of the zlib library has released 'pigz' or 'pig-zee' whichs adds just that: support for parallel compression. This dramatically improves the speed at which a file can be gzipped.

In this example, a 3 GB compressable file is gzipped:

Gzip:

root@Core7i:~# time gzip pigz.bin

real 1m58.994s

user 1m56.480s

sys 0m1.820s

Pigz:

root@Core7i:~# time pigz pigz.bin

real 0m31.524s

user 2m54.890s

sys 0m2.900s

This simple and a bit unscientific example shows a 400% speed improvement. Since the Core i7 has four real cores, this shows that pigz scales nicely.

Comments