Louwrentius

I'd like to announce the release of the distributed version of the Parallel Processing Shell Script (PPSS). PPSS is a bash script that allows you to run commands in parallel. It is written to make use of current multi-core CPUs.

The new distributed version of PPSS allows you to run PPSS on multiple hosts, that simultaneously work on a single group of files or other items. A central server is used for file locking. This way, nodes know which files are 'locked' by other nodes and/or have already been processed.

PPSS is written to be very easy to use. You can be up and running on a single host within 5 minutes. Distributed usage requires the (one-time) setup of some SSH accounts on your nodes and server, but that's about it. Please take a look at distributed PPSS for yourself at:

http://code.google.com/p/ppss

I used distributed PPSS to encode 400 GB of WAV files to MP3 using 4 computer systems. A total of 14 cores where available to PPSS to encode those files using Lame. However, PPSS is not limited to encoding WAV files. To the contrary, it is written to execute whatever command you want to execute. Gzip a bunch of files in parallel? No problem. It is totally up to your imagination.

Even today, Core 2 Duo processors clocked at 2 ghz are no slugs. However, the Core i7 920 is of a different kind. First, it is not only clocked at a higher speed (default 2,8 Ghz), it is also a quad-core processor. Thanks to the re- introduction of hyperthreading, this processor can handle 8 parallel proceses simultaneously.

Just how fast a Core i7 can be, especially if overclocked to 3,6 Ghz shows this diagram:

Using my still in development version of PPSS, four systems processed 400 GB of WAV files and converted them to MP3. This simple pie-chart shows that the Core i7 on it's own, using 8 parallel processes, managed to process 2/3 of the files. The Core i7 was way faster than the other 3 systems combined! This is marvelous, I think. And it seems all due to Hyperthreading. If an additional duo core system would have been added, the other systems combined would have also 8 parallel threads available and would have processed roughly 50% of the items. However, please note that the Core i7 is a quad core processor and has 'only' four physical cores...