Distributed Parallel Processing Shell Script (PPSS) Released

March 12, 2009 Category: Uncategorized

I'd like to announce the release of the distributed version of the Parallel Processing Shell Script (PPSS). PPSS is a bash script that allows you to run commands in parallel. It is written to make use of current multi-core CPUs.

The new distributed version of PPSS allows you to run PPSS on multiple hosts, that simultaneously work on a single group of files or other items. A central server is used for file locking. This way, nodes know which files are 'locked' by other nodes and/or have already been processed.

PPSS is written to be very easy to use. You can be up and running on a single host within 5 minutes. Distributed usage requires the (one-time) setup of some SSH accounts on your nodes and server, but that's about it. Please take a look at distributed PPSS for yourself at:

http://code.google.com/p/ppss

I used distributed PPSS to encode 400 GB of WAV files to MP3 using 4 computer systems. A total of 14 cores where available to PPSS to encode those files using Lame. However, PPSS is not limited to encoding WAV files. To the contrary, it is written to execute whatever command you want to execute. Gzip a bunch of files in parallel? No problem. It is totally up to your imagination.

Comments