Jérôme Belleman
Home  •  Tools  •  Posts  •  Talks  •  Travels  •  Graphics  •  About Me

Parallel Shells and Transfers

22 Mar 2014

PSSH isn't just about running SSH in parallel. It's also about performing parallel transfers with various tools. I was sold by this versatile toolkit.

1 parallel-ssh

I will not dwell on explaining the basics of using parallel-ssh, which are both straightforward and well documented in the man page. However, there is a slightly more advanced aspect which I found rather neat and involved controlling host authentication failure messages and suchlike. You can douse these messages which would otherwise appear as many times as there are hosts to connect to by passing the ssh command the LogLevel=QUIET option. For instance:

parallel-ssh -OLogLevel=QUIET -H'foo.example.com bar.example.com baz.example.com' 'uptime'

This is a mere example of how even more useful parallel-ssh can be made by passing SSH options with the -O switch. They're usually no pleasant options to type and before long, you'll consider setting up command aliases:

alias parallel-ssh='parallel-ssh -OLogLevel=QUIET'

2 parallel-rsync

While parallel-ssh is rather straightforward to use, parallel-rsync isn't, and the documentation is rather incomplete. In addition to the man page, you want to check parallel-rsync --help and might want to partly refer to the parallel-ssh documentation, as the arguments follow the same logic. In particular, you can use the -H option to specify an in-line list of hosts.

The first argument (foobar in the following example) is the source, local file you want to copy. The second argument is the destination file in each remote host:

parallel-rsync -x-t -x-b -H'foo.example.com bar.example.com baz.example.com' foobar /root/foobar

Note the -x arguments which let you pass options to rsync. In this example, -x-t has rsync preserve modification times and -x-b has it make a backup, as it would normally do if run directly. For some reason, -x-v won't make it more verbose, however, even with parallel-rsync's -v option. In fact, there isn't much feedback in general.

3 References