#distributedcomputing Hello Mastodon!

Are some of you using #GNUParallel or #SLURM to distribute computations across a set of network nodes?

Requirements:
- multiple executions of a single C++ app, with different arguments
- no communication between executions required
- approximately 20000 executions to distribute across 5-10 nodes
- possibility to run 2-4 tasks in parallel on the same node to speed things up
- retrieval of individual output files for further analysis

What do you suggest?

4/5 #gnuparallel is the simplest way to do some simple #parallel computing.
Several tasks in life are embarassingly parallel, and #gnuparallel is a great
way to take advantage of all the cores of my machines. I probably use 0.1% of
its features, but that's enough for me. It something takes #commandline inputs,
it can be trivially parallelized with #gnuparallel. Interestingly, #gnuparallel
is a single nearly 15,000-line #perl script.

Happy 20th birthday to GNU Parallel! Read Parallel author Ole Tange's reflections on the occasion here: https://u.fsf.org/3hi #gnuparallel

I couldn't get a single example shown in the man page of GNU Parallel working in bash or zsh.

Trying with version 20161222-1.1 on Debian testing.

Guess I'll stick to xargs -P4 for now. 😥

#GNUParallel #Xargs #shell