Welcome to Gogolist !

Gogolist is a tool which aims to ease the run and monitoring of embarassingly parallel processings.

It’s a wrapper which submits jobarrays to batch schedulers (torque/maui, oar, pbspro...) and manages the output logs and progress reports. It is like an “xargs -P n”, but distributed and with an easy monitoring.

Why Gogolist ?

Jobarray submission to batch schedulers is generally easy, but monitoring the job progress and accessing to the error logs of a few tasks among thousands is the same as finding a needle in a haystack... This wrapper was created to avoid spending more time in manual “logfile-mining” (grep, tail...) than the processing time itself. Some other needs it addresses :

• listing management avoids too big jobarrays (which crashes some batch schedulers)
• list the tasks in errors to ease their reprocessing
• allowing several execution modes : sequential, streaming (pipe)
• allowing multiple batch scheduler as backend
• having tasks status reporting and real-time monitoring
• having job history, meta-data & logfiles organization

Getting Help

Send an email to fpaf@ifremer.fr