About setimgr

Setimgr is a simple program to manage queuing of processing, collecting and sending the results of work units from the SETI@Home project using their UNIX client.

Setimgr features

UPDATED VERSION (0.04e)

(20th Mar 2002)

Changes

(Version 0.04d was a beta only distributed to a few people)

UPDATED VERSION (0.04c)

(3rd Mar 2002)

Brian Shaver has supplied the following bug fix and code/feature improvements:

Bug fix

Other changes

UPDATED VERSION (0.04b)

(24th Feb 2002)

Minor bug fix

Other changes

UPDATED VERSION (0.04a)

(19th Feb 2002)

Minor bug fix

NEW VERSION (0.04)

(11th Feb 2002)

Major Changes

Minor Changes If you want to be informed of updates/bug fixes, it is probably easiest to subscribe to setimgr on freshmeat.net

Portability

Platforms tested so far:

Download

The latest version (0.04e) can be downloaded here

(Older versions can also be downloaded 0.03a and 0.02c)

Using setimgr

Starting from scratch

There is a shell script simple_setup.sh included with the distribution which should do most of this work for non-complex setups, just create a sub-dir copy the setimgr distribution and the setiathome binary into it and run the script (from the sub-dir). (If you don't have a permanent connection then you will need to be dialed-in while running the script to download the initial work unit).

Or you can follow the instructions:

Upgrading from a previous version of setimgr

Upgrading from just running setiathome standalone

You should end up with set of directories like:
% ls -lR
.:
# Work unit directories proc1-4
drwxrwxr-x    2 user group    4096 Feb 24 07:50 proc1
drwxrwxr-x    2 user group    4096 Feb 24 12:47 proc2
drwxrwxr-x    2 user group    4096 Feb 24 07:50 proc3
drwxrwxr-x    2 user group    4096 Feb 24 07:51 proc4
# The setiathome binary
-r-xr-xr-x    1 user group  134072 Apr 14  2001 setiathome
# The setimgr binary
-rwxrwxr-x    1 user group   63137 Feb 24 08:20 setimgr
# The config file
-rw-rw-r--    1 user group     663 Feb 19 06:47 setimgr.conf
# This happens to be the name of the log file I use
-rw-rw-r--    1 user group    4942 Feb 24 19:20 setimgr.log
# The file in which setimgr puts its pid.
-rw-rw-r--    1 user group       5 Feb 24 08:20 setimgr.pid

./proc1:
# The work unit you copied in
-rw-rw-r--    1 user group     132 Feb 23 03:04 key.sah
-rw-r--r--    1 user group       0 Feb 16 05:46 lock.sah
-rw-rw-r--    1 user group       0 Feb 23 03:04 outfile.sah
-rw-rw-r--    1 user group       5 Feb 23 03:01 pid.sah
-rw-r--r--    1 user group    1528 Feb 21 15:08 process.log
-rw-rw-r--    1 user group     162 Feb 23 03:04 result_header.sah
-rw-rw-r--    1 user group       0 Feb 23 03:04 state.sah
-rw-rw-r--    1 user group     411 Feb 16 05:46 temp.sah
-rw-rw-r--    1 user group     367 Feb 23 03:02 user_info.sah
-rw-rw-r--    1 user group      32 Feb 16 05:46 version.sah
-rw-rw-r--    1 user group  356285 Feb 23 03:04 work_unit.sah
-rw-r--r--    1 user group    2113 Feb 23 03:04 xfer.log

./proc2:
# user_info.sah copied from proc1
-rw-rw-r--    1 user group     367 Feb 23 02:42 user_info.sah

./proc3:
-rw-rw-r--    1 user group     367 Feb 23 02:42 user_info.sah

./proc4:
-rw-rw-r--    1 user group     367 Feb 23 02:42 user_info.sah

The config file

Config files only needs to contain the options which you want to be different to the default.
The default file looks like:
# Number of seti instances to run simultaneously
# You would normally set this to be the number of physical processors in your system
nproc 1
# Max number of downloads to run in parallel
nxfer 3
# location of procN directories, defaults to current directory '.'
base_dir .
# Environment variable to set in the form VAR=value
# If you need a proxy set it here or delete this line
env HTTP_PROXY=some_proxy_host.domain
# Actions at startup ("load", "status" and/or "rescue")
# For the various actions see below in the notes section
start status,load
# Actions on receiving SIGHUP ("load", "status" or "load,status")
sighup load,status
# Actions on alarm timeout ("load", "status" or "load,status")
alarm status,load
# Actions on finishing a WU ("load" or "none")
finish load
# Timer interval (seconds)
timer 3600
# Delay after starting a transfer (seconds)
xfer_del 30
# Delay after starting a process (seconds)
process_del 5
# Name of file for logging transfers (or stdout or /dev/null)
xfer_log xfer.log
# Name of file for logging processing (or stdout or /dev/null)
process_log process.log
# Number of xfers which have to fail before we give up on a transfer attempt
max_fail 3
# Number of failed executions in a single directory before moving on
max_attempt 3
# Name of file where setimgr will store it's pid (or "none" to disable)
pid_file setimgr.pid
# Name of the SETI@Home binary, as run from the proc[n] directory
# You would only need to change this if you have several different machines
# sharing the same set of proc[n] directories which require different
# setiathome binaries (different architectures)
seti_binary ../setiathome
# Times that transfers are allowed - Default Day
# See notes below for examples
xfer_day_default y
# Times that transfers are allowed - Sunday
# xfer_day_sun
# Times that transfers are allowed - Monday
# xfer_day_mon
# Times that transfers are allowed - Tuesday
# xfer_day_tue
# Times that transfers are allowed - Wednesday
# xfer_day_wed
# Times that transfers are allowed - Thursday
# xfer_day_thu
# Times that transfers are allowed - Friday
# xfer_day_fri
# Times that transfers are allowed - Saturday
# xfer_day_sat

Notes:

Compilation options

There are 2 items which are configured in the source, which you might want to change before you compile.

MAXDIR - The max number of procN directories which setimgr will use (default 32)

LOCAL_TIME - If this is 1 (the default) status messages and the xfer time restrictions will use local time, if 0 or unset setimgr will use UTC.

PIDFILE This used to be a compilation option, but is now the config item pid_file in the config file, (use "none" if you don't want this feature).

How many proc directories ?

The basic plan is to create more procN directories than you have physical processors in your machine, if you have at least one more then you will be able to keep on processing while the upload/download of the previous WU is taking place.

If you have an intermittent connection (dialup etc.) create enough directories to provide enough work to last between dialups and allow for the occasional 'short' work unit eg. If you dial up about once a day and have a machine which takes ~8hours for a work unit, create four directories (proc0, proc1, proc2, proc3).

You may want more than this to allow for setiathome or your ISP having a fault. Having more than 1 weeks worth of WU's may not be worthwhile as results might be seen as stale when returned.

Settings for a typical dial up user

If you are on a dial-up connection you probably want the following config items:
start status            Print status when starting
sighup load,status      Down/up load WUs/results on receiving SIGHUP
alarm status            Print status on alarm (hourly)
finish none             Don't attempt down/up load when a WU finishes
Run the setimgr program and it should show you the status of the directories and run the setiathome processes. When you are dialed up send SIGHUP to the setimgr process:
kill -HUP `cat setimgr.pid`
or
setimgr -k hup
and it should attempt send any complete results and download new work units.

Settings for a user with a permanent connection

If you have a permanent internet connection, your setup should be similar but you probably want the following config items
start load,status       Print status and send/recv WUs when starting
sighup load,status      Down/up load WUs/results on receiving SIGHUP
alarm load,status       Print alarm hourly and send/recv WUs
finish load             Attempt down/up load when a WU finishes
Which should attempt to send units when complete and obtain new ones to keep the cache full.

The status messages

The program prints a report of jobs/started and completed, as well as an hourly status report of progress. The five possible statuses of each directory are:
None For some reason setimgr doesn't consider this a directory it can do anything with.
Run setimgr is running a setiathome -stop_after_process in this directory
Ready This directory contains a work unit which is ready to be processed
Xfer setimgr is running a setiathome -stop_after_xfer in this directory
Done This directory contains a result ready to be returned

If a directory in the state None has a valid user_info.sah then adding the rescue option to the start config item will cause setimgr to treat this directory as ready for download

xfer restrictions

See SETI@home bandwidth problems for the reasons why you should use this if you can.

If you want to restrict the times/days which setimgr uses for up/download then use the config items:

xfer_day_default
xfer_day_sun
xfer_day_mon
xfer_day_tue
xfer_day_wed
xfer_day_thu
xfer_day_fri
xfer_day_sat
Each represents a up/download allowed pattern for each day. If the pattern for a day is not set, it defaults to the default day.

The algorithm is simple, the string represents a complete day, if the character for the current time is a 'y' then up/download is allowed. eg.

xfer_day_default y
would allow up/downloads at any time (the default).
xfer_day_default ynnn
As the string is 4 characters, each character represents 6 hours (24/4), so up/downloads would be allowed 00:00:00->05:59:59, but not 06:00:00->23:59:59. If you want to allow on an hourly basis you would need a 24 character string. (The maximum length is 240 characters ie. 6 minute intervals).
xfer_day_default yn
xfer_day_sat	yyyyyn
xfer_day_sun	yny
Would allow transfers 00:00->11:59 on weekdays, 00:00->19:59 on Saturday and 00:00->07:59 and 18:00->23:59 on Sunday.

Setimgr does not wait for the start of a period, it carries on as normal with events when WU's finish and alarms arrive and only checks against the string when it wants to do an up/download. If the user sends SIGHUP to setimgr, the xfer restrictions are not checked.

With an hourly timer there is probably no point in having an xfer restriction with less than a 1 hour resolution (>24 char string).

Win32 Notes

I don't have a lot of experience with MS Windows, nor do I have a platform to test on, I managed to borrow an XP laptop for a couple of hours for testing. My guess is that this will work on all of the platforms which are supported by Cygwin, but I have no means to test this.

I compiled setimgr for MS Windows using the Cygwin cross compiler, running under Linux (RH 7.0). My Makefile looks like:

setimgr.exe: setimgr.c
        /usr/local/i386-pc-cygwin32/bin/i386-pc-cygwin32-gcc setimgr.c -o setimgr.exe
To install, you need put the setimgr.exe and setiathome.exe executables and the cygwin1.dll in your seti directory, and then create and populate the procN directories and create and edit the setimgr.conf file as above.

Running setimgr should process work units as normal.

There is only difference when compiled under cygwin is that the command line arguments to setiathome are slightly different (the UNIX version of setiathome understands -nice which is not there on the Win32 version). This assumes that the symbol WINNT is defined by the C pre-processor.

As there is not an equivalent of kill on MS Windows to send signals to setimgr, you will need to use setimgr -k hup to send SIGHUP, and setimgr -k term to send SIGTERM.

It is possible that having several setimgr's share the same set of (network shared) directories might work, it might even work if directories are shared from a UNIX box (with Samba?) to an MS Windows machine, or the other way around.

Warranty ?

NONE AT ALL

License text:

/*
 * This program may be freely used and distributed as modified or
 * unmodified in source or binary form provided that:
 * a) I am acknowledged as the copyright owner of the original source.
 * b) All source distributions contain this complete list of conditions
 *    copyright notice and disclaimer.
 * c) All distributions make it clear whether or not it is a version
 *    modified by someone other than myself.
 * d) All binary distributions make it clear that the source for the
 *    original unmodified version is available for free.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
 * THE AUTHOR BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
 * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF
 * OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 * SOFTWARE.
 *
 * Bernard Hatt - 11th February 2002
 * bmh at arkady dot demon dot co uk
 */

Example output

./setimgr
config: nproc = 1
config: putenv(HTTP_PROXY=local-proxy:80)
config: start = status,load
config: sighup = load,status
config: alarm = status,load
config: finish = load
config: timer = 3600
config: xfer_del = 10
config: process_del = 5
config: xfer_day_default = yynnyynn
config: xfer_day_sun = yynnnnnn
config: xfer_day_sat = yynnnnnn
./setimgr started pid=12926 at Tue Feb 12 19:18:20 2002
Found dir proc1, date=20020212.13:58:05 state=Ready
Found dir proc2, date=20020211.01:37:51 state=Ready
Found dir proc3, date=20020211.08:43:09 state=Ready
Found dir proc4, date=20020212.00:26:56 state=Ready
Found dir proc5, date=20020212.12:21:26 state=Ready
Found dir proc6, date=20020210.17:41:58 state=Ready
Found dir proc7, date=20020211.16:01:47 state=Ready
[20020212.19:18:20]:1:Ready(0.00%) 2:Ready(0.00%) 3:Ready(0.00%) 4:Ready(0.00%) 5:Ready(0.00%) 6:Ready(68.43%) 7:Ready(0.00%)
Started process pid=12927 in dir proc6 at Tue Feb 12 19:18:20 2002
[20020212.20:18:20]:1:Ready(0.00%) 2:Ready(0.00%) 3:Ready(0.00%) 4:Ready(0.00%) 5:Ready(0.00%) 6:Run(76.64%) 7:Ready(0.00%)
Finished process 12927 (dir proc6) exit(0) time 1h57m25s
Started process pid=13038 in dir proc2 at Tue Feb 12 21:15:45 2002
[20020212.21:18:20]:1:Ready(0.00%) 2:Run(1.07%) 3:Ready(0.00%) 4:Ready(0.00%) 5:Ready(0.00%) 6:Done 7:Ready(0.00%)
[20020212.22:18:20]:1:Ready(0.00%) 2:Run(15.87%) 3:Ready(0.00%) 4:Ready(0.00%) 5:Ready(0.00%) 6:Done 7:Ready(0.00%)
[20020212.23:18:20]:1:Ready(0.00%) 2:Run(27.89%) 3:Ready(0.00%) 4:Ready(0.00%) 5:Ready(0.00%) 6:Done 7:Ready(0.00%)
[20020213.00:18:20]:1:Ready(0.00%) 2:Run(39.95%) 3:Ready(0.00%) 4:Ready(0.00%) 5:Ready(0.00%) 6:Done 7:Ready(0.00%)
Started xfer pid=13081 in dir proc6 at Wed Feb 13 00:18:20 2002
Finished xfer 13081 (dir proc6) exit(0) time 0h01m40s
[20020213.01:18:20]:1:Ready(0.00%) 2:Run(50.50%) 3:Ready(0.00%) 4:Ready(0.00%) 5:Ready(0.00%) 6:Ready(0.00%) 7:Ready(0.00%)
[20020213.02:18:20]:1:Ready(0.00%) 2:Run(62.53%) 3:Ready(0.00%) 4:Ready(0.00%) 5:Ready(0.00%) 6:Ready(0.00%) 7:Ready(0.00%)
[20020213.03:18:20]:1:Ready(0.00%) 2:Run(74.32%) 3:Ready(0.00%) 4:Ready(0.00%) 5:Ready(0.00%) 6:Ready(0.00%) 7:Ready(0.00%)
[20020213.04:18:20]:1:Ready(0.00%) 2:Run(93.60%) 3:Ready(0.00%) 4:Ready(0.00%) 5:Ready(0.00%) 6:Ready(0.00%) 7:Ready(0.00%)
Finished process 13038 (dir proc2) exit(0) time 7h18m22s
Started process pid=13324 in dir proc3 at Wed Feb 13 04:34:07 2002
Started xfer pid=13325 in dir proc2 at Wed Feb 13 04:34:12 2002
Finished xfer 13325 (dir proc2) exit(0) time 0h00m42s
[20020213.05:18:20]:1:Ready(0.00%) 2:Ready(0.00%) 3:Run(13.82%) 4:Ready(0.00%) 5:Ready(0.00%) 6:Ready(0.00%) 7:Ready(0.00%)
[20020213.06:18:20]:1:Ready(0.00%) 2:Ready(0.00%) 3:Run(26.84%) 4:Ready(0.00%) 5:Ready(0.00%) 6:Ready(0.00%) 7:Ready(0.00%)
[20020213.07:18:20]:1:Ready(0.00%) 2:Ready(0.00%) 3:Run(39.97%) 4:Ready(0.00%) 5:Ready(0.00%) 6:Ready(0.00%) 7:Ready(0.00%)