I did some serious bug fixing and minor hacking to get pmake 2.1 to run as reliably as possible in the computing environment at ICSI (60+ Sparc machines, some old Sun3's, a NeXT, and a small cluster of DECstations). Recently there has also been extensive work porting to HP Snakes running HP-UX and SGI Iris. Platforms pmake/customs is known to run on (not necessarily tested for the most current version): Sun3, SunOS 3.5 and 4.1.1 Sun4, SunOS 4.1.1, 4.1.3, 4.1.4, 5.3, 5.4, 5.5, 5.6, 5.7 i386-PC, SunOS 5.5.1, 5.6 DECstation, Ultrix 4.0 and 4.3 NeXT, Mach Release 1.0 and 2.1 HP9000/700 (Snake), HP-UX 8.05, 8.07, and 9.01 SGI Iris, IRIX 4.0.1, 4.0.5F, 4.0.5H, 5.1.1, 5.2, 5.3, 6.2 DEC Alpha, OSF/1 V1.3, V2.0 MIPS M2000, RISC/os 4.52 (pmake only) BSDI BSD/386 1.1 i686-PC, Redhat Linux 2.0.32, 2.2.10 Major additions and changes include o In a typical NFS environment using an automounter, path names will not be identical all over the net, but one can set things up so everybody can access the same file using a `standard' path. customslib.c now has a function Customs_NormPath() that normalizes a pathname prior to exportation. This will have to be customized to the local environment. o The availability index computation was changed to be more realistic. Rather than combining all criteria into a magic number, we are only interested in the load and the relative speed of the machine as long as all other criteria are within their allowed ranges. The index is simply the load that is `left' (max - current), times the relative speed (say MIPS), divided by the number of currently imported jobs. o The os-dependent code was extended to support the new machines. It is now more robust in case some of the information can not be obtained (e.g., when debugging as non-root). Importquota shows only the criteria actually available to the system. o Several problems with byte-ordering surfaced as I ported the system to the DEC environment. These are believed to be all fixed now. o Code cleaned up in several places. Avoid conflicts with system header files and library functions. Exit with proper status. Make sure event handler functions get all the arguments they are supposed to. Customs and its clients (except pmake) now use xmalloc() instead of malloc() to avoid bad things in case virtual memory runs out. After all, customs runs as root so we better make sure there are no nasty surprises. o Pmake exit handling (lock file removal) didn't work on non-Sun systems. o A bad bug in the RPC system caused timeouts on long messages sent via TCP. o A work-around that really shouldn't be necessary to avoid losing stderr output from remote jobs on non-Sun systems. o Logd didn't work at all because of an incompatible data structure for the information given out by the MCAInfo RPC. o Reginfo got a nicer output format. o Makefiles were improved (but are still dependent on the compilation and installation environment). Compiler and options are now selected to be `best' for each machine. o Gratuitous incompatibilties with standard makes were removed (see patchlevel 12 below). o The notion of host architecture has been generalized to arbitrary "host attribute" strings and can be used to describe host resources. Both export and pmake support restricting the choice of remote host using combinations of attributes. For more details see the man pages, and the etc/ATTRIBUTES file for an overview. Details of all these changes can be found below. Andreas Stolcke stolcke@icsi.berkeley.edu 7/31/91 *** Patchlevel 4 *** o customs.conf script to generate startup flags automatically. Very site and machine specific, though. o Guard against process table overflow on target host using a new availability parameter `minimum no. of free process slots'. o Propagate process priorities and resource limits to exported job. o Implement CPU time limits for imported jobs. o Fixed SIGINT and SIGQUIT passing to remote jobs. o Allow remote killing of all imported jobs (cctrl -kill). o Check passwd entry before allowing job imports. A. Stolcke 2/18/92 *** Patchlevel 6 *** o -nice options for customs and importquota to run all imported jobs at lower priority. o The pmake.1 man page is now generated with Sprite man macros inserted automatically. o -idle and -cpu flags now allow time specs greater than 60 minutes. o New -help, -quiet and -default options for importquota. o Made malloc() used by liblst more secure (like xmalloc()). o Found and fixed one tricky source of core dumps in pmake. o Fixed kernel symbol names so os-bsd.c works on HP/UX. o Pmake support for Common Archive Format (e.g., HP/UX) added (see AR_COMMON in unix/config.h). o Pmake interprets negative -L arguments as preference for local execution. Can be used to force a certain number of local jobs. o Fixed handling of SIGTTIN/SIGTTOU in export. o Support active jobs eviction added (-evict flag). o Improved tracking of job status (stopped, signalled, continued) in pmake, including remote jobs. o Fixed a bug with bogus host handling in customs that could cause reginfo to core dump. o Fake a non-zero load value if load cannot be obtained from OS. Results in better spread of allocated hosts. o Can now deliver uncatchable signals (SIGCONT, SIGKILL) to remote jobs without killing customs child daemon (so we get status change reported back). o Allow for tighter security by restricting critical RPCs to reserved ports (see remarks in SECURITY file). A. Stolcke, 4/11/92 *** Patchlevel 7 *** o Added Customs_Port() to customslib for gnumake. o Use caller address to authenticate a Import_Kill. This allows export in slave mode to send signals. o Make Customs_NormPath() a noop if ORIG_CWD is defined (so can be called regardless of local ORIG_CWD setting). o Include patch for GNU make 3.62 Customs support. Also link export statically on Suns to speed things up a bit. o Use OPTIMIZE macro to customize CFLAGS in Makefiles (see config.mk). o Avoid problems with desynchronized clocks by transmitting elapsed job run time instead of start time to cctrl -jobs. o Fixed some inaccuracies and omissions in the INSTALL instructions. A. Stolcke, 4/21/92 *** Patchlevel 8 *** o All customs error logging done via xlog(). Allows routing to file or syslog(3). A. Stolcke, 7/1/92 *** Patchlevel 9 *** NOTE: The first four fixes below address various problems that have caused seemingly random failures, hanging, and core dumps of PMake under typically heavy load. Taken together, they have significantly increased the robustness of Pmake in my experience. o PMake: Fixed some bugs with tracking of remote jobs: queue corruption could lead to core dumps, "exits from unknown jobs". Also added new queue debugging mode (-d q) to help with future trouble. o Customs: Doublecheck import timeouts on the server side to prevent duplicate job creation and other bad things. Cf. DOUBLECHECK_TIMEOUT in config.h. NOTE: This new feature uses an incompatible change in the import protocol. Both customs and client programs have to be reinstalled and restarted. o PMake: Check that pipe() doesn't run out of file descriptors. Can result in major screwup in RPC module when handling remote streams! o Fix fileno() macro to prevent lossage with file numbers > 127. Cf. redefinition at the bottom of unix/config.h. o PMake: Fixed a bug with suid-ness that would initialize the directory cache before reverting to user's uid. o Customs: Cleaned up some of the syslogging related to imports. More info on security breaches. Log common, unavoidable errors as `warnings'. o Customs: For ps(1)'s benefit, set process title of child daemon to job id. o Customs: Detect and warn about reginfo exceeding max RPC size. o RPC: Check socket arguments for sanity. A. Stolcke, 7/23/92 *** Patchlevel 10 *** o Customs: log evictions. o Customs: kill orphans in process group of imported job. Also counteracts deliberate evasion of the eviction mechanism. o Reginfo: allow hostname specification other than localhost, so other customs networks can be monitored remotely. o PMake: allow runtime control over rechecking strategy (new -R option). Be smart and recheck only locally created targets by default. o PMake: enable -x option (used to be ignored, even in Make mode where it is useful). o PMake: prevent endless loops when -L 0 is specified and nothing can be exported. A. Stolcke, 7/30/92 *** Patchlevel 11 *** o RPC: make sure Rpc_Return() byte swaps its own data, not the caller's. This could mess up customs on certain RPC calls from machines with different byte order. o Customs: catch and log termination signals. o Cctrl: new -port options to select non-default port/service. A. Stolcke, 7/31/92 *** Patchlevel 12 *** o Reginfo: less redundant, columnized output. o Customs: link with -n to free up binary on diskless clients. o Need -DSYSV in hpux CFLAGS in makefile. o Customs: Now has full support for preserving resource limits in HP-UX. (Not very useful unless you have a shell that supports them too.) o Logd: man page added courtesy of . o PMake: Fixed glitch in HP-UX setuid() use that prevented uid switching from working when installed suid root. o PMake (system mk file): Use /bin/sh5 on ultrix system, where /bin/sh error control is broken. The following are small compatibility fixes so as to make PMake more like UNIX make and/or gnumake in cases where no conflict with PMake enhancements ensues. o PMake: Ignore all dot targets when searching for a default target. A dot target is one starting with `.' and not containing a `/'. This provides compatibility with other makes, especially when it comes to these the GNU makefiles with .PHONY at the top ... Previously only dot targets with a built-in special semantics were treated this way. o PMake: Strip initial ./ path components for purposes of target identification. o PMake: Let empty command disable transformation rule application to target (like in standard make and gnumake). o PMake: Don't automatically consider an archive file out-of-data just because if has no table of contents. o PMake: Accept options, variable assignments and targets on command line in any order. A. Stolcke, 11/3/92 *** Patchlevel 13 *** o Rpc: Send 'Access denied' error on non-local calls and report procedure number called. Allow non-local replies to be able to get the errors. o Rpc: Added workaround for systems where broadcasts don't reach the local host (like apparently HP-UX 8.x Snakes). Also fixed a memory allocation bug triggered by broadcasts on multiple networks. o Customs: Use undocumented pstat() call to get SETPROCTITLE working with HP-UX. o Customs: Avoid spurious 'downs' by setting the down event to one second after the next availability sampling. o Customs/Reginfo: Display the value of criterion that causes host to be unavailable, as well as down time. Also rearranged the order in which criteria are check to that most informative ones are `prefered' in justifying unavailability. o Customs: Modified the handling of cctrl -elect so that the target host will become new master with high probability. o On Ultrix systems use replacement for non-informative gethostid(). o Removed more ANSI nits so compiles cleanly with cc -Aa on HP-UX (except for wait() incompatibilities). A. Stolcke, 11/16/92 *** Patchlevel 14 *** o PMake: Added failure reason (errno) to various error messages. o PMake: Added convenience for root invocations: avoid export attempts and ignore lock file creation failure. Also don't create lock files in no-execute mode (-n). o PMake: Make rules for archive members behave in a more standard way by fixing the shorthands $% and $@ (for $(.MEMBER) and $(.ARCHIVE), respectively). o PMake: Now handles computed variable assignments ($(VAR) = foo) and substitutions ($($(VAR))). o PMake: Implemented the .CURDIR and MAKEOBJDIR variables found in 4.4BSD make (a derivative of pmake). Should be able to get reasonable BSD compatibility by defining SPECIAL_CHAR to '.'. o PMake: Force update on archive members to ensure library gets rebuilt after updating the external member file. Otherwise the MAKELIB rule will fail to do its thing under RECHECK operation. o PMake: Catch and warn about cycles in suffix dependencies. Used to leave pmake in infinite loops. o Customs: Fixed a potential bug in election protocol. Also modified handling of multiple master detection so as to make one of the existing master likely to stay in charge. This greatly reduces the broadcast storm formerly observed after rejoining partitioned networks, and speeds up resettling of the customs network. o Customs: Fixed a bug in the allocation procedure that could produce inaccurate records of availability on the master. The typical symptom was over-allocating jobs when the cluster runs out of available hosts. o Customs/Cctrl: List export permits issued in -jobs list (since these also count towards the jobs total). o Makefile: removed some unwanted files from distribution o gnumake: Some minor additions to the patch to make it work. Also added an sed script that will convert some of the gnumake directives (includes, conditionals) to pmake syntax. A. Stolcke, 1/22/93 *** Patchlevel 15 *** o gnumake: Included patch for GNU make-3.63. This patchlevel also incorporates all the differences found between the original pmake release of Sep 19, 1990 (on which these versions are based) and the newer release of May 6, 1992 from ginger.berkeley.edu. o Customs: Only daemons started with the -master flag are eligible for master duties. This is mainly a workaround for a poorly understood bug in the election protocol that fails to determined a master if a larger number of hosts are involved. It is also convenient to prevent certain (e.g., less powerful) machines from being recruited as masters. o Customs: A bug fix that should prevent customs restarts to fail because a socket address is still in use. o Customs: Under SunOS 4.x, include reserved pages in swap usage. o Rpc: Bug fix for reentrant Rpc_Wait calls. o PMake: Miscellaneous bug fixes. o Prefix: As before, this part of the distribution is included unchanged. A. Stolcke, 1/22/93 *** Patchlevel 16 *** Mainly lots of fixes to the Makefile parser. o PMake: Some fixes to parsing for compatibility with 4.4BSD make. Now warns about bad #include syntax, supports .OPTIONAL as a synonym for .DONTCARE, and should work correctly when SPECIAL_CHAR != '#'. o PMake: The # character for directives is now settable at run-time (-Z option). SPECIAL_CHAR provides the default. o PMake: Lines of whitespace (that don't start with a tab) are skipped just like empty lines, instead of triggering a 'needs a leading tab' warning and generating an empty command. o PMake: Variable assignment that start with a tab and don't occur in a dependency context are accepted with a warning (instead of generating a 'unattached command' fatal error). o PMake: Improved error detection in variable modifier parsing. In particular, no more infinite loops on missing right paren. o PMake: -i and .IGNORE now also work for jobs killed by signals (used to work in compat mode only). o PMake: Messages and diagnostics cleaned up and made more consistent. Error messages and the ones starting with '***' are now written to stderr rather than stdout. So is debugging output (except for -p output). o PMake: Added support for the SHELL variable as an alternate way of specifying the shell for command execution. SHELL settings are kept consistent with .SHELL targets and vice-versa. o PMake: SHELL or .SHELL now also affects both compatibility mode command execution and != assignments. #undef'ing SHELL or .SHELL: without dependencies reverts to default shell. o PMake: Fixed two bugs in := assignments: right-hand side used to ignore command line variables and left undefined variables unexpanded. o PMake: Archives without table of contents revisited: only consider them out-of-date if they depend on a member. o PMake: Include version info and exportation document string in pmake -h output. o tests: updated README file and included two new test for environment variables and SHELL. o PMake: man page updated (and some previous omissions fixed). o Reginfo: truncate hostnames to fit column format. o Makefiles: the -C flag is no longer required on the command line to install pmake. A. Stolcke, 1/29/93 *** Patchlevel 17 *** o PMake: Various fixes to -t, -q, -n, -s option processing. Execute -q on .MAKE targets recursively in PMake mode. Execute -n, -t recursively with commands that expand the MAKE variable in compat mode. o PMake: Fix to deal with NeXT's sorted ranlib tables. o PMake: Fixed command line parsing bug that could lead to infinite loops on machines where char is unsigned. o lib/mk: Cleaned up the way a system specific makefile is included. Tries to use the output of /bin/machine or similar to determine host envronment. Also eliminated the pmake define; use .PMAKE instead to #ifdef whether a Makefile is processed by pmake. o Customs: Added ifdefs for SGI port. o Customs/Reginfo/Export: Added support for general host attributes that may be set for customs as run-time, and which can be requested for exported jobs. o Cctrl: New -net option allows restricting broadcasts to a single customs network. o PMake: Support host attributes with .EXPORT= source targets. The sources of .EXPORT are used to specify globally required attributes. o PMake: Added .RESTART attribute to help in the control of long- running, continuable computations. Also automatically restart customs jobs that were evicted from their hosts (terminated with SIGXCPU). o Tests: Included sample for use of .RESTART and .EXPORT=, see test/count and test/count.mk. o Customs: Don't restart while jobs are running, return "system error" to caller. o Export/Customs: -force flag lets you export dispite availability. Use judiciously. o Customs: Fixed a bug whereby the host allocation routine could go into an infinite loop if allocations where denied by the importing host. o Reginfo/Customs: Use TCP for CUSTOMS_INFO call (has large return data). NOTE: I changed the optimization for gcc (version 2.3.3 at ICSI) from -O2 to -O. It would screw up the inlining of JobCanRetry in job.c. A. Stolcke, 2/11/93 *** Patchlevel 18 *** o Customs/Cctrl: Added simple statistics collection for imported jobs (total number, run times, etc.). Accessed with cctrl -stats. o Cctrl: Added printing of summaries (number of hosts responding etc.) for broadcasts (-all, -net). o Customs: Normalize signal number for network transmission to allow interoperation between BSD and SYSV systems. Also fixed the byte swapping of exit statuses that was broken in one of the early patches. o Customs: Bug fixes to SGI/mips load average reading, and memory mis-management in getusershell(). o Customs: Added /etc/utmp reading for better idle time support on system without /dev/kbd or equivalent. o Customs: Warn about illegal default availability criteria (compiled in or defined on command line). o Customs/Rpc: Use own extended Rpc_IsLocal() to obtain local inet address (instead of get_myaddress()). Allows error checking and guarantees that Local(&localAddr) is actually true. o Cctrl: Accept symbolic signal names as arguments to -kill. o PMake: Fixed a bug in the cycle printing routine that could lead to endless loops. o PMake: Fixed suid handling for IRIX. o system.mk: Added rule for .c -> .i (preprocessor output). Useful for debugging macros expansions etc. o Customslib: Don't normalize a local cwd that point to /tmp, /usr/tmp or /var/tmp. Just use the remote /tmp. Rationale: /tmp is only locally accessible anyway, so we might just be testing, for which it is convenient. o Removed some nits to make Sun's acc happy. o Documentation: closed some gaps in INSTALL and customs/README. Also included FAQ, a set of questions & answers compiled by gary@chinook.atd.ucar.EDU in the course of his attempts to get customs running. NOTE: If customs aborts soon after startup with the message "No local network address found", the problem may be with you compiler and how it expands certain ioctl constants. For example, gcc needs to have fixincludes run before it can do this correctly. A. Stolcke, 4/15/93 *** Patchlevel 19 *** o PMake: Extension of VPATH semantics: even non-source targets are found there. This seems to be what Sun, NeXT, and GNU Make are all doing, and recent GNU distributions seem to rely on it. o PMake: Fixed processing of System V-style 'F' and 'D' modified variables, so they work with PMake-style modifiers and '=' substitution. o PMake: Fixed bug whereby non-existent libs without dependencies were considered up-to-date. o PMake: Avoid running out of file descriptors when jobs are deferred. Also, try to resume deferred jobs as soon as any other job finishes. o PMake: Fix to handling of USELOCAL attribute, so it works when jobs as restarted. o PMake: Plugged file descriptor leak opened by exporting in -B mode. o PMake: Fixed the mechanism that forces a parent to be remade when a FORCE-type dummy child exists. Previously this could fail under rare circumstances. o PMake: Make sure jobs that are evicted or terminate under .RESTART are restarted immediately, without waiting for the next job to finish. o PMake: Fixed documentation that wrongly claimed that $* expands to filenames with directory removed. (It shouldn't, first, to be compatible with standard make, and second, since the :T modifier can be used to achieve that effect). o Reginfo: Added total count for available, but restricted hosts. o Reginfo: New -hosts flag to print just hostnames. o Customs: Also check /dev/mouse for idle time on Suns (patch thanks to Teemu Torma ). o Customs: Added note about Customs_NormPath configuration to config.h. o Customs/Reginfo: New -server flag to tie availability to server being up. o customs/libcustoms.a: This new library contains all objects needed for linking third-party customs client programs (such as gnumake). make install copies the library and required header files to system directories. o gnumake: Included patches for GNU make-3.64, 3.66, and 3.68 (they have their releases coming out fast, don't they :-). NOTE: On Suns and others, make-3.68 configure fails to include HAVE_SETREUID in config.h, which is needed for suid-ness to work! o Added tags files and makefile targets everywhere. o Moved miscellaneous files to the new etc/ subdirectory. o Minor tweaks for HP-UX 9.01 port. A. Stolcke 8/16/93 *** Patchlevel 20 *** o Rpc: Rpc_UdpCreate() always tries binding to reserved udp port. Use bindresvport() to avoid back-to-back use of same port by different invocations. o Rpc/Customs: Replaced the static LOCAL_NETS access control list with a dynamic interface to allow and deny host or network access to RPC system. See new -allow and -deny customs options. A default access list suitable for most cases is obtained automatically by inspecting the local network interfaces. (This also means that customs/config.h will rarely need any configuration now since LOCAL_NETS was often the only thing requiring editing.) o Customs: Added provisions for correcting load average upward when starting jobs, and downward when jobs finish. Should prevent double allocations and encourage repeated allocation after a machine becomes free. o Customs: By popular demand, allow attribute matching including negation and disjunction. Boolean combinations can be specified in disjunctive normal form: A string starting with ``!'' means the following attribute name must *not* be present. The attribute ``OR'' separates disjuncts. See etc/ATTRIBUTES for examples and some caveats. o Customs: Avoid availability index 0 values by requiring that load be strictly lower than max. o Customs: Added OS module for SunOS 5.x. o Customs: SIGHUP triggers restart (like cctrl -restart) instead of aborting. o Customs: Authenticate the CUSTOMS_PGRP child-to-parent RPC call to prevent forgeries. o Customs: The default MIN_IMPORTS value is now -1. This means that importquota -jobs -1 can be used to disable customs imports to a machine. (Note that -jobs 0 would simply remove the number of jobs as an availability criterion.) o Customs: Don't send EOF on tcp socket when process is just stopped, not terminated. o Customs: Added a generic os-default module that should allow rudimentary support for generic UNIX platforms. It can serve as a template in writing platform-specific versions. o Customs/Customslib: Make sure that inet addresses returned by various RPCs are valid for the caller, which can be a problem with multi-homed machines. A distinguished return value tells the caller to use as a result whatever address it used to reach the callee. Note: This make new customs agents and client progams incompatible with old ones, so everything has to be restarted. o Customs: Delay reply to a CUSTOMS_IMPORT call until after the environment for the user process has been fully set up. This allows errors to be returned to the caller (instead of simply causing termination of the job). In particular, this avoids imports hanging indefinitely on the chdir() call. o PMake: Be more careful with removing interrupted targets. Only do it if file is not a directory and was modified since the make started. o PMake: Identify remote host in banner lines printed ahead of remote output (in pipe mode). o PMake: Make it easier to identify shells spawned in pmake mode by including `(pmake)' in the zeroth argument. Can be disabled by #undefing MARK_SHELL_ARG0 in unix/config.h. o PMake: Archives in several formats (BSD ranlib'ed, Common archive, MIPS ranlib'ed) are now identified at runtime and (hopefully) handled correctly on all platforms, except for possible ranlib incompatibilities due to byte order and word size differences. Symbol table lookup still doesn't work on HP Snake systems, because, as all of that system, it's a funny mess: it uses an incomplete common archive format, with an undocumented symbol table format. o PMake: Fixed handling of '-' and '@' command modifiers so that they may appear interspersed with whitespace. o PMake: Handle System V makefile includes (enabled by SYSVINCLUDE in unix/config.h). This is an improved version of a pmake enhancement by Christos Zoulas (christos@deshaw.com). o PMake: Under certain circumstances a file specified with a relative path would be found relative to VPATH even though it was accessible from the current directory. The current dir now always takes precedence. o PMake: Explicit dependencies override implicit ones even in -M mode. This makes -M behavior consistent with native Pmake and all other makes I have access to. o PMake: Ensure makefiles read and included are not directories. Print debugging info about files read with -dp. o PMake: Fixed != variable assignment to accept arbitrary amounts of output from shell command. o PMake: The special .NULL and .EXPORT targets now have a meaning when given without sources. .NULL cancels the use of implicit null suffixes, whereas .EXPORT resets global export flags and attributes. o PMake: Fixed various inconsistencies and bugs with the setting of the .PREFIX ($*) variable. It now always includes the full path of the target being created, consistent with other makes (but contrary to the PMake tutorial). Note that the filename part can always be gotten using the :T modifier. There also was a genuine bug which led to the stripping of the .NULL suffix even if it wasn't textually part of the target's name. o PMake: Use the full path to the archive in $(.ARCHIVE). This is consistent with other dynamic variables and fixes various problems: libfoo.a(member) and -lfoo(member) now work as expected when the library is somewhere on the search path. Delay the archive symbol lookup until dependency checking phase, so path searches etc. can be used, and don't abort if the symbol is not found: simply complain about not knowing how to make the archive member. o PMake/Export: To increase reliability with long-running jobs, remote processes are now checked on at regular intervals to detect crashed machines etc (cf. REMOTE_CHECK_INTERVAL in customs/config.h). Also, jobs that are detected as lost this way are restarted as if evicted. o PMake/Export: Wait for EOF on remote output before terminating. Otherwise, depending on a race between the tcp stdio connection and the udp CUSTOMS_EXIT RPC the last bit of output could be lost. Also eliminated use of FIONREAD ioctl, which happens to hang Solaris 2.3. o PMake: Retry to export jobs that couldn't be exported the first time, at regular intervals (EXPORT_RETRY in customs/config.h). The previous strategy would only retry exports when some other job exits, which can waste a lot of time if jobs are long-running. (This generalizes a previous solution for jobs that could *only* be run remotely.) o PMake: Don't give up when running out of file descriptors. Lower the number of parallel jobs and continue. o PMake: Fixed bug whereby remote jobs could be restarted locally, which currently leads to their output being lost due to the way file descriptors are managed. o PMake: Fixed handling of a race condition between sending remote signals and registering remote exits, manifested as core dumps, waiting for non-existing jobs, and what not. o Customslib: When normalizing path names for export, leave NFS filesystems alone if they don't have the automounter prefix (suggested by pomeranz@aqm.com (Hal Pomeranz)). The default heuristic used by Customs_NormPath() thus becomes: 1) if path is in /tmp, /usr/tmp, or /var/tmp, leave it alone 2) if /tmp_mnt/..., remove /tmp_mnt/ from path 3) if /n/... or an NFS mount, do nothing 4) add /n/machinename/ and return The /tmp_mnt/ and /n/ prefixes can be configured by defining TMPMNT and NETMNT, respectively. o Customslib: Added convenience functions: Customs_Hostname() and Customs_ErrorMessage(). pmake and export changed to use these and give more consistent error diagnostics. o Customslib: Split client stubs over separate source files to get finer granularity during linking. Also allows the server to use routines from the library. o Rpc: Changed the fallback broadcast address from INADDR_ANY to INADDR_BROADCAST. (This only affects network interfaces who aren't able to report their broadcast address.) o Rpc: Added a special error code RPC_NOREPLY, to disable replies and acks on a given RPC. Useful to transfer responsibility for a reply to a child process. o Logd: Works again -- fixed info buffer parsing. o Makefiles: Create all installation directories if necessary. Handle build with MAKEOBJDIR set to put objects in target-specific subdirectories. The top-level Makefile sets MAKEOBJDIR to default values based on the system type. o libsprite: On second build, only include functions not in system libc (except for getopt, which is often broken). o etc/customs.conf: Added support for HP Snake (9000/700 series). o Customs, Export, PMake: Fixed to reinstall signal handlers on SYSV systems, so now working correctly on multiple suspend/continue etc. Cannot believe I had overlooked this until now! o Reginfo: Added -rate and -arch options to provide more selective listings. o Lots of small tweaks for SunOS 5.3 compile. Try to use POSIX interfaces where possible, and check available features by #ifdef'ing on the symbols belonging to an interface. o Lots of major tweaks for OSF/Alpha port. All data shipped over the net is now defined in terms of 'int16' and 'int32' types, defined in rpc.h. Also fixed a bug in WayBill swapping that caused the the import server to abort whenever one tried to `export pwd' between different endian machines. o Included our customs-aware version of xlock in the distribution. o BSD/386 port. This includes handling of variable-length struct sockaddr in rpc.c, enabled by -DHAVE_SA_LEN. Known problem: /bin/sh does weird input buffering, defeating the error and echo control scheme of pmake in native mode (not -M). Causes occasional "set" commands to be echoed. o Revised configuration (and INSTALL directions) slighly so that the system include directory can be edited in the top-level makefile (and config.mk) and will propagate to the compile. This means that unix/config.h should rarely have to be changed at all. o Revised PMake tutorial (doc/tutorial.psc) to reflect ICSI these CHANGES. A. Stolcke 6/23/94 *** Patchlevel 21 *** o customs/config.h: put MAX_DATA_SIZE override here and increase to 16K bytes. o Rpc: Handle incoming RPCs that exceed MAX_DATA_SIZE with RPC_TOOBIG. o Rpc: Handle RPCs of any size. By original design, return data is still limited to caller's preallocated buffer, though. o Rpc: Event created by Rpc_EventOnce is now freed automatically after being triggered. (Fixes minor memory leak in customs.) o Customs: some tweaks to the election/mca code that will hopefully avoid recurring "registering with new master: No such procedure" messages. o Customs: check clock synchrony between master and slave if DOUBLECHECK_TIMEOUT is in effect. This avoids trying to export to hosts whose clock is badly off and prevents timeout of the actual job import call. Reginfo will show the condition, thus helping to detect the troubled host (always assuming the master system clock is suitably synchronized). o Customs: handle certain out-of-memory cases with error returns instead of aborting the daemon. o Customs: workaround for "Address in use" startup error when trying to allocate customs service ports. See WAIT_FOR_FREE_ADDRESS in customs/config.h. o customslib: use CUSTOMS_NOCWD error code to indicate failed getcwd() or path normalization. Don't print error messages in Customs_NormPath, set errno instead. o customslib: guard against waybill size overflow due to large command and/or environment. o customslib: better check whether the current directory is NFS mounted (for path normalization purposes). o PMake: Fixed handling of the out-of-filedescriptors condition: lower paralellism to total number of jobs, waiting or running. o PMake: Plugged numerous memory leaks and arranged for per-target memory to be deallocated when target is done. Avoids pmake growing during excution phase. o PMake: Allow exported jobs to be restarted on local host. o PMake: More fixes to file descriptor handling for restartable jobs. o PMake: .EXPORT= attributes on specific targets no longer implies .EXPORT for that target (i.e., may be run locally if host satisfies attributes). o PMake: More efficient and robust error and echo control in -B mode. o PMake: -M mode tolerates dependency cycles, by ignoring the source closing the cycle. o PMake: Now uses SIGHLD to be notified of child status changes instead of polling. Dramatically reduces latency for short-lived local jobs. o PMake: Fixed archive symbol table lookup on Alphas (apparently use 32bit integer format for backwards compatibility). o PMake: Fixed interrupt handling to react promptly during dependency processing phase in no-execute modes (-q, -n, -t). o PMake: Workaround to let suid-ness of pmake binary be effective with executable pmake scripts (so exporting will work). Works only on systems that stick the full path in argv[0] on interpreter invocation (including Suns, but not Ultrix, IRIX, NeXT). o SunOS5.x optimizations: use vfork() while it's still there, 'cause it's a lot faster. Use statically linked /sbin/sh for SHELL 'cause it's a lot quicker to start up. A. Stolcke 9/27/94 *** Patchlevel 22 *** o export: avoid static memory free when -attr is used. o export, pmake: make sure remote exit status can be relayed even if job is running on non-local network. o RPC: change interface to Rpc_Allow/Rpc_Deny, added Rpc_IsAllowed. o customs, cctrl: added remote access list manipulation RPCs. A. Stolcke 10/1/94 *** Patchlevel 23 *** o PMake: avoid rare hanging, either due to missing wait() on children or blocking on pipe reads. o Customs, Export, PMake: fixed problem with unlimited resource values transmited between machines of different wordsizes. Added support for RLIMIT_VMEM (virtual memory limit). o Customs: New compile-time confuration option to allow any user to simulate evictions (ALLOW_EVICT_BY_USER). etc/cevict is a sample wrapper script using this "feature." o Customs: Revamped the idle time criterion to be more general and configurable at run-time. Check the -idlecrit option in customs and importquota. o Customs: Fixed various byte swapping glitches. o Reginfo: Added additional flags for host selection: -avail, -up, -down. Fixed column alignment for large INDEX values. NOTE: Protocol changes -- requires recompiling and restarting all daemons. A. Stolcke 11/24/94 *** Patchlevel 24 *** o Customs: Fixed bug with rlimit mapping that caused FSIZE limit to override CPU limit. o Customs: Added -memory limit on imported jobs. Rewrote descriptions of various import policies in man page. o Customs: When exporting from a machine withouth file descriptor resource limit, set the limit value to a finite value (1024). An infinite value may break things at the other end (e.g., /bin/csh in SunOS5.3 goes into an quasi-infinite loop trying to close all descriptor up to RLIM_INFINITY). o Customs: Update a machine's hostname when it changes and the machine re-registers with the master. o Customs: Evict jobs before daemon aborts. There is no eviction notice or grace period in this case, but at least pmake will restart them. A. Stolcke 1/10/95 *** Patchlevel 25 *** o lib/mk/sys-iris.mk: pmake -DSGI_IRIX4 generates binaries using the /usr/irix4 compatibility tools. o etc/gnumake: latest patch fixes problem whereby subsequent commands in a multi-command target would be exported even though no local command was running. o export: removed a redundant getservbyname() lookup that would often lead to NIS lossage under load. Identify error messages as coming from export. o RPC: Fixed typo that had effectively disabled the signal protection around event handling code. (Amazing that this apparently isn't necessary in practice.) o RPC: Use system-defined default FD_SETSIZE, so RPC module can make use of the larger ranges typically available on newer systems. Made fd_set handling more efficient by keeping track of the largest file descriptor in use. o RPC: Added a workaround for broken select() in IRIX 5.2. The symptom of this was that export would go into a tight loop instead of waiting for input. [The underlying bug is select() sets fd_set bits not requested by the caller.] o RPC: Added workaround for broken recvmsg() in SunOS5.4. The symptom were timeouts when the amount of information shipped in the CUSTOMS_IMPORT call exceeded approx. 2900 bytes. [The underlying bug is that recvmsg() on TCP streams discards the data from a message not fitting in the buffer passed to it. The workaround is to use read() on those streams instead.] o RPC: Plugged a potential file descriptor leak triggered by TCP RPC timeouts. [This was never an issue until the above problem came up.] o Customs: use local IP number for random seed computation, eliminating need for non-portable gethostid(). o Customs: Use ulimit() systems call for file size and descriptor limits on systems without getrlimit(). o Customs: Avoid core dump on cctrl -abort. o Customs, Importquota: Added support for SGI's non-degrading priorities. Check -npri flags. Also allows customs to be run under npri(1), since priority will be reset for user jobs. Using -npri, -cpu, -memory on systems that don't support them now generates diagnostics. o Customs: Changed rating formula to give multiprocessors same value as uniprocessors as long as they both have at least one CPU free. o Reginfo: Added -master flag. o Cctrl: Give preference to hostnames over network names when resolving non-numeric addresses. o PMake: Make sure the saved uid/gid also gets reset prior to exec-ing a command. Failure to do so causes bad lossage with /bin/csh scripts in SunOS5.x. o Customs, Export, PMake: Added another export option to request exclusive use of a machine, i.e., prevents multiple jobs on a single host. This is achieved with `export -exclusive' or `.EXPORT: EXCLUSIVE' (in pmake). o Cleaned up signal handling to use either BSD or POSIX interface (but not SYSV or a mixture of the above). Eliminates potential race conditions arising from old SYSV signal. o xautolock: Included modified version of this X11 idletime watcher. The program has been enhanced to update the access time of files in a manner similar to Sun's /dev/kbd or /dev/mouse. This in turn can be used to get accurate idle time values for customs purposes. A. Stolcke 2/18/95 *** Patchlevel 26 *** o RPC: Detect bogus message headers by checking magic numbers. o RPC: Prevent freeing of RPC cache entries while a handler procedure is running. This manifested itself as 'No reply sent' error messages and eventual pmake core dumps under heavy network load/delay conditions. o PMake: Prevent failure when pmake binary is both setuid and setgid (although setuid is all one should ever want). o PMake: Generalized the code that searches for libraries specified with -l . Now uses all the suffixes declared with .LIBS to locate the libraries. For example, SVR4 platforms have libraries ending in either .a or .so . o PMake: No longer silently ignores out-of-date or missing libraries given as dependencies. Only -l libraries are ignored if their full paths cannot be found. o PMake: Locate system makefiles only on system include path, not in general makefile search directories. Also allow MAKESYSPATH environment variable to override compiled-in system include path (suggested by someone at DEC-OSF). o PMake: Fixed exit code in compatibility mode to conform to standard (same a PMake mode): 1 for out-of-date, 2 for error or signal, and 0 if all is well. o PMake: various little bugs fixed that were pointed out by DEC-OSF folks, including a possible buffer overrun in handling long archive member specifications. o PMake: Eliminated POSIX compile-time option. The changes in the parser triggered by it are obsolete, and miscellaneous other effects are integrated in the default behavior. Most notably, pmake now interprets and sets both PMAKE and MAKEFLAGS environment variables. MAKEFLAGS is set to contain only the standard subset of flags understood by most make programs. This should allow interoperability with other make versions, such as when 'make' is called directly in a makefile. o xautolock: Upgraded to patchlevel 10. A. Stolcke 3/10/95 *** Patchlevel 27 *** o PMake: Fixed dynamic variable expansion so dependencies of suffix rules can use them correctly. o Customs: Prevent lossage when calling signal handlers before initialization. o RPC: Renamed int32, int16, etc. to Rpc_Long, Rpc_Short, .... Avoids conflict with vendor typedefs and is more consistent with general module naming conventions. o etc/gnumake: added patch for 3.73 code. A. Stolcke 4/22/95 *** Patchlevel 28 *** o Man pages: documented customs -bias flag. o lsf: (new) Simple pmake port to the LSF Load Sharing Facility. o xautolock: fixed memory corruption error that would stop updates from working in an apparently random fashion (or worse). o etc/gnumake: added patch for 3.74 code. o etc/customs.conf: various additions. A. Stolcke 8/20/95 *** Patchlevel 29 *** o RPC: fixed bad format string that could lead to core dump in rpc-debugging mode. o Customs: avoid sending OOB message on process exit: seems unnecessary and avoids 10 second client delay on Solaris hosts. A. Stolcke 11/17/95 *** Patchlevel 30 *** o More general fixes for 64 bit systems. o Customs: IRIX64 support. o Customs: Portable swap usage computation using swapctl(2). A. Stolcke 12/31/95 *** Patchlevel 31 *** o PMake: Report location of unclosed conditionals correctly. o Customs: Fixed -version message. o etc/customs.conf: Misc. changes to support new UltraSPARC systems. o etc/newvers.sh: Deal with multiple domain specs (use last one). A. Stolcke 4/18/96 *** Patchlevel 32 *** o Customs: Fixed minor problem in the way rlim_max was propagated. o Customs: Fixed free swap computation for SunOS5.x. o Customs: Optionally allow jobs from local clients (-localjobs 1). This is convenient to place all processes spawned by export or pmake under customs control. o Customs: Added rexport(1) script for simple-to-use parallel job exportation. o Customs: Prepending of NETMNT prefix can be disabled in config.h by leaving NETMNT undefined. This seems to be the right thing for many sites. o PMake: Added rule for C++ compiles. o etc/customs.conf: adapted to sysinfo 3.1 output format. o etc/gnumake: added patch for 3.75 code. o Found major bug in IRIX5.3: Processes executing suid root loose their resource limits, and thus export/pmake cannot propagate the user's resource limits when installed suid root as required for priviledged port authentication. This is fixed in IRIX6.2 by setting the kernel variable reset_limits_on_exec = 0. A. Stolcke, 8/30/96 *** Patchlevel 33 *** o Ported to Solaris-PC (i86pc). o Customs: use os-bsd module for Solaris since swap space code stopped working in SunOS5.6. o Ported to Linux PC (i686). o RPC: Fixed double-freeing bug in Rpc_Reset(). (Linux was first system to expose this problem by core dumping in ImportProcess().) A. Stolcke, 4/4/98 *** Patchlevel 34 *** o Customs: Handle master hosts with multiple network interfaces and routes. o Customs: Fixed nlist access in IRIX5/IRIX6. o Customs: Fixed bug in Linux OS module: -jobs parameter was being ignored. o Customs: Updated SunOS5.x OS module to work again. o Customs: Check that master can lookup own hostname. o rexport: Always use /bin/sh for command execution to avoid portability problems across different systems. o PMake: Workaround for missing _cleanup() function. o PMake: Fixed Y2K problem in debugging output. o RPC: Workaround for Linux bug (recvfrom() on unconnected TCP sockets returns EINVAL instead of ENOTCONN). o xlog: Linux 2.2 port. A. Stolcke 8/25/99 *** Patchlevel 35 *** o PMake: Added support for long names in GNU binutils archives. o PMake: Fixed archive symbol lookup on little-endian SYSV platforms. o Customs: Paths containing automount points are unchanged in normalization (assuming that automounter configuration is uniform across hosts). o Customs: Made handling of EXPORT_EXCLUSIVE more reliable by keeping state in affected customs agent, instead of in the master. Also, there are now a maximum number of 'exclusive' jobs per machine (by default equal to 1, mimicking old behavior), set by customs -exclusives switch. o Various fixes for regression tests on Linux. o rexport: Try to max out file descriptor limit before starting pmake. Fixed handling of shell variable expansions in commands. o Miscellaneous bug fixes. A. Stolcke 05/16/2001