• Condor may evict it, causing it to return to the queue and stay in the idle ("I") state.

executable = hugejob

• Condor may evict it, causing it to return to the queue and stay in the idle ("I") state. This is more likely to happen when there are a lot of other jobs queued. The job may then get relaunched on another machine, but all of its progress up to that point will be lost.

Running jobs larger than 1 GB

+RequiresWholeMachine = True

to your submit file. (Note the plus sign, which is required. Also, note that this attribute is a custom one for our site and may not be available on other Condor clusters.) You also will also want to tell Condor not to check your job's memory use, so it won't be evicted when it grows larger than 2 GB. This is easily done by adding your own memory constraint to your job's submit file; for example:

Requirements = (Memory > 0)

Finally, you may want to specify a minimum amount of total memory for the machine. This can be done by adding a TotalMemory requirement. (Both TotalMemory and Memory are measured in megabytes. Memory is the memory available per slot, while TotalMemory is the total amount of memory for the whole machine.)

Note: Be careful about being too specific with TotalMemory constraints. For various reasons (memory consumed by the OS, etc.) the TotalMemory constraint will probably be stricter than you expect. For example, our 4 gigabyte nodes actually report their total memory as 3950 MB, so a constraint of (TotalMemory >= (4*1024)) will exclude them.

+RequiresWholeMachine = True

Requirements = (Memory > 0)

+RequiresWholeMachine = True

Requirements = (Memory > 0)

• If the combination of jobs on a machine exceeds the amount of available RAM and swap, the kernel out of memory killer will kill processes until memory becomes available.

Running jobs larger than 2 GB

+RequiresWholeMachine = True

Requirements = (Memory > 0)

Here's an example submit script for an executable called hugejob, which requires at least 8 GB of memory to run:

universe = vanilla
executable = hugejob
getenv = true
input = hugejob.in
output = hugejob.out
error = hugejob.err
log = hugejob.log
+RequiresWholeMachine = True
Requirements = ( Memory > 0 && TotalMemory >= (8*1024) ) 

Interaction with other jobs

1. RequiresWholeMachine jobs will only start on Slot 1. Once the job is running, other slots will be marked as having "Owner" status, to prevent single-slot jobs from running in them and consuming memory. If no machines that match the job's requirements have Slot 1 available, the job will remain idle in the queue until Slot 1 opens up on a machine.
2. If a machine with no slots taken that matches the job's requirements is available, the job will start there.
3. If no machines are completely free, but an otherwise occupied machine has Slot 1 open, the job will start there and immediately go into the "Suspended" state. It will remain suspended until all the single-slot jobs on the machine complete, and then it will continue. (This more or less causes the job to claim "dibs" on a slot that might otherwise go to a single-slot job.) If the job remains suspended for at least two hours without running, it will become eligible for preemption and may return to the queue to wait for a new slot assignment. You can force this to happen at any time with the condor_vacate command.

Running Condor jobs with large memory requirements

Normally Condor assigns one job to each CPU on a node, dividing up the memory equally. On all of our current systems this results in 2 GB of RAM per slot. Ideally, you should structure your jobs to stay within this amount of memory.

If you job grows too large, one of two things will happen.

• If it exceeds 2 GB, Condor may evict it, causing it to return to the queue and stay in the idle ("I") state. This is more likely to happen when there are a lot of other jobs queued.
• If the combination of jobs on a machine exceeds the amount of available RAM and swap (about 8 GB, for our 2-CPU nodes), the kernel out of memory killer will kill processes until memory becomes available.

Eventually I will implement a custom submit file attribute to allow jobs to claim the entire machine, but this requires a newer version of Condor than we're currently running. My current target for this upgrade is the spring '09 term break. However, there are some stop-gap techniques that can help.

By adding the requirement "VirtualMachineID == 1" to your job, it will only run on the first CPU slot of any machine. This will not prevent other jobs from occupying other slots, but it will ensure that only one copy of your job (or any similarly flagged job) will run on each machine. Note: The name of this parameter changed to SlotID in condor 7.x, so when we upgrade in the spring any submit files that use this parameter will need to be changed.

By adding an explicit Memory requirement to your job, Condor will allow it to run on any slot with at least that amount of RAM, and will not evict it if it grows larger than 2 GB. (It's still vulnerable to the out-of-memory killer if it grows too large, however.) This requirement is measured in kilobytes and, for our purposes, can be set to any arbitrary number that's less than the smallest slot in the cluster -- currently 1976 KB.

Combining these two requirements, we end up with the following, which can be added to the submit file of your large job:

Requirements = (VirtualMachineID == 1 && Memory > 1024) 

-- brodbd - 23 Feb 2009