• Check the job's logfile for useful error messages.

Job immediately goes into the Held state

Often this means Condor is having trouble executing the job. Try using CGI condor_q -long and examining the HoldReason attribute. For example: CGI condor_q -long 13281 | grep 'HoldReason'.

Job runs for a while, then gets killed

This is usually because your job is consuming too much memory. The compute nodes each have 4 gigabytes of RAM and 4 gigabytes of swap. This is shared between up to two jobs running on the system. If the system runs out of memory, it will begin killing off large processes until the situation is resolved.

In other words, if your job requires more than 4 gigabytes of RAM, it will not perform well and may end up being killed off. You may need to process smaller data sets or restructure your code to store less data in RAM.

You can see Condor's estimate of your job's memory footprint, in megabytes, by running the condor_q command and looking at the SIZE column.

Job runs for a while, then goes idle (or gets evicted)

Troubleshooting Condor Job Problems

General suggestions

• Make sure you're giving the full path to the executable, in your submit file. (Unless the executable is in the same directory you're running condor_submit in.)
• Make sure the directory your logfile is in exists.
• Make sure your input and output files are correct.
• If you're running a script as your executable, make sure its execute bit is set and the Shebang line is correct. Try running it from the command line to make sure it works.

Guidelines for specific situations

Job runs for a while, then goes idle

This usually means the job has exceeded the requirements Condor set for it. You can check this with the CGI condor_q -better-analyze command. For example:

brodbd@patas:~$ condor_q -better-analyze 3727


-- Submitter: patas.ling.washington.edu : <192.168.100.50:44689> : patas.ling.washington.edu
---
3727.000:  Run analysis summary.  Of 44 machines,
	  44 are rejected by your job's requirements
		0 reject your job because of their own requirements
		0 match but are serving users with a better priority in the pool
		0 match but reject the job for unknown reasons
		0 match but will not currently preempt their existing job
		0 are available to run your job
	Last successful match: Tue Feb  5 20:55:24 2008
	Last failed match: Wed Feb  6 09:36:38 2008
	Reason for last match failure: no match found

WARNING:  Be advised:
	No resources matched request's constraints

The Requirements expression for your job is:

( target.Arch == "X86_64" ) && ( target.OpSys == "LINUX" ) &&
( target.Disk >= DiskUsage ) && ( ( target.Memory * 1024 ) >= ImageSize ) &&
( TARGET.FileSystemDomain == MY.FileSystemDomain )

	 Condition								 Machines Matched	 Suggestion
	 ---------								 ----------------	 ----------
1	( ( 1024 * target.Memory ) >= 2570000 )0						 REMOVE
2	( target.Arch == "X86_64" )		 44						 
3	( target.OpSys == "LINUX" )		 44						 
4	( target.Disk >= 10000 )			 44						 
5	( TARGET.FileSystemDomain == "ling.washington.edu" )
												  44						 

CGI requirements = (Memory > 1000)

This tells Condor to let the job run on any system with at least 1000 KB of RAM.

If you don't want to have to re-submit the job, you can use condor_qedit to change the requirements on the fly. First you need the current requirements line; you can get this by running CGI condor_q -long with the job number; e.g., CGI condor_q -long 3727. Then cut and paste the Requirements line into a condor_qedit command line, removing the equal sign and editing the Memory portion, like this:

brodbd@patas:~$ condor_qedit 3727 Requirements '(Arch == "X86_64") && (OpSys == "LINUX") && (Disk >= DiskUsage) && ((Memory) >= 1000) && (TARGET.FileSystemDomain == MY.FileSystemDomain)'

Be sure to put the requirements in single quotes, otherwise the shell will try to interpret the && signs and it won't work.

-- brodbd - 06 Feb 2008