Run parallel jobs on the compute grid - Revision history

Jadrian Miles: Initial update to remove references to SGE

2011-09-07T15:18:02Z

Initial update to remove references to SGE

Jadrian Miles: Run parallel jobs on the Sun Grid Engine moved to Run parallel jobs on the compute grid: SGE has been phased out.

2011-09-07T14:58:24Z

Run parallel jobs on the Sun Grid Engine moved to Run parallel jobs on the compute grid: SGE has been phased out.

← Older revision	Revision as of 14:58, 7 September 2011
(No difference)

Jadrian Miles: /* Recompiling for a 64-bit Architecture */

2010-03-07T21:05:02Z

Recompiling for a 64-bit Architecture

← Older revision		Revision as of 21:05, 7 March 2010
Line 34:		Line 34:

	<pre>		<pre>
	> ssh ~~sixg~~		> ssh sge
	> qlogin -l arch=lx24-amd64		> qlogin -l arch=lx24-amd64
	</pre>		</pre>

Jadrian Miles: Fixed greater-than-or-equal HTML entities /* Submit the job to SGE */

2009-12-03T16:03:50Z

Fixed greater-than-or-equal HTML entities Submit the job to SGE

← Older revision		Revision as of 16:03, 3 December 2009
Line 75:		Line 75:
	#** <tt>'''n'''</tt> --- don't get any emails. Monitor your job with <tt>qmon</tt> instead.		#** <tt>'''n'''</tt> --- don't get any emails. Monitor your job with <tt>qmon</tt> instead.
	#* <code>-t <RANGE></code> --- specify a range for the task IDs that will be sent to your parallel jobs. The range looks like this: <tt>'''MIN-MAX:STEP'''</tt>. This is a range specification similar to Matlab's; in particular, note that task IDs are one-indexed and the range is ''inclusive'' of its maximum value.		#* <code>-t <RANGE></code> --- specify a range for the task IDs that will be sent to your parallel jobs. The range looks like this: <tt>'''MIN-MAX:STEP'''</tt>. This is a range specification similar to Matlab's; in particular, note that task IDs are one-indexed and the range is ''inclusive'' of its maximum value.
	#** <code>MIN</code> is an integer &~~geq~~; 1 that represents the lowest ID that will be assigned.		#** <code>MIN</code> is an integer ≥ 1 that represents the lowest ID that will be assigned.
	#** <code>MAX</code> is an integer &~~geq~~; <code>MIN</code> that represents the highest ID that will be assigned.		#** <code>MAX</code> is an integer ≥ <code>MIN</code> that represents the highest ID that will be assigned.
	#** <code>STEP</code> is an integer &~~geq~~; 1 that represents the step size taken between assigned task IDs.		#** <code>STEP</code> is an integer ≥ 1 that represents the step size taken between assigned task IDs.
	#* <code>-wd <PATH></code> --- ''optional''. Set the working directory for the host executing your job to <tt><PATH></tt>.		#* <code>-wd <PATH></code> --- ''optional''. Set the working directory for the host executing your job to <tt><PATH></tt>.
	#* <code><SHELLSCRIPT></code> --- the shell script to be executed by the SGE hosts.		#* <code><SHELLSCRIPT></code> --- the shell script to be executed by the SGE hosts.

Jadrian Miles: /* See Also */

2009-03-31T05:00:43Z

Jadrian Miles: /* Submit the job to SGE */

2009-03-30T17:05:41Z

Submit the job to SGE

← Older revision		Revision as of 17:05, 30 March 2009
Line 61:		Line 61:
	=== Submit the job to SGE ===		=== Submit the job to SGE ===

	# <code>ssh sge</code> --- ~~Log into~~ the machine <tt>sge</tt>		# <code>ssh sge</code> --- You can only issue SGE commands from the particular machine <tt>sge</tt>, so log into that.
	# <code>cd</code> to the directory from which you want to run your shell script		# <code>cd</code> to the directory from which you want to run your shell script.
	# Submit the following command, replacing <code><STUFF_IN_BRACKETS></code> with the appropriate arguments:		# Submit the following command, replacing <code><STUFF_IN_BRACKETS></code> with the appropriate arguments:
	#; <pre>qsub -l arch=<ARCH> -q <QUEUE> -m <NOTIFY> -t <RANGE> -wd <PATH> <SHELLSCRIPT></pre>		#; <pre>qsub -l arch=<ARCH> -q <QUEUE> -m <NOTIFY> -t <RANGE> -wd <PATH> <SHELLSCRIPT></pre>

Jadrian Miles: /* Write a shell script interface to your executable */

2009-03-30T17:04:39Z

Write a shell script interface to your executable

← Older revision		Revision as of 17:04, 30 March 2009
Line 44:		Line 44:
	=== Write a shell script interface to your executable ===		=== Write a shell script interface to your executable ===

	To parallel jobs, SGE spawns a different instance of the job on every machine available, each time setting the environment variable <tt>$SGE_TASK_ID</tt> to a different value (see below for how to specify the range of values). The easiest way to run your executable is via a shell script that captures this and transforms it into a command-line argument for the program that's doing the real work. Here's an example script:		To run parallel jobs, SGE spawns a different instance of the job on every machine available, each time setting the environment variable <tt>$SGE_TASK_ID</tt> to a different value (see below for how to specify the range of values). The easiest way to run your executable is via a shell script that captures this and transforms it into a command-line argument for the program that's doing the real work. Here's an example script:

	<pre>		<pre>

Jadrian Miles: /* See Also */

2009-03-30T17:03:30Z

Jadrian Miles at 16:51, 30 March 2009

2009-03-30T16:51:52Z

← Older revision		Revision as of 16:51, 30 March 2009
Line 103:		Line 103:
	** [http://wikis.sun.com/display/GridEngine/Submitting+Extended+Jobs+and+Advanced+Jobs#SubmittingExtendedJobsandAdvancedJobs-HowtoSubmitanArrayJobFromtheCommandLine Submitting an array job]		** [http://wikis.sun.com/display/GridEngine/Submitting+Extended+Jobs+and+Advanced+Jobs#SubmittingExtendedJobsandAdvancedJobs-HowtoSubmitanArrayJobFromtheCommandLine Submitting an array job]
	* [http://cs.brown.edu/~dmcc/hogwash/ Hogwash], David McClosky's even simpler interface layer over SGE. Just write all the command you want run on separate lines of a text file, submit it, and you're done. There's a Python interface for scripting, too.		* [http://cs.brown.edu/~dmcc/hogwash/ Hogwash], David McClosky's even simpler interface layer over SGE. Just write all the command you want run on separate lines of a text file, submit it, and you're done. There's a Python interface for scripting, too.

			[[Category:Hardware]][[Category:HOWTO]]

Jadrian Miles: User:Jadrian Miles/Run parallel jobs on the Sun Grid Engine moved to Run parallel jobs on the Sun Grid Engine: Publishing this page to the main wiki

2009-03-30T16:50:49Z

User:Jadrian Miles/Run parallel jobs on the Sun Grid Engine moved to Run parallel jobs on the Sun Grid Engine: Publishing this page to the main wiki

← Older revision		Revision as of 16:50, 30 March 2009
(No difference)

← Older revision		Revision as of 05:00, 31 March 2009
Line 102:		Line 102:
	** [http://wikis.sun.com/display/GridEngine/Submitting+Batch+Jobs Submitting batch jobs]		** [http://wikis.sun.com/display/GridEngine/Submitting+Batch+Jobs Submitting batch jobs]
	** [http://wikis.sun.com/display/GridEngine/Submitting+Extended+Jobs+and+Advanced+Jobs#SubmittingExtendedJobsandAdvancedJobs-HowtoSubmitanArrayJobFromtheCommandLine Submitting an array job]		** [http://wikis.sun.com/display/GridEngine/Submitting+Extended+Jobs+and+Advanced+Jobs#SubmittingExtendedJobsandAdvancedJobs-HowtoSubmitanArrayJobFromtheCommandLine Submitting an array job]
	* [http://cs.brown.edu/~dmcc/hogwash/ Hogwash], David McClosky's even simpler interface layer over SGE. Just write all the ~~command~~ you want run on separate lines of a text file, submit it, and you're done. There's a Python interface for scripting, too.		* [http://cs.brown.edu/~dmcc/hogwash/ Hogwash], David McClosky's even simpler interface layer over SGE. Just write all the commands you want run on separate lines of a text file, submit it, and you're done. There's a Python interface for scripting, too.

	[[Category:IT Tools]][[Category:Hardware]][[Category:HOWTO]]		[[Category:IT Tools]][[Category:Hardware]][[Category:HOWTO]]

← Older revision		Revision as of 17:03, 30 March 2009
Line 104:		Line 104:
	* [http://cs.brown.edu/~dmcc/hogwash/ Hogwash], David McClosky's even simpler interface layer over SGE. Just write all the command you want run on separate lines of a text file, submit it, and you're done. There's a Python interface for scripting, too.		* [http://cs.brown.edu/~dmcc/hogwash/ Hogwash], David McClosky's even simpler interface layer over SGE. Just write all the command you want run on separate lines of a text file, submit it, and you're done. There's a Python interface for scripting, too.

	[[Category:Hardware]][[Category:HOWTO]]		[[Category:IT Tools]][[Category:Hardware]][[Category:HOWTO]]