jasmin_arc package¶
Submodules¶
jasmin_arc.arc_interface module¶
-
class
jasmin_arc.arc_interface.
ArcInterface
(config_path=None, log=<open file '<stdout>', mode 'w'>, log_level=<LogLevels.INFO: <class 'sphinx.ext.autodoc.INFO'>>)[source]¶ Bases:
object
Class to handle interactions with the ARC-CE server
-
__init__
(config_path=None, log=<open file '<stdout>', mode 'w'>, log_level=<LogLevels.INFO: <class 'sphinx.ext.autodoc.INFO'>>)[source]¶ Create an object to interface with the ARC server.
Parameters: - config_path – Path to config JSON file, or
None
to use the default settings - log – File-like object to write log messages to, or
None
to disable logging. Usesys.stdout
orsys.stderr
to print messages (default:sys.stdout
). - log_level – The level of detail logs should show (default:
LogLevels.INFO
). SeeLogLevels
for the available levels
Raises: InvalidConfigError – if config is not valid JSON or is otherwise invalid
- config_path – Path to config JSON file, or
-
cancel_job
(job_id)[source]¶ Cancel the given job
Parameters: job_id – ID of the job as returned by submit_job
Raises: JobNotFoundError – if no job with the given ID could be found
-
create_proxy
()[source]¶ Use
arcproxy
to create a proxy certificate from private key and certificate, and save it to the path given in the configRaises: ProxyGenerationError – if the certificate cannot be generated
-
create_user_config
()[source]¶ Create a user config for use with ARC client
Returns: An instance of arc.UserConfig
containing information about the necessary keys, certificates and proxy files
-
get_job
(job_id)[source]¶ Return an instance of
arc.Job
representing the job with the given IDParameters: job_id – ID of the job as returned by submit_job
Raises: JobNotFoundError – if no job with the given ID could be found Returns: Instance of arc.Job
representing the job
-
get_job_descriptions
(jsdl)[source]¶ Return an instance of
arc.JobDescriptionList
containing the job described by the given JSDLParameters: jsdl – String containing the job description in JSDL format
-
get_job_status
(job_id)[source]¶ Return the status of the given job
Parameters: job_id – ID of the job as returned by submit_job
Raises: JobNotFoundError – if no job with the given ID could be found Returns: The status of the job (see JobStatuses
for the available values)
-
get_user_config
()[source]¶ Return the cached user config, or create a new one. Also check if proxy has expired, and create a new one if so
Returns: An instance of arc.UserConfig
(seecreate_user_config
)
-
save_job_outputs
(job_id)[source]¶ Retrieve output files from a job and save them to a temp directory. The file/directory specified in
OUTPUT_FILE
will be downloaded, andstdout
andstderr
outputs are saved asstdout.txt
andstderr.txt
respectively.Parameters: job_id – ID of the job as returned by submit_job
Raises: JobNotFoundError – if no job with the given ID could be found Returns: Path to the directory the output files were saved in, or None
if no files were saved
-
submit_job
(executable, args=[], input_files=[])[source]¶ Submit a job and return the job ID
Parameters: - executable – The command to run on the LOTUS cluster
- args – List of arguments to pass to the executable
- input_files – A list of paths to local files to copy to the remote session directory (the directory the job will run from on JASMIN)
Raises: - InputFileError – if any of the specified input files do not exist or are directories
- NoTargetsAvailableError – if no execution targets can be found on the ARC server
- JobSubmissionError – if the job cannot be submitted to any targets
Returns: Job ID
-
jasmin_arc.config module¶
-
class
jasmin_arc.config.
ConnectionConfig
(config_dict, logger=None)[source]¶ Bases:
object
Class to define available config options and their default values
-
ARCPROXY_PATH
= '/usr/bin/arcproxy'¶ Path to the
arcproxy
binary, which is used to generate a proxy certificate from the private key and certificate
-
ARC_SERVER
= 'jasmin-ce.ceda.ac.uk:60000/arex'¶ URL to the ARC server
-
CERTS_DIR
= '/etc/grid-security/certificates'¶ Path to directory containing trusted CA certificates
-
CLIENT_CERT
= '~/.arc/usercert.pem'¶ Path to grid certificate file
-
CLIENT_KEY
= '~/.arc/userkey-nopass.pem'¶ Path to the private key file associated with your grid certificate
-
JOBS_INFO_FILE
= '~/.arc/jobs.dat'¶ Path to job information file used by ARC client tools (arcstat, arcget etc) to load information about submitted jobs
-
OUTPUT_FILE
= 'output'¶ The name of the file/directory to download when retrieving job outputs.
-
PROXY_FILE
= '/tmp/arcproxy_file'¶ Path to save the generated proxy certificate to
-
PROXY_RENEWAL_THRESHOLD
= 10¶ The number of seconds the proxy file can have till expiry before a new proxy is automatically generated
-
PROXY_VALIDITY_PERIOD
= 43200¶ Number of seconds to set the validity period to when generating a proxy file (default: 12 hours)
-
jasmin_arc.constants module¶
-
jasmin_arc.constants.
ARC_STATUS_MAPPING
= {'Undefined': <JobStatuses.FAILED: 'failed'>, 'Queuing': <JobStatuses.NOT_STARTED: 'not started'>, 'Failed': <JobStatuses.FAILED: 'failed'>, 'Other': <JobStatuses.FAILED: 'failed'>, 'Submitting': <JobStatuses.NOT_SUBMITTED: 'not submitted'>, 'Finishing': <JobStatuses.IN_PROGRESS: 'in progress'>, 'Accepted': <JobStatuses.NOT_SUBMITTED: 'not submitted'>, 'Deleted': <JobStatuses.FAILED: 'failed'>, 'Running': <JobStatuses.IN_PROGRESS: 'in progress'>, 'Finished': <JobStatuses.COMPLETED: 'completed'>, 'Preparing': <JobStatuses.NOT_SUBMITTED: 'not submitted'>, 'Hold': <JobStatuses.IN_PROGRESS: 'in progress'>, 'Killed': <JobStatuses.FAILED: 'failed'>}¶ Map job statuses as given by ARC to statuses in our simplified model
-
class
jasmin_arc.constants.
JobStatuses
[source]¶ Bases:
enum.Enum
Possible values for job status
-
COMPLETED
= 'completed'¶
-
FAILED
= 'failed'¶
-
IN_PROGRESS
= 'in progress'¶
-
NOT_STARTED
= 'not started'¶
-
NOT_SUBMITTED
= 'not submitted'¶
-
-
class
jasmin_arc.constants.
LogLevels
[source]¶ Bases:
enum.Enum
Log levels for specifying the level of details to include in the logs. These levels and descriptions come straight from the ARC library
-
DEBUG
= <class 'sphinx.ext.autodoc.DEBUG'>¶ DEBUG level designates finer-grained informational events which should only be used for debugging purposes
-
ERROR
= <class 'sphinx.ext.autodoc.ERROR'>¶ ERROR level designates error events that might still allow the application to continue running
-
FATAL
= <class 'sphinx.ext.autodoc.FATAL'>¶ FATAL level designates very severe error events that will presumably lead the application to abort
-
INFO
= <class 'sphinx.ext.autodoc.INFO'>¶ INFO level designates informational messages that highlight the progress of the application at coarse-grained level
-
VERBOSE
= <class 'sphinx.ext.autodoc.VERBOSE'>¶ VERBOSE level designates fine-grained informational events that will give additional information about the application
-
WARNING
= <class 'sphinx.ext.autodoc.WARNING'>¶ WARNING level designates potentially harmful situations
-
jasmin_arc.exceptions module¶
-
exception
jasmin_arc.exceptions.
InputFileError
[source]¶ Bases:
exceptions.Exception
Input file does not exist or is not a file
-
exception
jasmin_arc.exceptions.
InvalidConfigError
[source]¶ Bases:
exceptions.Exception
Exception thrown when config file is invalid
-
exception
jasmin_arc.exceptions.
InvalidJobDescription
[source]¶ Bases:
exceptions.Exception
Job description XML is invalid
-
exception
jasmin_arc.exceptions.
JobNotFoundError
[source]¶ Bases:
exceptions.Exception
A job with the given ID could not be found
-
exception
jasmin_arc.exceptions.
JobSubmissionError
[source]¶ Bases:
exceptions.Exception
Job could not be submitted to any targets