Reference

The wrapper includes the Firecrest class, which is in practice a very basic client. Together with the authorisation class it takes care of the token and makes the appropriate calls for each action.

The Firecrest class

class firecrest.Firecrest(firecrest_url, authorization, verify=None, sa_role='firecrest-sa')

Bases: object

This is the basic class you instantiate to access the FirecREST API v1. Necessary parameters are the firecrest URL and an authorization object. This object is responsible of handling the credentials and the only requirement for it is that it has a method get_access_token() that returns a valid access token.

Parameters

  • firecrest_url (string) – FirecREST’s URL

  • authorization (object) – the authorization object

  • verify (boolean or string, optional) – either a boolean, in which case it controls whether requests will verify the server’s TLS certificate, or a string, in which case it must be a path to a CA bundle to use (default True)

  • sa_role (string, optional) – this corresponds to the F7T_AUTH_ROLE configuration parameter of the site. If you don’t know how FirecREST is setup it’s better to leave the default.

all_services()

Returns a list containing all available micro services with a name, description, and status.

Calls

GET /status/services

Return type

list of dictionaries (one for each service)

service(service_name)

Returns information about a micro service. Returns the name, description, and status.

Parameters

service_name (string) – the service name

Calls

GET /status/services/{service_name}

Return type

list of dictionaries (one for each service)

all_systems()

Returns a list containing all available systems and response status.

Calls

GET /status/systems

Return type

list of dictionaries (one for each system)

system(system_name)

Returns information about a system. Returns the name, description, and status.

Parameters

system_name (string) – the system name

Calls

GET /status/systems/{system_name}

Return type

list of dictionaries (one for each system)

parameters()

Returns list of parameters that can be configured in environment files.

Calls

GET /status/parameters

Return type

list of parameters

list_files(machine, target_path, show_hidden=None)

Returns a list of files in a directory.

Parameters

  • machine (string) – the machine name where the filesystem belongs to

  • target_path (string) – the absolute target path

  • show_hidden (boolean, optional) – show hidden files

Calls

GET /utilities/ls

Return type

list of files

mkdir(machine, target_path, p=None)

Creates a new directory.

Parameters

  • machine (string) – the machine name where the filesystem belongs to

  • target_path (string) – the absolute target path

  • p (boolean, optional) – no error if existing, make parent directories as needed

Calls

POST /utilities/mkdir

Return type

None

mv(machine, source_path, target_path)

Rename/move a file, directory, or symlink at the source_path to the target_path on machine’s filesystem.

Parameters

  • machine (string) – the machine name where the filesystem belongs to

  • source_path (string) – the absolute source path

  • target_path (string) – the absolute target path

Calls

PUT /utilities/rename

Return type

None

chmod(machine, target_path, mode)

Changes the file mod bits of a given file according to the specified mode.

Parameters

  • machine (string) – the machine name where the filesystem belongs to

  • target_path (string) – the absolute target path

  • mode (string) – same as numeric mode of linux chmod tool

Calls

PUT /utilities/chmod

Return type

None

chown(machine, target_path, owner=None, group=None)

Changes the user and/or group ownership of a given file. If only owner or group information is passed, only that information will be updated.

Parameters

  • machine (string) – the machine name where the filesystem belongs to

  • target_path (string) – the absolute target path

  • owner (string, optional) – owner username for target

  • group (string, optional) – group username for target

Calls

PUT /utilities/chown

Return type

None

copy(machine, source_path, target_path)

Copies file from source_path to target_path.

Parameters

  • machine (string) – the machine name where the filesystem belongs to

  • source_path (string) – the absolute source path

  • target_path (string) – the absolute target path

Calls

POST /utilities/copy

Return type

None

file_type(machine, target_path)

Uses the file linux application to determine the type of a file.

Parameters

  • machine (string) – the machine name where the filesystem belongs to

  • target_path (string) – the absolute target path

Calls

GET /utilities/file

Return type

string

stat(machine, target_path, dereference=False)

Uses the stat linux application to determine the status of a file on the machine’s filesystem.

Parameters

  • machine (string) – the machine name where the filesystem belongs to

  • target_path (string) – the absolute target path

  • dereference (boolean, optional) – follow link (default False)

Calls

GET /utilities/stat

Return type

string

Creates a symbolic link.

Parameters

  • machine (string) – the machine name where the filesystem belongs to

  • target_path (string) – the absolute path that the symlink will point to

  • link_path (string) – the absolute path to the new symlink

Calls

POST /utilities/symlink

Return type

None

simple_download(machine, source_path, target_path)

Blocking call to download a small file. The maximun size of file that is allowed can be found from the parameters() call.

Parameters

  • machine (string) – the machine name where the filesystem belongs to

  • source_path (string) – the absolute source path

  • target_path (string or binary stream) – the target path in the local filesystem or binary stream

Calls

GET /utilities/download

Return type

None

simple_upload(machine, source_path, target_path)

Blocking call to upload a small file. The file that will be uploaded will have the same name as the source_path. The maximum size of file that is allowed can be found from the parameters() call.

Parameters

  • machine (string) – the machine name where the filesystem belongs to

  • source_path (string or binary stream) – the source path of the file or binary stream

  • target_path (string) – the absolute target path of the directory where the file will be uploaded

Calls

POST /utilities/upload

Return type

None

simple_delete(machine, target_path)

Blocking call to delete a small file.

Parameters

  • machine (string) – the machine name where the filesystem belongs to

  • target_path (string) – the absolute target path

Calls

DELETE /utilities/rm

Return type

None

checksum(machine, target_path)

Calculate the SHA256 (256-bit) checksum of a specified file.

Parameters

  • machine (string) – the machine name where the filesystem belongs to

  • target_path (string) – the absolute target path

Calls

GET /utilities/checksum

Return type

string

view(machine, target_path)

View the content of a specified file.

Parameters

  • machine (string) – the machine name where the filesystem belongs to

  • target_path (string) – the absolute target path

Calls

GET /utilities/checksum

Return type

string

whoami()

Returns the username that FirecREST will be using to perform the other calls. Will return None if the token is not valid.

Return type

string or None

submit(machine, job_script, local_file=True)

Submits a batch script to SLURM on the target system

Parameters

  • machine (string) – the machine name where the scheduler belongs to

  • job_script (string) – the path of the script (if it’s local it can be relative path, if it is on the machine it has to be the absolute path)

  • local_file (boolean, optional) – batch file can be local (default) or on the machine’s filesystem

Calls

POST /compute/jobs/upload or POST /compute/jobs/path

GET /tasks/{taskid}

Return type

dictionary

poll(machine, jobs=[], start_time=None, end_time=None)

Retrieves information about submitted jobs. This call uses the sacct command.

Parameters

  • machine (string) – the machine name where the scheduler belongs to

  • jobs (list of strings/integers, optional) – list of the IDs of the jobs (default [])

  • start_time (string, optional) – Start time (and/or date) of job’s query. Allowed formats are HH:MM[:SS] [AM|PM] MMDD[YY] or MM/DD[/YY] or MM.DD[.YY] MM/DD[/YY]-HH:MM[:SS] YYYY-MM-DD[THH:MM[:SS]]

  • end_time (string, optional) – End time (and/or date) of job’s query. Allowed formats are HH:MM[:SS] [AM|PM] MMDD[YY] or MM/DD[/YY] or MM.DD[.YY] MM/DD[/YY]-HH:MM[:SS] YYYY-MM-DD[THH:MM[:SS]]

Calls

GET /compute/acct

GET /tasks/{taskid}

Return type

dictionary

poll_active(machine, jobs=[])

Retrieves information about active jobs. This call uses the squeue -u <username> command.

Parameters

  • machine (string) – the machine name where the scheduler belongs to

  • jobs (list of strings/integers, optional) – list of the IDs of the jobs (default [])

Calls

GET /compute/jobs

GET /tasks/{taskid}

Return type

dictionary

cancel(machine, job_id)

Retrieves information about submitted jobs. This call uses the scancel command.

Parameters

  • machine (string) – the machine name where the scheduler belongs to

  • job_id (list of strings/integers, optional) – the absolute target path (default [])

Calls

DELETE /compute/jobs/{job_id}

GET /tasks/{taskid}

Return type

dictionary

submit_move_job(machine, source_path, target_path, job_name=None, time=None, stage_out_job_id=None, account=None)

Move files between internal CSCS file systems. Rename/Move source_path to target_path. Possible to stage-out jobs providing the SLURM ID of a production job. More info about internal transfer: https://user.cscs.ch/storage/data_transfer/internal_transfer/

Parameters

  • machine (string) – the machine name where the scheduler belongs to

  • source_path (string) – the absolute source path

  • target_path (string,) – the absolute target path

  • job_name (string, optional) – job name

  • time (string, optional) – limit on the total run time of the job. Acceptable time formats ‘minutes’, ‘minutes:seconds’, ‘hours:minutes:seconds’, ‘days-hours’, ‘days-hours:minutes’ and ‘days-hours:minutes:seconds’.

  • stage_out_job_id (string, optional) – transfer data after job with ID {stage_out_job_id} is completed

  • account (string, optional) – name of the bank account to be used in SLURM. If not set, system default is taken.

Calls

POST /storage/xfer-internal/mv

GET /tasks/{taskid}

Return type

dictionary with the jobid of the submitted job

submit_copy_job(machine, source_path, target_path, job_name=None, time=None, stage_out_job_id=None, account=None)

Copy files between internal CSCS file systems. Copy source_path to target_path. Possible to stage-out jobs providing the SLURM Id of a production job. More info about internal transfer: https://user.cscs.ch/storage/data_transfer/internal_transfer/

Parameters

  • machine (string) – the machine name where the scheduler belongs to

  • source_path (string) – the absolute source path

  • target_path (string,) – the absolute target path

  • job_name (string, optional) – job name

  • time (string, optional) – limit on the total run time of the job. Acceptable time formats ‘minutes’, ‘minutes:seconds’, ‘hours:minutes:seconds’, ‘days-hours’, ‘days-hours:minutes’ and ‘days-hours:minutes:seconds’.

  • stage_out_job_id (string, optional) – transfer data after job with ID {stage_out_job_id} is completed

  • account (string, optional) – name of the bank account to be used in SLURM. If not set, system default is taken.

Calls

POST /storage/xfer-internal/cp

GET /tasks/{taskid}

Return type

dictionary with the jobid of the submitted job

submit_rsync_job(machine, source_path, target_path, job_name=None, time=None, stage_out_job_id=None, account=None)

Transfer files between internal CSCS file systems. Transfer source_path to target_path. Possible to stage-out jobs providing the SLURM Id of a production job. More info about internal transfer: https://user.cscs.ch/storage/data_transfer/internal_transfer/

Parameters

  • machine (string) – the machine name where the scheduler belongs to

  • source_path (string) – the absolute source path

  • target_path (string,) – the absolute target path

  • job_name (string, optional) – job name

  • time (string, optional) – limit on the total run time of the job. Acceptable time formats ‘minutes’, ‘minutes:seconds’, ‘hours:minutes:seconds’, ‘days-hours’, ‘days-hours:minutes’ and ‘days-hours:minutes:seconds’.

  • stage_out_job_id (string, optional) – transfer data after job with ID {stage_out_job_id} is completed

  • account (string, optional) – name of the bank account to be used in SLURM. If not set, system default is taken.

Calls

POST /storage/xfer-internal/rsync

GET /tasks/{taskid}

Return type

dictionary with the jobid of the submitted job

submit_delete_job(machine, target_path, job_name=None, time=None, stage_out_job_id=None, account=None)

Remove files in internal CSCS file systems. Remove file in target_path. Possible to stage-out jobs providing the SLURM Id of a production job. More info about internal transfer: https://user.cscs.ch/storage/data_transfer/internal_transfer/

Parameters

  • machine (string) – the machine name where the scheduler belongs to

  • target_path (string,) – the absolute target path

  • job_name (string, optional) – job name

  • time (string, optional) – limit on the total run time of the job. Acceptable time formats ‘minutes’, ‘minutes:seconds’, ‘hours:minutes:seconds’, ‘days-hours’, ‘days-hours:minutes’ and ‘days-hours:minutes:seconds’.

  • stage_out_job_id (string, optional) – transfer data after job with ID {stage_out_job_id} is completed

  • account (string, optional) – name of the bank account to be used in SLURM. If not set, system default is taken.

Calls

POST /storage/xfer-internal/rm

GET /tasks/{taskid}

Return type

dictionary with the jobid of the submitted job

external_upload(machine, source_path, target_path)

Non blocking call for the upload of larger files.

Parameters

  • machine (string) – the machine where the filesystem belongs to

  • source_path (string) – the source path in the local filesystem

  • target_path (string) – the target path in the machine’s filesystem

Returns

an ExternalUpload object

Return type

ExternalUpload

external_download(machine, source_path)

Non blocking call for the download of larger files.

Parameters

  • machine (string) – the machine where the filesystem belongs to

  • source_path (string) – the source path in the local filesystem

Returns

an ExternalDownload object

Return type

ExternalDownload

all_reservations(machine)

List all active reservations and their status

Parameters

machine (string) – the machine name

Calls

GET /reservations

Return type

list of dictionaries (one for each reservation)

create_reservation(machine, reservation, account, number_of_nodes, node_type, start_time, end_time)

Creates a new reservation with {reservation} name for a given SLURM groupname

Parameters

  • machine (string) – the machine name

  • reservation (string) – the reservation name

  • account (string) – the account in SLURM to which the reservation is made for

  • number_of_nodes (string) – number of nodes needed for the reservation

  • node_type (string) – type of node

  • start_time (string) – start time for reservation (YYYY-MM-DDTHH:MM:SS)

  • end_time (string) – end time for reservation (YYYY-MM-DDTHH:MM:SS)

Calls

POST /reservations

Return type

None

update_reservation(machine, reservation, account, number_of_nodes, node_type, start_time, end_time)

Updates an already created reservation named {reservation}

Parameters

  • machine (string) – the machine name

  • reservation (string) – the reservation name

  • account (string) – the account in SLURM to which the reservation is made for

  • number_of_nodes (string) – number of nodes needed for the reservation

  • node_type (string) – type of node

  • start_time (string) – start time for reservation (YYYY-MM-DDTHH:MM:SS)

  • end_time (string) – end time for reservation (YYYY-MM-DDTHH:MM:SS)

Calls

PUT /reservations/{reservation}

Return type

None

delete_reservation(machine, reservation)

Deletes an already created reservation named {reservation}

Parameters

  • machine (string) – the machine name

  • reservation (string) – the reservation name

Calls

DELETE /reservations/{reservation}

Return type

None

The ExternalUpload class

class firecrest.ExternalUpload(client, task_id, previous_responses=[])

Bases: ExternalStorage

This class handles the external upload from a file.

Tracks the progress of the upload through the status of the associated task. Final states: 114 and 115.

Status

Description

110

Waiting for Form URL from Object Storage to be retrieved

111

Form URL from Object Storage received

112

Object Storage confirms that upload to Object Storage has finished

113

Download from Object Storage to server has started

114

Download from Object Storage to server has finished

115

Download from Object Storage error
Parameters

  • client (Firecrest) – FirecREST client associated with the transfer

  • task_id (string) – FirecrREST task associated with the transfer

finish_upload()

Finish the upload process. This call will upload the file to the staging area. Check with the method status or in_progress to see the status of the transfer. The transfer from the staging area to the systems’s filesystem can take several seconds to start to start.

Return type

None

property client

Returns the client that will be used to get information for the task.

Return type

Firecrest Object

property data

Returns the task information from the latest response.

Calls

GET /tasks/{taskid}

Return type

dictionary

property in_progress

Returns False when the transfer has been completed (succesfully or with errors), otherwise True.

Calls

GET /tasks/{taskid}

Return type

boolean

property object_storage_data

Returns the necessary information for the external transfer. The call is blocking and in cases of large file transfers it might take a long time.

Calls

GET /tasks/{taskid}

Return type

dictionary or string

property status

Returns status of the task that is associated with this transfer.

Calls

GET /tasks/{taskid}

Return type

string

The ExternalDownload class

class firecrest.ExternalDownload(client, task_id, previous_responses=[])

Bases: ExternalStorage

This class handles the external download from a file.

Tracks the progress of the download through the status of the associated task. Final states: 117 and 118.

Status

Description

116

Started upload from filesystem to Object Storage

117

Upload from filesystem to Object Storage has finished successfully

118

Upload from filesystem to Object Storage has finished with errors
Parameters

  • client (Firecrest) – FirecREST client associated with the transfer

  • task_id (string) – FirecrREST task associated with the transfer

Invalidate the temporary URL for downloading.

Calls

POST /storage/xfer-external/invalidate

Return type

None

finish_download(target_path)

Finish the download process.

Parameters

target_path (string or binary stream) – the local path to save the file

Return type

None

property client

Returns the client that will be used to get information for the task.

Return type

Firecrest Object

property data

Returns the task information from the latest response.

Calls

GET /tasks/{taskid}

Return type

dictionary

property in_progress

Returns False when the transfer has been completed (succesfully or with errors), otherwise True.

Calls

GET /tasks/{taskid}

Return type

boolean

property object_storage_data

Returns the necessary information for the external transfer. The call is blocking and in cases of large file transfers it might take a long time.

Calls

GET /tasks/{taskid}

Return type

dictionary or string

property status

Returns status of the task that is associated with this transfer.

Calls

GET /tasks/{taskid}

Return type

string

The ClientCredentialsAuth class

class firecrest.ClientCredentialsAuth(client_id, client_secret, token_uri, min_token_validity=10)

Bases: object

Client Credentials Authorization class.

Parameters

  • client_id (string) – name of the client as registered in the authorization server

  • client_secret (string) – secret associated to the client

  • token_uri (string) – URI of the token request in the authorization server (e.g. https://auth.your-server.com/auth/realms/cscs/protocol/openid-connect/token)

  • min_token_validity (float) – reuse OIDC token until {min_token_validity} sec before the expiration time (by default 10). Since the token will be checked by different microservices, setting more time in min_token_validity will ensure that the token doesn’t expire in the middle of the request.

get_access_token()

Returns an access token to be used for accessing resources. If the request fails the token will be None

Return type

string