Reference

The wrapper includes the Firecrest class, which is in practice a very basic client. Together with the authorisation class it takes care of the token and makes the appropriate calls for each action.

The Firecrest class

class firecrest.Firecrest(firecrest_url: str, authorization: Any, verify: str | bool | None = None, sa_role: str = 'firecrest-sa')

Bases: object

This is the basic class you instantiate to access the FirecREST API v1. Necessary parameters are the firecrest URL and an authorization object.

Parameters:

  • firecrest_url – FirecREST’s URL

  • authorization – the authorization object. This object is responsible of handling the credentials and the only requirement for it is that it has a method get_access_token() that returns a valid access token.

  • verify – either a boolean, in which case it controls whether requests will verify the server’s TLS certificate, or a string, in which case it must be a path to a CA bundle to use

  • sa_role – this corresponds to the F7T_AUTH_ROLE configuration parameter of the site. If you don’t know how FirecREST is setup it’s better to leave the default.

TOO_MANY_REQUESTS_CODE = 429
timeout: float | Tuple[float, float] | Tuple[float, None] | None

This attribute will be passed to all the requests that will be made. How many seconds to wait for the server to send data before giving up. After that time a requests.exceptions.Timeout error will be raised.

It can be a float or a tuple. More details here: https://requests.readthedocs.io.

num_retries_rate_limit: int | None

Number of retries in case the rate limit is reached. When it is set to None, the client will keep trying until it gets a different status code than 429.

all_services() → List[Service]

Returns a list containing all available micro services with a name, description, and status.

Calls:

GET /status/services

service(service_name: str) → Service

Returns information about a micro service. Returns the name, description, and status.

Parameters:

service_name – the service name

Calls:

GET /status/services/{service_name}

all_systems() → List[System]

Returns a list containing all available systems and response status.

Calls:

GET /status/systems

system(system_name: str) → System

Returns information about a system. Returns the name, description, and status.

Parameters:

system_name – the system name

Calls:

GET /status/systems/{system_name}

parameters() → Parameters

Returns configuration parameters of the FirecREST deployment that is associated with the client.

Calls:

GET /status/parameters

list_files(machine: str, target_path: str, show_hidden: bool = False) → List[LsFile]

Returns a list of files in a directory.

Parameters:

  • machine – the machine name where the filesystem belongs to

  • target_path – the absolute target path

  • show_hidden – show hidden files

Calls:

GET /utilities/ls

mkdir(machine: str, target_path: str, p: bool | None = None) → None

Creates a new directory.

Parameters:

  • machine – the machine name where the filesystem belongs to

  • target_path – the absolute target path

  • p – no error if existing, make parent directories as needed

Calls:

POST /utilities/mkdir

mv(machine: str, source_path: str, target_path: str) → None

Rename/move a file, directory, or symlink at the source_path to the target_path on machine’s filesystem.

Parameters:

  • machine – the machine name where the filesystem belongs to

  • source_path – the absolute source path

  • target_path – the absolute target path

Calls:

PUT /utilities/rename

chmod(machine: str, target_path: str, mode: str) → None

Changes the file mod bits of a given file according to the specified mode.

Parameters:

  • machine – the machine name where the filesystem belongs to

  • target_path – the absolute target path

  • mode – same as numeric mode of linux chmod tool

Calls:

PUT /utilities/chmod

chown(machine: str, target_path: str, owner: str | None = None, group: str | None = None) → None

Changes the user and/or group ownership of a given file. If only owner or group information is passed, only that information will be updated.

Parameters:

  • machine – the machine name where the filesystem belongs to

  • target_path – the absolute target path

  • owner – owner ID for target

  • group – group ID for target

Calls:

PUT /utilities/chown

copy(machine: str, source_path: str, target_path: str) → None

Copies file from source_path to target_path.

Parameters:

  • machine – the machine name where the filesystem belongs to

  • source_path – the absolute source path

  • target_path – the absolute target path

Calls:

POST /utilities/copy

file_type(machine: str, target_path: str) → str

Uses the file linux application to determine the type of a file.

Parameters:

  • machine – the machine name where the filesystem belongs to

  • target_path – the absolute target path

Calls:

GET /utilities/file

stat(machine: str, target_path: str, dereference: bool = False) → StatFile

Uses the stat linux application to determine the status of a file on the machine’s filesystem. The result follows: https://docs.python.org/3/library/os.html#os.stat_result.

Parameters:

  • machine – the machine name where the filesystem belongs to

  • target_path – the absolute target path

  • dereference – follow link (default False)

Calls:

GET /utilities/stat

Creates a symbolic link.

Parameters:

  • machine – the machine name where the filesystem belongs to

  • target_path – the absolute path that the symlink will point to

  • link_path – the absolute path to the new symlink

Calls:

POST /utilities/symlink

simple_download(machine: str, source_path: str, target_path: str | pathlib.Path | BytesIO) → None

Blocking call to download a small file. The maximun size of file that is allowed can be found from the parameters() call.

Parameters:

  • machine – the machine name where the filesystem belongs to

  • source_path – the absolute source path

  • target_path – the target path in the local filesystem or binary stream

Calls:

GET /utilities/download

simple_upload(machine: str, source_path: str | pathlib.Path | BytesIO, target_path: str, filename: str | None = None) → None

Blocking call to upload a small file. The file that will be uploaded will have the same name as the source_path. The maximum size of file that is allowed can be found from the parameters() call.

Parameters:

  • machine – the machine name where the filesystem belongs to

  • source_path – the source path of the file or binary stream

  • target_path – the absolute target path of the directory where the file will be uploaded

  • filename – naming target file to filename (default is same as the local one)

Calls:

POST /utilities/upload

simple_delete(machine: str, target_path: str) → None

Blocking call to delete a small file.

Parameters:

  • machine – the machine name where the filesystem belongs to

  • target_path – the absolute target path

Calls:

DELETE /utilities/rm

checksum(machine: str, target_path: str) → str

Calculate the SHA256 (256-bit) checksum of a specified file.

Parameters:

  • machine – the machine name where the filesystem belongs to

  • target_path – the absolute target path

Calls:

GET /utilities/checksum

head(machine: str, target_path: str, bytes: int | None = None, lines: int | None = None) → str

Display the beginning of a specified file. By default 10 lines will be returned. Bytes and lines cannot be specified simultaneously. The final result will be smaller than UTILITIES_MAX_FILE_SIZE bytes. This variable is available in the parameters command.

Parameters:

  • machine – the machine name where the filesystem belongs to

  • target_path – the absolute target path

  • lines – the number of lines to be displayed

  • bytes – the number of bytes to be displayed

Calls:

GET /utilities/head

tail(machine: str, target_path: str, bytes: int | None = None, lines: int | None = None) → str

Display the last part of a specified file. By default 10 lines will be returned. Bytes and lines cannot be specified simultaneously. The final result will be smaller than UTILITIES_MAX_FILE_SIZE bytes. This variable is available in the parameters command.

Parameters:

  • machine – the machine name where the filesystem belongs to

  • target_path – the absolute target path

  • lines – the number of lines to be displayed

  • bytes – the number of bytes to be displayed

Calls:

GET /utilities/head

view(machine: str, target_path: str) → str

View the content of a specified file. The final result will be smaller than UTILITIES_MAX_FILE_SIZE bytes. This variable is available in the parameters command.

Parameters:

  • machine – the machine name where the filesystem belongs to

  • target_path – the absolute target path

Calls:

GET /utilities/checksum

whoami() → str | None

Returns the username that FirecREST will be using to perform the other calls. Will return None if the token is not valid.

submit(machine: str, job_script: str, local_file: bool | None = True, account: str | None = None) → JobSubmit

Submits a batch script to SLURM on the target system

Parameters:

  • machine – the machine name where the scheduler belongs to

  • job_script – the path of the script (if it’s local it can be relative path, if it is on the machine it has to be the absolute path)

  • local_file – batch file can be local (default) or on the machine’s filesystem

  • account – submit the job with this project account

Calls:

POST /compute/jobs/upload or POST /compute/jobs/path

GET /tasks/{taskid}

poll(machine: str, jobs: Sequence[str | int] | None = None, start_time: str | None = None, end_time: str | None = None) → List[t.JobAcct]

Retrieves information about submitted jobs. This call uses the sacct command.

Parameters:

  • machine – the machine name where the scheduler belongs to

  • jobs – list of the IDs of the jobs

  • start_time – Start time (and/or date) of job’s query. Allowed formats are HH:MM[:SS] [AM|PM] MMDD[YY] or MM/DD[/YY] or MM.DD[.YY] MM/DD[/YY]-HH:MM[:SS] YYYY-MM-DD[THH:MM[:SS]]

  • end_time – End time (and/or date) of job’s query. Allowed formats are HH:MM[:SS] [AM|PM] MMDD[YY] or MM/DD[/YY] or MM.DD[.YY] MM/DD[/YY]-HH:MM[:SS] YYYY-MM-DD[THH:MM[:SS]]

Calls:

GET /compute/acct

GET /tasks/{taskid}

poll_active(machine: str, jobs: Sequence[str | int] | None = None) → List[t.JobQueue]

Retrieves information about active jobs. This call uses the squeue -u <username> command.

Parameters:

  • machine – the machine name where the scheduler belongs to

  • jobs – list of the IDs of the jobs

Calls:

GET /compute/jobs

GET /tasks/{taskid}

cancel(machine: str, job_id: str | int) → str

Cancels running job. This call uses the scancel command.

Parameters:

  • machine – the machine name where the scheduler belongs to

  • job_id – the ID of the job

Calls:

DELETE /compute/jobs/{job_id}

GET /tasks/{taskid}

submit_move_job(machine: str, source_path: str, target_path: str, job_name: str | None = None, time: str | None = None, stage_out_job_id: str | None = None, account: str | None = None) → JobSubmit

Move files between internal CSCS file systems. Rename/Move source_path to target_path. Possible to stage-out jobs providing the SLURM ID of a production job. More info about internal transfer: https://user.cscs.ch/storage/data_transfer/internal_transfer/

Parameters:

  • machine – the machine name where the scheduler belongs to

  • source_path – the absolute source path

  • target_path – the absolute target path

  • job_name – job name

  • time – limit on the total run time of the job. Acceptable time formats ‘minutes’, ‘minutes:seconds’, ‘hours:minutes:seconds’, ‘days-hours’, ‘days-hours:minutes’ and ‘days-hours:minutes:seconds’.

  • stage_out_job_id – transfer data after job with ID {stage_out_job_id} is completed

  • account – name of the bank account to be used in SLURM. If not set, system default is taken.

Calls:

POST /storage/xfer-internal/mv

GET /tasks/{taskid}

submit_copy_job(machine: str, source_path: str, target_path: str, job_name: str | None = None, time: str | None = None, stage_out_job_id: str | None = None, account: str | None = None) → JobSubmit

Copy files between internal CSCS file systems. Copy source_path to target_path. Possible to stage-out jobs providing the SLURM Id of a production job. More info about internal transfer: https://user.cscs.ch/storage/data_transfer/internal_transfer/

Parameters:

  • machine – the machine name where the scheduler belongs to

  • source_path – the absolute source path

  • target_path – the absolute target path

  • job_name – job name

  • time – limit on the total run time of the job. Acceptable time formats ‘minutes’, ‘minutes:seconds’, ‘hours:minutes:seconds’, ‘days-hours’, ‘days-hours:minutes’ and ‘days-hours:minutes:seconds’.

  • stage_out_job_id – transfer data after job with ID {stage_out_job_id} is completed

  • account – name of the bank account to be used in SLURM. If not set, system default is taken.

Calls:

POST /storage/xfer-internal/cp

GET /tasks/{taskid}

submit_rsync_job(machine: str, source_path: str, target_path: str, job_name: str | None = None, time: str | None = None, stage_out_job_id: str | None = None, account: str | None = None) → JobSubmit

Transfer files between internal CSCS file systems. Transfer source_path to target_path. Possible to stage-out jobs providing the SLURM Id of a production job. More info about internal transfer: https://user.cscs.ch/storage/data_transfer/internal_transfer/

Parameters:

  • machine – the machine name where the scheduler belongs to

  • source_path – the absolute source path

  • target_path – the absolute target path

  • job_name – job name

  • time – limit on the total run time of the job. Acceptable time formats ‘minutes’, ‘minutes:seconds’, ‘hours:minutes:seconds’, ‘days-hours’, ‘days-hours:minutes’ and ‘days-hours:minutes:seconds’.

  • stage_out_job_id – transfer data after job with ID {stage_out_job_id} is completed

  • account – name of the bank account to be used in SLURM. If not set, system default is taken.

Calls:

POST /storage/xfer-internal/rsync

GET /tasks/{taskid}

submit_delete_job(machine: str, target_path: str, job_name: str | None = None, time: str | None = None, stage_out_job_id: str | None = None, account: str | None = None) → JobSubmit

Remove files in internal CSCS file systems. Remove file in target_path. Possible to stage-out jobs providing the SLURM Id of a production job. More info about internal transfer: https://user.cscs.ch/storage/data_transfer/internal_transfer/

Parameters:

  • machine – the machine name where the scheduler belongs to

  • target_path – the absolute target path

  • job_name – job name

  • time – limit on the total run time of the job. Acceptable time formats ‘minutes’, ‘minutes:seconds’, ‘hours:minutes:seconds’, ‘days-hours’, ‘days-hours:minutes’ and ‘days-hours:minutes:seconds’.

  • stage_out_job_id – transfer data after job with ID {stage_out_job_id} is completed

  • account – name of the bank account to be used in SLURM. If not set, system default is taken.

Calls:

POST /storage/xfer-internal/rm

GET /tasks/{taskid}

external_upload(machine: str, source_path: str, target_path: str) → ExternalUpload

Non blocking call for the upload of larger files.

Parameters:

  • machine – the machine where the filesystem belongs to

  • source_path – the source path in the local filesystem

  • target_path – the target path in the machine’s filesystem

Returns:

an ExternalUpload object

external_download(machine: str, source_path: str) → ExternalDownload

Non blocking call for the download of larger files.

Parameters:

  • machine – the machine where the filesystem belongs to

  • source_path – the source path in the local filesystem

Returns:

an ExternalDownload object

all_reservations(machine: str) → List[dict]

List all active reservations and their status

Parameters:

machine – the machine name

Calls:

GET /reservations

create_reservation(machine: str, reservation: str, account: str, number_of_nodes: str, node_type: str, start_time: str, end_time: str) → None

Creates a new reservation with {reservation} name for a given SLURM groupname

Parameters:

  • machine – the machine name

  • reservation – the reservation name

  • account – the account in SLURM to which the reservation is made for

  • number_of_nodes – number of nodes needed for the reservation

  • node_type – type of node

  • start_time – start time for reservation (YYYY-MM-DDTHH:MM:SS)

  • end_time – end time for reservation (YYYY-MM-DDTHH:MM:SS)

Calls:

POST /reservations

update_reservation(machine: str, reservation: str, account: str, number_of_nodes: str, node_type: str, start_time: str, end_time: str) → None

Updates an already created reservation named {reservation}

Parameters:

  • machine – the machine name

  • reservation – the reservation name

  • account – the account in SLURM to which the reservation is made for

  • number_of_nodes – number of nodes needed for the reservation

  • node_type – type of node

  • start_time – start time for reservation (YYYY-MM-DDTHH:MM:SS)

  • end_time – end time for reservation (YYYY-MM-DDTHH:MM:SS)

Calls:

PUT /reservations/{reservation}

delete_reservation(machine: str, reservation: str) → None

Deletes an already created reservation named {reservation}

Parameters:

  • machine – the machine name

  • reservation – the reservation name

Calls:

DELETE /reservations/{reservation}

The ExternalUpload class

class firecrest.ExternalUpload(client: Firecrest, task_id: str, previous_responses: List[Response] | None = None)

Bases: ExternalStorage

This class handles the external upload from a file.

Tracks the progress of the upload through the status of the associated task. Final states: 114 and 115.

Status

Description

110

Waiting for Form URL from Object Storage to be retrieved

111

Form URL from Object Storage received

112

Object Storage confirms that upload to Object Storage has finished

113

Download from Object Storage to server has started

114

Download from Object Storage to server has finished

115

Download from Object Storage error
Parameters:

  • client – FirecREST client associated with the transfer

  • task_id – FirecrREST task associated with the transfer

finish_upload() → None

Finish the upload process. This call will upload the file to the staging area. Check with the method status or in_progress to see the status of the transfer. The transfer from the staging area to the systems’s filesystem can take several seconds to start to start.

property client: Firecrest

Returns the client that will be used to get information for the task.

property data: dict | None

Returns the task information from the latest response.

Calls:

GET /tasks/{taskid}

property in_progress: bool

Returns False when the transfer has been completed (succesfully or with errors), otherwise True.

Calls:

GET /tasks/{taskid}

property object_storage_data

Returns the necessary information for the external transfer. The call is blocking and in cases of large file transfers it might take a long time.

Calls:

GET /tasks/{taskid}

Return type:

dictionary or string

property status: str

Returns status of the task that is associated with this transfer.

Calls:

GET /tasks/{taskid}

property task_id: str

Returns the FirecREST task ID that is associated with this transfer.

The ExternalDownload class

class firecrest.ExternalDownload(client: Firecrest, task_id: str, previous_responses: List[Response] | None = None)

Bases: ExternalStorage

This class handles the external download from a file.

Tracks the progress of the download through the status of the associated task. Final states: 117 and 118.

Status

Description

116

Started upload from filesystem to Object Storage

117

Upload from filesystem to Object Storage has finished successfully

118

Upload from filesystem to Object Storage has finished with errors
Parameters:

  • client – FirecREST client associated with the transfer

  • task_id – FirecrREST task associated with the transfer

Invalidate the temporary URL for downloading.

Calls:

POST /storage/xfer-external/invalidate

finish_download(target_path: str | pathlib.Path | BufferedWriter) → None

Finish the download process.

Parameters:

target_path – the local path to save the file

property client: Firecrest

Returns the client that will be used to get information for the task.

property data: dict | None

Returns the task information from the latest response.

Calls:

GET /tasks/{taskid}

property in_progress: bool

Returns False when the transfer has been completed (succesfully or with errors), otherwise True.

Calls:

GET /tasks/{taskid}

property object_storage_data

Returns the necessary information for the external transfer. The call is blocking and in cases of large file transfers it might take a long time.

Calls:

GET /tasks/{taskid}

Return type:

dictionary or string

property status: str

Returns status of the task that is associated with this transfer.

Calls:

GET /tasks/{taskid}

property task_id: str

Returns the FirecREST task ID that is associated with this transfer.

The ClientCredentialsAuth class

class firecrest.ClientCredentialsAuth(client_id: str, client_secret: str, token_uri: str, min_token_validity: int = 10)

Bases: object

Client Credentials Authorization class.

Parameters:

  • client_id (string) – name of the client as registered in the authorization server

  • client_secret (string) – secret associated to the client

  • token_uri (string) – URI of the token request in the authorization server (e.g. https://auth.your-server.com/auth/realms/cscs/protocol/openid-connect/token)

  • min_token_validity (float) – reuse OIDC token until {min_token_validity} sec before the expiration time (by default 10). Since the token will be checked by different microservices, setting more time in min_token_validity will ensure that the token doesn’t expire in the middle of the request.

timeout: float | Tuple[float, float] | Tuple[float, None] | None

It will be passed to all the requests that will be made. How many seconds to wait for the server to send data before giving up. After that time a requests.exceptions.Timeout error will be raised.

It can be a float or a tuple. More details here: https://requests.readthedocs.io.

get_access_token() → str

Returns an access token to be used for accessing resources. If the request fails the token will be None

Custom types of the library

Types returned by the API.

See also: https://firecrest-api.cscs.ch

class firecrest.types.Parameter(*args, **kwargs)

A parameter record, from status/parameters/{name}

name: str
unit: str
value: Any
class firecrest.types.Parameters(*args, **kwargs)

A parameters record, from status/parameters

storage: list[Parameter]
utilities: list[Parameter]
class firecrest.types.Service(*args, **kwargs)

A service record, from status/services/{name}

description: str
service: str
status: str
status_code: int
class firecrest.types.System(*args, **kwargs)

A system record, from status/systems/{name}

description: str
status: str
system: str
class firecrest.types.Task(*args, **kwargs)

A task record, from /tasks

created_at: str
data: Any
description: str
hash_id: str
last_modify: str
service: str
status: str
task_id: str
updated_at: str
user: str
class firecrest.types.LsFile(*args, **kwargs)

A file listing record, from utilities/ls

group: str
last_modified: str
name: str
permissions: str
size: str
type: str
user: str
class firecrest.types.StatFile(*args, **kwargs)

A file stat record, from utilities/stat

Command is stat {deref} -c ‘%a %i %d %h %u %g %s %X %Y %Z

See also https://docs.python.org/3/library/os.html#os.stat_result

atime: int
ctime: int
dev: int
gid: int
ino: int
mode: int
mtime: int
size: int
uid: int
class firecrest.types.JobAcct(*args, **kwargs)

A job accounting record, from compute/acct

jobid: str
name: str
nodelist: str
nodes: str
partition: str
start_time: str
state: str
time: str
time_left: str
user: str
class firecrest.types.JobQueue(*args, **kwargs)

A job queue record, from compute/jobs

job_data_err: str
job_data_out: str
job_file: str
job_file_err: str
job_file_out: str
jobid: str
name: str
nodelist: str
nodes: str
partition: str
start_time: str
state: str
time: str
time_left: str
user: str
class firecrest.types.JobSubmit(*args, **kwargs)

A job submit record, from compute/jobs

job_data_err: str
job_data_out: str
job_file: str
job_file_err: str
job_file_out: str
jobid: int
result: str