Asynchronous FirecREST objects

The library also provides an asynchronous API for the client:

The `AsyncFirecrest` class

class firecrest.AsyncFirecrest(firecrest_url: str, authorization: Any, verify: str | bool | None = None, sa_role: str = 'firecrest-sa')

Bases: object

This is the basic class you instantiate to access the FirecREST API v1. Necessary parameters are the firecrest URL and an authorization object.

Parameters:

firecrest_url – FirecREST’s URL
authorization – the authorization object. This object is responsible of handling the credentials and the only requirement for it is that it has a method get_access_token() that returns a valid access token.
verify – either a boolean, in which case it controls whether requests will verify the server’s TLS certificate, or a string, in which case it must be a path to a CA bundle to use
sa_role – this corresponds to the F7T_AUTH_ROLE configuration parameter of the site. If you don’t know how FirecREST is setup it’s better to leave the default.

TOO_MANY_REQUESTS_CODE = 429

timeout: Any

This attribute will be passed to all the requests that will be made. How many seconds to wait for the server to send data before giving up. After that time a requests.exceptions.Timeout error will be raised.

It can be a float or a tuple. More details here: https://www.python-httpx.org/advanced/#fine-tuning-the-configuration.

num_retries_rate_limit: int | None: Number of retries in case the rate limit is reached. When it is set to None, the client will keep trying until it gets a different status code than 429.

time_between_calls: dict[str, float]: Seconds between requests in each microservice

set_api_version(api_version: str) → None: Set the version of the api of firecrest. By default it will be assumed that you are using version 1.13.1 or compatible. The version is parsed by the packaging library.

async close_session() → None: Close the httpx session

async create_new_session() → None: Create a new httpx session

property is_session_closed: bool: Check if the httpx session is closed

async all_services() → List[Service]

Returns a list containing all available micro services with a name, description, and status.

Calls:: GET /status/services

async service(service_name: str) → Service

Returns information about a micro service. Returns the name, description, and status.

Parameters:: service_name – the service name
Calls:: GET /status/services/{service_name}

async all_systems() → List[System]

Returns a list containing all available systems and response status.

Calls:: GET /status/systems

async system(system_name: str) → System

Returns information about a system. Returns the name, description, and status.

Parameters:: system_name – the system name
Calls:: GET /status/systems/{system_name}

async parameters() → Parameters

Returns configuration parameters of the FirecREST deployment that is associated with the client.

Calls:: GET /status/parameters

async filesystems(system_name: str | None = None) → dict[str, List[t.Filesystem]]

Returns the status of the filesystems per system.

Parameters:: system_name – the system name
Calls:: GET /status/filesystems
Calls:: GET /status/filesystems/{system_name}

Warning

This is available only for FirecREST>=1.15.0

async list_files(machine: str, target_path: str, show_hidden: bool = False) → List[LsFile]

Returns a list of files in a directory.

Parameters:

machine – the machine name where the filesystem belongs to
target_path – the absolute target path
show_hidden – show hidden files

Calls:

GET /utilities/ls

async mkdir(machine: str, target_path: str, p: bool | None = None) → str

Creates a new directory. When successful, the method returns a string with the path of the newly created directory.

Parameters:

machine – the machine name where the filesystem belongs to
target_path – the absolute target path
p – no error if existing, make parent directories as needed

Calls:

POST /utilities/mkdir

async mv(machine: str, source_path: str, target_path: str) → str

Rename/move a file, directory, or symlink at the source_path to the target_path on machine’s filesystem. When successful, the method returns a string with the new path of the file.

Parameters:

machine – the machine name where the filesystem belongs to
source_path – the absolute source path
target_path – the absolute target path

Calls:

PUT /utilities/rename

async chmod(machine: str, target_path: str, mode: str) → None

Changes the file mod bits of a given file according to the specified mode.

Parameters:

machine – the machine name where the filesystem belongs to
target_path – the absolute target path
mode – same as numeric mode of linux chmod tool

Calls:

PUT /utilities/chmod

async chown(machine: str, target_path: str, owner: str | None = None, group: str | None = None) → None

Changes the user and/or group ownership of a given file. If only owner or group information is passed, only that information will be updated.

Parameters:

machine – the machine name where the filesystem belongs to
target_path – the absolute target path
owner – owner ID for target
group – group ID for target

Calls:

PUT /utilities/chown

async copy(machine: str, source_path: str, target_path: str) → str

Copies file from source_path to target_path. When successful, the method returns a string with the path of the newly created file.

Parameters:

machine – the machine name where the filesystem belongs to
source_path – the absolute source path
target_path – the absolute target path

Calls:

POST /utilities/copy

async file_type(machine: str, target_path: str) → str

Uses the file linux application to determine the type of a file.

Parameters:

machine – the machine name where the filesystem belongs to
target_path – the absolute target path

Calls:

GET /utilities/file

async stat(machine: str, target_path: str, dereference: bool = False) → StatFile

Uses the stat linux application to determine the status of a file on the machine’s filesystem. The result follows: https://docs.python.org/3/library/os.html#os.stat_result.

Parameters:

machine – the machine name where the filesystem belongs to
target_path – the absolute target path
dereference – follow link (default False)

Calls:

GET /utilities/stat

async symlink(machine: str, target_path: str, link_path: str) → str

Creates a symbolic link. When successful, the method returns a string with the path of the newly created link.

Parameters:

machine – the machine name where the filesystem belongs to
target_path – the absolute path that the symlink will point to
link_path – the absolute path to the new symlink

Calls:

POST /utilities/symlink

async simple_download(machine: str, source_path: str, target_path: str | pathlib.Path | BytesIO) → None

Blocking call to download a small file. The maximun size of file that is allowed can be found from the parameters() call.

Parameters:

machine – the machine name where the filesystem belongs to
source_path – the absolute source path
target_path – the target path in the local filesystem or binary stream

Calls:

GET /utilities/download

async simple_upload(machine: str, source_path: str | pathlib.Path | BytesIO, target_path: str, filename: str | None = None) → None

Blocking call to upload a small file. The file that will be uploaded will have the same name as the source_path. The maximum size of file that is allowed can be found from the parameters() call.

Parameters:

machine – the machine name where the filesystem belongs to
source_path – the source path of the file or binary stream
target_path – the absolute target path of the directory where the file will be uploaded
filename – naming target file to filename (default is same as the local one)

Calls:

POST /utilities/upload

async simple_delete(machine: str, target_path: str) → None

Blocking call to delete a small file.

Parameters:

machine – the machine name where the filesystem belongs to
target_path – the absolute target path

Calls:

DELETE /utilities/rm

async checksum(machine: str, target_path: str) → str

Calculate the SHA256 (256-bit) checksum of a specified file.

Parameters:

machine – the machine name where the filesystem belongs to
target_path – the absolute target path

Calls:

GET /utilities/checksum

async head(machine: str, target_path: str, bytes: str | None = None, lines: str | None = None, skip_ending: bool | None = False) → str

Display the beginning of a specified file. By default 10 lines will be returned. Bytes and lines cannot be specified simultaneously. The final result will be smaller than UTILITIES_MAX_FILE_SIZE bytes. This variable is available in the parameters command.

Parameters:

machine – the machine name where the filesystem belongs to
target_path – the absolute target path
bytes – the number of bytes to be displayed
lines – the number of lines to be displayed
skip_ending – the output will be the whole file, without the last NUM bytes/lines of each file. NUM should be specified in the respective argument through bytes or lines. Equivalent to passing -NUM to the head command.

Calls:

GET /utilities/head

async tail(machine: str, target_path: str, bytes: str | None = None, lines: str | None = None, skip_beginning: bool | None = False) → str

Display the last part of a specified file. By default 10 lines will be returned. Bytes and lines cannot be specified simultaneously. The final result will be smaller than UTILITIES_MAX_FILE_SIZE bytes. This variable is available in the parameters command.

Parameters:

machine – the machine name where the filesystem belongs to
target_path – the absolute target path
bytes – the number of bytes to be displayed
lines – the number of lines to be displayed
skip_beginning – the output will start with byte/line NUM of each file. NUM should be specified in the respective argument through bytes or lines. Equivalent to passing +NUM to the tail command.

Calls:

GET /utilities/head

async view(machine: str, target_path: str) → str

View the content of a specified file. The final result will be smaller than UTILITIES_MAX_FILE_SIZE bytes. This variable is available in the parameters command.

Parameters:

machine – the machine name where the filesystem belongs to
target_path – the absolute target path

Calls:

GET /utilities/checksum

async whoami(machine=None) → str | None

Returns the username that FirecREST will be using to perform the other calls. In the case the machine name is passed in the arguments, a call is made to the respective endpoint and the command whoami is run on the machine. Otherwise, the library decodes the token and will return None if the token is not valid.

Calls:: GET /utilities/whoami

async groups(machine) → UserId

Returns the output of the id command, user and group ids.

Calls:: GET /utilities/whoami

Warning

This is available only for FirecREST>=1.15.0

Submits a batch script to SLURM on the target system. One of script_str, script_local and script_remote needs to be set.

Parameters:

machine – the machine name where the scheduler belongs to
job_script – [deprecated] use script_str, script_local_path or script_remote_path
local_file – [deprecated]
script_str – the content of the script to be submitted
script_local_path – the path of the script on the local file system
script_remote_path – the full path of the script on the remote file system
account – submit the job with this project account
env_vars – dictionary (varName, value) defining environment variables to be exported for the job

Calls:

POST /compute/jobs/upload or POST /compute/jobs/path

GET /tasks/{taskid}

Retrieves information about submitted jobs. This call uses the sacct command.

Parameters:

machine – the machine name where the scheduler belongs to
jobs – list of the IDs of the jobs
start_time – Start time (and/or date) of job’s query. Allowed formats are HH:MM[:SS] [AM|PM] MMDD[YY] or MM/DD[/YY] or MM.DD[.YY] MM/DD[/YY]-HH:MM[:SS] YYYY-MM-DD[THH:MM[:SS]]
end_time – End time (and/or date) of job’s query. Allowed formats are HH:MM[:SS] [AM|PM] MMDD[YY] or MM/DD[/YY] or MM.DD[.YY] MM/DD[/YY]-HH:MM[:SS] YYYY-MM-DD[THH:MM[:SS]]
page_size – number of entries returned (when page_number is not None, the default value is 25)
page_number – page number (if set to None the default value is 0)

Calls:

GET /compute/acct

GET /tasks/{taskid}

async poll_active(machine: str, jobs: Sequence[str | int] | None = None, page_size: int | None = None, page_number: int | None = None) → List[t.JobQueue]

Retrieves information about active jobs. This call uses the squeue -u <username> command.

Parameters:

machine – the machine name where the scheduler belongs to
jobs – list of the IDs of the jobs
page_size – number of entries returned (when page_number is not None, the default value is 25)
page_number – page number (if set to None the default value is 0)

Calls:

GET /compute/jobs

GET /tasks/{taskid}

async cancel(machine: str, job_id: str | int) → str

Cancels running job. This call uses the scancel command.

Parameters:

machine – the machine name where the scheduler belongs to
job_id – the ID of the job

Calls:

DELETE /compute/jobs/{job_id}

GET /tasks/{taskid}

async submit_move_job(machine: str, source_path: str, target_path: str, job_name: str | None = None, time: str | None = None, stage_out_job_id: str | None = None, account: str | None = None) → JobSubmit

Move files between internal CSCS file systems. Rename/Move source_path to target_path. Possible to stage-out jobs providing the SLURM ID of a production job. More info about internal transfer: https://user.cscs.ch/storage/data_transfer/internal_transfer/

Parameters:

machine – the machine name where the scheduler belongs to
source_path – the absolute source path
target_path – the absolute target path
job_name – job name
time – limit on the total run time of the job. Acceptable time formats ‘minutes’, ‘minutes:seconds’, ‘hours:minutes:seconds’, ‘days-hours’, ‘days-hours:minutes’ and ‘days-hours:minutes:seconds’.
stage_out_job_id – transfer data after job with ID {stage_out_job_id} is completed
account – name of the bank account to be used in SLURM. If not set, system default is taken.

Calls:

POST /storage/xfer-internal/mv

GET /tasks/{taskid}

async submit_copy_job(machine: str, source_path: str, target_path: str, job_name: str | None = None, time: str | None = None, stage_out_job_id: str | None = None, account: str | None = None) → JobSubmit

Copy files between internal CSCS file systems. Copy source_path to target_path. Possible to stage-out jobs providing the SLURM Id of a production job. More info about internal transfer: https://user.cscs.ch/storage/data_transfer/internal_transfer/

Parameters:

machine – the machine name where the scheduler belongs to
source_path – the absolute source path
target_path – the absolute target path
job_name – job name
time – limit on the total run time of the job. Acceptable time formats ‘minutes’, ‘minutes:seconds’, ‘hours:minutes:seconds’, ‘days-hours’, ‘days-hours:minutes’ and ‘days-hours:minutes:seconds’.
stage_out_job_id – transfer data after job with ID {stage_out_job_id} is completed
account – name of the bank account to be used in SLURM. If not set, system default is taken.

Calls:

POST /storage/xfer-internal/cp

GET /tasks/{taskid}

async submit_rsync_job(machine: str, source_path: str, target_path: str, job_name: str | None = None, time: str | None = None, stage_out_job_id: str | None = None, account: str | None = None) → JobSubmit

Transfer files between internal CSCS file systems. Transfer source_path to target_path. Possible to stage-out jobs providing the SLURM Id of a production job. More info about internal transfer: https://user.cscs.ch/storage/data_transfer/internal_transfer/

Parameters:

machine – the machine name where the scheduler belongs to
source_path – the absolute source path
target_path – the absolute target path
job_name – job name
time – limit on the total run time of the job. Acceptable time formats ‘minutes’, ‘minutes:seconds’, ‘hours:minutes:seconds’, ‘days-hours’, ‘days-hours:minutes’ and ‘days-hours:minutes:seconds’.
stage_out_job_id – transfer data after job with ID {stage_out_job_id} is completed
account – name of the bank account to be used in SLURM. If not set, system default is taken.

Calls:

POST /storage/xfer-internal/rsync

GET /tasks/{taskid}

async submit_delete_job(machine: str, target_path: str, job_name: str | None = None, time: str | None = None, stage_out_job_id: str | None = None, account: str | None = None) → JobSubmit

Remove files in internal CSCS file systems. Remove file in target_path. Possible to stage-out jobs providing the SLURM Id of a production job. More info about internal transfer: https://user.cscs.ch/storage/data_transfer/internal_transfer/

Parameters:

machine – the machine name where the scheduler belongs to
target_path – the absolute target path
job_name – job name
time – limit on the total run time of the job. Acceptable time formats ‘minutes’, ‘minutes:seconds’, ‘hours:minutes:seconds’, ‘days-hours’, ‘days-hours:minutes’ and ‘days-hours:minutes:seconds’.
stage_out_job_id – transfer data after job with ID {stage_out_job_id} is completed
account – name of the bank account to be used in SLURM. If not set, system default is taken.

Calls:

POST /storage/xfer-internal/rm

GET /tasks/{taskid}

async external_upload(machine: str, source_path: str, target_path: str) → AsyncExternalUpload

Non blocking call for the upload of larger files.

Parameters:

machine – the machine where the filesystem belongs to
source_path – the source path in the local filesystem
target_path – the target path in the machine’s filesystem

Returns:

an ExternalUpload object

async external_download(machine: str, source_path: str) → AsyncExternalDownload

Non blocking call for the download of larger files.

Parameters:

machine – the machine where the filesystem belongs to
source_path – the source path in the local filesystem

Returns:

an ExternalDownload object

async all_reservations(machine: str) → List[dict]

List all active reservations and their status

Parameters:: machine – the machine name
Calls:: GET /reservations

async create_reservation(machine: str, reservation: str, account: str, number_of_nodes: str, node_type: str, start_time: str, end_time: str) → None

Creates a new reservation with {reservation} name for a given SLURM groupname

Parameters:

machine – the machine name
reservation – the reservation name
account – the account in SLURM to which the reservation is made for
number_of_nodes – number of nodes needed for the reservation
node_type – type of node
start_time – start time for reservation (YYYY-MM-DDTHH:MM:SS)
end_time – end time for reservation (YYYY-MM-DDTHH:MM:SS)

Calls:

POST /reservations

async update_reservation(machine: str, reservation: str, account: str, number_of_nodes: str, node_type: str, start_time: str, end_time: str) → None

Updates an already created reservation named {reservation}

Parameters:

machine – the machine name
reservation – the reservation name
account – the account in SLURM to which the reservation is made for
number_of_nodes – number of nodes needed for the reservation
node_type – type of node
start_time – start time for reservation (YYYY-MM-DDTHH:MM:SS)
end_time – end time for reservation (YYYY-MM-DDTHH:MM:SS)

Calls:

PUT /reservations/{reservation}

async delete_reservation(machine: str, reservation: str) → None

Deletes an already created reservation named {reservation}

Parameters:

machine – the machine name
reservation – the reservation name

Calls:

DELETE /reservations/{reservation}

The `AsyncExternalDownload` class

class firecrest.AsyncExternalDownload(client: AsyncFirecrest, task_id: str, previous_responses: List[requests.Response] | None = None)

Bases: AsyncExternalStorage

This class handles the external download from a file.

Tracks the progress of the download through the status of the associated task. Final states: 117 and 118.

Status	Description
116	Started upload from filesystem to Object Storage
117	Upload from filesystem to Object Storage has finished successfully
118	Upload from filesystem to Object Storage has finished with errors

Parameters:

client – FirecREST client associated with the transfer
task_id – FirecrREST task associated with the transfer

async invalidate_object_storage_link() → None

Invalidate the temporary URL for downloading.

Calls:: POST /storage/xfer-external/invalidate

property object_storage_link: str

Get the direct download url for the file. The response from the FirecREST api changed after version 1.13.0, so make sure to set to older version, if you are using an older deployment.

Calls:: GET /tasks/{taskid}

property client: AsyncFirecrest: Returns the client that will be used to get information for the task.

property data: dict | None

Returns the task information from the latest response.

Calls:: GET /tasks/{taskid}

async finish_download(target_path: str | pathlib.Path | BufferedWriter) → None

Finish the download process.

Parameters:: target_path – the local path to save the file

property in_progress: bool

Returns False when the transfer has been completed (succesfully or with errors), otherwise True.

Calls:: GET /tasks/{taskid}

property object_storage_data

Returns the necessary information for the external transfer. The call is blocking and in cases of large file transfers it might take a long time.

Calls:: GET /tasks/{taskid}
Return type:: dictionary or string

property status: str

Returns status of the task that is associated with this transfer.

Calls:: GET /tasks/{taskid}

property task_id: str: Returns the FirecREST task ID that is associated with this transfer.

The `AsyncExternalUpload` class

class firecrest.AsyncExternalUpload(client: AsyncFirecrest, task_id: str, previous_responses: List[requests.Response] | None = None)

Bases: AsyncExternalStorage

This class handles the external upload from a file.

Tracks the progress of the upload through the status of the associated task. Final states: 114 and 115.

Status	Description
110	Waiting for Form URL from Object Storage to be retrieved
111	Form URL from Object Storage received
112	Object Storage confirms that upload to Object Storage has finished
113	Download from Object Storage to server has started
114	Download from Object Storage to server has finished
115	Download from Object Storage error

Parameters:

client – FirecREST client associated with the transfer
task_id – FirecrREST task associated with the transfer

async finish_upload() → None: Finish the upload process. This call will upload the file to the staging area. Check with the method status or in_progress to see the status of the transfer. The transfer from the staging area to the systems’s filesystem can take several seconds to start to start.

property client: AsyncFirecrest: Returns the client that will be used to get information for the task.

property data: dict | None

Returns the task information from the latest response.

Calls:: GET /tasks/{taskid}

property in_progress: bool

Returns False when the transfer has been completed (succesfully or with errors), otherwise True.

Calls:: GET /tasks/{taskid}

property object_storage_data

Returns the necessary information for the external transfer. The call is blocking and in cases of large file transfers it might take a long time.

Calls:: GET /tasks/{taskid}
Return type:: dictionary or string

property status: str

Returns status of the task that is associated with this transfer.

Calls:: GET /tasks/{taskid}

property task_id: str: Returns the FirecREST task ID that is associated with this transfer.

Asynchronous FirecREST objects

The AsyncFirecrest class

The AsyncExternalDownload class

The AsyncExternalUpload class

The `AsyncFirecrest` class

The `AsyncExternalDownload` class

The `AsyncExternalUpload` class