Asynchronous FirecREST objects

The library also provides an asynchronous API for the client:

The AsyncFirecrest class

class firecrest.v2.AsyncFirecrest(firecrest_url: str, authorization: Any, verify: str | bool | ssl.SSLContext = True)

Bases: object

This is the basic class you instantiate to access the FirecREST API v2. Necessary parameters are the firecrest URL and an authorization object.

Parameters:

  • firecrest_url – FirecREST’s URL

  • authorization – the authorization object. This object is responsible of handling the credentials and the only requirement for it is that it has a method get_access_token() that returns a valid access token.

  • verify – either a boolean, in which case it controls whether requests will verify the server’s TLS certificate, or a string, in which case it must be a path to a CA bundle to use

TOO_MANY_REQUESTS_CODE = 429
MAX_DIRECT_UPLOAD_SIZE = 1048576
MAX_S3_CONNECTIONS = 10
disable_client_logging: bool

Disable all logging from the client.

set_api_version(api_version: str) → None

Set the version of the api of firecrest. By default it will be assumed that you are using version 2.4.0 or compatible. The version is parsed by the packaging library.

async close_session() → None

Close the httpx session

async create_new_session() → None

Create a new httpx session

set_maximum_s3_connections(max_connections: int) → None

Set the maximum number of simultaneous connections to S3. By default it is set to 10.

property is_session_closed: bool

Check if the httpx session is closed

log(level: int, msg: Any) → None

Log a message with the given level on the client logger.

async server_version() → str | None

Returns the exact API version of the FirecREST server.

Calls:

GET /status/systems

async systems() → List[dict]

Returns available systems.

Calls:

GET /status/systems

async nodes(system_name: str) → List[dict]

Returns nodes of the system.

Parameters:

system_name – the system name where the nodes belong to

Calls:

GET /status/{system_name}/nodes

async reservations(system_name: str) → List[dict]

Returns reservations defined in the system.

Parameters:

system_name – the system name where the reservations belong to

Calls:

GET /status/{system_name}/reservations

async partitions(system_name: str) → List[dict]

Returns partitions defined in the scheduler of the system.

Parameters:

system_name – the system name where the partitions belong to

Calls:

GET /status/{system_name}/partitions

async userinfo(system_name: str) → dict

Returns user and groups information.

Calls:

GET /status/{system_name}/userinfo

async list_files(system_name: str, path: str, show_hidden: bool = False, recursive: bool = False, numeric_uid: bool = False, dereference: bool = False) → List[dict]

Returns a list of files in a directory.

Parameters:

  • system_name – the system name where the filesystem belongs to

  • path – the absolute target path

  • show_hidden – Show hidden files

  • recursive – recursively list directories encountered

  • dereference – when showing file information for a symbolic link, show information for the file the link references rather than for the link itself

Numeric_uid:

list numeric user and group IDs

Calls:

GET /filesystem/{system_name}/ops/ls

async head(system_name: str, path: str, num_bytes: int | None = None, num_lines: int | None = None, exclude_trailing: bool = False) → dict

Display the beginning of a specified file. By default 10 lines will be returned. num_bytes and num_lines cannot be specified simultaneously.

Parameters:

  • system_name – the system name where the filesystem belongs to

  • path – the absolute target path of the file

  • num_bytes – the output will be the first NUM bytes of each file

  • num_lines – the output will be the first NUM lines of each file

  • exclude_trailing – the output will be the whole file, without the last NUM bytes/lines of each file. NUM should be specified in the respective argument through bytes or lines.

Calls:

GET /filesystem/{system_name}/ops/head

async tail(system_name: str, path: str, num_bytes: int | None = None, num_lines: int | None = None, exclude_beginning: bool = False) → dict

Display the ending of a specified file. By default, 10 lines will be returned. num_bytes and num_lines cannot be specified simultaneously.

Parameters:

  • system_name – the system name where the filesystem belongs to

  • path – the absolute target path of the file

  • num_bytes – The output will be the last NUM bytes of each file

  • num_lines – The output will be the last NUM lines of each file

  • exclude_beginning – The output will be the whole file, without the first NUM bytes/lines of each file. NUM should be specified in the respective argument through num_bytes or num_lines.

Calls:

GET /filesystem/{system_name}/ops/tail

async view(system_name: str, path: str) → str

View full file content (up to 5MB files)

Parameters:

  • system_name – the system name where the filesystem belongs to

  • path – the absolute target path of the file

Calls:

GET /filesystem/{system_name}/ops/view

async checksum(system_name: str, path: str) → dict

Calculate the SHA256 (256-bit) checksum of a specified file.

Parameters:

  • system_name – the system name where the filesystem belongs to

  • path – the absolute target path of the file

Calls:

GET /filesystem/{system_name}/ops/checksum

async file_type(system_name: str, path: str) → str

Uses the file linux application to determine the type of a file.

Parameters:

  • system_name – the system name where the filesystem belongs to

  • path – the absolute target path of the file

Calls:

GET /filesystem/{system_name}/ops/file

async chmod(system_name: str, path: str, mode: str) → dict

Changes the file mod bits of a given file according to the specified mode.

Parameters:

  • system_name – the system name where the filesystem belongs to

  • path – the absolute target path of the file

  • mode – same as numeric mode of linux chmod tool

Calls:

PUT /filesystem/{system_name}/ops/chmod

async chown(system_name: str, path: str, owner: str, group: str) → dict

Changes the user and/or group ownership of a given file. If only owner or group information is passed, only that information will be updated.

Parameters:

  • system_name – the system name where the filesystem belongs to

  • path – the absolute target path of the file

  • owner – owner ID for target

  • group – group ID for target

Calls:

PUT /filesystem/{system_name}/ops/chown

async stat(system_name: str, path: str, dereference: bool = False) → dict

Uses the stat linux application to determine the status of a file on the system’s filesystem. The result follows: https://docs.python.org/3/library/os.html#os.stat_result.

Parameters:

  • system_name – the system name where the filesystem belongs to

  • path – the absolute target path

  • dereference – follow symbolic links

Calls:

GET /filesystem/{system_name}/ops/stat

Create a symbolic link.

Parameters:

  • system_name – the system name where the filesystem belongs to

  • source_path – the absolute path to the file the link points to

  • link_path – the absolute path to the symlink

Calls:

POST /filesystem/{system_name}/ops/symlink

async mkdir(system_name: str, path: str, create_parents: bool = False) → dict

Create a directory.

Parameters:

  • system_name – the system name where the filesystem belongs to

  • path – the absolute path to the new directory

  • create_parents – create intermediate parent directories

Calls:

POST /filesystem/{system_name}/ops/mkdir

async mv(system_name: str, source_path: str, target_path: str, account: str | None = None, blocking: bool = True, timeout: float | None = None) → dict

Rename/move a file, directory, or symlink at the source_path to the target_path on system_name’s filesystem. This operation runs in a job.

Parameters:

  • system_name – the system name where the filesystem belongs to

  • source_path – the absolute source path

  • target_path – the absolute target path

  • account – the account to be used for the transfer job

  • blocking – whether to wait for the job to complete

  • timeout – the maximum time to wait for the job to complete

Calls:

POST /filesystem/{system_name}/transfer/mv

async compress(system_name: str, source_path: str, target_path: str, dereference: bool = False, compression: str = 'gzip', match_pattern: str | None = None, account: str | None = None, blocking: bool = True) → None

Compress a directory or file.

Parameters:

  • system_name – the system name where the filesystem belongs to

  • source_path – the absolute path to source directory

  • target_path – the absolute path to the newly created compressed file

  • dereference – dereference links when compressing

  • compression – compression algorithm to use (one of “none”, “gzip”, “bzip2”, “xz”). Default is “gzip”.

  • match_pattern – regex pattern to filter files to compress

  • account – the account to be used for the transfer job (only relevant when the compression operation is not done directly)

  • blocking – whether to wait for the job to complete (only relevant when the compression operation is not done directly)

Calls:

POST /filesystem/{system_name}/ops/compress

POST /filesystem/{system_name}/transfer/compress

GET /jobs/{system_name}/{job_id}

async extract(system_name: str, source_path: str, target_path: str, compression: str = 'gzip', account: str | None = None, blocking: bool = True) → None

Extract compressed archives.

Parameters:

  • system_name – the system name where the filesystem belongs to

  • source_path – the absolute path to the archive

  • target_path – the absolute path to target directory

  • compression – compression algorithm to use (one of “none”, “gzip”, “bzip2”, “xz”). Default is “gzip”.

  • account – the account to be used for the transfer job (only relevant when the extract operation is not done directly)

  • blocking – whether to wait for the job to complete (only relevant when the extract operation is not done directly)

Calls:

POST /filesystem/{system_name}/ops/extract

POST /filesystem/{system_name}/transfer/extract

GET /jobs/{system_name}/{job_id}

async wait_for_job(system_name: str, job_id: str, timeout: float | None = None, not_found_timeout: float = 20.0) → List[Any]

Wait for a job to complete. When the job is completed, it will return the job information. :param system_name: the system name where the filesystem belongs to :param job_id: the ID of the job to wait for :param timeout: the maximum time to wait for the job to complete

in seconds. If the timeout is set and the job is not completed within this time, it will be cancelled.

Parameters:

not_found_timeout – the maximum time to wait for the job to be available in the scheduler, in seconds. Slurm or other scheduler will need a few seconds before updating the database. If the job is not found within this time, the function will raise an exception.

Calls:

GET /jobs/{system_name}/{job_id}

async cp(system_name: str, source_path: str, target_path: str, dereference: bool = False, account: str | None = None, blocking: bool = True, timeout: float | None = None) → dict

Copies file from source_path to target_path.

Parameters:

  • system_name – the system name where the filesystem belongs to

  • source_path – the absolute source path

  • target_path – the absolute target path

  • dereference – dereference links when copying

  • blocking – whether to wait for the job to complete

  • account – the account to be used for the transfer job

  • timeout – the maximum time to wait for the job to complete

Calls:

POST /filesystem/{system_name}/transfer/cp

async rm(system_name: str, path: str, account: str | None = None, blocking: bool = True, timeout: float | None = None) → dict | None

Delete a file. First the client will try to delete the file directly, and if it fails with a timeout, it will launch a job to delete the file.

Parameters:

  • system_name – the system name where the filesystem belongs to

  • path – the absolute target path

  • account – the account to be used for the transfer job (only relevant when the file is not deleted directly)

  • blocking – whether to wait for the job to complete (only relevant when the file is not deleted directly)

  • timeout – the maximum time to wait for the job to complete

Calls:

DELETE /filesystem/{system_name}/ops/rm

DELETE /filesystem/{system_name}/transfer/rm

GET /jobs/{system_name}/{job_id}

async upload(system_name: str, local_file: str | pathlib.Path, directory: str, filename: str, account: str | None = None, blocking: bool = True, transfer_method: str = 'wormhole') → AsyncExternalUpload | None

Upload a file to the system. Small files will be uploaded directly to FirecREST and will be immediately available. The function will return None in this case. Large files will be uploaded in parts to the staging area of FirecREST and then moved to the target directory in a job. The function will return the transfer job information in this case.

Parameters:

  • system_name – the system name where the filesystem belongs to

  • local_file – the local file’s path to be uploaded (can be relative)

  • directory – the absolut target path of the directory where the file will be uploaded

  • filename – the name of the file in the target directory

  • account – the account to be used for the transfer job (only relevant when the file is larger than MAX_DIRECT_UPLOAD_SIZE)

  • blocking – whether to wait for the job to complete (only relevant when the file is larger than MAX_DIRECT_UPLOAD_SIZE)

  • transfer_method – the method to be used for the upload of large files. Supported methods: “s3”, “streamer”, “wormhole”.

Calls:

POST /filesystem/{system_name}/transfer/upload

async download(system_name: str, source_path: str, target_path: str | pathlib.Path, account: str | None = None, blocking: bool = True, transfer_method: str = 'wormhole') → AsyncExternalDownload | None

Download a file from the remote system.

Parameters:

  • system_name – the system name where the filesystem belongs to

  • source_path – the absolute source path of the file

  • target_path – the target path in the local filesystem (can be relative path)

  • account – the account to be used for the transfer job (only relevant when the file is larger than MAX_DIRECT_UPLOAD_SIZE)

  • blocking – whether to wait for the job to complete

  • transfer_method – the method to be used for the download of large files. Supported methods: “s3”, “streamer”, “wormhole”.

Calls:

POST /filesystem/{system_name}/transfer/download

async submit(system_name: str, working_dir: str, script_str: str | None = None, script_local_path: str | None = None, script_remote_path: str | None = None, env_vars: dict[str, str] | None = None, account: str | None = None) → dict

Submit a job.

Parameters:

  • system_name – the system name where the filesystem belongs to

  • working_dir – the working directory of the job

  • script_str – the job script

  • script_local_path – path to the job script

  • script_remote_path – path to the job script on the remote filesystem

  • env_vars – environment variables to be set before running the job

  • account – the account to be used for the job

Calls:

POST /compute/{system_name}/jobs

async job_info(system_name: str, jobid: str | None = None, allusers: bool = False, account: str | None = None) → list

Get job information. When the job is not specified, it will return all the jobs.

Parameters:

  • system_name – the system name where the filesystem belongs to

  • jobid – the ID of the job

  • allusers – whether to return jobs of all users or only the current user

  • account – an account to filter the jobs by. It will only be taken into account when you are not specifying a jobid.

Calls:

GET /compute/{system_name}/jobs or GET /compute/{system_name}/jobs/{job}

async job_metadata(system_name: str, jobid: str) → dict

Get job metadata.

Parameters:

  • system_name – the system name where the filesystem belongs to

  • jobid – the ID of the job

Calls:

GET /compute/{system_name}/jobs/{jobid}/metadata

async cancel_job(system_name: str, jobid: str) → dict

Cancel a job.

Parameters:

  • system_name – the system name where the filesystem belongs to

  • jobid – the ID of the job to be cancelled

Calls:

DELETE /compute/{system_name}/jobs/{jobid}

async attach_to_job(system_name: str, jobid: str, command: str) → dict

Attach a process to a job.

Parameters:

  • system_name – the system name where the filesystem belongs to

  • jobid – the ID of the job

  • command – the command to be executed

Calls:

PUT /compute/{system_name}/jobs/{jobid}/attach

The AsyncExternalUpload class

class firecrest.v2.AsyncExternalUpload(client, transfer_info, local_file)

Bases: AsyncExternalTransfer

property transfer_data
property wormhole_code
async send_file_wormhole()
async upload_file_to_stage()
async upload_file_streamer()

The AsyncExternalDownload class

class firecrest.v2.AsyncExternalDownload(client, transfer_info, file_path)

Bases: AsyncExternalTransfer

property transfer_data
property wormhole_code
async receive_file_wormhole()
async download_file_from_stage(file_path=None)
async download_file_streamer(file_path=None)