Asynchronous FirecREST objects
The library also provides an asynchronous API for the client:
The AsyncFirecrest
class
- class firecrest.AsyncFirecrest(firecrest_url: str, authorization: Any, verify: str | bool | None = None, sa_role: str = 'firecrest-sa')
Bases:
object
This is the basic class you instantiate to access the FirecREST API v1. Necessary parameters are the firecrest URL and an authorization object.
- Parameters:
firecrest_url – FirecREST’s URL
authorization – the authorization object. This object is responsible of handling the credentials and the only requirement for it is that it has a method get_access_token() that returns a valid access token.
verify – either a boolean, in which case it controls whether requests will verify the server’s TLS certificate, or a string, in which case it must be a path to a CA bundle to use
sa_role – this corresponds to the F7T_AUTH_ROLE configuration parameter of the site. If you don’t know how FirecREST is setup it’s better to leave the default.
- TOO_MANY_REQUESTS_CODE = 429
- timeout: Any
This attribute will be passed to all the requests that will be made. How many seconds to wait for the server to send data before giving up. After that time a requests.exceptions.Timeout error will be raised.
It can be a float or a tuple. More details here: https://www.python-httpx.org/advanced/#fine-tuning-the-configuration.
- num_retries_rate_limit: int | None
Number of retries in case the rate limit is reached. When it is set to None, the client will keep trying until it gets a different status code than 429.
- time_between_calls: dict[str, float]
Seconds between requests in each microservice
- set_api_version(api_version: str) None
Set the version of the api of firecrest. By default it will be assumed that you are using version 1.13.1 or compatible. The version is parsed by the packaging library.
- async all_services() List[Service]
Returns a list containing all available micro services with a name, description, and status.
- Calls:
GET /status/services
- async service(service_name: str) Service
Returns information about a micro service. Returns the name, description, and status.
- Parameters:
service_name – the service name
- Calls:
GET /status/services/{service_name}
- async all_systems() List[System]
Returns a list containing all available systems and response status.
- Calls:
GET /status/systems
- async system(system_name: str) System
Returns information about a system. Returns the name, description, and status.
- Parameters:
system_name – the system name
- Calls:
GET /status/systems/{system_name}
- async parameters() Parameters
Returns configuration parameters of the FirecREST deployment that is associated with the client.
- Calls:
GET /status/parameters
- async filesystems(system_name: str | None = None) dict[str, List[t.Filesystem]]
Returns the status of the filesystems per system.
- Parameters:
system_name – the system name
- Calls:
GET /status/filesystems
- Calls:
GET /status/filesystems/{system_name}
Warning
This is available only for FirecREST>=1.15.0
- async list_files(machine: str, target_path: str, show_hidden: bool = False) List[LsFile]
Returns a list of files in a directory.
- Parameters:
machine – the machine name where the filesystem belongs to
target_path – the absolute target path
show_hidden – show hidden files
- Calls:
GET /utilities/ls
- async mkdir(machine: str, target_path: str, p: bool | None = None) str
Creates a new directory. When successful, the method returns a string with the path of the newly created directory.
- Parameters:
machine – the machine name where the filesystem belongs to
target_path – the absolute target path
p – no error if existing, make parent directories as needed
- Calls:
POST /utilities/mkdir
- async mv(machine: str, source_path: str, target_path: str) str
Rename/move a file, directory, or symlink at the source_path to the target_path on machine’s filesystem. When successful, the method returns a string with the new path of the file.
- Parameters:
machine – the machine name where the filesystem belongs to
source_path – the absolute source path
target_path – the absolute target path
- Calls:
PUT /utilities/rename
- async chmod(machine: str, target_path: str, mode: str) None
Changes the file mod bits of a given file according to the specified mode.
- Parameters:
machine – the machine name where the filesystem belongs to
target_path – the absolute target path
mode – same as numeric mode of linux chmod tool
- Calls:
PUT /utilities/chmod
- async chown(machine: str, target_path: str, owner: str | None = None, group: str | None = None) None
Changes the user and/or group ownership of a given file. If only owner or group information is passed, only that information will be updated.
- Parameters:
machine – the machine name where the filesystem belongs to
target_path – the absolute target path
owner – owner ID for target
group – group ID for target
- Calls:
PUT /utilities/chown
- async copy(machine: str, source_path: str, target_path: str) str
Copies file from source_path to target_path. When successful, the method returns a string with the path of the newly created file.
- Parameters:
machine – the machine name where the filesystem belongs to
source_path – the absolute source path
target_path – the absolute target path
- Calls:
POST /utilities/copy
- async file_type(machine: str, target_path: str) str
Uses the file linux application to determine the type of a file.
- Parameters:
machine – the machine name where the filesystem belongs to
target_path – the absolute target path
- Calls:
GET /utilities/file
- async stat(machine: str, target_path: str, dereference: bool = False) StatFile
Uses the stat linux application to determine the status of a file on the machine’s filesystem. The result follows: https://docs.python.org/3/library/os.html#os.stat_result.
- Parameters:
machine – the machine name where the filesystem belongs to
target_path – the absolute target path
dereference – follow link (default False)
- Calls:
GET /utilities/stat
- async symlink(machine: str, target_path: str, link_path: str) str
Creates a symbolic link. When successful, the method returns a string with the path of the newly created link.
- Parameters:
machine – the machine name where the filesystem belongs to
target_path – the absolute path that the symlink will point to
link_path – the absolute path to the new symlink
- Calls:
POST /utilities/symlink
- async simple_download(machine: str, source_path: str, target_path: str | pathlib.Path | BytesIO) None
Blocking call to download a small file. The maximun size of file that is allowed can be found from the parameters() call.
- Parameters:
machine – the machine name where the filesystem belongs to
source_path – the absolute source path
target_path – the target path in the local filesystem or binary stream
- Calls:
GET /utilities/download
- async simple_upload(machine: str, source_path: str | pathlib.Path | BytesIO, target_path: str, filename: str | None = None) None
Blocking call to upload a small file. The file that will be uploaded will have the same name as the source_path. The maximum size of file that is allowed can be found from the parameters() call.
- Parameters:
machine – the machine name where the filesystem belongs to
source_path – the source path of the file or binary stream
target_path – the absolute target path of the directory where the file will be uploaded
filename – naming target file to filename (default is same as the local one)
- Calls:
POST /utilities/upload
- async simple_delete(machine: str, target_path: str) None
Blocking call to delete a small file.
- Parameters:
machine – the machine name where the filesystem belongs to
target_path – the absolute target path
- Calls:
DELETE /utilities/rm
- async checksum(machine: str, target_path: str) str
Calculate the SHA256 (256-bit) checksum of a specified file.
- Parameters:
machine – the machine name where the filesystem belongs to
target_path – the absolute target path
- Calls:
GET /utilities/checksum
- async head(machine: str, target_path: str, bytes: str | None = None, lines: str | None = None, skip_ending: bool | None = False) str
Display the beginning of a specified file. By default 10 lines will be returned. Bytes and lines cannot be specified simultaneously. The final result will be smaller than UTILITIES_MAX_FILE_SIZE bytes. This variable is available in the parameters command.
- Parameters:
machine – the machine name where the filesystem belongs to
target_path – the absolute target path
bytes – the number of bytes to be displayed
lines – the number of lines to be displayed
skip_ending – the output will be the whole file, without the last NUM bytes/lines of each file. NUM should be specified in the respective argument through bytes or lines. Equivalent to passing -NUM to the head command.
- Calls:
GET /utilities/head
- async tail(machine: str, target_path: str, bytes: str | None = None, lines: str | None = None, skip_beginning: bool | None = False) str
Display the last part of a specified file. By default 10 lines will be returned. Bytes and lines cannot be specified simultaneously. The final result will be smaller than UTILITIES_MAX_FILE_SIZE bytes. This variable is available in the parameters command.
- Parameters:
machine – the machine name where the filesystem belongs to
target_path – the absolute target path
bytes – the number of bytes to be displayed
lines – the number of lines to be displayed
skip_beginning – the output will start with byte/line NUM of each file. NUM should be specified in the respective argument through bytes or lines. Equivalent to passing +NUM to the tail command.
- Calls:
GET /utilities/head
- async view(machine: str, target_path: str) str
View the content of a specified file. The final result will be smaller than UTILITIES_MAX_FILE_SIZE bytes. This variable is available in the parameters command.
- Parameters:
machine – the machine name where the filesystem belongs to
target_path – the absolute target path
- Calls:
GET /utilities/checksum
- async whoami(machine=None) str | None
Returns the username that FirecREST will be using to perform the other calls. In the case the machine name is passed in the arguments, a call is made to the respective endpoint and the command whoami is run on the machine. Otherwise, the library decodes the token and will return None if the token is not valid.
- Calls:
GET /utilities/whoami
- async groups(machine) UserId
Returns the output of the id command, user and group ids.
- Calls:
GET /utilities/whoami
Warning
This is available only for FirecREST>=1.15.0
- async submit(machine: str, job_script: str | None = None, local_file: bool | None = True, script_str: str | None = None, script_local_path: str | None = None, script_remote_path: str | None = None, account: str | None = None, env_vars: dict[str, Any] | None = None) t.JobSubmit
Submits a batch script to SLURM on the target system. One of script_str, script_local and script_remote needs to be set.
- Parameters:
machine – the machine name where the scheduler belongs to
job_script – [deprecated] use script_str, script_local_path or script_remote_path
local_file – [deprecated]
script_str – the content of the script to be submitted
script_local_path – the path of the script on the local file system
script_remote_path – the full path of the script on the remote file system
account – submit the job with this project account
env_vars – dictionary (varName, value) defining environment variables to be exported for the job
- Calls:
POST /compute/jobs/upload or POST /compute/jobs/path
GET /tasks/{taskid}
- async poll(machine: str, jobs: Sequence[str | int] | None = None, start_time: str | None = None, end_time: str | None = None, page_size: int | None = None, page_number: int | None = None) List[t.JobAcct]
Retrieves information about submitted jobs. This call uses the sacct command.
- Parameters:
machine – the machine name where the scheduler belongs to
jobs – list of the IDs of the jobs
start_time – Start time (and/or date) of job’s query. Allowed formats are HH:MM[:SS] [AM|PM] MMDD[YY] or MM/DD[/YY] or MM.DD[.YY] MM/DD[/YY]-HH:MM[:SS] YYYY-MM-DD[THH:MM[:SS]]
end_time – End time (and/or date) of job’s query. Allowed formats are HH:MM[:SS] [AM|PM] MMDD[YY] or MM/DD[/YY] or MM.DD[.YY] MM/DD[/YY]-HH:MM[:SS] YYYY-MM-DD[THH:MM[:SS]]
page_size – number of entries returned (when page_number is not None, the default value is 25)
page_number – page number (if set to None the default value is 0)
- Calls:
GET /compute/acct
GET /tasks/{taskid}
- async poll_active(machine: str, jobs: Sequence[str | int] | None = None, page_size: int | None = None, page_number: int | None = None) List[t.JobQueue]
Retrieves information about active jobs. This call uses the squeue -u <username> command.
- Parameters:
machine – the machine name where the scheduler belongs to
jobs – list of the IDs of the jobs
page_size – number of entries returned (when page_number is not None, the default value is 25)
page_number – page number (if set to None the default value is 0)
- Calls:
GET /compute/jobs
GET /tasks/{taskid}
- async cancel(machine: str, job_id: str | int) str
Cancels running job. This call uses the scancel command.
- Parameters:
machine – the machine name where the scheduler belongs to
job_id – the ID of the job
- Calls:
DELETE /compute/jobs/{job_id}
GET /tasks/{taskid}
- async submit_move_job(machine: str, source_path: str, target_path: str, job_name: str | None = None, time: str | None = None, stage_out_job_id: str | None = None, account: str | None = None) JobSubmit
Move files between internal CSCS file systems. Rename/Move source_path to target_path. Possible to stage-out jobs providing the SLURM ID of a production job. More info about internal transfer: https://user.cscs.ch/storage/data_transfer/internal_transfer/
- Parameters:
machine – the machine name where the scheduler belongs to
source_path – the absolute source path
target_path – the absolute target path
job_name – job name
time – limit on the total run time of the job. Acceptable time formats ‘minutes’, ‘minutes:seconds’, ‘hours:minutes:seconds’, ‘days-hours’, ‘days-hours:minutes’ and ‘days-hours:minutes:seconds’.
stage_out_job_id – transfer data after job with ID {stage_out_job_id} is completed
account – name of the bank account to be used in SLURM. If not set, system default is taken.
- Calls:
POST /storage/xfer-internal/mv
GET /tasks/{taskid}
- async submit_copy_job(machine: str, source_path: str, target_path: str, job_name: str | None = None, time: str | None = None, stage_out_job_id: str | None = None, account: str | None = None) JobSubmit
Copy files between internal CSCS file systems. Copy source_path to target_path. Possible to stage-out jobs providing the SLURM Id of a production job. More info about internal transfer: https://user.cscs.ch/storage/data_transfer/internal_transfer/
- Parameters:
machine – the machine name where the scheduler belongs to
source_path – the absolute source path
target_path – the absolute target path
job_name – job name
time – limit on the total run time of the job. Acceptable time formats ‘minutes’, ‘minutes:seconds’, ‘hours:minutes:seconds’, ‘days-hours’, ‘days-hours:minutes’ and ‘days-hours:minutes:seconds’.
stage_out_job_id – transfer data after job with ID {stage_out_job_id} is completed
account – name of the bank account to be used in SLURM. If not set, system default is taken.
- Calls:
POST /storage/xfer-internal/cp
GET /tasks/{taskid}
- async submit_rsync_job(machine: str, source_path: str, target_path: str, job_name: str | None = None, time: str | None = None, stage_out_job_id: str | None = None, account: str | None = None) JobSubmit
Transfer files between internal CSCS file systems. Transfer source_path to target_path. Possible to stage-out jobs providing the SLURM Id of a production job. More info about internal transfer: https://user.cscs.ch/storage/data_transfer/internal_transfer/
- Parameters:
machine – the machine name where the scheduler belongs to
source_path – the absolute source path
target_path – the absolute target path
job_name – job name
time – limit on the total run time of the job. Acceptable time formats ‘minutes’, ‘minutes:seconds’, ‘hours:minutes:seconds’, ‘days-hours’, ‘days-hours:minutes’ and ‘days-hours:minutes:seconds’.
stage_out_job_id – transfer data after job with ID {stage_out_job_id} is completed
account – name of the bank account to be used in SLURM. If not set, system default is taken.
- Calls:
POST /storage/xfer-internal/rsync
GET /tasks/{taskid}
- async submit_delete_job(machine: str, target_path: str, job_name: str | None = None, time: str | None = None, stage_out_job_id: str | None = None, account: str | None = None) JobSubmit
Remove files in internal CSCS file systems. Remove file in target_path. Possible to stage-out jobs providing the SLURM Id of a production job. More info about internal transfer: https://user.cscs.ch/storage/data_transfer/internal_transfer/
- Parameters:
machine – the machine name where the scheduler belongs to
target_path – the absolute target path
job_name – job name
time – limit on the total run time of the job. Acceptable time formats ‘minutes’, ‘minutes:seconds’, ‘hours:minutes:seconds’, ‘days-hours’, ‘days-hours:minutes’ and ‘days-hours:minutes:seconds’.
stage_out_job_id – transfer data after job with ID {stage_out_job_id} is completed
account – name of the bank account to be used in SLURM. If not set, system default is taken.
- Calls:
POST /storage/xfer-internal/rm
GET /tasks/{taskid}
- async external_upload(machine: str, source_path: str, target_path: str) AsyncExternalUpload
Non blocking call for the upload of larger files.
- Parameters:
machine – the machine where the filesystem belongs to
source_path – the source path in the local filesystem
target_path – the target path in the machine’s filesystem
- Returns:
an ExternalUpload object
- async external_download(machine: str, source_path: str) AsyncExternalDownload
Non blocking call for the download of larger files.
- Parameters:
machine – the machine where the filesystem belongs to
source_path – the source path in the local filesystem
- Returns:
an ExternalDownload object
- async all_reservations(machine: str) List[dict]
List all active reservations and their status
- Parameters:
machine – the machine name
- Calls:
GET /reservations
- async create_reservation(machine: str, reservation: str, account: str, number_of_nodes: str, node_type: str, start_time: str, end_time: str) None
Creates a new reservation with {reservation} name for a given SLURM groupname
- Parameters:
machine – the machine name
reservation – the reservation name
account – the account in SLURM to which the reservation is made for
number_of_nodes – number of nodes needed for the reservation
node_type – type of node
start_time – start time for reservation (YYYY-MM-DDTHH:MM:SS)
end_time – end time for reservation (YYYY-MM-DDTHH:MM:SS)
- Calls:
POST /reservations
- async update_reservation(machine: str, reservation: str, account: str, number_of_nodes: str, node_type: str, start_time: str, end_time: str) None
Updates an already created reservation named {reservation}
- Parameters:
machine – the machine name
reservation – the reservation name
account – the account in SLURM to which the reservation is made for
number_of_nodes – number of nodes needed for the reservation
node_type – type of node
start_time – start time for reservation (YYYY-MM-DDTHH:MM:SS)
end_time – end time for reservation (YYYY-MM-DDTHH:MM:SS)
- Calls:
PUT /reservations/{reservation}
- async delete_reservation(machine: str, reservation: str) None
Deletes an already created reservation named {reservation}
- Parameters:
machine – the machine name
reservation – the reservation name
- Calls:
DELETE /reservations/{reservation}
The AsyncExternalDownload
class
- class firecrest.AsyncExternalDownload(client: AsyncFirecrest, task_id: str, previous_responses: List[requests.Response] | None = None)
Bases:
AsyncExternalStorage
This class handles the external download from a file.
Tracks the progress of the download through the status of the associated task. Final states: 117 and 118.
Status
Description
116
Started upload from filesystem to Object Storage
117
Upload from filesystem to Object Storage has finished successfully
118
Upload from filesystem to Object Storage has finished with errors
- Parameters:
client – FirecREST client associated with the transfer
task_id – FirecrREST task associated with the transfer
- async invalidate_object_storage_link() None
Invalidate the temporary URL for downloading.
- Calls:
POST /storage/xfer-external/invalidate
- property object_storage_link: str
Get the direct download url for the file. The response from the FirecREST api changed after version 1.13.0, so make sure to set to older version, if you are using an older deployment.
- Calls:
GET /tasks/{taskid}
- property client: AsyncFirecrest
Returns the client that will be used to get information for the task.
- property data: dict | None
Returns the task information from the latest response.
- Calls:
GET /tasks/{taskid}
- async finish_download(target_path: str | pathlib.Path | BufferedWriter) None
Finish the download process.
- Parameters:
target_path – the local path to save the file
- property in_progress: bool
Returns False when the transfer has been completed (succesfully or with errors), otherwise True.
- Calls:
GET /tasks/{taskid}
- property object_storage_data
Returns the necessary information for the external transfer. The call is blocking and in cases of large file transfers it might take a long time.
- Calls:
GET /tasks/{taskid}
- Return type:
dictionary or string
- property status: str
Returns status of the task that is associated with this transfer.
- Calls:
GET /tasks/{taskid}
- property task_id: str
Returns the FirecREST task ID that is associated with this transfer.
The AsyncExternalUpload
class
- class firecrest.AsyncExternalUpload(client: AsyncFirecrest, task_id: str, previous_responses: List[requests.Response] | None = None)
Bases:
AsyncExternalStorage
This class handles the external upload from a file.
Tracks the progress of the upload through the status of the associated task. Final states: 114 and 115.
Status
Description
110
Waiting for Form URL from Object Storage to be retrieved
111
Form URL from Object Storage received
112
Object Storage confirms that upload to Object Storage has finished
113
Download from Object Storage to server has started
114
Download from Object Storage to server has finished
115
Download from Object Storage error
- Parameters:
client – FirecREST client associated with the transfer
task_id – FirecrREST task associated with the transfer
- async finish_upload() None
Finish the upload process. This call will upload the file to the staging area. Check with the method status or in_progress to see the status of the transfer. The transfer from the staging area to the systems’s filesystem can take several seconds to start to start.
- property client: AsyncFirecrest
Returns the client that will be used to get information for the task.
- property data: dict | None
Returns the task information from the latest response.
- Calls:
GET /tasks/{taskid}
- property in_progress: bool
Returns False when the transfer has been completed (succesfully or with errors), otherwise True.
- Calls:
GET /tasks/{taskid}
- property object_storage_data
Returns the necessary information for the external transfer. The call is blocking and in cases of large file transfers it might take a long time.
- Calls:
GET /tasks/{taskid}
- Return type:
dictionary or string
- property status: str
Returns status of the task that is associated with this transfer.
- Calls:
GET /tasks/{taskid}
- property task_id: str
Returns the FirecREST task ID that is associated with this transfer.