Work with environment variables and resource monitoring
An environment is the infrastructure used by the platform to store data and run computation. Once created, you can start working in the environments by creating pipelines and running them in the platform.
In addition, you can create and save environment variables that will allow you to parameterize certain variables in order to call them when running or deploying a pipeline.
| Function name | Method | Return type | Description |
|---|---|---|---|
| create_or_update_environment_variable | CraftAiSdk.create_or_update_environment_variable (environment_variable_name, environment_variable_value) | dict | To create or update an environment variable available for all pipelines executions. |
| list_environment_variables | CraftAiSdk.list_environment_variables() | List of dict | Get a list of all environments variables in the current environment. |
| delete_environment_variable | CraftAiSdk.delete_environment_variable(environment_variable_name) | dict | Delete a specified environment variable. |
Set up an environment variable
An environment variable is a value that can be passed to your pipeline code. It is used to store information that may be needed by the operating system or by applications that run on the platform (for example, by endpoints you deploy on the platform).
Tip
On this page, we will detail the usage of the SDK, but environment variables can be modified from a page in the web UI by going to: Environments > click on the three dots of the desired environment > Settings > Environment variables.
Create and update an environment variable
To create or update an environment variable available for all pipelines executions.
CraftAiSdk.create_or_update_environment_variable(environment_variable_name,
environment_variable_value)
Parameters
environment_variable_name(str) – Name of the environment variable to create.environment_variable_value(str) – Value of the environment variable to create.
Returns
A dict object containing the ID of environment variable (with keys “id”)
Get the list of environment variables
Get the list of all environment variables in the current environment.
Parameter
No parameter
Returns
List of dicts of environment variables (with keys “name” and “value”)
Delete an environment variable
Delete a specified environment variable.
Parameter
environment_variable_name(str) – Name of the environment variable to delete.
Returns
Dict (with keys “name” and “value”) of the deleted environment variable
Get resource metrics
Get resource metrics of the environment into dict format (by default) or in a .csv file. You can view these metrics in graph format using the Resource Metrics web page.
Parameter
start_date(datetime.datetime) - The beginning of the period.end_date(datetime.datetime) - The end of the period.csv(bool) - If True, it will return a csv file as bytes.
Returns
If csv is True, it will return bytes. Otherwise, dict with:
The resource metrics, with the following keys:
additional_data(dict): Additional data with the following keys:total_disk(int): Total disk size in bytes.total_ram(int): Total RAM size in bytes.total_vram(int): Total VRAM size in bytes if there is a GPU.cpu_usage(list of dict): The CPU usage in percent.disk_usage(list of dict): The disk usage in percent.ram_usage(list of dict): The RAM usage in percent.vram_usage(list of dict): The VRAM usage in percent if there is a GPU.gpu_usage(list of dict): The GPU usage in percent if there is a GPU.network_input_usage(list of dict): The network input usage in bytes.network_output_usage(list of dict): The network output usage in bytes.
Each element of the lists is a dict with the following keys:
metric(dict): Dictionary with the following key:worker(str): The worker name.
values(list of list): The values of the metrics in the following format: [[timestamp, value], ...].
Example with dict object
from datetime import datetime, timedelta
# Get the current time
current_time = datetime.now()
# Calculate the beginning of the period (last 2 hours)
begin_date = current_time - timedelta(minutes=2)
# Set end_date to the current time
end_date = current_time
# Print the results
print("begin_date:", begin_date)
print("end_date:", end_date)
res = sdk.get_resource_metrics(begin_date, end_date)
print (res)
Example of return object:
{'metrics': {'cpu_usage': [{'metric': {'worker': 'Worker-58d65a25'},
'values': [[1707132918930, 0.00464583333348214],
[1707132948930, 0.004708333333450027],
[1707132978930, 0.004763888888882371],
[1707133008930, 0.0047222222219369045],
[1707133038930, 0.004722222221995642]]},
{'metric': {'worker': 'Worker-6e32691a'},
'values': [[1707132918930, 0.007868055555652481],
[1707132948930, 0.007638888888662494],
[1707132978930, 0.00754861111112385],
[1707133008930, 0.0076597222224098135],
[1707133038930, 0.007930555555419681]]},
...
[1707132948930, 40546.24444444444],
[1707132978930, 48481.6111111111],
[1707133008930, 40639.65555555555],
[1707133038930, 48189.03333333333]]},
{'metric': {'worker': 'All cumulate'},
'values': [[1707132918930, 63759.26666666666],
[1707132948930, 56443.922222222216],
[1707132978930, 64879.22222222221],
[1707133008930, 46783.055555555555],
[1707133038930, 53954.188888888886]]}]},
'additional_data': {'total_ram': 16616214528, 'total_disk': 21462233088}}
Example with CSV file
from datetime import datetime, timedelta
# Get the current time
current_time = datetime.now()
# Calculate the beginning of the period (last 2 hours)
begin_date = current_time - timedelta(minutes=2)
# Set end_date to the current time
end_date = current_time
# Print the results
print("begin_date:", begin_date)
print("end_date:", end_date)
res = sdk.get_resource_metrics(begin_date, end_date,csv=True)
result_csv = res.decode('utf-8')
# Save the result to a CSV file
csv_filename = "resource_metrics.csv"
with open(csv_filename, "w") as csv_file:
csv_file.write(result_csv)
Example resource_metrics.csv file:
| timestamp | worker | CPU | RAM_percent | RAM | disk_percent | disk | network_input | network_output | |
|---|---|---|---|---|---|---|---|---|---|
| 1707132918930 | Worker-58d65a25 | 0.00464583333348214 | 0.0614234303655712 | 0.950531005859375 | 0.403168019866396 | 8.05862808227539 | 14646.7222222222 | 15505.5777777778 | |
| 1707132948930 | Worker-58d65a25 | 0.00470833333345003 | 0.0615619668532965 | 0.952674865722656 | 0.403168019866396 | 8.05862808227539 | 15029.0666666667 | 15897.6777777778 | |
| 1707132978930 | Worker-58d65a25 | 0.00476388888888237 | 0.0612654198875784 | 0.948085784912109 | 0.403217830899368 | 8.05962371826172 | 15577.8777777778 | 16397.6111111111 | |
| 1707133008930 | Worker-58d65a25 | 0.00472222222193691 | 0.061079554208317 | 0.945209503173829 | 0.40321668581815 | 8.05960083007813 | 5007 | 6143.4 | |
| 1707133038930 | Worker-58d65a25 | 0.00472222222199564 | 0.0612341135994269 | 0.947601318359374 | 0.40321668581815 | 8.05960083007813 | 4627.8 | 5765.15555555556 | |
| 1707132918930 | Worker-6e32691a | 0.00786805555565248 | 0.0948272398231762 | 1.46745681762695 | 0.831449961000442 | 16.6192398071289 | 47515.6111111111 | 48253.6888888889 | |
| 1707132948930 | Worker-6e32691a | 0.00763888888866249 | 0.0949657763109015 | 1.46960067749023 | 0.832236059256426 | 16.634952545166 | 39472.1555555555 | 40546.2444444444 | |
| 1707132978930 | Worker-6e32691a | 0.00754861111112385 | 0.0951188566647759 | 1.47196960449219 | 0.832231097237816 | 16.6348533630371 | 47657.8444444444 | 48481.6111111111 | |
| 1707133008930 | Worker-6e32691a | 0.00765972222240981 | 0.0950882898952422 | 1.47149658203125 | 0.832231097237816 | 16.6348533630371 | 39568.6777777778 | 40639.6555555556 |