Work with environment variables and resource monitoring

An environment is the infrastructure used by the platform to store data and run computation. Once created, you can start working in the environments by creating pipelines and running them in the platform.

In addition, you can create and save environment variables that will allow you to parameterize certain variables in order to call them when running or deploying a pipeline.

Function name	Method	Return type	Description
create_or_update_environment_variable	CraftAiSdk.create_or_update_environment_variable (environment_variable_name, environment_variable_value)	dict	To create or update an environment variable available for all pipelines executions.
list_environment_variables	CraftAiSdk.list_environment_variables()	List of dict	Get a list of all environments variables in the current environment.
delete_environment_variable	CraftAiSdk.delete_environment_variable(environment_variable_name)	dict	Delete a specified environment variable.

Set up an environment variable

An environment variable is a value that can be passed to your step code. It is used to store information that may be needed by the operating system or by applications that run on the platform (for example, by endpoints you deploy on the platform).

Create and update an environment variable

To create or update an environment variable available for all pipelines executions.

CraftAiSdk.create_or_update_environment_variable(environment_variable_name,
environment_variable_value)

Parameters

environment_variable_name (str) – Name of the environment variable to create.
environment_variable_value (str) – Value of the environment variable to create.

Returns

A dict object containing the ID of environment variable (with keys “id”)

Get the list of environment variables

Get the list of all environment variables in the current environment.

CraftAiSdk.list_environment_variables()

Parameter

No parameter

Returns

List of dicts of environment variables (with keys “name” and “value”)

Delete an environment variable

Delete a specified environment variable.

CraftAiSdk.delete_environment_variable(environment_variable_name)

Parameter

environment_variable_name (str) – Name of the environment variable to delete.

Returns

Dict (with keys “name” and “value”) of the deleted environment variable

Get resource metrics

Get resource metrics of the environment into dict format (by default) or in a .csv file. You can view these metrics in graph format using the Resource Metrics web page.

CraftAiSdk.get_resource_metrics(start_date, end_date, csv=False)

Parameter

start_date (datetime.datetime) - The beginning of the period.
end_date (datetime.datetime) - The end of the period.
csv (bool) - If True, it will return a csv file as bytes.

Returns

If csv is True, it will return bytes. Otherwise, dict with:

The resource metrics, with the following keys:

additional_data (dict): Additional data with the following keys:
- total_disk (int): Total disk size in bytes.
- total_ram (int): Total RAM size in bytes.
- total_vram (int): Total VRAM size in bytes if there is a GPU.
- cpu_usage (list of dict): The CPU usage in percent.
- disk_usage (list of dict): The disk usage in percent.
- ram_usage (list of dict): The RAM usage in percent.
- vram_usage (list of dict): The VRAM usage in percent if there is a GPU.
- gpu_usage (list of dict): The GPU usage in percent if there is a GPU.
- network_input_usage (list of dict): The network input usage in bytes.
- network_output_usage (list of dict): The network output usage in bytes.

Each element of the lists is a dict with the following keys:

metric (dict): Dictionary with the following key:
- worker (str): The worker name.
values (list of list): The values of the metrics in the following format: [[timestamp, value], ...].

Example with dict object

from datetime import datetime, timedelta

# Get the current time
current_time = datetime.now()

# Calculate the beginning of the period (last 2 hours)
begin_date = current_time - timedelta(minutes=2)

# Set end_date to the current time
end_date = current_time

# Print the results
print("begin_date:", begin_date)
print("end_date:", end_date)

res = sdk.get_resource_metrics(begin_date, end_date)

print (res)

Example of return object:

{'metrics': {'cpu_usage': [{'metric': {'worker': 'Worker-58d65a25'},
    'values': [[1707132918930, 0.00464583333348214],
     [1707132948930, 0.004708333333450027],
     [1707132978930, 0.004763888888882371],
     [1707133008930, 0.0047222222219369045],
     [1707133038930, 0.004722222221995642]]},
   {'metric': {'worker': 'Worker-6e32691a'},
    'values': [[1707132918930, 0.007868055555652481],
     [1707132948930, 0.007638888888662494],
     [1707132978930, 0.00754861111112385],
     [1707133008930, 0.0076597222224098135],
     [1707133038930, 0.007930555555419681]]},

    ...

     [1707132948930, 40546.24444444444],
     [1707132978930, 48481.6111111111],
     [1707133008930, 40639.65555555555],
     [1707133038930, 48189.03333333333]]},
   {'metric': {'worker': 'All cumulate'},
    'values': [[1707132918930, 63759.26666666666],
     [1707132948930, 56443.922222222216],
     [1707132978930, 64879.22222222221],
     [1707133008930, 46783.055555555555],
     [1707133038930, 53954.188888888886]]}]},
 'additional_data': {'total_ram': 16616214528, 'total_disk': 21462233088}}

Example with CSV file

from datetime import datetime, timedelta

# Get the current time
current_time = datetime.now()

# Calculate the beginning of the period (last 2 hours)
begin_date = current_time - timedelta(minutes=2)

# Set end_date to the current time
end_date = current_time

# Print the results
print("begin_date:", begin_date)
print("end_date:", end_date)

res = sdk.get_resource_metrics(begin_date, end_date,csv=True)

result_csv = res.decode('utf-8')

# Save the result to a CSV file
csv_filename = "resource_metrics.csv"
with open(csv_filename, "w") as csv_file:
    csv_file.write(result_csv)

Example resource_metrics.csv file:

timestamp	worker	CPU	RAM_percent	RAM	disk_percent	disk	network_input	network_output
1707132918930	Worker-58d65a25	0.00464583333348214	0.0614234303655712	0.950531005859375	0.403168019866396	8.05862808227539	14646.7222222222	15505.5777777778
1707132948930	Worker-58d65a25	0.00470833333345003	0.0615619668532965	0.952674865722656	0.403168019866396	8.05862808227539	15029.0666666667	15897.6777777778
1707132978930	Worker-58d65a25	0.00476388888888237	0.0612654198875784	0.948085784912109	0.403217830899368	8.05962371826172	15577.8777777778	16397.6111111111
1707133008930	Worker-58d65a25	0.00472222222193691	0.061079554208317	0.945209503173829	0.40321668581815	8.05960083007813	5007	6143.4
1707133038930	Worker-58d65a25	0.00472222222199564	0.0612341135994269	0.947601318359374	0.40321668581815	8.05960083007813	4627.8	5765.15555555556
1707132918930	Worker-6e32691a	0.00786805555565248	0.0948272398231762	1.46745681762695	0.831449961000442	16.6192398071289	47515.6111111111	48253.6888888889
1707132948930	Worker-6e32691a	0.00763888888866249	0.0949657763109015	1.46960067749023	0.832236059256426	16.634952545166	39472.1555555555	40546.2444444444
1707132978930	Worker-6e32691a	0.00754861111112385	0.0951188566647759	1.47196960449219	0.832231097237816	16.6348533630371	47657.8444444444	48481.6111111111
1707133008930	Worker-6e32691a	0.00765972222240981	0.0950882898952422	1.47149658203125	0.832231097237816	16.6348533630371	39568.6777777778	40639.6555555556