Save my data on the store
Each environment has its own storage to save and retrieve any amount of data at any time, from anywhere on the web. This storage is an Amazon S3 bucket that you can manage with the Python SDK of the platform.
You can upload any kind of raw data on the store. Furthermore, the results of the pipelines and metrics generated by the platform can also be saved on the data store.
Info: For the moment, it is not possible to copy data from an environment to another. You have to download it and to re-upload it. In future versions, you will be able to transfer your data.
Info
The graphical interface of the data store will arrive later on the platform.
Summary
| Function name | Method | Return type | Description |
|---|---|---|---|
| list_data_store_objects | CraftAiSdk.list_data_store_objects() | list of dict | Get the list of the objects stored in the data store. |
| upload_data_store_object | CraftAiSdk.upload_data_store_object(filepath_or_buffer, object_path_in_datastore) | None | Upload a file as an object into the data store. |
| download_data_store_object | CraftAiSdk.download_data_store_object(object_path_in_datastore, filepath_or_buffer) | None | Download an object in the data store and save it into a file. |
| delete_data_store_object | CraftAiSdk.delete_data_store_object(object_path_in_datastore) | dict | Delete an object on the data store. |
Get files
Function definition
Get the list of the objects stored in the data store.
Parameter
No parameter
Returns
List of objects (dict) in the data store, each object being represented as dict with :
- "path" (str): Location of the object in the data store.
- "last_modified" (str): The creation date or last modification date in ISO format.
- "size" (str): The size of the object with units of digital storage measurement (MB, GB, ...).
Get file info
Function definition
Get information about a single object in the data store.
Parameter
object_path_in_datastore(str) – Location of the object in the data store.
Returns
Object information, with the following keys:
- "path" (str): Location of the object in the data store.
- "last_modified" (str): The creation date or last modification date in ISO format.
- "size" (str): The size of the object with units of digital storage measurement (MB, GB, ...).
Upload a file
Function definition
Upload a file as an object into the data store.
Parameters
- filepath_or_buffer (str, or file-like object) – String, path to the file to be uploaded ; or file-like object implementing a read() method (e.g. via buildin open function). The file object must be opened in binary mode, not text mode.
- object_path_in_datastore (str) – Destination of the uploaded file.
Returns
This function returns None.
Download a file
Function definition
Download an object in the data store and save it into a file.
Parameters
object_path_in_datastore(str) – Location of the object to download from the data store.filepath_or_buffer(str or file-like object) – String, file path to save the file to ; or a file-like object implementing a write() method, (e.g. via builtin open function). The file object must be opened in binary mode, not text mode.
Returns
This function return an None.
Delete a file
Function definition
Delete an object on the data store.
Parameters
object_path_in_datastore(str) – Location of the object to delete in the data store.
Returns
Deleted object represented as dict (with key “path”).