🗞️ Data Management

Note: Check out our Dashboard (opens in a new tab) for a streamlined data management interface.

Managing Data Sources

Add a Source

To add a new data source:

client.add_source(source_name='City')

Optionally, to add a data source with metadata support:

client.add_source(source_name='City', metadata_fields={"author":"str", "views":"int", "date":"date"})

metadata is a dictionary that specifies the metadata fields and their types for a data source. We currently support three metadata types: str (String), int (Integer), date (Date).

You can also specify the embedding model to use for the data source:

client.add_source(source_name='City', embedding_model='bge-base-en-v1.5')

Please find the list of available embedding models here (opens in a new tab).

Delete a Source

To remove a data source:

client.delete_source(source_name='City')

List Sources

To get a list of available data sources:

sources = client.list_sources()

Get Source Info

To get useful meta information of a data source (e.g., embedding model and token limit):

results = client.get_source_info(source_name='City')

Set Source Embedding Model

To set or update the default embedding model for a data source:

client.set_source_embedding_model(source_name="City", embedding_model="text-embedding-3-large")

Managing Files

Add a File

To add a file to a specific source:

client.add_file(source_name='City', local_path='/Users/John/Boston.txt')

local_path specifies the local path to the file you wish to upload. source_name specifies the data source to store the uploaded file. Supported file types are text, PDF and Word files.

Optionally, you can also add a file with metadata:

client.add_file(source_name='City', local_path='/Users/John/Boston.txt', 
                metadata={'author':'John', 'views':4500, 'date':'2022-08-01'})

metadata is a dictionary that specifies the metadata fields and their values associated with the file.

Delete a File

To remove a file from a particular source:

client.delete_file(source_name='City', file_name='Boston.txt')

file_name is the name of the file to be deleted. source_name specifies the source of the file to delete.

Download a File

To download a file from a designated source:

client.download_file(source_name='City', file_name='Boston.txt', local_path='/Users/John/Boston.txt')

file_name is the name of the file to download. source_name specifies the source of the file. local_path specifies the local path to save the downloaded file.

Upsert File Metadata

To insert or update metadata for a specific file:

client.upsert_file_metadata(source_name='City', file_name='Boston.txt', 
                           metadata={'author':'Michael', 'views':6000, 'date':'2023-02-15'})

List Files

To get a list of available files in a specific source:

files = client.list_files(source_name='City')

Get File Retrieval Status

To check if a specific file from a source is ready for retrieval:

status = client.get_file_retrieval_status(source_name='City', file_name='Boston.txt')

Get Chunks of a File

To get the chunks of a certain file from a data source:

chunks = client.get_file_chunks(source_name='City', file_name='Boston.txt')

Managing Text

Add Text

To add text to a specific source:

client.add_text(source_name='City', text='Boston is the capital of Massachusetts.')

Optionally, to add text with metadata to a source with corresponding metadata fields:

client.add_text(source_name='City', text='Boston is the capital of Massachusetts.', 
                metadata={'author':'John', 'views':4500, 'date':'2022-08-01'})