File Formats
Video
The maximum file size is 1 GB, but it can be increased upon request. However, please note that 3rd party connectors may have their own file size limits, which you can check here.
We use Google’s Multimodal Embedding model on Vertex AI to generate video embeddings. This allows users to search through uploaded videos based on the similarity of a video snippet to the search query or the text within the video frames, similar to OCR.
Carbon supports the following video file formats:
File Formats |
---|
AVI |
FLV |
MKV |
MOV |
MP4 |
MPEG |
MPG |
WEBM |
WMV |
If you need to transcribe audio from video files, please refer to the documentation here.
Uploading Video Files
Below is an example of how to upload a video file via the uploadfile
endpoint:
“Carbon Demo.mp4” is just a file in the directory the request is being made from.
curl --request POST \
--url 'https://api.carbon.ai/uploadfile?media_type=VIDEO' \
--header 'Authorization: Bearer <YOUR TOKEN>' \
--header 'Content-Type: multipart/form-data' \
--header 'customer-id: <YOUR ID>' \
--form 'file=@Carbon Demo.mp4'
And the corresponding response will look like this:
{
"id": 33710,
"source": "MP4",
"organization_id": 1,
"organization_supplied_user_id": "TEST_ID",
"organization_user_data_source_id": null,
"external_file_id": "5sb0yDBJNsrOscAxszQqymzdZBMfMg4V",
"external_url": null,
"sync_status": "QUEUED_FOR_SYNC",
"sync_error_message": null,
"last_sync": null,
"tags": null,
"file_statistics": {
"file_format": "MP4",
"file_size": 16316780,
"num_characters": null,
"num_tokens": null,
"num_embeddings": null,
"mime_type": "video/mp4"
},
"file_metadata": null,
"embedding_properties": {
"VERTEX_MULTIMODAL": {
"chunk_size": null,
"chunk_overlap": null
}
},
"chunk_size": null,
"chunk_overlap": null,
"chunk_properties": {
"set_page_as_boundary": false,
"prepend_filename_to_chunks": false,
"max_items_per_chunk": null
},
"ocr_properties": {},
"ocr_job_started_at": null,
"name": "Carbon Demo.mp4",
"parent_id": null,
"enable_auto_sync": false,
"presigned_url": null,
"parsed_text_url": null,
"additional_presigned_urls": null,
"skip_embedding_generation": false,
"source_created_at": null,
"generate_sparse_vectors": false,
"request_id": null,
"sync_properties": {
"media_type": "VIDEO",
"detect_audio_language": false
},
"created_at": "2024-05-28T23:55:32.378376Z",
"updated_at": "2024-05-28T23:55:32.378376Z"
}
Querying Embeddings from Video Files
Here is an example on how to query that file once it’s uploaded:
curl --request POST \
--url https://api.carbon.ai/embeddings \
--header 'Content-Type: application/json' \
--header 'authorization: Bearer <YOUR TOKEN>' \
--header 'customer-id: <YOUR ID>' \
--data '{
"query": "what does Carbon do?",
"k": 3,
"file_ids": [33710],
"media_type": "VIDEO"
}'
And the corresponding response will look like this:
The important part of the response is
content_metadata
, where end_offset_sec
and start_offset_sec
denote the interval of the video that the chunk is referencing.{
"documents": [
{
"content": "VIDEO",
"file_id": 33710,
"parent_file_id": null,
"source": "Carbon Demo.mp4",
"source_url": null,
"source_type": "MP4",
"presigned_url": null,
"tags": null,
"vector": null,
"score": 0.09174753725528717,
"rank": 1,
"content_metadata": {
"end_offset_sec": 60,
"start_offset_sec": 45
},
"chunk_index": 3
},
{
"content": "VIDEO",
"file_id": 33710,
"parent_file_id": null,
"source": "Carbon Demo.mp4",
"source_url": null,
"source_type": "MP4",
"presigned_url": null,
"tags": null,
"vector": null,
"score": 0.08448819816112518,
"rank": 2,
"content_metadata": {
"end_offset_sec": 30,
"start_offset_sec": 15
},
"chunk_index": 1
},
{
"content": "VIDEO",
"file_id": 33710,
"parent_file_id": null,
"source": "Carbon Demo.mp4",
"source_url": null,
"source_type": "MP4",
"presigned_url": null,
"tags": null,
"vector": null,
"score": 0.08439408242702484,
"rank": 3,
"content_metadata": {
"end_offset_sec": 45,
"start_offset_sec": 30
},
"chunk_index": 2
}
]
}