The maximum file size is 1 GB, but it can be increased upon request. However, please note that 3rd party connectors may have their own file size limits, which you can check here.

We use Google’s Multimodal Embedding model on Vertex AI to generate video embeddings. This allows users to search through uploaded videos based on the similarity of a video snippet to the search query or the text within the video frames, similar to OCR.

Carbon supports the following video file formats:

File Formats
AVI
FLV
MKV
MOV
MP4
MPEG
MPG
WEBM
WMV

If you need to transcribe audio from video files, please refer to the documentation here.

​
Uploading Video Files

Below is an example of how to upload a video file via the uploadfile endpoint:

“Carbon Demo.mp4” is just a file in the directory the request is being made from.
curl --request POST \
  --url 'https://api.carbon.ai/uploadfile?media_type=VIDEO' \
  --header 'Authorization: Bearer <YOUR TOKEN>' \
  --header 'Content-Type: multipart/form-data' \
  --header 'customer-id: <YOUR ID>' \
  --form 'file=@Carbon Demo.mp4'

And the corresponding response will look like this:

{
    "id": 33710,
    "source": "MP4",
    "organization_id": 1,
    "organization_supplied_user_id": "TEST_ID",
    "organization_user_data_source_id": null,
    "external_file_id": "5sb0yDBJNsrOscAxszQqymzdZBMfMg4V",
    "external_url": null,
    "sync_status": "QUEUED_FOR_SYNC",
    "sync_error_message": null,
    "last_sync": null,
    "tags": null,
    "file_statistics": {
        "file_format": "MP4",
        "file_size": 16316780,
        "num_characters": null,
        "num_tokens": null,
        "num_embeddings": null,
        "mime_type": "video/mp4"
    },
    "file_metadata": null,
    "embedding_properties": {
        "VERTEX_MULTIMODAL": {
            "chunk_size": null,
            "chunk_overlap": null
        }
    },
    "chunk_size": null,
    "chunk_overlap": null,
    "chunk_properties": {
        "set_page_as_boundary": false,
        "prepend_filename_to_chunks": false,
        "max_items_per_chunk": null
    },
    "ocr_properties": {},
    "ocr_job_started_at": null,
    "name": "Carbon Demo.mp4",
    "parent_id": null,
    "enable_auto_sync": false,
    "presigned_url": null,
    "parsed_text_url": null,
    "additional_presigned_urls": null,
    "skip_embedding_generation": false,
    "source_created_at": null,
    "generate_sparse_vectors": false,
    "request_id": null,
    "sync_properties": {
        "media_type": "VIDEO",
        "detect_audio_language": false
    },
    "created_at": "2024-05-28T23:55:32.378376Z",
    "updated_at": "2024-05-28T23:55:32.378376Z"
}

​
Querying Embeddings from Video Files

Here is an example on how to query that file once it’s uploaded:

curl --request POST \
  --url https://api.carbon.ai/embeddings \
  --header 'Content-Type: application/json' \
  --header 'authorization: Bearer <YOUR TOKEN>' \
  --header 'customer-id: <YOUR ID>' \
  --data '{
  "query": "what does Carbon do?",
  "k": 3,
  "file_ids": [33710],
  "media_type": "VIDEO"
}'

And the corresponding response will look like this:

The important part of the response is content_metadata, where end_offset_sec and start_offset_sec denote the interval of the video that the chunk is referencing.
{
  "documents": [
    {
      "content": "VIDEO",
      "file_id": 33710,
      "parent_file_id": null,
      "source": "Carbon Demo.mp4",
      "source_url": null,
      "source_type": "MP4",
      "presigned_url": null,
      "tags": null,
      "vector": null,
      "score": 0.09174753725528717,
      "rank": 1,
      "content_metadata": {
        "end_offset_sec": 60,
        "start_offset_sec": 45
      },
      "chunk_index": 3
    },
    {
      "content": "VIDEO",
      "file_id": 33710,
      "parent_file_id": null,
      "source": "Carbon Demo.mp4",
      "source_url": null,
      "source_type": "MP4",
      "presigned_url": null,
      "tags": null,
      "vector": null,
      "score": 0.08448819816112518,
      "rank": 2,
      "content_metadata": {
        "end_offset_sec": 30,
        "start_offset_sec": 15
      },
      "chunk_index": 1
    },
    {
      "content": "VIDEO",
      "file_id": 33710,
      "parent_file_id": null,
      "source": "Carbon Demo.mp4",
      "source_url": null,
      "source_type": "MP4",
      "presigned_url": null,
      "tags": null,
      "vector": null,
      "score": 0.08439408242702484,
      "rank": 3,
      "content_metadata": {
        "end_offset_sec": 45,
        "start_offset_sec": 30
      },
      "chunk_index": 2
    }
  ]
}
AudioOCR