POST
/
scrape_sitemap
curl --request POST \
  --url https://api.carbon.ai/scrape_sitemap \
  --header 'Content-Type: application/json' \
  --header 'authorization: <api-key>' \
  --data '{
  "url": "<string>",
  "tags": {},
  "max_pages_to_scrape": 2,
  "chunk_size": 123,
  "chunk_overlap": 123,
  "skip_embedding_generation": true,
  "enable_auto_sync": true,
  "generate_sparse_vectors": true,
  "prepend_filename_to_chunks": true,
  "html_tags_to_skip": [
    "<string>"
  ],
  "css_classes_to_skip": [
    "<string>"
  ],
  "css_selectors_to_skip": [
    "<string>"
  ],
  "embedding_model": "OPENAI"
}'
"<any>"

Authorizations

authorization
string
headerrequired

token <token>, corresponds to temporary access tokens.

Body

application/json
url
string
required
tags
object | null
max_pages_to_scrape
integer | null
chunk_size
integer | null
chunk_overlap
integer | null
skip_embedding_generation
boolean | null
enable_auto_sync
boolean | null
generate_sparse_vectors
boolean | null
prepend_filename_to_chunks
boolean | null
html_tags_to_skip
string[] | null
css_classes_to_skip
string[] | null
css_selectors_to_skip
string[] | null
embedding_model
enum<string>
Available options:
OPENAI,
AZURE_OPENAI,
AZURE_ADA_LARGE_256,
AZURE_ADA_LARGE_1024,
AZURE_ADA_LARGE_3072,
AZURE_ADA_SMALL_512,
AZURE_ADA_SMALL_1536,
COHERE_MULTILINGUAL_V3,
VERTEX_MULTIMODAL,
OPENAI_ADA_LARGE_256,
OPENAI_ADA_LARGE_1024,
OPENAI_ADA_LARGE_3072,
OPENAI_ADA_SMALL_512,
OPENAI_ADA_SMALL_1536

Response

200 - application/json

The response is of type any.