POST
/
web_scrape
curl --request POST \
  --url https://api.carbon.ai/web_scrape \
  --header 'Content-Type: application/json' \
  --header 'authorization: <api-key>' \
  --data '[
  {
    "url": "<string>",
    "tags": {},
    "recursion_depth": 1,
    "max_pages_to_scrape": 2,
    "chunk_size": 123,
    "chunk_overlap": 123,
    "skip_embedding_generation": true,
    "enable_auto_sync": true,
    "generate_sparse_vectors": true,
    "prepend_filename_to_chunks": true,
    "html_tags_to_skip": [
      "<string>"
    ],
    "css_classes_to_skip": [
      "<string>"
    ],
    "css_selectors_to_skip": [
      "<string>"
    ],
    "embedding_model": "OPENAI"
  }
]'
"<any>"

Authorizations

authorization
string
headerrequired

token <token>, corresponds to temporary access tokens.

Body

application/json · object[]
url
string
required
tags
object | null
recursion_depth
integer | null
max_pages_to_scrape
integer | null
chunk_size
integer | null
chunk_overlap
integer | null
skip_embedding_generation
boolean | null
enable_auto_sync
boolean | null
generate_sparse_vectors
boolean | null
prepend_filename_to_chunks
boolean | null
html_tags_to_skip
string[] | null
css_classes_to_skip
string[] | null
css_selectors_to_skip
string[] | null
embedding_model
enum<string>
Available options:
OPENAI,
AZURE_OPENAI,
AZURE_ADA_LARGE_256,
AZURE_ADA_LARGE_1024,
AZURE_ADA_LARGE_3072,
AZURE_ADA_SMALL_512,
AZURE_ADA_SMALL_1536,
COHERE_MULTILINGUAL_V3,
VERTEX_MULTIMODAL,
OPENAI_ADA_LARGE_256,
OPENAI_ADA_LARGE_1024,
OPENAI_ADA_LARGE_3072,
OPENAI_ADA_SMALL_512,
OPENAI_ADA_SMALL_1536

Response

200 - application/json

The response is of type any.