Web Scrape
Web Scrape
Submit an URL for web scrape. Set a recursive depth and max number of pages to scrape.
POST
/
web_scrape
Authorizations
authorization
string
headerrequiredtoken <token>
, corresponds to temporary access tokens.
Body
application/json · object[]
url
string
requiredtags
object | null
recursion_depth
integer | null
max_pages_to_scrape
integer | null
chunk_size
integer | null
chunk_overlap
integer | null
skip_embedding_generation
boolean | null
enable_auto_sync
boolean | null
generate_sparse_vectors
boolean | null
prepend_filename_to_chunks
boolean | null
html_tags_to_skip
string[] | null
css_classes_to_skip
string[] | null
css_selectors_to_skip
string[] | null
embedding_model
enum<string>
Available options:
OPENAI
, AZURE_OPENAI
, AZURE_ADA_LARGE_256
, AZURE_ADA_LARGE_1024
, AZURE_ADA_LARGE_3072
, AZURE_ADA_SMALL_512
, AZURE_ADA_SMALL_1536
, COHERE_MULTILINGUAL_V3
, VERTEX_MULTIMODAL
, OPENAI_ADA_LARGE_256
, OPENAI_ADA_LARGE_1024
, OPENAI_ADA_LARGE_3072
, OPENAI_ADA_SMALL_512
, OPENAI_ADA_SMALL_1536
, SOLAR_1_MINI
url_paths_to_include
string[] | null
URL subpaths or directories that you want to include. For example if you want to only include URLs that start with /questions in stackoverflow.com, you will add /questions/ in this input
Response
200 - application/json
The response is of type any
.