# Welcome to Crawlbyte API Documentation Crawlbyte turns any website into structured data — with **zero complexity on your side**. Whether you're scraping public directories, protected booking systems, or large-scale listings, our engine handles it all — fast, clean, and reliably. ## Built Different * Crawlbyte is built for modern scraping — not just simple HTML dumps. * Unlike basic scraping platforms, Crawlbyte doesn’t stop at simple pages. * We solve **JavaScript rendering**, **anti-bot protections**, **fingerprinting**, and **rate limits** — all internally. ## Need something custom? * Have a tricky target or specific layout? * **We’ll build and deploy your custom template free of charge.** * No delays, no blockers — just plug it into your next API call. > → [Book a Free Setup Call](https://calendly.com/kristian-crawlbyte/crawlbyte-demo-call?preview_source=et_card\&month=2025-06) ## Credit-Based System * Crawlbyte runs on a flexible credit model — no fixed pricing, no guesswork. * Each task type has its own credit cost based on what it takes to return high-quality data:\ solving protections, handling rendering, and more. * You’ll always see the exact credit cost listed on the task page — no surprises. ## Start Building with Crawlbyte * **Fully managed scraping workflows** — no need to configure proxies, headers, retries, or bypass logic. * **Enterprise-grade performance** — consistently high success rates and rapid execution. * **Direct access to technical experts** — we implement what you need, when you need it. * **Custom templates on request** — included with your plan, no added fees or delays. # API Access To start integrating with the Crawlbyte API, you’ll first need to generate your unique API key. ## Getting Your API Key **Step 1: Create Your Account** * If you haven't already, [create your free Crawlbyte account here](https://www.crawlbyte.ai/auth/register). **Step 2: Navigate to API Settings** 1. Log into your Crawlbyte AI dashboard 2. Go to Settings in the main navigation 3. Select the API Access Tab **Step 3: Get your API Key** 1. Copy your API Key using Copy Button, you can reset your API Key anytime. Here’s where you’ll find the API key in your dashboard:

API Access Tab

*** ## ⚠️ **Important Security Notes** * Never expose your API key in client-side code or public repositories. * Always store your key in environment variables or secure vaults (e.g. `.env`, AWS Secrets Manager). * If you suspect your key is compromised, regenerate it immediately. # Authentication All Crawlbyte AI API requests require authentication using your unique API key. We use a simple but secure authentication method that works consistently across all our supported platforms. ## How Authentication Works * Include your API Key in the Request Headers for every API call: ``` Authorization: YOUR_API_KEY_HERE Content-type: application/json ``` ### Error Responses * If authentication fails, you'll receive one of these error responses: ``` 401 Unauthorized: Invalid or missing API key 403 Forbidden: API key valid but insufficient permissions 429 Too Many Requests: Rate limit exceeded for your API key — check your current package or Pay-as-you-go limits to view your allowed requests per second ``` # Chrome Extension The Crawlbyte Chrome Extension allows you to collect authentication cookies required for certain scraping tasks. ## How It Works Some tasks require session cookies to function properly. If cookies are needed, this will be clearly stated in the task’s documentation. You’ll then be prompted to extract cookies using the official Chrome Extension. ## Getting Started **Step 1: Install the Extension**\ Download and install the extension from our [Google Drive](https://drive.google.com/file/d/1_wng723YlBDDM3LPzoP770zAOQV9ch6M/view?usp=sharing). **Step 2: Visit the Site**\ Open the target site in your browser and ensure you're logged in. **Step 3: Open the Extension**\ Click the Crawlbyte Extension icon in your toolbar. It will automatically detect the site and fetch available cookies. **Step 4: Copy the Cookie String**\ Click the copy button to copy the full cookie string. **Step 5: Use in Crawlbyte**\ Paste the copied string into the `cookie` field in your API request or inside the **Cookie String** input field in the task UI if you’re using the dashboard. # Common Error Messages Crawlbyte's infrastructure is built for speed and resilience, but issues can still happen. Below is a list of common error messages you might encounter and what they mean. ## ⚡ Service & Connection Errors * **The service is temporarily unavailable, please try again shortly.**\ All workers are currently busy. Please retry in a few seconds. * **We're handling high load right now, please retry shortly.**\ Temporary congestion on our end. All workers are busy. * **Connection was interrupted, please try again.**\ A worker disconnected unexpectedly. This is rare. Retry should solve it. * **Unable to re-establish connection, please try again later.**\ Internal connection issue between API and backend. * **Not connected to the service, please try again.**\ API is currently disconnected from backend. *** ## 💳 Credit & Team Issues * **Insufficient balance.**\ Your workspace ran out of credits. Recharge via the dashboard. * **Your current balance is insufficient to complete this action.**\ The task requires more credits than are currently available. * **Requested team could not be found, please check and try again.**\ Your team may have been deleted or suspended. Double-check your API key or contact support.+ * **Rate limit exceeded.** * You’ve reached the request limit for your current plan. Check your dashboard for allowed requests per second or upgrade your package. *** ## ⚠️ Protection & Solving Errors * **Request blocked by protection system, please try again later.**\ We were blocked by the target site. We’re already working on bypassing it. * **We couldn't process this request at the moment, please retry or report this issue.**\ This is usually a temporary solving delay. Retry or contact support if persistent. * **We encountered an internal issue while preparing your request, please try again shortly.**\ Solver backend is updating or restarting. Please retry. * **A temporary internal check failed, we're working on it — please retry in a moment.**\ Internal validation failed. Our team has been notified. *** ## 🚫 Task & Payload Issues * **Operation was canceled.**\ The task was cancelled automatically. Review your payload. * **Invalid request setup detected, please contact support.**\ The input or configuration is invalid. * **An unexpected configuration issue occurred, please contact support.**\ An internal logic issue. Contact support. * **We were unable to create your request, please try again later.**\ General request creation error. Retry or contact us. * **Your request couldn’t be completed due to a temporary issue, please retry shortly.**\ Temporary failure. Usually resolves quickly with a retry. * **Unexpected response received, please try again later.**\ An unexpected output was returned. We’re automatically alerted and investigating. If you're stuck or receive repeated errors, reach out via your dashboard or open a support ticket. We monitor and respond to all high-priority failures automatically. # SDKs Crawlbyte offers official SDKs to help you interact with the API faster and more efficiently. Choose your preferred language and get started with installation, usage examples, and integration tips. ## Getting Started To run any scraping task via code, you’ll need to send API requests. The SDKs below make that process easier in your preferred language. Each SDK supports all task types listed in the Tasks section and includes everything you need to get started quickly. ## Go SDK Use the Crawlbyte SDK in Go projects with minimal setup. Built for speed and concurrency. * **Repository:** [crawlbyte-sdk-go](https://github.com/crawlbyte/crawlbyte-sdk-go) * **Install:** ```bash go get github.com/crawlbyte/crawlbyte-sdk-go ``` ## TypeScript / JavaScript SDK Use the Crawlbyte SDK from Node.js or frontend apps with built-in types and simple functions. * **Repository:** [crawlbyte-sdk-ts](https://github.com/crawlbyte/crawlbyte-sdk-ts) * **Install:** ```bash npm install crawlbyte # or yarn add crawlbyte ``` ## Python SDK Integrate Crawlbyte SDK easily into Python-based workflows. * **Repository:** [crawlbyte-sdk-py](https://github.com/crawlbyte/crawlbyte-sdk-py) * **Install:** ```bash pip install crawlbyte ``` ## What’s Next? Refer to the Tasks section for available scrapers and required parameters. Once you choose a task, use the SDK to create it, poll results, and handle the response - all without needing to manage raw requests manually. # Amazon Scraper Collects structured product data from Amazon listings — including titles, prices, ratings, and images. ## Endpoint ``` POST https://api.crawlbyte.ai/api/tasks ``` ## Basic Configuration (Required) ```json { "type": "amazon", "input": [ "B0BLS3Y632", "B09B2SBHQK" ], "multithread": false } ``` ### Parameters | Field | Type | Description | | ----------- | ------- | ------------------------------------------------------ | | type | string | Always `"amazon"` | | input | array | List of ASINs | | multithread | boolean | Use `true` for faster processing with multiple threads | ## Advanced Configuration (Optional) ```json { "type": "amazon", "input": [ "B0BLS3Y632", "B09B2SBHQK" ], "user_agent_preset": "chrome", "user_agent_custom": "", "headers": "{\"X-Test\":\"abc\"}", "cookie": "session=xyz", "proxy": "http://username:password@ip:port" } ``` ### Optional Parameters | Field | Type | Description | | ------------------- | ------ | -------------------------------------------------------------------------------------------------------------------- | | user\_agent\_preset | string | Preset user-agent. Options: `chrome`, `firefox`, `edge`, `opera`, `safari`, `ios-safari`, `android-chrome`, `custom` | | user\_agent\_custom | string | Used if `user_agent_preset` is `custom.` | | headers | string | JSON-formatted string of headers. | | cookie | string | `key=value;` | | proxy | string | `http://username:password@ip:port` | ## Pricing * **$0.075 per successful task**\ This is a pay-as-you-go pricing model — you’re only charged when an Amazon task successfully returns product data such as title, price, and availability. You can view your current credit balance and usage history in the [Crawlbyte Dashboard](https://dash.crawlbyte.ai/). ## Response The response contains metadata about the task. For `amazon` type, the most important fields are `status` and `result`. ```json { "id": "bd3e89ed-815e-4395-98a3-521ede71cc4d", "status": "completed", "result": { // Parsed availability and pricing data } } ``` * `result` is a **JSON object** object containing the final scraped product data — no further polling is required. ### Status Types | Status | Meaning | | ---------- | ----------------------------------------------------------------------------- | | queued | Task was accepted and added to the processing queue. | | processing | Task is currently running. | | completed | Task finished and product data was successfully collected. | | failed | Task encountered an error (e.g., invalid input, no results, or system issue). | ## Polling If `status` is `queued` or `processing`, continue polling the task until it's completed or failed. ``` GET https://api.crawlbyte.ai/api/tasks/:id ``` * You’ll receive the same structure with an updated `status`. * Only poll until you receive `completed` or `failed`. * Recommended interval: **every 2–3 seconds**. ## SDK Usage You can run this task using any official **Crawlbyte SDK**: * [Go SDK](https://github.com/crawlbyte/crawlbyte-sdk-go) * [TypeScript SDK](https://github.com/crawlbyte/crawlbyte-sdk-ts) * [Python SDK](https://github.com/crawlbyte/crawlbyte-sdk-py) Each SDK provides a simple way to: * Create the task * Poll for status * Handle the final result Refer to the [SDKs section](/sdks) for installation, examples, and setup instructions. ## Notes * Only valid input objects with correct ASINs will return results. * Crawlbyte handles retries, rendering, fingerprinting, and anti-bot logic internally — no need to manage it yourself. * Use `multithread: true` in advanced settings if running large volumes. * Ensure all required fields like region and the list of ASINs are properly structured. * The Amazon response includes **all relevant product data** such as title, price, availability, and seller details. * Average task duration is **\~2 seconds.** # Amtrak Scraper Scrapes train availability and pricing from Amtrak’s booking system using structured input parameters like route and travel dates. ## Endpoint ``` POST https://api.crawlbyte.ai/api/tasks ``` ## Basic Configuration (Required) ```json { "type": "amtrak", "input": [ "{\"origin\":\"NYP\",\"destination\":\"PHL\",\"departure\":\"2025-08-01\",\"return\":\"2025-08-08\",\"passengers\":{\"adult\":1,\"seniors\":1,\"youth\":1,\"child\":1,\"infant\":1}}" ], "multithread": false } ``` ### Parameters | Field | Type | Description | | ----------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | type | string | Always `"amtrak"` | | input | array | Array of JSON strings containing route, date, and passenger details (e.g., origin, destination, departure, return, and passenger breakdown) | | multithread | boolean | Use `true` for faster processing with multiple threads | ## Input Builder Notes You must structure the input as a JSON string and insert it into the `input` array. Use the following fields: | Parameter | Meaning | Example | | ----------- | --------------------------- | -------------------------------------------------------- | | origin | Origin Station Code | NYP | | destination | Destination Station Code | PHL | | departure | Departure Date (YYYY-MM-DD) | 2025-08-01 | | return | Return Date *(optional)* | 2025-08-08 | | passengers | Passenger Count by Type | `{"adult":1,"seniors":1,"youth":1,"child":1,"infant":1}` | * All values must be wrapped in a **JSON string** (not object) inside the array. * `return` is optional – omit it for one-way trips. ## Advanced Configuration (Optional) ```json { "type": "amtrak", "input": [ "{\"origin\":\"NYP\",\"destination\":\"PHL\",\"departure\":\"2025-08-01\",\"return\":\"2025-08-08\",\"passengers\":{\"adult\":1,\"seniors\":1,\"youth\":1,\"child\":1,\"infant\":1}}" ], "user_agent_preset": "chrome", "user_agent_custom": "", "headers": "{\"X-Test\":\"abc\"}", "cookie": "session=xyz", "proxy": "http://username:password@ip:port" } ``` ### Optional Parameters | Field | Type | Description | | ------------------- | ------ | -------------------------------------------------------------------------------------------------------------------- | | user\_agent\_preset | string | Preset user-agent. Options: `chrome`, `firefox`, `edge`, `opera`, `safari`, `ios-safari`, `android-chrome`, `custom` | | user\_agent\_custom | string | Used if `user_agent_preset` is `custom.` | | headers | string | JSON-formatted string of headers. | | cookie | string | `key=value;` | | proxy | string | `http://username:password@ip:port` | ## Pricing * **$0.01 per successful task**\ This is a pay-as-you-go pricing model — you're only charged when an Amtrak task successfully returns train availability and fare data. You can view your current credit balance and usage history in the [Crawlbyte Dashboard](https://dash.crawlbyte.ai/). ## Response The response contains metadata about the task. For `amtrak` type, the most important fields are `status` and `result`. ```json { "id": "bd3e89ed-815e-4395-98a3-521ede71cc4d", "status": "completed", "result": { // Parsed availability and pricing data } } ``` * `result` is a **JSON object** containing the final scraped train availability and fare data — no further polling is needed. ### Status Types | Status | Meaning | | ---------- | ----------------------------------------------------------------------------- | | queued | Task was accepted and added to the processing queue. | | processing | Task is currently running. | | completed | Task finished and train data was successfully collected. | | failed | Task encountered an error (e.g., invalid input, no results, or system issue). | ## Polling If `status` is `queued` or `processing`, continue polling the task until it's completed or failed. ``` GET https://api.crawlbyte.ai/api/tasks/:id ``` * You’ll receive the same structure with an updated `status`. * Only poll until you receive `completed` or `failed`. * Recommended interval: **every 2–4 seconds**. ## SDK Usage You can run this task using any official **Crawlbyte SDK**: * [Go SDK](https://github.com/crawlbyte/crawlbyte-sdk-go) * [TypeScript SDK](https://github.com/crawlbyte/crawlbyte-sdk-ts) * [Python SDK](https://github.com/crawlbyte/crawlbyte-sdk-py) Each SDK provides a simple way to: * Create the task * Poll for status * Handle the final result Refer to the [SDKs section](/sdks) for installation, examples, and setup instructions. ## Notes * Only valid input objects with correct station codes and date formats will return results. * Crawlbyte handles retries, rendering, fingerprinting, and anti-bot logic internally — no need to manage it yourself. * Use `multithread: true` in advanced settings if running large volumes. * Ensure all required fields like `origin`, `destination`, `departure`, and `passengers` are properly structured. * The Amtrak response includes **all relevant train data**, including schedule and fare breakdowns. * Average task duration is **\~8 seconds**, primarily due to Amtrak’s slower API response — this is expected and fully supported. # Beehiiv Subscriber Subscribe users to a Beehiiv newsletter with just a few inputs. This task supports single or multithreaded processing, along with full customization via headers, user-agents, cookies, and proxy. ## Endpoint ``` POST https://api.crawlbyte.ai/api/tasks ``` ## Basic Configuration (Required) ```json { "type": "beehiiv", "url": "https://link.beehiiv.com/subscribe", "input": [ "test@example.com", "hello@crawlbyte.ai" ], "multithread": false } ``` ### Parameters | Field | Type | Description | | ----------- | ------- | ------------------------------------------------------ | | type | string | Always `"beehiiv"` | | url | string | Target Beehiiv subscribe form URL | | input | array | Array of emails to be subscribed | | multithread | boolean | Use `true` for faster processing with multiple threads | ## Advanced Configuration (Optional) ```json { "type": "beehiiv", "url": "https://link.beehiiv.com/subscribe", "input": [ "test@example.com" ], "multithread": true, "headers": "{\"Authorization\":\"Bearer abc123\",\"X-Custom-Header\":\"value\"}", "cookie": "session_id=xyz123; token=abc456", "proxy": "http://username:password@ip:port" } ``` ### Optional Parameters | Field | Type | Description | | ------- | ------ | ---------------------------------- | | headers | string | JSON-formatted string of headers. | | cookie | string | `key=value;` | | proxy | string | `http://username:password@ip:port` | ## Pricing * **$0.01 per successful task**\ This is a pay-as-you-go pricing model — you're only charged when a Beehiiv task successfully submits email addresses. You can view your current credit balance and usage history in the [Crawlbyte Dashboard](https://dash.crawlbyte.ai/). ## Response The response object contains metadata about the task, but for `beehiiv` type, the most relevant field is `status`, which determines task progress. ```json { "id": "bd3e89ed-815e-4395-98a3-521ede71cc4d", "status": "queued" } ``` ### Status Types | Status | Meaning | | ---------- | -------------------------------------------------------------- | | queued | Task was accepted and added to the processing queue. | | processing | Task is currently running. | | completed | Task finished successfully (emails were submitted to Beehiiv). | | failed | Task encountered an error (e.g. bad proxy, invalid URL, etc.). | ## Polling If the initial `status` is `queued` or `processing`, you should poll for task completion. ``` GET https://api.crawlbyte.ai/api/tasks/:id ``` * You’ll receive the same structure with an updated `status`. * Only poll until you receive `completed` or `failed`. * Recommended interval: **every 3–5 seconds**. ## SDK Usage You can run this task using any official **Crawlbyte SDK**: * [Go SDK](https://github.com/crawlbyte/crawlbyte-sdk-go) * [TypeScript SDK](https://github.com/crawlbyte/crawlbyte-sdk-ts) * [Python SDK](https://github.com/crawlbyte/crawlbyte-sdk-py) Each SDK provides a simple way to: * Create the task * Poll for status * Handle the final result Refer to the [SDKs section](/sdks) for installation, examples, and setup instructions. ## Notes * If you leave advanced settings blank, Crawlbyte will automatically assign optimal headers, user-agent, and fingerprinting. * `multithread: true` is recommended for faster processing — especially when submitting larger email batches. * Incorrect `proxy`, malformed `headers`, or invalid `cookie` format will cause the task to fail. * Only use custom proxy, headers, or cookies if needed — default settings work best in most cases. # Best Buy Scraper ## Endpoint ``` POST https://api.crawlbyte.ai/api/tasks ``` ## Basic Configuration (Required) 
{
  "type": "bestbuy",
  "input": [
    "6613487"
  ],
  "multithread": false
}
### Parameters | Field | Type | Description | | ----------- | ------- | ------------------------------------------------------ | | type | string | Always `"bestbuy"` | | input | array | Array of Best Buy Product ID's | | multithread | boolean | Use `true` for faster processing with multiple threads | ## Advanced Configuration (Optional) ```json { "type": "bestbuy", "input": [ "6613487" ], "fields": [ "title", "price", "availability", "url", "image", "description", "rating", "upc", "model" ], "user_agent_preset": "chrome", "user_agent_custom": "", "proxy": "http://username:password@ip:port" } ``` ### Optional Parameters | Field | Type | Description | | ------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | fields | array | List of fields to scrape. Available options: `title`, `price`, `availability`, `url`, `image`, `description`, `rating`, `upc`, `model`. Leave empty to scrape all. | | user\_agent\_preset | string | Preset user-agent. Options: `chrome`, `firefox`, `edge`, `opera`, `safari`, `ios-safari`, `android-chrome`, `custom` | | user\_agent\_custom | string | Used if `user_agent_preset` is `custom.` | | proxy | string | `http://username:password@ip:port` | ## Pricing * **$0.0050 per successful task**\ This is a pay-as-you-go pricing model — you're only charged when a Best Buy task successfully returns product data. You can view your current credit balance and usage history in the [Crawlbyte Dashboard](https://dash.crawlbyte.ai/). ## Response The response contains metadata about the task. For `bestbuy` type, the most important fields are `status` and `result`. ```json { "id": "bd3e89ed-815e-4395-98a3-521ede71cc4d", "status": "completed", "result": { // product data } } ``` * `result` is a **JSON object,** that's the final scraped product data — no further polling is needed. ### Status Types | Status | Meaning | | ---------- | -------------------------------------------------------------------------- | | queued | Task was accepted and added to the processing queue. | | processing | Task is currently running. | | completed | Task finished and product data was successfully collected. | | failed | Task encountered an error (e.g., invalid product ID, access denied, etc.). | ## Polling If `status` is `queued` or `processing`, continue polling the task until it's completed or failed. ``` GET https://api.crawlbyte.ai/api/tasks/:id ``` * You’ll receive the same structure with an updated `status`. * Only poll until you receive `completed` or `failed`. * Recommended interval: **every 3–5 seconds**. ## SDK Usage You can run this task using any official **Crawlbyte SDK**: * [Go SDK](https://github.com/crawlbyte/crawlbyte-sdk-go) * [TypeScript SDK](https://github.com/crawlbyte/crawlbyte-sdk-ts) * [Python SDK](https://github.com/crawlbyte/crawlbyte-sdk-py) Each SDK provides a simple way to: * Create the task * Poll for status * Handle the final result Refer to the [SDKs section](/sdks) for installation, examples, and setup instructions. ## Notes * Only valid Best Buy product IDs will return results. * Crawlbyte handles retries, fingerprinting, and anti-bot logic internally — no need to configure anything manually. * Use `multithread: true` in advanced settings when processing large volumes. * If a specific field is missing in the result, it means the data wasn’t available or couldn’t be parsed — usually all supported fields are returned. * Need more fields? Just contact our support and we’ll add them. * Average task duration is \~1–2 seconds per product. # FlyFrontier Scraper Scrapes flight availability and pricing from FlyFrontier’s booking system using full booking URLs with departure and destination parameters. ## Endpoint ``` POST https://api.crawlbyte.ai/api/tasks ``` ## Basic Configuration (Required) ```json { "type": "flyfrontier", "input": [ "https://booking.flyfrontier.com/Flight/InternalSelect?o1=ATL&d1=DEN&dd1=May%2028%2C%202025&dd2=May%2029%2C%202025&ADT=1&r=true&mon=true&promo=&ftype=STD" ], "multithread": false } ``` ### Parameters | Field | Type | Description | | ----------- | ------- | ------------------------------------------------------ | | type | string | Always `"flyfrontier"` | | input | array | Array of full FlyFrontier booking URLs | | multithread | boolean | Use `true` for faster processing with multiple threads | ## URL Builder Notes You must build the full FlyFrontier URL manually and insert it into the `input` array. Use the following key query parameters: | Parameter | Meaning | Example | | --------- | ------------------------ | ------------ | | o1 | Origin Airport Code | ATL | | d1 | Destination Airport Code | DEN | | dd1 | Departure Date | May 28, 2025 | | dd2 | Return Date | May 29, 2025 | * **Ensure dates are URL-encoded** (e.g., `May%2028%2C%202025`) ## Advanced Configuration (Optional) ```json { "type": "flyfrontier", "input": [ "https://booking.flyfrontier.com/Flight/InternalSelect?o1=ATL&d1=DEN&dd1=May%2028%2C%202025&dd2=May%2029%2C%202025&ADT=1&r=true&mon=true&promo=&ftype=STD" ], "proxy": "http://username:password@ip:port" } ``` ### Optional Parameters | Field | Type | Description | | ----- | ------ | ---------------------------------- | | proxy | string | `http://username:password@ip:port` | ## Pricing * **$0.01 per successful task**\ This is a pay-as-you-go pricing model — you're only charged when a FlyFrontier task successfully returns flight data. You can view your current credit balance and usage history in the [Crawlbyte Dashboard](https://dash.crawlbyte.ai/). ## Response The response contains metadata about the task. For `flyfrontier` type, the most important fields are `status` and `result`. ```json { "id": "bd3e89ed-815e-4395-98a3-521ede71cc4d", "status": "completed", "result": { "inboundSchedule": { // inbound schedule here }, "outboundSchedule": { // outbound schedule here }, "results": { // flight data here } } } ``` * `result` is a **JSON object,** that's the final scraped flight data — no further polling is needed. ### Status Types | Status | Meaning | | ---------- | ---------------------------------------------------- | | queued | Task was accepted and added to the processing queue. | | processing | Task is currently running. | | completed | Task finished and flight data was collected. | | failed | Task encountered an error (invalid URL, etc.). | ## Polling If `status` is `queued` or `processing`, continue polling the task until it's completed or failed. ``` GET https://api.crawlbyte.ai/api/tasks/:id ``` * You’ll receive the same structure with an updated `status`. * Only poll until you receive `completed` or `failed`. * Recommended interval: **every 3–5 seconds**. ## SDK Usage You can run this task using any official **Crawlbyte SDK**: * [Go SDK](https://github.com/crawlbyte/crawlbyte-sdk-go) * [TypeScript SDK](https://github.com/crawlbyte/crawlbyte-sdk-ts) * [Python SDK](https://github.com/crawlbyte/crawlbyte-sdk-py) Each SDK provides a simple way to: * Create the task * Poll for status * Handle the final result Refer to the [SDKs section](/sdks) for installation, examples, and setup instructions. ## Notes * Only valid FlyFrontier booking URLs will return results. * Crawlbyte handles retries, rendering, fingerprinting, and anti-bot logic internally — no need to manage it yourself. * Use `multithread: true` in advanced settings if running large volumes. * Make sure airport codes and dates are correct and properly URL-encoded. * FlyFrontier response includes **all relevant flight data**, including the full date range (from `RetrieveSchedule` endpoint). * Average task duration is **\~5 seconds**, due to FlyFrontier’s slower site architecture — this is normal and fully supported. # FlyFrontier App Scraper Scrapes flight availability and pricing from FlyFrontier’s App ## Endpoint ``` POST https://api.crawlbyte.ai/api/tasks ``` ## Basic Configuration for GET Request (Required) ```json { "type": "flyfrontierapp", "input": [ "https://mtier.flyfrontier.com/flightavailabilityssv/GetTripSchedule?Origin=AUS&Destination=ATL&BeginDate=2025-08-24&EndDate=2026-08-28&Type=5" ], "multithread": false } ``` ### Parameters | Field | Type | Description | | ----------- | ------- | ------------------------------------------------------ | | type | string | Always `"flyfrontierapp"` | | input | array | Array of full FlyFrontier app URLs you want to scrape | | multithread | boolean | Use `true` for faster processing with multiple threads | ## Advanced Configuration (Optional) ```json { "type": "flyfrontierapp", "input": [ "https://mtier.flyfrontier.com/flightavailabilityssv/FlightAvailabilitySimpleSearch" ], "method": "POST", "body": "{\"flightAvailabilityRequestModel\":{\"passengers\":{\"types\":[{\"type\":\"ADT\",\"count\":1}],\"residentCountry\":\"US\"},\"filters\":{\"maxConnections\":20,\"fareInclusionType\":\"Default\",\"type\":\"All\",\"includeAllotments\":true,\"bundleControlFilter\":\"2\"},\"codes\":{\"currencyCode\":\"USD\"},\"origin\":\"ATL\",\"destination\":\"AUS\",\"beginDate\":\"2025-08-24\",\"endDate\":\"2025-08-26\"}}", "proxy": "http://username:password@ip:port" } ``` ### Optional Parameters | Field | Type | Description | | ------ | ------ | ---------------------------------- | | method | string | GET, POST, PUT, PATCH | | body | string | Stringified JSON | | proxy | string | `http://username:password@ip:port` | ## Pricing * **$0.002 per successful task**\ This is a pay-as-you-go pricing model — you're only charged when a FlyFrontier task successfully returns flight data. You can view your current credit balance and usage history in the [Crawlbyte Dashboard](https://dash.crawlbyte.ai/). ## Response The response contains metadata about the task. For `flyfrontierapp` type, the most important fields are `status` and `result`. ```json { "id": "f5da7606-2dd1-4f2f-bd24-f8d70680df97", "type": "flyfrontierapp", "status": "completed", "input": [ "https://mtier.flyfrontier.com/flightavailabilityssv/GetTripSchedule?Origin=AUS&Destination=ATL&BeginDate=2025-08-25&EndDate=2026-08-28&Type=5" ], "method": "GET", "result": [ "{\"status\":200,\"message\":\"Success\",\"errorDetails\":null,\"data\":[{\"departureDate\":\"2025-08-25T00:00:00\",\"earliest\":\"2025-08-25T06:00:00\",\"latest\":\"2025-08-25T21:33:00\",\"flights\":9,\"noteCode\":null},{\"departureDate\":\"2025-08-26T00:00:00\",\"earliest\":\"2025-08-26T07:55:00\",\"latest\":\"2025-08-26T07:55:00\",\"flights\":2,\"noteCode\":null},{\"departureDate\":\"2025-08-27T00:00:00\",\"earliest\":\"2025-08-27T07:55:00\",\"latest\":\"2025-08-27T07:55:00\",\"flights\":2,\"noteCode\":null},{\"departureDate\":\"2025-08-28T00:00:00\",\"earliest\":\"2025-08-28T06:00:00\",\"latest\":\"2025-08-28T21:56:00\",\"flights\":14,\"noteCode\":null},{\"departureDate\":\"2025-08-29T00:00:00\",\"earliest\":\"2025-08-29T06:00:00\",\"latest\":\"2025-08-29T21:35:00\",\"flights\":8,\"noteCode\":null},{\"departureDate\":\"2025-08-30T00:00:00\",\"earliest\":\"2025-08-30T07:44:00\",\"latest\":\"2025-08-30T11:00:00\",\"flights\":5,\"noteCode\":null},{\"departureDate\":\"2025-08-31T00:00:00\",\"earliest\":\"2025-08-31T06:00:00\",\"latest\":\"2025-08-31T21:33:00\",\"flights\":15,\"noteCode\":null},{\"departureDate\":\"2025-09-01T00:00:00\",\"earliest\":\"2025-09-01T06:00:00\",\"latest\":\"2025-09-01T21:33:00\",\"flights\":10,\"noteCode\":null},{\"departureDate\":\"2025-09-02T00:00:00\",\"earliest\":\"2025-09-02T07:55:00\",\"latest\":\"2025-09-02T07:55:00\",\"flights\":2,\"noteCode\":null},{\"departureDate\":\"2025-09-03T00:00:00\",\"earliest\":\"2025-09-03T07:55:00\",\"latest\":\"2025-09-03T07:55:00\",\"flights\":1,\"noteCode\":null},{\"departureDate\":\"2025-09-04T00:00:00\",\"earliest\":\"2025-09-04T06:30:00\",\"latest\":\"2025-09-04T19:35:00\",\"flights\":15,\"noteCode\":null},{\"departureDate\":\"2025-09-05T00:00:00\",\"earliest\":\"2025-09-05T06:30:00\",\"latest\":\"2025-09-05T21:23:00\",\"flights\":9,\"noteCode\":null},{\"departureDate\":\"2025-09-06T00:00:00\",\"earliest\":\"2025-09-06T07:50:00\",\"latest\":\"2025-09-06T07:50:00\",\"flights\":2,\"noteCode\":null},{\"departureDate\":\"2025-09-07T00:00:00\",\"earliest\":\"2025-09-07T06:30:00\",\"latest\":\"2025-09-07T21:35:00\",\"flights\":18,\"noteCode\":null},{\"departureDate\":\"2025-09-08T00:00:00\",\"earliest\":\"2025-09-08T06:30:00\",\"latest\":\"2025-09-08T21:23:00\",\"flights\":9,\"noteCode\":null},{\"departureDate\":\"2025-09-09T00:00:00\",\"earliest\":\"2025-09-09T08:00:00\",\"latest\":\"2025-09-09T08:00:00\",\"flights\":1,\"noteCode\":null},{\"departureDate\":\"2025-09-10T00:00:00\",\"earliest\":\"2025-09-10T08:00:00\",\"latest\":\"2025-09-10T08:00:00\",\"flights\":2,\"noteCode\":null},{\"departureDate\":\"2025-09-11T00:00:00\",\"earliest\":\"2025-09-11T06:30:00\",\"latest\":\"2025-09-11T19:35:00\",\"flights\":15,\"noteCode\":null},{\"departureDate\":\"2025-09-12T00:00:00\",\"earliest\":\"2025-09-12T06:30:00\",\"latest\":\"2025-09-12T21:23:00\",\"flights\":9,\"noteCode\":null},{\"departureDate\":\"2025-09-13T00:00:00\",\"earliest\":\"2025-09-13T07:50:00\",\"latest\":\"2025-09-13T07:50:00\",\"flights\":2,\"noteCode\":null},{\"departureDate\":\"2025-09-14T00:00:00\",\"earliest\":\"2025-09-14T06:30:00\",\"latest\":\"2025-09-14T21:41:00\",\"flights\":18,\"noteCode\":null},{\"departureDate\":\"2025-09-15T00:00:00\",\"earliest\":\"2025-09-15T06:30:00\",\"latest\":\"2025-09-15T21:23:00\",\"flights\":9,\"noteCode\":null},{\"departureDate\":\"2025-09-16T00:00:00\",\"earliest\":\"2025-09-16T08:00:00\",\"latest\":\"2025-09-16T08:00:00\",\"flights\":1,\"noteCode\":null},{\"departureDate\":\"2025-09-17T00:00:00\",\"earliest\":\"2025-09-17T08:00:00\",\"latest\":\"2025-09-17T08:00:00\",\"flights\":2,\"noteCode\":null},{\"departureDate\":\"2025-09-18T00:00:00\",\"earliest\":\"2025-09-18T06:30:00\",\"latest\":\"2025-09-18T19:35:00\",\"flights\":15,\"noteCode\":null},{\"departureDate\":\"2025-09-19T00:00:00\",\"earliest\":\"2025-09-19T06:30:00\",\"latest\":\"2025-09-19T21:23:00\",\"flights\":8,\"noteCode\":null},{\"departureDate\":\"2025-09-20T00:00:00\",\"earliest\":\"2025-09-20T07:50:00\",\"latest\":\"2025-09-20T07:50:00\",\"flights\":2,\"noteCode\":null},{\"departureDate\":\"2025-09-21T00:00:00\",\"earliest\":\"2025-09-21T06:30:00\",\"latest\":\"2025-09-21T21:41:00\",\"flights\":18,\"noteCode\":null},{\"departureDate\":\"2025-09-22T00:00:00\",\"earliest\":\"2025-09-22T06:30:00\",\"latest\":\"2025-09-22T21:23:00\",\"flights\":9,\"noteCode\":null},{\"departureDate\":\"2025-09-23T00:00:00\",\"earliest\":\"2025-09-23T08:00:00\",\"latest\":\"2025-09-23T08:00:00\",\"flights\":1,\"noteCode\":null},{\"departureDate\":\"2025-09-24T00:00:00\",\"earliest\":\"2025-09-24T08:00:00\",\"latest\":\"2025-09-24T08:00:00\",\"flights\":2,\"noteCode\":null},{\"departureDate\":\"2025-09-25T00:00:00\",\"earliest\":\"2025-09-25T06:30:00\",\"latest\":\"2025-09-25T19:35:00\",\"flights\":15,\"noteCode\":null},{\"departureDate\":\"2025-09-26T00:00:00\",\"earliest\":\"2025-09-26T06:30:00\",\"latest\":\"2025-09-26T21:23:00\",\"flights\":8,\"noteCode\":null},{\"departureDate\":\"2025-09-27T00:00:00\",\"earliest\":\"2025-09-27T07:50:00\",\"latest\":\"2025-09-27T07:50:00\",\"flights\":2,\"noteCode\":null},{\"departureDate\":\"2025-09-28T00:00:00\",\"earliest\":\"2025-09-28T06:30:00\",\"latest\":\"2025-09-28T21:41:00\",\"flights\":18,\"noteCode\":null},{\"departureDate\":\"2025-09-29T00:00:00\",\"earliest\":\"2025-09-29T06:30:00\",\"latest\":\"2025-09-29T21:23:00\",\"flights\":9,\"noteCode\":null},{\"departureDate\":\"2025-09-30T00:00:00\",\"earliest\":\"2025-09-30T08:00:00\",\"latest\":\"2025-09-30T08:00:00\",\"flights\":1,\"noteCode\":null},{\"departureDate\":\"2025-10-01T00:00:00\",\"earliest\":\"2025-10-01T08:00:00\",\"latest\":\"2025-10-01T08:00:00\",\"flights\":2,\"noteCode\":null},{\"departureDate\":\"2025-10-02T00:00:00\",\"earliest\":\"2025-10-02T06:30:00\",\"latest\":\"2025-10-02T19:35:00\",\"flights\":15,\"noteCode\":null},{\"departureDate\":\"2025-10-03T00:00:00\",\"earliest\":\"2025-10-03T06:30:00\",\"latest\":\"2025-10-03T21:23:00\",\"flights\":8,\"noteCode\":null},{\"departureDate\":\"2025-10-04T00:00:00\",\"earliest\":\"2025-10-04T07:50:00\",\"latest\":\"2025-10-04T07:50:00\",\"flights\":2,\"noteCode\":null},{\"departureDate\":\"2025-10-05T00:00:00\",\"earliest\":\"2025-10-05T06:30:00\",\"latest\":\"2025-10-05T21:41:00\",\"flights\":18,\"noteCode\":null},{\"departureDate\":\"2025-10-06T00:00:00\",\"earliest\":\"2025-10-06T06:30:00\",\"latest\":\"2025-10-06T22:54:00\",\"flights\":15,\"noteCode\":null},{\"departureDate\":\"2025-10-07T00:00:00\",\"earliest\":\"2025-10-07T06:00:00\",\"latest\":\"2025-10-07T14:00:00\",\"flights\":3,\"noteCode\":null},{\"departureDate\":\"2025-10-08T00:00:00\",\"earliest\":\"2025-10-08T06:00:00\",\"latest\":\"2025-10-08T14:00:00\",\"flights\":3,\"noteCode\":null},{\"departureDate\":\"2025-10-09T00:00:00\",\"earliest\":\"2025-10-09T06:00:00\",\"latest\":\"2025-10-09T22:19:00\",\"flights\":19,\"noteCode\":null},{\"departureDate\":\"2025-10-10T00:00:00\",\"earliest\":\"2025-10-10T06:00:00\",\"latest\":\"2025-10-10T22:47:00\",\"flights\":16,\"noteCode\":null},{\"departureDate\":\"2025-10-11T00:00:00\",\"earliest\":\"2025-10-11T07:45:00\",\"latest\":\"2025-10-11T20:56:00\",\"flights\":4,\"noteCode\":null},{\"departureDate\":\"2025-10-12T00:00:00\",\"earliest\":\"2025-10-12T06:00:00\",\"latest\":\"2025-10-12T22:24:00\",\"flights\":19,\"noteCode\":null},{\"departureDate\":\"2025-10-13T00:00:00\",\"earliest\":\"2025-10-13T06:00:00\",\"latest\":\"2025-10-13T22:30:00\",\"flights\":10,\"noteCode\":null},{\"departureDate\":\"2025-10-14T00:00:00\",\"earliest\":\"2025-10-14T06:00:00\",\"latest\":\"2025-10-14T14:00:00\",\"flights\":3,\"noteCode\":null},{\"departureDate\":\"2025-10-15T00:00:00\",\"earliest\":\"2025-10-15T06:00:00\",\"latest\":\"2025-10-15T14:00:00\",\"flights\":3,\"noteCode\":null},{\"departureDate\":\"2025-10-16T00:00:00\",\"earliest\":\"2025-10-16T06:00:00\",\"latest\":\"2025-10-16T21:01:00\",\"flights\":16,\"noteCode\":null},{\"departureDate\":\"2025-10-17T00:00:00\",\"earliest\":\"2025-10-17T06:00:00\",\"latest\":\"2025-10-17T22:47:00\",\"flights\":8,\"noteCode\":null},{\"departureDate\":\"2025-10-18T00:00:00\",\"earliest\":\"2025-10-18T07:45:00\",\"latest\":\"2025-10-18T20:56:00\",\"flights\":4,\"noteCode\":null},{\"departureDate\":\"2025-10-19T00:00:00\",\"earliest\":\"2025-10-19T06:00:00\",\"latest\":\"2025-10-19T21:06:00\",\"flights\":16,\"noteCode\":null},{\"departureDate\":\"2025-10-20T00:00:00\",\"earliest\":\"2025-10-20T06:00:00\",\"latest\":\"2025-10-20T22:30:00\",\"flights\":10,\"noteCode\":null},{\"departureDate\":\"2025-10-21T00:00:00\",\"earliest\":\"2025-10-21T06:00:00\",\"latest\":\"2025-10-21T14:00:00\",\"flights\":3,\"noteCode\":null},{\"departureDate\":\"2025-10-22T00:00:00\",\"earliest\":\"2025-10-22T06:00:00\",\"latest\":\"2025-10-22T14:00:00\",\"flights\":3,\"noteCode\":null},{\"departureDate\":\"2025-10-23T00:00:00\",\"earliest\":\"2025-10-23T06:00:00\",\"latest\":\"2025-10-23T21:01:00\",\"flights\":16,\"noteCode\":null},{\"departureDate\":\"2025-10-24T00:00:00\",\"earliest\":\"2025-10-24T06:00:00\",\"latest\":\"2025-10-24T22:47:00\",\"flights\":13,\"noteCode\":null},{\"departureDate\":\"2025-10-25T00:00:00\",\"earliest\":\"2025-10-25T07:45:00\",\"latest\":\"2025-10-25T20:56:00\",\"flights\":4,\"noteCode\":null},{\"departureDate\":\"2025-10-26T00:00:00\",\"earliest\":\"2025-10-26T06:00:00\",\"latest\":\"2025-10-26T21:06:00\",\"flights\":16,\"noteCode\":null},{\"departureDate\":\"2025-10-27T00:00:00\",\"earliest\":\"2025-10-27T06:00:00\",\"latest\":\"2025-10-27T22:30:00\",\"flights\":7,\"noteCode\":null},{\"departureDate\":\"2025-10-28T00:00:00\",\"earliest\":\"2025-10-28T07:45:00\",\"latest\":\"2025-10-28T07:45:00\",\"flights\":2,\"noteCode\":null},{\"departureDate\":\"2025-10-29T00:00:00\",\"earliest\":\"2025-10-29T07:45:00\",\"latest\":\"2025-10-29T07:45:00\",\"flights\":2,\"noteCode\":null},{\"departureDate\":\"2025-10-30T00:00:00\",\"earliest\":\"2025-10-30T06:00:00\",\"latest\":\"2025-10-30T20:51:00\",\"flights\":13,\"noteCode\":null},{\"departureDate\":\"2025-10-31T00:00:00\",\"earliest\":\"2025-10-31T06:00:00\",\"latest\":\"2025-10-31T19:07:00\",\"flights\":10,\"noteCode\":null},{\"departureDate\":\"2025-11-01T00:00:00\",\"earliest\":\"2025-11-01T07:45:00\",\"latest\":\"2025-11-01T16:09:00\",\"flights\":3,\"noteCode\":null},{\"departureDate\":\"2025-11-02T00:00:00\",\"earliest\":\"2025-11-02T06:00:00\",\"latest\":\"2025-11-02T20:51:00\",\"flights\":13,\"noteCode\":null},{\"departureDate\":\"2025-11-03T00:00:00\",\"earliest\":\"2025-11-03T06:00:00\",\"latest\":\"2025-11-03T19:02:00\",\"flights\":10,\"noteCode\":null},{\"departureDate\":\"2025-11-04T00:00:00\",\"earliest\":\"2025-11-04T07:45:00\",\"latest\":\"2025-11-04T07:45:00\",\"flights\":2,\"noteCode\":null},{\"departureDate\":\"2025-11-05T00:00:00\",\"earliest\":\"2025-11-05T07:45:00\",\"latest\":\"2025-11-05T07:45:00\",\"flights\":2,\"noteCode\":null},{\"departureDate\":\"2025-11-06T00:00:00\",\"earliest\":\"2025-11-06T06:00:00\",\"latest\":\"2025-11-06T20:51:00\",\"flights\":13,\"noteCode\":null},{\"departureDate\":\"2025-11-07T00:00:00\",\"earliest\":\"2025-11-07T06:00:00\",\"latest\":\"2025-11-07T19:02:00\",\"flights\":10,\"noteCode\":null},{\"departureDate\":\"2025-11-08T00:00:00\",\"earliest\":\"2025-11-08T07:45:00\",\"latest\":\"2025-11-08T16:09:00\",\"flights\":3,\"noteCode\":null},{\"departureDate\":\"2025-11-09T00:00:00\",\"earliest\":\"2025-11-09T06:00:00\",\"latest\":\"2025-11-09T20:51:00\",\"flights\":13,\"noteCode\":null},{\"departureDate\":\"2025-11-10T00:00:00\",\"earliest\":\"2025-11-10T06:00:00\",\"latest\":\"2025-11-10T19:02:00\",\"flights\":10,\"noteCode\":null},{\"departureDate\":\"2025-11-11T00:00:00\",\"earliest\":\"2025-11-11T07:45:00\",\"latest\":\"2025-11-11T07:45:00\",\"flights\":2,\"noteCode\":null},{\"departureDate\":\"2025-11-12T00:00:00\",\"earliest\":\"2025-11-12T07:45:00\",\"latest\":\"2025-11-12T07:45:00\",\"flights\":2,\"noteCode\":null},{\"departureDate\":\"2025-11-13T00:00:00\",\"earliest\":\"2025-11-13T06:00:00\",\"latest\":\"2025-11-13T20:51:00\",\"flights\":13,\"noteCode\":null},{\"departureDate\":\"2025-11-14T00:00:00\",\"earliest\":\"2025-11-14T06:00:00\",\"latest\":\"2025-11-14T19:02:00\",\"flights\":10,\"noteCode\":null},{\"departureDate\":\"2025-11-15T00:00:00\",\"earliest\":\"2025-11-15T07:45:00\",\"latest\":\"2025-11-15T16:09:00\",\"flights\":3,\"noteCode\":null},{\"departureDate\":\"2025-11-16T00:00:00\",\"earliest\":\"2025-11-16T06:00:00\",\"latest\":\"2025-11-16T20:51:00\",\"flights\":13,\"noteCode\":null},{\"departureDate\":\"2025-11-17T00:00:00\",\"earliest\":\"2025-11-17T06:00:00\",\"latest\":\"2025-11-17T19:02:00\",\"flights\":10,\"noteCode\":null},{\"departureDate\":\"2025-11-18T00:00:00\",\"earliest\":\"2025-11-18T07:45:00\",\"latest\":\"2025-11-18T07:45:00\",\"flights\":2,\"noteCode\":null},{\"departureDate\":\"2025-11-19T00:00:00\",\"earliest\":\"2025-11-19T07:45:00\",\"latest\":\"2025-11-19T07:45:00\",\"flights\":2,\"noteCode\":null},{\"departureDate\":\"2025-11-20T00:00:00\",\"earliest\":\"2025-11-20T06:45:00\",\"latest\":\"2025-11-20T17:22:00\",\"flights\":8,\"noteCode\":null},{\"departureDate\":\"2025-11-21T00:00:00\",\"earliest\":\"2025-11-21T06:45:00\",\"latest\":\"2025-11-21T20:55:00\",\"flights\":13,\"noteCode\":null},{\"departureDate\":\"2025-11-22T00:00:00\",\"earliest\":\"2025-11-22T06:45:00\",\"latest\":\"2025-11-22T20:25:00\",\"flights\":9,\"noteCode\":null},{\"departureDate\":\"2025-11-23T00:00:00\",\"earliest\":\"2025-11-23T06:45:00\",\"latest\":\"2025-11-23T20:25:00\",\"flights\":11,\"noteCode\":null},{\"departureDate\":\"2025-11-24T00:00:00\",\"earliest\":\"2025-11-24T06:45:00\",\"latest\":\"2025-11-24T20:55:00\",\"flights\":15,\"noteCode\":null},{\"departureDate\":\"2025-11-25T00:00:00\",\"earliest\":\"2025-11-25T06:45:00\",\"latest\":\"2025-11-25T20:25:00\",\"flights\":10,\"noteCode\":null},{\"departureDate\":\"2025-11-26T00:00:00\",\"earliest\":\"2025-11-26T06:45:00\",\"latest\":\"2025-11-26T20:55:00\",\"flights\":13,\"noteCode\":null},{\"departureDate\":\"2025-11-27T00:00:00\",\"earliest\":\"2025-11-27T06:45:00\",\"latest\":\"2025-11-27T17:22:00\",\"flights\":6,\"noteCode\":null},{\"departureDate\":\"2025-11-28T00:00:00\",\"earliest\":\"2025-11-28T06:45:00\",\"latest\":\"2025-11-28T20:55:00\",\"flights\":13,\"noteCode\":null},{\"departureDate\":\"2025-11-29T00:00:00\",\"earliest\":\"2025-11-29T06:45:00\",\"latest\":\"2025-11-29T20:25:00\",\"flights\":9,\"noteCode\":null},{\"departureDate\":\"2025-11-30T00:00:00\",\"earliest\":\"2025-11-30T06:45:00\",\"latest\":\"2025-11-30T20:25:00\",\"flights\":11,\"noteCode\":null},{\"departureDate\":\"2025-12-01T00:00:00\",\"earliest\":\"2025-12-01T06:45:00\",\"latest\":\"2025-12-01T20:55:00\",\"flights\":11,\"noteCode\":null},{\"departureDate\":\"2025-12-02T00:00:00\",\"earliest\":\"2025-12-02T08:00:00\",\"latest\":\"2025-12-02T08:00:00\",\"flights\":1,\"noteCode\":null},{\"departureDate\":\"2025-12-03T00:00:00\",\"earliest\":\"2025-12-03T08:00:00\",\"latest\":\"2025-12-03T08:00:00\",\"flights\":2,\"noteCode\":null},{\"departureDate\":\"2025-12-04T00:00:00\",\"earliest\":\"2025-12-04T06:45:00\",\"latest\":\"2025-12-04T22:38:00\",\"flights\":16,\"noteCode\":null},{\"departureDate\":\"2025-12-05T00:00:00\",\"earliest\":\"2025-12-05T06:45:00\",\"latest\":\"2025-12-05T21:09:00\",\"flights\":9,\"noteCode\":null},{\"departureDate\":\"2025-12-06T00:00:00\",\"earliest\":\"2025-12-06T06:45:00\",\"latest\":\"2025-12-06T06:45:00\",\"flights\":2,\"noteCode\":null},{\"departureDate\":\"2025-12-07T00:00:00\",\"earliest\":\"2025-12-07T06:45:00\",\"latest\":\"2025-12-07T22:38:00\",\"flights\":18,\"noteCode\":null},{\"departureDate\":\"2025-12-08T00:00:00\",\"earliest\":\"2025-12-08T06:45:00\",\"latest\":\"2025-12-08T21:09:00\",\"flights\":12,\"noteCode\":null},{\"departureDate\":\"2025-12-09T00:00:00\",\"earliest\":\"2025-12-09T08:00:00\",\"latest\":\"2025-12-09T08:00:00\",\"flights\":1,\"noteCode\":null},{\"departureDate\":\"2025-12-10T00:00:00\",\"earliest\":\"2025-12-10T08:00:00\",\"latest\":\"2025-12-10T08:00:00\",\"flights\":2,\"noteCode\":null},{\"departureDate\":\"2025-12-11T00:00:00\",\"earliest\":\"2025-12-11T06:45:00\",\"latest\":\"2025-12-11T22:38:00\",\"flights\":16,\"noteCode\":null},{\"departureDate\":\"2025-12-12T00:00:00\",\"earliest\":\"2025-12-12T06:45:00\",\"latest\":\"2025-12-12T21:09:00\",\"flights\":9,\"noteCode\":null},{\"departureDate\":\"2025-12-13T00:00:00\",\"earliest\":\"2025-12-13T06:45:00\",\"latest\":\"2025-12-13T06:45:00\",\"flights\":2,\"noteCode\":null},{\"departureDate\":\"2025-12-14T00:00:00\",\"earliest\":\"2025-12-14T06:45:00\",\"latest\":\"2025-12-14T22:38:00\",\"flights\":18,\"noteCode\":null},{\"departureDate\":\"2025-12-15T00:00:00\",\"earliest\":\"2025-12-15T06:45:00\",\"latest\":\"2025-12-15T21:09:00\",\"flights\":12,\"noteCode\":null},{\"departureDate\":\"2025-12-16T00:00:00\",\"earliest\":\"2025-12-16T08:00:00\",\"latest\":\"2025-12-16T08:00:00\",\"flights\":1,\"noteCode\":null},{\"departureDate\":\"2025-12-17T00:00:00\",\"earliest\":\"2025-12-17T08:00:00\",\"latest\":\"2025-12-17T08:00:00\",\"flights\":2,\"noteCode\":null},{\"departureDate\":\"2025-12-18T00:00:00\",\"earliest\":\"2025-12-18T06:45:00\",\"latest\":\"2025-12-18T19:13:00\",\"flights\":14,\"noteCode\":null},{\"departureDate\":\"2025-12-19T00:00:00\",\"earliest\":\"2025-12-19T06:45:00\",\"latest\":\"2025-12-19T18:57:00\",\"flights\":7,\"noteCode\":null},{\"departureDate\":\"2025-12-20T00:00:00\",\"earliest\":\"2025-12-20T06:45:00\",\"latest\":\"2025-12-20T18:57:00\",\"flights\":5,\"noteCode\":null},{\"departureDate\":\"2025-12-21T00:00:00\",\"earliest\":\"2025-12-21T06:45:00\",\"latest\":\"2025-12-21T19:13:00\",\"flights\":14,\"noteCode\":null},{\"departureDate\":\"2025-12-22T00:00:00\",\"earliest\":\"2025-12-22T06:45:00\",\"latest\":\"2025-12-22T18:57:00\",\"flights\":7,\"noteCode\":null},{\"departureDate\":\"2025-12-23T00:00:00\",\"earliest\":\"2025-12-23T06:45:00\",\"latest\":\"2025-12-23T19:13:00\",\"flights\":13,\"noteCode\":null},{\"departureDate\":\"2025-12-24T00:00:00\",\"earliest\":\"2025-12-24T06:45:00\",\"latest\":\"2025-12-24T17:18:00\",\"flights\":3,\"noteCode\":null},{\"departureDate\":\"2025-12-25T00:00:00\",\"earliest\":\"2025-12-25T06:45:00\",\"latest\":\"2025-12-25T19:13:00\",\"flights\":13,\"noteCode\":null},{\"departureDate\":\"2025-12-26T00:00:00\",\"earliest\":\"2025-12-26T06:45:00\",\"latest\":\"2025-12-26T18:57:00\",\"flights\":7,\"noteCode\":null},{\"departureDate\":\"2025-12-27T00:00:00\",\"earliest\":\"2025-12-27T06:45:00\",\"latest\":\"2025-12-27T18:57:00\",\"flights\":5,\"noteCode\":null},{\"departureDate\":\"2025-12-28T00:00:00\",\"earliest\":\"2025-12-28T06:45:00\",\"latest\":\"2025-12-28T19:13:00\",\"flights\":14,\"noteCode\":null},{\"departureDate\":\"2025-12-29T00:00:00\",\"earliest\":\"2025-12-29T06:45:00\",\"latest\":\"2025-12-29T18:57:00\",\"flights\":7,\"noteCode\":null},{\"departureDate\":\"2025-12-30T00:00:00\",\"earliest\":\"2025-12-30T06:45:00\",\"latest\":\"2025-12-30T19:13:00\",\"flights\":13,\"noteCode\":null},{\"departureDate\":\"2025-12-31T00:00:00\",\"earliest\":\"2025-12-31T06:45:00\",\"latest\":\"2025-12-31T17:18:00\",\"flights\":3,\"noteCode\":null},{\"departureDate\":\"2026-01-01T00:00:00\",\"earliest\":\"2026-01-01T06:45:00\",\"latest\":\"2026-01-01T19:13:00\",\"flights\":14,\"noteCode\":null},{\"departureDate\":\"2026-01-02T00:00:00\",\"earliest\":\"2026-01-02T06:45:00\",\"latest\":\"2026-01-02T18:57:00\",\"flights\":7,\"noteCode\":null},{\"departureDate\":\"2026-01-03T00:00:00\",\"earliest\":\"2026-01-03T06:45:00\",\"latest\":\"2026-01-03T18:57:00\",\"flights\":5,\"noteCode\":null},{\"departureDate\":\"2026-01-04T00:00:00\",\"earliest\":\"2026-01-04T06:45:00\",\"latest\":\"2026-01-04T19:13:00\",\"flights\":14,\"noteCode\":null},{\"departureDate\":\"2026-01-05T00:00:00\",\"earliest\":\"2026-01-05T06:45:00\",\"latest\":\"2026-01-05T18:57:00\",\"flights\":4,\"noteCode\":null}]}" ] } ``` ### Status Types | Status | Meaning | | ---------- | ---------------------------------------------------- | | queued | Task was accepted and added to the processing queue. | | processing | Task is currently running. | | completed | Task finished and flight data was collected. | | failed | Task encountered an error (invalid URL, etc.). | ## Polling If `status` is `queued` or `processing`, continue polling the task until it's completed or failed. ``` GET https://api.crawlbyte.ai/api/tasks/:id ``` * You’ll receive the same structure with an updated `status`. * Only poll until you receive `completed` or `failed`. * Recommended interval: **every 3–5 seconds**. ## SDK Usage You can run this task using any official **Crawlbyte SDK**: * [Go SDK](https://github.com/crawlbyte/crawlbyte-sdk-go) * [TypeScript SDK](https://github.com/crawlbyte/crawlbyte-sdk-ts) * [Python SDK](https://github.com/crawlbyte/crawlbyte-sdk-py) Each SDK provides a simple way to: * Create the task * Poll for status * Handle the final result Refer to the [SDKs section](/sdks) for installation, examples, and setup instructions. ## Notes * Only valid FlyFrontier App URLs will return results. * Crawlbyte handles retries, rendering, fingerprinting, and anti-bot logic internally — no need to manage it yourself. * Use `multithread: true` in advanced settings if running large volumes. # HAR Scraper Scrape real estate listings directly from HAR using structured input data. This task supports flexible filtering by city and price, and allows sorting the results by listing date or price. ## Endpoint ``` POST https://api.crawlbyte.ai/api/tasks ``` ## Basic Configuration (Required) ```json { "type": "har", "input": [ "{\"city\":\"Dallas\",\"priceMin\":10000,\"priceMax\":50000}" ], "sortBy": "CHEAPEST_FIRST" } ``` ### Parameters | Field | Type | Description | | ----------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | type | string | Always `"har"` | | input | array | Array of JSON-formatted strings with city and optional price filters | | sortBy | string |

Controls result order:
• NEWEST\_FIRST
• OLDEST\_FIRST
• CHEAPEST\_FIRST
• EXPENSIVE\_FIRST

| | multithread | boolean | Use `true` for faster processing with multiple threads | ## Input Object Format ``` "{\"city\":\"Houston\",\"priceMin\":100000,\"priceMax\":200000}" ``` * `city` (required): Target city name * `priceMin` (optional): Minimum price * `priceMax` (optional): Maximum price You may use only `priceMin` or `priceMax` if desired. ## Advanced Configuration (Optional) ```json { "type": "har", "input": [ "{\"city\":\"Dallas\",\"priceMin\":10000,\"priceMax\":50000}" ], "sortBy": "CHEAPEST_FIRST", "user_agent_preset": "chrome", "user_agent_custom": "", "headers": "{\"X-Test\":\"abc\"}", "cookie": "session=xyz", "proxy": "http://username:password@ip:port" } ``` ### Optional Parameters | Field | Type | Description | | ------------------- | ------ | -------------------------------------------------------------------------------------------------------------------- | | user\_agent\_preset | string | Preset user-agent. Options: `chrome`, `firefox`, `edge`, `opera`, `safari`, `ios-safari`, `android-chrome`, `custom` | | user\_agent\_custom | string | Used if `user_agent_preset` is `custom.` | | headers | string | JSON-formatted string of headers. | | cookie | string | `key=value;` | | proxy | string | `http://username:password@ip:port` | ## Pricing * **$0.005 per successful task**\ This is a pay-as-you-go pricing model — you're only charged when a HAR task successfully returns listings. You can view your current credit balance and usage history in the [Crawlbyte Dashboard](https://dash.crawlbyte.ai/). ## Response The response contains metadata about the task. For `har` type, the most important fields are `status` and `result`. ```json { "id": "bd3e89ed-815e-4395-98a3-521ede71cc4d", "status": "completed", "result": { // all listings here } } ``` * `result` is a **JSON object,** that's the final scraped results — no further polling is needed. ### Status Types | Status | Meaning | | ---------- | ---------------------------------------------------- | | queued | Task was accepted and added to the processing queue. | | processing | Task is currently running. | | completed | Task finished and listings were collected. | | failed | Task encountered an error (invalid URL, etc.). | ## Polling If `status` is `queued` or `processing`, continue polling the task until it's completed or failed. ``` GET https://api.crawlbyte.ai/api/tasks/:id ``` * You’ll receive the same structure with an updated `status`. * Only poll until you receive `completed` or `failed`. * Recommended interval: **every 3–5 seconds**. ## SDK Usage You can run this task using any official **Crawlbyte SDK**: * [Go SDK](https://github.com/crawlbyte/crawlbyte-sdk-go) * [TypeScript SDK](https://github.com/crawlbyte/crawlbyte-sdk-ts) * [Python SDK](https://github.com/crawlbyte/crawlbyte-sdk-py) Each SDK provides a simple way to: * Create the task * Poll for status * Handle the final result Refer to the [SDKs section](/sdks) for installation, examples, and setup instructions. ## Notes * Input values must be valid — misspelled cities or malformed price fields will result in empty results or failure. * The `input` must always be a JSON string, even if it contains only the city field. * `sortBy` is required and must exactly match one of the supported options. * Crawlbyte automatically handles pagination, anti-bot protections, and request sequencing. * `multithread: true` can be used in advanced settings for faster bulk processing. # LinkedIn Scraper Scrapes LinkedIn profile data from either individual profiles or companies. The task accepts LinkedIn profile URLs and returns structured data. ## Endpoint ``` POST https://api.crawlbyte.ai/api/tasks ``` ## Basic Configuration (Required) ```json { "type": "linkedin", "input": [ "https://www.linkedin.com/in/some-person/", "https://www.linkedin.com/company/some-company/" ], "dataType": "profiles", "multithread": false, "cookie": "li_at=YOUR_COOKIE_HERE; JSESSIONID=YOUR_COOKIE_HERE" } ``` ### Parameters | Field | Type | Description | | ----------- | ------- | ----------------------------------------------------------------- | | type | string | Always `"linkedin"` | | input | array | Array of full LinkedIn URLs | | dataType | string | Either `"profiles"` for people or `"companies"` for organizations | | multithread | boolean | Use `true` for faster processing with multiple threads | | cookie | string | `key=value;` | ## Advanced Configuration (Optional) ```json { "type": "linkedin", "input": [ "https://www.linkedin.com/in/some-person/" ], "dataType": "profiles", "user_agent_preset": "chrome", "headers": "{\"X-Test\":\"abc\"}", "proxy": "http://username:password@ip:port" } ``` ### Optional Parameters | Field | Type | Description | | ------------------- | ------ | -------------------------------------------------------------------------------------------------------------------- | | user\_agent\_preset | string | Preset user-agent. Options: `chrome`, `firefox`, `edge`, `opera`, `safari`, `ios-safari`, `android-chrome`, `custom` | | user\_agent\_custom | string | Used if `user_agent_preset` is `custom.` | | headers | string | JSON-formatted string of headers. | | proxy | string | `http://username:password@ip:port` | ## Pricing * **$0.0075 per successful task**\ This is a pay-as-you-go pricing model — you're only charged when a LinkedIn task successfully returns public data. You can view your current credit balance and usage history in the [Crawlbyte Dashboard](https://dash.crawlbyte.ai/). ## Response The response contains metadata about the task. For linkedin type, the most important fields are `status` and `result`. ```json { "id": "af3e12f2-8f45-43b0-8a7b-cabbbb94c1e9", "status": "completed", "result": "JSON_RESULT_HERE" } ``` * If `result` is a **JSON object**, the data is already available — no polling needed. ### Status Types | Status | Meaning | | ---------- | -------------------------------------------------------------- | | queued | Task was accepted and added to the processing queue. | | processing | Task is currently running. | | completed | Task finished successfully. | | failed | Task encountered an error (e.g. bad proxy, invalid URL, etc.). | ## Polling If the initial `status` is `queued` or `processing`, you should poll for task completion. ``` GET https://api.crawlbyte.ai/api/tasks/:id ``` * You’ll receive the same structure with an updated `status`. * Only poll until you receive `completed` or `failed`. * Recommended interval: **every 2–3 seconds**. ## SDK Usage You can run this task using any official **Crawlbyte SDK**: * [Go SDK](https://github.com/crawlbyte/crawlbyte-sdk-go) * [TypeScript SDK](https://github.com/crawlbyte/crawlbyte-sdk-ts) * [Python SDK](https://github.com/crawlbyte/crawlbyte-sdk-py) Each SDK provides a simple way to: * Create the task * Poll for status * Handle the final result Refer to the [SDKs section](/sdks) for installation, examples, and setup instructions. ## Notes * This task supports all LinkedIn profiles and company pages. * A session cookie is typically required. Use the [Crawlbyte Chrome Extension](https://developers.crawlbyte.ai/linkedin-scraper) to extract it. * While a session cookie is typically needed, alternative handling may be available in advanced setups. Feel free to reach out to explore what works best for your use case. * Make sure any session usage complies with Crawlbyte’s Terms of Service. * You can safely run large batches using `multithread: true`. # Target Scraper Scrapes product information such as title, price, availability, images, descriptions, and other related data based on the provided product ID/TCIN. ## Endpoint ``` POST https://api.crawlbyte.ai/api/tasks ``` ## Basic Configuration (Required) 
{
  "type": "target",
  "input": [
    "1003664214",
    "87975302",
  ],
  "multithread": false
}
### Parameters | Field | Type | Description | | ----------- | ------- | ------------------------------------------------------ | | type | string | Always `"target"` | | input | array | Array of Target product ID/TCIN's | | multithread | boolean | Use `true` for faster processing with multiple threads | ## Advanced Configuration (Optional) ```json { "type": "target", "input": [ "1003664214", "87975302", ], "fields": [ "title", "price", "availability", "url", "image", "description", "rating", "upc", "variations" ], "user_agent_preset": "chrome", "user_agent_custom": "", "proxy": "http://username:password@ip:port" } ``` ### Optional Parameters | Field | Type | Description | | ------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | fields | array | List of fields to scrape. Available options: `title`, `price`, `availability`, `url`, `image`, `description`, `rating`, `upc`, `variations`. Leave empty to scrape all. | | user\_agent\_preset | string | Preset user-agent. Options: `chrome`, `firefox`, `edge`, `opera`, `safari`, `ios-safari`, `android-chrome`, `custom` | | user\_agent\_custom | string | Used if `user_agent_preset` is `custom.` | | proxy | string | `http://username:password@ip:port` | ## Pricing * **$0.0050 per successful task**\ This is a pay-as-you-go pricing model — you're only charged when a Target task successfully returns product data. You can view your current credit balance and usage history in the [Crawlbyte Dashboard](https://dash.crawlbyte.ai/). ## Response The response contains metadata about the task. For `target` type, the most important fields are `status` and `result`. ```json { "id": "bd3e89ed-815e-4395-98a3-521ede71cc4d", "status": "completed", "result": { // product data } } ``` * `result` is a **JSON object,** that's the final scraped product data — no further polling is needed. ### Status Types | Status | Meaning | | ---------- | -------------------------------------------------------------------------- | | queued | Task was accepted and added to the processing queue. | | processing | Task is currently running. | | completed | Task finished and product data was successfully collected. | | failed | Task encountered an error (e.g., invalid product ID, access denied, etc.). | ## Polling If `status` is `queued` or `processing`, continue polling the task until it's completed or failed. ``` GET https://api.crawlbyte.ai/api/tasks/:id ``` * You’ll receive the same structure with an updated `status`. * Only poll until you receive `completed` or `failed`. * Recommended interval: **every 3–5 seconds**. ## SDK Usage You can run this task using any official **Crawlbyte SDK**: * [Go SDK](https://github.com/crawlbyte/crawlbyte-sdk-go) * [TypeScript SDK](https://github.com/crawlbyte/crawlbyte-sdk-ts) * [Python SDK](https://github.com/crawlbyte/crawlbyte-sdk-py) Each SDK provides a simple way to: * Create the task * Poll for status * Handle the final result Refer to the [SDKs section](/sdks) for installation, examples, and setup instructions. ## Notes * Only valid Target product IDs (TCINs) will return results. * Crawlbyte handles retries, fingerprinting, and anti-bot logic internally — no need to configure anything manually. * Use `multithread: true` in advanced settings when processing large volumes. * If a specific field is missing in the result, it means the data wasn’t available or couldn’t be parsed — usually all supported fields are returned. * Need more fields? Just contact our support and we’ll add them. * Average task duration is \~1–2 seconds per product, depending on Target’s site response time. # Universal Scraper Scrape any public website using the Crawlbyte universal task engine. Designed for flexibility, it supports GET, POST, and custom selectors, JS rendering, headers, proxies, and more. ## Endpoint ``` POST https://api.crawlbyte.ai/api/tasks ``` ## Basic Configuration (Required) ```json { "type": "universal", "input": [ "https://example.com/page" ], "multithread": false } ``` ### Parameters | Field | Type | Description | | ----------- | ------- | ------------------------------------------------------ | | type | string | Always `"universal"` | | input | array | Array of valid URLs | | multithread | boolean | Use `true` for faster processing with multiple threads | ## Advanced Configuration (Optional) ```json { "type": "universal", "input": [ "https://example.com/page" ], "multithread": true, "jsRendering": true, "customSelector": "#main-content", "method": "GET", "headers": "{\"X-Test\":\"abc\"}", "cookie": "session=xyz", "proxy": "http://username:password@ip:port" } ``` ### Optional Parameters | Field | Type | Description | | ------------------- | ------- | -------------------------------------------------------------------------------------------------------------------- | | jsRendering | boolean | Enable full-page JavaScript rendering. | | customSelector | string | CSS selector (e.g., `#main-content`) to extract specific HTML. | | method | string | HTTP method (`GET`, `POST`, `PUT`, `PATCH`) | | user\_agent\_preset | string | Preset user-agent. Options: `chrome`, `firefox`, `edge`, `opera`, `safari`, `ios-safari`, `android-chrome`, `custom` | | user\_agent\_custom | string | Used if `user_agent_preset` is `custom` | | headers | string | JSON-formatted string of headers. | | cookie | string | `key=value;` | | proxy | string | `http://username:password@ip:port` | ## Pricing * **$0.005 per successful task**\ This is a pay-as-you-go pricing model — you're only charged when a Universal task successfully returns HTML or JSON content. You can view your current credit balance and usage history in the [Crawlbyte Dashboard](https://dash.crawlbyte.ai/). ## Response The response contains metadata about the task. For `universal` type, the most important fields are `status` and `result`. ```json { "id": "af3e12f2-8f45-43b0-8a7b-cabbbb94c1e9", "status": "completed", "result": "HTML or JSON RESULT_HERE" } ``` * If `result` is **raw HTML** or a **JSON object**, no further polling is needed — this is the final data. ### Status Types | Status | Meaning | | ---------- | -------------------------------------------------------------- | | queued | Task was accepted and added to the processing queue. | | processing | Task is currently running. | | completed | Task finished successfully (emails were submitted to Beehiiv). | | failed | Task encountered an error (e.g. bad proxy, invalid URL, etc.). | ## Polling If the initial `status` is `queued` or `processing`, you should poll for task completion. ``` GET https://api.crawlbyte.ai/api/tasks/:id ``` * You’ll receive the same structure with an updated `status`. * Only poll until you receive `completed` or `failed`. * Rendered pages (with `jsRendering: true`) may take slightly longer. ## SDK Usage You can run this task using any official **Crawlbyte SDK**: * [Go SDK](https://github.com/crawlbyte/crawlbyte-sdk-go) * [TypeScript SDK](https://github.com/crawlbyte/crawlbyte-sdk-ts) * [Python SDK](https://github.com/crawlbyte/crawlbyte-sdk-py) Each SDK provides a simple way to: * Create the task * Poll for status * Handle the final result Refer to the [SDKs section](/sdks) for installation, examples, and setup instructions. ## Notes * The universal scraper is ideal for **public webpages** that don’t require a login. * Crawlbyte handles fingerprinting, headers, JS execution, bot detection, and more automatically. * For complex or highly protected pages, the task may fail. If this happens: [**Book a free setup call**](https://calendly.com/kristian-crawlbyte/crawlbyte-demo-call?preview_source=et_card) — we’ll configure a custom scraper template for your target site at no cost. * `jsRendering` is optional but may be required for dynamic sites (e.g., React, Vue, Angular). * If a `cookie` string is provided, it’s used as-is — **you are fully responsible for its usage** in accordance with our [Terms of Service](https://www.crawlbyte.ai/terms-of-service). # Walmart Scraper Scrapes product information such as title, price, availability, images, descriptions, and other related data based on the provided product ID. ## Endpoint ``` POST https://api.crawlbyte.ai/api/tasks ``` ## Basic Configuration (Required) ```json { "type": "walmart", "input": [ "14091452016", "7778210362", "11653805194" ], "multithread": false } ``` ### Parameters | Field | Type | Description | | ----------- | ------- | ------------------------------------------------------ | | type | string | Always `"walmart"` | | input | array | Array of Walmart product IDs | | multithread | boolean | Use `true` for faster processing with multiple threads | ## Advanced Configuration (Optional) ```json { "type": "walmart", "input": [ "14091452016", "7778210362", "11653805194" ], "fields": [ "title", "price", "availability", "url", "image", "description", "rating", "upc", "model" ], "user_agent_preset": "chrome", "user_agent_custom": "", "proxy": "http://username:password@ip:port" } ``` ### Optional Parameters | Field | Type | Description | | ------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | fields | array | List of fields to scrape. Available options: `title`, `price`, `availability`, `url`, `image`, `description`, `rating`, `upc`, `model`. Leave empty to scrape all. | | user\_agent\_preset | string | Preset user-agent. Options: `chrome`, `firefox`, `edge`, `opera`, `safari`, `ios-safari`, `android-chrome`, `custom` | | user\_agent\_custom | string | Used if `user_agent_preset` is `custom.` | | proxy | string | `http://username:password@ip:port` | ## Pricing * **$0.01 per successful task**\ This is a pay-as-you-go pricing model — you're only charged when a Walmart task successfully returns product data. You can view your current credit balance and usage history in the [Crawlbyte Dashboard](https://dash.crawlbyte.ai/). ## Response The response contains metadata about the task. For `walmart` type, the most important fields are `status` and `result`. ```json { "id": "bd3e89ed-815e-4395-98a3-521ede71cc4d", "status": "completed", "result": { // product data } } ``` * `result` is a **JSON object,** that's the final scraped product data — no further polling is needed. ### Status Types | Status | Meaning | | ---------- | -------------------------------------------------------------------------- | | queued | Task was accepted and added to the processing queue. | | processing | Task is currently running. | | completed | Task finished and product data was successfully collected. | | failed | Task encountered an error (e.g., invalid product ID, access denied, etc.). | ## Polling If `status` is `queued` or `processing`, continue polling the task until it's completed or failed. ``` GET https://api.crawlbyte.ai/api/tasks/:id ``` * You’ll receive the same structure with an updated `status`. * Only poll until you receive `completed` or `failed`. * Recommended interval: **every 3–5 seconds**. ## SDK Usage You can run this task using any official **Crawlbyte SDK**: * [Go SDK](https://github.com/crawlbyte/crawlbyte-sdk-go) * [TypeScript SDK](https://github.com/crawlbyte/crawlbyte-sdk-ts) * [Python SDK](https://github.com/crawlbyte/crawlbyte-sdk-py) Each SDK provides a simple way to: * Create the task * Poll for status * Handle the final result Refer to the [SDKs section](/sdks) for installation, examples, and setup instructions. ## Notes * Only valid Walmart product IDs will return results. * Crawlbyte handles retries, fingerprinting, and anti-bot logic internally — no need to configure anything manually. * Use `multithread: true` in advanced settings when processing large volumes. * If a specific field is missing in the result, it means the data wasn’t available or couldn’t be parsed — usually all supported fields are returned. * Need more fields? Just contact our support and we’ll add them. * Average task duration is \~2–3 seconds per product, depending on Walmart’s site response time. # Yelp Scraper Scrapes public data from Yelp business profiles or review listings. Crawlbyte handles fingerprinting, anti-bot protections, and pagination — so you can collect structured data from Yelp effortlessly. ## Endpoint ``` POST https://api.crawlbyte.ai/api/tasks ``` ## Basic Configuration (Required) #### **Business Profile** ```json { "type": "yelp", "input": [ "https://www.yelp.com/biz/some-restaurant" ], "dataType": "profiles", "multithread": false } ``` #### **Reviews** ```json { "type": "yelp", "input": [ "https://www.yelp.com/biz/some-restaurant" ], "dataType": "reviews", "sortBy": "NEWEST_FIRST", "multithread": false } ``` ### Parameters | Field | Type | Description | | ----------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | type | string | Always `"yelp"` | | input | array | Array of Yelp business URLs | | dataType | string | `"profiles"` or `"reviews"` | | sortBy | string |

Only for reviews — controls review order:
• NEWEST\_FIRST
• OLDEST\_FIRST
• HIGHEST\_RATED
• LOWEST\_RATED

| | multithread | boolean | Use `true` for faster processing with multiple threads | ## Advanced Configuration (Optional) ```json { "type": "yelp", "input": [ "https://www.yelp.com/biz/some-restaurant" ], "dataType": "reviews", "sortBy": "NEWEST_FIRST", "user_agent_preset": "chrome", "headers": "{\"X-Test\":\"abc\"}", "cookie": "", "proxy": "http://username:password@ip:port" } ``` ### Optional Parameters | Field | Type | Description | | ------------------- | ------ | -------------------------------------------------------------------------------------------------------------------- | | user\_agent\_preset | string | Preset user-agent. Options: `chrome`, `firefox`, `edge`, `opera`, `safari`, `ios-safari`, `android-chrome`, `custom` | | user\_agent\_custom | string | Used if `user_agent_preset` is `custom.` | | headers | string | JSON-formatted string of headers. | | cookie | string | `key=value;` | | proxy | string | `http://username:password@ip:port` | ## Pricing * **$0.005 per successful task**\ This is a pay-as-you-go pricing model — you're only charged when a Yelp task successfully returns listings or reviews. You can view your current credit balance and usage history in the [Crawlbyte Dashboard](https://dash.crawlbyte.ai/). ## Response The response contains metadata about the task. For the `yelp` type, the key fields in the response are `status` and `result`. ```json { "id": "af3e12f2-8f45-43b0-8a7b-cabbbb94c1e9", "status": "completed", "result": "JSON_RESULT_HERE" } ``` * If `result` is a **hash**, you must poll `/api/tasks/:id` to retrieve the full data. * If `result` is a **JSON object**, the data is already available — no polling needed. ### Status Types | Status | Meaning | | ---------- | -------------------------------------------------------------- | | queued | Task was accepted and added to the processing queue. | | processing | Task is currently running. | | completed | Task finished successfully and data was collected from Yelp. | | failed | Task encountered an error (e.g. bad proxy, invalid URL, etc.). | ## Polling If the initial `status` is `queued` or `processing`, you should poll for task completion. ``` GET https://api.crawlbyte.ai/api/tasks/:id ``` * You’ll receive the same structure with an updated `status`. * Only poll until you receive `completed` or `failed`. * Average time: **2–4 seconds**, but longer for reviews (full scraping). ## SDK Usage You can run this task using any official **Crawlbyte SDK**: * [Go SDK](https://github.com/crawlbyte/crawlbyte-sdk-go) * [TypeScript SDK](https://github.com/crawlbyte/crawlbyte-sdk-ts) * [Python SDK](https://github.com/crawlbyte/crawlbyte-sdk-py) Each SDK provides a simple way to: * Create the task * Poll for status * Handle the final result Refer to the [SDKs section](/sdks) for installation, examples, and setup instructions. ## Notes * This task supports **public Yelp business pages and review listings**. * If `dataType` is set to `"reviews"`, the `sortBy` field is **required**. Accepted values: `NEWEST_FIRST`, `OLDEST_FIRST`, `HIGHEST_RATED`, `LOWEST_RATED`. * When scraping reviews, Crawlbyte fetches **all available reviews across all pages**, which may take longer for listings with high volume. * Crawlbyte handles fingerprinting, bot detection, and pagination internally — no need to configure anything manually. * You can batch multiple business URLs using `multithread: true`.