What is HuluFlow: productized scraping instead of one-off scripts

Why productize scraping instead of writing another script

In many teams, web scraping stays stuck as a pile of one-off scripts. Marketing needs a directory list; an engineer ships a crawler. Two weeks later the markup changes and the script dies. A month after that, ops wants scheduled price checks, so another repo and another cron job appear. Scripts scatter, environments are hard to reproduce, field names drift, and “notifications” mean someone pasting a spreadsheet into chat. Ownership of proxies, rate limits, public-page-only policy, and durable storage is equally fuzzy.

Throwaway scripts are fine for exploration. They break down when the real requirement is weekly runs, stable fields, exportable tables, change alerts, and API access for downstream systems. HuluFlow exists for that productized path: describe scrape jobs as editable, schedulable, metered workflows; land rows in Datasets; manage plans and credentials in one console.

What HuluFlow is

HuluFlow is a public-web scraping SaaS. English marketing lives at /; Chinese at /zh/. The product owns accounts and credit plans, a visual workflow editor, Datasets, schedules, email notify, a REST API, and a Chrome extension. The crawl engine is external crawld-web; this app focuses on orchestration, quotas, and delivery.

You do not need to operate a headless browser fleet or keep brittle selectors in a business repo. The usual path is: paste or generate URLs → discover fields and preview samples → wire nodes on the canvas → run manually or on an interval → land rows in a Dataset, optionally email on change, then pull via API keys into internal tools.

Start with the docs and use cases. After sign-up, the console at /app centers on workflows and datasets.

Four core nodes in a workflow

A Workflow is a DAG stored as graph_json. The Drawflow editor deliberately limits node types so the model stays teachable and billing stays clear.

1. url_gen — build the URL list

url_gen has no upstream inputs. It emits URLs from a range template (pagination) or a pasted list, with a hard cap (for example 500) so a single run cannot explode. Price monitors, catalog paging, and fixed shop lists all start here.

2. scrape — list or detail fetch

scrape fetches via crawld-web in list or detail mode. It can take a URL or fan out over upstream items[].url|link. Successful output rows consume credits; detail pages typically cost more. Field discovery and preview let you finalize columns from samples instead of hard-coding selectors first.

3. store — upsert into a Dataset

store upserts rows into a logical Dataset, keyed by fields such as url and link. Browse in the console, export CSV/JSON, or page through the API. Scrapes become living tables instead of one-shot downloads.

4. notify — email on new rows or field changes

notify can fire on when=new or when=field_change. A common price-monitor chain is list scrape → detail scrape → store → notify when price moves. Operators should not have to diff spreadsheets every morning.

A typical pipeline is url_gen → list scrape → detail scrape → store → notify. Wiring rules live in the workflows guide.

Credits, plans, and schedules

Usage is metered in page-request credits: successful scrape work consumes from a monthly (or yearly prepaid) pool. Public tiers range from a small Free allowance to Starter, Pro, and Business. Free is enough to validate workflows and the extension; serious monitoring and large lists need a paid pool so jobs do not stop mid-run.

Schedules use interval_minutes and next_run_at. An in-process worker (or standalone python -m app.worker) picks up due workflows. Cloud schedules cover “run while nobody has a browser open”; the Chrome extension covers “I am looking at this page right now—analyze and save a workflow.” They complement each other.

API keys (Authorization: Bearer hulu_…) let you trigger runs, inspect run history, and read Datasets for ETL or reporting. See the API reference and API-first guide.

Versus one-off scripts and in-house crawler platforms

Custom scripts win on total control: arbitrary parsers, storage, and schedulers. You also own retries, markup drift, secrets, alerting, and multi-person collaboration. An internal crawler platform is heavier still—queues, sandboxes, authz, and multi-tenant metering become another product.

HuluFlow sits in the middle: crawld-web executes fetches; the product layer owns the graph, Datasets, credits, and API. You gain faster time-to-value and less ops surface; you give up unbounded custom parse code. Deep login walls, CAPTCHA warfare, or aggressive anti-bot bypass are explicitly out of scope. The product targets public pages you are allowed to collect, expressed as discoverable fields and workflows.

For most growth, ops, intel, and data teams, that boundary is the useful one: lists become tables, price moves trigger mail, pagination lands in Datasets, APIs sync warehouses—without opening a new crawler repo per request.

Who it is for (and who it is not)

A fit for operators and analysts who need structured data from public pages on a cadence; engineers who prefer preview-driven workflows over hand-maintained selectors; internal platform teams that want Datasets and APIs; individuals and small teams who build flows from the page they are already viewing.

Not a fit for large-scale anti-bot evasion, fully offline local browser farms, or any collection that violates site terms or law. Product and store copy both stress: only public information you are permitted to collect.

Compliance boundaries

Capability is not authorization. Confirm pages are publicly reachable, your use complies with site terms and applicable law, and your request rate is reasonable. Datasets may hold sensitive business fields—rotate API keys and treat account access seriously. Notify emails reach real inboxes; do not point test workflows at production alert lists.

Compliance language in docs, pricing, and the extension listing is intentional: public pages, reusable workflows, auditable credit spend—not a universal bypass toolkit.

Console editing and AI-assisted graphs

In /app/workflows you drag nodes, wire edges, preview scrape samples, and save the graph. When model keys are configured, toolbar AI / side-panel AI chat can help analyze pages or validate structure and stream canvas updates—you still confirm and save. AI accelerates; it does not replace compliance or field sign-off. Sample rows must be business-usable before you trust a schedule.

Non-engineers often start from the public home analyze demo or the extension, then finish store/notify in the editor. Engineers may create workflows via API or reuse node presets. Whatever the entrance, the saved Workflow and Dataset are authoritative—not a chat transcript.

Next steps

Build intuition quickly: try the public analyze demo on the home page, or read Getting started and the API quick start. The next post covers the Chrome extension path from the current tab to a reusable workflow; the third walks price monitoring through Datasets and the API.

HuluFlow’s goal is concrete: stop rewriting scrapers for every ask, and maintain workflow assets that are schedulable, metered, exportable, and notifiable. If that is where you are stuck, shipping the first Free-tier pipeline usually beats opening yet another crawler repository. Turning scraping from project work into owned assets is the organizational habit this product is built to unlock.