Workflows and node configuration

What is a workflow, in plain language?

A workflow is a small assembly line for web data. You place boxes (nodes) on a canvas and connect them with arrows. When the line runs, each box does one job and hands its results to the next box.

Typical jobs: build many page URLs → read each page into rows → save rows into a table → email you when something is new or a price moved. HuluFlow hosts the line in the cloud, so it can run on a schedule while you are offline.

You are not writing a crawler. You configure what to read and where to put it. The product discovers columns from a sample page; you choose which columns to keep.

Workflow-level settings

These settings belong to the whole pipeline, not to a single node. You set them in the workflow header or via the API when creating/updating a workflow.

Runs need remaining credits (saving the canvas does not). HTTP 402 means the pool is exhausted — upgrade or wait for the next period.

Tip: name workflows by business purpose (“Competitor A prices”), not by technical nicknames. Future you will thank present you when the inbox fills up.

How the graph is stored

Visually you see boxes and arrows. Internally HuluFlow stores a graph: a list of nodes and a list of edges. Each node needs a unique id string (the editor creates these), a type, and a config object.

Edges use from and to (node ids). from is upstream. Never use source/target — the API will not understand them. The runner sorts nodes so upstream always finishes before downstream. Loops are rejected with an error.

{
  "nodes": [
    { "id": "s1", "type": "scrape", "config": { } },
    { "id": "st1", "type": "store", "config": { } }
  ],
  "edges": [
    { "from": "s1", "to": "st1" }
  ]
}

If you copy JSON from docs into the API, keep ids stable when you also use notify — notify compares to the previous output of the same node id.

How data moves along edges

Most nodes produce an object that includes items (rows) and often fields (column definitions). The next node receives that as input. If several arrows point into one node, the runner merges available upstream outputs according to the engine rules — in practice, keep graphs simple: one clear path.

URL generator ignores inputs; it only produces URLs. Store and notify look for data rows on their connected upstream (or the first rows available in the run). They do not re-scrape the web.

List→detail is special: the detail scrape reads URLs from each upstream row (via input_field or link/url/href) and merges parent list fields into each detail row so you keep title+price from the list and description from the detail page.

URL generator

Purpose: create many list-page URLs without typing them by hand. It never downloads pages and does not use credits. Think of it as a spreadsheet formula that fills page=1…N.

Output shape is always items: [{url}, {url}, …] with a simple url field. Connect the output only to a scrape node. Cap: 500 URLs per generator per configuration.

When to use it: You need page 1, page 2, page 3… of the same list pattern, or you have a fixed list of seed URLs to paste.

Example configuration

{
  "mode": "range",
  "template": "https://example.com/list?page={page}",
  "param": "page",
  "start": 1,
  "end": 10,
  "step": 1
}

{
  "mode": "list",
  "urls_text": "https://example.com/a\nhttps://example.com/b"
}

After configuring, use preview/validate in the UI to see how many URLs will be generated before you Apply. Then raise the downstream scrape limit if you generate more than 20 URLs.

Scrape

Purpose: open page(s) and extract structured rows. Each webpage request consumes 1 credit when the workflow runs. Without a working scrape, store and notify have nothing useful.

Two-phase mental model: (1) Discovery — you provide requirement in plain language; the engine proposes fields and a fetch_profile. (2) Extract — later runs use your selected fields (+ profile) to pull those columns repeatedly. Always Apply selected fields before you trust production runs.

list mode returns many items from a listing page. detail mode returns one richer row per URL and can merge fields from an upstream list row. Default limit is 20 URLs per run of this node — raise it when a generator feeds more pages.

When to use it: Any time you need data from a web page. Start with one scrape; add a second scrape only when you need detail pages.

Advanced keys you may see: scope_xpath, item_xpath (from discovery), url_from (force a URL list). Inputs should come from URL generator or scrape — not from store or notify.

Example configuration

{
  "url": "https://example.com/list",
  "mode": "list",
  "requirement": "title, price, url",
  "limit": 20,
  "fields": [
    { "name": "title", "label": "title", "type": "text" },
    { "name": "url", "label": "url", "type": "url" }
  ]
}

{
  "mode": "detail",
  "input_field": "url",
  "requirement": "description, sku",
  "limit": 20,
  "fields": [
    { "name": "description", "label": "description", "type": "text" }
  ]
}

Credits follow webpage requests. Test the list scrape alone first. If rows look wrong, fix discovery before adding store.

Store

Purpose: save upstream items into a dataset (table) you can browse, export, and query via API. Store does not use a quota slot. It is not offered as a reusable preset because it points at a specific table.

Each incoming row is upserted by hashing key_fields. Same key → update the existing row. New key → insert. That is how daily monitors refresh prices without duplicating products.

When to use it: Whenever you want durable history or exports — almost every serious workflow ends with store.

Example configuration

{
  "dataset_name": "products",
  "key_fields": ["url"],
  "store_fields": ["title", "price", "url"]
}

After the first successful store, open Datasets to confirm columns and a few rows. Export CSV/JSON from there when you need a spreadsheet.

Notify

Purpose: email you when something changed compared to the previous run of this same notify node. It does not crawl the web; it reads items from upstream (usually scrape or store).

First run often has nothing to compare to — you may get a full set of “new” rows or a baseline with no mail, depending on when. Run twice when testing alerts. Node test supports dry_run (no email sent).

When to use it: Monitoring jobs: new listings, price drops, or any watched field change. Skip notify for pure one-time collects.

Example configuration

{
  "email": "ops@example.com",
  "when": "field_change",
  "watch_fields": ["price"],
  "subject": "[HuluFlow] price change"
}

Wire notify after the scrape (or store) you care about. For price alerts, choose “When watched fields change”, set watch fields to price, and keep a stable URL key upstream.

How to wire nodes (patterns that work)

The editor validates many bad links (for example URL generator cannot feed store directly). Prefer these patterns:

Keep one main spine. Extra branches are possible but harder to debug as a beginner.

• URL generator → scrape (list) — generated URLs become the pages to open.
• Scrape (list) → scrape (detail) — set the link field to the list’s link/URL column; detail merges list + detail fields.
• Scrape → store — rows land in a dataset with upsert keys.
• Scrape or store → notify — notify reads upstream rows; choose “When watched fields change” when you care about value changes on known keys.

Story example: generate catalog pages 1–5 → list scrape (title, price, url) → detail scrape (description) → store keyed by url → notify on price field change. Credits bill by webpage requests.

Limits, quota, and exports

URL generator: max 500 URLs per node. Scrape: default 20 URLs per node per run (configurable via limit). Credits: webpage requests by scrape nodes this period — not “one seat per scrape node”. Out of credits → HTTP 402.

Datasets: console pagination and API page size limits apply when browsing; export supports CSV/JSON up to 100,000 rows (413 if larger). Failed crawls surface as node errors inside the run — open the run detail.