HuluFlow · Scenario
Generate pagination URLs and scrape in batch
Use the URL generator instead of hand-editing query strings.
Pagination means the site splits a catalog across page=1, page=2, …. The URL generator builds those addresses; a scrape node reads each page. This is easier than maintaining a handwritten list.
Beginners who already completed one single-URL scrape and now need more list pages.
-
01 Range vs list
Range mode fills {page} (or your param name) from start to end with a step. Use it when URLs are predictable. List mode pastes one full URL per line when the pattern is irregular.
Always preview the generated count. If the preview shows hundreds of URLs, confirm you really need them all in one run.
-
02 Connect to scrape
The generator has no inputs. Draw from its output port to a scrape node’s input. Put the scrape in list mode. The scrape reads the url field from each generated item.
You do not type the seed URL on the scrape when the generator is connected — the upstream list supplies URLs. You still must Fetch fields using a representative sample (often page 1) so columns exist.
-
03 Cap and limit
Two different caps apply. Generator: at most 500 URLs. Scrape: default 20 URLs processed per run of that node. If you generate 40 pages, raise scrape.limit to at least 40 or split into two generators/workflows.
Credits are spent on webpage requests, not on how many URLs you generate. Generate 40 pages but scrape 5 → 5 credits.
-
04 Then store
Add store with key url (or your product link) so reruns upsert. Optional notify afterward. Open Datasets after the first successful run.
If some pages fail, open the run’s scrape node output/error — one bad page can be a blocked URL or a layout change mid-catalog.
Credits and failures
If a run fails, open that run and read the failing node’s error and output — do not guess from the workflow list alone. URL generator caps at 500 URLs. Each scrape node defaults to 20 URLs per run (raise limit when you generate more pages). Credits are granted by plan and deducted by webpage requests. Out of credits → HTTP 402.
On failure, open that run’s node error — do not guess from the workflow list alone.
What next
After batch lists work, add detail scrapes or a price monitor.