Price monitoring: schedules, field-change email, datasets, and API

Price monitoring needs a loop, not a one-off spreadsheet

“Watch competitor prices” usually means four jobs: maintain product URLs, fetch list and detail pages on a cadence, store price fields as a queryable table, and notify humans (plus let internal systems pull via API). A single Excel export fails next week’s review and fails overnight patrols.

HuluFlow folds those jobs into one workflow asset. See the price monitor guide and the price monitoring use case. This post wires schedules, notify, Datasets, and API into an operational path, including credits and compliance.

If you also run lead lists or content intel, the node combo is similar—only watched fields and notify conditions change. Get price monitoring working, then clone the workflow and retune fields. That is usually cheaper than three unrelated scripts. Productized compounding comes from reusable graphs.

Recommended graph: from URLs to an alertable table

A classic monitor pipeline:

url_gen → list scrape → detail scrape → store → notify

url_gen

Paste a fixed SKU/URL list, or use a range template for paginated catalogs—always start small under the URL cap. Pagination detail: pagination guide.

list scrape and detail scrape

Lists discover cards: title, list price, detail link. Details refine the price you actually care about, stock copy, and specs. Whether you need both stages depends on whether the list already exposes every watched field. Many sites disagree between list and detail prices—name the field after the business meaning.

store

Pick a Dataset and key_fields (usually url or link). Upserts refresh the same product across runs instead of duplicating rows. Unstable keys break field_change notify.

notify

Prefer when=field_change on price columns; add or combine when=new if new listings matter. Use inboxes that will actually act on alerts. Before enabling the schedule, trigger one manual run so you can read the mail template and click through—then turn on automation.

Schedules and credit budgets

Set interval_minutes and next run time. The worker executes due jobs; console or API run covers ad-hoc refreshes. Match frequency to the business: daily competitor scans and hourly flash-sale watches differ by an order of magnitude in credit burn.

Credits meter successful scrape page requests. List-to-detail fan-out multiplies cost. Before go-live, estimate list pages + detail pages per run × runs per day against your plan. Free validates the graph; production monitors need enough quota so midnight runs do not leave holes.

Cloud schedules and extension manual runs can coexist: validate post-redesign fields in the extension by day; keep the night schedule. Datasets and run history remain authoritative.

Field-change notify: humans handle exceptions

Notify exists so people do not open the sheet every morning. Mail when watched fields change versus the last stored row. For changes to mean something, keep keys stable, field semantics stable, and noisy columns (second-level timestamps) out of the watch set.

Treat alerts as ticket entries: open the Dataset, spot-check the live page if needed, then adjust price strategy or dismiss. Do not invent another chat thread for screenshots.

Datasets: browse, export, system of record for the snapshot

Use the console Dataset view to spot-check upserts. CSV/JSON export serves teammates who will not call the API. For engineering, treat the Dataset as the current snapshot table: page via API into your warehouse or DB if you need longer history yourselves.

Datasets are not infinite blob storage; they hold structured workflow output. When scrape fields change, update downstream consumers so reports do not silently read empty columns.

Connect results to internal systems via API

Create an API key in the console; send Authorization: Bearer hulu_…. Typical calls:

• POST /api/v1/workflows/{id}/run — force a refresh before a launch.
• GET /api/v1/workflows/{id}/runs — inspect recent status.
• GET /api/v1/datasets/{id}/rows and …/export — pull or export rows.

See the API-first guide and API reference. Split keys by environment; rotate on leak; never commit keys to public front-end repos.

A clean split: HuluFlow turns public pages into tables plus exception email on a schedule; your warehouse owns long history and heavy analytics. Connect them with Dataset API or scheduled export.

Post-launch ops checklist

• Weekly spot-check product pages for field drift.
• Watch failing runs; pause schedules on repeated failure to stop burning credits.
• Review notify noise—too frequent usually means bad watch fields or inherently volatile pages.
• Reconcile plan quota vs monthly burn; upgrade before incidents.
• Compliance: only public pages you may collect; keep rates reasonable; respect terms and law.

Put a light “monitor health” review on the team calendar: ten minutes on failure rate, alert volume, and credit burn. Unattended schedules often fail silently for two weeks until someone asks why nobody saw a competitor drop. Workflow assets need owners the way services need on-call—lightweight, but named.

Common mistakes and fixes

Mixing list price and detail price in one watched field. Pick a business definition; split columns if needed and watch only one.

Exploding url_gen to thousands of links “to see if credits last.” Calibrate on 5–20 URLs first, then scale.

Routing notify to a personal inbox with no weekend coverage. Use a shared alias or duty list and agree that off-hours may wait.

Treating CSV mail attachments as the archive of record. CSV is for sharing; the system of record should be Dataset + API or your warehouse sync.

Ignoring compliance and rate. Monitoring is not a load test. Set intervals to what the business truly needs—cheaper and more sustainable.

After fixing these, you do not get a flashier crawler—you get an explainable supply chain: public URLs in, tables/mail/API out, every step visible in the console.

Next steps

Build the minimal graph from the price monitor guide, prove store plus one test notify with a manual run, then enable the schedule. Use the extension when you need page-context cold start. For positioning versus DIY scripts, see the first Blog post in this series.

Once price monitoring is a workflow asset, the team debates thresholds, fields, and frequency—not whose laptop holds the crawler. That is the operating model HuluFlow aims for: public pages into Datasets, changes into inboxes, snapshots into APIs—one workflow, three delivery lines. The first morning a field-change email beats accidentally opening a competitor page, you will feel why the loop deserves to be productized.