Forecast — data provenance and method

What this is

Longitudinal price, stock and assortment measurement for direct-to-consumer

brand storefronts (Shopify, WooCommerce, Magento and other open-feed

platforms). It is a time series, not a product catalogue.

What it is not

Marketplaces are never a source. Amazon, Walmart and other marketplaces and

aggregators are excluded at both probe and selection time, so nothing here

overlaps with marketplace price trackers. There is also no attempt to compete

with shopping indexes on current-state breadth.

How it is collected

• Structured feeds only — JSON-LD, Open Graph, microdata and platform product

APIs. No free-text scraping of marketing copy.

• Every outbound fetch passes a robots.txt check first, honouring `*` and `$`

wildcards with Google precedence. Fetching fails closed on an explicit

Disallow and open only when robots.txt itself cannot be retrieved.

• Explicit blocks are honoured. A 403 is final: no user-agent re-spoofing, no

referer spoofing, no TLS impersonation to defeat a block.

• An identifying user agent is sent, with per-domain rate limiting.

What is stored

Structured facts: name, SKU, URL, brand, category, material, colour, size,

price, stock, rating, review count, variants, images. No personal data of any

kind is collected, stored or served.

Honesty constraints applied to every published figure

• Prices are grouped per currency and NEVER synthetically converted. Roughly

100 currencies appear with partial FX coverage, and a converted average is a

fabricated number.

• Discount statistics count only listings that actually carry a discount.
• Groups below a minimum size are omitted rather than published thin.
• Stock figures (current state) and flow figures (counted within a period) are

labelled separately, because publishing one under the other's heading makes

the number right and the sentence wrong.

• A published period is frozen. A figure that changes between two readings is

not citable.

• Units are stored only when the source states them. NULL means "not

published"; zero is a claim. Out-of-range values are dropped, never clamped.

Known limitations, stated rather than papered over

• Our own first crawl of a catalogue cannot be distinguished from a merchant

adding products, so new-listing velocity counts a storefront only after a

settling period and excludes storefronts whose catalogue grew implausibly

fast within the period.

• Fact rows are not removed when a product stops being seen, so

assortment-wide aggregates can include since-delisted listings.

• Category assignment is measured, not assumed, and its precision is published

alongside it. Categories measured below the precision floor ship as

unassigned rather than as a probably-wrong label.

---

This statement is served by the same process that answers the tool calls, and get_methodology returns the identical text — so the document and the data cannot drift apart.