How a Nine-Market Retailer Cut Residential Spend on Price Crawls
A nine-market retailer rebuilt its price crawler with exit tiering and sticky checkout sessions, cutting residential traffic spend while keeping comparable storefront measurements.
A mid-size European retailer came in with nine storefronts, 180,000 SKUs, and a residential bill that had doubled in two quarters. Their price crawls still missed the hour a competitor actually moved, and checkout probes dropped the cart whenever the exit address changed. This note is the rebuild: they stopped treating every URL as a stealth problem.
Why the price crawls got expensive
Three jobs were sharing one residential pool.
Catalogue scans for public list prices. Logged-out storefront checks that only needed local tax and currency. Checkout probes that had to keep the same exit from the product page through payment.
Residential exits belong on the third job. They are a costly default for the first two. The team rotated every request, including the ones that needed affinity, so carts vanished and residential traffic sat at 78% of the mix.
What changed in the first sprint
Work was split by what the page requires, not by a single stealth preset.
Datacenter for the catalogue. Open listing pages without member prices moved to datacenter exits, capped per hostname, with a retry onto ISP only after a 403 or a challenge. That volume left the residential pool.
ISP for the storefront measurement. The comparable series — same country, same device class, same logged-out state — now leaves through ISP addresses pinned to the storefront city. The record kept exit city, currency, tax flag and promotion label. Mixing a Munich desktop reading into a Berlin mobile series had produced several false alerts.
Residential sticky sessions for checkout. Cart and payment paths pin one residential exit for the length of the probe, then drop the cookie jar. Reusing that jar on a later visit from a different IP is how two marketplaces had banned them.
The split matches residential versus datacenter proxies and the price monitoring setup. The pipeline for comparable prices covers observation fields; this case is about the invoice.
Numbers after six weeks
| Metric | Before | After |
|---|---|---|
| Residential share of requests | 78% | 22% |
| Residential gigabytes per week | 410 | 96 |
| Checkout probes that lost the cart | 18% | 3% |
| False price alerts per week | 40–50 | 8–12 |
| Cost per comparable observation | 100% baseline | 39% |
The catalogue did not shrink. Recrawl cadence on the hot set stayed at two to four hours. What changed was which exit paid for which page.
Two side effects mattered more than the invoice. The pricing desk started trusting the German and French series again, because the observation point was stable. Marketplace blocks on checkout stopped arriving in clusters; they became isolated 403s that a retry on a fresh sticky session usually cleared.
What we refused to do
The retailer asked twice for a full residential crawl "just to be safe". We said no. Safety on a listing page is a well-formed request and a backoff on 429, not a household IP. Buying more residential traffic would have hidden the session bug for another quarter.
We also refused a cookie jar that lived across days. Member prices are a different product: they need an account, a consistent fingerprint, and a legal basis. Public list prices do not.
How to copy the pattern
If your own price crawls look like this one:
- Label every URL with the job it serves: catalogue, storefront measurement, or checkout probe.
- Pin the observation point before you pick an exit. Country, device class and login state belong in the record.
- Use a sticky residential session only for the path that loses state when the IP changes.
- Recrawl the hot set on a short interval; do not recrawl the long tail at the same cost.
- Alert on deltas at a fixed observation point, not on mixed-origin levels.
The team still runs nine markets. The residential pool is now a scarce tool for pages that actually need a household exit, which is the only way that pool stays cheap as the catalogue grows.
FAQ
- Did they stop using residential proxies?
- No. Residential exits stayed on checkout probes and a few challenge pages. Catalogue scans and most storefront measurements moved off that pool.
- How long did a sticky session last?
- One checkout probe. The cookie jar was dropped afterwards so the next visit could not reuse a fingerprint tied to a dead IP.
About the author
Head of Data Engineering, SableProxy
Daniel ran price intelligence for a retail group across nine markets before joining SableProxy. He now spends most of his time helping teams re-architect crawlers whose costs grew faster than their data volume — usually by tiering exits and classifying responses instead of buying more residential traffic.
