Consultation
We respond within 15 minutes
A commercial tool for a wholesale auto parts distributor: automatically cross-references prices from an internal catalog against live listings on a major B2B platform and generates clear reports for sales and procurement teams.
The goal is not a one-off scrape, but a robust, industrial-scale run across a large catalog — thousands of SKUs, unstable site rendering, anti-bot measures, proxy limits, and the ability to retry only failed rows without losing already-collected data.
What the system does
Loads the catalog from JSON (part number, manufacturer, name, own price). For each item, opens the marketplace search page via Playwright (Chromium), waits for the offers table to render, and parses the HTML with BeautifulSoup. Matches products by part number and manufacturer, distinguishes OEM from aftermarket, finds the minimum competitor price, and compares it against the client’s price (LOWER / EQUAL / HIGHER). Splits results into two streams: successful comparisons and problem cases (no results, out of stock, failed to retrieve data). Runs in batches with –offset / –limit, saves progress after each item, and supports re-scanning only failed rows in a separate batch. Generates summary reports: Excel and HTML, a short plain-text summary, and a breakdown by result category.
Technical decisions
| Area | Solution |
|---|---|
| Bypassing blocks | Residential/datacenter proxy pool with upfront validation of live addresses |
| Reliability | Up to 5 retries per item, deferred retry queue, configurable timeouts via ENV |
| UI resilience | Waits for the price grid to appear; handles both fast and slow load scenarios |
| Scale | ~10000 SKUs in catalog; runs of 500–1,000 SKUs; batch merging with recency priority |
| Output | openpyxl — color-coded comparison highlighting; HTML — same dataset for browser viewing |
Stack: Python 3 · Playwright · BeautifulSoup4 · openpyxl · JSON · batch scripts for operator workflows
Consultation
We respond within 15 minutes
