The Challenge
What PriceZ Was Facing
PriceZ powers a price intelligence product for US retailers. Their original pull-based scraping pipeline could not scale, frequently missed updates from major sources, and had no observability into pipeline failures. They needed a reliable ingestion architecture that could pull from 200+ retail sources — each with different schemas, rate limits, and reliability profiles — and normalize them into a single queryable product catalog.
The Solution
What We Built
We replaced polling with an event-driven ingestion system built on Apache Kafka. Each retailer source published price-change events to a dedicated topic, consumed by a normalization worker that resolved product identity via barcode and fuzzy-match algorithms. The serving layer used Redis for hot-path caching and PostgreSQL with a read replica for analytical queries. Prometheus and Grafana gave the team full per-source visibility into ingestion lag, error rates, and data freshness SLAs.
Results