CRM data pipeline for a German retail network

Our client operates a retail network in Germany with a B2B wholesale arm selling to trade customers. Customer and order data lived in several systems: an on-premise ERP, an e-commerce platform, point-of-sale exports and a Salesforce org used by the field sales team.

Salesforce was meant to be the single view of each trade customer. In practice it was fed by manual CSV imports, and the sales team had learned not to trust it.

Account records were duplicated across regions, contact data went stale within months, and consent flags were maintained by hand — a GDPR exposure as much as a sales problem. Pipeline reporting disagreed with the ERP, so forecasting meetings started with an argument about whose numbers were right. The client did not need a new CRM; it needed engineering discipline applied to the data feeding the one it had.

We treated the engagement as a data engineering project. After mapping every source system and field, we built an ingestion and transformation layer: Python services orchestrated with Apache Airflow, running on Azure Kubernetes Service in an EU region, with a PostgreSQL staging database as the intermediate store. The ERP is read through scheduled OData extracts; the e-commerce platform and POS exports arrive through event-based and file-based ingestion.

Deduplication runs deterministic matching on registry identifiers first, then fuzzy matching on name and address, with explicit survivorship rules deciding which source wins for each field — rules the client’s sales operations team reviewed and signed off. Enrichment validates VAT numbers and legal-entity data against public registers. Consent and marketing-permission flags are modelled as first-class fields with full history, so a withdrawal propagates to every connected system instead of surviving in a forgotten copy.

Cleaned, merged records sync to Salesforce through the Bulk API 2.0, and Change Data Capture events carry field-level updates from Salesforce back into the pipeline for bidirectional consistency. Data-quality checks run on every load with Great Expectations; a failed check blocks propagation and alerts our on-call team rather than silently writing bad records into the CRM.