Real-Time Data Pipeline Development for Enhanced Customer Insights

We are a growing SME in the E-commerce & Online Retail sector looking to elevate our data management capabilities. Our current batch processing system limits our ability to derive real-time insights, leading to slower decision-making processes and missed opportunities for customer engagement. This project aims to develop a real-time data pipeline using technologies such as Apache Kafka for event streaming and Spark for real-time data processing. The solution will integrate with Snowflake or BigQuery for centralized data storage and use Airflow and dbt for orchestration and transformation. By implementing this pipeline, we expect to gain instant access to customer behavior insights, optimize inventory management, and personalize marketing strategies. The ideal candidate will have strong expertise in setting up data pipelines, knowledge of MLOps for deploying machine learning models, and experience in data observability to monitor data flow and quality.