Real-Time Data Pipeline for Enhanced Customer Insights

As a growing startup in the Data Analytics & Science industry, we are faced with the challenge of keeping up with the increasing demand for real-time customer insights. We are seeking a data engineering expert to develop a robust and scalable real-time data pipeline. This pipeline will integrate with our existing data infrastructure, aggregate customer data from multiple sources, and provide actionable insights using Apache Kafka, Spark, and Airflow. Our goal is to transform raw data into a valuable asset that aids in immediate decision-making processes. The project involves setting up event streaming with Kafka, batch processing with Spark, orchestrating workflows with Airflow, and ensuring data quality and transformation with dbt. Integration with Snowflake or BigQuery for efficient data storage is essential. The freelancer will work closely with our in-house analytics team to ensure that the data pipeline supports our business objectives effectively. Successful completion of this project will result in improved customer segmentation, personalized marketing strategies, and increased customer satisfaction. The freelancer should have expertise in real-time analytics, data observability, and a strong understanding of MLOps principles to ensure continuous integration and deployment of analytics models.