Real-time Data Pipeline Optimization for Enhanced Analytics Insights

As a rapidly expanding company in the Research & Analytics industry, we are confronted with the challenge of managing and deriving insights from increasing volumes of data in real-time. Our existing data pipeline struggles with latency and inefficiencies, which hinders our ability to provide timely analytics to our clients. We are looking for a skilled data engineer to overhaul our current system by implementing a real-time data processing architecture using Apache Kafka for event streaming and Spark for processing. The project will also involve integrating Airflow for workflow management and dbt for transforming data in our Snowflake and BigQuery environments. The primary goal is to establish a scalable data mesh architecture that ensures data observability and supports our transition towards MLOps practices. Deliverables include a comprehensive blueprint of the new data architecture, a fully operational data pipeline, and documentation for future scalability.