Real-time Data Pipeline Optimization for Enhanced Viewer Insights

As a rapidly growing streaming platform, understanding viewer behavior in real-time is critical to maintaining a competitive edge. This project involves optimizing our existing data pipeline to support real-time analytics and data observability. We aim to integrate advanced technologies such as Apache Kafka for efficient event streaming, and Spark for processing high volumes of data quickly. Your role will be to assess our current setup and identify bottlenecks, implementing solutions that enhance data throughput and reliability. An essential part of this project is ensuring data quality and observability, employing tools like Airflow for workflow orchestration, dbt for data transformation, and Snowflake or BigQuery for warehousing. Additionally, we are looking to implement a data mesh architecture to democratize data access across different teams, enhancing collaboration and innovation. The ideal candidate will have a deep understanding of MLOps practices to facilitate machine learning integration, providing our analytics team with the tools needed to derive actionable insights promptly.