Real-time Data Pipeline Optimization for Enhanced Production Insights

In the fast-moving world of electronics manufacturing, real-time data is crucial for maintaining competitive advantage and operational efficiency. Our growing company has recognized the need to upgrade our existing data infrastructure to better support real-time analytics and decision-making across the production floor and supply chain management. We are seeking a skilled data engineer to lead this transformation by optimizing our event streaming capabilities and integrating them with advanced analytics and machine learning operations (MLOps). The project will involve building a robust data mesh framework that ensures seamless data observability across all departments. Key tasks will include configuring Apache Kafka for efficient event streaming, utilizing Spark and Databricks for real-time data processing, and implementing Airflow and dbt for orchestrating data workflows. The improved infrastructure will be hosted on cloud platforms like Snowflake and BigQuery, enabling high scalability and performance. This initiative is intended to empower our team with actionable insights, thereby driving productivity and innovation.