Real-time Data Pipeline Optimization for Quality Control in Steel Production

In the competitive landscape of steel production, ensuring product quality and efficient operations are paramount. Our scale-up company, operating within the Steel & Metals industry, is experiencing rapid growth and seeks to enhance its data infrastructure for real-time quality control and production monitoring. The project entails designing and implementing an optimized data pipeline that will facilitate real-time data collection and processing from various manufacturing equipment and sensors. By integrating technologies like Apache Kafka for event streaming and Apache Spark for large-scale data processing, we aim to create a robust solution that supports real-time analytics and decision-making. The data pipeline will also incorporate Airflow for orchestrating workflows and dbt for transformations. Data will be stored in Snowflake, enabling scalable and performant data queries, with potential integrations into BigQuery for advanced analytical capabilities. The project will empower our engineering and quality assurance teams to detect anomalies promptly and maintain product standards, driving competitive advantage and operational efficiency.