Real-Time Energy Consumption Data Infrastructure for Optimized Renewable Resource Allocation

As a rapidly growing company in the Renewable Energy industry, we face the challenge of efficiently managing and distributing energy derived from renewable resources. Our goal is to establish a real-time data infrastructure that allows us to monitor energy consumption patterns dynamically. We seek an experienced Data Engineer to design and implement this system using cutting-edge technologies such as Apache Kafka for event streaming, Spark for large-scale data processing, and Airflow for orchestrating data workflows. Our current architecture needs enhancement to accommodate the increasing volume and variety of data, necessitating a shift towards a more scalable and flexible data mesh architecture. Additionally, we aim to incorporate MLOps for continuous deployment and monitoring of machine learning models that predict consumption trends, alongside ensuring data observability using tools like dbt and Snowflake or BigQuery for robust analytics and reporting. This project will not only streamline our operational processes but also empower us to make data-driven decisions that maximize the efficiency and sustainability of our energy distribution.