Real-Time Data Pipeline for Carbon Credit Trading Optimization

As a scale-up in the Carbon Credits & Trading industry, our company is focused on advancing our data infrastructure to support real-time trading decisions. We are seeking an experienced data engineer to build a comprehensive data pipeline that integrates various data streams from market prices, regulatory updates, and environmental data sources. Utilizing technologies such as Apache Kafka for event streaming, Spark for data processing, and Snowflake for storage, the solution should deliver real-time analytics to our trading platform. Additionally, the implementation of a data mesh approach will enable decentralized data accessibility across teams, enhancing collaborative decision-making. The project requires expertise in MLOps to facilitate the integration of machine learning models for predictive analytics, as well as data observability to ensure reliability and accuracy. The objective is to enhance our trading strategies, maintain compliance with regulatory standards, and secure a competitive edge in the rapidly evolving carbon trading market.