Real-Time Data Pipeline Implementation for Non-Profit Impact Analytics

As a rapidly growing non-profit organization, we face the challenge of efficiently analyzing the impact of our diverse programs. We are seeking an experienced data engineer to build a robust real-time data pipeline. This pipeline will integrate various data sources, leveraging technologies such as Apache Kafka for event streaming and Spark for data processing. Our goal is to enable real-time analytics, providing our stakeholders with timely insights into program outcomes and donor engagement. The project involves setting up a data mesh architecture, ensuring data observability, and implementing MLOps practices for scalable machine learning operations. The successful candidate will design workflows using Airflow, establish transformation scripts with dbt, and configure data storage solutions using Snowflake or BigQuery. This project is critical to our mission of maximizing impact and transparency, as it will streamline our reporting processes and enhance our capabilities to attract and retain donors. Time is of the essence, as we aim to implement this solution within a 4-6 week timeframe.