Real-time Data Pipeline for Enhanced Animal Welfare Monitoring

As a leading scale-up in the animal welfare sector, we are committed to ensuring the highest standards of care for animals under our supervision. To achieve this, we need to harness the power of data. Our current infrastructure consists of siloed data systems that limit our ability to perform timely analysis and react swiftly to animal welfare needs. We aim to implement a comprehensive real-time data pipeline that will aggregate data across various sources, including IoT sensors, veterinary records, and shelter management systems. This pipeline will leverage technologies such as Apache Kafka for event streaming, Spark for processing large datasets, and Airflow for orchestrating data workflows. Our goal is to create a seamless flow of information into a centralized data warehouse (Snowflake or BigQuery), where analysts and decision-makers can visualize and act upon data insights in real-time. By integrating machine learning operations (MLOps) and data observability practices, we aim to ensure the reliability and accuracy of our analytical models, ultimately leading to improved animal welfare outcomes. We invite experienced data engineers to collaborate with us on this pivotal project, contributing to a meaningful cause while working with cutting-edge technology.