Real-time Data Pipeline Development for Enhanced Pet Health Insights

As a dynamic and rapidly growing veterinary startup, we are committed to leveraging technology to improve pet health. We seek a skilled data engineer to build a robust real-time data pipeline. This project will involve integrating disparate data sources, such as health records, IoT pet devices, and lab results, using Apache Kafka for event streaming. We require the implementation of a data mesh architecture to ensure that data can be accessed and utilized efficiently across our organization. The project will utilize tools like Spark for data processing, Airflow for workflow orchestration, and dbt for data transformation. Snowflake and BigQuery will serve as our data warehouses, while Databricks will be used for advanced analytics. Our goal is to enable real-time analytics and insights for veterinarians, which will facilitate prompt decision-making and improve patient outcomes. The project's success will mean enhanced data observability and operational efficiency, positioning us competitively in the veterinary services market.