Real-Time Data Pipeline Optimization for Enhanced Telemedicine Analytics

In the competitive landscape of telemedicine, our company is experiencing exponential growth, and there's a compelling need to leverage real-time analytics for improved decision-making and service delivery. We are seeking a seasoned data engineer to revamp our existing data infrastructure. The primary goal is to build a robust data mesh that supports real-time event streaming and analytics. Key tasks include designing and implementing a scalable and efficient data pipeline using Apache Kafka for real-time data ingestion and transformation, Spark for data processing, and Airflow for orchestrating workflows. The project will also involve leveraging Snowflake or BigQuery for cloud data warehousing, ensuring seamless integration with our existing analytics tools, and utilizing dbt for data transformations. This initiative is critical to improve our data architecture, enhance data quality, and support our expansion into new markets. By optimizing our data infrastructure, we aim to provide better insights into patient care, operational efficiencies, and ultimately, a superior user experience. The project is expected to last between 8-12 weeks and requires a high urgency level due to increasing market demands and competitive pressures.