Real-time Data Infrastructure Optimization for Personalized Learning Insights

As a rapidly growing educational technology company, we aim to provide personalized learning experiences by harnessing the power of real-time data analytics. Our current data infrastructure limits our ability to deliver instant insights, which are critical for tailoring learning paths for students. We are seeking an expert data engineer to lead a project that will transform our data architecture into a real-time, scalable system. The project involves designing and implementing a data mesh architecture using Apache Kafka for event streaming, Spark for processing, and Snowflake for storage. Additionally, we aim to integrate MLOps practices to streamline our machine learning model deployment and monitoring processes. Data observability will be crucial to ensure high data quality and system reliability. The successful execution of this project will empower educators with timely and actionable insights, enabling them to adapt educational content dynamically, thus enhancing student engagement and outcomes.