Real-Time Data Pipeline for Enhanced Passenger Experience in Airline Operations

Our startup, aiming to revolutionize passenger experience in the aviation industry, seeks a skilled data engineer to build a cutting-edge real-time data pipeline. The project involves integrating multiple data sources such as flight status, weather conditions, and passenger preferences using technologies like Apache Kafka for event streaming, Spark for real-time analytics, and Snowflake for data warehousing. The goal is to create a data mesh architecture that supports decentralized data ownership and access, improving operational efficiency and customer satisfaction. Additionally, the pipeline should incorporate MLOps principles to streamline machine learning model deployment for predictive insights into passenger needs. Key deliverables include data observability dashboards for monitoring data flow and quality, ensuring system reliability and accuracy. This project is crucial for providing timely insights to staff and passengers, ultimately enhancing the overall travel experience.