Data Engineer - DaAI

About the Role

As a Data Engineer at Infosys, you will be responsible for designing, developing, and maintaining large-scale data pipelines using various technologies such as PySpark, Apache Spark, and Airflow. You will work with structured data from various sources, including databases, APIs, and enterprise applications, and ensure data quality and integrity throughout the pipeline.

You will also be working with cloud data platforms such as Databricks, Snowflake, and BigQuery, and have exposure to data modelling, schema design, joins, keys, relationships, data validation, and data quality concepts.

Key Responsibilities

• Design and develop data pipelines using PySpark, Apache Spark, and Airflow to extract, transform, and load data from various sources.
• Work with structured data from databases, APIs, and enterprise applications to ensure data quality and integrity.
• Develop ETL/ELT workflows using Python, SQL, PySpark, Apache Spark, Airflow, dbt, Dagster, and cloud-native services.
• Implement data quality checks using frameworks such as Great Expectations, dbt tests, AWS Glue DataBrew, and custom validation scripts.
• Support data labelling, contextualization, harmonization, enrichment, and classification workflows required for AI agent configuration.
• Prepare data outputs for downstream AI consumption, including embeddings, metadata, semantic tags, graph-ready datasets, and retrieval-ready document chunks.
• Collaborate with cross-functional teams to ensure data pipelines meet business requirements and are scalable.

Skills & Qualifications

• Strong understanding of data modelling, schema design, joins, keys, relationships, data validation, and data quality concepts.
• Experience with PySpark, Apache Spark, and Airflow for data pipeline development.
• Knowledge of cloud data platforms such as Databricks, Snowflake, and BigQuery.
• Understanding of data profiling, cleansing, transformation, and reconciliation.
• Practical experience with Git, CI/CD basics, unit testing, and production-grade engineering practices.
• Bachelor's degree in Computer Science, Information Technology, or related field.
• Strong problem-solving skills and attention to detail.

What You'll Learn

As a Data Engineer at Infosys, you will have the opportunity to learn and work with cutting-edge technologies such as PySpark, Apache Spark, and Airflow. You will also gain experience working with cloud data platforms and developing ETL/ELT workflows. Additionally, you will learn about data quality checks and data labelling, contextualization, harmonization, enrichment, and classification workflows required for AI agent configuration.

Resume Tip

When applying for this role, make sure to highlight your experience with data pipeline development using PySpark, Apache Spark, and Airflow. Also, be sure to mention any experience you have with cloud data platforms and data quality checks. Additionally, include any relevant certifications or training programs you have completed in data engineering or related fields.