Data Engineer

About the Role

This Data Engineer role at Capgemini Invent involves designing, developing, and maintaining scalable data pipelines using Snowflake. You will write efficient and maintainable code in Python and PySpark, and develop and optimize SQL queries for data extraction and transformation.

As a Data Engineer, you will implement data ingestion strategies in Snowflake using Snowpipe and COPY command to load data into base tables with Streams and Tasks. You will also extract data from various source systems and load data into Snowflake using Python, PySpark scripts.

Additionally, you will configure role-based access on Snowflake objects and process data sharing and cloning between Snowflake accounts. You will create models in DBT and integrate data quality rules and tests to ensure data accuracy, consistency, and completeness.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Snowflake.
• Write efficient and maintainable code in Python and PySpark.
• Develop and optimize SQL queries for data extraction and transformation.
• Implement data ingestion strategies in Snowflake using Snowpipe and COPY command.
• Extract data from various source systems and load data into Snowflake using Python, PySpark scripts.
• Configure role-based access on Snowflake objects.
• Process data sharing and cloning between Snowflake accounts.
• Create models in DBT and integrate data quality rules and tests.

Skills & Qualifications

• Proficiency in Python and PySpark.
• SQL skills, including query optimization and performance tuning.
• Familiarity with data warehousing concepts and ETL processes.
• Experience with big data tools: Hadoop, Spark, Kafka, etc.
• Experience with relational databases such as Microsoft SQL Server, MySQL, PostgreSQL, Oracle, and NoSQL databases such as Hadoop, Cassandra, MongoDB.
• Excellent problem-solving skills with an emphasis on sustainable and reusable development.
• Bachelor's degree in Computer Science or related field.

What You'll Learn

In this role, you will gain hands-on experience with designing, developing, and maintaining scalable data pipelines using Snowflake. You will learn how to write efficient and maintainable code in Python and PySpark, and develop and optimize SQL queries for data extraction and transformation.

You will also learn how to implement data ingestion strategies in Snowflake using Snowpipe and COPY command, and configure role-based access on Snowflake objects. Additionally, you will gain experience with creating models in DBT and integrating data quality rules and tests.

Resume Tip

When applying for this Data Engineer role, make sure to highlight your experience with data warehousing concepts and ETL processes. Emphasize your proficiency in Python and PySpark, and your ability to write efficient and maintainable code. Also, be sure to mention any experience you have with big data tools, relational databases, and NoSQL databases.

Additionally, highlight any relevant certifications or training you have received in data engineering, such as AWS Certified Data Engineer or Google Cloud Certified - Professional Data Engineer.