Hadoop Data Engineer

Technology

Bengaluru

About the job

The purpose of this role is to maintain, improve, clean and manipulate data in the business’s operational and analytics databases. The Data Engineer works with the business’s software engineers, data analytics teams, data scientists and data warehouse engineers in order to understand and aid in the implementation of database requirements, analyse performance, and troubleshoot any existent issues.

Responsibilities:

• Design, develop, and maintain scalable data pipelines and systems for data processing.

• Utilize Hadoop and related technologies to manage large-scale data processing.

• Perform data ingestion using Kafka & spark, Sqoop and various file formats and process data into Hive using Beeline/spark.

• Develop and maintain shell scripts for automation of data processing tasks.

• Implement full and incremental data loading strategies to ensure data consistency and availability.

• Orchestrate and monitor workflows using Apache Airflow.

• Ensure code quality and version control using Git.

• Troubleshoot and resolve data-related issues in a timely manner.

• Stay up-to-date with the latest industry trends and technologies to continuously improve our data infrastructure.

Requirements:

• Proven experience as a Data Engineer (ETL, data warehousing).

• Strong knowledge of Hadoop and its ecosystem (HDFS, YARN, MapReduce, Tez and spark).

• Proficiency in Kafka & spark, Sqoop and Hive.

• Experience with shell scripting for automation.

• Expertise in full and incremental data loading techniques.

• Excellent problem-solving skills and attention to detail.

• Ability to work collaboratively in a team environment and communicate effectively with stakeholders.

Good to have:

• Understanding of PySpark and its application in real-time data processing.

• Hands-on experience with Apache Airflow for workflow orchestration.

• Proficiency with Git for version control

• Experience on Postgres SQL or SQL server or MSBI

Beware of Job Scams

We are aware of several scams targeting job seekers and candidates. Please be vigilant. All communication throughout the recruitment process will be from an official member of the dentsu recruitment team, using corporate email addresses (e.g., @dentsu.com or @merkle.com). We will never ask you to send money or vouchers to secure employment. If you suspect you have been a victim of a scam, please report the incident to your bank, local police, or fraud protection authority immediately. Additionally, you can report the scam to us at jobfraud@dentsu.com so we can take appropriate action to request the website is taken down. Please note that Merkle and dentsu are not responsible for any losses incurred as a result of these scams. We advise all individuals to exercise caution and verify the authenticity of any job offers or communications received.