This role is for one of Weekday’s clients
Min Experience: 3.5+ years
Location: India, Pune
JobType: full-time
We are looking for an experienced Senior Data Engineer with 3.5+ years of experience in designing, developing, and maintaining scalable data pipelines and robust data engineering solutions. The ideal candidate will have strong expertise in Databricks, Azure Data Services, and programming languages such as Python and Scala. You will be responsible for building efficient data processing workflows, optimizing data performance, and ensuring data quality, security, and reliability across the data platform.
The role requires hands-on experience with Lakehouse architecture, Delta Lake, Unity Catalog, and ETL/ELT processes. You should be comfortable working with large datasets, troubleshooting production issues, and implementing best practices for data governance, automation, and performance optimization.
Requirements
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Databricks, Apache Spark, Python, and/or Scala.
- Build and optimize data ingestion, transformation, and processing workflows using ETL/ELT methodologies.
- Implement and maintain Lakehouse architecture and Delta Lake solutions to support reliable and efficient data storage and processing.
- Work with Azure cloud services, particularly Azure Data Lake Storage Gen2 (ADLS Gen2), to manage and process large volumes of data.
- Optimize Spark jobs and data pipelines for performance, scalability, and cost efficiency.
- Implement data modeling practices to support analytical workloads, reporting, and downstream data consumption.
- Use Unity Catalog and related governance capabilities to manage data access, security, permissions, and compliance.
- Develop and maintain CI/CD pipelines to automate deployment, testing, and release processes for data engineering solutions.
- Monitor production data pipelines, troubleshoot failures, and resolve performance and data quality issues.
- Collaborate with data scientists, analysts, architects, and other engineering teams to deliver dependable data solutions.
- Follow best practices for data quality, documentation, version control, security, and production support.
Must-Have Skills
- Strong hands-on experience with Databricks and Azure data engineering services.
- Experience working with Azure Data Lake Storage Gen2 (ADLS Gen2).
- Proficiency in Python and/or Scala for data processing and pipeline development.
- Minimum 3.5 years of relevant experience in data engineering.
- Strong understanding of data pipeline development, data integration, and cloud-based data platforms.
- Experience with CI/CD workflows, data modeling, governance, and production support.
Good-to-Have Skills
- Hands-on experience with Apache Spark and distributed data processing.
- Knowledge of Delta Lake, Unity Catalog, and Lakehouse architecture.
- Experience with ETL/ELT development and data transformation frameworks.
- Expertise in Spark performance tuning, query optimization, and efficient resource utilization.
- Familiarity with data quality monitoring, pipeline orchestration, and automated testing.
Experience and Qualifications
- Experience: 3.5+ years in data engineering or a related field.
- Strong analytical and problem-solving skills, with the ability to troubleshoot complex data issues.
- Ability to work collaboratively in a fast-paced environment and take ownership of data engineering deliverables.
- A focus on building scalable, secure, maintainable, and production-ready data solutions.