Lead Data Engineer
Primary Skills
- Athena, Step Functions, Spark - Pyspark, ETL Fundamentals, SQL (Basic + Advanced), Glue, Python, Lambda, Data Warehousing, EBS /EFS, AWS EC2, Lake Formation, Aurora, S3, Modern Data Platform Fundamentals, PLSQL, Data Modelling Fundamentals, Cloud front
Specialization
- AWS Data EngineerIng Basic: Senior Data Engineer
Job requirements
Job Description – AWS Senior Data Engineer
Company: Brillio Technologies
Role: Senior Data Engineer
Experience: 5–10 Years
Location: India (Hybrid/Remote)
Employment Type: Full-time
Role Overview
We are looking for a Senior Data Engineer with strong Databricks and Big Data expertise to design, build, and optimize scalable data pipelines and platforms. The ideal candidate will have hands-on experience with Spark, PySpark, AWS ecosystem, and modern data architectures, along with exposure to APIs and real-time data processing.
Key Responsibilities
- Design and build scalable data pipelines using Spark / PySpark on Databricks
- Develop and optimize ETL/ELT workflows for large-scale data processing
- Work with AWS EMR, S3, and Hadoop ecosystem for distributed data processing
- Build and maintain data lake and data warehouse solutions
- Develop and integrate APIs for data ingestion and consumption
- Implement data processing workflows using Airflow / Autosys
- Optimize data performance using partitioning, caching, and query tuning
- Handle structured and semi-structured data from multiple sources
- Ensure data quality, governance, and reliability of pipelines
- Collaborate with cross-functional teams (Analytics, Product, Engineering)
Technical Skills (Must-Have)
- Strong expertise in:
- Apache Spark / PySpark
- Databricks (mandatory)
- SQL (PostgreSQL or similar)
- Hands-on experience with:
- AWS EMR, S3
- Hadoop, Hive ecosystem
- Programming skills:
- Python (must-have)
- Scala (good exposure)
- Strong knowledge of:
- UNIX / Shell scripting
- ETL pipelines and data engineering fundamentals
Additional Skills
- Experience with:
- Elasticsearch (data storage & retrieval)
- Workflow orchestration tools (Airflow, Autosys)
- Exposure to:
- API development & integration
- Version control tools (Git / SVN)
- Basic knowledge of HTML (for web/data tasks)
Experience Required
- Minimum 5+ years of Data Engineering experience
- Strong hands-on experience in Databricks + Big Data stack
- Proven experience working on large-scale distributed systems
Education
- Bachelor’s or Master’s degree in Computer Science, Engineering, or related field
Why Join Brillio
- Work on cloud-native, large-scale data platforms
- Exposure to cutting-edge Databricks & AWS ecosystems
- Collaborative and high-performance engineering culture