The role of a Senior Data Engineer is to design, develop, test, and implement enterprise-wide software applications. Conduct code reviews and testing in ongoing sprints. Gather data from multiple sources, aggregating it in the right formats assuring that it adhere to data quality standards, and assuring that downstream users can get the data quickly. The role manages to develop, construct and maintain architectures such as databases and high scalable data processing systems. Develop processes for data modelling, mining and production data sets. Integrate new data management technologies and software engineering tools into existing structures. Build process to improve data reliability, efficiency and quality.
Critical Business Activities
As a Senior Data Engineer you (candidate) will be responsible for,
- Development and deployment of highly scalable OLAP systems for advanced analytics and data processing
- Building a scalable and robust data pipeline architecture
- Identifying, designing, and implementing internal process improvements, automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
- Working with stakeholders including the Executive, Product, Data and Design teams to assist with data requirements and infrastructure needs
- Keeping our data separated and secure across multiple data centers and AWS regions for business and operational compliance and continuity
- Optimizing or even re-designing the data architecture and design to support our next generation of products and data initiatives
- Integrate new data management technologies and software engineering tools into existing structures
- Delivering data with consistency by implementing automation testing and strong QA processes.
Competencies and Experience
Technical Skills and Experience
- E/B.Tech or MCA graduate with 5+ years of experience in Software Engineering/Data Warehousing
- Expertise in Python programming and Shell scripting
- Proficient with writing basic and advanced SQL queries
- Excellent analytical and problem solving skills
- Hands on experience on any of the ETL tools
- Expert understanding of data warehousing concepts, business intelligence and big data management
- Software development in a Linux/Unix environment
- Good understanding of operating systems and computer fundamentals
- Good understanding of NoSQL databases
- Knowledge on Hadoop, Hive, HBase, and Pig scripting is a plus
- Reliability: Gets around large obstacles on their own, finds solutions to most of the technical issues and should be very dependable.
- Execution: Ability to present great quality deliverables (designs, code, test cases, etc.) with hardly any iterations/guidance.
- Management: Breaks down complex tasks into a daily plan and execute it efficiently. Never lose track of action items and tasks. Should have excellent understanding of their own progress against their plans.
- Communication: Communicates complex concepts in terms understandable to other technical and business people.
- Leadership: Ability to lead peers in adopting a new process or innovation.
- Judgement: Makes expert technical decisions on design and coding tasks. Understands module level impacts of design and coding decisions.