06/10/2021
Position: Data Engineer
Experience: 3 – 5 Yrs
Location: Flipkart, Bangalore (Client Deputation from Algonomy to Flipkart)
JOB DESCRIPTION
Specific responsibilities include:
• You should have good hands-on experience in designing, implementing, and operating stable, scalable, solutions to flow data from production systems into analytical data platform (big data tech stack + MPP) and into end-user facing applications for both real-time and batch use cases
• You should be able to work with business customers in a fast paced environment understanding the business requirements and implementing analytical solutions.
• You should have good experience in the design, creation, management, and business use of large datasets
• Do high level design with guidance; Functional modelling, break-down of a module
• Thinking platforms & reuse
• Build and execute data modeling projects across multiple tech stacks i.e. big data, MPP, OLAP using agile development techniques
• Challenge status quo and propose innovative ways to process, model, consume data when it comes to tech stack choices or design principles
• Build and integrate robust data processing pipelines for enterprise-level business analytics.
• Strong engineering mindset - build automated monitoring, alerting, self-healing (restart ability/graceful failures) features while building the consumption pipelines
• Translate business requirements into technical specification (fact/ dimension/ filters/ derivations/ aggregations)
• As needed, assist other staff with reporting, debugging data accuracy issues and other related functions.
• An ideal candidate will have excellent communication skills to be able to work with engineering, product and business owners to develop and define key business questions and to build data sets that answer those questions
• Above all, you should bring your passion for working with huge data sets and bringing datasets together to answer business questions and drive change
Desired Competencies and Skill sets include:
• 3-5 years’ experience with Bachelor's Degree in Computer Science, Engineering, Technology or related field required
• 2 to 3 years of relevant software development experience with sound skills in database modeling (relational, multi-dimensional) & optimization and data architecture – databases e.g. Vertica
• Good understanding of streaming technologies like Kafka, Spark Streaming.
• Experience with Enterprise Business Intelligence Platform/Data platform sizing, tuning, optimization and system landscape integration in large-scale, enterprise deployments.
• Proficiency in one of the programming language preferably Java, Scala or Python
• Good knowledge of Agile, SDLC/CICD practices and tools
• Must have proven experience with Hadoop, Mapreduce, Hive, Spark, Scala programming.
• Must have in-depth knowledge of performance tuning/optimizing data processing jobs, debugging time consuming jobs.
• Proven experience in development of conceptual, logical, and physical data models for Hadoop, relational, EDW (enterprise data warehouse) and OLAP database solutions.
• Good understanding of distributed systems
• Experience working extensively in multi-petabyte DW environment
• Experience in engineering large-scale systems in a product environment
Please email your profile at [email protected]