Job Description
Responsibilities:
- Design, develop, and deploy scalable and reliable data pipelines to ingest, process, and analyze large volumes of structured and unstructured data.
- Collaborate with cross-functional teams to understand data requirements and design efficient models and schemas.3. Implement data processing workflows using technologies like Informatica BDM, Apache Spark, Apache Flink, or similar distributed computing frameworks.
- Optimize and tune data pipelines for performance, reliability, and scalability.
- Ensure data quality and integrity throughout the data pipeline by implementing data validation and monitoring processes.
- Manage and maintain data infrastructure, including data warehouses, data lakes, and ETL tools.
- Work closely with data scientists and analysts to provide them with clean, reliable, and accessible data for analysis and reporting.
- Stay up-to-date with emerging technologies and trends in data engineering and recommend new tools and techniques to improve efficiency and productivity.
Requirements:
- Bachelor's degree in Computer Science, Engineering, or a related field.
- 2-3 years of experience in data engineering or a similar role.
- Proficiency in programming languages such as Python, Java, or Scala.
- Hands-on experience with big data technologies such as Hadoop, Spark, Kafka, etc.
- Experience working with cloud platforms such as AWS, Azure, or GCP.
- Strong SQL skills and experience with relational and NoSQL databases.
- Solid understanding of data warehousing concepts and best practices.
- Hands on experience in Informatica BDM, SSIS is must
- Excellent problem-solving and analytical skills.
- Strong communication and collaboration skills, with the ability to work effectively in a team environment.
Preferred Qualifications:
- Experience with containerization and orchestration tools such as Docker and Kubernetes.
- Familiarity with machine learning frameworks and libraries.
- Experience with data visualization tools like Tableau, Power BI, etc.
- Certifications in relevant technologies (e.g., AWS Certified Big Data - Specialty, Google Professional Data Engineer, etc.).