AI Data Engineer

Diverse Lynx is recruiting for a confidential employer seeking an AI Data Engineer. The role designs and maintains scalable data pipelines, prepares data for AI/ML and generative AI applications, and supports production deployments and data operations.

Responsibilities

• Design, develop and maintain scalable end-to-end ETL/ELT and data pipelines
• Build data processing and transformation solutions using Python, SQL and PySpark
• Develop batch and streaming pipelines to integrate data from databases, APIs, files and enterprise source systems
• Implement data-quality checks, validation, reconciliation, deduplication and exception handling
• Eligible to workimize data pipelines, SQL queries and Spark workloads for high-volume processing
• Prepare and transform datasets required for AI/ML and Generative AI applications
• Develop data ingestion and preparation pipelines supporting LLM/RAG solutions and vector databases
• Collaborate with AI/ML engineers, application teams and business stakeholders on downstream data requirements
• Implement automated deployment and monitoring through CI/CD and DevOps practices
• Support production deployments, monitoring, troubleshooting and root-cause analysis for data-related issues

Company

• Enterprise staffing and contingent workforce solutions
• Princeton, New Jersey, USA
• http://www.diverselynx.com/
• Founded in 2002
• 1001-5000 employees

Qualifications

• Strong hands-on experience in Python, SQL and data engineering
• Experience designing and developing ETL/ELT pipelines for large-scale data processing
• Strong knowledge of PySpark / Spark for data transformation and processing
• Experience with AWS data services such as Glue, S3, Redshift and related cloud services
• Hands-on experience with Apache Airflow or equivalent orchestration tools
• Strong understanding of data transformation, cleansing, validation, reconciliation and data-quality frameworks
• Experience working with structured, semi-structured and unstructured data
• Knowledge of batch and streaming data pipelines, preferably Kafka
• Experience building data pipelines supporting AI/ML, GenAI, LLM and RAG use cases
• Experience with CI/CD, Git, Docker and Kubernetes/EKS for production deployments
• Good understanding of data modeling, performance Eligible to workimization and production support
• Diverselynx IT Consulting Services.
• Princeton, New Jersey, USA
• http://www.diverselynx.com
• Late Stage
• Leadership Team
• Sujata Varma
• CEO
• Recent News
• LinkedIn
• Diverse Lynx on LinkedIn: #wbenc #diverselynx #wbenc #amplify #wbencnetwork #wbencconf…
• 2024-04-07
• Company data provided by
• AI Tools
• Customize Your Resume
• Maximize your interview chances
• Build Cover Letter
• Make your application stand out
• Analyze How Well You Fit
• Understand your strength & weakness

Required skills

• AI/ML Data Pipelines, AWS Glue, Amazon Redshift, Amazon S3, Apache Airflow, CI/CD, Data Engineering, Docker, ETL/ELT Pipelines, Kafka, Kubernetes, PySpark, Python, SQL

Experience

• Mid level