Data Engineer Consultant

Indicium AI is hiring a Data Engineer Consultant to build and deploy production-grade data pipelines for enterprise clients. The role focuses on modernizing legacy data systems into cloud-native Lakehouses and delivering reliable, AI-ready data solutions in collaboration with client teams and engineering squads.

Responsibilities

• Build Lakehouse Modernization Pipelines: Convert legacy data setups (Informatica, PL/SQL, legacy data warehouses) into high-performance PySpark, Databricks SQL, and Delta Live Tables (DLT) using Medallion Architecture standards (Bronze, Silver, Gold)
• Data Modeling & Transformation: Write clean, modular, and tested dbt models and advanced SQL to transform raw client data into trusted, business-ready datasets
• Automate Infrastructure (DataOps): Use Terraform to provision cloud resources (AWS/GCP) programmatically and enforce data governance using Unity Catalog
• Integrate & Orchestrate: Build and monitor automated ELT/ETL workflows using tools like Apache Airflow and Databricks Workflows, ensuring data quality, lineage, and strict SLA compliance
• Client Engagement & Collaboration: Embedded directly within client projects, participating in daily standups, code reviews, and working closely with our nearshore delivery squads to deliver on time
• Troubleshoot & Optimize: Debug failing pipelines, tune complex SQL/Spark queries for performance, and reduce cloud computing costs for our clients

Preferred

• Databricks, AWS, or dbt certifications
• Experience working on migration projects from legacy warehouses (Teradata, Netezza, Snowflake, Redshift) to Databricks
• Familiarity with streaming tools like Apache Kafka or CDC (Change Data Capture) pipelines
• Databricks Certification
• AWS Certification
• dbt Certification
• Data Warehouse Migration
• Teradata
• Netezza
• Snowflake
• Redshift
• Databricks
• Apache Kafka
• Streaming Data
• Change Data Capture (CDC) Pipelines

Benefits

• May be eligible for an annual discretionary bonus.

Company

• Enterprise AI consultancy delivering production AI products and platforms
• New York, New York, USA
• https://indicium.ai/
• Founded in 2017
• 501-1000 employees

Qualifications

• Strong background writing production code in **Python** and advanced **SQL**
• Hands-on experience building on the Databricks Lakehouse platform (Delta Lake, PySpark, Unity Catalog, Workflows)
• Solid experience using **dbt** for data transformation, testing, and documentation
• Practical experience writing **Terraform** scripts to deploy AWS or GCP cloud data infrastructure
• Proficient with **Git** (GitHub/GitLab) for CI/CD workflows and experience with schedulers like **Airflow**
• Ability to communicate technical decisions clearly to client teams and collaborate in a fast-paced consulting environment
• Ability to jump into unfamiliar legacy codebases, figure out how the data flows, and rebuild it cleanly without needing a rigid playbook
• Passion for writing testable, documented, and reusable code
• Databricks, AWS, or dbt certifications
• Experience working on migration projects from legacy warehouses (Teradata, Netezza, Snowflake, Redshift) to Databricks
• Familiarity with streaming tools like Apache Kafka or CDC (Change Data Capture) pipelines

Required skills

• Python, SQL, Databricks, PySpark, Delta Lake, Unity Catalog, dbt, Terraform, AWS, GCP, Git, Apache Airflow

Preferred skills

• Databricks, AWS, or dbt certifications

Experience

• Mid level