Databricks Data Engineering Consultant

SDG Group is a data, analytics, and AI consulting firm that delivers data solutions for its clients. The Databricks Data Engineering Consultant designs, develops, and delivers scalable data solutions, with an emphasis on Databricks SQL and SQL-based engineering, while collaborating with clients and delivery teams.

Responsibilities

• Databricks Development: Design, build, test, and maintain scalable data solutions within Databricks, with a strong focus on SQL-based transformation and data processing workloads
• SQL Development: Write, troubleshoot, and optimize complex SQL queries used for data transformation, validation, reconciliation, and analytical workloads
• Data Engineering & Development: Build and maintain scalable data pipelines and processing solutions that support client analytics and business requirements
• Data Integration: Integrate data from a variety of structured and unstructured sources using modern ETL/ELT methodologies and data integration tools
• Data Modeling & Warehousing: Develop and maintain data models and warehouse structures that support reporting, analytics, and downstream applications
• Cloud Data Solutions: Contribute to the implementation and modernization of cloud-based data platforms using Databricks and related cloud technologies
• Technical Delivery: Transform, validate, and manage enterprise data while following development, testing, documentation, and deployment best practices
• Data Quality & Performance: Identify, troubleshoot, and resolve data quality issues while improving reliability, scalability, and query performance
• Client Collaboration: Work directly with clients and internal stakeholders to gather requirements, understand business challenges, communicate technical concepts, and deliver solutions aligned with client needs
• Team Collaboration: Work closely with data engineers, analysts, architects, data scientists, and business stakeholders throughout the project lifecycle
• Continuous Learning: Stay current with evolving Databricks capabilities, data engineering methodologies, and industry best practices while contributing knowledge and ideas to the broader SDG community

Preferred

• Experience within consulting or another client-facing environment is a plus
• Experience with Matillion and/or dbt is highly preferred
• Databricks SQL Warehouses
• Delta Lake / Delta Tables
• Databricks Workflows and orchestration
• Unity Catalog
• Databricks query and performance optimization
• Matillion
• dbt
• Python / PySpark
• Snowflake
• Experience supporting enterprise-scale data migration or modernization initiatives
• Experience working within pharmaceutical, life sciences, healthcare, or another highly regulated industry
• Consulting experience
• Client-facing experience
• Delta Lake
• Delta Tables
• Databricks Workflows
• Databricks query optimization
• Databricks performance optimization
• PySpark
• Enterprise-scale data migration
• Data modernization
• Pharmaceutical industry experience
• Life sciences experience
• Healthcare industry experience
• Regulated industry experience

Benefits

• 401(k) retirement plan with employer match
• Medical, Dental, Vision, and Disability coverage
• 3+ weeks of paid vacation
• Flexible holiday schedule
• Hybrid working arrangements that balance on-site and remote work
• Career advancement and professional development
• Continuous training and exposure to modern Data, Analytics, and AI technologies

Company

• AI, data and analytics consulting firm
• Bedminster , New Jersey, US
• http://www.sdggroup.com/
• Founded in 1994
• 1001-5000 employees

Qualifications

• Approximately 2–5+ years of experience in data engineering, data warehousing, data integration, analytics engineering, or a related technical discipline
• Hands-on professional experience working within Databricks is required
• with demonstrated use of the platform for production data engineering and analytical workloads
• Strong experience using Databricks SQL
• including writing, troubleshooting, and optimizing queries used for data transformation, validation, reconciliation, and downstream analytics
• Strong SQL skills, including joins, CTEs, window functions, aggregations, and complex transformations
• Working proficiency in Python or another relevant programming or scripting language
• Strong understanding of data pipelines, ETL/ELT processes, data integration, data warehousing, and data transformation concepts
• Understanding of dimensional modeling and modern data modeling concepts used within analytical environments
• Experience working with modern cloud environments and enterprise-scale data platforms
• Experience with modern data engineering, orchestration, or transformation tools
• Experience validating, reconciling, and troubleshooting data across multiple source and target systems
• Strong analytical skills with the ability to troubleshoot data issues, understand complex requirements, and translate business needs into technical solutions
• Strong written and verbal communication skills with the ability to collaborate effectively with clients, technical teams, and business stakeholders
• A proactive, collaborative approach and willingness to continuously learn new tools, technologies, and methodologies
• With demonstrated use of the platform for production data engineering and analytical workloads
• Including writing, troubleshooting, and optimizing queries used for data transformation, validation, reconciliation, and downstream analytics
• Experience within consulting or another client-facing environment is a plus
• Experience with Matillion and/or dbt is highly preferred
• Databricks SQL Warehouses
• Delta Lake / Delta Tables
• Databricks Workflows and orchestration
• Unity Catalog
• Databricks query and performance optimization
• Matillion
• Dbt
• Python / PySpark
• Snowflake
• Experience supporting enterprise-scale data migration or modernization initiatives
• Experience working within pharmaceutical, life sciences, healthcare, or another highly regulated industry
• Approximately 2–5+ years of experience in data engineering, data warehousing, data integration, analytics engineering, or a related technical discipline.

Required skills

• Databricks, Databricks SQL, SQL, Python, Data Pipelines, ETL/ELT, Data Integration, Data Warehousing, Dimensional Data Modeling, Data Quality Validation, Matillion, dbt

Preferred skills

• Experience with Matillion and/or dbt is highly preferred, Databricks SQL Warehouses, Delta Lake / Delta Tables, Databricks Workflows and orchestration, Unity Catalog, Databricks query and performance optimization, dbt, Python / PySpark, Snowflake

Experience

• 2+ years