Specialist Data Engineer

Responsibilities

• Design, develop, and implement data solutions to meet business requirements and data ingestion needs facilitating accurate and timely data availability for analysis and decision-making
• Extract, load and Transform (ELT/ETL) data from various sources, including on-premises and cloud-based systems, APIs, databases, and files
• Write well designed, efficient code, that adheres to security standards
• Monitor and troubleshoot data pipelines to identify and resolve issues in a timely manner to minimize disruptions in data processing for on premise and cloud environments
• Implement data quality checks and validation processes to ensure accuracy and completeness of data. collaborate with the data team to resolve them
• Writes complex queries and scripts to efficiently manipulate, transform, and process raw data
• Creates and executes data validation processes to ensure the reliability and consistency of incoming data. Build processes to monitor data quality
• Continuously optimize data pipelines for performance, scalability, and reliability
• Create and maintain technical documentation. Contribute to building and maintaining data catalog and lineage
• Design and develop CI/CD processes that ensure high availability and agility
• Develop cloud data services provisioning automation with integrated capabilities of IAM, network, security policies as code, and observability
• Build tools and services to support data discovery, lineage, resiliency, and privacy compliance across the data platform
• Stay updated with the latest trends and best practices in data engineering, cloud computing, and Azure services to suggest innovative solutions to continually improve the organization's data intelligence capabilities
• Monitors and reports on supplier performance, customer satisfaction, adherence to security requirements and market intelligence
• May mentor less experienced staff

Preferred

• Microsoft Certified: Azure Data Engineer Associate is preferred
• Microsoft Certified: Azure Data Fundamentals is preferred
• Experience with Microsoft Purview is a plus
• Data Structures and Algorithms (Thorough Knowledge/Fully Proficient)
• Cloud Computing (Thorough Knowledge/Fully Proficient)

Benefits

• This position is eligible for teleworking, which is currently 2 days per week. New hires are eligible to apply 30 days after their effective date of hire.
• Glassdoor
• 3.6
• The Metropolitan Transportation Authority is North America's largest transportation network, serving a population of 15.3 million people in the 5,000-square-mile area fanning out from New York City through Long Island, southeastern New York State, and Connecticut.
• New York, NY, US
• https:/mta.info/
• Late Stage
• Leadership Team
• Janno Lieber
• Chair & CEO
• Sam Tabikh, P.E.
• Facilities Engineer and Projects CEO
• Recent News
• Crain's New York Business
• MTA targets overweight trucks with automated tickets to reduce $30M in yearly bridge damage
• 2025-10-07
• San Francisco Chronicle
• Kids in New York keep dying while 'subway surfing' on top of trains. Can they be stopped?
• 2025-10-05
• The Real Deal
• The Daily Dirt: A new path to affordable housing takes shape in Soho
• 2025-10-04
• Company data provided by
• AI Tools
• Customize Your Resume
• Maximize your interview chances
• Build Cover Letter
• Make your application stand out
• Analyze How Well You Fit
• Understand your strength & weakness

Qualifications

• Bachelor's Degree in Arts/Sciences (BA/BS). An equivalent combination of education and experience may be considered in lieu of a degree
• Minimum 3 years of relevant experience
• Strong knowledge of Big Data architectures, large data warehouse and Data Lake solutions
• Experience designing and implementing Modern Data and Analytics solutions including Lakehouse architecture, Medallion architecture
• Proficient in cloud services including Azure Databricks. Azure Synapse Analytics, Azure Data Factory, Azure Data Lake Store, Microsoft Purview and Power BI
• Experience deploying and running cloud-based data solutions and infrastructure-as-code frameworks like Terraform
• Proficient in cloud deployments (MS Azure preferred) in an agile SDLC environment, leveraging modern programming languages, DevOps
• Experience with major database platforms including Oracle, SQL Server as well as Cloud databases and NOSQL Databases
• Strong understanding of data engineering concepts, ELT/ETL principles, and data modeling
• Experience with data integration techniques for both structured and unstructured data
• Solid programming skills in languages such as Python, Pyspark, SQL
• Experience with Airflow
• Experience in DevOps, Git Repos and CI/CD pipelines for code deployment
• Experience in programming and infra-automation tools – Terraform, Ansible, python, etc
• Strong knowledge of Microsoft Azure Cloud. AWS and GCP desirable
• Functional knowledge of Microsoft Power BI
• Experience with Jira, Confluence
• Demonstrated ability to work independently and navigate organizational ambiguity
• Effective written and verbal communication skills
• Hands-on programming experience in a business setting
• Proficiency in at least one software engineering methodology, including but not limited to Agile, Scrum, DevOps, Extreme Programming (XP), Kanban, Lean, Rapid Application Development (RAD)
• Experience applying a structured validation and testing method, including but not limited to: Unit Testing, Integration Testing, System Testing, Acceptance Testing, Regression Testing
• Proficient collaboration skills to ensure design specifications are seamlessly implemented by the development team
• Microsoft Certified: Azure Data Engineer Associate is preferred
• Microsoft Certified: Azure Data Fundamentals is preferred
• Experience with Microsoft Purview is a plus
• Data Structures and Algorithms (Thorough Knowledge/Fully Proficient)
• Cloud Computing (Thorough Knowledge/Fully Proficient)
• Minimum 3 years of relevant experience.

Required skills

• Microsoft Azure, Azure Databricks, Azure Synapse Analytics, Azure Data Factory, Azure Data Lake Store, Microsoft Purview, Power BI, Python, PySpark, SQL, Terraform, Apache Airflow

Preferred skills

• Microsoft Certified: Azure Data Fundamentals is preferred, Experience with Microsoft Purview is a plus, Cloud Computing (Thorough Knowledge/Fully Proficient)

Experience

• 3+ years