Senior Data Engineer

As a Senior Data Engineer on the AI Enablement team within the Data and AI Center of Excellence (DAICOE) at IDEXX, you will build robust, fault-tolerant data pipelines that collect, transform, and deliver veterinary clinical and diagnostic data to support the development of AI-powered diagnostic tools. Embedded within a cross-functional project team, you will take full ownership of end-to-end data pipeline development, quickly establishing yourself as the technical data engineering lead with minimal ramp-up.

Your work will be anchored in dbt Core, with deep hands-on involvement in data transformation, testing, and pipeline quality on Databricks. Our dbt environment spans projects at varying stages of maturity, all guided by consistent best practices and standards. A real opportunity exists to help shape the future of our dbt projects, building toward modular, templatized dbt structures and shared dbt packages that make our standards easier to apply and scale across the team.

This role sits within our R&D organization. You will work primarily with veterinary clinical and diagnostic data, including cell annotation data produced by pathologists, partnering closely with ML engineers and data scientists. You will collaborate with application teams and centralized ITS data engineering teams where workstreams intersect.

Core technology stack: dbt Core, Databricks, Databricks CLI, Spark SQL, AWS (S3), Python, SQL, GitHub

This is a hybrid role requiring 8 days per month in office, with all Tuesdays as a required in-office day. Our office is located in Westbrook, Maine.

Applicants must be authorized to work for ANY employer in the U.S. We are unable to sponsor or take over sponsorship of an employment visa for this role.

In This Role You Will

Manage Data Pipelines

• Build and maintain sophisticated ELT pipelines using dbt Core, SQL, and Python on Databricks, with a strong emphasis on transformation layer design, data modeling, and code quality.

• Following a medallion architecture, own end-to-end data pipeline development from ingestion of structured data sources through transformation and delivery to data consumers for your primary project.

• Implement Databricks-native streaming ingestion patterns and materializations, working with Parquet and JSON file formats sourced from AWS S3-backed Databricks Volumes, including schema evolution handling.

Ensure Data Quality & Governance

• Write and maintain dbt tests and unit tests to ensure data accuracy, completeness, and reliability across pipeline layers.

• Monitor and alert on pipeline performance, data quality, and system health. Identify issues or optimization opportunities and implement solutions independently.

• Apply and contribute to data governance practices, including data documentation, lineage, and access standards, ensuring pipelines meet organizational and compliance requirements.

Uphold Deployment & Engineering Standards

• Implement and maintain CI/CD practices for dbt pipelines, including version control workflows, automated testing, and deployment processes via GitHub.

• Use Databricks Asset Bundles (DABs) to define, deploy, and manage Databricks resources as versioned, source-controlled infrastructure across environments.

• Develop documentation and reusable standards that enable consistent dbt project onboarding and self-service capabilities across the team. Author and maintain shared assets (e.g., dbt Jinja macros) to drive modularization and templatization at scale.

Collaborate Cross-Functionally

• Adhere to established technical standards and best practices set by data engineering leadership, collaborating with technical leads to align project work with broader organizational objectives and actively contributing improvements based on project experience.

• Work closely with ML engineers and data scientists to support ML-adjacent data needs, including feature preparation and data delivery for model development and evaluation.

• Provide informal technical guidance and mentorship to less experienced engineers on your project teams.

What You Need to Succeed

Required

• 4-6 years of experience in data engineering, with demonstrated ability to own end-to-end pipeline development.

• Bachelor's degree in engineering, analytics, computer science, information systems, or a related field.

• Strong hands-on experience with dbt Core or dbt Cloud including data modeling, transformation, testing, documentation, and deployment.

• Proven experience building and maintaining data pipelines on a modern cloud data platform (e.g., Databricks, Snowflake). A strong understanding of how compute, storage, and pipeline execution interact within that platform.

• Deep understanding of data warehousing concepts and ELT architectural patterns, including medallion architecture.

• Expert proficiency in SQL as the primary language for data transformation and pipeline logic. Proficiency using Spark SQL for distributed data processing. Proficiency in Python for pipeline development and automation.

• Streaming ingestion patterns, including schema evolution handling and working with Parquet and JSON file formats sourced from cloud storage.

• Building and maintaining CI/CD pipelines for data workflows (GitHub Actions or equivalent).

• Unit testing and data quality testing frameworks within a dbt or Python context.

• Cloud-based data pipelines on AWS, particularly working with S3.

• Ability to quickly embed within new cross-functional teams and take ownership of technical decisions with minimal support.

• Collaboration and communication skills. Comfortable working alongside ML engineers, data scientists, and researchers in an R&D environment.

Nice to Have

• Infrastructure-as-code deployment tools (e.g., Terraform, Databricks Asset Bundles).

• Apache Iceberg or other open table formats in a lakehouse context.

• Authoring or maintaining a shared dbt package or reusable macro library.

• Life sciences, clinical, veterinary, or scientific data.

• Agile and Scrum methodologies.

Why IDEXX?

We're proud of the work we do because our work matters. An innovation leader in every industry we serve, we follow our Purpose and Guiding Principles to help pet owners worldwide keep their companion animals healthy and happy, to ensure safe drinking water for billions, and to help farmers protect livestock and poultry from disease. We have customers in over 175 countries and a global workforce of over 10,000 talented people.

So, what does that mean for you? We enrich the livelihoods of our employees with a positive and respectful work culture that embraces challenges and encourages learning and discovery. At IDEXX, you will be supported by competitive compensation, incentives, and benefits while enjoying purposeful work that drives improvement.

Let's pursue what matters together.

IDEXX values a diverse workforce and workplace and strongly encourages women, people of color, LGBTQ+ individuals, people with disabilities, members of ethnic minorities, foreign-born residents, and veterans to apply.

IDEXX is an equal opportunity employer. Applicants will not be discriminated against because of race, color, creed, sex, sexual orientation, gender identity or expression, age, religion, national origin, citizenship status, disability, ancestry, marital status, veteran status, medical condition, or any protected category prohibited by local, state, or federal laws.

#LI-KS1