AI QA Specialist

Acentra Health exists to empower better health outcomes through technology, services, and clinical expertise. Our mission is to innovate health solutions that deliver maximum value and impact.

Lead the Way is our rallying cry at Acentra Health. Think of it as an open invitation to embrace the mission of the company; to actively engage in problem-solving; and to take ownership of your work every day. Acentra Health offers you unparalleled opportunities. In fact, you have all you need to take charge of your career and accelerate better outcomes - making this a great time to join our team of passionate individuals dedicated to being a vital partner for health solutions in the public sector.

Acentra Health is looking for an AI QA Specilalist to join our growing team.

As an AI QA Specialist within Acentra Health's AI & Analytics team, you will ensure the accuracy, reliability, and validity of our AI, ML, and Generative AI (GenAI) products and services. Leveraging your expertise in quality assurance methodologies, you will perform Functional, UI, and API testing to effectively validate products such as AI chatbots, ML/GenAI-generated clinical documents, microservices, and other software solutions. Your role will tackle unique challenges in GenAI environments, including output quality assurance and hallucination detection, ensuring stringent standards are consistently met. Aligned with this, you will support and contribute to our maturing AI Inter-Rater Reliability (AI-IRR) process and services.

Collaborating closely with data scientists, software developers, engineers, and stakeholders, you will design and execute comprehensive test plans and strategies specific to our AI and GenAI platforms. Responsibilities include creating and refining detailed test cases, verifying prompt-response accuracy, and ensuring the ethical and responsible use of AI outputs, ultimately contributing to improved patient outcomes.

Responsibilities:

• Develop and implement test strategies, detailed test plans, and test cases tailored to AI, ML, and GenAI solutions
• Conduct thorough Functional, UI, and API testing, including GenAI output quality, hallucination detection, and user acceptance testing to ensure optimal application performance
• Work closely with data scientists, software developers, engineers, and project managers to validate algorithms and integrate QA best practices into development processes
• Implement automation testing frameworks to optimize testing efforts, including automated GenAI evaluation workflows
• Verify training data integrity, perform bias detection, and conduct prompt-response validations for AI and GenAI model training
• Review and verify AI-generated outputs for consistency and accuracy. Collaborate with AI/ML engineers to enhance automated evaluation systems and pipelines
• Document test results, identify defects, collaborate to resolve issues, and prepare comprehensive reports with actionable insights for continuous improvement
• Ensure adherence to industry standards, regulatory requirements, and Acentra Health standards around ethical and safe AI usage
• Stay current with emerging AI testing methodologies and tools; contribute to evolving QA processes and strategies

• Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field; equivalent work experience will be considered
• 3+ years of quality assurance experience, with a strong background in traditional QA and UAT methodologies
• Proven experience in testing AI, ML, and Generative AI outputs, including Functional, UI, and API testing
• Proficiency in RESTful APIs and testing tools such as Postman, specifically for testing GenAI model APIs like OpenAI and Hugging Face
• Experience verifying training data integrity, bias detection, and prompt-response validation for AI models
• Familiarity with data annotation tools, quality control processes, and GenAI-specific annotation for RAG and LLM fine-tuning
• Demonstrated experience in testing complex software applications
• Proven experience designing, documenting, and executing comprehensive test plans for AI-driven products
• Strong analytical thinking, attention to detail, excellent communication skills, and the ability to thrive in a fast-paced, cross-functional team environment

Technical Skills:

• Strong experience with QA tools and frameworks like PyTest, Selenium, or JUnit, including Gen AI evaluation frameworks like RAGAS
• Understanding of AI, ML, and GenAI concepts, algorithms, and output validation techniques
• Familiarity with data annotation tools and quality control processes for GenAI-specific tasks
• Solid knowledge of Python, with experience in Gen AI libraries like LangChain and LlamaIndex
• Experience with GenAI model evaluation metrics such as perplexity and BLEU
• Solid knowledge of software development life cycle (SDLC) and agile methodologies

Preferred Qualifications

• Experience in healthcare technology or data analytics environments
• Familiarity with regulatory and ethical standards related to AI and healthcare solutions

Why us?

We are a team of experienced and caring leaders, clinicians, pioneering technologists, and industry professionals who come together to redefine expectations for the healthcare industry. State and federal healthcare agencies, providers, and employers turn to us as their vital partner to ensure better healthcare and improve health outcomes.

We do this through our people.

You will have meaningful work that genuinely improves people's lives across the country. We are a company that cares about our employees, and we give you the tools and encouragement you need to achieve the finest work of your career.

Benefits

Benefits are a key component of your rewards package. Our benefits are designed to provide you with additional protection, security, and support for both your career and your life away from work. Our benefits include comprehensive health plans, paid time off, retirement savings, corporate wellness, educational assistance, corporate discounts, and more.

Thank You!

We know your time is valuable and we thank you for applying for this position. Due to the high volume of applicants, only those who are chosen to advance in our interview process will be contacted. We sincerely appreciate your interest in Acentra Health and invite you to apply to future openings that may be of interest. Best of luck in your search!

~ The Acentra Health Talent Acquisition Team

Visit us at Acentra.com/careers/

EEO AA M/F/Vet/Disability

Acentra Health is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran or any other status protected by applicable Federal, State or Local law.

Compensation

The pay for this position is listed below.

"Based on our compensation philosophy, an applicant's position placement in the pay range will depend on various considerations, such as years of applicable experience and skill level."

#LI-SF1