Overview
We are seeking an experienced Business Test Analyst specializing in AI products to join our team on a contract basis. This hybrid role will focus on designing and implementing a comprehensive testing and evaluation framework for AI solutions, ensuring high quality and user satisfaction across various AI products. The analyst will work closely with engineering, product leads, and business users to establish rigorous testing protocols and create sustainable practices that uphold the integrity of our AI services.
Responsibilities
- Design and implement a comprehensive AI testing and evaluation framework for AI solutions.
- Define and document quality standards for semantic accuracy, factual consistency, and bias.
- Develop reusable testing templates, data sets, and evaluation methods for scaling and maintenance.
- Conduct hands-on testing of AI prototypes and production tools for performance assessment.
- Collaborate with business users to guide practical testing and feedback processes.
- Deliver training materials to empower internal staff for sustaining the framework post-contract.
- Support vendor evaluations and POC assessments with robust test protocols.
- Establish baseline metrics and dashboards to measure ongoing AI quality.
Requirements
- Strong hands-on experience in testing AI or software systems, particularly with NLP or LLM applications.
- Understanding of prompt evaluation, semantic search, and LLM behavior such as accuracy and bias.
- Familiarity with tools like Trulens, HumanLoop, or similar, with experience in GenAI QA approaches.
- Knowledge of modern AI architectures including RAG pipelines and API integrations.
- Experience in designing and implementing structured test regimes in dynamic environments.
- Excellent communication skills for engaging both technical and business audiences.
- Proven ability to create sustainable frameworks, documentation, and training content.