Overview
We are seeking an experienced Business Test Analyst to develop and implement a comprehensive testing and evaluation framework for AI products on a contract basis. This contract role involves defining quality standards, conducting hands-on testing, and collaborating with business users to ensure the reliability and user focus of AI services. The successful candidate will play a crucial role in shaping AI quality practices within the organization while working in a hybrid setting, combining remote work with on-site collaboration.
Responsibilities
- Design and implement an AI testing and evaluation framework for various AI solutions.
- Define and document quality standards focusing on accuracy, consistency, bias, and relevance.
- Develop reusable templates and evaluation methods for internal teams.
- Conduct hands-on testing of AI prototypes and production tools.
- Collaborate with business users to enhance testing and feedback processes.
- Create training materials for internal staff to maintain testing frameworks post-contract.
- Support vendor evaluations and POC assessments with robust testing protocols.
- Establish metrics and dashboards to monitor ongoing AI quality.
Requirements
- Strong experience in testing and evaluating AI or software systems, ideally with NLP or LLM applications.
- Understanding of prompt evaluation, semantic search, and LLM behavior.
- Familiarity with tools like Trulens, HumanLoop, or PromptLayer.
- Knowledge of AI architectures including RAG pipelines and API integrations.
- Experience in designing structured test regimes in dynamic environments.
- Excellent communication skills for engaging technical and business audiences.
- Proven ability to create sustainable documentation and training materials.