Testing Lead-QA

Job Title: Testing Lead – Deep Learning / LLM / VLM

Location

Remote /

Hybrid / On-site

Experience

3–4 years

(hands-on ownership in DL / LLM / GenAI testing)

Employment Type

Full-time

Role Overview

We are

• seeking a hands-on Testing Lead to own quality and documentation for our Deep Learning, LLM, and Vision-Language Model (VLM) products. You will
• define how we test, measure, document, and communicate AI quality—working
• closely with ML, Engineering, and Product teams in a fast-paced startup
• environment.
• This role
• is ideal for someone who believes clear documentation is as critical as good
• testing, especially for non-deterministic AI systems.

What You’ll Do

Own Quality & Documentation End-to-End

• Define testing strategy for LLMs,
• VLMs, and DL pipelines.
• Create and maintain clear,
• lightweight documentation covering:
• Model testing strategies
• and assumptions
• Evaluation metrics and
• acceptance criteria
• Known limitations, risks,
• and failure modes
• Release readiness and
• quality sign-off
• Ensure documentation evolves
• with models, data, and prompts.

LLM / GenAI Testing

Design tests for

• Prompt templates and prompt
• changes
• RAG pipelines (retrieval
• quality, grounding, hallucination control)
• Multi-turn conversations
• and long-context behaviour
• Maintain golden datasets,
• regression test suites, and test result summaries.
• Document prompt behaviour,
• edge cases, and known model quirks.

Vision & Multimodal Testing

• Test VLMs for image-text
• alignment, OCR, captioning, and reasoning.
• Document model performance
• across different image types, quality levels, and domains.
• Track and publish model
• behaviour changes between versions.

Automation, MLOps & Reporting

Build Python-based

• automation for evaluation and regression testing.
• Integrate tests into CI/CD
• and MLOps pipelines.
• Produce readable quality
• reports and dashboards for engineers and leadership.
• Monitor and document
• production issues such as model/data drift and degradation.
• Build a Quality-First Culture
• Establish QA and
• documentation standards that scale with a startup.
• Mentor engineers on writing
• testable code and meaningful documentation.
• Act as the single source
• of truth for AI quality, testing, and known risks.

What we’re looking For

Must-Have

• Strong background in software
• testing with lead or ownership experience.
• Hands-on experience testing LLMs,
• DL models, or GenAI systems.
• Strong Python skills
• for test automation and data validation.
• Proven ability to write clear,
• structured technical documentation.

Understanding of

• Transformer-based models
• and DL workflows
• Model evaluation metrics
• and non-deterministic system testing
• Comfortable working in
• ambiguity and moving fast in a startup.

Nice-to-Have

• Experience with VLMs,
• multimodal models, or computer vision.
• Exposure to RAG
• architectures, vector databases, and embeddings.
• Familiarity with tools like
• LangChain, LlamaIndex, MLflow, or similar.

Experience documenting AI

risks, limitations, or compliance requirements.