AI Test Engineer (SME)

NeatDevs LLCAI Test Engineer (SME)

Ashburn, VA

·

Full time

Apply for AI Test Engineer (SME)

NeatDevs LLC is a small federal subcontractor with a long-running seat on a U.S. Customs and Border Protection (CBP) program in Ashburn, VA. We are hiring an SME-level AI Test Engineer to own testing of AI components - Gemini skills, AI agents, and MCP integrations - defining how non-deterministic AI behavior is evaluated and building the automated evaluation and regression suites that run in CI/CD. U.S. citizenship is required (federal contract); you must be able to obtain a DHS/CBP Public Trust - an active or recent clearance shortens your start date but is not a gate. Hybrid: on-site in Ashburn, VA about one day per week, remote the rest.

About NeatDevs LLC

NeatDevs, LLC is a dynamic federal contracting company dedicated to providing cutting-edge technology solutions across custom software development, artificial intelligence, cybersecurity, and data analytics. We excel by delivering innovative, secure, and mission-critical applications tailored specifically to the unique needs of federal agencies.

At NeatDevs, our standout culture emphasizes collaboration, continuous learning, and a commitment to excellence. Our employees thrive in an environment that promotes innovation, supports professional growth, and fosters work-life balance. We proudly leverage agile methodologies and advanced technology stacks to drive impactful solutions that protect national interests and enhance government efficiency.

Join NeatDevs and become part of a passionate team making meaningful contributions to our nation’s security and technological advancement. Our employees benefit from:

An engaging, collaborative, and inclusive team environment

Robust professional development and mentorship opportunities

Excellent work-life balance and flexible work arrangements

Competitive compensation and comprehensive benefits, including immediate 401K matching

If you’re looking to grow your career with a company that values innovation, integrity, and teamwork, NeatDevs is the place for you!

Description

What you'll do

• Define and own test strategy, standards, and quality gates for AI components
• Build evaluation approaches for non-deterministic output: golden datasets, scoring rubrics, acceptance thresholds, LLM-as-judge evaluation
• Test agents end to end: task completion, tool selection and trajectory, guardrails, human-in-the-loop checkpoints
• Test MCP servers and clients: schema conformance, error handling, authentication, least-privilege access
• Perform adversarial testing: prompt injection, jailbreaks, data leakage; measure latency, token usage, and cost
• Maintain automated suites in CI/CD; track test activity in Jira/Xray; mentor the team on AI testing

Required

• U.S. citizenship and ability to obtain a DHS/CBP Public Trust; must live in the DC/MD/VA area
• Bachelor's degree (BA/BS) in Computer Science or related field
• 10+ years of software QA and testing experience, including 5+ years in test automation
• 2+ years hands-on testing AI/ML or LLM-based applications (chat assistants, RAG, agents, tool integrations)
• Strong Python test development (pytest); strong API testing (REST, JSON schemas, auth)
• Working knowledge of agent concepts and the Model Context Protocol (MCP)
• CI/CD test integration (GitLab, Jenkins)
• Nice to have
• Gemini / Vertex AI; Google ADK or LangGraph agent evaluation
• LLM eval frameworks (Vertex AI GenAI eval, promptfoo, DeepEval, Ragas)
• OWASP LLM Top 10, NIST AI RMF; Section 508 (ANDI); ISTQB AI Testing certification

Pay and benefits

$104,000 - $119,000 per year depending on experience and clearance status. Full-time W-2 role with health benefits, 401(k), and paid federal holidays.

NeatDevs LLC is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or protected veteran status.

Salary

$104,000 - $119,000 per year

Apply for AI Test Engineer (SME)