Senior QA Engineer
About the company
Joist AI is a technology company revolutionising the way professionals in the architecture, engineering, and construction (AEC) industry manage marketing and revenue operations. Our AI-powered software streamlines workflows, making it easier for teams to collaborate, innovate, and succeed.
About the Role
Joist AI is looking for a strategic and highly technical Senior QA Engineer to own the quality roadmap for our AI-powered RevOps platform. Unlike traditional software, our product relies on complex document processing pipelines, multi-model LLM architectures (OpenAI, Anthropic, Gemini), and intricate knowledge graphs.
You won’t just be finding bugs; you will be architecting the testing frameworks that ensure our AI's "hit rate" and compliance checks are flawless for our enterprise customers. Reporting to the Engineering Manager, you will lead the QA effort to ensure Joist AI remains the best-in-class platform for the built world.
What You’ll Do
QA Strategy & Leadership: Define and implement the end-to-end testing strategy for Joist AI, focusing on the Proposal Processing Pipeline and Knowledge Graph building.
AI/LLM Validation: Develop novel ways to test and monitor LLM-generated narratives and chat responses for accuracy, hallucination reduction, and compliance.
Pipeline Oversight: Own the quality of our extraction services (Adobe API), object/image detection, and captioning services.
Automation Architecture: Lead the development of automated test suites using Playwright (browser) and API automation (FastAPI) integrated into our CI/CD pipelines.
Defect Intelligence: Use Linear to track defects through the lifecycle and collaborate with leads (Bala, Sharat, Chandan) to prioritize fixes based on customer impact.
Cross-Functional Collaboration: Partner with Data Science teams to validate image processing (face/label detection) and with the GTM team to understand real-world proposal workflows.
What You’ll Bring
Experience: 6+ years of professional QA experience, with at least 2 years in a startup or high-growth AI/Data-heavy environment.
Technical Mastery:
Deep proficiency in SQL for validating complex RDS and OpenSearch data structures.
Advanced automation skills in Python or Java.
Expertise in Postman/Swagger for API testing within a microservices (AWS Lambda/FastAPI) architecture.
AI Knowledge: Familiarity with the STLC as it applies to non-deterministic systems, including RAG (Retrieval-Augmented Generation) and Agentic workflows.
Tools: Mastery of Linear (issue tracking), PostHog (user event tracking), and AWS environments (Cloudwatch, S3).
Attributes: You are an "Everyday Innovator" who thinks outside the box to solve the unique challenges of testing AI for the built world.
Experience We’d Be Particularly Excited About
AEC Industry Knowledge: Familiarity with the SMPS proposal process or RFP workflows.
AI Tooling: Experience with Langchain or evaluating LLM performance metrics.
DevOps Leanings: Interest in compliance or improving Developer Experience (DevEx).
What to expect
We conduct a rigorous interview process based on integrity, talent, and drive. We trust our teammates from day one and move quickly to evaluate whether you are fit for the role. The entire interview process typically takes two weeks. Here's what to expect:
A 30 minute Zoom meeting to talk about Joist AI, your background, and answer any questions about the role.
A 30 minute Zoom meeting with another one of our team members to hear more about your experience and how you'd approach working in the role.
A take home project to assess your functional expertise for the role you're applying for.
A 60 minute Zoom call to review your project and answer any outstanding questions.