Quality Analyst, AI Business
Quality Analyst, AI Business
Team: Client Services
Prolific
Prolific isn’t just enabling AI innovation – we’re redefining it. While foundational AI technologies are becoming commoditized, Prolific’s human data infrastructure provides the high-quality, diverse data required to train the next generation of AI models. Through our platform, we empower researchers and companies to access a global, ethically curated participant base, ensuring cutting-edge AI research and training grounded in inclusivity and precision.
The Role
As a Quality Analyst within Prolific AI Data Services, you will play a critical role in ensuring the consistency, reliability, and integrity of the human data we deliver to customers. As task types and volumes grow, quality can’t rely on spot checks or tribal knowledge—this role helps operationalise quality through measurement, early detection, and clear feedback loops.
You’ll work closely with Operations, reviewers, and Quality and Delivery leads to run day-to-day quality measurement, surface issues early, and help turn fixes into repeatable workflows that scale across studies and customers.
What You’ll Be Doing
- Run day-to-day quality measurement for managed service studies, including sampling, rubric-based review, and structured reporting.
- Apply rubrics and guidelines consistently, flag ambiguities, and propose clear, actionable clarifications as task types evolve.
- Maintain and update golden sets, and support calibration efforts by preparing materials, tracking outcomes, and documenting decisions.
- Track defects and failure modes by categorising issues, assessing severity, and keeping defect taxonomies current.
- Run automated quality checks using SQL and Python or R, including completeness checks, schema and format validation, anomaly detection, and label distribution monitoring.
- Support launch gates by preparing readiness artefacts such as rubric versioning, gold status, sampling plans, and early pilot findings.
- Investigate quality drops by pulling examples, quantifying scope, and supporting root-cause analysis and corrective and preventive actions (CAPAs).
- Build dashboards and weekly updates covering defect rates, rework, agreement, throughput versus quality trade-offs, and SLA risk.
- Partner with Operations, reviewers, and—when needed—Product and Engineering to improve task design, reviewer workflows, and escalation paths.
What You’ll Bring to the Role
- 1–3 years of experience in analysis, quality, operations, or data-focused roles, or equivalent academic projects or internships.
- A background in computer science, data science, statistics, engineering, or a related quantitative discipline.
- Strong SQL skills (required), with the ability to independently query, join, and summarise datasets to answer quality and operational questions.
- Working proficiency in Python and/or R, with experience writing scripts or notebooks to clean data, compute metrics, and automate checks.
- Strong analytical rigour, with a high bar for precision, consistency, and clear documentation of decisions.
- Solid statistical fundamentals, including intuition for sampling, distributions, confidence, and agreement concepts.
- Clear and structured communication skills, with the ability to explain issues using concrete examples and propose practical fixes.
- A collaborative mindset and willingness to partner cross-functionally to improve quality outcomes and delivery reliability.
Even Better if you have - Experience with human data workflows such as annotation or data labelling, or ML/LLM evaluation (rubric-based evals, preference data, red teaming).
- Familiarity with QA concepts such as defect severity, leakage, root-cause analysis, and CAPA processes.
- Experience building or maintaining dashboards in BI tools such as Looker, Tableau, or Mode.
Why Prolific is a great place to work
We've built a unique platform that connects researchers and companies with a global pool of participants, enabling the collection of high-quality, ethically sourced human behavioral data and feedback. This data is the cornerstone of developing more accurate, nuanced, and aligned AI systems.
We believe that the next leap in AI capabilities won't come solely from scaling existing models, but from integrating diverse human perspectives and behaviors into AI development. By providing this crucial human data infrastructure, Prolific is positioning itself at the forefront of the next wave of AI innovation – one that reflects the breath and the best of humanity.
Working for us will place you at the forefront of AI innovation, providing access to our unique human data platform and opportunities for groundbreaking research.
Join us to enjoy a competitive salary, benefits, and hybrid working within our impactful, mission-driven culture. At Prolific, our compensation packages for eligible roles include base salary, equity, and benefits. Many roles also include the opportunity to earn a cash variable element, such as a bonus or commission. Your recruiter will also be happy to share the specific salary range for your preferred location during the hiring process.
Links to More Information
Privacy Statement
By submitting your application, you agree that Prolific may collect your personal data for recruiting and global organisation planning. Prolific's Candidate Privacy Notice explains what personal information Prolific may process, where Prolific may process your personal information, its purposes for processing your personal information, and the rights you can exercise over Prolific use of your personal information.