HireHere
Apolloresearch

Backend Engineer (Monitoring)

Apolloresearch London Today
engineering
Application deadline: We accept submissions until 16 January 2026. We review applications on a rolling basis and encourage early submissions.


THE OPPORTUNITY

Join our new AGI safety monitoring team and help transform complex AI research into practical tools that reduce risks from AI. As a Backend Engineer, you'll work closely with our CEO, monitoring engineers and Evals team software engineers to build tools that make AI agent safety accessible at scale. We are building tools that monitor AI coding agents for safety and security failures.

You will join a small team and will have significant ability to shape the team & tech, and have the ability to earn responsibility quickly. You will like this opportunity if you care about building tools that genuinely make AI agents safe and thrive in high-paced environments as well as enjoy closely working with researchers.


KEY RESPONSIBILITIES

Infrastructure & Architecture
- Design and implement scalable backend systems capable of processing and analyzing large volumes of AI agent logs in real-time
- Build and maintain data processing pipelines that extract, transform, and store agent trajectory data efficiently
- Architect database schemas and data models optimized for both high-throughput writes and complex analytical queries
- Design for reliability, implementing robust error handling, retry logic, and graceful degradation strategies
- Monitor system performance and optimize bottlenecks to ensure sub-second latency for critical monitoring operations

API Development
- Develop secure, well-documented RESTful APIs that allow users to integrate our monitoring tools into their workflows
- Implement authentication, authorization, and rate limiting to protect users data and ensure fair resource usage
- Build webhook systems and real-time notification services to alert users about critical safety events
- Design API interfaces that are intuitive for developers while remaining flexible for diverse user use cases
- Design and implement integrations with Security Information and Event Management (SIEM) systems, enabling users to stream monitoring alerts and security events into their existing security operations workflows

Data Systems
- Implement efficient storage solutions for both structured data (monitoring results, metadata) and unstructured data (agent logs, code outputs)
- Build data processing systems that can handle everything from streaming real-time monitoring to batch analysis of historical data
- Design and implement caching strategies to optimize frequent queries and reduce infrastructure costs
- Create data retention and archival policies that balance users needs with storage efficiency

Monitoring & Observability
- Build comprehensive logging, metrics, and tracing systems to ensure visibility into system health and performance
- Implement alerting systems that notify the team of infrastructure issues before they impact users
- Create dashboards and tools that help the team understand system behavior and diagnose issues quickly
- Design systems that make debugging production issues straightforward and minimize time-to-resolution

Collaboration & Quality
- Work closely with our researchers to understand their needs and translate research prototypes into production-ready systems
- Collaborate with frontend engineers to design APIs and data structures that enable excellent user experiences
- Participate in code reviews to maintain high standards for code quality, security, and performance
- Document architectural decisions, API specifications, and system behaviors to facilitate knowledge sharing
- Contribute to technical discussions about technology choices, trade-offs, and implementation approaches

JOB REQUIREMENTS

  • 4+ years of experience building production backend systems at scale
  • Strong Python proficiency with experience in frameworks like FastAPI, Flask, or Django
  • Experience designing and implementing RESTful APIs with clear documentation
  • Solid understanding of database design and optimization (SQL and/or NoSQL)
  • Experience with cloud platforms (AWS, Google Cloud, or Azure) and containerization technologies (Docker, Kubernetes)
  • Experience building data-intensive applications or processing large-scale log data
  • Strong understanding of system design principles, including scalability, reliability, and security
  • Experience with asynchronous processing, message queues, and distributed systems
  • Demonstrated ability to write clean, well-tested, maintainable code

    • Bonus
  • Familiarity with real-time data processing frameworks (Kafka, Redis Streams, etc.)
  • Experience with ML/AI infrastructure or building tools for AI applications
  • Previous work on developer tools, monitoring systems, or security tools
  • Experience with infrastructure-as-code (Terraform, CloudFormation, etc.)
  • Familiarity with AI safety concepts or evaluation frameworks like Inspect
  • Contributions to open-source backend infrastructure projects
  • Experience building security-centric tools
  • Experience with code analysis platforms
  • Experience with Golang

    • We want to emphasize that people who feel they don't fulfill all of these characteristics but think they would be a good fit for the position nonetheless are strongly encouraged to apply. We believe that excellent candidates can come from a variety of backgrounds and are excited to give you opportunities to shine.

    REPRESENTATIVE PROJECT

  • Real-time agent monitoring infrastructure: Design and build the backend system that processes AI coding agent outputs in real-time to detect safety and security issues. Start by implementing a scalable ingestion pipeline that can accept agent logs via API, then build a processing system that routes logs through various monitors based on their characteristics. Implement a storage layer that efficiently handles both recent high-frequency queries and historical analysis. Add a notification system that alerts users when monitors detect concerning behaviors, with configurable thresholds and delivery methods. Throughout the project, ensure the system maintains sub-second p95 latency for critical operations while gracefully handling traffic spikes and partial system failures.

    • BENEFITS

  • Salary: 100k - 180k GBP (~135k - 245k USD)
  • Flexible work hours and schedule
  • Unlimited vacation
  • Unlimited sick leave
  • Lunch, dinner, and snacks are provided for all employees on workdays
  • Paid work trips, including staff retreats, business trips, and relevant conferences
  • A yearly $1,000 (USD) professional development budget.

    • LOGISTICS

  • Start Date: Target of 2-3 months after the first interview
  • Time Allocation: Full-time
  • Location: The office is in London, and the building is next to the London Initiative for Safe AI (LISA) offices. This is an in-person role. In rare situations, we may consider partially remote arrangements on a case-by-case basis
  • Work Visas: We can sponsor UK visas
    • ABOUT THE TEAM

    The monitoring team is a new team. Especially early on, you will work closely with Marius Hobbhahn (CEO), Jeremy Neiman (engineer) and others on the monitoring team. You’ll also sometimes work with our SWEs, Rusheb Shah, Andrei Matveiakin, Alex Kedrik, and Glen Rodgers to translate our internal tools into externally usable tools. Furthermore you will interact with our researchers, since we intend to be “our own customer” by using our tools internally for our research work. You can find our full team here.

    ABOUT APOLLO

    The rapid rise in AI capabilities offer tremendous opportunities, but also present significant risks. At Apollo Research, we’re primarily concerned with risks from Loss of Control, i.e. risks coming from the model itself rather than e.g. humans misusing the AI. We’re particularly concerned with deceptive alignment / scheming, a phenomenon where a model appears to be aligned but is, in fact, misaligned and capable of evading human oversight. We work on the detection of scheming (e.g., building evaluations), the science of scheming (e.g., model organisms), and scheming mitigations (e.g., anti-scheming and control). We closely work with multiple frontier AI companies, e.g. to test their models before deployment or collaborate on scheming mitigations. 

    We’re now also developing tools that make it easier to prevent harms from AI systems widely deployed AI systems. We specifically target coding agent safety since coding agents are the most advanced agents and tasked with high-stakes decisions.

    At Apollo, we aim for a culture that emphasizes truth-seeking, being goal-oriented, giving and receiving constructive feedback, and being friendly and helpful. If you’re interested in more details about what it’s like working at Apollo, you can find more information here.

    Equality Statement: Apollo Research is an Equal Opportunity Employer. We value diversity and are committed to providing equal opportunities to all, regardless of age, disability, gender reassignment, marriage and civil partnership, pregnancy and maternity, race, religion or belief, sex, or sexual orientation.

    HOW TO APPLY

    Please complete the application form with your CV. The provision of a cover letter is neither required nor encouraged. Please also feel free to share links to relevant work samples.

    About the interview process: Our multi-stage process includes a screening interview, a take-home test (approx. 3 hours), 3 technical interviews, and a final interview with Marius (CEO). The technical interviews will be closely related to tasks the candidate would do on the job. There are no leetcode-style general coding interviews. If you want to prepare for the interviews, we suggest getting familiar with the evaluations framework Inspect, or by building simple monitors for coding agents and running them on your own Claude Code / Cursor / Codex / etc. traffic.

    Your Privacy and Fairness in Our Recruitment Process: We are committed to protecting your data, ensuring fairness, and adhering to workplace fairness principles in our recruitment process. To enhance hiring efficiency, we use AI-powered tools to assist with tasks such as resume screening. These tools are designed and deployed in compliance with internationally recognized AI governance frameworks. Your personal data is handled securely and transparently. We adopt a human-centred approach: all resumes are screened by a human and final hiring decisions are made by our team. If you have questions about how your data is processed or wish to report concerns about fairness, please contact us at info@apolloresearch.ai.

    Similar Jobs

    Binance

    Senior Backend Engineer - Service Monitoring & Observability

    Binance

    Engineering
    Databricks

    Platform Monitoring Engineer

    Databricks

    Engineering
    Databricks

    Platform Monitoring Engineer

    Databricks

    Engineering
    GitLab

    Intermediate Backend Engineer (Golang), Monitor: Platform Insights

    GitLab

    Engineering Remote
    KPN

    Consultant-engineer Monitoring (ELK Stack)

    KPN

    Engineering
    Report this job
    Sign In to Apply