Staff Software Engineer - Grafana Cloud Observability, Kubernetes Monitoring | USA - EST only | Remote

This is a remote opportunity and we would be interested in applicants based in the Unites States, EST timezones at this time.

Staff Software Engineer - Grafana Cloud Observability, Kubernetes Monitoring

The Opportunity: 

Grafana Cloud is our composable observability platform that integrates metrics, logs, and traces with Grafana. It allows our customers to leverage the best open source observability software – including Prometheus, Mimir, Loki, and Tempo – without the overhead of installing, maintaining and scaling their own observability stack

The Observability department is focused on enabling developers to understand the health and performance of their applications and infrastructure in any environment by providing tools to instrument their code, ingest observability data into Grafana Cloud and visualize and explore it.

In this role, you will be part of the team that builds our Cloud Observability stack that allows customers to collect and visualize metrics from various systems and applications. We build and maintain the backend of opinionated applications such as Cloud Provider Observability, Database Observability, and Kubernetes Monitoring. This includes the dashboards, alerts, documentation, and infrastructure while working closely with other teams to ensure seamless experiences. We also strive to incorporate OSS contributions in our work by contributing to projects such as Alloy, Prometheus, OpenTelemetry, and Beyla. The Observability department provides a core building block for customers using Grafana Cloud.

As a company we are remote-first and global, we embrace people of different experiences and backgrounds to build diverse teams where every person brings a unique perspective to the software. We are looking for Engineers that are passionate about communicating with data and providing seamless experiences for our customers to join our growing team! Engineers at Grafana also have the opportunity to contribute to Open Source communities.

What You’ll Be Doing:

At Grafana Labs, our engineers have a dedicated career path and do not have to become managers to progress in their career. Staff Software Engineers at Grafana have a large amount of experience across multiple areas. They are able to estimate, plan, coordinate and deliver large tasks spanning multiple systems. They actively coach and mentor other team members in their team and are able to identify and resolve issues with technology and product processes.

You will bring your passion for observability and software engineering expertise to help us take our infrastructure monitoring capabilities within Grafana Cloud to the next level. This will include working with our Kubernetes monitoring solution.

• Design and implement high-quality, scalable integrations for various infrastructure components, applications, and data ingestion pipelines 
• Create middleware components and libraries that simplify development and maintenance of observability solutions
• When necessary, represent Grafana Labs in open source forums, working groups, and events
• Work with product teams, in addition to design and docs, to develop features that align with wider product strategy and customer needs
• Lead the technical direction and vision of the team, contributing to strategic discussions and future development of observability solutions
• Work with other departments including Sales, Product, and Support teams to deliver a holistic product experience
• Take ownership of the services you’re running by deploying well tested clean code
• Embrace our open-source culture and contribute to other projects that may not directly fall within your team’s scope

As we are remote-first and our engineering organization is entirely remote, we provide guidance and meet regularly using video calls, so an independent attitude, good communication skills, and transparency are a must.

We invest heavily in developer productivity. You can use modern AI coding assistants as part of your daily workflow (your choice of tools, within security guidelines), backed by a company-funded usage budget so you can iterate quickly without unnecessary friction. We encourage pragmatic AI-assisted development: faster prototyping, test generation, refactors, documentation, and incident follow-ups—always paired with strong code review and quality standards. You’ll also have access to frontier models (e.g., GPT-Codex 5/3, Claude Opus 4.6, Gemini 3 Pro).

What Makes You a Great Fit:

• You have a passion for observability and like to share your knowledge by writing documentation and blog posts.
• You love to engage with customers and help them out.
• You have excellent communication skills.
• You have relevant open source experience, ideally in the observability domain.
• You are willing to become an active member of the OpenTelemetry and Prometheus communities.
• You’re curious and you enjoy learning new programming languages and frameworks, setting up examples, and figuring out how things work.
• You have a good understanding of typical production environments. Ideally you have been responsible for operating production services and organizing on-call.
• You actively mentor other team members, identifying areas for focus and improvement.

Requirements:

• Strong 8+ years of experience with at least one programming language - any major language (Python, .NET, Java, Go, Rust, etc) is acceptable
• Demonstrated working experience in operating high-scale production systems running on Kubernetes and monitoring it, including on-call participation, incident response, and postmortem practices
• Familiarity with observability tooling (e.g. Grafana)
• Strong understanding of time-series data, metrics cardinality challenges, and cost/performance tradeoffs/optimizations in observability systems
• Experience in a hands-on technical leadership role - setting technical direction, leading project teams, and influencing architectural decisions beyond your immediate team
• Deep understanding of distributed systems concepts including scalability, consistency, high availability, and failure modes in large-scale systems
• Experience writing clean, maintainable, robust, and performant software
• Experience with delivering projects from start to finish in a self-driven manner
• Excellent problem-solving and debugging skills
• Strong mentoring and leadership skills

Bonus Points For:

• Experience operating or scaling Prometheus in high-cardinality, multi-tenant environments
• Experience working with OpenTelemetry Collector pipelines or similar telemetry ingestion systems
• Certified Kubernetes Administrator (CKA)/ Certified Kubernetes Application Developer (CKAD) or any other Kubernetes related certification from CNCF
• Experience developing Kubernetes operators, controllers, or custom resources
• Strong understanding of metrics collection, visualization, and alerting concepts
• Experience contributing to or maintaining open source projects, with evidence of successful pull requests and community collaboration
• Experience designing and building observability backends for various systems and applications

Compensation & Rewards:

In the United States, the compensation range for this role is USD 174,986 - 209,983. Actual compensation may vary based on level, experience, and skillset as assessed throughout the interview process. All of our roles include Restricted Stock Units (RSUs), giving every team member ownership in Grafana Labs' success. We believe in shared outcomes—RSUs help us stay aligned and invested as we scale globally.