Technical Operations Engineer, Validators & Rollups
Quicknode is a cloud-based infrastructure company that powers the blockchain ecosystem.
Our mission is to be the indispensable utility that empowers companies and innovators globally to build next-generation, Web3 enabled businesses & applications using blockchain technology. Quicknode is backed by some of the world's best investors including Tiger Global, Y Combinator, SoftBank, and the Seven Seven Six Fund. The Quicknode team has over 120 people maintaining high performance global data infrastructure for amazing customers serving billions of requests daily.
We are a global remote company with an HQ in Miami, Florida.
Location
Remote (with regional coverage for 24hr operations). Limited travel may be required for conferences or offsites, generally less than 10 days per year.
The Role
Keep validators online and un‑slashable. We run production validators across L1/L2s and a focused set of rollups. You’ll run K8s at scale, ship IaC, lead incidents, and write the tooling that makes production boring—while guarding keys, latencies, and SLOs like they matter (because they do).
What We Actually Need
Linux + Kubernetes: you debug real production—networking, storage, rollouts, performance.
IaC (Terraform/Helm/Ansible): you ship repeatable infra and zero‑touch deploys.
Go or Python (plus Bash): you automate noise away and build small, sharp tools/CLIs.
Validator operations: operate/tune validators and sentry topologies; avoid slashing; manage peers; design backups/snapshots/state‑sync; handle client diversity and version skew.
Key management & signing: Vault/HSM/KMS; Web3Signer/TMKMS/Horcrux; slashing‑protection DBs; rotation and break‑glass procedures.
Observability that matters: SLOs/error budgets; Prometheus/Grafana; alerts that only page when users hurt.
Networking & systems depth: DNS/TLS/LB, P2P specifics (ports/NAT/conn limits), kernel/IO tuning, capacity & cost modeling that survives contact with reality.
Rollups (targeted scope): deploy/operate OP Stack/Arbitrum/Polygon CDK components (sequencer/batcher/indexer), wire DA endpoints (Celestia/Avail/EigenDA/AnyTrust), upgrade safely.
What You’ll Do
Run validators: deploy, upgrade, and tune EVM/Cosmos/Solana validators; manage client diversity (e.g., Prysm/Lighthouse/Teku + Geth/Nethermind/Erigon); integrate MEV‑Boost/relays where relevant; keep latency & attestation/commit rates high.
Protect keys & stake: implement secure remote signing, slashing protection, and audited change windows; test restores and failovers regularly.
Automate everything: IaC modules, golden images, CI/CD, snapshot pipelines, state‑sync validation, replay/load harnesses.
Operate K8s at scale: safe rollouts, HPA tuning, storage choices that won’t bite you; multi‑region capacity and budgets.
Lead incidents (SEV0–2): isolate fast, roll forward/back, publish crisp RCAs, and ship the change that prevents a replay.
Limited rollups support: stand up/upgrade sequencers & batchers, wire DA, manage contracts and params, and keep L1/L2 bridges healthy.
Build signal, not noise: SLOs/error budgets, useful dashboards, alerts that only page when users hurt.
Code where it counts: write/extend tools (snapshots, replay/load, state sync checks); occasionally patch client bugs that bite production and upstream when it’s worth it.
Why This Role Stands Out
Impact: your work lights up production; chain launches, reliability wins, performance gains.
Pace > Ceremony: tight reviews, practical standards, minimal meetings, async‑first.
Growth: own big surfaces; learn protocol internals while leveling up distributed systems chops.
Remote‑first: follow‑the‑sun coverage; humane on‑call.
Compensation & benefits: region‑aligned, bonus-eligible and shared early; no bait‑and‑switch.
The Bar: Signals We Care About
Production validator ownership: you’ve run at least one of:
Ethereum/EVM: consensus + execution clients in prod (e.g., Prysm/Lighthouse/Teku + Geth/Erigon/Nethermind/Besu), MEV‑Boost/relays, slashing‑protection DB hygiene.
Cosmos/CometBFT: validators + sentries, TMKMS/Horcrux remote signing, upgrades/param changes, governance tooling.
Solana: Agave/Jito, QUIC tuning, accounts/runtime tuning, high‑throughput storage/I/O decisions.
Key security in practice: Vault/HSM/KMS, rotation/backup/restore tested; principle of least privilege enforced.
SLO thinking with results: before/after on alert noise, latency, attestation/commit rates, missed slots, MTTR.
Tooling & code impact: Go/Python utilities (snapshot/restore, state‑sync verification, health checks), plus IaC modules you’d reuse.
K8s & systems trade‑offs: requests/limits/QoS that match workloads; storage & kernel tuning; P2P conn limits/peer selection; change windows/canaries/rollbacks that are real.
Targeted rollups experience: shipped/operated OP Stack/Arbitrum/Polygon CDK components and integrated DA endpoints safely.
Process
Steps may vary slightly by region/seniority; we keep it lean.
30‑min intro + context with Talent Acquisition
60‑min technical deep dive with the Hiring Manager (incidents you led, K8s/IaC trade‑offs, tooling you built)
60‑min hands‑on with Team Members (pair on a small plan/code review or targeted tool fix)
30-min meet a Founder
New Hire Onboarding: Week 1-12
Week 1–2: ship an IaC change + a validator dashboard/alert revamp someone actually uses; verify slashing‑DB backups and restores.
Week 3–6: own 2–3 validator fleets (plus a small rollup stack), kill a flaky alert, and remove a recurring papercut with code/automation.
Week 7–12: lead one upgrade/cutover (validator client or rollup component) with clean rollback; publish an RCA and the platform change that prevents recurrence.
The Quicknode compensation philosophy includes pillars to ensure fair and unbiased compensation for all employees. To design and deliver total reward offerings that are employee-centric. To offer a competitive benefit package in all locations where we operate. To prioritize attracting and retaining the best talent globally. To maintain a high-performing and flexible way of working.
During the hiring process, we are committed to discussing compensation openly and honestly. We encourage candidates to share their salary expectations and requirements early, allowing for an individualized discussion. We know that our total rewards practices impact the lives and wellbeing of our employees. Therefore, we will never stop learning about the market, our business, your needs, and how best to achieve our goals through thoughtful and data-driven practices. If you have any questions or require further information about the compensation for this position, please don't hesitate to reach out to your Recruiter.
We at Quicknode are an equal opportunity employer and all qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity or expression, pregnancy, age, national origin, disability status, genetic information, protected veteran status, or any other characteristic protected by law.