Staff Software Engineer

Summary

Join Grafana Labs, a remote-first, open-source company, as a Senior Engineer in GenAI & ML Evaluation Frameworks. You will design and implement robust evaluation frameworks for GenAI and LLM-based systems, develop tooling for automated evaluation, define and refine evaluation metrics, and lead dataset management processes. This role requires experience designing and implementing evaluation frameworks for AI/ML systems, familiarity with prompt engineering and LLM systems, and high autonomy to collaborate and translate team goals into testable criteria. The position offers a competitive salary (CAD 153,729 - CAD 184,475), RSUs, and a 100% remote global culture. Grafana Labs values transparency, autonomy, and trust, offering career growth pathways and a supportive work environment. The position is open to applicants from Canada time zones only.

Requirements

• Experience designing and implementing evaluation frameworks for AI/ML systems
• Familiarity with prompt engineering, structured output evaluation, and context-window management in LLM systems
• High autonomy to collaborate and translate team goals into clear, testable criteria supported by effective tooling

Responsibilities

• Design and implement robust evaluation frameworks for GenAI and LLM-based systems, including golden test sets, regression tracking, LLM-as-judge methods, and structured output verification
• Develop tooling to enable automated, low-friction evaluation of model outputs, prompts, and agent behaviors
• Define and refine metrics for both structure and semantics, ensuring alignment with realistic use cases and operational constraints
• Lead the development of dataset management processes and guide teams across Grafana in best practices for GenAI evaluation

Preferred Qualifications

• Experience working in environments with rapid iteration and experimental development
• A pragmatic mindset that values reproducibility, developer experience, and thoughtful trade-offs when scaling GenAI systems
• A passion for minimizing human toil and building AI systems that actively support engineers

Benefits

• In Canada, the Base compensation range for this role is CAD 153,729 - CAD 184,475
• All of our roles include Restricted Stock Units (RSUs), giving every team member ownership in Grafana Labs' success
• 100% Remote, Global Culture - As a remote-only company, we bring together talent from around the world, united by a culture of collaboration and shared purpose
• Scaling Organization – Tackle meaningful work in a high-growth, ever-evolving environment
• Transparent Communication – Expect open decision-making and regular company-wide updates
• Innovation-Driven – Autonomy and support to ship great work and try new things
• Open Source Roots – Built on community-driven values that shape how we work
• Empowered Teams – High trust, low ego culture that values outcomes over optics
• Career Growth Pathways – Defined opportunities to grow and develop your career
• Approachable Leadership – Transparent execs who are involved, visible, and human
• Passionate People – Join a team of smart, supportive folks who care deeply about what they do
• In-Person onboarding - We want you to thrive from day 1 with your fellow new ‘Grafanistas’ to learn all about what we do and how we do it
• Balance is Key - We operate a global annual leave policy of 30 days per annum. 3 days of your annual leave entitlement are reserved for Grafana Shutdown Days to allow the team to really disconnect