Principal Engineer, Agentic Evals and Guardrails

Job Description

What's the opportunity?

As Principal Engineer for Agentic Evals and Guardrails, you'll be the architect of trust and safety for RBC's agentic AI systems. In this groundbreaking role, you'll design and build the critical evaluation frameworks and guardrail systems that ensure our AI agents operate safely, reliably, and within defined boundaries-protecting the bank from rogue agent behavior while enabling innovation at scale.

This is a high-impact opportunity to establish the foundational infrastructure that will govern how autonomous AI agents behave at runtime across RBC. You'll create sophisticated monitoring, evaluation, and control systems that detect anomalies, enforce policies, and prevent agents from taking unsafe actions. Working at the intersection of AI safety, platform engineering, and enterprise governance, you'll set the standard for responsible agentic AI deployment in financial services. If you're passionate about AI safety, real-time systems, and building the guardrails that enable fearless innovation, this role offers unparalleled impact.

Your responsibilities include:

• Design Runtime Guardrail Systems: Architect and implement multi-layered control planes that monitor and govern AI agent behavior in real-time, preventing unauthorized actions and ensuring compliance with enterprise policies
• Build Comprehensive Evaluation Frameworks: Create robust evaluation systems (evals) that continuously assess agent performance, accuracy, safety, and alignment with intended objectives across diverse use cases
• Develop Safety Control Mechanisms: Design kill-switches, circuit breakers, budget caps, and access controls that enable rapid intervention when agents exhibit unexpected or unsafe behavior
• Implement Real-Time Monitoring: Build observability systems that provide continuous visibility into agent decision-making, tool invocations, and behavioral patterns to detect anomalies and deviations
• Create Policy Enforcement Infrastructure: Develop systems that enforce least-privilege access, data governance policies, ethical constraints, and regulatory compliance requirements at runtime
• Establish Agent Behavioral Standards: Define metrics, benchmarks, and KPIs for agent behavior, including hallucination detection, output quality assessment, and alignment verification
• Build Defense Against Adversarial Attacks: Implement guardrails that protect against prompt injection, jailbreaking, data leakage, and other security threats targeting agentic systems
• Lead Technical Innovation: Drive the adoption of cutting-edge AI safety techniques, including behavioral monitoring, constraint validation, and dynamic risk assessment
• Collaborate Across Platforms: Partner with GenAI, MLOps, and platform teams to integrate evaluation and guardrail capabilities seamlessly into the broader Lumina ecosystem
• Provide Technical Leadership: Mentor engineers and establish best practices for building safe, reliable, and trustworthy agentic AI systems
• Drive Governance Integration: Work with compliance, risk, and security teams to ensure guardrail systems align with regulatory requirements (e.g., NIST AI RMF, ISO/IEC 42001, financial services regulations)
• Enable Rapid Response: Create dashboards, alerting systems, and diagnostic tools that enable quick identification and resolution of agent failures

You're our ideal candidate if you have:

• Deep AI/ML Expertise: 10+ years of software engineering experience with significant focus on machine learning, AI systems, or LLM/GenAI applications
• Agentic AI Understanding: Strong knowledge of agentic AI architectures, autonomous systems, and the unique safety challenges they present
• Real-Time Systems Experience: Proven track record building low-latency monitoring, observability, and control systems that operate at scale
• AI Safety & Evaluation Knowledge: Experience with LLM evaluation frameworks, guardrail implementation, prompt engineering, and AI safety best practices
• Security & Compliance Background: Understanding of AI security threats (prompt injection, jailbreaking, data leakage) and regulatory compliance requirements in financial services
• Platform Engineering Skills: Proficiency with cloud platforms (Azure, AWS, GCP), container orchestration (Kubernetes), and infrastructure-as-code
• Observability & Monitoring: Experience with observability tools, telemetry systems, real-time analytics, and anomaly detection platforms
• Policy & Governance Systems: Ability to translate business policies and compliance requirements into technical controls and automated enforcement mechanisms
• Programming Excellence: Expert-level programming skills (Python, Go, or similar) with experience building production-grade systems
• MLOps & DevOps Practices: Strong understanding of CI/CD, model deployment, and operational best practices for AI systems
• Leadership Experience: Proven ability to lead technical initiatives, influence cross-functional teams, and drive alignment across multiple stakeholders
• Analytical & Problem-Solving Skills: Exceptional ability to reason about complex systems, identify edge cases, and design resilient solutions
• Communication Skills: Excellent ability to articulate complex AI safety concepts to technical and non-technical audiences, including risk and compliance teams

Nice to have

• Experience with AI governance frameworks (NIST AI RMF, ISO/IEC 42001, EU AI Act)
• Background in adversarial ML, red-teaming, or security testing
• Experience with workflow orchestration platforms (Temporal, Airflow)
• Knowledge of financial services regulations and risk management
• Contributions to AI safety research or open-source projects
• Experience with prompt engineering and LLM fine-tuning

What's in it for you?

• Become part of a team that thinks progressively and works collaboratively. We care about seeing each other reach full potential.
• A comprehensive Total Rewards Program including bonuses and flexible benefits, competitive compensation, commissions, and stock options where applicable.
• Leaders who support your development through coaching and managing opportunities.
• Ability to make a difference and lasting impact from a local-to-global scale.
• Pioneer AI safety in financial services and establish the standards for responsible agentic AI deployment at one of Canada's largest institutions.
• Work on cutting-edge challenges at the forefront of AI safety, governance, and autonomous systems.
• Shape the future of trustworthy AI and enable RBC to deploy agentic AI confidently and responsibly.

About RBC Borealis

RBC Borealis is the driving force behind Royal Bank of Canada's AI and data innovation. As part of Canada's largest financial institution, we bring together a team of architects, engineers, scientists, and product experts on a mission to revolutionize finance through world-class research, solutions, and a resilient data platform. With locations across Toronto, Waterloo, Montreal, Calgary, and Vancouver, we're at the forefront of AI research and platform development. With a focus on cutting-edge research in areas like time series forecasting, causal machine learning, and responsible AI, we are seamlessly integrating AI research and data engineering to solve critical challenges in the financial industry. We are building intelligent and scalable data-driven solutions that will help communities thrive and drive innovation for our customers across the bank.

Inclusion and Equal Opportunity Employment

RBC is an equal opportunity employer committed to diversity and inclusion. We are pleased to consider all qualified applicants for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, protected veterans status, Aboriginal/Native American status or any other legally-protected factors. Disability-related accommodations during the application process are available upon request.

#TECHPJ

#LI-POST

Job Skills

Active Learning, Agile Methodology, Decision Making, Emerging Technologies, Enterprise Application Delivery, Group Problem Solving, Long Term Planning, Programming Languages, Software Development Life Cycle (SDLC)

Additional Job Details

Address:

RBC WATERPARK PLACE, 88 QUEENS QUAY W:TORONTO

City:

Toronto

Country:

Canada

Work hours/week:

37.5

Employment Type:

Full time

Platform:

TECHNOLOGY AND OPERATIONS

Job Type:

Regular

Pay Type:

Salaried

Posted Date:

2026-03-13

Application Deadline:

2026-03-31

Note: Applications will be accepted until 11:59 PM on the day prior to the application deadline date above

Our Employment Opportunities

At RBC, we are guided by living shared values of Client First, Integrity, Collaboration, Respect and Excellence and winning together as One RBC. We believe an inclusive workplace that has diverse perspectives is core to our continued growth as one of the largest and most successful banks in the world. Maintaining a workplace where our employees feel supported to perform at their best, effectively collaborate, drive innovation, and grow professionally helps to bring our Purpose to life and create value for our clients and communities. RBC strives to deliver this through policies and programs intended to foster a workplace based on respect, belonging and opportunity for all.

Join our Talent Community

Stay in-the-know about great career opportunities at RBC. Sign up and get customized info on our latest jobs, career tips and Recruitment events that matter to you.

Expand your limits and create a new future together at RBC. Find out how we use our passion and drive to enhance the well-being of our clients and communities at jobs.rbc.com.

RBC is presently inviting candidates to apply for this existing vacancy. Applying to this posting allows you to express your interest in this current career opportunity at RBC. Qualified applicants may be contacted to review their resume in more detail.