RBC has taken a holistic approach on its cloud journey, as it enables technologies across our enterprise. Our RBC Capital Markets business is constantly evolving our technology stack to meet the needs of our clients. One of the key enablers of this evolution has been our adoption of Kubernetes for application deployment.

The need for Kubernetes in capital markets

Capital markets operate in a highly dynamic environment where milliseconds can make a difference. Our applications need to be highly available, scalable, and resilient to failures. Kubernetes, with its capabilities, is an industry standard for managing containerized applications at scale. It allows us to:

  • Scale applications dynamically based on market demand
  • Ensure high availability with self-healing capabilities
  • Roll out updates seamlessly with zero downtime
  • Optimize resource utilization across our infrastructure

However, managing Kubernetes at scale, especially in a multi-tenant environment like ours, comes with its own set of challenges. This is where FluxCD comes into play.

Introducing FluxCD: GitOps for Kubernetes

FluxCD is a GitOps tool that helps us automate the deployment and management of applications on Kubernetes. It operates on the principle of declarative infrastructure as code, where the desired state of our applications and infrastructure is defined in Open Container Initiative (OCI) repositories. FluxCD continuously monitors these repositories and ensures that the actual state of the cluster matches the desired state. It allows us to:

  • Automate deployments across multiple environments
  • Enforce consistency by using OCI as the single source of truth
  • Simplify rollbacks by reverting to previous OCI version
  • Enable self-service for tenants while maintaining strict governance

Our deployment architecture

RBC Capital Markets operates across multiple teams (e.g., trading desks, risk management teams, equities application team),  each of which deploys and manages its own applications on our Kubernetes platform. Here’s how we’ve structured our deployment pipeline using FluxCD:

1. Platform deployment

The platform layer includes all the shared services and infrastructure components that are required to run our applications. This includes:

  • Ingress controllers for routing traffic
  • Monitoring and logging tools like Prometheus, Loki and Grafana
  • Cert management and network configurations
  • Custom resource definitions (CRDs) for tenant-specific configurations

We use a dedicated OCI repository to define the desired state of our platform. FluxCD monitors this repository and applies any changes to the Kubernetes cluster automatically. This ensures that our platform is always up-to-date and consistent across all environments.

2. Tenant application deployment

Each tenant has its own OCI repository where they define the desired state of their applications. These repositories include:

  • Kubernetes manifests (Deployments, Services, ConfigMaps, etc.)
  • Helm charts for packaging applications
  • Environment-specific configurations (e.g., dev, qa, production)

FluxCD is configured to monitor these tenant repositories and deploy applications to their respective namespaces in the Kubernetes cluster. This allows tenants to manage their applications independently while adhering to the platform’s governance policies.

3. Multi-tenant isolation

To ensure isolation between tenants, we use Kubernetes namespaces and role-based access control (RBAC). Each tenant is assigned a dedicated namespace where they can deploy and manage their applications. FluxCD enforces RBAC policies to ensure that tenants can only modify resources within their own namespaces.

4. Continuous delivery pipeline

FluxCD integrates seamlessly with our CI/CD (Continuous Integration/Continuous Dployment) pipeline. When a tenant pushes changes to their Git repository, FluxCD automatically picks up the changes and deploys them to the appropriate environment. This enables us to achieve continuous delivery with minimal manual intervention.

Benefits of using FluxCD at RBC Capital Markets

The improvements we have seen in our deployment process are:

  • Faster time-to-market: Automated deployments reduce the time it takes to roll out new features and updates.
  • Improved reliability: GitOps ensures that our deployments are consistent and reproducible, reducing the risk of errors.
  • Enhanced collaboration: Tenants can manage their applications independently, while platform teams focus on maintaining the underlying infrastructure.
  • Better governance: Git and OCI serves as an audit trail, allowing us to track changes and enforce compliance.

Challenges and lessons learned

While FluxCD has been a great fit for our use case, we’ve also encountered some challenges along the way:

  • Managing large Repositories: As the number of tenants and applications grow, managing large Git repositories can become cumbersome. We’ve addressed this by breaking down repositories into smaller, more manageable components.
  • Sensitive Information: Managing information requires careful consideration. We’ve integrated FluxCD and external management tools like HasiCorp Vault to improve processing of information.
  • Onboarding tenants: Educating tenants on GitOps practices and Kubernetes concepts has been an ongoing effort. We’ve invested in training and documentation to help tenants get up to speed.

Looking ahead

As we continue to scale our Kubernetes platform, we’re exploring new ways to enhance our deployment process. Some of the areas we’re focusing on include:

  • Advanced GitOps patterns: Implementing canary deployments, blue-green deployments, and other advanced deployment strategies.
  • Policy enforcement: Using tools like Open Policy Agent (OPA) to enforce policies across tenants.
  • Observability: Enhancing our monitoring and alerting capabilities to provide better visibility into the health of our applications.

In a highly competitive environment, we will continue to innovate and are excited to see how these technologies evolve.

This article was originally posted on the Cloud Native Computing Foundation website.

This article offers general information only and is not intended as legal, financial or other professional advice. A professional advisor should be consulted regarding your specific situation. While information presented is believed to be factual and current, its accuracy is not guaranteed and it should not be regarded as a complete analysis of the subject matter discussed. All expressions of opinion reflect the judgment of the author(s) as of the date of publication and are subject to change. No endorsement of any third parties or their advice, opinions, information, products or services is expressly given or implied by Royal Bank of Canada or its affiliates.