Introduction
In the last 5-10 years, enabling technologies like Docker and Kubernetes and their respective open-source communities have driven the growth and maturity of cloud native development. The ease with which systems these days can be built to make use of cloud resources, and scale according to usage patterns, is incredibly empowering. This, in turn, has led to a boom in open-source software developed around open cloud native standards. Interfaces like the Container Runtime Interface (CRI), Container Network Interface (CNI), Container Storage Interface (CSI), and the Kubernetes API itself, allow developers to create complex applications agnostic to the underlying cloud vendor. And although the ecosystem and communities are maturing and big changes in the Kubernetes project are becoming sparse, the cloud native landscape is still vast and expanding.
Figure 1. Cloud native landscape. Image taken from https://landscape.cncf.io/.
Federated Cloud
As it has become relatively easy to create and manage Kubernetes clusters, organisations are growing them both in size and number. This phenomenon, known as kube-sprawl, is not just due to the relative effortlessness of cluster management. Often, organisations hit the scalability limits of a single cluster (at time of writing, Kubernetes recommends using no more than 5000 nodes in a cluster). Other times, they are forced to maintain multiple clusters due to organisational structure and security implications, geographical constraints (for example due to GDPR requirements), or explicit multi-cloud strategies to reduce vendor lock-in.
These are all solid reasons to want to move towards a multi-cluster setup. However, multi-cluster is also expensive, as compute resources often sit idle and clusters need to be managed separately.
Federated cloud is aimed at joining multiple clusters together to form a single interconnected cloud (at least, from a user's point of view), allowing users to share compute resources and move workloads around seamlessly. In this way, workload scheduling can be further optimized, and both data and applications can be migrated in a standardized cloud native way.
However, federated cloud also poses technical challenges that are not solved by Kubernetes itself (Kubernetes is currently scoped to the single cluster boundary, although with the advent of multi-cluster services that is slightly changing). The main ones are:
- Inter-cluster networking. The problem of establishing network connection between heterogeneous clusters (each with a different networking and cluster configuration) to allow inter-cluster Pod-to-Pod and Service-to-Pod routing.
- Multi-cluster deployment orchestration. The problem of providing a single control plane and resource model to deploy and manage your applications across clusters in a Kubernetes-native way.
- Multi-cluster data management. The problem of managing stateful workloads by providing data locality and robust native migration of different data sources and formats.
- Multi-cluster security models. The problem of dynamically restricting access to applications, data, and compute resources in a multi-tenant scenario to the agreed upon scope.
Multi-cluster topologies
There are different models for establishing a federated cloud. The simplest model is a hierarchical one, where a single cluster is used as a host cluster, and other clusters are able to join the federation. In this setup, the host cluster serves as the control plane for the federation. Applications are scheduled to the host cluster, which determines the best placement, and pushes the request to one of the joined clusters. A tiered waterfall structure can also be created where a downstream cluster in turn acts as a host cluster for other clusters and delegates any incoming deployment requests.
An alternative is a decentralized model, where no single host cluster is identified. Rather, peer-to-peer connections are established between clusters, and any cluster is able to delegate deployments to another. Note that this does not have to mean that clusters are fully interconnected, or that peering is always bi-directional. A cluster can request compute resources in another cluster, but does not necessarily have to accomodate for that itself. This model is significantly more flexible, as we can have clusters that are able to make use of another cluster's resources in an ad-hoc fashion, even though it might not be able to share resources itself. Moreover, we no longer have a single point of failure in the central host cluster. It also introduces additional complexity however, for example in the case of loops, additional communication overhead, and elaborate multi-cluster topologies.
Another way is a hybrid form, where part of the federated cloud is applying a centralized model, and other parts are set up in a decentralized manner.
Figure 2. Multi-cluster topologies. Images taken from https://admiralty.io/docs/concepts/topologies/.
Technologies
There are a number of projects within the cloud native landscape that tackle the above-mentioned challenges in the federated cloud space. Some focus on establishing inter-cluster network connection, some aim at creating a better deployment model and multi-cluster control plane, and some try to solve both.
Specific challenges in the multi-cloud networking domain are inter-cluster service discovery, inter-cluster service invocation, and inter-cluster network policies. There are solutions that are baked into the container network interface - for example Cilium ClusterMesh - or in a service mesh - for example Istio, that routes it traffic through sidecar proxies. Other solutions are deployed seperately. Submariner is a well-known project that provides cross-cluster L3 connectivity and multi-cluster service discovery through a broker architecture.
Considering workload orchestration, Kubefed was a Kubernetes subproject aimed at providing a federated deployment model The downside was that it did not provide a way to work with Kubernetes-native resources, and that it only supported a centralized topology. It is now discontinued but has a successor in the novel Work API proposal. Other solutions support decentralized (peer-to-peer) topologies, such as Admiralty and Liqo. The Liqo paper goes into extensive detail about their approach.
GAIA-X
One special federated cloud project is the GAIA-X project. GAIA-X is a European initiative, aimed at establishing a secure and federated data infrastructure. Building an open federated cloud that "stands for European values", it should provide digital sovereignty to citizens. The idea is to create an alternative to the large US-based hyperscalers (Amazon, Google, Microsoft), by allowing European cloud providers to inter-connect their cloud infrastructure and data platforms through common standards. End-users would then be able to run their applications on top, without having to worry about which cloud their software would end up on. They would just provide their specification (i.e. performance and network constraints, geographical constraints, cost constraints, carbon emission constraints, etc.). It is still uncertain if the project will be a success, but the aim is admirable.
Figure 3. Liqo multi-cluster model. Liqo employs the Virtual Kubelet project to represent other clusters as a big node within a cluster, allowing for native scheduling. Liqo also sports a nifty data fabric feature that simplifies stateful workload orchestration.
Edge environments
Now that we have learned about federated cloud, we would like to start applying it to our systems. Specifically, it would be great to be able to apply these technologies to edge platforms such as UAVs. That way, we could seamlessly connect fleets of drones that can share compute capacity or machine learning models, for example for wildlife or wildfire surveillance, or to establish an ad-hoc network in the case of environmental disasters (e.g. the recent floodings in north-west Europe).
Unfortuntely, most of the aforementioned solutions assume a static and high-bandwidth environment. They are not developed for constrained and dynamic edge environments where we have clusters joining and leaving on the fly. More, the network can be flaky and drop completely at times. This is not an ideal environment for standard solutions. There are a few technologies that facilitate edge orchestration in resource-constrained environments (such as KubeEdge, FLEDGE, or KaiS), but those do not support multiple clusters.
Security
Hardening a single Kubernetes cluster is important, and the project has - especially in these last few years - made significant steps in improving its security methods (we could be close to effortly running core Kubernetes components rootless 🎉). The NSA has published a report with some sane default steps to improve cluster security.
Full cluster separation is obviously the safest, and a direction mamy companies are now taking (hence the kube-sprawl). However, what if we do want to share some of our resources and applications with others? Kubernetes has the concept of namespaces, but those provide only very weak isolation.
Allowing other tenants to run workloads on your own cluster introduces many additional security considerations. Tenants should be restricted to the resources they are granted, and not see any of your own compute resources, data, and applications, nor any of the other tenants'. Additionally, you should not be able to see the data and applications of the tenants using your cluster.
One project that allows multiple tenants to share a single cluster is the vcluster project by Loft. Tenant Pods end up on the same shared infrastructure, but tenants can only view and access their own virtual cluster (control plane included). Vcluster employs Pod Security Standards, as well as Network Policies and other measures such as enforcing rootless containers to secure tenant workloads.
Further ways of hardening involve employing container runtimes with an improved security model, such as Kata Containers, although adding additional security measures typically comes at a performance cost and does not necessarily reduce the attack surface. Container runtimes that advertise themselves as more secure, often expose a much wider interface to the host and invoke more calls to the host kernel. They also introduces a performance tradeoff that is especially important in resource-constrained edge environments.
Experimentation
To experiment with secure federated cloud technology in edge environments, we built a local test bed. Through a mix of Vagrant, libvirt, Terraform, and kubeadm, we could run multiple clusters on our local workstations. That sped up development significantly.
To simulate reduced network performance, we used Chaos Mesh, a chaos engineering tool that is able to inject faults into your system. These could be application or Kubernetes component failures, or network issues such as increased latency, packet loss, and reduced bandwidth. Under the hood it uses tc to apply these rules on your cluster hosts. At the time Chaos Mesh did not support multi-cluster functionality, so this was custom built. With this setup we could run specific edge scenario's in a reproducible manner.
We used Liqo as the federated orchestration solution, providing both a Wireguard-tunnel-based peer-to-peer network solution, as well as a simple Kubernetes-native deployment model.
Next, we needed to be able to react to changes in our cloud environment. The default Kubernetes scheduler does not take into account network conditions when deciding on wich node Pods should land. This severely impacts performance for Pods with specific network requirements requirements. We integrated Intel's Telemetry Aware Scheduling (TAS) solution to add custom scheduling rules that take into consideration cluster-to-cluster network latency, packet loss, and bandwidth. To report these metrics to the TAS scheduler, we need to observe them. This was done through OLSR, a routing protocol optimized for mobile ad-hoc networks that exposes network quality information.
Conclusion and Future work
Federated cloud technology is still in its infancy, but it has great potential to further push the interoperability and ubiquity of cloud computing, enabling seamless processing capabilities at the edges of our world, and perhaps beyond.
Its remaining challenges - including those in edge environments - are rapidly being addressed by both academia and industry, and it's exciting to see what's next. These are just a few examples of inspiring future directions:
- Fleet management: There are currently few projects that address the UI/UX aspect of multi-cluster, multi-device, and really multi-anything environments. Kubernetes is a great platform to build platforms, and more and more solutions are popping up to manage not just compute resources, but even physical objects. We really need a better standardized management interface for that purpose.
- Orbital Edge Computing: Federated cloud, moving towards robust edge application, is an ideal enabler for future low-earth-orbit (nano-)satellite communication. Specific mention to Denby et al., where I first saw the term.
- Sustainable Computing: Federated clouds, such as those envisioned by the GAIA-X project, would open the market and increase competition on environmentally oriented performance signals, such as carbon emission. It could allow end users to easily choose the most environmentally friendly cloud to run their workloads on. If you're interested in this, be sure to check out one of my other projects.
Further reading
Two amazing colleagues gave a presentation on the work we did on federated cloud while at TNO at KubeCon + CloudNativeCon Europe 2023. You can find it here.