Kuber­netes Al­tern­at­ives

Kuber­netes is the market leader in dis­trib­uted cluster or­ches­trat­ors. In­ter­est­ingly, a number of al­tern­at­ives now exist to give Kuber­netes a run for its money. Each has its own specific ad­vant­ages and dis­ad­vant­ages, as well as different usage options: from a simple local cluster for your own de­vel­op­ment en­vir­on­ment to a dis­trib­uted system with tens of thousands of nodes. We’ll present the best solutions here.

Following the triumph of Docker as a platform for container vir­tu­al­isa­tion, Kuber­netes es­tab­lished itself as the standard tool for container or­ches­tra­tion. However, Kuber­netes’ features go far beyond managing groups of con­tain­ers. Kuber­netes automates de­ploy­ment, man­age­ment and scaling of con­tain­er­ised ap­plic­a­tions. Ap­plic­a­tions managed with Kuber­netes are scalable and pre­dict­able. Kuber­netes supports the following features:

• Automated rollout and rollback of changes
• Service discovery within the network
• Mass storage or­ches­tra­tion
• Scaling of ap­plic­a­tions and services
• Batch pro­cessing of data

Not all Kuber­netes al­tern­at­ives cover the entire spectrum of Kuber­netes’ features, however many of the al­tern­at­ives offer features beyond the scope of Kuber­netes. The most popular ones integrate con­veni­ent features related to user man­age­ment, mon­it­or­ing, and security. Some of the al­tern­at­ives use Kuber­netes behind the scenes, while others can be used in con­junc­tion with Kuber­netes. Let’s first get an overview of the most important aspects of Kuber­netes and its al­tern­at­ives.

Different ap­proaches exist for operating clusters and con­tain­ers. The de­ploy­ment para­met­ers describe how resources are allocated and managed. They also determine the en­vir­on­ments in which the systems can be set up and the types of ap­plic­a­tions that can be used:

To operate a cluster of con­tain­ers, it is not enough to just manage them. A cluster is only fully func­tion­al and useful if other tasks are performed in addition to the man­age­ment:

Kuber­netes and its al­tern­at­ives are software for managing dis­trib­uted clusters. There are different ap­proaches to structure this kind of system:

There are now several systems on the market that provide a service com­par­able to Kuber­netes. We’ve dis­tin­guished between three large classes:

Fur­ther­more, there are Kuber­netes al­tern­at­ives such as Apache Mesos and Azk, which do not fit into any of the three classes. We’ve taken a look at the following systems and compared them with Kuber­netes:

• Docker Swarm
• Apache Mesos
• AWS Fargate
• OpenShift
• Rancher
• Nomad
• Azk

First, let’s start with a brief overview of Kuber­netes.

Kuber­netes enables the execution and or­ches­tra­tion of contained ap­plic­a­tions. The feature set includes cluster man­age­ment, schedul­ing, service discovery, mon­it­or­ing, and secret man­age­ment. Kuber­netes can be used to implement clusters of up to 5,000 nodes and 300,000 con­tain­ers. Ori­gin­ally developed by Google, the software is now under the umbrellas of the Cloud Native Computing Found­a­tion (CNCF). In addition to Google, industry giants such as Red Hat, Docker, and IBM are involved in its ongoing further de­vel­op­ment.

As an industry standard, Kuber­netes im­ple­ment­a­tions can be found in the public cloud in­fra­struc­tures of major providers. New, useful de­vel­op­ments and tools such as ‘minikube’, ‘kubeadm’ and ‘K3s’ are con­stantly being added. These provide con­veni­ent functions, simplify the in­stall­a­tion and ad­min­is­tra­tion of Kuber­netes and make the software possible on weaker hardware.

The biggest dis­ad­vant­age of Kuber­netes is the com­plex­ity of the software. A complete Kuber­netes in­stall­a­tion involves half a dozen in­ter­act­ing services. Setup requires some effort and the running software is resource-consuming.

Kuber­netes is used for big data and machine learning projects that require the Kuber­netes ecosystem or Helm charts. Due to the com­plex­ity of the software, Kuber­netes is best suited for larger projects with a multi-year duration and a generous budget. If an in-house in­stall­a­tion is operated, a full-time team should be available for ad­min­is­tra­tion. Al­tern­at­ively, managed Kuber­netes in a public cloud in­fra­struc­ture works for many busi­nesses.

Docker Swarm is Docker’s own tool for or­ches­trat­ing clusters that consist of Docker engines. A Swarm cluster consists of Swarm manager nodes, which or­ches­trate and manage the cluster, and worker nodes, which are given jobs to process by the manager nodes.

The biggest advantage of Docker Swarm is the tight in­teg­ra­tion of the software into the Docker ecosystem. Once the Docker Engine is installed, you can start using Docker Swarm right away. For control, you’ll fall back on the familiar Docker command line interface. Docker Swarm interacts with the func­tion­al­ity of Docker and Docker Compose, using its own Swarm API. The software supports scaling, load balancing, service discovery, and multi-host net­work­ing.

In direct com­par­is­on with Kuber­netes, Docker Swarm offers fewer options for cus­tom­isz­a­tions and ex­ten­sions. In general, the range of functions is smaller.

Docker Swarm is par­tic­u­larly well suited to handle smaller workloads. If an existing Docker en­vir­on­ment exists, it can be easily leveraged with Docker Swarm to or­ches­trate container clusters.

Apache Mesos is based on the same prin­ciples as the Linux kernel. However, the two are im­ple­men­ted at a different level of ab­strac­tion. The Mesos kernel runs on every machine in the cluster and provides APIs for ap­plic­a­tions such as Hadoop, Spark, Kafka, and Elast­ic­search. Mesos allows resource man­age­ment and schedul­ing to be dis­trib­uted across data centres and cloud en­vir­on­ments.

Apache Mesos abstracts computing resources such as CPU, memory, and storage away from the actual machines. This allows clusters to scale to tens of thousands of nodes. Fault-tolerant, redundant manager and agent nodes are used, which are supplied with updates without in­ter­rup­tion if required. Cloud-native and Legay ap­plic­a­tions can run side-by-side within a cluster. The software runs on Linux, macOS, and Windows and includes a web interface for managing clusters and con­tain­ers.

Mesos is designed for systems with many clusters and high scalab­il­ity. The software operates at a lower level of ab­strac­tion than Kuber­netes. Thus, it is not as suited for small clusters with less than a dozen nodes.

Apache Mesos can be used to build fault-tolerant and elast­ic­ally dis­trib­uted systems. The software is par­tic­u­larly suitable for large systems that are designed for maximum re­dund­ancy.

AWS Fargate is a ‘server­less’ engine for con­tain­ers and follows the container-as-a-service model. When using Fargate, the focus is on the ap­plic­a­tion, rather than the in­fra­struc­ture. Fargate takes care of the man­age­ment of the required resources. The software uses the sister tech­no­lo­gies Amazon Elastic Container Service (ECS) and Amazon Elastic Kuber­netes Service (EKS).

The biggest advantage of AWS Fargate is its strong focus on the ap­plic­a­tion layer. Since you don’t have to worry about pro­vi­sion­ing the in­fra­struc­ture, you don’t have the overhead as­so­ci­ated with ad­min­is­ter­ing and scaling servers. Fargate scales the required resources elast­ic­ally, so that no over­pro­vi­sion­ing of servers for peak loads is needed. That way, you only pay for the computing capacity you actually use. Fargate stands out with integral ap­plic­a­tion isolation for enhanced security.

AWS Fargate makes the use of Kuber­netes more com­fort­able. However, this may also result in higher costs. Fur­ther­more, the shielding of the user from the un­der­ly­ing tech­no­logy means that there are fewer pos­sib­il­it­ies for control and influence.

AWS Fargate can be used to implement ap­plic­a­tions, APIs, mi­croservices ar­chi­tec­tures, and data batch pro­cessing. Fargate runs con­tain­er­ised ap­plic­a­tions and scales them on demand. The software is also used for ar­ti­fi­cial in­tel­li­gence (AI) and machine learning (ML) training.

The OpenShift platform from vendor Red Hat uses Kuber­netes behind the scenes. OpenShift allows or­ches­tra­tion of con­tain­ers and networks and enables schedul­ing and man­age­ment of dis­trib­uted, scalable resources. The software can be rolled out across multiple cloud en­vir­on­ments. This includes private clouds on its own compute in­fra­struc­ture, as well as Amazon AWS and Microsoft Azure cloud in­fra­struc­tures.

OpenShift has a web interface on board to control func­tion­al­ity in addition to the ob­lig­at­ory command line tools. Following the GitOps model, de­velopers are required to keep all con­fig­ur­a­tion as code. Despite a similar name, OpenShift is not related to OpenStack.

The biggest advantage of OpenShift is that it provides de­vel­op­ment teams with a con­sist­ent and powerful working en­vir­on­ment. Re­gard­less of the en­vir­on­ment used - public, private, or hybrid cloud - de­velopers have the same ap­proaches to building, testing, and running ap­plic­a­tions. This includes in par­tic­u­lar the ‘OpenShift Pipelines’ a solution for con­tinu­ous de­vel­op­ment.

Another strength of OpenShift is its built-in mon­it­or­ing and logging tools. These include Pro­meth­eus for managing mon­it­or­ing and no­ti­fic­a­tions, Jaeger for trans­ac­tion tracking, Kali for in­ter­faces and visu­al­isa­tion, and Istion for im­ple­ment­ing dis­trib­uted mi­croservice ar­chi­tec­tures. Fur­ther­more, OpenShift accesses the ‘Red Hat Container Catalog’, which contains container images tested and certified by Red Hat.

Like many products from Red Hat, OpenShift relies on the com­mer­cial Linux dis­tri­bu­tion ‘Red Hat En­ter­prise Linux’ (RHEL). This de­pend­ency results in a certain Vendor Lock-In.

OpenShift is par­tic­u­larly suitable for im­ple­ment­ing hybrid cloud strategies and building and scaling the con­tain­er­ised ap­plic­a­tions required for this. Or­gan­isa­tions that are already part of the Red Hat ecosystem will benefit from using the software.

Rancher provides another al­tern­at­ive based on Kuber­netes. The tool is a complete software stack for teams that rely on con­tain­ers. It addresses the op­er­a­tion­al and security chal­lenges of running multiple Kuber­netes clusters. Rancher provides DevOps teams with a cent­ral­ised control interface and unifies the man­age­ment of all Kuber­netes clusters in an or­gan­isa­tion. Cluster pro­vi­sion­ing, upgrades, and user and policy man­age­ment are included in the feature set.

Rancher sim­pli­fies the de­ploy­ment of Kuber­netes clusters on physical machines, as well as in public and private cloud en­vir­on­ments. Clusters are protected using global security policies; cent­ral­ised systems are used for au­then­tic­a­tion, access control and trace­ab­il­ity. The software prevents cluster failures and performs rescue and recovery op­er­a­tions as needed. Rancher works with various Linux and Kuber­netes dis­tri­bu­tions, pro­tect­ing against vendor lock-in. In terms of con­veni­ence features, Active Directory, LDAP and SAML support, as well as DNS pro­vi­sion­ing are included.

In contrast to Kuber­netes, the community around Rancher is con­sid­er­ably smaller. Fur­ther­more, the software only runs on Linux.

Rancher is par­tic­u­larly well suited for managing multiple Kuber­netes clusters with a cent­ral­ised interface. User rights can be assigned across multiple clusters.

With Nomad an in­ter­est­ing tool for cluster man­age­ment and schedul­ing is available from HashiCorp. The software can be used in con­junc­tion with Kuber­netes as well as a stan­dalone al­tern­at­ive. Nomad joins the company’s existing range of cloud and pro­vi­sion­ing software. Following the Unix philo­sophy, HashiCorp prefers a modular approach, with each tool covering a narrowly defined set of tasks. These include Terraform for in­fra­struc­ture-as-code (IaC), Consul for service discovery, and Vault for secrets man­age­ment.

In addition to the usual con­tain­er­ised ap­plic­a­tions, Nomad also supports vir­tu­al­ised as well as tra­di­tion­al ‘stan­dalone’ ap­plic­a­tions. Unlike Kuber­netes, Nomad does not consist of multiple, in­ter­act­ing system com­pon­ents. Rather, the software exists as a single, light­weight binary. This is deployed in local de­vel­op­ment en­vir­on­ments, cloud en­vir­on­ments, and private or public data centres.

Re­gard­less of the deployed en­vir­on­ment, Nomad provides the same op­er­a­tion­al con­veni­ence. It can be used to deploy clusters of tens of thousands of nodes across multiple avail­ab­il­ity zones, regions, and data centres. Multi-cluster de­ploy­ments are also included in the feature set.

Since Nomad is a re­l­at­ively new de­vel­op­ment, there is a much weaker ecosystem in contrast to Kuber­netes. As a result, there are no Managed Nomad solutions from the major cloud providers.

Nomad is par­tic­u­larly well suited for small and medium-sized teams that lack the capacity to maintain a complex or­ches­trat­or. Nomad is also ideal when running a mix of con­tain­er­ised and non-con­tain­er­ised (Windows, Java) ap­plic­a­tions. The sim­pli­city of the software allows teams to move quickly and respond to changing business needs.

With Azk, de­velopers have an exciting Kuber­netes al­tern­at­ive at their disposal. The tool installs locally and allows or­ches­tra­tion of de­vel­op­ment en­vir­on­ments on your own machine. Azk provides a command-line interface; for each project, a con­fig­ur­a­tion file (called Azkfile.js) is created that defines the de­vel­op­ment en­vir­on­ment. This includes Docker container images, network and mass storage settings, and en­vir­on­ment variables.

The biggest advantage of Azk is to make it easier for de­velopers to set up local de­vel­op­ment en­vir­on­ments. In addition to a central re­pos­it­ory that contains images of the most used systems, the Azkfile file format is the main con­trib­ut­or to a good user ex­per­i­ence. An Azkfile is written in an easy-to-un­der­stand JavaS­cript dialect and acts as a recipe for a local de­vel­op­ment en­vir­on­ment. Fur­ther­more, Azk offers the pos­sib­il­ity to auto­mat­ic­ally generate an Azkfile based on some spe­cific­a­tions.

Since Azk is mainly aimed at pro­fes­sion­al web de­velopers, there is no support for Windows so far.

Azk is mainly suited to automate the in­stall­a­tion and or­ches­tra­tion of de­vel­op­ment en­vir­on­ments. This makes it possible to define con­sist­ent en­vir­on­ments for different projects within a team.