Kubernetes - 10. Designing for High Availability and Scalability

In this 10th chapter, we cover the following:

- Introduction to High Availability
- High Availability Best Practices
- Multi-region setup
- Security Best Practices
- HA setup for hosted Kubernetes PaaS
- Cluster lifecycle events
- How to use Admission Controllers
- Introduction to the Workload API
- What are Custom Resource Definitions (CRDs)?

High Availability

In the industry, High Availability (HA) refers to a very high level of availability, often referred to as "five nines" availability (99.999%).

Basically, availability is calculated as follows:

Availability (%) = (Uptime / (Uptime + Downtime)) x 100

Availability for uptime is calculated using the following formulas:

MTBF (Mean Time Between Failures) = Value of 1 year in hours / Number of failures in 1 year
MTTR (Mean Time To Repair) = (Number of failures x System repair time) / Total number of failures
Uptime Availability = MTBF / (MTTR + MTBF)
Annual Downtime (per hour) = (1 - Uptime ratio) x 365 x 24

The guaranteed availability levels for Service Level Agreements (SLA) are as follows:

1. If availability is 99.9%, downtime is 8 hours, 45 minutes, 57.0 seconds per year.

2. If availability is 99.99%, downtime is 52 minutes, 35.7 seconds per year.

3. If availability is 99.999%, downtime is 5 minutes, 15.6 seconds per year.

To guarantee "five nines" availability, you must operate your Kubernetes cluster very tightly.

HA Best Practices

To build a Kubernetes system that guarantees high availability, keep in mind that "availability is often as much about people and processes as it is about technical errors."

First, there is a term you should know: the concept of graceful degradation.

Graceful degradation is the concept of building functionality by distributing it across multiple layers and modules. Even if a critical error occurs in a part of the system, it continues to provide a certain level of availability.

There are two ways to handle graceful degradation in Kubernetes:

Infrastructure Degradation : This degradation method relies on complex algorithms and software to handle unexpected hardware or VM errors. We will explore how to secure high availability for the essential Kubernetes components needed to provide this degradation method.

Application Degradation : While this is largely dependent on the aforementioned Microservices (MS) best practice strategies, there are several patterns to ensure user success.

You should use core Kubernetes strategies to isolate underlying infrastructure failures, while building caching, failover, and rollback mechanisms for application failures, and ensuring high availability for Kubernetes components.

Antifragility

In simple terms, 'antifragility' is the property where performance actually increases in the face of external chaos or pressure.

To cope with the complexity of Kubernetes systems and maintain systems using large-scale Kubernetes, you need to know a few key concepts.

1. Redundancy
2. Triggering failure scenarios, then responding to, analyzing, exploring, and improving them. (Netflix's Chaos Monkey is a standard, well-organized approach for testing complex system stability: https://github.com/Netflix/chaosmonkey)
3. Introducing appropriate patterns into the system. (Retry, load balancing, circuit breakers, timeouts, health checks, and concurrent connection checks are key patterns for antifragility. At a higher level, there is service mesh, such as Istio: https://techcafe.tistory.com/133)

HA Approaches for Kubernetes

Kubernetes HA configurations include the "stacked master" approach, which combines etcd and control plane nodes, and the approach where etcd and control plane nodes are separated.

Installation of Kubernetes is omitted.

Cluster Lifecycle

Let's learn how to extend the cluster using Admission Controllers, Workloads, and CRDs.

Admission Controllers

Admission controllers can intercept calls to the Kubernetes API server after authentication and authorization are complete.

The following two admission controllers are particularly important:

MutatingAdmissionWebhook is executed only when the cluster is in the mutation phase, and it calls webhooks that modify requests sequentially. Use this controller when you want to customize approval logic for operations such as CREATE, DELETE, or UPDATE to inject business logic into the cluster. You can perform tasks such as automating storage provisioning using StorageClass.

ValidatingAdmissionWebhook is executed by the admission controller during the validation phase. It calls webhooks that verify "request validity," such as a webhook that validates quota increases. Keep in mind that any webhook called by this controller cannot modify the original object.

Workload API

In the early days of Kubernetes, pods and workloads were tightly coupled with containers, sharing CPU, networking, storage, and lifecycle events. Kubernetes introduced concepts such as replication, deployment, and labels to manage the 12-factor app methodology for cloud applications, and introduced StatefulSets to help Kubernetes operators handle stateful workloads.

Over time, Kubernetes workload concepts have been divided into several types:

Pod
ReplicationController
ReplicaSet
Deployment
DaemonSet
StatefulSet

These various elements are the result of Kubernetes rationally adjusting workload types, but unfortunately, the API was distributed throughout various parts of the Kubernetes codebase. After months of effort, including sacrificing some backward compatibility, we were able to consolidate all code into the apps/v1 API.

Important decisions made during the consolidation process are as follows:

Default Selector : If a label selector is not specified, it defaults to a selector automatically generated from the template labels.
Immutable Selector : While changing selectors can be useful for Deployments, mutating selectors is contrary to Kubernetes recommendations. It has been changed to a method where Kubernetes orchestrates canary deployments and swaps out pod labels.
Rolling Update : Rolling updates have become the default at the request of Kubernetes programmers.
Garbage Collection: In version 1.9 and apps/v1, garbage collection is more aggressive. If you delete a DaemonSet, ReplicaSet, StatefulSet, or Deployment, the associated pods are also deleted.

Custom Resource Definitions

Custom resources extend the Kubernetes API and complement admission controllers. You can use custom resources to improve running Kubernetes clusters.

You can apply features such as: