Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications
Kubernetes is also referred to as k8s (pronounced Kate's), as there are 8 characters between k and s.
Kubernetes is highly inspired by the Google Borg system, a container and workload orchestrator for its global operations for more than a decade. It is an open source project written in the Go language and licensed under the Apache License, Version 2.0.
Kubernetes was started by Google and, with its v1.0 release in July 2015, Google donated it to the Cloud Native Computing Foundation (CNCF), one of the largest sub-foundations of the Linux Foundation.
New Kubernetes versions are released in 4 month cycles. The current stable version is 1.26 (as of December 2022).
Kubernetes Features
Kubernetes offers a very rich set of features for container orchestration. Some of its fully supported features are:
• Automatic bin packing
Kubernetes automatically schedules containers based on resource needs and constraints, to maximize utilization without sacrificing availability.
• Designed for extensibility
A Kubernetes cluster can be extended with new custom features without modifying the upstream source code.
• Self-healing
Kubernetes automatically replaces and reschedules containers from failed nodes. It terminates and then restarts containers that become unresponsive to health checks, based on existing rules/policy. It also prevents traffic from being routed to unresponsive containers.
• Horizontal scaling
With Kubernetes applications are scaled manually or automatically based on CPU or custom metrics utilization.
• Service discovery and load balancing
Containers receive IP addresses from Kubernetes, while it assigns a single Domain Name System (DNS) name to a set of containers to aid in load-balancing requests across the containers of the set.
Additional fully supported Kubernetes features are:
• Automated rollouts and rollbacks
Kubernetes seamlessly rolls out and rolls back application updates and configuration changes, constantly monitoring the application's health to prevent any downtime.
• Secret and configuration management
Kubernetes manages sensitive data and configuration details for an application separately from the container image, in order to avoid a rebuild of the respective image. Secrets consist of sensitive/confidential information passed to the application without revealing the sensitive content to the stack configuration, like on GitHub.
• Storage orchestration
Kubernetes automatically mounts software-defined storage (SDS) solutions to containers from local storage, external cloud providers, distributed storage, or network storage systems.
• Batch execution
Kubernetes supports batch execution, long-running jobs, and replaces failed containers.
• IPv4/IPv6 dual-stack
Kubernetes supports both IPv4 and IPv6 addresses.
There are many additional features currently in alpha or beta phase. They will add great value to any Kubernetes deployment once they become stable features. For example, support for role-based access control (RBAC) is stable only as of the Kubernetes 1.8 release.
Why Use Kubernetes?
Another one of Kubernetes' strengths is portability. It can be deployed in many environments such as local or remote Virtual Machines, bare metal, or in public/private/hybrid/multi-cloud setups.
Popular graduated projects of CNCF(cloud netwoking computing foundation):
• Kubernetes container orchestrator
• etcd distributed key-value store
• CoreDNS DNS server
• containerd container runtime
• Envoy cloud native proxy
• Fluentd for unified logging
• Harbor registry
• Helm package management for Kubernetes
• Linkerd service mesh for Kubernetes
• Open Policy Agent policy engine
• Prometheus monitoring system and time series DB
• Rook cloud native storage orchestrator for Kubernetes
Kubernetes Architecture
In this chapter, we will explore the Kubernetes architecture, the components of a control plane node, the role of the worker nodes, the cluster state management with etcd and the network setup requirements. We will also learn about the Container Network Interface (CNI), as Kubernetes' network specification
At a very high level, Kubernetes is a cluster of compute systems categorized by their distinct roles:
• One or more control plane nodes
One or more worker nodes (optional, but recommended)
Control Plane Node Overview
The control plane node provides a running environment for the control plane agents responsible for managing the state of a Kubernetes cluster, and it is the brain behind all operations inside the cluster. The control plane components are agents with very distinct roles in the cluster's management. In order to communicate with the Kubernetes cluster, users send requests to the control plane via a Command Line Interface (CLI) tool, a Web User-Interface (Web UI) Dashboard, or an Application Programming Interface (API).
It is important to keep the control plane running at all costs. Losing the control plane may introduce downtime, causing service disruption to clients, with possible loss of business. To ensure the control plane's fault tolerance, control plane node replicas can be added to the cluster, configured in High-Availability (HA) mode. While only one of the control plane nodes is dedicated to actively managing the cluster, the control plane components stay in sync across the control plane node replicas. This type of configuration adds resiliency to the cluster's control plane, should the active control plane node fail.
To persist the Kubernetes cluster's state, all cluster configuration data is saved to a distributed key-value store which only holds cluster state related data, no client workload generated data. The key-value store may be configured on the control plane node (stacked topology), or on its dedicated host (external topology) to help reduce the chances of data store loss by decoupling it from the other control plane agents
A control plane node runs the following essential control plane components and agents:
• API Server
• Scheduler
• Controller Managers
• Key-Value Data Store.
In addition, the control plane node runs:
• Container Runtime
• Node Agent
• Proxy
• Optional addons for cluster-level monitoring and logging.
Control Plane Node Components: API Server
All the administrative tasks are coordinated by the kube-apiserver, a central control plane component running on the control plane node. The API Server intercepts RESTful calls from users, administrators, developers, operators and external agents, then validates and processes them. During processing the API Server reads the Kubernetes cluster's current state from the key-value store, and after a call's execution, the resulting state of the Kubernetes cluster is saved in the key-value store for persistence. The API Server is the only control plane component to talk to the key-value store, both to read from and to save Kubernetes cluster state information - acting as a middle interface for any other control plane agent inquiring about the cluster's state.
The API Server is highly configurable and customizable. It can scale horizontally, but it also supports the addition of custom secondary API Servers, a configuration that transforms the primary API Server into a proxy to all secondary, custom API Servers, routing all incoming RESTful calls to them based on custom defined rules.
Control Plane Node Components: Scheduler
The role of the kube-scheduler is to assign new workload objects, such as pods encapsulating containers, to nodes - typically worker nodes. During the scheduling process, decisions are made based on current Kubernetes cluster state and new workload object's requirements. The scheduler obtains from the key-value store, via the API Server, resource usage data for each worker node in the cluster. The scheduler also receives from the API Server the new workload object's requirements which are part of its configuration data.
Control Plane Node Components: Controller Managers
The controller managers are components of the control plane node running controllers or operator processes to regulate the state of the Kubernetes cluster. Controllers are watch-loop processes continuously running and comparing the cluster's desired state (provided by objects' configuration data) with its current state (obtained from the key-value store via the API Server). In case of a mismatch, corrective action is taken in the cluster until its current state matches the desired state.
The kube-controller-manager runs controllers or operators responsible to act when nodes become unavailable, to ensure container pod counts are as expected, to create endpoints, service accounts, and API access tokens.
The cloud-controller-manager runs controllers or operators responsible to interact with the underlying infrastructure of a cloud provider when nodes become unavailable, to manage storage volumes when provided by a cloud service, and to manage load balancing and routing.
Control Plane Node Components: Key-Value Data Store
etcd is an open source project under the Cloud Native Computing Foundation (CNCF). etcd is a strongly consistent, distributed key-value data store used to persist a Kubernetes cluster's state. New data is written to the data store only by appending to it, data is never replaced in the data store. Obsolete data is compacted (or shredded) periodically to minimize the size of the data store.
Out of all the control plane components, only the API Server is able to communicate with the etcd data store.
etcd's CLI management tool - etcdctl, provides snapshot save and restore capabilities which come in handy especially for a single etcd instance Kubernetes cluster - common in Development and learning environments. However, in Stage and Production environments, it is extremely important to replicate the data stores in HA mode, for cluster configuration data resiliency.
Some Kubernetes cluster bootstrapping tools, such as kubeadm, by default, provision stacked etcd control plane nodes, where the data store runs alongside and shares resources with the other control plane components on the same control plane node.
Worker Node Overview
A worker node provides a running environment for client applications. These applications are microservices running as application containers. In Kubernetes the application containers are encapsulated in Pods, controlled by the cluster control plane agents running on the control plane node. Pods are scheduled on worker nodes, where they find required compute, memory and storage resources to run, and networking to talk to each other and the outside world. A Pod is the smallest scheduling work unit in Kubernetes. It is a logical collection of one or more containers scheduled together, and the collection can be started, stopped, or rescheduled as a single unit of work.
Also, in a multi-worker Kubernetes cluster, the network traffic between client users and the containerized applications deployed in Pods is handled directly by the worker nodes, and is not routed through the control plane node.
Worker Node Components
A worker node has the following components:
• Container Runtime
• Node Agent - kubelet
• Proxy - kube-proxy
• Addons for DNS, Dashboard user interface, cluster-level monitoring and logging.
Worker Node Components: Container Runtime
Although Kubernetes is described as a "container orchestration engine", it lacks the capability to directly handle and run containers. In order to manage a container's lifecycle, Kubernetes requires a container runtime on the node where a Pod and its containers are to be scheduled. Runtimes are required on all nodes of a Kubernetes cluster, both control plane and worker. Kubernetes supports several container runtimes:
• CRI-O
A lightweight container runtime for Kubernetes, supporting quay.io and Docker Hub image registries.
• containerd
A simple, robust, and portable container runtime.
• Docker Engine
A popular and complex container platform which uses containerd as a container runtime.
• Mirantis Container Runtime
Formerly known as the Docker Enterprise Edition.
Worker Node Components: Node Agent - kubelet
The kubelet is an agent running on each node, control plane and workers, and communicates with the control plane. It receives Pod definitions, primarily from the API Server, and interacts with the container runtime on the node to run containers associated with the Pod. It also monitors the health and resources of Pods running containers.
The kubelet connects to container runtimes through a plugin based interface - the Container Runtime Interface (CRI). The CRI consists of protocol buffers, gRPC API, libraries, and additional specifications and tools. In order to connect to interchangeable container runtimes, kubelet uses a CRI shim, an application which provides a clear abstraction layer between kubelet and the container runtime.
Worker Node Components: Proxy - kube-proxy
The kube-proxy is the network agent which runs on each node, control plane and workers, responsible for dynamic updates and maintenance of all networking rules on the node. It abstracts the details of Pods networking and forwards connection requests to the containers in the Pods.
The kube-proxy is responsible for TCP, UDP, and SCTP stream forwarding or random forwarding across a set of Pod backends of an application, and it implements forwarding rules defined by users through Service API objects.
Worker Node Components: Addons
Addons are cluster features and functionality not yet available in Kubernetes, therefore implemented through 3rd-party pods and services.
• DNS
Cluster DNS is a DNS server required to assign DNS records to Kubernetes objects and resources.
• Dashboard
A general purpose web-based user interface for cluster management.
• Monitoring
Collects cluster-level container metrics and saves them to a central data store.
• Logging
Collects cluster-level container logs and saves them to a central log store for analysis.
Container-to-Container Communication Inside Pods
Making use of the underlying host operating system's kernel virtualization features, a container runtime creates an isolated network space for each container it starts. On Linux, this isolated network space is referred to as a network namespace. A network namespace can be shared across containers, or with the host operating system.
When a grouping of containers defined by a Pod is started, a special infrastructure Pause container is initialized by the Container Runtime for the sole purpose of creating a network namespace for the Pod. All additional containers, created through user requests, running inside the Pod will share the Pause container's network namespace so that they can all talk to each other via localhost.
Pod-to-Pod Communication Across Nodes
In a Kubernetes cluster Pods, groups of containers, are scheduled on nodes in a nearly unpredictable fashion. Regardless of their host node, Pods are expected to be able to communicate with all other Pods in the cluster, all this without the implementation of Network Address Translation (NAT). This is a fundamental requirement of any networking implementation in Kubernetes.
The Kubernetes network model aims to reduce complexity, and it treats Pods as VMs on a network, where each VM is equipped with a network interface - thus each Pod receiving a unique IP address. This model is called "IP-per-Pod" and ensures Pod-to-Pod communication, just as VMs are able to communicate with each other on the same network.
Let's not forget about containers though. They share the Pod's network namespace and must coordinate ports assignment inside the Pod just as applications would on a VM, all while being able to communicate with each other on localhost - inside the Pod. However, containers are integrated with the overall Kubernetes networking model through the use of the Container Network Interface (CNI) supported by CNI plugins. CNI is a set of specifications and libraries which allow plugins to configure the networking for containers. While there are a few core plugins, most CNI plugins are 3rd-party Software Defined Networking (SDN) solutions implementing the Kubernetes networking model. In addition to addressing the fundamental requirement of the networking model, some networking solutions offer support for Network Policies. Flannel, Weave, Calico are only a few of the SDN solutions available for Kubernetes clusters.
Pod-to-External World Communication
A successfully deployed containerized application running in Pods inside a Kubernetes cluster may require accessibility from the outside world. Kubernetes enables external accessibility through Services, complex encapsulations of network routing rule definitions stored in iptables on cluster nodes and implemented by kube-proxy agents. By exposing services to the external world with the aid of kube-proxy, applications become accessible from outside the cluster over a virtual IP address and a dedicated port number.
Infrastructure for Kubernetes Installation
Once we decide on the installation type, we need to decide on the infrastructure. Infrastructure related decisions are typically guided by the desired environment type, either learning or production environment. For infrastructure, we need to decide on the following:
• Should we set up Kubernetes on bare metal, public cloud, private, or hybrid cloud?
• Which underlying OS should we use? Should we choose a Linux distribution - Red Hat-based or Debian-based, or Windows?
• Which networking solution (CNI) should we use?
Hosted Solutions
Hosted Solutions providers fully manage the provided software stack, while the user pays hosting and management charges. Popular vendors providing hosted solutions for Kubernetes are (listed in alphabetical order):
• Alibaba Cloud Container Service for Kubernetes (ACK)
• Amazon Elastic Kubernetes Service (EKS)
• Azure Kubernetes Service (AKS)
• DigitalOcean Kubernetes
• Google Kubernetes Engine (GKE)
• IBM Cloud Kubernetes Service
• Oracle Container Engine for Kubernetes (OKE)
• Platform9 Managed Kubernetes (PMK)
• Red Hat OpenShift
• VMware Tanzu Kubernetes Grid
What is Minikube?
Minikube is one of the easiest, most flexible and popular methods to run an all-in-one or a multi-node local Kubernetes cluster directly on our local workstations. It installs and runs on any native OS such as Linux, macOS, or Windows.
Requirements for Running Minikube
kubectl
kubectl is a binary used to access and manage any Kubernetes cluster. It is installed through Minikube and accessed through the minikube kubectl command, or it can be installed separately and run as a standalone tool. We will explore kubectl installation and usage in future chapters.
Accessing Minikube: Command Line Interface (CLI)
kubectl is the Kubernetes Command Line Interface (CLI) client to manage cluster resources and applications. It is very flexible and easy to integrate with other systems, therefore it can be used standalone, or part of scripts and automation tools. Once all required credentials and cluster access points have been configured for kubectl, it can be used remotely from anywhere to access a cluster.
We will be using kubectl extensively to deploy applications, manage and configure Kubernetes resources.
kubectl Configuration File
To access the Kubernetes cluster, the kubectl client needs the control plane node endpoint and appropriate credentials to be able to securely interact with the API Server running on the control plane node. While starting Minikube, the startup process creates, by default, a configuration file, config, inside the .kube directory (often referred to as the kubeconfig), which resides in the user's home directory. The configuration file has all the connection details required by kubectl. By default, the kubectl binary parses this file to find the control plane node's connection endpoint, along with the required credentials. Multiple kubeconfig files can be configured with a single kubectl client. To look at the connection details, we can either display the content of the ~/.kube/config file (on Linux) or run the following command (the output is redacted for readability):
root@systest-runner:~/.kube[65120]# ls -lrt
total 12
drwxr-x--- 4 root root 4096 Sep 1 2021 cache
-rw------- 1 root root 6390 Jan 11 22:06 config
root@systest-runner:~/.kube[65120]#
root@systest-runner:~/.kube[65120]# cat config
apiVersion: v1
clusters:
- cluster:
insecure-skip-tls-verify: true
server: https://20.20.90.50:6443. ++++++ API server end point
name: 20.20.90.50
- cluster:
insecure-skip-tls-verify: true
server: https://30.30.1.17:6443
name: 30.30.1.17
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM2akNDQWRLZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJeU1URXdPREE1TVRRME9Gb1hEVE15TVRFd05UQTVNVGswT0Zvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTmZtCk1RZXJaZnVVckZqV1REVitYejNqOGE0amF2WDdmRHVsMWkrUnB1ZGJ2eTZ6TDNIUWRKRFFJTWMwRjU3cnY2aGkKcXlOWVRDazB4RFJrSC9kZDI3cFlJQ3h2VGZNcWU1WFI4R0VFRWMrV2REdmhPNFlxNjFCQ0VabTRxWStCekFLVApqLzJiQzJpb2lYSlRYRzVleTkwQy9YTjRCZTJoTUdoaUxzU0owTnFYSVBTdktQaTRrWnVTdFJ0MUJhQU5Ta2xwCnMwU2IrRUJkN2N0V0Z1cTRNd01HcDhTaU5rTHJSaG1nY0xHenB5U3pXeVlhZE1HRGxZa1RWUCtPZUJwaUNLaW0KVzlWLy90UnZ3TzNTdzdkMU9PeVdIVmtnZjh1Z0FCazAyMWZ1OHNVNCtqbUVtTFNyQ1RPUlBDeTdYZlpiY1VicwpiVW1CNERQU3VRV21TTWRlcHpzQ0F3RUFBYU5GTUVNd0RnWURWUjBQQVFIL0JBUURBZ0trTUJJR0ExVWRFd0VCCi93UUlNQVlCQWY4Q0FRQXdIUVlEVlIwT0JCWUVGQUtlNWZBZWRNWHVQWm9PNWk0TmZnaVBZQU40TUEwR0NTcUcKU0liM0RRRUJDd1VBQTRJQkFRQzVGalpqSlhoSjNYRFVMNjRCTWVaMzZuTDZYa1hjdS9TejJyN1Y5TlNJT0xlRQpXWHREWUxQcWJIQzNGV09Kd0pvdlNFWkVTblBaWXRTSmttcDQ4ZnlCbmhMSXdnRVczcVcveDZqaUNHM29aTlpnCkM5S29lbWx6RjROSFdhTlUydkRhSVdSc1RTc1hhT0xpZEdhYW85MlhUZ2dPRnRuZUxTVHJZamtUUVp1UVVGWDYKTTQrdTRYd1Y4RzgvT1o0czJBK2YxanphR1AwWmdYRU5SWEozNjBhaVM1cy9sc1RLVG1aVmhBb3prbGlhMXhnSQpRRzR0clAwdThTS0NZaFRTRmFzeVNZN0toZ3B1M1ZYRlJKRkVob0kyOTE0d0xscUg4L3p0ZUwzOEFsWG1ZSVVqCjZLNm5oaXp0VGxDSHBtVXFKUGJoc2Z6Wms1WXBqSHlYRGM0ZVBobkwKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
server: https://30.30.1.18:6443
name: 30.30.1.18
- cluster:
insecure-skip-tls-verify: true
server: https://20.20.90.50:6443
name: supervisor-20.20.90.50
contexts:
- context:
cluster: 20.20.90.50
user: wcp:20.20.90.50:administrator@vsphere.local
name: 20.20.90.50
- context:
cluster: 30.30.1.17
namespace: intelligence
user: wcp:30.30.1.17:administrator@vsphere.local
name: intelligence
- context:
cluster: 30.30.1.18
user: wcp:30.30.1.18:administrator@vsphere.local
name: intelligencecluster
current-context: intelligencecluster
kind: Config
preferences: {}
users:
- name: wcp:20.20.90.50:administrator@vsphere.local
user:
token: eyJraWQiOiJBRTE3ODc3NEM3OUU3NEVBRjNFQUJBRjRGNjUzNkJFNDUyNjFGNUUzIiwiYWxnIjoiUlMyNTYifQ.eyJzdWIiOiJBZG1pbmlzdHJhdG9yQHZzcGhlcmUubG9jYWwiLCJhdWQiOiJ2bXdhcmUtdGVzOnZjOnZuczprOHMiLCJkb21haW4iOiJ2c3BoZXJlLmxvY2FsIiwiaXNzIjoiaHR0cHM6XC9cLzEwLjkwLjEzNy4xOTZcL29wZW5pZGNvbm5lY3RcL3ZzcGhlcmUubG9jYWwiLCJncm91cF9uYW1lcyI6WyJMaWNlbnNlU2VydmljZS5BZG1pbmlzdHJhdG9yc0B2c3BoZXJlLmxvY2FsIiwiU3lzdGVtQ29uZmlndXJhdGlvbi5SZWFkT25seUB2c3BoZXJlLmxvY2FsIiwiU3lzdGVtQ29uZmlndXJhdGlvbi5TdXBwb3J0VXNlcnNAdnNwaGVyZS5sb2NhbCIsIkFkbWluaXN0cmF0b3JzQHZzcGhlcmUubG9jYWwiLCJFdmVyeW9uZUB2c3BoZXJlLmxvY2FsIiwiQ0FBZG1pbnNAdnNwaGVyZS5sb2NhbCIsIlN5c3RlbUNvbmZpZ3VyYXRpb24uQWRtaW5pc3RyYXRvcnNAdnNwaGVyZS5sb2NhbCIsIlN5c3RlbUNvbmZpZ3VyYXRpb24uQmFzaFNoZWxsQWRtaW5pc3RyYXRvcnNAdnNwaGVyZS5sb2NhbCIsIlVzZXJzQHZzcGhlcmUubG9jYWwiXSwiZXhwIjoxNjczNTM5NTU0LCJpYXQiOjE2NzM1MDM1NTQsImp0aSI6Ijk5OTVkYjlmLWRhMzMtNDAxNC04ZjhmLWYyOTExZTRkMjcwOCIsInVzZXJuYW1lIjoiQWRtaW5pc3RyYXRvciJ9.Hqv_83I6Y3CTqRfBzwXb_0lqhtGxJZUydTkWCx24_wXOHBOKVzMxvUDHhbxmt7k5f_QlD5J8bZFq6k4RuylTOLNYMlUNZxIvRvyYr6_3OrBNJb6cejNbhhGrYBmJt_Px0ya5czkLEcGEfUFpW0u_fS7wUOvcli3Qaf3dKMjdyykc5Oa019NyigyvxLEvVxeD6dvMFQHjGrWA_2Txyc0Gj2awxw8KSeLdNxjJlY1pewEpRbjAjSrkFxoWKNvfWzQiYuw4BPpEvvMOIcdU9c6yCYKVzXGx2TACeXk9cSKDUTbN5y9Xl1XGp3AW2T1YRQ3RF1RpOa5ORjfEcWYuDa2w8w
- name: wcp:30.30.1.17:administrator@vsphere.local
user:
token: eyJraWQiOiJBRTE3ODc3NEM3OUU3NEVBRjNFQUJBRjRGNjUzNkJFNDUyNjFGNUUzIiwiYWxnIjoiUlMyNTYifQ.eyJzdWIiOiJBZG1pbmlzdHJhdG9yQHZzcGhlcmUubG9jYWwiLCJhdWQiOiJ2bXdhcmUtdGVzOnZjOnZuczprOHMiLCJkb21haW4iOiJ2c3BoZXJlLmxvY2FsIiwiaXNzIjoiaHR0cHM6XC9cLzEwLjkwLjEzNy4xOTZcL29wZW5pZGNvbm5lY3RcL3ZzcGhlcmUubG9jYWwiLCJncm91cF9uYW1lcyI6WyJMaWNlbnNlU2VydmljZS5BZG1pbmlzdHJhdG9yc0B2c3BoZXJlLmxvY2FsIiwiU3lzdGVtQ29uZmlndXJhdGlvbi5SZWFkT25seUB2c3BoZXJlLmxvY2FsIiwiU3lzdGVtQ29uZmlndXJhdGlvbi5TdXBwb3J0VXNlcnNAdnNwaGVyZS5sb2NhbCIsIkFkbWluaXN0cmF0b3JzQHZzcGhlcmUubG9jYWwiLCJFdmVyeW9uZUB2c3BoZXJlLmxvY2FsIiwiQ0FBZG1pbnNAdnNwaGVyZS5sb2NhbCIsIlN5c3RlbUNvbmZpZ3VyYXRpb24uQWRtaW5pc3RyYXRvcnNAdnNwaGVyZS5sb2NhbCIsIlN5c3RlbUNvbmZpZ3VyYXRpb24uQmFzaFNoZWxsQWRtaW5pc3RyYXRvcnNAdnNwaGVyZS5sb2NhbCIsIlVzZXJzQHZzcGhlcmUubG9jYWwiXSwiZXhwIjoxNjczNTM5NTgzLCJpYXQiOjE2NzM1MDM1ODMsImp0aSI6IjEyYzQwNjVlLWQzNDUtNDNjOC05MWE5LWU3OGIxZTk2NTE4ZSIsInVzZXJuYW1lIjoiQWRtaW5pc3RyYXRvciJ9.JeOpOc_Vt-_1TQKGMJ2B1R-l9JHfXoDrdY6dxATHtgEFA8yztsR0lut5GfSzDIi3TDrOqsoOT3EkBLSMf0_Esf-syGwvhFsogN1YPPQ-7b_XvI2w0hNyvq4yMiwEXUaYBJ9YnQJum3HOMpzNSlYGWm86j1Zf7c12OM96LI8TqlNpBCnCZmGxw3v-RcnOxcahmzqNWx69tIpii-C559AAMqp6L-1djt3BsXy_iOQTt-Ww9ao8jkIIDwCH6-kv2irU7Jdi0YRg53fSLuJBlP4wPW83K6aaEzefQgGjyCEAMft_Nwo2KC8r3TVYRUMiNEEipLyiikN2KCWCpuK4rFhsiQ
- name: wcp:30.30.1.18:administrator@vsphere.local
user:
token: eyJraWQiOiJBRTE3ODc3NEM3OUU3NEVBRjNFQUJBRjRGNjUzNkJFNDUyNjFGNUUzIiwiYWxnIjoiUlMyNTYifQ.eyJzdWIiOiJBZG1pbmlzdHJhdG9yQHZzcGhlcmUubG9jYWwiLCJhdWQiOiJ2bXdhcmUtdGVzOnZjOnZuczprOHM6MWNhZjJhNGUtMTU2Yy00MzhjLThjYWQtMDk0M2E3ZTc1YzcyIiwiZG9tYWluIjoidnNwaGVyZS5sb2NhbCIsImlzcyI6Imh0dHBzOlwvXC8xMC45MC4xMzcuMTk2XC9vcGVuaWRjb25uZWN0XC92c3BoZXJlLmxvY2FsIiwiZ3JvdXBfbmFtZXMiOlsiTGljZW5zZVNlcnZpY2UuQWRtaW5pc3RyYXRvcnNAdnNwaGVyZS5sb2NhbCIsIlN5c3RlbUNvbmZpZ3VyYXRpb24uUmVhZE9ubHlAdnNwaGVyZS5sb2NhbCIsIlN5c3RlbUNvbmZpZ3VyYXRpb24uU3VwcG9ydFVzZXJzQHZzcGhlcmUubG9jYWwiLCJBZG1pbmlzdHJhdG9yc0B2c3BoZXJlLmxvY2FsIiwiRXZlcnlvbmVAdnNwaGVyZS5sb2NhbCIsIkNBQWRtaW5zQHZzcGhlcmUubG9jYWwiLCJTeXN0ZW1Db25maWd1cmF0aW9uLkFkbWluaXN0cmF0b3JzQHZzcGhlcmUubG9jYWwiLCJTeXN0ZW1Db25maWd1cmF0aW9uLkJhc2hTaGVsbEFkbWluaXN0cmF0b3JzQHZzcGhlcmUubG9jYWwiLCJVc2Vyc0B2c3BoZXJlLmxvY2FsIl0sImV4cCI6MTY3MzUzOTU4MiwiaWF0IjoxNjczNTAzNTgyLCJqdGkiOiJjYTk1ZWZhNS1mMGIzLTRiZTUtOTg4NS1mMTQ4NzMyNDZkODkiLCJ1c2VybmFtZSI6IkFkbWluaXN0cmF0b3IifQ.ahwI-jHYT3RK51Ud66nij1IjzvZaCG2EF75KDYKY7z6QQR_fBrMT3464-3OaH-W_ExtO9l_10DeIPGzYGlFZTl2HegOPfzNZtm6ThfQ2KVCS-5c3otbq4jWt6teGVA1AZaB8PekYe0D0R81gon7DzLJ4ULidUvuuKS2JyTgdp80J-Ig0jYj5EjzwH849GmpMi2-UtH5ytuVjtfulCWBx0mYYsy8gdhjBNO0OJ3pvxCob84dAh7Gv7iQXH-7mvrmBqU4ykIPkUvncvdwtjv0k_p7EIoQ2CmAhAMoJ0ftgawWr7Ul2V3lID_ZqnjQxq3zt4MzPROkJbXFxwKZGA4rvmg
kubectl cluster-info
Kubernetes master is running at https://30.30.1.18:6443
CoreDNS is running at https://30.30.1.18:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Kubernetes Object Model
With each object, we declare our intent, or the desired state of the object, in the spec section. The Kubernetes system manages the status section for objects, where it records the actual state of the object. At any given point in time, the Kubernetes Control Plane tries to match the object's actual state to the object's desired state. An object definition manifest must include other fields that specify the version of the API we are referencing as the apiVersion, the object type as kind, and additional data helpful to the cluster or users for accounting purposes - the metadata.
Examples of Kubernetes object types are Nodes, Namespaces, Pods, ReplicaSets, Deployments, DaemonSets, etc. We will explore them next.
When creating an object, the object's configuration data section from below the spec field has to be submitted to the Kubernetes API Server. The API request to create an object must have the spec section, describing the desired state, as well as other details. Although the API Server accepts object definitions in a JSON format, most often we provide such definition manifests in a YAML format which is converted by kubectl in a JSON payload and sent to the API Server.
Nodes
Nodes are virtual identities assigned by Kubernetes to the systems part of the cluster - whether Virtual Machines, bare-metal, Containers, etc. These identities are unique to each system, and are used by the cluster for resources accounting and monitoring purposes, which helps with workload management throughout the cluster.
Each node is managed with the help of two Kubernetes node agents - kubelet and kube-proxy, while it also hosts a container runtime. The container runtime is required to run all containerized workload on the node - control plane agents and user workloads. The kubelet and kube-proxy node agents are responsible for executing all local workload management related tasks - interact with the runtime to run containers, monitor containers and node health, report any issues and node state to the API Server, and managing network traffic to containers.
Based on their pre-determined functions, there are two distinct types of nodes - control plane and worker. A typical Kubernetes cluster includes at least one control plane node, but it may include multiple control plane nodes for Highly Available (HA) control plane. In addition, the cluster includes one or mode worker nodes to provide resource redundancy in the cluster. There are cases when a single all-in-one cluster is bootstrapped as a single node on a single VM, bare-metal, or Container, when high availability and resource redundancy are not of importance. These are hybrid or mixed nodes hosting both control plane agents and user workload on the same system. Minikube allows us to bootstrap multi-node clusters with distinct, dedicated control plane nodes, however, if our host system has a limited amount of physical resources, we can easily bootstrap a single all-in-one cluster as a single node on a single VM or Container, and still be able to explore most of the topics covered in this course, with the exception of features specific to multi-node clusters, such as DaemonSets, multi node networking, etc.
Node identities are created and assigned during the cluster bootstrapping process by the tool responsible to initialize the cluster agents. Minikube is using the default kubeadm bootstrapping tool, to initialize the control plane node during the init phase and grow the cluster by adding worker or control plane nodes with the join phase.
The control plane nodes run the control plane agents, such as the API Server, Scheduler, Controller Managers, and etcd in addition to the kubelet and kube-proxy node agents, the container runtime, and addons for container networking, monitoring, logging, DNS, etc.
Worker nodes run the kubelet and kube-proxy node agents, the container runtime, and addons for container networking, monitoring, logging, DNS, etc.
Namespaces
If multiple users and teams use the same Kubernetes cluster we can partition the cluster into virtual sub-clusters using Namespaces. The names of the resources/objects created inside a Namespace are unique, but not across Namespaces in the cluster.
To list all the Namespaces, we can run the following command:
$ kubectl get namespaces
NAME STATUS AGE
default Active 11h
kube-node-lease Active 11h
kube-public Active 11h
kube-system Active 11h
Generally, Kubernetes creates four Namespaces out of the box: kube-system, kube-public, kube-node-lease, and default. The kube-system Namespace contains the objects created by the Kubernetes system, mostly the control plane agents. The default Namespace contains the objects and resources created by administrators and developers, and objects are assigned to it by default unless another Namespace name is provided by the user. kube-public is a special Namespace, which is unsecured and readable by anyone, used for special purposes such as exposing public (non-sensitive) information about the cluster. The newest Namespace is kube-node-lease which holds node lease objects used for node heartbeat data. Good practice, however, is to create additional Namespaces, as desired, to virtualize the cluster and isolate users, developer teams, applications, or tiers:
Pods
A Pod is the smallest Kubernetes workload object. It is the unit of deployment in Kubernetes, which represents a single instance of the application. A Pod is a logical collection of one or more containers, enclosing and isolating them to ensure that they:
• Are scheduled together on the same host with the Pod.
• Share the same network namespace, meaning that they share a single IP address originally assigned to the Pod.
• Have access to mount the same external storage (volumes) and other common dependencies.
Pods are ephemeral in nature, and they do not have the capability to self-heal themselves. That is the reason they are used with controllers, or operators (controllers/operators are used interchangeably), which handle Pods' replication, fault tolerance, self-healing, etc. Examples of controllers are Deployments, ReplicaSets, DaemonSets, Jobs, etc. When an operator is used to manage an application, the Pod's specification is nested in the controller's definition using the Pod Template.
Below is an example of a stand-alone Pod object's definition manifest in YAML format, without an operator:
apiVersion: v1
kind: Pod
metadata:
name: nginx-pod
labels:
run: nginx-pod
spec:
containers:
- name: nginx
image: nginx:1.22.1
ports:
- containerPort: 80
The apiVersion field must specify v1 for the Pod object definition. The second required field is kind specifying the Pod object type. The third required field metadata, holds the object's name and optional labels and annotations. The fourth required field spec marks the beginning of the block defining the desired state of the Pod object - also named the PodSpec. Our Pod creates a single container running the nginx:1.22.1 image pulled from a container image registry, in this case from Docker Hub. The containerPort field specifies the container port to be exposed by Kubernetes resources for inter-application access or external client access - to be explored in the Services chapter. The contents of spec are evaluated for scheduling purposes, then the kubelet of the selected node becomes responsible for running the container image with the help of the container runtime of the node. The Pod's name and labels are used for workload accounting purposes.
How to Run Applications with Pods
root@systest-runner:~[65117]# cat tkg.yaml
apiVersion: run.tanzu.vmware.com/v1alpha1
kind: TanzuKubernetesCluster
metadata:
name: intelligencecluster
namespace: intelligence
spec:
topology:
controlPlane:
count: 1
class: best-effort-xlarge # vmclass to be used for workers(s)
storageClass: wcpglobalstorageprofile
workers:
count: 4
class: best-effort-3xlarge # vmclass to be used for workers(s)
storageClass: wcpglobalstorageprofile
volumes:
- name: containerd
capacity:
storage: 64Gi
mountPath: /var/lib/containerd
distribution:
#fullVersion: v1.21.6+vmware.1-tkg.1.b3d708a
# version: v1.21.6+vmware.1-tkg.1.b3d708a
fullVersion: v1.21.2+vmware.1-tkg.1.ee25d55
version: v1.21.2+vmware.1-tkg.1.ee25d55
root@systest-runner:~[65091]#
root@systest-runner:~/nlekhak/nsx-qe/spark[64257](master)# kubectl config view
apiVersion: v1
clusters:
- cluster:
insecure-skip-tls-verify: true
server: https://20.20.90.50:6443
name: 20.20.90.50
- cluster:
insecure-skip-tls-verify: true
server: https://30.30.1.17:6443
name: 30.30.1.17
- cluster:
certificate-authority-data: DATA+OMITTED
server: https://30.30.1.18:6443
name: 30.30.1.18
- cluster:
insecure-skip-tls-verify: true
server: https://20.20.90.50:6443
name: supervisor-20.20.90.50
contexts:
- context:
cluster: 20.20.90.50
user: wcp:20.20.90.50:administrator@vsphere.local
name: 20.20.90.50
- context:
cluster: 30.30.1.17
namespace: intelligence
user: wcp:30.30.1.17:administrator@vsphere.local
name: intelligence
- context:
cluster: 30.30.1.18
user: wcp:30.30.1.18:administrator@vsphere.local
name: intelligencecluster
current-context: intelligencecluster
kind: Config
preferences: {}
users:
- name: wcp:20.20.90.50:administrator@vsphere.local
user:
token: REDACTED
- name: wcp:30.30.1.17:administrator@vsphere.local
user:
token: REDACTED
- name: wcp:30.30.1.18:administrator@vsphere.local
user:
token: REDACTED
root@systest-runner:~/nlekhak/nsx-qe/spark[64258](master)#
Connecting Users or Applications to Pods
To access the application, a user or another application need to connect to the Pods. As Pods are ephemeral in nature, resources like IP addresses allocated to them cannot be static. Pods could be terminated abruptly or be rescheduled based on existing requirements.
Let's take, for example, a scenario where an operator manages a set of Pods and a user/client is accessing the Pods directly by using their individual IP addresses.
Unexpectedly, one of the Pods the user/client is accessing is terminated, and a new Pod is created by the controller. The new Pod will have a new IP address, that will not be immediately known by the user/client.
To overcome this situation, Kubernetes provides a higher-level abstraction called Service, which logically groups Pods and defines a policy to access them. This grouping is achieved via Labels and Selectors
Services
Labels and Selectors use a key-value pair format. In the following graphical representation, app is the Label key, frontend and db are Label values for different Pods.
Using the selectors app==frontend and app==db, we group Pods into two logical sets: one set with 3 Pods, and one set with a single Pod.
We assign a name to the logical grouping, referred to as a Service. The Service name is also registered with the cluster's internal DNS service. In our example, we create two Services, frontend-svc, and db-svc, and they have the app==frontend and the app==db Selectors, respectively.
Services can expose single Pods, ReplicaSets, Deployments, DaemonSets, and StatefulSets. When exposing the Pods managed by an operator, the Service's Selector may use the same label(s) as the operator.
Service Object Example
The following is an example of a Service object definition:
apiVersion: v1
kind: Service
metadata:
name: frontend-svc
spec:
selector:
app: frontend
ports:
- protocol: TCP
port: 80
targetPort: 5000
In this example, we are creating a frontend-svc Service by selecting all the Pods that have the Label key=app set to value=frontend. By default, each Service receives an IP address routable only inside the cluster, known as ClusterIP. In our example, we have 172.17.0.4 and 172.17.0.5 as ClusterIPs assigned to our frontend-svc and db-svc Services, respectively.
The user/client now connects to a Service via its ClusterIP, which forwards traffic to one of the Pods attached to it. A Service provides load balancing by default while selecting the Pods for traffic forwarding.
While the Service forwards traffic to Pods, we can select the targetPort on the Pod which receives the traffic. In our example, the frontend-svc Service receives requests from the user/client on port: 80 and then forwards these requests to one of the attached Pods on the targetPort: 5000. If the targetPort is not defined explicitly, then traffic will be forwarded to Pods on the port on which the Service receives traffic. It is very important to ensure that the value of the targetPort, which is 5000 in this example, matches the value of the containerPort property of the Pod spec section.
A logical set of a Pod's IP address, along with the targetPort is referred to as a Service endpoint. In our example, the frontend-svc Service has 3 endpoints: 10.0.1.3:5000, 10.0.1.4:5000, and 10.0.1.5:5000. Endpoints are created and managed automatically by the Service, not by the Kubernetes cluster administrator.
kubectl get deployments -n nsxi-platform
NAME READY UP-TO-DATE AVAILABLE AGE
authserver 1/1 1 1 25h
cloud-connector-check-license-status 1/1 1 1 24h
cloud-connector-file-server 1/1 1 1 24h
cloud-connector-proxy 1/1 1 1 24h
cloud-connector-update-license-status 1/1 1 1 24h
cluster-api 1/1 1 1 25h
common-agent 1/1 1 1 25h
data-archiver 1/1 1 1 24h
data-collection 1/1 1 1 24h
druid-broker 1/1 1 1 25h
druid-config-broker 1/1 1 1 25h
druid-coordinator 1/1 1 1 25h
druid-overlord 1/1 1 1 25h
druid-router 2/2 2 2 25h
intelligence-ui 1/1 1 1 24h
latestflow 3/3 3 3 24h
malware-prevention-feature-switch-watcher-notifier-ndr 1/1 1 1 24h
malware-prevention-ui 1/1 1 1 24h
metrics-app-server 1/1 1 1 25h
metrics-db-helper 1/1 1 1 25h
metrics-manager 1/1 1 1 25h
metrics-nsx-config 1/1 1 1 25h
metrics-postgresql-ha-pgpool 2/2 2 2 25h
metrics-query-server 1/1 1 1 25h
monitor 1/1 1 1 25h
nsx-config 1/1 1 1 24h
nsx-ndr-feature-switch-watcher-notifier-ndr 1/1 1 1 24h
nsx-ndr-upload-config 1/1 1 1 24h
nsx-ndr-worker-file-event-processor 1/1 1 1 24h
nsx-ndr-worker-file-event-uploader 1/1 1 1 24h
nsx-ndr-worker-ids-event-processor 1/1 1 1 24h
nsx-ndr-worker-monitored-host-processor 1/1 1 1 24h
nsx-ndr-worker-monitored-host-uploader 1/1 1 1 24h
nsx-ndr-worker-ndr-event-processor 1/1 1 1 24h
nsx-ndr-worker-ndr-event-uploader 1/1 1 1 24h
nsx-ndr-worker-nta-event-processor 1/1 1 1 24h
nta-server 1/1 1 1 24h
platform-ui 1/1 1 1 25h
postgresql-ha-pgpool 2/2 2 2 25h
pubsub 1/1 1 1 24h
recommendation 1/1 1 1 24h
reputation-service 1/1 1 1 24h
reputation-service-feature-switch-watcher-notifier-dependencies 1/1 1 1 24h
routing-controller 1/1 1 1 25h
sa-asds 1/1 1 1 24h
sa-events-processor 1/1 1 1 24h
sa-scheduler-services 1/1 1 1 24h
sa-web-services 1/1 1 1 24h
spark-job-manager 1/1 1 1 24h
spark-operator 1/1 1 1 25h
telemetry 1/1 1 1 25h
trust-manager 1/1 1 1 25h
visualization 1/1 1 1 24h
workload 1/1 1 1 24h
kubectl get replicasets -n nsxi-platform
NAME DESIRED CURRENT READY AGE
authserver-545db5d5c 1 1 1 24h
authserver-5f97669b8f 0 0 0 25h
authserver-77c5686858 0 0 0 25h
cloud-connector-check-license-status-7ccb96879f 1 1 1 24h
cloud-connector-file-server-7966bbbd79 1 1 1 24h
cloud-connector-proxy-b454654f 1 1 1 24h
cloud-connector-update-license-status-57d6bb99fd 1 1 1 8h
cloud-connector-update-license-status-6799bf5764 0 0 0 8h
cloud-connector-update-license-status-67bd69bbd 0 0 0 24h
cloud-connector-update-license-status-865db9c584 0 0 0 23h
cluster-api-5cc68d4c7c 1 1 1 25h
common-agent-6c78786947 1 1 1 25h
data-archiver-558dd5dd89 1 1 1 24h
data-collection-5885c4b8f6 1 1 1 24h
data-collection-5ff64bf494 0 0 0 24h
druid-broker-85988689d4 1 1 1 25h
druid-config-broker-6f7569568d 1 1 1 25h
druid-coordinator-57f5b6947b 1 1 1 25h
druid-overlord-7d79bbfb4 1 1 1 25h
druid-router-7fc9587fb8 2 2 2 25h
intelligence-ui-86b79945b4 1 1 1 24h
latestflow-78fb895bfc 3 3 3 24h
malware-prevention-feature-switch-watcher-notifier-ndr-7bfcc649c6 1 1 1 24h
malware-prevention-ui-6c456f8854 1 1 1 24h
metrics-app-server-b575f5654 1 1 1 25h
metrics-db-helper-b875bd58b 1 1 1 25h
metrics-manager-ddb9c68b4 1 1 1 25h
metrics-nsx-config-5585d6c975 1 1 1 25h
metrics-postgresql-ha-pgpool-58957c6d8f 0 0 0 25h
metrics-postgresql-ha-pgpool-7ccc8bcd7c 0 0 0 25h
metrics-postgresql-ha-pgpool-f69979f4b 2 2 2 25h
metrics-query-server-755fcfd476 1 1 1 25h
monitor-5bb6f585c4 1 1 1 25h
monitor-764d68bd6 0 0 0 25h
monitor-8479fd98bf 0 0 0 25h
nsx-config-7b7f586cc5 1 1 1 24h
nsx-config-87f6d8c58 0 0 0 24h
nsx-ndr-feature-switch-watcher-notifier-ndr-86997d545 1 1 1 24h
nsx-ndr-upload-config-5b8bd78465 1 1 1 24h
nsx-ndr-worker-file-event-processor-67b59b597b 1 1 1 24h
nsx-ndr-worker-file-event-uploader-64848ff99b 1 1 1 24h
nsx-ndr-worker-ids-event-processor-77875449cf 1 1 1 24h
nsx-ndr-worker-monitored-host-processor-77f86c57b5 1 1 1 24h
nsx-ndr-worker-monitored-host-uploader-6844bf6bb5 1 1 1 24h
nsx-ndr-worker-ndr-event-processor-7679f8666d 1 1 1 24h
nsx-ndr-worker-ndr-event-uploader-59949655fd 1 1 1 24h
nsx-ndr-worker-nta-event-processor-79485bb555 1 1 1 24h
nta-server-5b5bd7c977 0 0 0 24h
nta-server-8668f7845f 1 1 1 24h
platform-ui-5949bd7758 1 1 1 25h
postgresql-ha-pgpool-749864fcff 2 2 2 25h
pubsub-5754b5b7d4 0 0 0 24h
pubsub-64cbc4b959 1 1 1 24h
recommendation-69b46b7589 1 1 1 24h
recommendation-6dd888d6c9 0 0 0 24h
reputation-service-598ccb4b66 0 0 0 24h
reputation-service-65b66f4cc6 0 0 0 24h
reputation-service-676857d595 0 0 0 24h
reputation-service-68fd48b488 1 1 1 8h
reputation-service-74f945675d 0 0 0 23h
reputation-service-7c886f9c58 0 0 0 8h
reputation-service-feature-switch-watcher-notifier-dependencies-84d8498f57 1 1 1 24h
routing-controller-68448bc487 1 1 1 25h
sa-asds-5989ff664b 1 1 1 24h
sa-events-processor-7d4f68b68b 1 1 1 24h
sa-scheduler-services-5c5d845766 1 1 1 24h
sa-web-services-766c6f84c5 1 1 1 24h
sa-web-services-7c6c6586ff 0 0 0 24h
spark-job-manager-57c54844f6 1 1 1 24h
spark-operator-bd54c67b9 1 1 1 25h
telemetry-75ff485c8d 1 1 1 25h
trust-manager-97f745597 1 1 1 25h
visualization-5cdbc9d56c 0 0 0 24h
visualization-5df4848df9 1 1 1 24h
workload-8b9d5687b 1 1 1 24h
workload-9d8794dfd 0 0 0 24h
kubectl get pods -n nsxi-platform
NAME READY STATUS RESTARTS AGE
anomalydetectionstreamingjob-74298b85bace5ef7-exec-1 1/1 Running 0 23h
anomalydetectionstreamingjob-74298b85bace5ef7-exec-2 1/1 Running 0 23h
authserver-545db5d5c-27qg7 1/1 Running 0 24h
cloud-connector-check-license-status-7ccb96879f-ckmcz 2/2 Running 0 24h
cloud-connector-file-server-7966bbbd79-2448m 1/1 Running 0 24h
cloud-connector-proxy-b454654f-7kgm7 2/2 Running 0 24h
cloud-connector-register-k4hc8 0/2 Completed 0 24h
cloud-connector-update-license-status-57d6bb99fd-qvpn8 2/2 Running 0 8h
cluster-api-5cc68d4c7c-fwvkj 2/2 Running 0 25h
common-agent-6c78786947-kz6m2 1/1 Running 0 25h
common-agent-create-kafka-topic-kwzps 0/1 Completed 0 25h
configure-druid-l4zw2 0/1 Completed 0 25h
contextcorrelator-92c07a85bac3278f-exec-1 2/2 Running 0 23h
continuousmonitoringjob-driver 0/1 Completed 0 56m
correlated-flow-daily-reindex-27898680-wfb4z 0/1 Completed 0 11h
correlated-flow-hourly-reindex-27899340-crnxk 0/1 Completed 0 26m
correlated-flow-rec-daily-reindex-27898740-xtmc4 0/1 Completed 0 10h
correlated-flow-viz-daily-reindex-27898740-wp8sq 0/1 Completed 0 10h
create-kubeapi-networkpolicy-job-zpznt 0/1 Completed 0 25h
data-archiver-558dd5dd89-lv5fg 1/1 Running 0 24h
data-collection-5885c4b8f6-z67b9 1/1 Running 0 24h
druid-broker-85988689d4-7bqzj 1/1 Running 2 25h
druid-cleaner-27899040-fhdqx 0/1 Completed 0 5h26m
druid-config-broker-6f7569568d-xnjct 1/1 Running 2 25h
druid-config-historical-0 1/1 Running 0 25h
druid-coordinator-57f5b6947b-2t5p4 1/1 Running 2 25h
druid-historical-0 1/1 Running 0 25h
druid-historical-1 1/1 Running 0 25h
druid-middle-manager-0 1/1 Running 0 25h
druid-middle-manager-1 1/1 Running 0 25h
druid-middle-manager-2 1/1 Running 0 25h
druid-overlord-7d79bbfb4-gkkzm 1/1 Running 1 25h
druid-router-7fc9587fb8-jvt7j 1/1 Running 0 25h
druid-router-7fc9587fb8-sq6g2 1/1 Running 0 25h
druid-rule-monitor-27899360-fqslh 0/1 Completed 0 6m58s
feature-service-flow-feature-creator-27899220-hzxfn 0/1 Completed 0 146m
feature-service-flow-feature-creator-27899280-bk7zp 0/1 Completed 0 86m
feature-service-flow-feature-creator-27899340-kt75s 0/1 Completed 0 26m
feature-service-name-feature-creator-27899220-2wdfk 0/1 Completed 0 146m
feature-service-name-feature-creator-27899280-m7k8b 0/1 Completed 0 86m
feature-service-name-feature-creator-27899340-jkw9h 0/1 Completed 0 26m
feature-service-setup-minio-job-drszm 0/1 Completed 0 24h
fluentd-0 1/1 Running 0 25h
infraclassifier-pod-cleaner-27899365-d57j2 0/1 Completed 0 118s
intelligence-ui-86b79945b4-x9pbn 1/1 Running 0 24h
kafka-0 1/1 Running 0 24h
kafka-1 1/1 Running 0 24h
kafka-2 1/1 Running 0 24h
latestflow-78fb895bfc-pdxfb 1/1 Running 0 23h
latestflow-78fb895bfc-sclzm 1/1 Running 0 23h
latestflow-78fb895bfc-zmjvm 1/1 Running 0 23h
llanta-detectors-0 4/4 Running 6 24h
llanta-setup-minio-job-42r5f 0/1 Completed 0 24h
malware-prevention-create-kafka-topic-89dsg 0/1 Completed 0 24h
malware-prevention-feature-switch-watcher-notifier-ndr-7bfww5bv 1/1 Running 0 24h
malware-prevention-ui-6c456f8854-df5d5 1/1 Running 0 24h
metrics-app-server-b575f5654-75fd2 1/1 Running 0 25h
metrics-db-helper-b875bd58b-wxhc9 1/1 Running 1 25h
metrics-manager-ddb9c68b4-v5457 1/1 Running 1 25h
metrics-nsx-config-5585d6c975-d5bht 1/1 Running 0 25h
metrics-nsx-config-create-kafka-topic-sqf4q 0/1 Completed 0 25h
metrics-postgresql-ha-pgpool-f69979f4b-czj4g 1/1 Running 3 25h
metrics-postgresql-ha-pgpool-f69979f4b-t9vj9 1/1 Running 0 24h
metrics-postgresql-ha-postgresql-0 1/1 Running 0 25h
metrics-postgresql-ha-postgresql-1 1/1 Running 2 25h
metrics-postgresql-ha-postgresql-2 1/1 Running 2 25h
metrics-query-server-755fcfd476-khldx 1/1 Running 2 25h
minio-0 1/1 Running 0 25h
minio-1 1/1 Running 0 25h
minio-2 1/1 Running 0 25h
minio-3 1/1 Running 0 25h
minio-make-bucket-for-druid-cztmt 0/1 Completed 2 25h
monitor-5bb6f585c4-tnknb 1/1 Running 0 25h
monitor-create-kafka-topic-zwxqm 0/1 Completed 0 25h
mps-post-install-jobs-ntjkr 0/1 Completed 0 24h
nsx-config-7b7f586cc5-rq5bs 1/1 Running 0 24h
nsx-config-create-kafka-topic-85bq5 0/1 Completed 0 24h
nsx-ndr-enable-ids-zxn4h 0/1 Completed 0 24h
nsx-ndr-feature-switch-watcher-notifier-ndr-86997d545-4ngjt 1/1 Running 0 24h
nsx-ndr-setup-kafka-jwxq9 0/1 Completed 0 24h
nsx-ndr-upload-config-5b8bd78465-pbpjd 2/2 Running 0 24h
nsx-ndr-worker-file-event-processor-67b59b597b-5gb72 2/2 Running 0 24h
nsx-ndr-worker-file-event-uploader-64848ff99b-xsjws 2/2 Running 0 24h
nsx-ndr-worker-ids-event-processor-77875449cf-rmp9g 2/2 Running 0 24h
nsx-ndr-worker-monitored-host-processor-77f86c57b5-jwbb9 2/2 Running 1 24h
nsx-ndr-worker-monitored-host-uploader-6844bf6bb5-n9rnz 2/2 Running 0 24h
nsx-ndr-worker-ndr-event-processor-7679f8666d-6fqg9 2/2 Running 0 24h
nsx-ndr-worker-ndr-event-uploader-59949655fd-4v4v9 2/2 Running 0 24h
nsx-ndr-worker-nta-event-processor-79485bb555-4fm4v 2/2 Running 1 24h
nsxi-platform-fluent-bit-4gdhn 1/1 Running 0 25h
nsxi-platform-fluent-bit-7rrhv 1/1 Running 0 25h
nsxi-platform-fluent-bit-9mwxx 1/1 Running 0 25h
nsxi-platform-fluent-bit-sks6s 1/1 Running 0 25h
nsxi-post-install-jobs-9vbn7 0/1 Completed 0 24h
nta-flow-driver 1/1 Running 0 23h
nta-server-8668f7845f-xcjqv 2/2 Running 1 24h
nta-server-create-kafka-topic-job-bqt5t 0/1 Completed 0 24h
overflowcorrelator-e1ac2585bac55989-exec-1 2/2 Running 0 23h
overflowcorrelator-e1ac2585bac55989-exec-2 2/2 Running 0 23h
overflowcorrelator-e1ac2585bac55989-exec-3 2/2 Running 0 23h
platform-ui-5949bd7758-lq2sb 1/1 Running 0 25h
pod-cleaner-27899340-q9pds 0/1 Completed 0 26m
postgresql-ha-pgpool-749864fcff-bqdp8 1/1 Running 0 25h
postgresql-ha-pgpool-749864fcff-rmpqs 1/1 Running 0 25h
postgresql-ha-postgresql-0 1/1 Running 0 25h
processing-create-kafka-topic-job-5cmbp 0/1 Completed 0 24h
processing-pod-cleaner-27899365-wjpwr 0/1 Completed 0 118s
processing-setup-minio-job-fm967 0/1 Completed 0 24h
pubsub-64cbc4b959-jshxw 1/1 Running 0 24h
rawflowcorrelator-49a10385bae18f98-exec-1 2/2 Running 0 23h
rawflowcorrelator-49a10385bae18f98-exec-2 2/2 Running 0 23h
rawflowcorrelator-49a10385bae18f98-exec-3 2/2 Running 0 23h
recommendation-69b46b7589-bx2dp 2/2 Running 0 24h
recommendation-configure-druid-6r4zr 0/1 Completed 0 24h
recommendation-upgrade-configure-druid-fk9mn 0/1 Completed 0 24h
redis-master-0 1/1 Running 0 25h
redis-slave-0 1/1 Running 0 25h
redis-slave-1 1/1 Running 0 25h
reputation-service-68fd48b488-xws58 1/1 Running 0 8h
reputation-service-feature-switch-watcher-notifier-dependek9rwl 1/1 Running 0 24h
routing-controller-68448bc487-ctd5m 1/1 Running 0 25h
sa-asds-5989ff664b-9m2z5 1/1 Running 0 24h
sa-events-processor-7d4f68b68b-fgsbh 1/1 Running 0 24h
sa-scheduler-services-5c5d845766-n6gg5 1/1 Running 0 24h
sa-web-services-766c6f84c5-jgtcj 1/1 Running 0 24h
spark-app-context-driver 2/2 Running 0 24h
spark-app-overflow-driver 2/2 Running 0 24h
spark-app-rawflow-driver 2/2 Running 0 23h
spark-job-manager-57c54844f6-rh9cf 1/1 Running 0 24h
spark-operator-bd54c67b9-b72x2 1/1 Running 0 25h
spark-operator-webhook-init-lx68b 0/1 Completed 0 25h
telemetry-75ff485c8d-wwbjs 1/1 Running 0 25h
telemetry-create-kafka-topic-t8km2 0/1 Completed 0 25h
trust-manager-97f745597-xkv2n 1/1 Running 1 25h
visualization-5df4848df9-s2n9g 1/1 Running 0 24h
visualization-create-kafka-topic-job-5skrg 0/1 Completed 0 24h
visualization-postgres-purge-cronjob-27898560-2sw7z 0/1 Completed 0 13h
workload-8b9d5687b-wt22j 1/1 Running 0 24h
zookeeper-0 1/1 Running 0 25h
zookeeper-1 1/1 Running 0 25h
zookeeper-2 1/1 Running 0 25h
For labels check describe pod
kubectl describe pod zookeeper-0 -n nsxi-platform
kubectl get services
Liveness and Readiness Probes
While containerized applications are scheduled to run in pods on nodes across our cluster, at times the applications may become unresponsive or may be delayed during startup. Implementing Liveness and Readiness Probes allows the kubelet to control the health of the application running inside a Pod's container and force a container restart of an unresponsive application. When defining both Readiness and Liveness Probes, it is recommended to allow enough time for the Readiness Probe to possibly fail a few times before a pass, and only then check the Liveness Probe. If Readiness and Liveness Probes overlap there may be a risk that the container never reaches ready state, being stuck in an infinite re-create - fail loop.
Liveness
If a container in the Pod has been running successfully for a while, but the application running inside this container suddenly stopped responding to our requests, then that container is no longer useful to us. This kind of situation can occur, for example, due to application deadlock or memory pressure. In such a case, it is recommended to restart the container to make the application available.
Rather than restarting it manually, we can use a Liveness Probe. Liveness Probe checks on an application's health, and if the health check fails, kubelet restarts the affected container automatically.
Liveness Probes can be set by defining:
• Liveness command
• Liveness HTTP request
• TCP Liveness probe.
##A Pod with containers that do not report ready status will not receive traffic from Kubernetes Services.
Volumes
As we know, containers running in Pods are ephemeral in nature. All data stored inside a container is deleted if the container crashes. However, the kubelet will restart it with a clean slate, which means that it will not have any of the old data.
To overcome this problem, Kubernetes uses Volumes, storage abstractions that allow various storage technologies to be used by Kubernetes and offered to containers in Pods as storage media. A Volume is essentially a mount point on the container's file system backed by a storage medium. The storage medium, content and access mode are determined by the Volume Type.
In Kubernetes, a Volume is linked to a Pod and can be shared among the containers of that Pod. Although the Volume has the same life span as the Pod, meaning that it is deleted together with the Pod, the Volume outlives the containers of the Pod - this allows data to be preserved across container restarts.
ConfigMaps
ConfigMaps allow us to decouple the configuration details from the container image. Using ConfigMaps, we pass configuration data as key-value pairs, which are consumed by Pods or any other system components and controllers, in the form of environment variables, sets of commands and arguments, or volumes. We can create ConfigMaps from literal values, from configuration files, from one or more files or directories.
According to kubernetes.io,
"An Ingress is a collection of rules that allow inbound connections to reach the cluster Services."
To allow the inbound connection to reach the cluster Services, Ingress configures a Layer 7 HTTP/HTTPS load balancer for Services and provides the following:
With Ingress, users do not connect directly to a Service. Users reach the Ingress endpoint, and, from there, the request is forwarded to the desired Service.
An Ingress Controller is an application watching the Control Plane Node's API server for changes in the Ingress resources and updates the Layer 7 Load Balancer accordingly. Ingress Controllers are also know as Controllers, Ingress Proxy, Service Proxy, Revers Proxy, etc. Kubernetes supports an array of Ingress Controllers, and, if needed, we can also build our own. GCE L7 Load Balancer Controller and Nginx Ingress Controller are commonly used Ingress Controllers. Other controllers are Contour, HAProxy Ingress, Istio Ingress, Kong, Traefik, etc. In order to ensure that the ingress controller is watching its corresponding ingress resource, the ingress resource definition manifest needs to include an ingress class annotation with the name of the desired controller kubernetes.io/ingress.class: "nginx" (for an nginx ingress controller).
Network Policies
Kubernetes was designed to allow all Pods to communicate freely, without restrictions, with all other Pods in cluster Namespaces. In time it became clear that it was not an ideal design, and mechanisms needed to be put in place in order to restrict communication between certain Pods and applications in the cluster Namespace. Network Policies are sets of rules which define how Pods are allowed to talk to other Pods and resources inside and outside the cluster. Pods not covered by any Network Policy will continue to receive unrestricted traffic from any endpoint.
Network Policies are very similar to typical Firewalls. They are designed to protect mostly assets located inside the Firewall but can restrict outgoing traffic as well based on sets of rules and policies.
Monitoring, Logging, and Troubleshooting
In Kubernetes, we have to collect resource usage data by Pods, Services, nodes, etc., to understand the overall resource consumption and to make decisions for scaling a given application. Two popular Kubernetes monitoring solutions are the Kubernetes Metrics Server and Prometheus.
• Metrics Server
Metrics Server is a cluster-wide aggregator of resource usage data - a relatively new feature in Kubernetes.
• Prometheus
Prometheus, now part of CNCF (Cloud Native Computing Foundation), can also be used to scrape the resource usage from different Kubernetes components and objects. Using its client libraries, we can also instrument the code of our application.
A popular method to collect logs is using Elasticsearch together with Fluentd with custom configuration as an agent on the nodes. Fluentd is an open source data collector, which is also part of CNCF.
Helm
To deploy a complex application, we use a large number of Kubernetes manifests to define API resources such as Deployments, Services, PersistentVolumes, PersistentVolumeClaims, Ingress, or ServiceAccounts. It can become counter productive to deploy them one by one. We can bundle all those manifests after templatizing them into a well-defined format, along with other metadata. Such a bundle is referred to as Chart. These Charts can then be served via repositories, such as those that we have for rpm and deb packages.
Helm is a package manager (analogous to yum and apt for Linux) for Kubernetes, which can install/update/delete those Charts in the Kubernetes cluster.
Helm is a CLI client that may run side-by-side with kubectl on our workstation, that also uses kubeconfig to securely communicate with the Kubernetes API server.
The helm client queries the Chart repositories for Charts based on search parameters, downloads a desired Chart, and then it requests the API server to deploy in the cluster the resources defined in the Chart. Charts submitted for Kubernetes are available here.
No comments:
Post a Comment