Cluster configuration

There are a lot of options possible for your cluster. Most options have a sane default however could be overridden on request.

A default cluster comes with 3 control plane and 3 worker nodes. To connect all nodes we create a network, default (10.128.0.0/22). We also deploy monitoring to ensure functionality of all cluster components. However most things are just a default and could be overridden.

Common options

Nodes

The standard configuration consists of the following:

• Three control plane nodes, one in each of our availability zones. Flavor: v2-c2-m8-d80
• Three worker nodes, one in each of our availability zones, in a single nodegroup. Flavor: v2-c2-m8-d80

Minimal configuration

• Three control plane nodes, one in each of our availability zones. Flavor: v2-c2-m8-d80

• One worker node, Flavor: v2-c2-m8-d80

  This is the minimal configuration offered. Scaling to larger flavors and adding nodes are supported. Autoscaling is not supported with a single worker node.

  Note: SLA is different for minimal configuration type of cluster. SLA’s can be found here.

Nodegroups and multiple flavors

A nodegroup contains of one or multiple nodes with the same flavor and a list of availability zones to deploy nodes in. Clusters are default delivered with a single nodegroup containing 3 nodes, one in each AZ. Each nodegroup is limited to one flavor.

You could have multiple nodegroups, if you for example want to target workload on separate nodes or in case you wish to consume multiple flavors.

A few examples of nodegroups:

Name	Flavour	AZ list	Min node count	Max node count (autoscaling)
worker	v2-c2-m8-d80	STO1, STO2, STO3	3	0
database	d2-c8-m120-d1.6k	STO1, STO2, STO3	3	0
frontend	v2-c4-m16-d160	STO1, STO2, STO3	3	12
jobs	v2-c4-m16-d160	STO1	1	3

In the examples we could see worker our default nodegroup and an example of having separate nodes for databases and frontend where the database is running on dedicated nodes and the frontend is running on smaller nodes but can autoscale between 3 and 12 nodes based on current cluster request. We also have a jobs nodegroup where we have one node in sto1 but can scale up to 3 nodes where all are placed inside STO1. You can read more about autoscaling here.

Nodegroups can be changed at any time. Please also note that we have auto-healing meaning in case any of your nodes for any reason stops working we will replace them. More about autohealing could be found here.

Worker nodes Floating IPs

By default, our clusters come with nodes that do not have any Floating IPs attached to them. If, for any reason, you require Floating IPs on your workload nodes, please inform us, and we can configure your cluster accordingly. It’s worth noting that the most common use case for Floating IPs is to ensure predictable source IPs. However, please note that enabling or disabling Floating IPs will necessitate the recreation of all your nodes.

Since during upgrades we create a new node prior to removing an old node you would need to have an additional IP adress on standby. Thus, for a 3 worker nodes, with with autoscaling up to 5 nodes, we will allocate 6 IPs.

Firewall rules per nodegroup

By default all nodes in a cluster share the same firewall rules. If you need to reach the nodes in one nodegroup directly, without opening the same access to every other node in the cluster, we can add firewall rules that apply to that nodegroup only. The most common reason to want this is a nodegroup whose nodes have Floating IPs, where a service is reached on the node address rather than through a load balancer.

This is not enabled by default. Contact support and tell us which nodegroup the rules belong to and what should be allowed, and we will configure it for you.

Each rule consists of:

• A direction, either incoming traffic to the node or outgoing traffic from it. Incoming is the usual case.
• A protocol, TCP, UDP or ICMP. A rule may also be written without a protocol, in which case it matches all of them.
• A port or port range, for TCP and UDP. For ICMP you give a type, and optionally a code.
• The address range the traffic comes from, written as a CIDR. We recommend narrowing this to the networks that actually need the access rather than leaving it open.
• An optional description, so that whoever looks at the rule later can tell what it is for.

A few things worth knowing before you ask for them:

• The rules apply to the nodes in the nodegroup you name, and to no other nodes in the cluster. If you want the same access on several nodegroups, each one needs its own rules.
• We create and manage the firewall for you. You cannot point us at a security group you have already created in your own project and have us attach it.
• Outgoing rules only ever add to what a node is allowed to send. They cannot be used to restrict outgoing traffic, because nodes are allowed to send anything by default. If you need to restrict what leaves your nodes, talk to us about it separately.
• There is a limit of 32 rules per nodegroup. If you find yourself wanting more than that, it is usually a sign the access belongs in front of the cluster rather than on the nodes, and we are happy to look at it with you.
• A description is recorded when the rule is created and is not updated afterwards. If the text has to change, we remove the rule and add it again.
• Rules can be added, changed and removed at any time, and doing so does not replace your nodes.

Note that Floating IPs are a separate thing to ask for, covered under Worker nodes Floating IPs above. A firewall rule opens access to an address; it does not give the node one.

Network

By default we create a node network (10.128.0.0/22). However we could use another subnet per customer request. The most common scenario is when customer request another subnet is when exposing multiple Kubernetes clusters over a VPN.

Please make sure to inform us if you wish to use a custom subnet during the ordering process since we cannot replace the network after creation, meaning we would then need to recreate your entire cluster.

We currently only support cidr in the 10.0.0.0/8 subnet range and at least a /24. Both nodes and loadbalancers are using IPs for this range meaning you need to have a sizable network from the beginning.

Cluster domain

We default all clusters to “cluster.local”. If you wish to have another cluster domain please let us know during the ordering procedure since it cannot be replaced after cluster creation.

OIDC

If you wish to integrate with your existing OIDC compatible IDP, example Microsoft AD And Google Workspace that is supported directy in the kubernetes api service.

By default we ship clusters with this option disabled however if you wish to make use of OIDC just let us know when order the cluster or afterwards. OIDC can be enabled, disabled or changed at any time.

Kubelet configurations and resource reservations

We make a few adaptations to Kubernetes vanilla settings.

• NodeDrainVolume and NodeDrainTimeout: 5 -> 15min

  • Increased duration to 15 minutes to allow more time for graceful shutdown and controlled startup of workload on new nodes, while respecting PodDisruptionBudgets.

• podPidsLimit: 0 → 4096

  • Added safety net of a maximum of Per-pod PIDs (process IDs), that is limited and enforced by the kubelet. We used to not have any limitation. Setting this to 4096 limits how many PIDs a single pod may create, which helps mitigate runaway processes or fork-bombs.

• serializeImagePulls: true → false

  • Allows the kubelet to pull multiple images in parallel, speeding up startup times.

• maxParallelImagePulls: 0 → 10

  • Controls the maximum number of image pulls the kubelet will perform in parallel.

Resource reservations on worker nodes

To improve stability and predictability of the core Kubernetes functionality during heavy load, we introduce node reservations for CPU, memory, and ephemeral storage.

The reservation model follows proven hyperscaler formulas but is tuned conservatively, ensuring more allocatable resources.

Hyperscalers tend to not make a distinction of systemReserved and kubeReserved, and bundle all reservations into and kubeReserved. We make use of both, but skewed towards kube reservations to align closer with Hyperscalers, but still maintain the reservations of the system. We calculate the reservations settings based on cpu cores, memory and storage of each flavor dynamically.

Here we’ve provided a sample of what to expect:

CPU Reservations Table

Cores (int)	System reserved (millicores)	Kube reserved (millicores)	Allocatable of node (%)
2	35	120	92%
4	41	180	94%
8	81	240	96%
16	83	320	97%
32	88	480	98%
64	98	800	99%

Memory Reservations

Memory (Gi)	System reserved (Gi)	Kube reserved (Gi)	Reserved total (Gi)	Eviction Soft (Gi)	Eviction Hard (Gi)	Allocatable of node (%)
8	0.4	1.0	1.4	0.00	0.25	79%
16	0.4	1.8	2.2	0.00	0.25	85%
32	0.4	3.4	3.8	0.00	0.25	87%
64	0.4	3.7	4.1	0.00	0.25	93%
120	0.4	4.3	4.7	0.00	0.25	96%
240	0.4	4.5	4.9	0.00	0.25	98%
384	0.4	6.9	7.3	0.00	0.25	98%
512	0.4	8.2	8.6	0.00	0.25	98%

Ephemeral Disk Reservations

NOTE: We use the default of nodefs.available at 10%.

Storage (Gi)	System reserved (Gi)	Kube reserved (Gi)	Reserved total (Gi)	Eviction Soft (Gi)	Eviction Hard (Gi)	Allocatable of node (%)
60	12.0	1.0	13.0	0.0	6.0	68%
80	12.0	1.0	13.0	0.0	8.0	74%
120	12.0	1.0	13.0	0.0	12.0	79%
240	12.0	1.0	13.0	0.0	24.0	85%
1600	12.0	1.0	13.0	0.0	160.0	89%

Cluster add-ons

We currently offer managed cert-manager, NGINX Ingress and elx-nodegroup-controller.

Cert-manager

Cert-manager (link to cert-manager.io) helps you to manage TLS certificates. A common use case is to use lets-encrypt to “automatically” generate certificates for web apps. However the functionality goes much deeper. We also have usage instructions and have a guide if you wish to deploy cert-manager yourself.

Ingress

An ingress controller in a Kubernetes cluster manages how external traffic reaches your services. It routes requests based on rules, handles load balancing, and can integrate with cert-manager to manage TLS certificates. This simplifies traffic handling and improves scalability and security compared to exposing each service individually. We have a usage guide with examples that can be found here.

We have chosen to use ingress-nginx and to support ingress, we limit what custom configurations can be made per cluster. We offer two “modes”. One that we call direct mode, which is the default behavior. This mode is used when end-clients connect directly to your ingress. We also have a proxy mode for when a proxy (e.g., WAF) is used in front of your ingress. When running in proxy mode, we also have the ability to limit traffic from specific IP addresses, which we recommend doing for security reasons. If you are unsure which mode to use or how to handle IP whitelisting, just let us know and we will help you choose the best options for your use case.

If you are interested in removing any limitations, we’ve assembled guides with everything you need to install the same IngressController as we provide. This will give you full control. The various resources give configuration examples and instructions for lifecycle management. These can be found here.

elx-nodegroup-controller

The nodegroup controller lets you declaratively manage labels and taints across groups of nodes. It supports targeting nodes by explicit name or by matching name segments (useful for nodegroup-based naming conventions). See the nodegroup controller guide for usage examples and configuration options. The controller can also be found on GitHub if you wish to inspect the code or deploy it yourself.