Skip to main content

Ingress

The ingress system of Kubernetes is specifically designed to route external HTTP and HTTPS traffic into the cluster. It is composed of the ingress resource itself and a ingress controller which implements the needed logic. We offer a managed controller in the form of Nginx Ingress that you can choose to deploy to your NKE cluster in Cockpit. You can also control various features by adding annotations to your ingress object.

Availability​

Nginx Ingress is available as an optional service for NKE. It can be deployed on an existing NKE cluster using Cockpit.

Usage​

The basic usage and structure of an ingress resource is documented in the official Kubernetes documentation.

Ingress DNS Name​

We also provide a DNS name which will always point to your nginx ingress controllers IP. You can use it to point your own domain hostnames to the ingress. DNS is already set up, you will find the name in the Nginx Ingress view in Cockpit.

To use it just create a Alias or CNAME record in your own domain and point it to our provided ingress DNS.

Wildcard DNS Name​

Additionally we also create a wildcard DNS record *.<ingress-dns-name>. It is meant for quick application tests in development. You can use any hostname of that wildcard zone in your ingress resources to quickly get something up and running without requiring to mess with DNS records.

Automatic TLS Certificates​

NKE ships with cert-manager and Let's Encrypt preconfigured, so securing an ingress with free TLS certificates is as simple as adding an annotation.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    # this tells cert-manager to issue a certificate for
    # all hosts specified in the tls section
    kubernetes.io/tls-acme: "true"
  name: hello-ingress
spec:
  ingressClassName: nginx
  rules:
    - host: app.example.org
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: hello-ingress
                port:
                  number: 80
  tls:
    - hosts:
        - app.example.org
      secretName: hello-ingress

Access Logs​

The access logs of your ingress requests can be viewed in your Grafana instance in the Loki Explore view. The ingress logs are available under the label app="ingress-nginx". To only get the logs of a specific ingress instance, you can filter by using the additional label ingress. The label is in the schema of <namespace>-<ingress-name>-<backend-port>. Here's an example query to get all the logs of the ingress frontend with the port 80 in the namespace shop-prod:

{app="ingress-nginx",ingress="shop-prod-frontend-80"}

Additionally the ingress logs can be filtered by these labels:

  • method the HTTP method of a request
  • status the HTTP status code of the request

For more information on the usage of Loki, refer to the specific support article.

Nginx ingress features​

The nginx ingress controller provides many features like rate limiting, IP filtering, custom default backends, temporary or permanent redirects, etc. All of the annotations which can be used to control those features can be found in the official nginx ingress controller documentation.

Documentation for the most used features can be found below.

Basic authentification​

You can add basic authentication to your ingress resource by providing the credentials in a Kubernetes secret. Here are some instructional steps:

1 set some env variables for easier processing

USERNAME=<YOUR USERNAME>
SECRET_NAMESPACE=<THE NAMESPACE FOR THE SECRET>
INGRESS_NAMESPACE=<THE NAMESPACE OF YOUR INGRESS RESOURCE>
INGRESS=<THE NAME OF YOUR INGRESS RESOURCE>

2 create the Kubernetes secret which contains the credentials for basic auth. It can also be created in a different namespace than your ingress resource is stored. You will need the mkpasswd tool installed locally (can be found in the whois package in Debian/Ubuntu).

kubectl create secret generic basic-auth-secret --namespace=$SECRET_NAMESPACE --from-literal=auth=$USERNAME:$(mkpasswd -m sha-512)

3 add some annotations to your ingress object

kubectl --namespace=$INGRESS_NAMESPACE annotate ingress $INGRESS nginx.ingress.kubernetes.io/auth-type=basic
kubectl --namespace=$INGRESS_NAMESPACE annotate ingress $INGRESS nginx.ingress.kubernetes.io/auth-secret=$SECRET_NAMESPACE/basic-auth-secret
kubectl --namespace=$INGRESS_NAMESPACE annotate ingress $INGRESS nginx.ingress.kubernetes.io/auth-realm='Authentication required'

Rate limiting​

You have various ways of putting rate limits on your ingresses. All available options are documented in the official nginx ingress docs.

Temporary and persistent redirects​

To enable a temporary redirect to another URL for your ingress you can use the following annotation:

nginx.ingress.kubernetes.io/temporal-redirect: <YOUR URL>

The redirect will use the HTTP status code of 302.

If you want to have a permanent redirect you can use:

nginx.ingress.kubernetes.io/permanent-redirect: <YOUR URL>

HTTPS redirect​

If TLS is enabled for the given ingress, the nginx ingress controller will automatically redirect to the equivalent HTTPS URL of the ingress. To disable this redirect use:

nginx.ingress.kubernetes.io/ssl-redirect: "false"

IP whitelisting​

You can whitelist the IP addresses which are allowed to connect to your ingress resource. You can specify them in CIDR notation in the following annotation:

nginx.ingress.kubernetes.io/whitelist-source-range: <YOUR CIDR RANGE>

Caching​

The nginx ingress controllers allows to enable basic caching of backend resources, which can be particularly useful for static content. You can enable caching on specific ingress resources by setting the following annotations in your ingress definition (metadata.annotations):

nginx.ingress.kubernetes.io/proxy-buffering: "on"
nginx.ingress.kubernetes.io/configuration-snippet: |
  proxy_cache static-cache;
  proxy_cache_valid 10m;
  proxy_cache_use_stale error timeout updating http_404 http_500 http_502 http_503 http_504;
  proxy_cache_bypass $http_x_purge;
  add_header X-Cache-Status $upstream_cache_status;

In this example, the cache is invalidated after 10 minutes, and only HTTP status codes 200, 301 and 302 are cached. If another behaviour should be desired, proxy_cache_valid can also take a list of status codes in front of the time:

proxy_cache_valid 404 1m;

would cache 404 responses for one minute. There is also the special code any, which can be specified to cache any responses. Additionally, multiple proxy_cache_valid statements can be added on one ingress to specify different cache times for different status codes.

The default cache size is 100MB. Should you have other requirements, please do not hesitate to get in contact with us.

To check that the caching worked, you can use cURL to inspect the header of the returned resource. For that, you will need to execute the command twice, and at the second request, the resource should be cached:

curl --head <URL>

should print the header x-cache-status: HIT.

If you want to enable caching only for a sub-path of your application (for example an endpoint /static), you will need to create two separate ingress resources. To further configure caching, you can use the proxy_cache options that are valid for location blocks. These options can be found here.

Custom default backend​

The default backend is responsible for showing a 404 error page if a request arrives on the nginx ingress controller for which no ingress rule was specified. You can create an own custom default backend (+ kubernetes service) and refer to it on your ingress object.

The default backend only has 2 requirements:

  • it needs to serve a 404 page/code on the path /
  • it needs to serve a 200 HTTP code on the path /healthz

The implementation of the default nginx ingress controller backend can be found here.

Once you built and deployed your default backend service in the same namespace as your ingress resource you can refer to it via the following annotation on your ingress:

nginx.ingress.kubernetes.io/default-backend: <SERVICE NAME OF YOUR DEFAULT BACKEND>
Additional custom error pages​

To be able to additionally display custom error pages on the default backend (for example if the service your ingress points to is not available) you can use the following annotation:

nginx.ingress.kubernetes.io/custom-http-errors: <ERROR CODES> # for example: "404,415,503"

The nginx ingress controller will forward error information via certain HTTP headers to your default backend, which then can return the best possible error representation. More information about this can be found in the official documentation. An example of a default backend which can display custom error pages can be found here.

Static IPs for ingress and egress​

The IP of every ingress controller is static. This means all ingress resources using that controller will share that IP. In addition, it is possible to deploy multiple ingress controllers per NKE cluster and you can choose the controller with the ingressClass field in the ingress resource.

While ingress IPs remain static, it's important to note that egress IPs are currently dynamic due to operational considerations and therefore it is not possible to have static egress IPs. Specifically, during node replacements, such as underlying OS version upgrades, NKE initiates the creation of a new node. The existing workloads are seamlessly moved to the new node, which then inherits a new IP address. As a result, the egress IPs are subject to change in this process. However, we are working on a solution to make egress IPs static in future.

## IP ranges for egress

We currently use the following IP ranges for our NKE clusters:

178.209.59.64/26 185.88.237.64/26 94.230.211.128/26

You can use this to implement IP-based access for external services.