Log Forwarding allows you to send all logs of your containers running on GKE to an existing centralized logging solution.

Details

We deploy a Fluent Bit instance that forwards all your pod logs to a predefined address. On this address you need to run a receiver that supports the Fluentd Forward Protocol. Typically this would be Fluent Bit for simple use-cases or Fluentd for more flexibility. Usually this receiver should run within the GKE cluster. If you have the need to forward the logs to an external host, contact us to discuss the options.

Availability

Log Forwarding is available as an additional service with nine Managed GKE.

Usage

Let us know to which address we should forward the logs to. After we complete the installation of the service, the details will be shown on runway.

Example receiver with Fluentd

Here’s a basic example for a possible receiver that uses Fluentd and sends the logs to stdout. This configuration is just for debugging purposes to show that logs are coming in properly. In a real world scenario the logs should be forwarded to something like a Elasticsearch instance.

This example creates a ConfigMap, a Service and a Deployment in the namespace logging. All the logs that we send will start with the tag kube. so you have to create a match kube.** to select all the logs. With this configuration the receiver address results in fluentd.logging.svc:24224, which is what you would need to tell us so we can forward the logs to that address.

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: logging
data:
  fluent.conf: |-
    <source>
      @type forward
      port 24224
      bind 0.0.0.0
    </source>

    <match kube.**>
      # here you would usually configure something like elasticsearch
      @type stdout
    </match>
---
apiVersion: v1
kind: Service
metadata:
  name: fluentd
  namespace: logging
spec:
  selector:
    app: fluentd
  ports:
    - protocol: TCP
      port: 24224
      targetPort: 24224
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fluentd
  namespace: logging
spec:
  replicas: 1
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
      annotations:
        # we exclude this pod so we don't create a logging loop
        fluentbit.io/exclude: "true"
    spec:
      volumes:
      - name: fluentd-config
        configMap:
          name: fluentd-config
      containers:
      - name: fluentd
        image: fluent/fluentd:v1.9
        ports:
          - containerPort: 24224
        volumeMounts:
        - name: fluentd-config
          mountPath: /fluentd/etc

Documentation and Links

• Fluentd forward input
• Fluent Bit forward input