Skip to main content

Log Forwarding

Log Forwarding allows you to send all logs of your containers running on GKE to an existing centralized logging solution.

Details

We deploy a Fluent Bit instance that forwards all your pod logs to a predefined address. On this address you need to run a receiver that supports the Fluentd Forward Protocol. Typically this would be Fluent Bit for simple use-cases or Fluentd for more flexibility. Usually this receiver should run within the GKE cluster. If you have the need to forward the logs to an external host, contact us to discuss the options.

Availability

Log Forwarding is available as an additional service with nine Managed GKE.

Usage

Let us know to which address we should forward the logs to. After we complete the installation of the service, the details will be shown on runway.

Example receiver with Fluentd

Here's a basic example for a possible receiver that uses Fluentd and sends the logs to stdout. This configuration is just for debugging purposes to show that logs are coming in properly. In a real world scenario the logs should be forwarded to something like a Elasticsearch instance.

This example creates a ConfigMap, a Service and a Deployment in the namespace logging. All the logs that we send will start with the tag kube. so you have to create a match kube.** to select all the logs. With this configuration the receiver address results in fluentd.logging.svc:24224, which is what you would need to tell us so we can forward the logs to that address.

apiVersion: v1
kind: ConfigMap
metadata:
name: fluentd-config
namespace: logging
data:
fluent.conf: |-
<source>
@type forward
port 24224
bind 0.0.0.0
</source>

<match kube.**>
# here you would usually configure something like elasticsearch
@type stdout
</match>
---
apiVersion: v1
kind: Service
metadata:
name: fluentd
namespace: logging
spec:
selector:
app: fluentd
ports:
- protocol: TCP
port: 24224
targetPort: 24224
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: fluentd
namespace: logging
spec:
replicas: 1
selector:
matchLabels:
app: fluentd
template:
metadata:
labels:
app: fluentd
annotations:
# we exclude this pod so we don't create a logging loop
fluentbit.io/exclude: "true"
spec:
volumes:
- name: fluentd-config
configMap:
name: fluentd-config
containers:
- name: fluentd
image: fluent/fluentd:v1.9
ports:
- containerPort: 24224
volumeMounts:
- name: fluentd-config
mountPath: /fluentd/etc