Forwarding Logs without the Gateway

This document describes how to send application, infrastructure, and audit logs to the Loki Distributor as different tenants using Fluentd or Vector.

Please read the hacking guide before proceeding with the following instructions.

Note: This document only applies to OpenShift-based deployments.

Disclaimer: This document helps to connect the forwarder (fluentd or vector) to the LokiStack by going around the authentication gateway. This is not a normal configuration for an OpenShift-based deployments and should only be used for testing if going through the gateway is no option.

OpenShift Logging#

OpenShift Logging supports forwarding logs to an external Loki instance using fluentd or vector as log forwarders. The below step-by-step guide will help you to send application, infrastructure, and audit logs to the LokiStack through the Distributor endpoint. The steps remain same for both fluentd and vector.

In order to enable communication between the log forwarder and the distributor, follow these steps:

• Deploy the Loki Operator and a lokistack instance for OpenShift.

• Deploy the OpenShift Logging Operator from the Operator Hub or using the following command locally:

  make deploy-image deploy-catalog install
  

• Create a Cluster Logging instance in the openshift-logging namespace with only collection defined.

  For fluentd:

  apiVersion: logging.openshift.io/v1
  kind: ClusterLogging
  metadata:
    name: instance
    namespace: openshift-logging
  spec:
    collection:
      logs:
        type: fluentd
        fluentd: {}
  

  For vector:

  apiVersion: logging.openshift.io/v1
  kind: ClusterLogging
  metadata:
    name: instance
    namespace: openshift-logging
  spec:
    collection:
      logs:
        type: vector
        fluentd: {}
  

• By default, TLS is enabled on all components deployed by loki-operator. Because the service certificates are signed by a cluster-internal CA you need to set up a secret that enables the collector to validate the certificate returned by the distributor. The secret must exist in the openshift-logging namespace, and must have a key ca-bundle.crt.

  The CA certificate is part of a ConfigMap that gets created by loki-operator as part of the LokiStack. Unfortunately this ConfigMap can not be used directly and has to be converted to a Secret readable by the collector.

  Fetch the ca-bundle.crt using:

  kubectl -n openshift-logging get cm lokistack-dev-gateway-ca-bundle -o jsonpath="{.data.service-ca\.crt}" > <FILE_NAME>
  

  where <FILE_NAME> can be ca_bundle.crt and used directly to create secret in the next step.

• Once secret is fetched, create a new secret file:

  kubectl -n openshift-logging create secret generic loki-distributor-ca \
  --from-file=ca-bundle.crt=<PATH/TO/CA_BUNDLE.CRT>
  

  where <PATH/TO/CA_BUNDLE.CRT> is the file path where the ca_bundle.crt was copied to.

• Now create a ClusterLogForwarder CR to forward logs to LokiStack:

  apiVersion: logging.openshift.io/v1
  kind: ClusterLogForwarder
  metadata:
    name: instance
    namespace: openshift-logging
  spec:
    outputs:
     - name: loki-operator
       type: loki
       url: https://lokistack-dev-distributor-http.openshift-logging.svc:3100
       secret:
         name: loki-distributor-ca
       loki:
         tenantKey: log_type
    pipelines:
     - name: send-logs
       inputRefs:
       - application
       - audit
       - infrastructure
       outputRefs:
       - loki-operator
  

  Note: The tenantKey: log_type gets resolved as application, audit or infrastructure by fluentd and vector based on the type of logs being collected. This is later used as different tenants for storing logs in Loki.

Troubleshooting#

Log Entries Out of Order#

If the forwarder is configured to send too much data in a short span of time, Loki will back-pressure the forwarder and respond to the POST requests with 429 errors. In order to alleviate this, follow this documentation.