Loki Operator

Forwarding Logs to LokiStack

This document will describe how to send application, infrastructure, audit and network logs to the LokiStack Gateway as different tenants using Promtail or Fluentd. The built-in gateway provides secure access to the distributor (and query-frontend) via consulting an OAuth/OIDC endpoint for the request subject.

Please read the hacking guide before proceeding with the following instructions.

Note: While this document will only give instructions for two methods of log forwarding into the gateway, the examples given in the Promtail and Fluentd sections can be extrapolated to other log forwarders.

OpenShift Logging

OpenShift Logging supports forwarding logs to an external Loki instance. This can also be used to forward logs to LokiStack gateway.

  • Deploy the Loki Operator and an lokistack instance with the gateway flag enabled.

  • Deploy the OpenShift Logging Operator from the Operator Hub or using the following command locally:

    make deploy-image deploy-catalog install

  • Create a Cluster Logging instance in the openshift-logging namespace with only collection defined.

    apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  name: instance
  namespace: openshift-logging
spec:
  collection:
    logs:
      type: fluentd
      fluentd: {}

  • The LokiStack Gateway requires a bearer token for communication with fluentd. Therefore, create a secret with token key and the path to the file.

    kubectl -n openshift-logging create secret generic lokistack-gateway-bearer-token \
  --from-literal=token="$(kubectl -n openshift-logging get secret logcollector-token --template='{{.data.token | base64decode}}')"  \
  --from-literal=ca-bundle.crt="$(kubectl -n openshift-logging get configmap openshift-service-ca.crt --template='{{index .data "service-ca.crt"}}')"

  • Create the following ClusterRole and ClusterRoleBinding which will allow the cluster to authenticate the user(s) submitting the logs:

    ---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: lokistack-dev-tenant-logs
rules:
- apiGroups:
  - 'loki.grafana.com'
  resources:
  - application
  - infrastructure
  - audit
  resourceNames:
  - logs
  verbs:
  - 'create'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: lokistack-dev-tenant-logs
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: lokistack-dev-tenant-logs
subjects:
- kind: ServiceAccount
  name: logcollector
  namespace: openshift-logging

  • Now create a ClusterLogForwarder CR to forward logs to LokiStack:

    apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  outputs:
   - name: loki-app
     type: loki
     url: https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/application
     secret:
       name: lokistack-gateway-bearer-token
   - name: loki-infra
     type: loki
     url: https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/infrastructure
     secret:
       name: lokistack-gateway-bearer-token
   - name: loki-audit
     type: loki
     url: https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit
     secret:
       name: lokistack-gateway-bearer-token
  pipelines:
   - name: send-app-logs
     inputRefs:
     - application
     outputRefs:
     - loki-app
   - name: send-infra-logs
     inputRefs:
     - infrastructure
     outputRefs:
     - loki-infra
   - name: send-audit-logs
     inputRefs:
     - audit
     outputRefs:
     - loki-audit

    Note: You can add/remove any pipeline from the ClusterLogForwarder spec in case if you want to limit the logs being sent.

Network Observability

Network Observability also require an external loki instance and is compatible with LokiStack Gateway. You must use a separate instance than openshift-logging one.

The Network Observability Operator can automatically install and configure dependent operators. However, if you need to configure these manually, follow the step below.

  • Deploy the Loki Operator and a dedicated lokistack for Network Observability:
  • open Openshift admin console
  • go to Operators -> OperatorHub and install Loki Operator from Red Hat catalog if not already installed
  • ensure you are in network-observability namespace
  • go to Operators -> Installed Operators -> Loki Operator -> LokiStack
  • click on Create LokiStack
  • set name to lokistack-network
  • set Object Storage -> Secret check object storage documentation
  • set Tenants Configuration -> Mode to openshift-network

  • Create the following ClusterRole and ClusterRoleBinding which allow flowlogs-pipeline and network-observability-plugin service accounts to read and write the network logs:

    ---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: lokistack-network-tenant-logs
rules:
- apiGroups:
  - 'loki.grafana.com'
  resources:
  - network
  resourceNames:
  - logs
  verbs:
  - 'get'
  - 'create'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: lokistack-network-tenant-logs
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: lokistack-network-tenant-logs
subjects:
- kind: ServiceAccount
  name: flowlogs-pipeline
  namespace: network-observability
- kind: ServiceAccount
  name: network-observability-plugin
  namespace: network-observability

  • Deploy the Network Observability Operator following the Getting started documentation either from Operator Hub or commands available in the repository.

  • Apply the following configuration in FlowCollector for network tenant of lokistack-network:

  loki:
    tenantID: network
    sendAuthToken: true
    url: 'https://lokistack-network-gateway-http.network-observability.svc.cluster.local:8080/api/logs/v1/network/'

Check config samples for all options.

Forwarding Clients

In order to enable communication between the client(s) and the gateway, follow these steps:

  1. Deploy the Loki Operator and an lokistack instance with the gateway flag enabled.

  2. Create a ServiceAccount to generate the Secret which will be used to authorize the forwarder.

kubectl -n openshift-logging create serviceaccount <SERVICE_ACCOUNT_NAME>

  1. Configure the forwarder and deploy it to the openshift-logging namespace.

  2. Create the following ClusterRole and ClusterRoleBinding which will allow the cluster to authenticate the user(s) submitting the logs:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: lokistack-dev-tenant-logs-role
rules:
- apiGroups:
  - 'loki.grafana.com'
  resources:
  - application
  - infrastructure
  - audit
  resourceNames:
  - logs
  verbs:
  - 'get'
  - 'create'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: lokistack-dev-tenant-logs-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: lokistack-dev-tenant-logs-role
subjects:
- kind: ServiceAccount
  name: "<SERVICE_ACCOUNT_NAME>"
  namespace: openshift-logging

Promtail

Promtail is an agent managed by Grafana which forwards logs to a Loki instance. The Grafana documentation can be consulted for configuring and deploying an instance of Promtail in a Kubernetes cluster.

To configure Promtail to send application, audit, and infrastructure logs, add the following clients to the Promtail configuration

clients:
  - # ...
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    tls_config:
      ca_file: /run/secrets/kubernetes.io/serviceaccount/service-ca.crt
    url: https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit/loki/api/v1/push
  - # ...
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    tls_config:
      ca_file: /run/secrets/kubernetes.io/serviceaccount/service-ca.crt
    url: https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/application/loki/api/v1/push
  - # ...
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    tls_config:
      ca_file: /run/secrets/kubernetes.io/serviceaccount/service-ca.crt
    url: https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/infrastructure/loki/api/v1/push

The rest of the configuration can be configured to the developer’s desire.

Fluentd

Loki can receive logs from Fluentd via the Grafana plugin.

The Fluentd configuration can be overrided to target the application endpoint to send those log types.

<match **>
  @type loki
  # ...
  bearer_token_file /var/run/secrets/kubernetes.io/serviceaccount/token
  ca_cert /run/secrets/kubernetes.io/serviceaccount/service-ca.crt
  url https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/application
</match>

Troubleshooting

Log Entries Out of Order

If the forwarder is configured to send too much data in a short span of time, Loki will back-pressure the forwarder and respond to the POST requests with 429 errors. In order to alleviate this, a few changes could be made to the spec:

  • Consider moving up a t-shirt size. This will bring in addition resources and have a higher ingestion rate.
kubectl -n openshift-logging edit lokistack

size: 1x.medium
  • Manually change the ingestion rate (global or tenant) can be changed via configuration changes to lokistack:
kubectl -n openshift-logging edit lokistack

limits:
    tenants:
        <TENANT_NAME>:
            IngestionLimits:
                IngestionRate: 15

where <TENANT_NAME> can be application, audit or infrastructure.

← Connect Grafana to an in-cluster LokiStack
Forwarding Logs without the Gateway →