Hacking using make run

This document demonstrates how to use Loki Operator for development and testing locally on Kind and OpenShift using the make run command.

Note: This is helpful when you don’t want to deploy the Loki Operator image everytime you make slight changes to the operator code.

Hacking using kind#

kind is a tool for running local Kubernetes clusters using Docker container “nodes”. kind was primarily designed for testing Kubernetes itself, but may be used for local development or CI.

Requirements#

• Install kubectl or Openshift CLI for communicating with the cluster. The guide below will be using kubectl for the same.
• Create a running Kubernetes cluster using kind.

Installation of Loki Operator#

• Install the CRDs into the cluster:

  make install
  

  This will create a custom resource definition with the name lokistacks.loki.grafana.com which can be verified using:

  kubectl get crd lokistacks.loki.grafana.com
  

• Create a minio deployment in the cluster using:

  kubectl apply -k config/overlays/development/minio
  

  This creates minio’s deployment, service, pvc and secret in the default namespace.

• Now create a LokiStack instance using:

  kubectl apply -f hack/lokistack_dev.yaml
  

• Now run the operator locally on your machine:

  make run
  

  This will start the loki operator locally, recognize the LokiStack CRD instance and also creates distributor, compactor, ingester, querier and query-frontend components.

  Confirm that all components are up and running for deployments using:

  kubectl rollout status deployment/<DEPLOYMENT_NAME>  
  

  where <DEPLOYMENT_NAME> is the name of the deployment and can be found using:

  kubectl get deployments
  

  Confirm that all are up and running for statefulsets using:

  kubectl rollout status statefulset/<STATEFULSET_NAME>
  

  where <STATEFULSET_NAME> is the name of the statefulset and can be found using:

  kubectl get statefulsets
  

• If you make some changes to the operator’s code, then just stop the operator locally using CTRL + C, update the code and rerun the operator locally:

  make run
  

  This saves time by not deploying the operator again and again.

• When everything works fine, for the final testing deploy everything to the cluster using this document.

Cleanup#

• Stop the operator from running locally by pressing CTRL + C.

• Cleanup the LokiStack instance, CRDs, deployments on the cluster using:

  make uninstall
  

• Cleanup the minio deployment using:

  kubectl delete -k config/overlays/development/minio
  

Hacking on OpenShift#

Requirements#

• Install kubectl or Openshift CLI for communicating with the cluster. The guide below will be using kubectl for the same.
• Create a running OpenShift cluster on AWS.
• Create an S3 bucket in one of the AWS Regions.

Installation of Loki Operator#

• Install the CRDs into the cluster:

  make install
  

  This will create a custom resource definition with the name lokistacks.loki.grafana.com which can be verified using:

  kubectl get crd lokistacks.loki.grafana.com
  

• Create the openshift-logging namespace in the cluster:

  kubectl create ns openshift-logging
  

• Now you need to create a storage secret for the operator. This can be done using:

  ./hack/deploy-aws-storage-secret.sh <BUCKET_NAME>
  

  This secret will be available in openshift-logging namespace. You can check the hack/deploy-aws-storage-secret.sh file to check the content of the secret. By default, the script will pull credential information using the aws cli. However, these values can be overwritten. For example:

  REGION=us-west-1 ./hack/deploy-aws-storage-secret.sh <BUCKET_NAME>
  

• Once the object storage secret is created, you can now create a LokiStack instance:

  kubectl -n openshift-logging apply -f hack/lokistack_dev.yaml
  

• Now run the operator locally on your machine:

  make run
  

  This will create distributor, compactor, ingester, querier and query-frontend components only.

  Confirm that all are up and running for deployments using:

  kubectl -n openshift-logging rollout status deployment/<DEPLOYMENT_NAME>
  

  where <DEPLOYMENT_NAME> is the name of the deployment and can be found using:

  kubectl -n openshift-logging get deployments
  

  Confirm that all are up and running for statefulsets using:

  kubectl -n openshift-logging rollout status statefulset/<STATEFULSET_NAME>  
  

  where <STATEFULSET_NAME> is the name of the statefulset and can be found using:

  kubectl -n openshift-logging get statefulsets
  

• If you want lokistack-gateway component [1] to be deployed then you need to create a gateway secret [2] for the operator. This can be done using:

  kubectl -n openshift-logging create secret generic test1 \
    --from-literal=clientID="<CLIENT_ID>" \
    --from-literal=clientSecret="<CLIENT_SECRET>" \
    --from-literal=issuerCAPath="<ISSUER_CA_PATH>"
  

• Now create a LokiStack instance using:

  kubectl -n openshift-logging apply -f hack/lokistack_gateway_dev.yaml
  

• Edit the main file to set the flag values to true and rerun the operator using:

  make run
  

  This will create distributor, compactor, ingester, querier, query-frontend and lokistack-gateway components.

• If you make some changes to the operator’s code, then just stop the operator locally using CTRL + C, update the code and rerun the operator locally:

  make run
  

  This saves time by not deploying the operator again and again.

• When everything works fine, for the final testing deploy everything to the cluster using this document.

Cleanup#

• Stop the operator from running locally by pressing CTRL + C.

• Cleanup the LokiStack instance, CRDs, deployments on the cluster using:

  make uninstall
  

Notes#

[1] lokistack-gateway is an optional component deployed as part of Loki Operator. It provides secure access to Loki’s distributor (i.e. for pushing logs) and query-frontend (i.e. for querying logs) via consulting an OAuth/OIDC endpoint for the request subject.

[2] The OIDC configuration expects clientID, clientSecret and issuerCAPath which should be provided via a Kubernetes secret that the LokiStack admin provides upfront.

Each tenant Secret is required to match:

• metadata.name with TenantsSecretsSpec.Name.
• metadata.namespace with LokiStack.metadata.namespace.

Basic Troubleshooting on Hacking on Loki Operator#

kubectl using old context#

It is possible that when you use two different clusters - one is kind cluster and the other is OpenShift cluster, you might need to switch between clusters to test your changes. There is a possibility that once you switch between clusters, the kubectl might not switch the context automatically and hence you might need to do this manually to correctly communicate with your cluster.

• List all the available context:

  kubectl config get-contexts
  

  The * mark against the context shows the one in use currently.

• Set the context name you want to use now:

  kubectl config use-context $CONTEXTNAME
  

  where $CONTEXTNAME is the context name you want to use now from the previous step.

The Loki Operator giving Missing Secrets / Invalid Secrets error#

You have probably forgotten to create the gateway secrets because of which the operator runs in degraded condition. Follow the steps mentioned in the step-by-step guide to create the gateway secret first. Once done, you can now create the LokiStack instance.

Verify this by checking the conditions field:

kubectl get lokistack lokistack-dev -o yaml

For OpenShift, the above command would be:

kubectl -n openshift-logging get lokistack lokistack-dev -o yaml

The Loki Operator giving Mandatory Configuration / Incompatible Configuration error#

This usually happens when the LokiStack CR is wrongly configured for the lokistack-gateway. Please read the enhancement proposal to figure out the correct way to configure it.