System Admin Share

Kubernetes CrashLoopBackOff: A Practical Debugging Playbook

noreply@blogger.com (LiveStream) — Fri, 17 Jul 2026 11:24:27 +0530

CrashLoopBackOff is one of the first pod states every Kubernetes operator learns to dread, and also one of the most misunderstood. It is not an error in itself — it is Kubernetes telling you that a container keeps starting, crashing, and being restarted, and that the kubelet is now deliberately waiting longer between each restart. This playbook walks through exactly how to read that signal, find the real root cause, and fix the most common ones, with the commands you actually run.

The key mindset shift: CrashLoopBackOff is a symptom, never a diagnosis. Your job is to get from the status to the crashed container's own logs and exit code, because that is where the truth lives.

What CrashLoopBackOff actually means

When a container exits, the kubelet restarts it according to the pod's restartPolicy (Always by default for Deployments). If the container keeps exiting, the kubelet applies an exponential back-off delay so it is not hammering a broken process: 10s, then 20s, 40s, 80s, and so on, capped at 5 minutes. While the pod is sitting in that penalty box, its status reads CrashLoopBackOff.

So the loop is: container starts → container exits (crash) → kubelet waits (back-off) → restarts → repeat. The RESTARTS count climbs, and the delay grows. Fix the reason the container exits and the loop ends on its own.

Prerequisites

kubectl configured against the target cluster and the right namespace (kubectl config current-context).
Read access to pods, logs, and events in that namespace.
The pod name — grab it from kubectl get pods.

Step 1: Confirm the state and the restart count

Start broad. List the pods and look at the STATUS and RESTARTS columns:

kubectl get pods -n my-namespace
kubectl get pod my-pod -n my-namespace -o wide

A high, steadily-climbing RESTARTS value confirms the loop. The -o wide view also shows the node — useful later if the problem turns out to be node-level (memory pressure, disk).

Step 2: Read the crashed container's logs

This is the single most important step, and the one people skip. A crashed container's current logs are often empty because it just restarted. You almost always want the previous instance's logs:

kubectl logs my-pod -n my-namespace
kubectl logs my-pod -n my-namespace --previous

If the pod has more than one container, target the crashing one explicitly with -c:

kubectl logs my-pod -n my-namespace -c app-container --previous

Nine times out of ten the application prints the reason it died — a stack trace, a "connection refused", a "config file not found", a missing environment variable — right here.

Step 3: Describe the pod for events and exit codes

If the logs are silent (the process died before it could log, or the crash is outside the app), describe the pod. Read the Last State block and the Events at the bottom:

kubectl describe pod my-pod -n my-namespace

Look for a section like Last State: Terminated with a Reason and an Exit Code. You can pull the exit code directly:

kubectl get pod my-pod -n my-namespace -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}{"\n"}'

And scan recent cluster events, newest last:

kubectl get events -n my-namespace --sort-by=.lastTimestamp

Exit codes worth memorising

Exit code	Usual meaning
0	Clean exit. If a long-running service exits 0, its command probably finished instead of staying up.
1	General application error — check the app logs.
126	Command found but not executable (permissions / wrong binary format).
127	Command not found — bad `command`/`args` or missing binary in the image.
137	Killed by SIGKILL (128+9). Almost always OOMKilled — out of memory.
139	SIGSEGV (128+11) — segmentation fault in the process.
143	SIGTERM (128+15) — graceful termination that the app did not handle in time.

CrashLoopBackOff vs other failing states

It is easy to waste time chasing the wrong problem because several pod states look similar in kubectl get pods. Knowing which one you have narrows the search immediately:

Status	What it really means
CrashLoopBackOff	The container starts but keeps exiting; the kubelet is backing off between restarts. The image is fine — the process dies. Look at logs and exit code.
Error	The container exited non-zero on its most recent run and has not yet entered back-off. Same investigation as CrashLoopBackOff, just earlier in the loop.
ImagePullBackOff / ErrImagePull	Kubernetes cannot even pull the image — wrong tag, private registry without credentials, or a typo. Nothing to do with your code.
CreateContainerConfigError	A referenced ConfigMap or Secret key is missing, so the container never starts. Fix the reference, not the app.
RunContainerError	The container runtime failed to start it — often a bad mount, device, or securityContext.

The practical rule: if the container runs and then dies, it is a CrashLoopBackOff investigation (logs + exit code). If it never runs, it is a pull or config problem, and the fix is in the manifest or the registry, not the application.

Common root causes and how to fix each

1. The application errors on startup

The most common cause. A bad database URL, an unreachable dependency, a malformed config, a failed migration — the app throws on boot and exits non-zero. The fix is driven entirely by the log line from Step 2. Reproduce locally with the same image and env if the message is cryptic.

2. A missing or wrong ConfigMap / Secret / env var

If the container needs a value that is not mounted or is empty, it often crashes immediately. Confirm the referenced objects actually exist:

kubectl get configmap,secret -n my-namespace
kubectl describe pod my-pod -n my-namespace | grep -A3 -i "env\|mount"

A CreateContainerConfigError in describe is a dead giveaway that a referenced ConfigMap/Secret key is missing.

3. Exit code 137 — OOMKilled

The container exceeded its memory limit and the kernel killed it. describe shows Reason: OOMKilled. Either the limit is too low or the app has a leak. Raise the limit (and request) sensibly:

resources:
  requests:
    memory: "256Mi"
  limits:
    memory: "512Mi"

Then confirm the node actually has headroom with kubectl describe node <node> before assuming the limit alone is the fix.

4. A failing liveness probe restarts a healthy app

An over-aggressive livenessProbe can kill an app that is simply slow to start, producing a crash loop where the app itself is fine. Give it time to boot and be realistic about the endpoint:

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 3

Symptom to watch for in describe: Liveness probe failed events immediately before each restart, with the app logs otherwise clean.

5. A bad command, args, or entrypoint

Exit code 127 (command not found) or 126 (not executable) points at the command/args in the manifest or a broken image entrypoint. Verify the binary exists and the path is right; a quick kubectl run debug pod with the same image and a shell entrypoint lets you poke around:

kubectl run debug --rm -it --image=my-image:tag --restart=Never -- sh

6. A dependency is not ready yet

If the app crashes because the database or a downstream service is not up, the honest fix is application-level ret/back-off, not fighting Kubernetes. As a stopgap, an initContainer that waits for the dependency, or a readiness gate on the dependency, keeps the loop from starting.

7. Permissions and read-only filesystems

Containers running as non-root (or with readOnlyRootFilesystem: true) crash when they try to write where they cannot. Check the securityContext and mount a writable emptyDir for scratch paths the app insists on writing.

Step 4: Verify the fix

After you change the manifest, roll it out and watch the pod stabilise — the RESTARTS count should stop climbing and the status should settle on Running:

kubectl apply -f deployment.yaml
kubectl rollout status deployment/my-app -n my-namespace
kubectl get pods -n my-namespace -w

Give it a couple of minutes past the longest back-off you saw. A pod that survives well beyond its previous crash interval is genuinely fixed, not just early in the next back-off window.

Key Takeaways

CrashLoopBackOff is a symptom. The real answer is always in the crashed container's logs or its exit code.
Always use kubectl logs --previous — the current instance's logs are usually empty right after a restart.
Exit code 137 = OOMKilled. Fix the memory limit or the leak; 127/126 mean a bad command or permissions.
An aggressive liveness probe can crash a healthy app. Tune initialDelaySeconds and failureThreshold.
Verify past the back-off window. Watch RESTARTS stop climbing before you call it fixed.

Frequently Asked Questions

What does CrashLoopBackOff mean in Kubernetes?

It means a container in the pod keeps starting and then exiting, and the kubelet is now waiting progressively longer (up to 5 minutes) between restart attempts. It is a status describing the restart loop, not the underlying cause of the crash.

How do I see the logs of a container that already crashed?

Use kubectl logs <pod> --previous to read the previous (crashed) instance. Add -c <container> for a specific container in a multi-container pod.

What causes exit code 137 in a pod?

Exit code 137 is the process being killed by SIGKILL (128 + 9). In Kubernetes this is almost always OOMKilled — the container exceeded its memory limit. Raise the memory limit or fix the memory leak.

Why does my healthy app keep restarting?

Usually a failing livenessProbe. If the app is slow to start or the probe path/port is wrong, Kubernetes kills a perfectly healthy container. Increase initialDelaySeconds/failureThreshold and confirm the probe endpoint actually responds.

What is the difference between CrashLoopBackOff and ImagePullBackOff?

They fail at different stages. ImagePullBackOff means Kubernetes cannot pull the container image at all (bad tag, missing registry credentials), so the container never starts. CrashLoopBackOff means the image pulled fine and the container ran, but the process keeps exiting. The first is a manifest/registry fix; the second is a logs-and-exit-code investigation.

🛒 Recommended gear on Amazon

Disclosure: some links above are affiliate links — if you buy through them I may earn a small commission at no extra cost to you. Thanks for supporting the channel!

Simplifying Helm Configurations for Grafana Deployment using classic loadbalancer

noreply@blogger.com (System Engineer) — Tue, 18 Jun 2024 17:56:00 +0530

🛒 Recommended gear on Amazon

Disclosure: some links above are affiliate links — if you buy through them I may earn a small commission at no extra cost to you. Thanks for supporting the channel!

Deploying Grafana using Helm can be complex, especially when managing the configuration of the Kubernetes service. Ensuring your configurations are accurate is crucial for the security and functionality of your deployment. This guide will help you understand the key components of both old and new configurations, highlight common mistakes, and ensure a smooth transition to the new configuration.

The AWS EKS Load Balancer Controller (AWS LB Controller) can sometimes change settings due to several reasons related to its management of Elastic Load Balancers (ELBs) and Network Load Balancers (NLBs) within an Amazon EKS cluster. Here are some common reasons why settings might be changed:

1. **Automatic Updates**: The AWS LB Controller automatically manages and updates load balancer configurations based on changes in the Kubernetes resources (like Services and Ingresses) or changes in associated annotations and configurations.

2. **Annotation Changes**: Annotations on Kubernetes resources (such as Services or Ingresses) can trigger updates or changes in the load balancer configuration. For example, modifying annotations related to SSL certificates, health checks, or backend protocols can lead to corresponding changes in the AWS load balancer settings.

3. **Resource Lifecycle Events**: Events such as scaling events (scaling up or down pods or nodes), changes in service configurations (like changing service types or annotations), or updates to the Kubernetes resources that affect load balancer behavior can cause the AWS LB Controller to adjust settings to maintain consistency or reflect the updated configurations.

4. **Integration with AWS Services**: The AWS LB Controller integrates closely with AWS services like ACM (AWS Certificate Manager) for SSL certificates, IAM (Identity and Access Management) for permissions, and CloudFormation for resource provisioning. Changes or updates in these AWS services or their configurations can influence how the AWS LB Controller manages and updates load balancer settings.

5. **Policy and Compliance**: AWS LB Controller might enforce certain policies or best practices by automatically adjusting settings. For example, ensuring that SSL certificates are correctly configured and up-to-date, enforcing health check configurations, or adhering to AWS security group rules.

6. **Bug Fixes and Enhancements**: Updates to the AWS LB Controller itself, including bug fixes, security patches, or enhancements, may result in changes to load balancer settings to improve stability, security, or performance.

7. **Operator Intervention**: In some cases, manual interventions by administrators or operators to change load balancer configurations via Kubernetes resources or directly through AWS APIs can lead to changes managed by the AWS LB Controller.

Understanding these reasons helps in troubleshooting and managing configurations effectively when using the AWS LB Controller with Amazon EKS. It's essential to monitor changes and understand how annotations, Kubernetes resource configurations, and AWS service integrations impact load balancer settings to maintain the desired operational state and security posture.

When setting up Grafana with Helm, the old configuration involved several critical settings to establish a LoadBalancer service. Here’s a detailed look at the old configuration:

 
service:
  enabled: true
  type: LoadBalancer
  loadBalancerIP: ""
  loadBalancerClass: ""

  loadBalancerAnnotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:eu-west-2:REDACTED:certificate/REDACTED"  # Replace Me******
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"  # Expose port 443 (HTTPS)
    service.beta.kubernetes.io/aws-load-balancer-ssl-protocol: "TLS"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"  # Forward traffic to port 80 internally
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "80"  # Health check on port 80
    service.beta.kubernetes.io/aws-load-balancer-security-groups: "sg-REDACTED"  # Replace Me******
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
    service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "grafana-service=true"
    service.beta.kubernetes.io/aws-load-balancer-security-policy: "ELBSecurityPolicy-TLS-1-2-2017-01"
  labels: {}
  portName: service
  appProtocol: ""

Key Components of the Old Configuration

enabled: Activates the service.
type: Specifies the service type as LoadBalancer.
loadBalancerIP and loadBalancerClass: Initially left empty, these can be set for specific IP addresses or classes.
loadBalancerSourceRanges: Defines the IP ranges allowed to access the service.
loadBalancerAnnotations: Contains various AWS Load Balancer settings such as SSL certificate ARN, scheme, ports, protocols, security groups, health checks, and resource tags.

Common Mistakes in the Old Configuration

Incorrect SSL Certificate ARN: Not replacing placeholder values.
Improper Security Groups: Failing to update security group IDs.
Misconfigured Source Ranges: Allowing unintended IP ranges access.

These mistakes can lead to security vulnerabilities and connectivity issues, making it essential to handle these configurations with care.

Transitioning to the New Configuration

The new configuration aims to streamline and improve the management of the LoadBalancer service. Here's the updated configuration:

yaml 
service:
  enabled: true
  type: LoadBalancer
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:eu-west-2:REDACTED:certificate/REDACTED"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
    service.beta.kubernetes.io/aws-load-balancer-ssl-protocol: "HTTPS"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
    service.beta.kubernetes.io/aws-load-balancer-security-groups: "sg-REDACTED"
    service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "grafana-service=true"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: "tcp"
  labels: {}
  ports:
    - name: http
      port: 80
      targetPort: 3000
      protocol: TCP
    - name: https
      port: 443
      targetPort: 3000
      protocol: TCP
extraExposePorts:
  - name: https
    port: 443
    targetPort: 3000
healthCheck:
  path: "/"  # Health check path for ALB to monitor Grafana health
  intervalSeconds: 10
  timeoutSeconds: 5
  unhealthyThresholdCount: 6
  healthyThresholdCount: 2
externalTrafficPolicy: Cluster  # Use Cluster to preserve client source IP

Key Improvements in the New Configuration

annotations: Consolidates LoadBalancer settings into annotations, making the configuration cleaner.
ports: Defines both HTTP and HTTPS ports, specifying target ports and protocols.
extraExposePorts: Ensures HTTPS traffic is correctly routed to the appropriate target port.
healthCheck: Configures detailed health check settings for better monitoring.
externalTrafficPolicy: Uses the Cluster policy to preserve the client source IP.

Benefits of the New Configuration

Simplified Management: Consolidated settings reduce complexity.
Enhanced Security: Explicit annotations improve clarity and reduce the risk of misconfiguration.
Better Monitoring: Detailed health check settings allow for more effective monitoring of Grafana's health.

Conclusion

Transitioning from the old to the new configuration for Grafana deployment using Helm can significantly enhance your deployment’s security and manageability. By understanding and correctly implementing these configurations, you can avoid common pitfalls and ensure a robust and efficient setup.

Remember, always double-check sensitive data like SSL certificates and security group IDs to avoid common mistakes. With this guide, you’re now better equipped to handle these configurations and ensure a smooth and secure Grafana deployment.

https://github.com/grafana/helm-charts/issues/1251#issuecomment-2167881691

##
################################################################################################################

service:
enabled: true
type: LoadBalancer
annotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:eu-west-2:851465799644:certificate/f1"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
    service.beta.kubernetes.io/aws-load-balancer-ssl-protocol: "HTTPS"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
    service.beta.kubernetes.io/aws-load-balancer-security-groups: "sg-0bxxxxx"
    service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "grafana-service=true"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: "tcp"
labels: {}
ports:
    - name: http
      port: 80
      targetPort: 3000
      protocol: TCP
    - name: https
      port: 443
      targetPort: 3000
       protocol: TCP

extraExposePorts:
- name: https
port: 443
targetPort: 3000

healthCheck:
path: "/" # Health check path for ALB to monitor Grafana health
intervalSeconds: 10
timeoutSeconds: 5
unhealthyThresholdCount: 6
healthyThresholdCount: 2

externalTrafficPolicy: Cluster # Use Cluster to preserve client source IP

let's break down each section of the Helm chart configuration for the service component:

1. `service` Section

yaml

<?XML:NAMESPACE PREFIX = "[default] http://www.w3.org/2000/svg" NS = "http://www.w3.org/2000/svg" />Copy code

service: enabled: true type: LoadBalancer annotations: service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:eu-west-2:851465799644:certificate/f1" service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing" service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443" service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*" service.beta.kubernetes.io/aws-load-balancer-ssl-protocol: "HTTPS" service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http" service.beta.kubernetes.io/aws-load-balancer-security-groups: "sg-0bxxxxx" service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "grafana-service=true" service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: "tcp" labels: {}

enabled: true: Indicates that the service should be enabled.
type: LoadBalancer: Specifies that this service should be exposed externally using an AWS Elastic Load Balancer (ELB) configured by Kubernetes.
annotations: This section provides specific instructions to Kubernetes on how to configure the AWS ELB associated with this service:
- service.beta.kubernetes.io/aws-load-balancer-ssl-cert: Specifies the ACM (AWS Certificate Manager) ARN for the SSL certificate used for HTTPS.
- service.beta.kubernetes.io/aws-load-balancer-scheme: Specifies whether the ELB should be internet-facing or internal.
- service.beta.kubernetes.io/aws-load-balancer-ssl-ports: Specifies the ports on which SSL should be enabled.
- service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: Enables the Proxy Protocol to pass the client's information to the backend service.
- service.beta.kubernetes.io/aws-load-balancer-ssl-protocol: Specifies the SSL protocol to be used (HTTPS).
- service.beta.kubernetes.io/aws-load-balancer-backend-protocol: Specifies the protocol used between the ELB and the backend service (HTTP).
- service.beta.kubernetes.io/aws-load-balancer-security-groups: Specifies the AWS security group to associate with the ELB.
- service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: Adds additional AWS tags to the ELB for easier management.
- service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: Specifies the protocol used for health checks by the ELB (TCP).
labels: {}: This section allows you to specify Kubernetes labels for the service, which can be used for querying and selecting objects.

2. `ports` Section

yaml

Copy code

ports: - name: http port: 80 targetPort: 3000 protocol: TCP - name: https port: 443 targetPort: 3000 protocol: TCP

ports: Defines the ports that the service exposes:
- name: Friendly name for the port.
- port: Port number exposed on the ELB.
- targetPort: Port on the container where traffic is sent (in this case, Grafana's port).
- protocol: Protocol used (TCP in this case).

3. `extraExposePorts` Section

yaml

Copy code

extraExposePorts: - name: https port: 443 targetPort: 3000

extraExposePorts: Additional ports to expose beyond those defined in the ports section. In this case, it reiterates the HTTPS configuration, specifying port 443 to target Grafana's port 3000.

4. `healthCheck` Section

yaml

Copy code

healthCheck: path: "/" intervalSeconds: 10 timeoutSeconds: 5 unhealthyThresholdCount: 6 healthyThresholdCount: 2

healthCheck: Configures health checks for the AWS ELB:
- path: The endpoint path used for health checks (root path "/" in this case).
- intervalSeconds: Interval between health checks.
- timeoutSeconds: Timeout for each health check.
- unhealthyThresholdCount: Number of consecutive health check failures before considering the instance unhealthy.
- healthyThresholdCount: Number of consecutive health check successes before considering the instance healthy again.

5. `externalTrafficPolicy` Section

yaml

Copy code

externalTrafficPolicy: Cluster

externalTrafficPolicy: Specifies how the external traffic is routed to the service:
- Cluster: Preserves the client source IP, ensuring that the source IP is maintained when the traffic hits the backend service.

Summary

This Helm chart configuration snippet effectively sets up a Grafana service on AWS EKS with an AWS ELB configured for HTTPS, SSL termination, health checks, and traffic preservation policies. Each parameter plays a crucial role in ensuring the secure and reliable operation of Grafana within a Kubernetes cluster on AWS.

This breakdown provides insights into configuring AWS EKS load balancer settings via Helm for deploying Grafana securely with HTTPS support.

# Grafana Helm Chart

* Installs the web dashboarding system [Grafana](http://grafana.org/)

## Get Repo Info

  <br /> helm repo add grafana <a href="https://grafana.github.io/helm-charts">https://grafana.github.io/helm-charts</a>

  <br /> helm repo update

  <br />

_See [helm repo](https://helm.sh/docs/helm/helm_repo/) for command documentation._

## Installing the Chart

To install the chart with the release name `my-release`:

  <br /> helm install my-release grafana/grafana

  <br />

## Uninstalling the Chart

To uninstall/delete the my-release deployment:

  <br /> helm delete my-release

  <br />

The command removes all the Kubernetes components associated with the chart and deletes the release.

## Upgrading an existing Release to a new major version

A major chart version change (like v1.2.3 -> v2.0.0) indicates that there is an
incompatible breaking change needing manual actions.

### To 4.0.0 (And 3.12.1)

This version requires Helm >= 2.12.0.

### To 5.0.0

You have to add --force to your helm upgrade command as the labels of the chart have changed.

### To 6.0.0

This version requires Helm >= 3.1.0.

### To 7.0.0

For consistency with other Helm charts, the `global.image.registry` parameter was renamed
to `global.imageRegistry`. If you were not previously setting `global.image.registry`, no action
is required on upgrade. If you were previously setting `global.image.registry`, you will
need to instead set `global.imageRegistry`.

## Configuration

| Parameter                                 | Description                                   | Default                                                 |
|-------------------------------------------|-----------------------------------------------|---------------------------------------------------------|
| `replicas`                                | Number of nodes                               | `1`                                                     |
| `podDisruptionBudget.minAvailable`        | Pod disruption minimum available              | `nil`                                                   |
| `podDisruptionBudget.maxUnavailable`      | Pod disruption maximum unavailable            | `nil`                                                   |
| `podDisruptionBudget.apiVersion`          | Pod disruption apiVersion                     | `nil`                                                   |
| `deploymentStrategy`                      | Deployment strategy                           | `{ "type": "RollingUpdate" }`                           |
| `livenessProbe`                           | Liveness Probe settings                       | `{ "httpGet": { "path": "/api/health", "port": 3000 } "initialDelaySeconds": 60, "timeoutSeconds": 30, "failureThreshold": 10 }` |
| `readinessProbe`                          | Readiness Probe settings                      | `{ "httpGet": { "path": "/api/health", "port": 3000 } }`|
| `securityContext`                         | Deployment securityContext                    | `{"runAsUser": 472, "runAsGroup": 472, "fsGroup": 472}` |
| `priorityClassName`                       | Name of Priority Class to assign pods         | `nil`                                                   |
| `image.registry`                          | Image registry                                | `docker.io`                                       |
| `image.repository`                        | Image repository                              | `grafana/grafana`                                       |
| `image.tag`                               | Overrides the Grafana image tag whose default is the chart appVersion (`Must be >= 5.0.0`) | ``                                                      |
| `image.sha`                               | Image sha (optional)                          | ``                                                      |
| `image.pullPolicy`                        | Image pull policy                             | `IfNotPresent`                                          |
| `image.pullSecrets`                       | Image pull secrets (can be templated)         | `[]`                                                    |
| `service.enabled`                         | Enable grafana service                        | `true`                                                  |
| `service.type`                            | Kubernetes service type                       | `ClusterIP`                                             |
| `service.port`                            | Kubernetes port where service is exposed      | `80`                                                    |
| `service.portName`                        | Name of the port on the service               | `service`                                               |
| `service.appProtocol`                     | Adds the appProtocol field to the service     | ``                                                      |
| `service.targetPort`                      | Internal service is port                      | `3000`                                                  |
| `service.nodePort`                        | Kubernetes service nodePort                   | `nil`                                                   |
| `service.annotations`                     | Service annotations (can be templated)        | `{}`                                                    |
| `service.labels`                          | Custom labels                                 | `{}`                                                    |
| `service.clusterIP`                       | internal cluster service IP                   | `nil`                                                   |
| `service.loadBalancerIP`                  | IP address to assign to load balancer (if supported) | `nil`                                            |
| `service.loadBalancerSourceRanges`        | list of IP CIDRs allowed access to lb (if supported) | `[]`                                             |
| `service.externalIPs`                     | service external IP addresses                 | `[]`                                                    |
| `service.externalTrafficPolicy`           | change the default externalTrafficPolicy | `nil`                                            |
| `headlessService`                         | Create a headless service                     | `false`                                                 |
| `extraExposePorts`                        | Additional service ports for sidecar containers| `[]`                                                   |
| `hostAliases`                             | adds rules to the pod's /etc/hosts            | `[]`                                                    |
| `ingress.enabled`                         | Enables Ingress                               | `false`                                                 |
| `ingress.annotations`                     | Ingress annotations (values are templated)    | `{}`                                                    |
| `ingress.labels`                          | Custom labels                                 | `{}`                                                    |
| `ingress.path`                            | Ingress accepted path                         | `/`                                                     |
| `ingress.pathType`                        | Ingress type of path                          | `Prefix`                                                |
| `ingress.hosts`                           | Ingress accepted hostnames                    | `["chart-example.local"]`                                                    |
| `ingress.extraPaths`                      | Ingress extra paths to prepend to every host configuration. Useful when configuring [custom actions with AWS ALB Ingress Controller](https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.6/guide/ingress/annotations/#actions). Requires `ingress.hosts` to have one or more host entries. | `[]`                                                    |
| `ingress.tls`                             | Ingress TLS configuration                     | `[]`                                                    |
| `ingress.ingressClassName`                | Ingress Class Name. MAY be required for Kubernetes versions >= 1.18 | `""`                              |
| `resources`                               | CPU/Memory resource requests/limits           | `{}`                                                    |
| `nodeSelector`                            | Node labels for pod assignment                | `{}`                                                    |
| `tolerations`                             | Toleration labels for pod assignment          | `[]`                                                    |
| `affinity`                                | Affinity settings for pod assignment          | `{}`                                                    |
| `extraInitContainers`                     | Init containers to add to the grafana pod     | `{}`                                                    |
| `extraContainers`                         | Sidecar containers to add to the grafana pod | `""`                                                    |
| `extraContainerVolumes`                   | Volumes that can be mounted in sidecar containers | `[]`                                                |
| `extraLabels`                             | Custom labels for all manifests               | `{}`                                                    |
| `schedulerName`                           | Name of the k8s scheduler (other than default) | `nil`                                                  |
| `persistence.enabled`                     | Use persistent volume to store data           | `false`                                                 |
| `persistence.type`                        | Type of persistence (`pvc` or `statefulset`) | `pvc`                                                   |
| `persistence.size`                        | Size of persistent volume claim               | `10Gi`                                                  |
| `persistence.existingClaim`               | Use an existing PVC to persist data (can be templated) | `nil`                                          |
| `persistence.storageClassName`            | Type of persistent volume claim               | `nil`                                                   |
| `persistence.accessModes`                 | Persistence access modes                      | `[ReadWriteOnce]`                                       |
| `persistence.annotations`                 | PersistentVolumeClaim annotations             | `{}`                                                    |
| `persistence.finalizers`                  | PersistentVolumeClaim finalizers              | `[ "kubernetes.io/pvc-protection" ]`                    |
| `persistence.extraPvcLabels`              | Extra labels to apply to a PVC.               | `{}`                                                    |
| `persistence.subPath`                     | Mount a sub dir of the persistent volume (can be templated) | `nil`                                     |
| `persistence.inMemory.enabled`            | If persistence is not enabled, whether to mount the local storage in-memory to improve performance | `false`                                                   |
| `persistence.inMemory.sizeLimit`          | SizeLimit for the in-memory local storage     | `nil`                                                   |
| `initChownData.enabled`                   | If false, don't reset data ownership at startup | true                                                  |
| `initChownData.image.registry`            | init-chown-data container image registry      | `docker.io`                                               |
| `initChownData.image.repository`          | init-chown-data container image repository    | `busybox`                                               |
| `initChownData.image.tag`                 | init-chown-data container image tag           | `1.31.1`                                                |
| `initChownData.image.sha`                 | init-chown-data container image sha (optional)| `""`                                                    |
| `initChownData.image.pullPolicy`          | init-chown-data container image pull policy   | `IfNotPresent`                                          |
| `initChownData.resources`                 | init-chown-data pod resource requests & limits | `{}`                                                   |
| `schedulerName`                           | Alternate scheduler name                      | `nil`                                                   |
| `env`                                     | Extra environment variables passed to pods    | `{}`                                                    |
| `envValueFrom`                            | Environment variables from alternate sources. See the API docs on [EnvVarSource](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#envvarsource-v1-core) for format details. Can be templated | `{}` |
| `envFromSecret`                           | Name of a Kubernetes secret (must be manually created in the same namespace) containing values to be added to the environment. Can be templated | `""` |
| `envFromSecrets`                          | List of Kubernetes secrets (must be manually created in the same namespace) containing values to be added to the environment. Can be templated | `[]` |
| `envFromConfigMaps`                       | List of Kubernetes ConfigMaps (must be manually created in the same namespace) containing values to be added to the environment. Can be templated | `[]` |
| `envRenderSecret`                         | Sensible environment variables passed to pods and stored as secret. (passed through [tpl](https://helm.sh/docs/howto/charts_tips_and_tricks/#using-the-tpl-function))   | `{}`                               |
| `enableServiceLinks`                      | Inject Kubernetes services as environment variables. | `true`                                           |
| `extraSecretMounts`                       | Additional grafana server secret mounts       | `[]`                                                    |
| `extraVolumeMounts`                       | Additional grafana server volume mounts       | `[]`                                                    |
| `extraVolumes`                            | Additional Grafana server volumes             | `[]`                                                    |
| `automountServiceAccountToken`            | Mounted the service account token on the grafana pod. Mandatory, if sidecars are enabled | `true`      |
| `createConfigmap`                         | Enable creating the grafana configmap         | `true`                                                  |
| `extraConfigmapMounts`                    | Additional grafana server configMap volume mounts (values are templated) | `[]`                         |
| `extraEmptyDirMounts`                     | Additional grafana server emptyDir volume mounts | `[]`                                                 |
| `plugins`                                 | Plugins to be loaded along with Grafana       | `[]`                                                    |
| `datasources`                             | Configure grafana datasources (passed through tpl) | `{}`                                               |
| `alerting`                                | Configure grafana alerting (passed through tpl) | `{}`                                                  |
| `notifiers`                               | Configure grafana notifiers                   | `{}`                                                    |
| `dashboardProviders`                      | Configure grafana dashboard providers         | `{}`                                                    |
| `dashboards`                              | Dashboards to import                          | `{}`                                                    |
| `dashboardsConfigMaps`                    | ConfigMaps reference that contains dashboards | `{}`                                                    |
| `grafana.ini`                             | Grafana's primary configuration               | `{}`                                                    |
| `global.imageRegistry`                    | Global image pull registry for all images.    | `null`                                   |
| `global.imagePullSecrets`                 | Global image pull secrets (can be templated). Allows either an array of {name: pullSecret} maps (k8s-style), or an array of strings (more common helm-style). | `[]`                                                    |
| `ldap.enabled`                            | Enable LDAP authentication                    | `false`                                                 |
| `ldap.existingSecret`                     | The name of an existing secret containing the `ldap.toml` file, this must have the key `ldap-toml`. | `""` |
| `ldap.config`                             | Grafana's LDAP configuration                  | `""`                                                    |
| `annotations`                             | Deployment annotations                        | `{}`                                                    |
| `labels`                                  | Deployment labels                             | `{}`                                                    |
| `podAnnotations`                          | Pod annotations                               | `{}`                                                    |
| `podLabels`                               | Pod labels                                    | `{}`                                                    |
| `podPortName`                             | Name of the grafana port on the pod           | `grafana`                                               |
| `lifecycleHooks`                          | Lifecycle hooks for podStart and preStop [Example](https://kubernetes.io/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/#define-poststart-and-prestop-handlers)     | `{}`                                                    |
| `sidecar.image.registry`                  | Sidecar image registry                        | `quay.io`                          |
| `sidecar.image.repository`                | Sidecar image repository                      | `kiwigrid/k8s-sidecar`                          |
| `sidecar.image.tag`                       | Sidecar image tag                             | `1.26.0`                                                |
| `sidecar.image.sha`                       | Sidecar image sha (optional)                  | `""`                                                    |
| `sidecar.imagePullPolicy`                 | Sidecar image pull policy                     | `IfNotPresent`                                          |
| `sidecar.resources`                       | Sidecar resources                             | `{}`                                                    |
| `sidecar.securityContext`                 | Sidecar securityContext                       | `{}`                                                    |
| `sidecar.enableUniqueFilenames`           | Sets the kiwigrid/k8s-sidecar UNIQUE_FILENAMES environment variable. If set to `true` the sidecar will create unique filenames where duplicate data keys exist between ConfigMaps and/or Secrets within the same or multiple Namespaces. | `false`                           |
| `sidecar.alerts.enabled`             | Enables the cluster wide search for alerts and adds/updates/deletes them in grafana |`false`       |
| `sidecar.alerts.label`               | Label that config maps with alerts should have to be added | `grafana_alert`                               |
| `sidecar.alerts.labelValue`          | Label value that config maps with alerts should have to be added | `""`                                |
| `sidecar.alerts.searchNamespace`     | Namespaces list. If specified, the sidecar will search for alerts config-maps inside these namespaces. Otherwise the namespace in which the sidecar is running will be used. It's also possible to specify ALL to search in all namespaces. | `nil`                               |
| `sidecar.alerts.watchMethod`         | Method to use to detect ConfigMap changes. With WATCH the sidecar will do a WATCH requests, with SLEEP it will list all ConfigMaps, then sleep for 60 seconds. | `WATCH` |
| `sidecar.alerts.resource`            | Should the sidecar looks into secrets, configmaps or both. | `both`                               |
| `sidecar.alerts.reloadURL`           | Full url of datasource configuration reload API endpoint, to invoke after a config-map change | `"http://localhost:3000/api/admin/provisioning/alerting/reload"` |
| `sidecar.alerts.skipReload`          | Enabling this omits defining the REQ_URL and REQ_METHOD environment variables | `false` |
| `sidecar.alerts.initAlerts`          | Set to true to deploy the alerts sidecar as an initContainer. This is needed if skipReload is true, to load any alerts defined at startup time. | `false` |
| `sidecar.alerts.extraMounts`         | Additional alerts sidecar volume mounts. | `[]`                               |
| `sidecar.dashboards.enabled`              | Enables the cluster wide search for dashboards and adds/updates/deletes them in grafana | `false`       |
| `sidecar.dashboards.SCProvider`           | Enables creation of sidecar provider          | `true`                                                  |
| `sidecar.dashboards.provider.name`        | Unique name of the grafana provider           | `sidecarProvider`                                       |
| `sidecar.dashboards.provider.orgid`       | Id of the organisation, to which the dashboards should be added | `1`                                   |
| `sidecar.dashboards.provider.folder`      | Logical folder in which grafana groups dashboards | `""`                                                |
| `sidecar.dashboards.provider.folderUid`   | Allows you to specify the static UID for the logical folder above | `""`                                |
| `sidecar.dashboards.provider.disableDelete` | Activate to avoid the deletion of imported dashboards | `false`                                       |
| `sidecar.dashboards.provider.allowUiUpdates` | Allow updating provisioned dashboards from the UI | `false`                                          |
| `sidecar.dashboards.provider.type`        | Provider type                                 | `file`                                                  |
| `sidecar.dashboards.provider.foldersFromFilesStructure`        | Allow Grafana to replicate dashboard structure from filesystem.                                 | `false`                                                  |
| `sidecar.dashboards.watchMethod`          | Method to use to detect ConfigMap changes. With WATCH the sidecar will do a WATCH requests, with SLEEP it will list all ConfigMaps, then sleep for 60 seconds. | `WATCH` |
| `sidecar.skipTlsVerify`                   | Set to true to skip tls verification for kube api calls | `nil`                                         |
| `sidecar.dashboards.label`                | Label that config maps with dashboards should have to be added | `grafana_dashboard`                                |
| `sidecar.dashboards.labelValue`                | Label value that config maps with dashboards should have to be added | `""`                                |
| `sidecar.dashboards.folder`               | Folder in the pod that should hold the collected dashboards (unless `sidecar.dashboards.defaultFolderName` is set). This path will be mounted. | `/tmp/dashboards`    |
| `sidecar.dashboards.folderAnnotation`     | The annotation the sidecar will look for in configmaps to override the destination folder for files | `nil`                                                  |
| `sidecar.dashboards.defaultFolderName`    | The default folder name, it will create a subfolder under the `sidecar.dashboards.folder` and put dashboards in there instead | `nil`                                |
| `sidecar.dashboards.searchNamespace`      | Namespaces list. If specified, the sidecar will search for dashboards config-maps inside these namespaces. Otherwise the namespace in which the sidecar is running will be used. It's also possible to specify ALL to search in all namespaces. | `nil`                                |
| `sidecar.dashboards.script`               | Absolute path to shell script to execute after a configmap got reloaded. | `nil`                                |
| `sidecar.dashboards.reloadURL`            | Full url of dashboards configuration reload API endpoint, to invoke after a config-map change | `"http://localhost:3000/api/admin/provisioning/dashboards/reload"` |
| `sidecar.dashboards.skipReload`           | Enabling this omits defining the REQ_USERNAME, REQ_PASSWORD, REQ_URL and REQ_METHOD environment variables | `false` |
| `sidecar.dashboards.resource`             | Should the sidecar looks into secrets, configmaps or both. | `both`                               |
| `sidecar.dashboards.extraMounts`          | Additional dashboard sidecar volume mounts. | `[]`                               |
| `sidecar.datasources.enabled`             | Enables the cluster wide search for datasources and adds/updates/deletes them in grafana |`false`       |
| `sidecar.datasources.label`               | Label that config maps with datasources should have to be added | `grafana_datasource`                               |
| `sidecar.datasources.labelValue`          | Label value that config maps with datasources should have to be added | `""`                                |
| `sidecar.datasources.searchNamespace`     | Namespaces list. If specified, the sidecar will search for datasources config-maps inside these namespaces. Otherwise the namespace in which the sidecar is running will be used. It's also possible to specify ALL to search in all namespaces. | `nil`                               |
| `sidecar.datasources.watchMethod`         | Method to use to detect ConfigMap changes. With WATCH the sidecar will do a WATCH requests, with SLEEP it will list all ConfigMaps, then sleep for 60 seconds. | `WATCH` |
| `sidecar.datasources.resource`            | Should the sidecar looks into secrets, configmaps or both. | `both`                               |
| `sidecar.datasources.reloadURL`           | Full url of datasource configuration reload API endpoint, to invoke after a config-map change | `"http://localhost:3000/api/admin/provisioning/datasources/reload"` |
| `sidecar.datasources.skipReload`          | Enabling this omits defining the REQ_URL and REQ_METHOD environment variables | `false` |
| `sidecar.datasources.initDatasources`     | Set to true to deploy the datasource sidecar as an initContainer in addition to a container. This is needed if skipReload is true, to load any datasources defined at startup time. | `false` |
| `sidecar.notifiers.enabled`               | Enables the cluster wide search for notifiers and adds/updates/deletes them in grafana | `false`        |
| `sidecar.notifiers.label`                 | Label that config maps with notifiers should have to be added | `grafana_notifier`                               |
| `sidecar.notifiers.labelValue`            | Label value that config maps with notifiers should have to be added | `""`                                |
| `sidecar.notifiers.searchNamespace`       | Namespaces list. If specified, the sidecar will search for notifiers config-maps (or secrets) inside these namespaces. Otherwise the namespace in which the sidecar is running will be used. It's also possible to specify ALL to search in all namespaces. | `nil`                               |
| `sidecar.notifiers.watchMethod`           | Method to use to detect ConfigMap changes. With WATCH the sidecar will do a WATCH requests, with SLEEP it will list all ConfigMaps, then sleep for 60 seconds. | `WATCH` |
| `sidecar.notifiers.resource`              | Should the sidecar looks into secrets, configmaps or both. | `both`                               |
| `sidecar.notifiers.reloadURL`             | Full url of notifier configuration reload API endpoint, to invoke after a config-map change | `"http://localhost:3000/api/admin/provisioning/notifications/reload"` |
| `sidecar.notifiers.skipReload`            | Enabling this omits defining the REQ_URL and REQ_METHOD environment variables | `false` |
| `sidecar.notifiers.initNotifiers`         | Set to true to deploy the notifier sidecar as an initContainer in addition to a container. This is needed if skipReload is true, to load any notifiers defined at startup time. | `false` |
| `smtp.existingSecret`                     | The name of an existing secret containing the SMTP credentials. | `""`                                  |
| `smtp.userKey`                            | The key in the existing SMTP secret containing the username. | `"user"`                                 |
| `smtp.passwordKey`                        | The key in the existing SMTP secret containing the password. | `"password"`                             |
| `admin.existingSecret`                    | The name of an existing secret containing the admin credentials (can be templated). | `""`                                 |
| `admin.userKey`                           | The key in the existing admin secret containing the username. | `"admin-user"`                          |
| `admin.passwordKey`                       | The key in the existing admin secret containing the password. | `"admin-password"`                      |
| `serviceAccount.automountServiceAccountToken` | Automount the service account token on all pods where is service account is used | `false` |
| `serviceAccount.annotations`              | ServiceAccount annotations                    |                                                         |
| `serviceAccount.create`                   | Create service account                        | `true`                                                  |
| `serviceAccount.labels`                   | ServiceAccount labels                         | `{}`                                                    |
| `serviceAccount.name`                     | Service account name to use, when empty will be set to created account if `serviceAccount.create` is set else to `default` | `` |
| `serviceAccount.nameTest`                 | Service account name to use for test, when empty will be set to created account if `serviceAccount.create` is set else to `default` | `nil` |
| `rbac.create`                             | Create and use RBAC resources                 | `true`                                                  |
| `rbac.namespaced`                         | Creates Role and Rolebinding instead of the default ClusterRole and ClusteRoleBindings for the grafana instance | `false` |
| `rbac.useExistingRole`                    | Set to a rolename to use existing role - skipping role creating - but still doing serviceaccount and rolebinding to the rolename set here. | `nil` |
| `rbac.pspEnabled`                         | Create PodSecurityPolicy (with `rbac.create`, grant roles permissions as well) | `false`                |
| `rbac.pspUseAppArmor`                     | Enforce AppArmor in created PodSecurityPolicy (requires `rbac.pspEnabled`) | `false`                   |
| `rbac.extraRoleRules`                     | Additional rules to add to the Role           | []                                                      |
| `rbac.extraClusterRoleRules`              | Additional rules to add to the ClusterRole    | []                                                      |
| `command`                                 | Define command to be executed by grafana container at startup | `nil`                                   |
| `args`                                    | Define additional args if command is used     | `nil`                                                   |
| `testFramework.enabled`                   | Whether to create test-related resources      | `true`                                                  |
| `testFramework.image.registry`            | `test-framework` image registry.            | `docker.io`                                             |
| `testFramework.image.repository`          | `test-framework` image repository.            | `bats/bats`                                             |
| `testFramework.image.tag`                 | `test-framework` image tag.                   | `v1.4.1`                                                |
| `testFramework.imagePullPolicy`           | `test-framework` image pull policy.           | `IfNotPresent`                                          |
| `testFramework.securityContext`           | `test-framework` securityContext              | `{}`                                                    |
| `downloadDashboards.env`                  | Environment variables to be passed to the `download-dashboards` container | `{}`                        |
| `downloadDashboards.envFromSecret`        | Name of a Kubernetes secret (must be manually created in the same namespace) containing values to be added to the environment. Can be templated | `""` |
| `downloadDashboards.resources`            | Resources of `download-dashboards` container | `{}`                                                    |
| `downloadDashboardsImage.registry`        | Curl docker image registry                    | `docker.io`                                       |
| `downloadDashboardsImage.repository`      | Curl docker image repository                  | `curlimages/curl`                                       |
| `downloadDashboardsImage.tag`             | Curl docker image tag                         | `7.73.0`                                                |
| `downloadDashboardsImage.sha`             | Curl docker image sha (optional)              | `""`                                                    |
| `downloadDashboardsImage.pullPolicy`      | Curl docker image pull policy                 | `IfNotPresent`                                          |
| `namespaceOverride`                       | Override the deployment namespace             | `""` (`Release.Namespace`)                              |
| `serviceMonitor.enabled`                  | Use servicemonitor from prometheus operator   | `false`                                                 |
| `serviceMonitor.namespace`                | Namespace this servicemonitor is installed in |                                                         |
| `serviceMonitor.interval`                 | How frequently Prometheus should scrape       | `1m`                                                    |
| `serviceMonitor.path`                     | Path to scrape                                | `/metrics`                                              |
| `serviceMonitor.scheme`                   | Scheme to use for metrics scraping            | `http`                                                  |
| `serviceMonitor.tlsConfig`                | TLS configuration block for the endpoint      | `{}`                                                    |
| `serviceMonitor.labels`                   | Labels for the servicemonitor passed to Prometheus Operator      | `{}`                                |
| `serviceMonitor.scrapeTimeout`            | Timeout after which the scrape is ended       | `30s`                                                   |
| `serviceMonitor.relabelings`              | RelabelConfigs to apply to samples before scraping.     | `[]`                                      |
| `serviceMonitor.metricRelabelings`        | MetricRelabelConfigs to apply to samples before ingestion. | `[]`                                      |
| `revisionHistoryLimit`                    | Number of old ReplicaSets to retain           | `10`                                                    |
| `imageRenderer.enabled`                    | Enable the image-renderer deployment & service                                     | `false`                          |
| `imageRenderer.image.registry`             | image-renderer Image registry                                                      | `docker.io` |
| `imageRenderer.image.repository`           | image-renderer Image repository                                                    | `grafana/grafana-image-renderer` |
| `imageRenderer.image.tag`                  | image-renderer Image tag                                                           | `latest`                         |
| `imageRenderer.image.sha`                  | image-renderer Image sha (optional)                                                | `""`                             |
| `imageRenderer.image.pullPolicy`           | image-renderer ImagePullPolicy                                                     | `Always`                         |
| `imageRenderer.env`                        | extra env-vars for image-renderer                                                  | `{}`                             |
| `imageRenderer.envValueFrom`               | Environment variables for image-renderer from alternate sources. See the API docs on [EnvVarSource](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#envvarsource-v1-core) for format details. Can be templated | `{}` |
| `imageRenderer.serviceAccountName`         | image-renderer deployment serviceAccountName                                       | `""`                             |
| `imageRenderer.securityContext`            | image-renderer deployment securityContext                                          | `{}`                             |
| `imageRenderer.podAnnotations `            | image-renderer image-renderer pod annotation                                       | `{}`                             |
| `imageRenderer.hostAliases`                | image-renderer deployment Host Aliases                                             | `[]`                             |
| `imageRenderer.priorityClassName`          | image-renderer deployment priority class                                           | `''`                             |
| `imageRenderer.service.enabled`            | Enable the image-renderer service                                                  | `true`                           |
| `imageRenderer.service.portName`           | image-renderer service port name                                                   | `http`                           |
| `imageRenderer.service.port`               | image-renderer port used by deployment                                             | `8081`                           |
| `imageRenderer.service.targetPort`         | image-renderer service port used by service                                        | `8081`                           |
| `imageRenderer.appProtocol`                | Adds the appProtocol field to the service                                          | ``                               |
| `imageRenderer.grafanaSubPath`             | Grafana sub path to use for image renderer callback url                            | `''`                             |
| `imageRenderer.podPortName`                | name of the image-renderer port on the pod                                         | `http`                           |
| `imageRenderer.revisionHistoryLimit`       | number of image-renderer replica sets to keep                                      | `10`                             |
| `imageRenderer.networkPolicy.limitIngress` | Enable a NetworkPolicy to limit inbound traffic from only the created grafana pods | `true`                           |
| `imageRenderer.networkPolicy.limitEgress` | Enable a NetworkPolicy to limit outbound traffic to only the created grafana pods | `false`                          |
| `imageRenderer.resources`                  | Set resource limits for image-renderer pods                                        | `{}`                             |
| `imageRenderer.nodeSelector`               | Node labels for pod assignment                | `{}`                                                    |
| `imageRenderer.tolerations`                | Toleration labels for pod assignment          | `[]`                                                    |
| `imageRenderer.affinity`                   | Affinity settings for pod assignment          | `{}`                                                    |
| `networkPolicy.enabled`                    | Enable creation of NetworkPolicy resources.                                                                              | `false`             |
| `networkPolicy.allowExternal`              | Don't require client label for connections                                                                               | `true`              |
| `networkPolicy.explicitNamespacesSelector` | A Kubernetes LabelSelector to explicitly select namespaces from which traffic could be allowed                           | `{}`                |
| `networkPolicy.ingress`                    | Enable the creation of an ingress network policy             | `true`    |
| `networkPolicy.egress.enabled`             | Enable the creation of an egress network policy              | `false`   |
| `networkPolicy.egress.ports`               | An array of ports to allow for the egress                    | `[]`    |
| `enableKubeBackwardCompatibility`          | Enable backward compatibility of kubernetes where pod's defintion version below 1.13 doesn't have the enableServiceLinks option | `false`     |

### Example ingress with path

With grafana 6.3 and above

  <br />grafana.ini:

  <br />&nbsp; server:

  <br />&nbsp;&nbsp;&nbsp;&nbsp; domain: monitoring.example.com

  <br />&nbsp;&nbsp;&nbsp; root_url: "%(protocol)s://%(domain)s/grafana"

  <br />&nbsp;&nbsp;&nbsp; serve_from_sub_path: true

  <br /> ingress:

  <br />&nbsp; enabled: true

  <br />&nbsp; hosts:

  <br />&nbsp;&nbsp;&nbsp; - "monitoring.example.com"

  <br />&nbsp; path: "/grafana"

  <br />

### Example of extraVolumeMounts and extraVolumes

Configure additional volumes with `extraVolumes` and volume mounts with `extraVolumeMounts`.

Example for `extraVolumeMounts` and corresponding `extraVolumes`:

  <br />extraVolumeMounts:

  <br />&nbsp; - name: plugins

  <br />&nbsp;&nbsp;&nbsp; mountPath: /var/lib/grafana/plugins

  <br />&nbsp;&nbsp;&nbsp; subPath: configs/grafana/plugins

  <br />&nbsp;&nbsp;&nbsp; readOnly: false

  <br />&nbsp; - name: dashboards

  <br />&nbsp;&nbsp;&nbsp; mountPath: /var/lib/grafana/dashboards

  <br />&nbsp;&nbsp;&nbsp; hostPath: /usr/shared/grafana/dashboards

  <br />&nbsp;&nbsp;&nbsp; readOnly: false</p>

<p>extraVolumes:
  <br />&nbsp; - name: plugins

  <br />&nbsp;&nbsp;&nbsp; existingClaim: existing-grafana-claim

  <br />&nbsp; - name: dashboards

  <br />&nbsp;&nbsp;&nbsp; hostPath: /usr/shared/grafana/dashboards

  <br />

Volumes default to `emptyDir`. Set to `persistentVolumeClaim`,
`hostPath`, `csi`, or `configMap` for other types. For a
`persistentVolumeClaim`, specify an existing claim name with
`existingClaim`.

## Import dashboards

There are a few methods to import dashboards to Grafana. Below are some examples and explanations as to how to use each method:

  <br /> dashboards:

  <br />&nbsp; default:

  <br />&nbsp;&nbsp;&nbsp; some-dashboard:

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; json: |

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; {

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; "annotations":</p>

<p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ...
  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; # Complete json file here

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ...</p>

<p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; "title": "Some Dashboard",
  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; "uid": "abcd1234",

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; "version": 1

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }

  <br />&nbsp;&nbsp;&nbsp; custom-dashboard:

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; # This is a path to a file inside the dashboards directory inside the chart directory

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; file: dashboards/custom-dashboard.json

  <br />&nbsp;&nbsp;&nbsp; prometheus-stats:

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; # Ref: <a href="https://grafana.com/dashboards/2">https://grafana.com/dashboards/2</a>

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; gnetId: 2

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; revision: 2

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; datasource: Prometheus

  <br />&nbsp;&nbsp;&nbsp; loki-dashboard-quick-search:

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; gnetId: 12019

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; revision: 2

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; datasource:

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - name: DS_PROMETHEUS

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; value: Prometheus

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - name: DS_LOKI

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; value: Loki

  <br />&nbsp;&nbsp;&nbsp; local-dashboard:

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; url: <a href="https://raw.githubusercontent.com/user/repository/master/dashboards/dashboard.json">https://raw.githubusercontent.com/user/repository/master/dashboards/dashboard.json</a>

  <br />

## BASE64 dashboards

Dashboards could be stored on a server that does not return JSON directly and instead of it returns a Base64 encoded file (e.g. Gerrit)
A new parameter has been added to the url use case so if you specify a b64content value equals to true after the url entry a Base64 decoding is applied before save the file to disk.
If this entry is not set or is equals to false not decoding is applied to the file before saving it to disk.

### Gerrit use case

Gerrit API for download files has the following schema: <https://yourgerritserver/a/{project-name}/branches/{branch-id}/files/{file-id}/content> where {project-name} and
{file-id} usually has '/' in their values and so they MUST be replaced by %2F so if project-name is user/repo, branch-id is master and file-id is equals to dir1/dir2/dashboard
the url value is <https://yourgerritserver/a/user%2Frepo/branches/master/files/dir1%2Fdir2%2Fdashboard/content>

## Sidecar for dashboards

If the parameter `sidecar.dashboards.enabled` is set, a sidecar container is deployed in the grafana
pod. This container watches all configmaps (or secrets) in the cluster and filters out the ones with
a label as defined in `sidecar.dashboards.label`. The files defined in those configmaps are written
to a folder and accessed by grafana. Changes to the configmaps are monitored and the imported
dashboards are deleted/updated.

A recommendation is to use one configmap per dashboard, as a reduction of multiple dashboards inside
one configmap is currently not properly mirrored in grafana.

Example dashboard config:

  <br />apiVersion: v1

  <br /> kind: ConfigMap

  <br /> metadata:

  <br />&nbsp; name: sample-grafana-dashboard

  <br />&nbsp; labels:

  <br />&nbsp;&nbsp;&nbsp;&nbsp; grafana_dashboard: "1"

  <br /> data:

  <br />&nbsp; k8s-dashboard.json: |-

  <br />&nbsp; [...]

  <br />

## Sidecar for datasources

If the parameter `sidecar.datasources.enabled` is set, an init container is deployed in the grafana
pod. This container lists all secrets (or configmaps, though not recommended) in the cluster and
filters out the ones with a label as defined in `sidecar.datasources.label`. The files defined in
those secrets are written to a folder and accessed by grafana on startup. Using these yaml files,
the data sources in grafana can be imported.

Should you aim for reloading datasources in Grafana each time the config is changed, set `sidecar.datasources.skipReload: false` and adjust `sidecar.datasources.reloadURL` to `http://<svc-name>.<namespace>.svc.cluster.local/api/admin/provisioning/datasources/reload`.

Secrets are recommended over configmaps for this usecase because datasources usually contain private
data like usernames and passwords. Secrets are the more appropriate cluster resource to manage those.

Example values to add a postgres datasource as a kubernetes secret:

  <br />apiVersion: v1

  <br /> kind: Secret

  <br />metadata:

  <br />&nbsp; name: grafana-datasources

  <br />&nbsp; labels:

  <br />&nbsp;&nbsp;&nbsp; grafana_datasource: 'true' # default value for: sidecar.datasources.label

  <br />stringData:

  <br />&nbsp; pg-db.yaml: |-

  <br />&nbsp;&nbsp;&nbsp; apiVersion: 1

  <br />&nbsp;&nbsp;&nbsp; datasources:

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - name: My pg db datasource

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; type: postgres

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; url: my-postgresql-db:5432

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; user: db-readonly-user

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; secureJsonData:

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; password: 'SUperSEcretPa$$word'

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; jsonData:

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; database: my_datase

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; sslmode: 'disable' # disable/require/verify-ca/verify-full

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; maxOpenConns: 0 # Grafana v5.4+

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; maxIdleConns: 2 # Grafana v5.4+

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; connMaxLifetime: 14400 # Grafana v5.4+

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; postgresVersion: 1000 # 903=9.3, 904=9.4, 905=9.5, 906=9.6, 1000=10

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; timescaledb: false

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; # &lt;bool&gt; allow users to edit datasources from the UI.

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; editable: false

  <br />

Example values to add a datasource adapted from [Grafana](http://docs.grafana.org/administration/provisioning/#example-datasource-config-file):

  <br />datasources:

  <br />&nbsp; datasources.yaml:

  <br />&nbsp;&nbsp; apiVersion: 1

  <br />&nbsp;&nbsp; datasources:

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; # &lt;string, required&gt; name of the datasource. Required

  <br />&nbsp;&nbsp;&nbsp; - name: Graphite

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; # &lt;string, required&gt; datasource type. Required

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; type: graphite

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; # &lt;string, required&gt; access mode. proxy or direct (Server or Browser in the UI). Required

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; access: proxy

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; # &lt;int&gt; org id. will default to orgId 1 if not specified

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; orgId: 1

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; # &lt;string&gt; url

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; url: <a href="http://localhost:8080">http://localhost:8080</a>

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; # &lt;string&gt; database password, if used

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; password:

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; # &lt;string&gt; database user, if used

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; user:

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; # &lt;string&gt; database name, if used

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; database:

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; # &lt;bool&gt; enable/disable basic auth

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; basicAuth:

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; # &lt;string&gt; basic auth username

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; basicAuthUser:

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; # &lt;string&gt; basic auth password

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; basicAuthPassword:

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; # &lt;bool&gt; enable/disable with credentials headers

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; withCredentials:

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; # &lt;bool&gt; mark as default datasource. Max one per org

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; isDefault:

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; # &lt;map&gt; fields that will be converted to json and stored in json_data

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; jsonData:

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; graphiteVersion: "1.1"

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; tlsAuth: true

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; tlsAuthWithCACert: true

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; # &lt;string&gt; json object of data that will be encrypted.

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; secureJsonData:

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; tlsCACert: "..."

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; tlsClientCert: "..."

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; tlsClientKey: "..."

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; version: 1

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; # &lt;bool&gt; allow users to edit datasources from the UI.

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; editable: false

  <br />

## Sidecar for notifiers

If the parameter `sidecar.notifiers.enabled` is set, an init container is deployed in the grafana
pod. This container lists all secrets (or configmaps, though not recommended) in the cluster and
filters out the ones with a label as defined in `sidecar.notifiers.label`. The files defined in
those secrets are written to a folder and accessed by grafana on startup. Using these yaml files,
the notification channels in grafana can be imported. The secrets must be created before
`helm install` so that the notifiers init container can list the secrets.

Secrets are recommended over configmaps for this usecase because alert notification channels usually contain
private data like SMTP usernames and passwords. Secrets are the more appropriate cluster resource to manage those.

Example datasource config adapted from [Grafana](https://grafana.com/docs/grafana/latest/administration/provisioning/#alert-notification-channels):

  <br /> notifiers:

  <br />&nbsp; - name: notification-channel-1

  <br />&nbsp;&nbsp;&nbsp; type: slack

  <br />&nbsp;&nbsp;&nbsp; uid: notifier1

  <br />&nbsp;&nbsp;&nbsp; # either

  <br />&nbsp;&nbsp;&nbsp; org_id: 2

  <br />&nbsp;&nbsp;&nbsp; # or

  <br />&nbsp;&nbsp;&nbsp; org_name: Main Org.

  <br />&nbsp;&nbsp;&nbsp; is_default: true

  <br />&nbsp;&nbsp;&nbsp; send_reminder: true

  <br />&nbsp;&nbsp;&nbsp; frequency: 1h

  <br />&nbsp;&nbsp;&nbsp; disable_resolve_message: false

  <br />&nbsp;&nbsp;&nbsp; # See `Supported Settings` section for settings supporter for each

  <br />&nbsp;&nbsp;&nbsp; # alert notification type.

  <br />&nbsp;&nbsp;&nbsp; settings:

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; recipient: 'XXX'

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; token: 'xoxb'

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; uploadImage: true

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; url: <a href="https://slack.com">https://slack.com</a></p>

<p>delete_notifiers:
  <br />&nbsp; - name: notification-channel-1

  <br />&nbsp;&nbsp;&nbsp; uid: notifier1

  <br />&nbsp;&nbsp;&nbsp; org_id: 2

  <br />&nbsp; - name: notification-channel-2

  <br />&nbsp;&nbsp;&nbsp; # default org_id: 1

  <br />

## Sidecar for alerting resources

If the parameter `sidecar.alerts.enabled` is set, a sidecar container is deployed in the grafana
pod. This container watches all configmaps (or secrets) in the cluster (namespace defined by `sidecar.alerts.searchNamespace`) and filters out the ones with
a label as defined in `sidecar.alerts.label` (default is `grafana_alert`). The files defined in those configmaps are written
to a folder and accessed by grafana. Changes to the configmaps are monitored and the imported alerting resources are updated, however, deletions are a little more complicated (see below).

This sidecar can be used to provision alert rules, contact points, notification policies, notification templates and mute timings as shown in [Grafana Documentation](https://grafana.com/docs/grafana/next/alerting/set-up/provision-alerting-resources/file-provisioning/).

To fetch the alert config which will be provisioned, use the alert provisioning API ([Grafana Documentation](https://grafana.com/docs/grafana/next/developers/http_api/alerting_provisioning/)).
You can use either JSON or YAML format.

Example config for an alert rule:

  <br />apiVersion: v1

  <br /> kind: ConfigMap

  <br /> metadata:

  <br />&nbsp; name: sample-grafana-alert

  <br />&nbsp; labels:

  <br />&nbsp;&nbsp;&nbsp;&nbsp; grafana_alert: "1"

  <br /> data:

  <br />&nbsp; k8s-alert.yml: |-

  <br />&nbsp;&nbsp;&nbsp; apiVersion: 1

  <br />&nbsp;&nbsp;&nbsp; groups:

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - orgId: 1

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; name: k8s-alert

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; [...]

  <br />

To delete provisioned alert rules is a two step process, you need to delete the configmap which defined the alert rule
and then create a configuration which deletes the alert rule.

Example deletion configuration:

  <br />apiVersion: v1

  <br /> kind: ConfigMap

  <br />metadata:

  <br />&nbsp; name: delete-sample-grafana-alert

  <br />&nbsp; namespace: monitoring

  <br />&nbsp; labels:

  <br />&nbsp;&nbsp;&nbsp; grafana_alert: "1"

  <br /> data:

  <br />&nbsp; delete-k8s-alert.yml: |-

  <br />&nbsp;&nbsp;&nbsp; apiVersion: 1

  <br />&nbsp;&nbsp;&nbsp; deleteRules:

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - orgId: 1

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; uid: 16624780-6564-45dc-825c-8bded4ad92d3

  <br />

## Statically provision alerting resources
If you don't need to change alerting resources (alert rules, contact points, notification policies and notification templates) regularly you could use the `alerting` config option instead of the sidecar option above.
This will grab the alerting config and apply it statically at build time for the helm file.

There are two methods to statically provision alerting configuration in Grafana. Below are some examples and explanations as to how to use each method:

  <br /> alerting:

  <br />&nbsp; team1-alert-rules.yaml:

  <br />&nbsp;&nbsp;&nbsp; file: alerting/team1/rules.yaml

  <br />&nbsp; team2-alert-rules.yaml:

  <br />&nbsp;&nbsp;&nbsp; file: alerting/team2/rules.yaml

  <br />&nbsp; team3-alert-rules.yaml:

  <br />&nbsp;&nbsp;&nbsp; file: alerting/team3/rules.yaml

  <br />&nbsp; notification-policies.yaml:

  <br />&nbsp;&nbsp;&nbsp; file: alerting/shared/notification-policies.yaml

  <br />&nbsp; notification-templates.yaml:

  <br />&nbsp;&nbsp;&nbsp; file: alerting/shared/notification-templates.yaml

  <br />&nbsp; contactpoints.yaml:

  <br />&nbsp;&nbsp;&nbsp; apiVersion: 1

  <br />&nbsp;&nbsp;&nbsp; contactPoints:

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - orgId: 1

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; name: Slack channel

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; receivers:

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - uid: default-receiver

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; type: slack

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; settings:

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; # Webhook URL to be filled in

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; url: ""

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; # We need to escape double curly braces for the tpl function.

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; text: '{{ `{{ template "default.message" . }}` }}'

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; title: '{{ `{{ template "default.title" . }}` }}'

  <br />

The two possibilities for static alerting resource provisioning are:

* Inlining the file contents as shown for contact points in the above example.
* Importing a file using a relative path starting from the chart root directory as shown for the alert rules in the above example.

### Important notes on file provisioning

* The format of the files is defined in the [Grafana documentation](https://grafana.com/docs/grafana/next/alerting/set-up/provision-alerting-resources/file-provisioning/) on file provisioning.
* The chart supports importing YAML and JSON files.
* The filename must be unique, otherwise one volume mount will overwrite the other.
* In case of inlining, double curly braces that arise from the Grafana configuration format and are not intended as templates for the chart must be escaped.
* The number of total files under `alerting:` is not limited. Each file will end up as a volume mount in the corresponding provisioning folder of the deployed Grafana instance.
* The file size for each import is limited by what the function `.Files.Get` can handle, which suffices for most cases.

## How to serve Grafana with a path prefix (/grafana)

In order to serve Grafana with a prefix (e.g., <http://example.com/grafana>), add the following to your values.yaml.

  <br /> ingress:

  <br />&nbsp; enabled: true

  <br />&nbsp; annotations:

  <br />&nbsp;&nbsp;&nbsp; kubernetes.io/ingress.class: "nginx"

  <br />&nbsp;&nbsp;&nbsp; nginx.ingress.kubernetes.io/rewrite-target: /$1

  <br />&nbsp;&nbsp;&nbsp; nginx.ingress.kubernetes.io/use-regex: "true"</p>

<p>&nbsp; path: /grafana/?(.*)
  <br />&nbsp; hosts:

  <br />&nbsp;&nbsp;&nbsp; - k8s.example.dev</p>

<p>grafana.ini:
  <br />&nbsp; server:

  <br />&nbsp;&nbsp;&nbsp; root_url: <a href="http://localhost:3000/grafana">http://localhost:3000/grafana</a> # this host can be localhost

  <br />

## How to securely reference secrets in grafana.ini

This example uses Grafana [file providers](https://grafana.com/docs/grafana/latest/administration/configuration/#file-provider) for secret values and the `extraSecretMounts` configuration flag (Additional grafana server secret mounts) to mount the secrets.

In grafana.ini:

  <br />grafana.ini:

  <br />&nbsp; [auth.generic_oauth]

  <br />&nbsp; enabled = true

  <br />&nbsp; client_id = $__file{/etc/secrets/auth_generic_oauth/client_id}

  <br />&nbsp; client_secret = $__file{/etc/secrets/auth_generic_oauth/client_secret}

  <br />

Existing secret, or created along with helm:

  <br /> ---

  <br />apiVersion: v1

  <br /> kind: Secret

  <br /> metadata:

  <br />&nbsp; name: auth-generic-oauth-secret

  <br /> type: Opaque

  <br />stringData:

  <br />&nbsp; client_id: &lt;value&gt;

  <br />&nbsp; client_secret: &lt;value&gt;

  <br />

Include in the `extraSecretMounts` configuration flag:

  <br /> - extraSecretMounts:

  <br />&nbsp; - name: auth-generic-oauth-secret-mount

  <br />&nbsp;&nbsp;&nbsp; secretName: auth-generic-oauth-secret

  <br />&nbsp;&nbsp;&nbsp; defaultMode: 0440

  <br />&nbsp;&nbsp;&nbsp; mountPath: /etc/secrets/auth_generic_oauth

  <br />&nbsp;&nbsp;&nbsp; readOnly: true

  <br />

### extraSecretMounts using a Container Storage Interface (CSI) provider

This example uses a CSI driver e.g. retrieving secrets using [Azure Key Vault Provider](https://github.com/Azure/secrets-store-csi-driver-provider-azure)

  <br /> - extraSecretMounts:

  <br />&nbsp; - name: secrets-store-inline

  <br />&nbsp;&nbsp;&nbsp; mountPath: /run/secrets

  <br />&nbsp;&nbsp;&nbsp; readOnly: true

  <br />&nbsp;&nbsp;&nbsp; csi:

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; driver: secrets-store.csi.k8s.io

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; readOnly: true

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; volumeAttributes:

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; secretProviderClass: "my-provider"

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; nodePublishSecretRef:

  <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; name: akv-creds

  <br />

## Image Renderer Plug-In

This chart supports enabling [remote image rendering](https://github.com/grafana/grafana-image-renderer/blob/master/README.md#run-in-docker)

  <br />imageRenderer:

  <br />&nbsp; enabled: true

  <br />

### Image Renderer NetworkPolicy

By default the image-renderer pods will have a network policy which only allows ingress traffic from the created grafana instance

### High Availability for unified alerting

If you want to run Grafana in a high availability cluster you need to enable
the headless service by setting `headlessService: true` in your `values.yaml`
file.

As next step you have to setup the `grafana.ini` in your `values.yaml` in a way
that it will make use of the headless service to obtain all the IPs of the
cluster. You should replace ``{{ Name }}`` with the name of your helm deployment.

  <br />grafana.ini:

  <br />&nbsp; ...

  <br />&nbsp; unified_alerting:

  <br />&nbsp;&nbsp;&nbsp; enabled: true

  <br />&nbsp;&nbsp;&nbsp; ha_peers: {{ Name }}-headless:9094

  <br />&nbsp;&nbsp;&nbsp; ha_listen_address: ${POD_IP}:9094

  <br />&nbsp;&nbsp;&nbsp; ha_advertise_address: ${POD_IP}:9094</p>

<p>&nbsp; alerting:
  <br />&nbsp;&nbsp;&nbsp; enabled: false

  <br />

Configuring Velero for Cluster Backup on AWS

noreply@blogger.com (System Engineer) — Fri, 17 May 2024 19:25:00 +0530

Velero is an open-source tool for backing up and restoring Kubernetes cluster resources and persistent volumes. This guide will walk you through configuring Velero on AWS to ensure your cluster data is securely backed up.

🛒 Recommended gear on Amazon

Disclosure: some links above are affiliate links — if you buy through them I may earn a small commission at no extra cost to you. Thanks for supporting the channel!

Introduction: In the ever-evolving landscape of cloud-native infrastructure, ensuring the resilience and integrity of your data is paramount. With Kubernetes becoming the de facto standard for container orchestration, it's essential to have robust backup and recovery mechanisms in place. Enter Velero – an open-source tool designed to streamline the backup, restore, and migration of Kubernetes resources and persistent volumes. In this comprehensive guide, we'll explore everything you need to know about Velero and how it can help safeguard your Kubernetes data.

1. Understanding the Need for Kubernetes Backup Solutions:

Discuss the importance of data protection in Kubernetes environments.
Highlight common data loss scenarios and their impact on operations.
Introduce the concept of disaster recovery and the role of backup solutions like Velero in mitigating risks.

2. Getting Started with Velero:

Step-by-step guide to installing Velero on your Kubernetes cluster.
Configuring Velero to work with your preferred cloud provider or storage backend.
Exploring Velero's command-line interface (CLI) and RESTful API for managing backups and restores.

3. Key Features and Functionality:

Deep dive into Velero's core features, including:
- Customizable backup policies for selective resource inclusion/exclusion.
- Support for both cluster-wide and individual persistent volume backups.
- Incremental backups to optimize storage and reduce backup duration.
- Extensibility through plugins for various cloud providers and storage solutions.
- Cross-cluster migration capabilities for seamless workload portability.
Real-world examples illustrating how these features address common use cases and challenges.

4. Best Practices for Velero Deployment:

Recommendations for optimizing Velero's performance and reliability.
Strategies for securely managing backup data, including encryption and access control.
Backup lifecycle management tips to ensure efficient resource utilization and cost savings.
Scalability considerations for large-scale Kubernetes deployments.

5. Integrating Velero into Your Workflow:

Exploring Velero's integration with CI/CD pipelines for automated backup and recovery.
Leveraging Velero's API for seamless integration with monitoring and alerting systems.
Incorporating Velero into disaster recovery plans and testing procedures.

6. Community and Support Ecosystem:

Overview of the Velero community, including contributors, forums, and resources.
Where to find documentation, tutorials, and best practices for Velero deployment and usage.
Available support channels and options for engaging with the Velero community for assistance and collaboration.

Setup

To set up Velero on AWS, you:

You can also use this plugin to migrate PVs across clusters or create an additional Backup Storage Location.

If you do not have the aws CLI locally installed, follow the user guide to set it up.

Create S3 bucket

Velero requires an object storage bucket to store backups in, preferably unique to a single Kubernetes cluster (see the FAQ for more details). Create an S3 bucket, replacing placeholders appropriately:

BUCKET=<YOUR_BUCKET>
REGION=<YOUR_REGION>
aws s3api create-bucket \
    --bucket $BUCKET \
    --region $REGION \
    --create-bucket-configuration LocationConstraint=$REGION

NOTE: us-east-1 does not support a LocationConstraint. If your region is us-east-1, omit the bucket configuration:

aws s3api create-bucket \
    --bucket $BUCKET \
    --region us-east-1

Set permissions for Velero

For more information, see the AWS documentation on IAM users.

Create the IAM user:
```
aws iam create-user --user-name velero
```
If you'll be using Velero to backup multiple clusters with multiple S3 buckets, it may be desirable to create a unique username per cluster rather than the default velero.

Attach policies to give velero the necessary permissions:

cat > velero-policy.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeVolumes",
                "ec2:DescribeSnapshots",
                "ec2:CreateTags",
                "ec2:CreateVolume",
                "ec2:CreateSnapshot",
                "ec2:DeleteSnapshot"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:PutObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::${BUCKET}/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::${BUCKET}"
            ]
        }
    ]
}
EOF

aws iam put-user-policy \
  --user-name velero \
  --policy-name velero \
  --policy-document file://velero-policy.json

Create an access key for the user:

aws iam create-access-key --user-name velero

The result should look like:

{
  "AccessKey": {
        "UserName": "velero",
        "Status": "Active",
        "CreateDate": "2017-07-31T22:24:41.576Z",
        "SecretAccessKey": <AWS_SECRET_ACCESS_KEY>,
        "AccessKeyId": <AWS_ACCESS_KEY_ID>
  }
}

Create a Velero-specific credentials file (credentials-velero) in your local directory:
```
[default]
aws_access_key_id=<AWS_ACCESS_KEY_ID>
aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
```
where the access key id and secret are the values returned from the create-access-key request.

Install and start Velero

Download Velero

Install Velero, including all prerequisites, into the cluster and start the deployment. This will create a namespace called velero, and place a deployment named velero in it.

If using IAM user and access key:

velero install \
    --provider aws \
    --plugins velero/velero-plugin-for-aws:v1.10.0 \
    --bucket $BUCKET \
    --backup-location-config region=$REGION \
    --snapshot-location-config region=$REGION \
    --secret-file ./credentials-velero

velero.exe install \

> --provider aws We are closing this ticket now.

> --plugins velero/velero-plugin-for-aws:v1.9.0 \

> --bucket velero-xfi-test \

> --backup-location-config region=eu-west-2\

> --snapshot-location-config region=eu-west-2 \

> --secret-file ./credentials-velero

;6bd1439b-3bea-441f-a5bf-b5390b6174ceCustomResourceDefinition/backuprepositories.velero.io: attempting to create resource

CustomResourceDefinition/backuprepositories.velero.io: attempting to create resource client

CustomResourceDefinition/backuprepositories.velero.io: already exists, proceeding

CustomResourceDefinition/backuprepositories.velero.io: created

CustomResourceDefinition/backups.velero.io: attempting to create resource

CustomResourceDefinition/backups.velero.io: attempting to create resource client

CustomResourceDefinition/backups.velero.io: already exists, proceeding

CustomResourceDefinition/backups.velero.io: created

CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource

CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource client

CustomResourceDefinition/backupstoragelocations.velero.io: already exists, proceeding

CustomResourceDefinition/backupstoragelocations.velero.io: created

CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource

CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource client

CustomResourceDefinition/deletebackuprequests.velero.io: already exists, proceeding

CustomResourceDefinition/deletebackuprequests.velero.io: created

CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource

CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource client

CustomResourceDefinition/downloadrequests.velero.io: already exists, proceeding

CustomResourceDefinition/downloadrequests.velero.io: created

CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource

CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource client

CustomResourceDefinition/podvolumebackups.velero.io: already exists, proceeding

CustomResourceDefinition/podvolumebackups.velero.io: created

CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource

CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource client

CustomResourceDefinition/podvolumerestores.velero.io: already exists, proceeding

CustomResourceDefinition/podvolumerestores.velero.io: created

CustomResourceDefinition/restores.velero.io: attempting to create resource

CustomResourceDefinition/restores.velero.io: attempting to create resource client

CustomResourceDefinition/restores.velero.io: already exists, proceeding

CustomResourceDefinition/restores.velero.io: created

CustomResourceDefinition/schedules.velero.io: attempting to create resource

CustomResourceDefinition/schedules.velero.io: attempting to create resource client

CustomResourceDefinition/schedules.velero.io: already exists, proceeding

CustomResourceDefinition/schedules.velero.io: created

CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource

CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource client

CustomResourceDefinition/serverstatusrequests.velero.io: already exists, proceeding

CustomResourceDefinition/serverstatusrequests.velero.io: created

CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource

CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource client

CustomResourceDefinition/volumesnapshotlocations.velero.io: already exists, proceeding

CustomResourceDefinition/volumesnapshotlocations.velero.io: created

CustomResourceDefinition/datadownloads.velero.io: attempting to create resource

CustomResourceDefinition/datadownloads.velero.io: attempting to create resource client

CustomResourceDefinition/datadownloads.velero.io: already exists, proceeding

CustomResourceDefinition/datadownloads.velero.io: created

CustomResourceDefinition/datauploads.velero.io: attempting to create resource

CustomResourceDefinition/datauploads.velero.io: attempting to create resource client

CustomResourceDefinition/datauploads.velero.io: already exists, proceeding

CustomResourceDefinition/datauploads.velero.io: created

Waiting for resources to be ready in cluster...

Namespace/velero: attempting to create resource

Namespace/velero: attempting to create resource client

Namespace/velero: created

ClusterRoleBinding/velero: attempting to create resource

ClusterRoleBinding/velero: attempting to create resource client

ClusterRoleBinding/velero: created

ServiceAccount/velero: attempting to create resource

ServiceAccount/velero: attempting to create resource client

ServiceAccount/velero: created

Secret/cloud-credentials: attempting to create resource

Secret/cloud-credentials: attempting to create resource client

Secret/cloud-credentials: created

BackupStorageLocation/default: attempting to create resource

BackupStorageLocation/default: attempting to create resource client

BackupStorageLocation/default: created

VolumeSnapshotLocation/default: attempting to create resource

VolumeSnapshotLocation/default: attempting to create resource client

VolumeSnapshotLocation/default: created

Deployment/velero: attempting to create resource

Deployment/velero: attempting to create resource client

Deployment/velero: created

Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.

Reference: https://github.com/vmware-tanzu/velero-plugin-for-aws

Important Commands:

velero.exe schedule create name --schedule="* 18 * * *" --include-namespaces vector

velero.exe backup get

velero.exe backup create new --include-namespaces vector

velero.exe backup logs 16may-my-immediate-backup

Using Helm to Install Velero

Alternatively, you can install Velero using Helm:

helm install my-velero vmware-tanzu/velero --version 6.2.0
helm upgrade --reset-values my-velero vmware-tanzu/velero -f values-velero.yaml --version 6.2.0
kubectl create secret generic velero-credentials --from-file=cloud=file.txt
helm upgrade --reset-values my-velero vmware-tanzu/velero -f values-velero.yaml --version 6.2.0 --set-file credentials.secretContents.cloud=file.txt

Add the Helm repository and install Velero:

helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts helm install my-velero vmware-tanzu/velero --version 6.2.0

Create the credentials secret:

kubectl create secret generic velero-credentials --from-file=cloud=credentials-velero

Upgrade Velero with the credentials:

helm upgrade --set-file credentials.secretContents.cloud=credentials-velero my-velero vmware-tanzu/velero --version 6.2.0

Reference:

https://artifacthub.io/packages/helm/vmware-tanzu/velero

Take a Backup

$ velero.exe backup create full-cluster-backup
Backup request "full-cluster-backup" submitted successfully.
Run `velero backup describe full-cluster-backup` or `velero backup logs full-cluster-backup` for more details.



$ velero.exe get backup
NAME                        STATUS            ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
16may-my-immediate-backup   Failed            0        0          2024-05-16 17:34:02 +0530 IST   29d       default            <none>
full-cluster-backup         Completed         0        0          2024-05-16 19:16:34 +0530 IST   29d       default            <none>
my-immediate-backup         PartiallyFailed   6        0          2024-05-16 16:01:15 +0530 IST   29d       default            <none>
new                         Completed         0        0          2024-05-16 18:13:46 +0530 IST   29d       default            <none>
test-my-immediate-backup    Completed         0        0          2024-05-16 17:36:59 +0530 IST   29d       default            <none>


velero.exe backup create new --include-namespaces vector

Restore a Backup

To restore a backup: Create a restore from a backup

velero restore create --from-backup <backup-name> --include-namespaces <namespace>

Check the status of the restore:

velero restore describe <restore-name> velero restore logs <restore-name>

Restore Demo:

For testing, I previously deleted the Grafana Deployment and PVC and restored them with the original data.

velero.exe restore create --from-backup new --include-namespaces vector  Restore request "new-20240516184747" submitted successfully.
Run `velero restore describe new-20240516184747` or `velero restore logs new-20240516184747` for more details.

$ kubectl get pvc -n vector
NAME                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
data-loki-backend-0   Bound    pvc-a9d7164c-1640-48af-9417-741fb29c8d1b   10Gi       RWO            gp2            <unset>                 16d
data-loki-backend-1   Bound    pvc-9185e1e5-8b3d-4089-8414-b7425dcd4680   10Gi       RWO            gp2            <unset>                 16d
data-loki-write-0     Bound    pvc-18cfeda2-a5c9-4faa-8cc9-7648217c0871   10Gi       RWO            gp2            <unset>                 57d
data-loki-write-1     Bound    pvc-76739fcf-f999-4a46-ada9-a5fbe71add07   10Gi       RWO            gp2            <unset>                 57d
data-loki-write-2     Bound    pvc-840d28a2-0d32-4122-8896-89e0a8c18443   10Gi       RWO            gp2            <unset>                 57d

$ kgpo -n vector
NAME                                              READY   STATUS    RESTARTS      AGE
loki-backend-0                                    2/2     Running   0             16d
loki-backend-1                                    2/2     Running   0             16d
loki-canary-dd8zs                                 1/1     Running   0             16d
loki-canary-ddcm2                                 1/1     Running   0             16d
loki-canary-ntqkk                                 1/1     Running   0             16d
loki-canary-pbj9q                                 1/1     Running   0             16d
loki-gateway-5f5dfb657c-w5pcb                     1/1     Running   0             20h
loki-read-8f6b77b99-9zfwx                         1/1     Running   0             20h
loki-read-8f6b77b99-hzl4x                         1/1     Running   0             20h
loki-write-0                                      1/1     Running   0             16d
loki-write-1                                      1/1     Running   0             16d
my-loki-grafana-agent-operator-7f58c9bd66-56w6h   1/1     Running   0             20h
my-loki-logs-bhtn2                                2/2     Running   0             16d
my-loki-logs-dbf5g                                2/2     Running   0             16d
my-loki-logs-mpmgt                                2/2     Running   0             16d
my-loki-logs-sbqsw                                2/2     Running   0             16d
my-vector-0                                       1/1     Running   0             16d
my-velero-74ff4d56d-hbrdc                         1/1     Running   2 (11m ago)   11m

$ kubectl get pvc -n vector
NAME                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
data-loki-backend-0   Bound    pvc-a9d7164c-1640-48af-9417-741fb29c8d1b   10Gi       RWO            gp2            <unset>                 16d
data-loki-backend-1   Bound    pvc-9185e1e5-8b3d-4089-8414-b7425dcd4680   10Gi       RWO            gp2            <unset>                 16d
data-loki-write-0     Bound    pvc-18cfeda2-a5c9-4faa-8cc9-7648217c0871   10Gi       RWO            gp2            <unset>                 57d
data-loki-write-1     Bound    pvc-76739fcf-f999-4a46-ada9-a5fbe71add07   10Gi       RWO            gp2            <unset>                 57d
data-loki-write-2     Bound    pvc-840d28a2-0d32-4122-8896-89e0a8c18443   10Gi       RWO            gp2            <unset>                 57d
my-grafana            Bound    pvc-39f24317-e484-4e98-b4c9-625de1b9367b   10Gi       RWO            gp2            <unset>                 28s

$ kubectl get pods -n vector
NAME                                              READY   STATUS    RESTARTS   AGE
loki-backend-0                                    2/2     Running   0          16d
loki-backend-1                                    2/2     Running   0          16d
loki-canary-dd8zs                                 1/1     Running   0          16d
loki-canary-ddcm2                                 1/1     Running   0          16d
loki-canary-ntqkk                                 1/1     Running   0          16d
loki-canary-pbj9q                                 1/1     Running   0          16d
loki-gateway-5f5dfb657c-w5pcb                     1/1     Running   0          20h
loki-read-8f6b77b99-9zfwx                         1/1     Running   0          20h
loki-read-8f6b77b99-hzl4x                         1/1     Running   0          20h
loki-write-0                                      1/1     Running   0          16d
loki-write-1                                      1/1     Running   0          16d
my-grafana-69c9db76dc-cwhmm                       1/1     Running   0          54s
my-loki-grafana-agent-operator-7f58c9bd66-56w6h   1/1     Running   0          20h
my-loki-logs-bhtn2                                2/2     Running   0          16d
my-loki-logs-dbf5g                                2/2     Running   0          16d
my-loki-logs-mpmgt                                2/2     Running   0          16d
my-loki-logs-sbqsw                                2/2     Running   0          16d
my-vector-0                                       1/1     Running   0          16d

System Admin Share

Kubernetes CrashLoopBackOff: A Practical Debugging Playbook

What CrashLoopBackOff actually means

Prerequisites

Step 1: Confirm the state and the restart count

Step 2: Read the crashed container's logs

Step 3: Describe the pod for events and exit codes

Exit codes worth memorising

CrashLoopBackOff vs other failing states

Common root causes and how to fix each

1. The application errors on startup

2. A missing or wrong ConfigMap / Secret / env var

3. Exit code 137 — OOMKilled

4. A failing liveness probe restarts a healthy app

5. A bad command, args, or entrypoint

6. A dependency is not ready yet

7. Permissions and read-only filesystems

Step 4: Verify the fix

Key Takeaways

Frequently Asked Questions

What does CrashLoopBackOff mean in Kubernetes?

How do I see the logs of a container that already crashed?

What causes exit code 137 in a pod?

Why does my healthy app keep restarting?

What is the difference between CrashLoopBackOff and ImagePullBackOff?

Simplifying Helm Configurations for Grafana Deployment using classic loadbalancer

When setting up Grafana with Helm, the old configuration involved several critical settings to establish a LoadBalancer service. Here’s a detailed look at the old configuration:

Key Components of the Old Configuration

Common Mistakes in the Old Configuration

Transitioning to the New Configuration

Key Improvements in the New Configuration

Benefits of the New Configuration

Conclusion

1. service Section

2. ports Section

3. extraExposePorts Section

4. healthCheck Section

5. externalTrafficPolicy Section

Summary

Configuring Velero for Cluster Backup on AWS

Setup

Create S3 bucket

Set permissions for Velero

Install and start Velero

Important Commands:

Using Helm to Install Velero

Reference:

Take a Backup

Restore a Backup

1. `service` Section

2. `ports` Section

3. `extraExposePorts` Section

4. `healthCheck` Section

5. `externalTrafficPolicy` Section