Monitor Spring Boot Custom Metrics with Kubernetes using Prometheus and Grafana

7 min readFeb 14, 2023

In this article, we will deploy our spring boot application and Prometheus server into Kubernetes cluster for monitoring custom metrics.

This is series of articles that you can check previous article about “Monitor Spring Boot Custom Metrics with Micrometer and Prometheus using Docker”.

I have just published a new course — Design Microservices Architecture with Patterns & Principles.

Background

In this tutorial series, we will learn how horizontally auto-scale spring boot microservice applications with using Prometheus custom metrics and KEDA — Kubernetes Event-driven Auto-scaler. Here you can find the 3 main article that we are going to follow:

Monitor Spring Boot Custom Metrics with Micrometer and Prometheus using Docker
Monitor Custom Metrics with deploying Kubernetes using Prometheus (this article)
Auto-scaling Kubernetes apps with Prometheus and KEDA

Prerequisites

As you can understand from the first image, we have some prerequisite for monitoring Spring Boot application. Those are;

Spring Boot — Java applications
Actuator, Micrometer libraries
Minikube
Helm Charts
Prometheus and Grafana for monitoring tools

In previous article, we have developed and containerize Spring Boot application. So in this tutorial we will focus on deploying Kubernetes and use Helm Charts to activate Prometheus and Grafana for monitoring custom metrics.

Pushing Local Docker Images to Minikube

By default, Kubernetes retrieve images from Docker Hub or other container registries. Since we are developing local poc application, we should define local docker image to our minikube.

For that purpose we can run several commands on project directory after finished the installments:

& minikube -p minikube docker-env --shell powershell | Invoke-Expression
docker ps
docker build -t demoapp:1 -f Dockerfile .
kubectl run demoapp --image=demoapp:1 --image-pull-policy=Never
kubectl get pods

This will provide to run our spring boot demo application on minikube Kubernetes cluster. The important part is here, we have build docker images with demoapp:1 name and tag information in our local environment. We will use this image in Kubernetes soon. We can see metrics on url below: (after port-redirection)

http://localhost:8080/actuator/prometheus

But we need to deploy whole application with Spring Boot and Prometheus and also we can add here Grafana later.

Deploy Spring Boot Application in Kubernetes

We will start to deploy our demo app spring boot application into minikube Kubernetes. We’ll need to deploy 2 things: a Deployment to run the application, a Service to access the application.

Create k8s folder into spring boot application and create demoapp.yaml file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demoapp
  labels:
    app: demoapp
spec:
  selector:
    matchLabels:
      app: demoapp
  template:
    metadata:
      labels:
        app: demoapp
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/actuator/prometheus"
    spec:
      containers:
        - name: demoapp
          image: demoapp:1
          imagePullPolicy: Never
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: demoapp
  labels:
    app: demoapp
spec:
  ports:
    - protocol: TCP
      name: http-traffic
      port: 8080
      targetPort: 8080
  selector:
    app: demoapp

So this yaml file includes required deployment objects in Kubernetes for our Spring Boot application. Important notes in this yaml file;

We retrieve local Docker image which is demoapp:1. See configurations should set imagePullPolicy: Never in order to avoid to search this image from the internet.
Give the name of service protocol as a http-trafic that we will use this port name when communicating with Prometheus.

Deploy Prometheus in Kubernetes with installing Prometheus Helm Charts

We will start to deploy Prometheus in Kubernetes. We will use Helm charts for this action so before this step you should download helm on your computer.

To install Prometheus, you first need to add the Bitnami Helm repository by using the helm repo add command followed by the helm repo update command to pull in the latest metadata:

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

Before you install Prometheus, check out the configuration options you have, because there are a lot of them. In this example, it’s assumed that you’ll be installing with the default configuration into a Kubernetes cluster with no specific requirements. As you can see, there are options for everything from how each component is exposed (i.e., the type of ingress used, if it’s behind a load balancer, etc.) to how data is stored and more. If this was a production installation, you’d want to thoroughly sort out these options, but for the purposes of this demo, you can install Prometheus with the default configuration using the helm install command:

helm install prometheus bitnami/kube-prometheus

After a few moments, you’ll see a few new pods created:

alertmanager-prometheus-prometheus-oper-alertmanager-0   2/2     Running   0          25m
prometheus-kube-state-metrics-68cb46fdd4-gk4jh           1/1     Running   0          25m
prometheus-node-exporter-rkg84                           1/1     Running   0          25m
prometheus-prometheus-oper-operator-745f4b599c-xjjsn     1/1     Running   0          25m
prometheus-prometheus-prometheus-oper-prometheus-0       3/3     Running   1          25m

There are a few ways you can access Prometheus, but it largely depends on how your Kubernetes cluster is configured. As the Prometheus documentation points out, traditionally you would expose the server through a reverse proxy, such as nginx. But since the default configuration of the Prometheus Helm chart only exposes it to other pods in the Kubernetes cluster, you can instead take advantage of the kubectl port-forward command. Open a new terminal and keep it open after running the following:

kubectl port-forward --namespace default svc/prometheus-kube-prometheus-prometheus 9090:9090

Great! The above command forwards all traffic to port 9090 on your machine to the prometheus-server pod, which you can see by visiting http://localhost:9090.

Connecting Between Prometheus and Spring Boot App Metrics

Now we have deployed 2 application on Kubernetes, but our demo application metrics can’t scrape from Prometheus. In order to define integration, we should create ServiceMonitor object in kubernetes that provide to listen prometheus metrics on service definition.

Create ServiceMonitor Object in Kubernetes

When we define the Service also receives a label of app: demoapp, which the ServiceMonitor will use to find the Service. One thing to note is that the port receives a name of http-traffic, which you can see the ServiceMonitor reference below:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: demoapp-service-monitor
  labels:
    release: prometheus-operator
spec:
  selector:
    matchLabels:
      app: demoapp
  endpoints:
    - port: http-traffic
      path: "/actuator/prometheus"

Here a ServiceMonitor is created, which looks for a Service with the label app: demoapp. It then defines an endpoint to use to scrape metrics, referring to the port named http-traffic and the path /actuator/prometheus, which as you saw is where Spring Boot exposes the Prometheus-formatted metrics.

Now we have 2 deployment file under k8s folder which are demoapp.yaml and service_monitor.yaml file.

kubectl apply -f demoapp.yaml
kubectl apply -f service_monitor.yaml
kubectl get svc
kubectl port-forward service/demoapp 8080:8080
kubectl port-forward service/prometheus-kube-prometheus-prometheus 9090:9090

If you apply these 2 files with kubectl appy command, you can run both Spring Boot and Prometheus apps. You can verify that Prometheus is scraping the new endpoint by checking the targets it has registered, found under “Status” drop-down menu:

So far we have finished the first part of tutorial which is “2- Monitor Custom Metrics with deploying Kubernetes using Prometheus”. Now we can go forward to last part.

Test Links:
http://localhost:8080/actuator/prometheus

Send Http POST
http://localhost:8080/books

http://localhost:9090/targets?search=
serviceMonitor/default/demoapp-service-monitor/0 (1/1 up)

Source Code

Get the Source Code from Github — Clone or fork this repository, if you like don’t forget the star. If you find or ask anything you can directly open issue on repository.

Afterwards

In this tutorial series, we will learn how horizontally auto-scale spring boot microservice applications with using Prometheus custom metrics and KEDA — Kubernetes Event-driven Auto-scaler. Here you can find the 3 main article that we are going to follow:

Monitor Spring Boot Custom Metrics with Micrometer and Prometheus using Docker
Monitor Custom Metrics with deploying Kubernetes using Prometheus (this article)
Auto-scaling Kubernetes apps with Prometheus and KEDA

Step by Step Design Architectures w/ Course

I have just published a new course — Design Microservices Architecture with Patterns & Principles.

In this course, we’re going to learn how to Design Microservices Architecture with using Design Patterns, Principles and the Best Practices. We will start with designing Monolithic to Event-Driven Microservices step by step and together using the right architecture design patterns and techniques.