Application Monitoring

Accessing Grafana

Via internal services

Grafana is available at grafana.internal_services.domain

The internal_services.domain is in the config.yaml for the environment you want to access grafana for in your Environments Repo.

You can quickly access it by running:

corectl env open <env> grafana

How can I monitor application resources?

This dashboard will allow a team to monitor their application namespaces and check their status. It will show data like:

• CPU Usage
• Memory usage
• Pod status
• Pod restart count

How can I monitor the whole cluster?

The global view dashboard will give you visibility over the cluster as a whole. This will show you data like:

• Nodes status
• CPU and Memory usage
• CPU and Memory Limits
• CPU and Memory Reserved
• Pod and namespace count
• Pods status

How can do I know if the environment is stable?

The platform-monitoring module also deploys a continuous load. This will create k6 injectors and pods with podinfo, always with a stable throughput allowing us to monitor with enough data the different percentils and any errors that occur to ensure that we can be proactive in investigating and fixing any issues.

Application-Specific Metrics

Examples of Application-Specific Metrics:

• Go GC statistics
• Request processing latency
• Number of processed jobs

Prerequisite

You should have deployed monitoring stack for your tenant. This monitoring stack consists of preconfigured Grafana and Prometheus instances.

To install monitoring stack, run the following command:

# Add helm repository
helm repo add coreeng https://coreeng.github.io/core-platform-assets
helm repo update
 
# Install the chart
helm -n {{ target-ns }} install monitoring-stack coreeng/monitoring-stack --set tenantName={{ your-tenant-name }}

To upgrade installed monitoring stack, run the following command:

#Upgrade installed chart
helm -n {{ target-ns }} upgrade monitoring-stack coreeng/monitoring-stack --set tenantName={{ your-tenant-name }}

Shared Monitoring Stack For Multiple Delivery Units

Use a dedicated monitoring delivery unit when one Prometheus, Grafana, and Alertmanager stack should monitor several delivery units. The monitoring delivery unit owns the monitoring infrastructure. Each monitored delivery unit owns its metrics endpoint, alerts, dashboards, and the RBAC grant that lets Prometheus discover resources in that namespace.

This pattern assumes:

• the monitored applications use the core-platform-app Helm chart
• the monitoring delivery unit is created from the monitoring-stack template
• each app exposes Prometheus metrics on an internal endpoint
• app-specific alerts and dashboards stay in the app repository

Responsibilities

The monitoring delivery unit owns:

• Prometheus, Grafana, and Alertmanager
• Slack or notification routing
• the namespaces Prometheus should watch
• the app instances Prometheus should select
• selectors for ServiceMonitor, PodMonitor, PrometheusRule, and GrafanaDashboard resources

Each monitored delivery unit owns:

• its metrics endpoint
• app metrics configuration passed to the app Helm chart
• app alert definitions
• app dashboard definitions
• the namespace-local RBAC grant for the monitoring Prometheus service account

Create The Monitoring Delivery Unit

Create the monitoring delivery unit from the monitoring-stack template. Configure each environment with the namespaces and app instances that the shared Prometheus instance should discover.

prometheus:
  targetNamespaces:
    - <monitoring-namespace>
    - <app-a-namespace>
    - <app-b-namespace>
  targetInstances:
    - <monitoring-instance-name>
    - <app-a-instance-name>
    - <app-b-instance-name>

For example, an integration environment can include:

prometheus:
  targetNamespaces:
    - elevate-monitoring-integration
    - training-backend-integration
    - insights-backend-integration
  targetInstances:
    - elevate-monitoring
    - training-backend
    - insights-backend

Use the production namespaces for production, such as elevate-monitoring-prod and training-backend-prod.

Configure Alertmanager routing in the monitoring delivery unit. App alerts should set labels and annotations; they should not configure Slack webhooks directly.

alertmanager:
  slack:
    channel: "#core-elevate"
    webhookSecretName: alertmanager-slack-webhook
    webhookSecretKey: url

Configure Each Monitored Delivery Unit

Enable metrics in the app’s common configuration:

metrics:
  enabled: true
  port: 8081
  path: /prometheus
  interval: 30s
  scrapeTimeout: 10s

For Spring Boot services, expose Prometheus metrics through Actuator and Micrometer on the configured management port and path.

Grant the monitoring Prometheus service account read-only discovery access from each app namespace. Configure this per environment:

metrics:
  prometheusAccess:
    enabled: true
    serviceAccountName: <monitoring-prometheus-service-account>
    namespace: <monitoring-namespace>

For example:

metrics:
  prometheusAccess:
    enabled: true
    serviceAccountName: elevate-monitoring-prometheus
    namespace: elevate-monitoring-integration

The app chart creates a namespace-local Role and RoleBinding so Prometheus can read services, endpoints, pods, and PrometheusRule resources. Metrics endpoints should stay internal-only.

Add App Alerts And Dashboards

Keep app-owned alerts in p2p/config/common-alerts.yaml.

---
alerts:
  enabled: true
  rules:
    - name: http-errors
      rules:
        - alert: ${p2p_app_name}-${p2p_environment}-http-5xx-errors
          expr: sum(rate(http_server_requests_seconds_count{job="${p2p_app_name}", namespace="${p2p_namespace}", status=~"5.."}[5m])) > 0
          for: 1m
          labels:
            severity: critical
            job: ${p2p_app_name}
          annotations:
            summary: "HTTP 5xx errors detected"
            description: "Service is returning HTTP 5xx errors"
            source: "${p2p_app_name} (${p2p_environment}) - namespace ${p2p_namespace}"

Keep app-owned dashboards in p2p/config/common-dashboards.yaml. Dashboard queries should use the shared monitoring Prometheus datasource UID and must filter by namespace="${p2p_namespace}".

Use these conventions:

• scope PromQL queries by job="${p2p_app_name}"
• scope PromQL queries by namespace="${p2p_namespace}"
• include ${p2p_environment} in alert names and dashboard UIDs
• keep business-specific alerts in the app that owns the metric
• keep notification routing in Alertmanager

Deploy And Verify

Deploy monitored apps before deploying or reconciling the monitoring delivery unit.

Recommended order:

1. Deploy or update the monitored delivery units.
2. Confirm each app namespace contains the expected RoleBinding for the monitoring Prometheus service account.
3. Deploy or update the monitoring delivery unit.
4. Confirm Prometheus target discovery.
5. Confirm PrometheusRule discovery.
6. Confirm Grafana dashboard discovery.
7. Confirm Alertmanager routing.

To add another delivery unit to the shared stack:

1. Expose Prometheus metrics in the app.
2. Add common app metrics config.
3. Add environment-specific metrics.prometheusAccess config.
4. Add app-owned alerts in p2p/config/common-alerts.yaml.
5. Add app-owned dashboards in p2p/config/common-dashboards.yaml if dashboards are required.
6. Add the app namespace to the monitoring stack prometheus.targetNamespaces for each environment.
7. Add the app instance name to the monitoring stack prometheus.targetInstances for each environment.
8. Deploy the app, then deploy or reconcile the monitoring stack.
9. Verify targets, alert rules, dashboards, and Alertmanager delivery.

Common failures:

• Prometheus cannot see app targets because metrics.prometheusAccess.enabled is missing or false.
• Prometheus is watching the wrong namespaces because prometheus.targetNamespaces is incomplete.
• Prometheus selectors do not match because the app instance is missing from prometheus.targetInstances.
• Alerts are created but not evaluated because PrometheusRule resources are not discoverable.
• Dashboards show cross-environment data because PromQL does not filter by namespace.
• Alert names collide because they omit ${p2p_environment}.

Monitoring Application

To monitor start monitoring your application, you have to create ServiceMonitor or PodMonitor resource for your application and put monitoring/instance:: {{ your-tenant-name }} label on it. By using tenant label, you define the target prometheus instance. Here is the example:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: some-service
  labels:
    app.kubernetes.io/name: some-service
    monitoring/instance: { { your-tenant-name } }
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: some-service
  endpoints:
    - interval: 30s
      port: metrics
      path: /metrics

This configuration assumes that your application exposes metrics on metrics endpoint in Prometheus format.

Accessing Grafana and Prometheus

Grafana and Prometheus are not exposed by default. You have two options:

• Create Ingress resources for internal domain
• Port forward Grafana and Prometheus

Through Ingress

To enable ingress for monitoring services, you have to set the respective values during the installation of the monitoring-stack chart. It should look like this:

#Install helm chart
helm -n {{ target-ns }} install monitoring-stack coreeng/monitoring-stack \
    --set tenantName={{ your-tenant-name }} \
    --set internalServicesDomain={{ internal-services-domain }} \
    --set prometheus.ingress.enabled=true \
    --set grafana.ingress.enabled=true

To upgrade installed monitoring stack, run the following command:

#Upgrade installed chart
helm -n {{ target-ns }} upgrade monitoring-stack coreeng/monitoring-stack \
    --set tenantName={{ your-tenant-name }} \
    --set internalServicesDomain={{ internal-services-domain }} \
    --set prometheus.ingress.enabled=true \
    --set grafana.ingress.enabled=true

• Grafana should be accessible by the URL: {{ your-tenant-name }}-grafana.{{ internal-services-domain }}
• Prometheus should be accessible by the URL: {{ your-tenant-name }}-prometheus.{{ internal-services-domain }}

Through port forward

To port forward Grafana, run the command below and access it on http://localhost:3000 

kubectl -n {{ target-ns }} port-forward service/grafana-{{ your-tenant-name }}-service 3000

To port forward Prometheus, run the command below and access it on http://localhost:9090 :

kubectl -n {{ target-ns }} port-forward service/prometheus-{{ your-tenant-name }} 9090

Creating Grafana Dashboards

To create Grafana Dashboard you have to create GrafanaDashboard CR. If you create it directly with UI, the changes will not be persisted for long, so it’s advised to use Grafana UI for designing your Dashboard and then export it to CR.

When creating GrafanaDashboard, you have to specify Grafana instance selector, so Grafana Operator can inject your dashboard in Grafana instance:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: my-grafana-dashboard
spec:
  instanceSelector:
    matchLabels:
      grafana: grafana-{{ your-tenant-name }}

In addition, in case your GrafanaDashboard is not in the same namespace as your monitoring stack, you have to specify an additional field:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: my-grafana-dashboard
spec:
  instanceSelector:
    matchLabels:
      grafana: grafana-{{ your-tenant-name }}
  allowCrossNamespaceImport: "true"

Creating Grafana Datasources

To create Grafana Datasources you have to create GrafanaDatasource CR. If you create it directly with UI, the changes will not be persisted for long, so it’s advised to use Grafana UI for designing your Datasource and then export it to CR.

When creating GrafanaDatasource, you have to specify Grafana instance selector, so Grafana Operator can inject your datasource in Grafana instance:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
  name: my-datasource
spec:
  instanceSelector:
    matchLabels:
      grafana: grafana-{{ your-tenant-name }}

In addition, in case your GrafanaDatasource is not in the same namespace as your monitoring stack, you have to specify an additional field:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
  name: my-datasource
spec:
  instanceSelector:
    matchLabels:
      grafana: grafana-{{ your-tenant-name }}
  allowCrossNamespaceImport: "true"

Using Grafana Plugins for Dashboards/Datasources

To create Grafana plugins for dashboards or datasources, you need to specify an additional field, plugins.

Here is an example of a plugin for a datasource:

kind: GrafanaDatasource
metadata:
  name: my-questdb
spec:
  instanceSelector:
    matchLabels:
      grafana: grafana-{{ your-tenant-name }}
  plugins:
    - name: questdb-questdb-datasource
      version: 0.1.4
  datasource:
    name: questdb
    access: proxy
    type: questdb-questdb-datasource
    database: qdb
    url: my-questdb-headless.questdb.svc:8812
    user: admin
    jsonData:
      sslmode: disable
      maxOpenConns: 100
      maxIdleConns: 100
      connMaxLifetime: 14400
    secureJsonData:
      password: quest

Here is an example of a plugin for dashboards:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: keycloak-dashboard
spec:
  instanceSelector:
    matchLabels:
      grafana: grafana-{{ your-tenant-name }}
  plugins:
    - name: grafana-piechart-panel
      version: 1.3.9
  json: >
  {
    ...
  }