Overview on Prometheus and Grafana Monitoring

Prometheus and Grafana Monitoring is a powerful combination for collecting, analyzing, and visualizing server metrics in real-time. Prometheus is an open-source monitoring and alerting toolkit, while Grafana provides a flexible dashboard to visualize the collected data.

This guide covers the installation and configuration of Prometheus and Grafana on Debian 12. You will learn how to set up exporters for data collection, configure Prometheus to scrape metrics, and create insightful dashboards in Grafana.

Requirements:

• Two Debian servers:
  • Server 1: Prometheus (primary node) and Node Exporter.
  • Server 2: Grafana and Node Exporter.

Step 1: Install Prometheus on Server 1

1. Download the Prometheus binary:

cd /opt
wget https://github.com/prometheus/prometheus/releases/download/v3.2.1/prometheus-3.2.1.linux-amd64.tar.gz

2. Extract the archive:

tar xvf prometheus-3.2.1.linux-amd64.tar.gz
mv prometheus-3.2.1.linux-amd64 prometheus

3. Set up directories for Prometheus:

sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo cp prometheus/prometheus /usr/local/bin/
sudo cp prometheus/promtool /usr/local/bin/
sudo cp -r prometheus/consoles /etc/prometheus
sudo cp -r prometheus/console_libraries /etc/prometheus
sudo cp prometheus/prometheus.yml /etc/prometheus/

4. Configure Prometheus:

Edit the Prometheus configuration file:

sudo nano /etc/prometheus/prometheus.yml

Replace the contents with:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node_exporter"
    static_configs:
      - targets: ["localhost:9100", "<Server2_IP>:9100"]

5. Set Up Prometheus as a Service:

sudo nano /etc/systemd/system/prometheus.service

Add the following configuration:

[Unit]
Description=Prometheus Monitoring
Wants=network-online.target
After=network-online.target

[Service]
User=root
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus/ \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

6. Start and Enable Prometheus:

sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus

1. Verify Prometheus:
   • Open a browser and go to http://<Server1_IP>:9090 to access the Prometheus dashboard.

Step 2: Install Node Exporter on Both Servers

Node Exporter collects hardware and OS metrics from Linux servers.

1. Download Node Exporter:

cd /opt
wget https://github.com/prometheus/node_exporter/releases/download/v1.9.0/node_exporter-1.9.0.linux-amd64.tar.gz

2. Extract and Install Node Exporter:

tar xvf node_exporter-1.9.0.linux-amd64.tar.gz
sudo mv node_exporter-1.9.0.linux-amd64/node_exporter /usr/local/bin/

3. Set Up Node Exporter as a Service:

sudo nano /etc/systemd/system/node_exporter.service

Add the following configuration:

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=root
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=default.target

4. Start and Enable Node Exporter:

sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter

5. Verify Node Exporter:

• Open a browser and go to http://<Server1_IP>:9100/metrics (and replace with Server2 IP to check the second node).

Step 3: Install Grafana on Server 2

Grafana will be used to visualize and monitor the metrics collected by Prometheus.

1. Install Grafana:

sudo apt-get install -y software-properties-common
sudo apt-get install -y apt-transport-https
sudo apt-get install -y wget
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt update
sudo apt install grafana -y

2. Start and Enable Grafana:

sudo systemctl enable grafana-server
sudo systemctl start grafana-server

3. Access Grafana:

• Open a web browser and go to http://<Server2_IP>:3000.
• Log in with default credentials (admin/admin) and change the password as prompted.

Step 4: Configure Prometheus as a Data Source in Grafana

1. Log into Grafana on Server 2.
2. Add Prometheus as a Data Source:

• Go to Configuration > Data Sources > Add data source.
• Choose Prometheus.
• In the URL field, enter http://<Server1_IP>:9090.
• Click Save & Test to verify the connection.
• Prometheus best practices

Step 5: Create Grafana Dashboards for Visualization

1. Create a New Dashboard:
   • Go to + (Create) > Dashboard > Add new panel.
2. Select Metrics to Visualize:
   • Choose metrics collected by Prometheus (e.g., node_cpu_seconds_total or node_memory_MemAvailable_bytes).
   • Customize the panel options, visualization type (graph, gauge, heatmap), and threshold settings.
3. Save the Dashboard:
   • Save the dashboard to monitor CPU usage, memory availability, disk space, etc., across your servers.

Step 6: Set Up Alerts in Grafana

1. Go to the Dashboard Panel you created.
2. Configure Alerts:
   • Click on Alert > Create Alert.
   • Set up the alert rule (e.g., when CPU usage is over 80%).
   • Define notification channels (such as email, Slack, etc., set up under Grafana Alerting > Notification channels).
3. Save Alert Configuration:
   • Grafana will now monitor this metric and alert you if the threshold is breached.

Conclusion on Prometheus and Grafana Monitoring

You’ve successfully set up a Prometheus and Grafana Monitoring cluster on Debian 12. This environment is a powerful, real-time aggregation and monitoring system for tracking server performance metrics across multiple nodes, which can be easily scaled by adding more nodes with Node Exporter installed.

Learn more about aggregation and about monitoring.