Prometheus and Grafana

Cluster Overview

Prometheus: Collects, stores, and aggregates metrics.
Grafana: Visualizes metrics collected by Prometheus and creates alerts.
Node Exporter: A Prometheus agent that collects server metrics.

Requirements:

Two Debian servers:
- Server 1: Prometheus (primary node) and Node Exporter.
- Server 2: Grafana and Node Exporter.

Step 1: Install Prometheus on Server 1

1. Download the Prometheus binary:

cd /opt
wget https://github.com/prometheus/prometheus/releases/download/v2.45.0/prometheus-2.45.0.linux-amd64.tar.gz

2. Extract the archive:

tar xvf prometheus-2.45.0.linux-amd64.tar.gz
mv prometheus-2.45.0.linux-amd64 prometheus

3. Set up directories for Prometheus:

sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo cp prometheus/prometheus /usr/local/bin/
sudo cp prometheus/promtool /usr/local/bin/
sudo cp -r prometheus/consoles /etc/prometheus
sudo cp -r prometheus/console_libraries /etc/prometheus
sudo cp prometheus/prometheus.yml /etc/prometheus/

4. Configure Prometheus:

Edit the Prometheus configuration file:

sudo nano /etc/prometheus/prometheus.yml

Replace the contents with:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node_exporter"
    static_configs:
      - targets: ["localhost:9100", "<Server2_IP>:9100"]

5. Set Up Prometheus as a Service:

sudo nano /etc/systemd/system/prometheus.service

Add the following configuration:

[Unit]
Description=Prometheus Monitoring
Wants=network-online.target
After=network-online.target

[Service]
User=root
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus/ \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

6. Start and Enable Prometheus:

sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus

Verify Prometheus:
- Open a browser and go to http://<Server1_IP>:9090 to access the Prometheus dashboard.

Step 2: Install Node Exporter on Both Servers

Node Exporter collects hardware and OS metrics from Linux servers.

1. Download Node Exporter:

cd /opt
wget https://github.com/prometheus/node_exporter/releases/download/v1.5.0/node_exporter-1.5.0.linux-amd64.tar.gz

2. Extract and Install Node Exporter:

tar xvf node_exporter-1.5.0.linux-amd64.tar.gz
sudo mv node_exporter-1.5.0.linux-amd64/node_exporter /usr/local/bin/

3. Set Up Node Exporter as a Service:

sudo nano /etc/systemd/system/node_exporter.service

Add the following configuration:

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=root
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=default.target

4. Start and Enable Node Exporter:

sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter

5. Verify Node Exporter:

Open a browser and go to http://<Server1_IP>:9100/metrics (and replace with Server2 IP to check the second node).

Step 3: Install Grafana on Server 2

Grafana will be used to visualize and monitor the metrics collected by Prometheus.

1. Install Grafana:

sudo apt-get install -y software-properties-common
sudo apt-get install -y apt-transport-https
sudo apt-get install -y wget
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt update
sudo apt install grafana -y

2. Start and Enable Grafana:

sudo systemctl enable grafana-server
sudo systemctl start grafana-server

3. Access Grafana:

Open a web browser and go to http://<Server2_IP>:3000.
Log in with default credentials (admin/admin) and change the password as prompted.

Step 4: Configure Prometheus as a Data Source in Grafana

Log into Grafana on Server 2.
Add Prometheus as a Data Source:
- Go to Configuration > Data Sources > Add data source.
- Choose Prometheus.
- In the URL field, enter http://<Server1_IP>:9090.
- Click Save & Test to verify the connection.

Step 5: Create Grafana Dashboards for Visualization

Create a New Dashboard:
- Go to + (Create) > Dashboard > Add new panel.
Select Metrics to Visualize:
- Choose metrics collected by Prometheus (e.g., node_cpu_seconds_total or node_memory_MemAvailable_bytes).
- Customize the panel options, visualization type (graph, gauge, heatmap), and threshold settings.
Save the Dashboard:
- Save the dashboard to monitor CPU usage, memory availability, disk space, etc., across your servers.

Step 6: Set Up Alerts in Grafana

Go to the Dashboard Panel you created.
Configure Alerts:
- Click on Alert > Create Alert.
- Set up the alert rule (e.g., when CPU usage is over 80%).
- Define notification channels (such as email, Slack, etc., set up under Grafana Alerting > Notification channels).
Save Alert Configuration:
- Grafana will now monitor this metric and alert you if the threshold is breached.

Conclusion

You’ve successfully set up a Prometheus and Grafana aggregation cluster on Debian 12. This environment is a powerful, real-time aggregation and monitoring system for tracking server performance metrics across multiple nodes, which can be easily scaled by adding more nodes with Node Exporter installed.

Aggregation