Tag: Grafana

[Series] Homelab

Bài viết này viết về homelab cơ bản mình đang sử dụng. Vẫn còn cần nhiều improve, tuy nhiên do 1 số hạn chế kiến thức nên vẫn chưa được đầy đủ.

Diagram

https://lucid.app/lucidchart/af457311-6e50-4e83-9249-80f7df36b249/edit?viewport_loc=-98%2C102%2C2083%2C1334%2C0_0&invitationId=inv_1a10f138-8d09-4814-b475-f15a1d1c316a

Homelab diagram
Continue reading [Series] Homelab

Ubuntu install Grafana

sudo apt-get install -y apt-transport-https software-properties-common wget

sudo mkdir -p /etc/apt/keyrings/

wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null

echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

# Updates the list of available packages
sudo apt-get update

# Installs the latest Enterprise release:
sudo apt-get install grafana-enterprise

Prometheus

#!/bin/bash

echo 'Create tmp dir'
mkdir -p /tmp/prometheus

echo 'Change directory'
cd /tmp/prometheus

echo 'Download Prometheus'
PROMETHEUS_VERSION=$(curl -Ls https://api.github.com/repos/prometheus/prometheus/releases/latest | jq ".tag_name" | xargs | cut -c2-)
wget "https://github.com/prometheus/prometheus/releases/download/v${PROMETHEUS_VERSION}/prometheus-${VERSION}.linux-amd64.tar.gz"
tar xvzf prometheus-${VERSION}.linux-amd64.tar.gz
cd ./prometheus-${VERSION}.linux-amd64

echo 'Create directories'
for i in rules rules.d files_sd; do sudo mkdir -p /etc/prometheus/${i}; done
sudo mkdir /var/lib/prometheus

echo 'Create user & group'
sudo groupadd --system prometheus
sudo useradd -s /sbin/nologin --no-create-home --system -g prometheus prometheus

echo 'Deploy'
sudo mv prometheus promtool /usr/local/bin/
sudo mv prometheus.yml /etc/prometheus/prometheus.yml
sudo mv consoles/ console_libraries/ /etc/prometheus/

echo 'Chown'
sudo chown -R prometheus:prometheus /usr/local/bin/prometheus
sudo chown -R prometheus:prometheus /usr/local/bin/promtool
sudo chown -R prometheus:prometheus /etc/prometheus
sudo chown -R prometheus:prometheus /var/lib/prometheus

echo 'Create service'
sudo tee /etc/systemd/system/prometheus.service<<EOF
[Unit]
Description=Prometheus
Documentation=https://prometheus.io/docs/introduction/overview/
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecReload=/bin/kill -HUP \$MAINPID
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=0.0.0.0:9090 \
  --web.external-url=

SyslogIdentifier=prometheus
Restart=always

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl enable prometheus

Grafana – Install Loki

A Loki-based logging stack consists of 3 components:

  • Promtail is the agent, responsible for gathering logs and sending them to Loki.
  • Loki is the main server, responsible for storing logs and processing queries.
  • Grafana for querying and displaying the logs.
wget https://github.com/grafana/loki/releases/download/v2.9.8/loki-linux-amd64.zip
chmod a+x loki-linux-amd64
sudo cp loki-linux-amd64 /usr/local/bin/loki
loki --version
  • Create Systemd service
sudo nano /etc/systemd/system/loki.service
[Unit]
Description=Loki is a log aggregation system designed to store and query logs from all your applications and infrastructure.
Document=https://github.com/grafana/loki/releases
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/loki -config.file /mnt/data/loki/loki-config.yaml
Restart=on-failure
RestartSec=10
StandardOutput=append:/mnt/data/loki/logs/loki.log
StandardError=append:/mnt/data/loki/logs/loki.log

[Install]
WantedBy=multi-user.target

# This is a complete configuration to deploy Loki backed by the filesystem.
# The index will be shipped to the storage via tsdb-shipper.

auth_enabled: false

server:
  http_listen_port: 3100

common:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory
  replication_factor: 1
  path_prefix: /mnt/data/loki
  storage:
    filesystem:
      chunks_directory: /mnt/data/loki/chunks
      rules_directory: /mnt/data/loki/rules

schema_config:
  configs:
  - from: 2020-05-15
    store: tsdb
    object_store: filesystem
    schema: v13
    index:
      prefix: index_
      period: 24h

storage_config:
  filesystem:
    directory: /mnt/data/loki/chunks

Grafana – Monitor các thiết bị

Ở các bài trước ta đã setup được cơ bản server cho web hosting, gatekeeper. Vậy post này ta setup Grafana trên `gatekeeper` để monitor các server(s).

Đầu tiên là install Grafana . Cái này quite simple chỉ cần follow up guide trên website của hãng là okay. By default Grafana sử dụng port 3000

Sau đó install Prometheus

  • Setup Group & User
sudo groupadd prometheus
sudo useradd -s /sbin/nologin --system -g prometheus prometheus
  • Setup các directories cần thiết
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus

Directory “““/var/lib/prometheus sẽ dùng chứa data do đó suggest store trên HDD thay vì SSD chính để bảo toàn tuổi thọ cho SSD, cũng như không làm ảnh hưởng performance của SSD cho các việc khác liên quan tới OS.

Do đó trong case này mình chuyển về /mnt/data

sudo mkdir /mnt/data/prometheus

Latest version hiện tại là 2.52.02 / 2024-05-07

mkdir -p /tmp/prometheus
cd /tmp/prometheus
curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest | grep browser_download_url | grep linux-amd64 | cut -d '"' -f 4 | wget -qi -
tar xvf prometheus*.tar.gz
cd prometheus*/

Tiếp theo là cd vào directory vừa unzip ra và move binaries về “`/usr/local/bin

sudo mv prometheus /usr/local/bin
sudo mv promtool /usr/local/bin
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool

Sau đó tạo file config ( ở directory song song với data )

sudo mv consoles /etc/prometheus
sudo mv console_libraries /etc/prometheus
sudo mv prometheus.yml /etc/prometheus
  • Tạo Systemd Service
[Unit]
Description=Power your metrics and alerting with the leading open-source monitoring solution
Documentation=https://prometheus.io/
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/prometheus \
    --config.file=/etc/prometheus/prometheus.yml \
    --storage.tsdb.path=/mnt/data/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries \
    --web.listen-address=0.0.0.0:9090 \
    --web.enable-lifecycle \
    --log.level=info

[Install]
WantedBy=multi-user.target

https://prometheus.io/docs/prometheus/latest/command-line/prometheus

Chú ý thay đổi path phù hợp

Enable service khi boot và start

sudo systemctl daemon-reload
sudo systemctl enable prometheus.service
sudo systemctl start prometheus.service
sudo systemctl status prometheus.service

Như vậy là xong. Prometheus sẽ chạy ở port 9090

Thực tế https://prometheus.jooservices.com

Ở các bài tiếp theo sẽ setup tiếp các exporters / loki