Installation of Prometheus and related tools

Network Monitoring and Management

Installation of Prometheus and related tools

In the labs, all these tools were pre-installed - you just had to start them. Here we document how they were installed, so you can repeat this at home.

Whilst there are some packages available in the Ubuntu repositories, these contain rather old versions so we recommend you install the release binaries as described here (or in the case of grafana, use their apt repository).

These steps all assume you are running as the root user, so use sudo -s if necessary to get a root shell:

$ sudo -s
# cd /root

Install Prometheus

Download and install

All the prometheus components are available as pre-built binaries.

Fetch and unpack the latest release from the releases page. You should go here first and then replace the “XX” and the “Y” below in the link with the current Prometheus version number that is listed on the releases page (scroll down a bit to find the link).

# cd /var/tmp
# wget https://github.com/prometheus/prometheus/releases/download/v2.XX.Y/prometheus-2.XX.Y.linux-amd64.tar.gz
# tar -C /opt -xvzf prometheus-2.XX.Y.linux-amd64.tar.gz

This may take a few minutes depending on your network speed.

Create a symbolic link from /opt/prometheus pointing to the version you downloaded (this makes it easier to upgrade and switch between versions). Remember to replace “XX” and “Y” with the version of prometheus you have downloaded.

If the prometheus symlink already exists, because you are upgrading from a previous version, then remove it first:

# rm /opt/prometheus

Then create a new symlink:

# ln -s prometheus-2.XX.Y.linux-amd64 /opt/prometheus

The symbolic link allows us to create the systemd unit file without needing to update it each time we upgrade Prometheus.

Configure prometheus

If you are installing Prometheus for the first time, then create a user for prometheus to run as, and a data directory:

# useradd --system -d /var/lib/prometheus prometheus
# mkdir /var/lib/prometheus
# chown prometheus:prometheus /var/lib/prometheus

Use a text editor to create a systemd unit file /etc/systemd/system/prometheus.service, for example:

# nano /etc/systemd/system/prometheus.service

with the following contents:

[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network-online.target

[Service]
User=prometheus
Restart=on-failure
RestartSec=5
TimeoutStopSec=300
WorkingDirectory=/opt/prometheus
EnvironmentFile=/etc/default/prometheus
ExecStart=/opt/prometheus/prometheus $OPTIONS
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

Tell systemd to read this new file:

# systemctl daemon-reload

Also create an options file /etc/default/prometheus with the following contents:

OPTIONS='--config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus/data/ --storage.tsdb.wal-compression --web.external-url=http://srv1-campusX.ws.nsrc.org/prometheus'

(Adjust campusX as appropriate). One of the consequences of this is that the prometheus API will response to /prometheus instead of /, which allows us to proxy easily from the Apache web server.

Create the initial default configuration:

# mkdir /etc/prometheus
# cp /opt/prometheus/prometheus.yml /etc/prometheus/

Edit /etc/prometheus/prometheus.yml and under job_name: 'prometheus' add:

    metrics_path: '/prometheus/metrics'

So, that the section of the file looks like:

  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"
    metrics_path: '/prometheus/metrics'

Finally, start it and check that it is running:

# systemctl enable prometheus
# systemctl start prometheus
# systemctl status prometheus

Install node_exporter

Fetch and unpack the latest release from the releases page and create a symlink so that /opt/node_exporter refers to the current version.

# cd /var/tmp
# wget https://github.com/prometheus/node_exporter/releases/download/vX.Y.Z/node_exporter-X.Y.Z.linux-amd64.tar.gz
# tar -C /opt -xvzf node_exporter-X.Y.Z.linux-amd64.tar.gz

If node exporter was already installed, then remove the existing symbolic link first:

# rm /opt/node_exporter

And, now create the symbolic link to the current node_exporter version:

# ln -s node_exporter-X.Y.Z.linux-amd64 /opt/node_exporter

The symbolic link allows us to create systemd unit file without needing to update it each time we upgrade node_exporter.

Use a text editor to create a systemd unit file /etc/systemd/system/node_exporter.service with the following contents:

[Unit]
Description=Prometheus Node Exporter
Documentation=https://github.com/prometheus/node_exporter
After=network-online.target

[Service]
User=root
EnvironmentFile=/etc/default/node_exporter
ExecStart=/opt/node_exporter/node_exporter $OPTIONS
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Tell systemd to read this new file:

# systemctl daemon-reload

Also create an options file /etc/default/node_exporter with the following contents:

OPTIONS=''

Finally, start it and check that it is running:

# systemctl enable node_exporter
# systemctl start node_exporter
# systemctl status node_exporter

Install snmp_exporter

Fetch and unpack the latest release from the releases page and create a symlink so that /opt/snmp_exporter refers to the current version.

# cd /var/tmp
# wget https://github.com/prometheus/snmp_exporter/releases/download/vX.Y.Z/snmp_exporter-X.Y.Z.linux-amd64.tar.gz
# tar -C /opt -xvzf snmp_exporter-X.Y.Z.linux-amd64.tar.gz
# rm /opt/snmp_exporter    # if it already exists
# ln -s snmp_exporter-X.Y.Z.linux-amd64 /opt/snmp_exporter

Use a text editor to create a systemd unit file /etc/systemd/system/snmp_exporter.service with the following contents:

[Unit]
Description=Prometheus SNMP Exporter
Documentation=https://github.com/prometheus/snmp_exporter
After=network-online.target

[Service]
User=prometheus
Restart=on-failure
RestartSec=5
EnvironmentFile=/etc/default/snmp_exporter
ExecStart=/opt/snmp_exporter/snmp_exporter $OPTIONS
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

Tell systemd to read this new file:

# systemctl daemon-reload

Also create an options file /etc/default/snmp_exporter with the following contents:

OPTIONS='--config.file=/etc/prometheus/snmp/snmp.yml --web.listen-address=127.0.0.1:9116'

Create the initial default configuration:

# mkdir /etc/prometheus/snmp
# cp /opt/snmp_exporter/snmp.yml /etc/prometheus/snmp/

Then start it and check that it is running:

# systemctl enable snmp_exporter
# systemctl start snmp_exporter
# systemctl status snmp_exporter

Install grafana

Grafana is available pre-packaged in many formats, including deb packages for Ubuntu.

curl -fsSL https://apt.grafana.com/gpg.key >/etc/apt/trusted.gpg.d/grafana.asc
echo "deb https://apt.grafana.com stable main" >/etc/apt/sources.list.d/grafana.list
apt update
apt install grafana

Note that the package is called “grafana” but the systemd service is “grafana-server”.

If you want grafana to run under the path prefix /grafana like we do in the labs, you will need to uncomment and set some values in /etc/grafana/grafana.ini:

domain = srv1-campusX.ws.nsrc.org

root_url = %(protocol)s://%(domain)s:%(http_port)s/grafana/

serve_from_sub_path = true

Install alertmanager

Fetch and unpack the latest release from the releases page and create a symlink so that /opt/alertmanager refers to the current version.

# cd /var/tmp
# wget https://github.com/prometheus/alertmanager/releases/download/vX.Y.Z/alertmanager-X.Y.Z.linux-amd64.tar.gz
# tar -C /opt -xvzf alertmanager-X.Y.Z.linux-amd64.tar.gz
# rm /opt/alertmanager     # if it already exists
# ln -s alertmanager-X.Y.Z.linux-amd64 /opt/alertmanager

Create a data directory:

# mkdir /var/lib/alertmanager
# chown prometheus:prometheus /var/lib/alertmanager

Use a text editor to create a systemd unit file /etc/systemd/system/alertmanager.service with the following contents:

[Unit]
Description=Prometheus Alertmanager
Documentation=https://prometheus.io/docs/alerting/alertmanager/
After=network-online.target

[Service]
User=prometheus
Restart=on-failure
RestartSec=5
WorkingDirectory=/var/lib/alertmanager
EnvironmentFile=/etc/default/alertmanager
ExecStart=/opt/alertmanager/alertmanager $OPTIONS
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

Tell systemd to read this new file:

# systemctl daemon-reload

Also create an options file /etc/default/alertmanager with the following contents:

OPTIONS='--config.file=/etc/prometheus/alertmanager.yml --web.external-url=http://srv1-campusX.ws.nsrc.org/alertmanager'

(adjust campusX as appropriate)

Create the initial default configuration:

# cp /opt/alertmanager/alertmanager.yml /etc/prometheus/

Finally, start it and check that it is running:

# systemctl enable alertmanager
# systemctl start alertmanager
# systemctl status alertmanager

Install karma

# cd /var/tmp
# wget https://github.com/prymitive/karma/releases/download/vX.Y/karma-linux-amd64.tar.gz
# mkdir /opt/karma
# tar -C /opt/karma -xvzf karma-linux-amd64.tar.gz

Create /etc/systemd/system/karma.service

[Service]
User=prometheus
Restart=on-failure
RestartSec=5
WorkingDirectory=/var/lib/alertmanager
EnvironmentFile=/etc/default/karma
ExecStart=/opt/karma/karma-linux-amd64 $OPTIONS

[Install]
WantedBy=multi-user.target

Tell systemd to read this new file:

# systemctl daemon-reload

Also create an options file /etc/default/karma with the following contents:

OPTIONS='--config.file=/etc/prometheus/karma.yml'

In order to start karma, you will first need to create the configuration file /etc/prometheus/karma.yml. See the alertmanager exercise for an example.

Install blackbox_exporter

Fetch and unpack the latest release from the releases page and create a symlink so that /opt/blackbox_exporter refers to the current version.

# cd /var/tmp
# wget https://github.com/prometheus/blackbox_exporter/releases/download/vX.Y.Z/blackbox_exporter-X.Y.Z.linux-amd64.tar.gz
# tar -C /opt -xvzf blackbox_exporter-X.Y.Z.linux-amd64.tar.gz
# rm /opt/blackbox_exporter     # if it already exists
# ln -s blackbox_exporter-X.Y.Z.linux-amd64 /opt/blackbox_exporter

Use a text editor to create a systemd unit file /etc/systemd/system/blackbox_exporter.service with the following contents:

[Unit]
Description=Prometheus Blackbox Exporter
Documentation=https://github.com/prometheus/blackbox_exporter
After=network-online.target

[Service]
User=prometheus
Restart=on-failure
RestartSec=5
EnvironmentFile=/etc/default/blackbox_exporter
AmbientCapabilities=CAP_NET_RAW
ExecStart=/opt/blackbox_exporter/blackbox_exporter $OPTIONS
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

Tell systemd to read this new file:

# systemctl daemon-reload

Also create an options file /etc/default/blackbox_exporter with the following contents:

OPTIONS='--web.listen-address=127.0.0.1:9115 --config.file=/etc/prometheus/blackbox.yml'

In order to start it, first you will need to create a configuration file in /etc/prometheus/blackbox.yml. There is a sample in /opt/blackbox_exporter/blackbox.yml that you can copy and edit.

Install VictoriaMetrics

# cd /var/tmp
# wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/vX.Y.Z/victoria-metrics-vX.Y.Z.tar.gz
# mkdir /opt/victoria-metrics
# tar -C /opt/victoria-metrics -xvzf victoria-metrics-vX.Y.Z.tar.gz

# mkdir -p /var/lib/victoria-metrics/data
# chown prometheus:prometheus /var/lib/victoria-metrics/data

Create /etc/systemd/system/victoria-metrics.service

[Unit]
Description=VictoriaMetrics server
Documentation=https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/Single-server-VictoriaMetrics#operation
After=network-online.target
# Fix shutdown delays: if prometheus is running on the same host,
# VictoriaMetrics should start first, and shutdown after.
Before=prometheus.service

[Service]
User=prometheus
Restart=on-failure
RestartSec=5
WorkingDirectory=/var/lib/victoria-metrics
EnvironmentFile=/etc/default/victoria-metrics
ExecStart=/opt/victoria-metrics/victoria-metrics-prod $OPTIONS

[Install]
WantedBy=multi-user.target

Tell systemd to read this new file:

# systemctl daemon-reload

Create /etc/default/victoria-metrics

OPTIONS='-storageDataPath=/var/lib/victoria-metrics/data -retentionPeriod=6 -httpAuth.username=admin -httpAuth.password=password123 -http.pathPrefix'

(this sets a 6 months retention period, and enables simple username/password authentication)

To start:

# systemctl enable victoria-metrics
# systemctl start victoria-metrics
# systemctl status victoria-metrics