Loki is a scalable log storage engine which is simple to install and run. It proves an API for adding and querying logs, and can be queried via a Grafana web interface.

You are going to work with your classmates to configure the following logging pipeline:

    (devices) ----> rsyslog ----> promtail ----> loki

All the commands in this sheet can be run as “root”. Use `sudo -s` to get a root shell if necessary.

# Start loki

Loki has been pre-installed on your campus server instance (srv1-campusX.ws.nsrc.org) inside the “prometheus” container, and promtail directly in srv1 itself.

ONE of the users in your group should log into your srv1 instance and start the prometheus container:

    incus start prometheus

Enter the container (this gives you a root shell inside it):

    incus shell prometheus

Check that loki is running:

    systemctl status loki

You can check it is running by making a query to its API:

    curl localhost:3100/ready

It should respond with “ready”, but it may take a minute or two before it gets into that state. Until then, you will see responses like:

    Ingester not ready: waiting for 15s after being ready
    Pattern Ingester not ready: waiting for 15s after being ready

Repeat the curl command until you get “ready”.

Now exit the prometheus container to get back to srv1:

    exit

# Ingesting logs

The tool “promtail” can be used to pick up logs and forward them to loki. It supports reading logs from:

- syslog (framed TCP)
- local logfiles (e.g. Apache logs)
- journald

Again, ONE user in your group should do the following. Make sure you are logged into srv1, and switch to the root user if not already:

    sudo -s

On srv1, you should find there’s a configuration we’ve already provided in `/etc/loki/promtail.yaml`. Check that it’s there, using `cat /etc/loki/promtail.yaml`

    # Promtail can expose prometheus metrics
    server:
      http_listen_address: 127.0.0.1
      http_listen_port: 9080
      grpc_listen_port: 0

    # How to find the loki server(s)
    clients:
      - url: http://prometheus.nmm.internal:3100/loki/api/v1/push

    # Logs to read and forward
    scrape_configs:
      - job_name: syslog
        syslog:
          listen_address: 127.0.0.1:5140
        relabel_configs:
          - source_labels: [__syslog_message_severity]
            target_label: severity
          - source_labels: [__syslog_message_facility]
            target_label: facility
          - source_labels: [__syslog_message_hostname]
            target_label: host
          - source_labels: [__syslog_message_app_name]
            target_label: app

This configuration listens for syslog messages on TCP port 5140 and forwards them to loki. Now start it:

    systemctl enable promtail   # start on future boots
    systemctl start promtail    # start now
    journalctl -eu promtail     # check for no errors during startup

Now we need to configure rsyslog on srv1 to forward to promtail. Create a file `/etc/rsyslog.d/25-promtail.conf` with the following contents:

    *.*     action(type="omfwd" protocol="tcp"
                   target="127.0.0.1" port="5140"
                   Template="RSYSLOG_SyslogProtocol23Format"
                   TCP_Framing="octet-counted")

(Note that `*.*` matches messages of all facility and severity, so we will forward both router logs and srv1’s own system logs to loki)

Restart rsyslog and check for errors:

    systemctl restart rsyslog
    journalctl -eu rsyslog

Now send a test message:

    logger "testing 123"

You should see that this still appears in the files that rsyslog itself writes:

    grep testing /var/log/syslog

However, hopefully it has also been sent via promtail to loki. To check this, we need to perform some queries.

# Querying

This part can be done by everyone in the group.

The command-line tool for querying loki is called “logcli”. Log into srv1, then enter the prometheus container:

    incus shell prometheus

then type “logcli” by itself to get a big page of help.

Every log message is categorised using one or more “labels”. Assuming that at least one log message has been ingested, we should be able to query for labels now. Try this command:

    logcli labels

The response should look something like this:

    http://localhost:3100/loki/api/v1/labels?end=1729001335837139798&start=1728997735837139798
    __aggregated_metric__
    app
    facility
    host
    level
    service_name
    severity

Note that the first line of logcli output is the API URL that it is communicating with. The remainder are the labels which exist in the log index. Hopefully there will be labels “host”, “facility”, “severity”. If not, see “Troubleshooting” below.

To check what different values of this label loki has seen, run this:

    logcli labels host

In other words, this is the list of all the hosts which loki has seen messages from. Hopefully you’ll see “srv1-campusY” here - if not, see “troubleshooting” below. (Again, the first line is just the URL that the query used)

Now you can create a log query using logQL. The simplest query is just to show all log messages with a given label value:

    logcli query '{host="srv1-campusY"}'

(change “Y” to your campus)

You should now see some log messages. Notice how each line starts with a timestamp and a list of labels. If there are any labels which are the same on all the messages shown, they are extracted as “common labels” to reduce clutter.

logcli by default will show you the last 30 matching logs over the last hour. There are flags available to change this:

    logcli query '{host="srv1-campusY"}' --limit=10 --since=2h

## Troubleshooting

If you are not able to see any logs, here are some things to check on srv1:

- Check that services are running

      incus exec prometheus -- systemctl status loki
      systemctl status promtail
      systemctl status rsyslog

- Check for errors in service logs

      incus exec prometheus -- journalctl -eu loki
      journalctl -eu promtail
      journalctl -eu rsyslog

- Run tcpdump on “localhost” socket

  rsyslog->promtail communicates on the loopback interface (localhost, 127.0.0.1) so you can check this using tcpdump:

      tcpdump -i lo -nnA -s0

  Now in another window generate a log message

      logger "testing 123"

  Do you see any connection on tcp port 5140 (rsyslog to promtail)? Similarly, to check promtail to loki, which is in a different container reached via the ‘bridge0’ interface:

      tcpdump -i bridge0 -nnA -s0 tcp port 3100

# Using LogQL

Here are some examples you can try of LogQL, loki’s log query language.

You can select messages which match multiple labels simultaneously:

    logcli query '{host="srv1-campusY",severity="notice"}'

You can filter for messages which contain a given string:

    logcli query '{host="srv1-campusY"} |="test"'

Multiple filter conditions can be chained: you will get messages which match all the conditions simultaneously. Here you will see messages which contain the string “test” and contain the string “123”.

    logcli query '{host="srv1-campusY"} |="test" |="123"'

Or for messages which do not contain a given string:

    logcli query '{host="srv1-campusY"} !="test"'

For more advanced uses, use regular expression operators with `~` instead of `=`. For example:

    # severity label is "warning", "error" or "critical"
    logcli query '{severity=~"warning|error|critical"}'

    # message contains "test" or "done", but does not contain "plugins" or "alerts"
    logcli query '{host="srv1-campusY"} |~"test|done" !~"plugins|alerts"'

## Live tailing

“Live tailing” lets you see logs as they arrive in real time. Run this command:

    logcli query '{host="srv1-campusY"}' -t

Now login to another console and type:

    logger boom

Go back to the original console where logcli is running, and the message should appear.

# Aggregated logs

If you have already done the rsyslog exercise, then your network device logs will already be arriving at srv1 - and therefore should also be arriving in loki. Can you see them?

Hints:

- Check which label values you have for the “host” label
- Check which label values you have for the “facility” label (remembering that you configured devices to send messages with facility “local0”)
- Query logs for a specific device, or query all logs with facility “local0”

If you need any help, please ask your instructor. Remember that you’ll only see logs which arrived after you configured rsyslog to talk to promtail/loki, so if necessary you can login to one of your devices and make a config change, to force it to send some logs.

# Grafana

Grafana can serve as a web interface to loki, including LogQL and live tailing.

## Start Grafana

The grafana server should already be running inside the prometheus container. You can check by getting a shell inside the prometheus container and checking the status:

    incus shell prometheus
    journalctl -eu grafana-server     # check for no errors during startup
    exit

Or as a one-liner from srv1:

    incus exec prometheus -- systemectl status grafana-server

## Configure Grafana

ONE of the users in your group should do the following configuration.

Login to the Grafana interface at http://oob-srv1-campusX.ws.nsrc.org/grafana. The initial login is “admin” and “admin”. It will prompt you to change the password - use the class password that everyone knows.

On the left-hand side, open the “Connections” menu and under here select “Data sources”.

Click the blue button “Add data source”.

From the list of data sources shown, under “Logging & document databases”, click on “Loki”

Under HTTP, in the URL field, type `http://localhost:3100` (it may look like it’s already there, but you still need to type over it)

Click the blue `Save & test`. If successful, you should get a green banner saying “Data source successfully connected”.

## Browse Grafana

ALL of the users in your group can now do the remaining exercise.

Login to the Grafana web interface as above, using the class password.

From the left-hand set of icons, click on “Explore”.

At the very top of the page, next to “Outline”, ensure that “Loki” is selected from the drop-down menu.

You can now explore logs by selecting the “Log labels” menu, selecting any particular label (for example “host”) and then selecting one of the labels offered (e.g. “srv1-campusY”). Then click “Run query” at the top right.

This will build a query for you: `{host="srv1-campusY"}` and display the matching logs. You will get a bar graph showing the number of matching logs per second, and the logs themselves shown at the end.

Toggle the “Unique labels” switch to see the labels next to each message.

When you click on an individual message to expand it, there are filter buttons which let you drill down to logs with matching labels. For example, if you look at a cron line and it has label “app” value “CRON”, then clicking the plus-magnifying glass will adjust the query so that it shows only logs with `app="CRON"`.

Try a query which matches on the content of the message. Under “Line contains” (Text to find), enter “pam” (without the quotes). This should build the following query, which shows any message which contains the string “pam”:

    {host="srv1-campusY"} |="pam"

At the very top-right, next to Run Query, is a button “Live”. Click on this to get live log tailing. Login to srv1 from another machine, and the web browser should show additional matching logs as they occur.

Grafana has an experimental “Explore Logs” plugin to make it easier to browse logs without using PromQL. If you want to try it out, click “Logs” under the expanded “Explore” in the left-hand menu:

    v  🧭 Explore
          Metrics
          Logs     << click here

# Additional exercises

These are optional improvements to the logging configuration - you can try them out if you have time, or just keep them as a reference.

## Logging source IP addresses

The “hostname” field within a syslog message is easily forged, and it would be more secure if we could also record the originating IP address of each message.

We can do this by getting rsyslog to send `hostname/ipaddress` in the hostname field - this is a convention started by syslog-ng - and getting promtail to split this into two labels.

Edit `/etc/loki/promtail.yaml` and replace the `relabel_configs` section of the syslog job, so it looks like this:

    ... leave the first part ...

    scrape_configs:
      - job_name: syslog
        syslog:
          listen_address: 127.0.0.1:5140
        relabel_configs:
          - source_labels: [__syslog_message_severity]
            target_label: severity
          - source_labels: [__syslog_message_facility]
            target_label: facility
          - source_labels: [__syslog_message_hostname]
            regex: '([^/]*)'
            target_label: host
          - source_labels: [__syslog_message_hostname]
            regex: '(.*)/(.*)'
            replacement: '$1'
            target_label: host
          - source_labels: [__syslog_message_hostname]
            regex: '(.*)/(.*)'
            replacement: '$2'
            target_label: ip
          - source_labels: [__syslog_message_app_name]
            target_label: app

Hostnames which don’t contain a slash are copied to the “host” label as before. But for hostnames which do contain a slash, the part before goes to “host” and the part afterwards to “ip”.

Restart promtail and check for errors:

    systemctl restart promtail
    journalctl -eu promtail

Now replace `/etc/rsyslog.d/25-promtail.conf` with the following:

    template(name="Custom_SyslogProtocol23Format" type="string"
         string="<%PRI%>1 %TIMESTAMP:::date-rfc3339% %HOSTNAME%/%FROMHOST-IP% %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n")

    *.*     action(type="omfwd" protocol="tcp"
                   target="127.0.0.1" port="5140"
                   Template="Custom_SyslogProtocol23Format"
                   TCP_Framing="octet-counted")

This replaces the built-in RSYSLOG_SyslogProtocol23Format template with a custom version with `%HOSTNAME%/%FROMHOST-IP%` in the hostname field.

Restart rsyslog and check for errors:

    systemctl restart rsyslog
    journalctl -eu rsyslog

Make some logs (e.g. login to router and do `conf t`, `exit`)

Check to see if a new “ip” label has been seen, and what values it has:

    incus shell prometheus   # to enter the prometheus container
    logcli labels
    logcli labels ip

Query your logs using the source IP label as a filter:

    logcli query '{ip="100.68.X.Y"}'

(replacing X and Y as appropriate)

You can match multiple values with a regular expression. In a regular expression, `.` matches any character, so you need `\.` to match a literal dot. `\d` matches any single digit (0-9), and `\d+` matches any sequence of one or more digits.

However in LogQL the backslashes need to be doubled-up to make a valid query:

    logcli query '{ip=~"100\\.68\\.\\d+\\.\\d+"}'

To avoid this issue, you can surround the entire value with backticks instead of double-quotes (new feature in loki 1.5):

    logcli query '{ip=~`100\.68\.\d+\.\d+`}'

!!! Note

    In all cases, the single-quotes outside the expression are very important.
    These stop the shell from doing its own expansions on backslashes and
    backticks.

## Improving rsyslog parsing

Unfortunately, there is wide variation of how different vendor devices format logs. The “standard” defines a tag field:

    timestamp hostname tag message

The tag can include the process name and pid, like `sshd[12345]:`. But some devices don’t include a tag: this means you may get a device which sends a message like this:

    Invalid sequence number

but it gets interpreted by rsyslog (and forwarded to loki) as:

    app="Invalid"
    msg=" sequence number"

This displays wrongly in loki and is harder to search.

You can fix this by tweaking some settings in rsyslog, so that it only splits out the tag field if it ends in a colon. To do this, edit `/etc/rsyslog.conf`, find this section:

    # provides UDP syslog reception
    module(load="imudp")
    input(type="imudp" port="514")

and add the following after it (replacing any RulesetParser section you already have):

    # Some devices miss the "tag" field from syslog messages
    # https://www.rsyslog.com/doc/v8-stable/configuration/modules/pmrfc3164.html
    parser(name="custom.rfc3164"
          type="pmrfc3164"
          force.tagEndingByColon="on"
          permit.squareBracketsInHostname="on"
          detect.YearAfterTimestamp="on")
    # Replace the default parser chain
    # https://www.rsyslog.com/doc/v8-stable/configuration/ruleset/rsconf1_rulesetparser.html
    module(load="pmciscoios")
    $RulesetParser rsyslog.ciscoios
    $RulesetParser rsyslog.rfc5424
    $RulesetParser custom.rfc3164

This still allows devices to send using the newer and more strictly defined rfc5424 format, but if not, falls back to our customised rfc3164 parser which requires tags to end with a colon.

(Why isn’t this the default behaviour? It’s because the original “standard” RFC 3164 doesn’t require a colon at the end of the tag, so rsyslog follows the standard. But in practice, most devices which send a tag do end it with a colon)

## Application log aggregation

On srv1, there is a log file `/var/log/apache2/access.log`. Can you get this into loki too? Yes - by using promtail to read it.

Edit `/etc/loki/promtail.yaml` and add the following to the end under the `scrape_jobs` section. Alignment is important: the dash before `job_name` must align exactly with the dash of the previous `job_name`.

      - job_name: apache2_access
        static_configs:
          - targets: [localhost]
            labels:
              job: apache2
              __path__: /var/log/apache2/access.log

Restart promtail and check for errors:

    systemctl restart promtail
    journalctl -eu promtail

Using your web browser, refresh a page on srv1, or generate a web request from the command line:

    # do this on srv1, not inside the prometheus container
    curl localhost

Now inside the prometheus container check your logs:

    incus shell prometheus
    logcli query '{job="apache2"}'

The results should look something like this:

    Common labels: {filename="/var/log/apache2/access.log", job="apache2"}
    2020-03-06T14:39:01Z {} 127.0.0.1 - - [06/Mar/2020:14:39:01 +0000] "GET / HTTP/1.1" 200 1856 "-" "curl/7.47.0"

You are storing the raw log lines.

It is possible to perform additional processing in a pipeline, to extract fields from this into labels. However, you need to be careful to avoid “high cardinality” labels - that means labels which can have many different values. Loki assembles logs with the same labels into the same chunks, but if you have many different labels then loki can be forced to track many small chunks which makes it very inefficient and may crash.

For example, the source IP address may be high cardinality if your web site could be accessed from anywhere on the Internet - this means it is not a good candidate for a label. However the status code (e.g. “200”) only has a small number of possible values so is safe in a label, and also useful (e.g. you can quickly search for requests which generated a “500” status)

To test this, add the following to the end of the scrape config. The new section “pipeline_stages” should line up with “static_configs” above it.

        pipeline_stages:
          - regex:
              expression: '^\S+ \S+ \S+ \[[^\]]*\] "[^"]*" (?P<code>\d+) \d+'
          - labels:
              code:

Restart promtail again. Generate some requests:

    curl localhost
    curl localhost/foobar

Finally, check the logs again:

    logcli query '{job="apache2"}'

The results should look something like this:

    Common labels: {filename="/var/log/apache2/access.log", job="apache2"}
    2020-03-06T15:02:28Z {code="404"} 127.0.0.1 - - [06/Mar/2020:15:02:28 +0000] "GET /foobar HTTP/1.1" 404 432 "-" "curl/7.47.0"
    2020-03-06T15:02:27Z {code="200"} 127.0.0.1 - - [06/Mar/2020:15:02:27 +0000] "GET / HTTP/1.1" 200 1856 "-" "curl/7.47.0"

Now you can quickly search for logs which have an error code starting 4 or 5:

    logcli query '{job="apache2",code=~"[45].+"}'

## Linux syslog aggregation

If you still have spare time, remember that you also have other Linux hosts (host1-6) in your campus. Can you configure them to log centrally as well?

For syslog there are two main options: you could get the host’s rsyslog server to send to rsyslog on srv1 (TCP or UDP port 514), or you could get it to write syslog messages directly to loki (TCP port 5140).

You could also run promtail on the remote hosts, which would be useful if you also needed to pick up application logs from those hosts.

How to configure any of these options is left as an exercise.

# Further reading

- Grafana loki datasource
- Loki docs
- LogQL
- GrafanaCONline video on Loki

To control how long Loki keeps logs for, see Loki Storage Retention. Old logs can be automatically deleted by setting `retention_deletes_enabled`, `retention_period` and `max_look_back_period`.

For storing large volumes of logs, Loki can be configured to use remote storage, including the ability to store log chunks into S3-compatible storage such as garage or seaweedfs.

# Reference: installation

This section is for reference only - it documents how loki and promtail are installed.

All the loki components are available as pre-built binaries.

Fetch and unpack the latest release from the releases page:

    wget https://github.com/grafana/loki/releases/download/vX.Y.Z/loki-linux-amd64.zip
    wget https://github.com/grafana/loki/releases/download/vX.Y.Z/logcli-linux-amd64.zip
    wget https://github.com/grafana/loki/releases/download/vX.Y.Z/promtail-linux-amd64.zip

    mkdir /opt/loki
    unzip loki-linux-amd64.zip -d /opt/loki
    unzip logcli-linux-amd64.zip -d /opt/loki
    unzip promtail-linux-amd64.zip -d /opt/loki

    ln -s /opt/loki/logcli-linux-amd64 /usr/local/bin/logcli

Create a data directory for loki to use:

    mkdir /etc/loki
    mkdir /var/lib/loki
    chown syslog:syslog /var/lib/loki

Use a text editor to create a systemd unit file `/etc/systemd/system/loki.service` with the following contents:

    [Unit]
    Description=Loki logging server
    Documentation=https://grafana.com/oss/loki/
    After=network-online.target
    # Fix shutdown delays: if promtail is running on the same host,
    # loki should start first, and shutdown after.
    Before=promtail.service

    [Service]
    User=syslog
    Group=syslog
    Restart=on-failure
    RestartSec=5
    WorkingDirectory=/var/lib/loki
    EnvironmentFile=/etc/default/loki
    ExecStart=/opt/loki/loki-linux-amd64 $OPTIONS

    [Install]
    WantedBy=multi-user.target

Also create an options file `/etc/default/loki` with the following contents:

    OPTIONS='-config.file=/etc/loki/loki.yaml -log.level=warn'

(`-log.level=warn` is to suppress excessive log messages from loki’s storage backend)

Create the initial `/etc/loki/loki.yaml` with the following contents:

    auth_enabled: false

    server:
      http_listen_port: 3100
      grpc_listen_port: 9096

    common:
      path_prefix: /var/lib/loki
      storage:
        filesystem:
          chunks_directory: /var/lib/loki/chunks
          rules_directory: /var/lib/loki/rules
      replication_factor: 1
      ring:
        instance_addr: 127.0.0.1
        kvstore:
          store: inmemory

    schema_config:
      configs:
        - from: 2020-10-24
          store: boltdb-shipper
          object_store: filesystem
          schema: v11
          index:
            prefix: index_
            period: 24h

    ruler:
      alertmanager_url: http://localhost:9093/alertmanager

    compactor:
      retention_enabled: true

    limits_config:
      retention_period: 91d
      enforce_metric_name: false
      reject_old_samples: true
      reject_old_samples_max_age: 168h

    # By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
    # analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
    #
    # Statistics help us better understand how Loki is used, and they show us performance
    # levels for most users. This helps us prioritize features and documentation.
    # For more information on what's sent, look at
    # https://github.com/grafana/loki/blob/main/pkg/usagestats/stats.go
    # Refer to the buildReport method to see what goes into a report.
    #
    # If you would like to disable reporting, uncomment the following lines:
    analytics:
      reporting_enabled: false

Note that this configuration is insecure. In practice, you should front loki with an authentication proxy, or use iptables to limit access.

Create `/etc/systemd/system/promtail.service`:

    [Unit]
    Description=Promtail
    Documentation=https://github.com/grafana/loki/tree/master/docs/clients/promtail
    After=network-online.target

    [Service]
    User=syslog
    Group=syslog
    Restart=on-failure
    RestartSec=5
    WorkingDirectory=/var/log
    EnvironmentFile=/etc/default/promtail
    ExecStart=/opt/loki/promtail-linux-amd64 $OPTIONS

    [Install]
    WantedBy=multi-user.target

Create `/etc/default/promtail`:

    OPTIONS='-config.file=/etc/loki/promtail.yaml'

Create `/etc/loki/promtail.yaml`:

    # Promtail can expose prometheus metrics
    server:
      http_listen_address: 127.0.0.1
      http_listen_port: 9080
      grpc_listen_port: 0

    # How to find the loki server(s)
    clients:
      - url: http://127.0.0.1:3100/loki/api/v1/push

    # Logs to read and forward
    scrape_configs:
      - job_name: syslog
        syslog:
          listen_address: 127.0.0.1:5140
        relabel_configs:
          - source_labels: [__syslog_message_severity]
            target_label: severity
          - source_labels: [__syslog_message_facility]
            target_label: facility
          - source_labels: [__syslog_message_hostname]
            target_label: host
          - source_labels: [__syslog_message_app_name]
            target_label: app

Finally, tell systemd to read our new service files:

    systemctl daemon-reload

Grafana is available pre-packaged in many formats, including deb packages for Ubuntu:

    curl -fsSL https://apt.grafana.com/gpg.key >/etc/apt/trusted.gpg.d/grafana.asc
    echo "deb https://apt.grafana.com stable main" >/etc/apt/sources.list.d/grafana.list
    apt-get update
    apt-get install grafana