Loki and Promtail

LOKI

Install Loki

cd /usr/local/bin

sudo curl -O -L “https://github.com/grafana/loki/releases/download/v2.0.0/loki-linux-amd64.zip”

unzip loki-linux-amd64.zip

And allow the execute permission on the Loki binary 

sudo chmod a+x loki 

 
sudo nano config-loki.yml 

SAMPLE config files are here

wget https://raw.githubusercontent.com/grafana/loki/master/cmd/loki/loki-local-config.yaml

wget https://raw.githubusercontent.com/grafana/loki/master/cmd/promtail/promtail-local-config.yaml

Start Loki

./loki-linux-amd64 -config.file=loki-local-config.yaml

Loki as a service

sudo nano /etc/systemd/system/loki.service

[Unit] 

Description=Loki service 

After=network.target 

[Service] 

Type=simple 

User=loki 

ExecStart=/usr/local/bin/loki -config.file /usr/local/bin/config-loki.yml 

[Install] 

WantedBy=multi-user.target 

$sudo service loki start

$sudo service loki status

$sudo curl localhost:3100/ready , check if loki is ready

PROMTAIL

cd /usr/local/bin 

sudo  curl -O -L "https://github.com/grafana/loki/releases/download/v2.0.0/promtail-linux-amd64.zip"                         

sudo unzip promtail-linux-amd64.zip 

allow the execute permission on the Promtail binary

$sudo chmod a+x promtail

Promtail config

$sudo nano config-promtail.yml

Promtail as a service

Now we will configure Promtail as a service so that we can keep it running in the background.

Create a file called promtail.service

$sudo nano /etc/systemd/system/promtail.service

And add this script, if u are using vagrant replace promtail user with vagrant

[Unit] 

Description=Promtail service 

After=network.target 

[Service] 

Type=simple 

User=promtail 

ExecStart=/usr/local/bin/promtail -config.file /usr/local/bin/config-promtail.yml 

[Install] 

WantedBy=multi-user.target 

sudo service promtail start

sudo service promtail status

usermod -a -G systemd-journal promtail

If you ever need to stop the new Promtail service, then type

sudo service promtail stop

sudo service promtail status

Grafana integration with Loki

Login to grafana URL : http://127.0.0.1:3000

Create a new Data Source in the Grafana User Interface, and select

Name : Loki URL : http://127.0.0.1:3100

Note : If you installed Loki on a different server than your local Grafana

server, then the address will be different. eg,

http://your-loki-server-ip-address:3100

Leave everything else default.

Then, visit the Explore menu option on the left.

Select Loki as the Data Source

And in the Log labels text area, try these LogQL examples one by one

LOGQL

{job=“mysql”} |= “error”

{name=“kafka”} |~ "tsdb-ops.*io:2003"

{name=“cassandra”} |~ error=\w+

{instance=~“kafka-[23]”,name=“kafka”} != “kafka.server:type=ReplicaManager”

Get the top 10 applications by the highest log throughput:

topk(10,sum(rate({region="us-east1"}[5m])) by (name))

Get the count of logs for the last five minutes, grouping by level:

sum(count_over_time({job="mysql"}[5m])) by (level)

Get the rate of HTTP GET of /home requests from NGINX logs by region:

avg(rate(({job="nginx"} |= "GET" | json | path="/home")[10s])) by (region)

This example counts all the log lines within the last five minutes for the MySQL job.

sum by (host) (rate({job="mysql"} |= "error" != "timeout" | json | duration > 10s [1m]))

This will add a dummy test message to log

echo 'abc123 this is a fake error def678' | systemd-cat