Adding log monitoring to our setup

So far in this series we’ve covered an overview of the stack we’re trying to create, looked at DNS filtering with PiHole, gone over the theory of how this all works, and setup the majority of the DNS filtering and monitoring stack.

This post (the final post on this subject for now!) is going to add in the log collection from our network devices so we can get a better view of what’s happening across the network.

Reviewing the Container Management Stack

As a refresher, let’s look at the logging part of the over-all platform.

Server01 is the physical server running Arch Linux. You can do this with a number of Raspberry Pi’s instead of a single server, but that adds some more complexity and reduces the amount of resources available to us, so we’ll focus on a dedicated server here.

Server01 has a number of services running on it:

  • FluentBit is used for log processing. This runs on the physical server and processes log events before sending them on to the logging storage engine. The logging storage engine runs in a container, so we’ll cover that in a bit.
  • Loki runs in a container and is used for logging storage and querying
  • Grafana also runs in a container and provides a user-interface to tie our metrics from Prometheus and our logs from Loki into a single location

How do I get this working?

Getting all this working is really simple.

First of all, install fluent-bit then download the configuration files from the git repository and move them into the appropriate location.

For fluent-bit, you’ll need the following files:

Place all these files in /etc/fluent-bit (or the configuration directory for your installation of fluent-bit depending on the version of linux you’re running).

Finally, check that you have a file called parsers.conf in that same directory, and that it has a ruleset called iptables. If either of these do not exist, download the latest from the fluend-bit git repository and place that into the configuration directory as well.

Now restart fluend-bit and check that it is listening on the correct ports:

  • 1514/UDP

You can check this by running ss -ntplau | grep -E "1514|8888", and you should see the following output:

# ss -ntplau | grep 1514
udp   UNCONN    0      0                   0.0.0.0:1514               0.0.0.0:*     users:(("fluent-bit",pid=507409,fd=19))                  

If you don’t get this output, check the logs using journalctl -fu fluent-bit and look for errors.

Once you’ve confirmed that the service is up and running, access the configuration settings of your Unifi Controller (or other network router) and configure the remote SYSLOG server to point to your server IP address and port 1514 (the green box in the image below)

If you’ve followed the whole series and are running my nomad configurations then Loki should already be accepting logs via fFluent-bit, so now we need to log into Grafana and configure that.

As before fill in the Loki address, and if you’re following the entire series then you can use https://loki.service.my.domain/ here.

Save and test the settings, ensuring that everything is working, then go into the “explorer” mode of grafana using the compass icon on the left-hand menu bar.

You should see a job called “fluentbit”, and when you run the query all your logs should appear:

If you’ve got this far, congratulations, there’s just one more step to do!

Creating cool dashboards

You should now have all your metrics and logging data in one place, enabling you to build all kinds of dashboards.

To get you started, I’ve shared my home dashboard on the Grafana.com website, just import that into Grafana, select your data sources as appropriate, and off you go!

Summary

This marks the end of our journey through setting up logging and monitoring for our smart-home, from now on I’ll be focusing on the devices that I’m fitting to the house and how to get them to work with the occupants to enhance the way they act with their surroundings.

If you find any errors or issues, please log them on the Github Issue Tracker and I’ll get to them as soon as I can!

Share Comments
comments powered by Disqus