This post continues from Part 7: Proxies and Caching
As this series comes to an end there is one last thing I wanted to cover: monitoring. This is another one of those not-necessary-but-nice-to-have features. If you don't really care about pretty graphs of network traffic then just stop here and call it a day. Personally though I like being able to see and track my network's performance. We rely on our home internet connections so heavily why not make sure it's running like a well oiled machine?
To do the monitoring we'll need a few different pieces of software, each of which fulfils one role of a complete monitoring system. We'll be using Prometheus for the database, Prometheus' node exporter for metrics collection, and Grafana for dashboarding and alerting.
Prometheus is what's known as a time series database (TSDB). It pulls stats down from different places at regular intervals so the metric values can be looked at over time. The metrics it pulls are exposed by different types of "exporters". An exporter is just a piece of software that collects data from all kinds of different sources and aggregates it into a format Prometheus can understand. In this case we'll be using the node exporter to export statistics like CPU usage, memory usage, and of course networking stats. There are plenty of other exporters available though for things like DNS, Apache, and Nginx. You can even instrument your own code and make it into an exporter. Once all of the data is in Prometheus we'll need a way to look at it. This is where Grafana comes in. It has native support for querying Prometheus data sources and can transform that data into pretty graphs. The graphs can be arranged into neat dashboards and it can even alert you when something goes haywire.
So let's just get to it. Before starting you'll need to get Docker installed on your router. While all of the applications we'll be using can be run on bare metal, setup and maintenance is far easier using Docker containers. If you need some help check out the installation docs from Docker or just Google around a bit.
Once you've got Docker up and running the first step will be to get a Prometheus container running. There's two things you'll need to get ready before running the container: a place to store data and a configuration file.
Make a directory for data storage in your home directory and within it add a directory for prometheus:
$> mkdir ~/data $> mkdir ~/data/prometheus
Next create a basic configuration file for Prometheus like the one below called
prometheus.yml. I've included comments inline to explain what it is doing:
# prometheus.yml global: # By default, scrape targets every 15 seconds. scrape_interval: 15s external_labels: # This label will be added to all stats scraped by this instance of Prometheus monitor: 'pine-monitor' # A configuration for where to pull stats from scrape_configs: # Name for stats pulled by this configuration - job_name: 'prometheus' static_configs: # Pull stats from router.home:9090. This is where # Prometheus will be running. By default it exports some # stats about itself. - targets: ['router.home:9090']
Now that you have the prereqs met, you can actually run the container. If you're using an ARM based system like I am make sure to use the
braingamer/prometheus-arm Docker image. If you're running on a regular Intel or AMD architecture though just use the official
docker run -d \ -p 9090:9090 \ -v ~/data/prometheus:/prometheus \ -v ~/data/prometheus.yml:/etc/prometheus/prometheus.yml \ --name prometheus \ braingamer/prometheus-arm
This creates the container, maps port 9090 on the host machine to port 9090 (Prometheus) in the container, maps the data directory on the host to the data directory in the container, and copies the config to the container. Try navigating to
http://router.home:9090/ in your browser. If all is well you should see Prometheus' console. You can navigate to the settings and look at the targets it is capturing stats from. You should see the one target you configured and it should be up. Once it's all working the last step is to create a service for it so it will run on boot.
Stop the container so that the service can be used to start it:
$> docker stop prometheus
Add a standard Docker container service to your
/lib/systemd/system/ directory. For example:
#/etc/systemd/system/prometheus.service [Unit] Description=Prometheus container Requires=docker.service After=docker.service [Service] Restart=always ExecStart=/usr/bin/docker start -a prometheus ExecStop=/usr/bin/docker stop prometheus [Install] WantedBy=default.target
And finally start and enable the service:
$> sudo systemctl start prometheus $> sudo systemctl enable prometheus
Alrighty, step one done. Let's move on to the node exporter that will let us collect things like CPU, memory, and network statistics.
Running Node Exporter
Node exporter is the one piece of this puzzle that will run on bare metal instead of as a Docker container. It is possible to run it in a container, but it needs access to so many parts of the host system for proper metrics gathering that it isn't really worth it.
It is written in, and so obviously depends on, Go. You'll have to install it first with:
$> sudo apt-get install golang
You'll also need to setup a
GOPATH for it. This is where it will download packages to. Make a directory and set it as the
$> mkdir -p ~/src/go $> export GOPATH=~/src/go
You can then use
go get to download the node exporter package from Github:
$> go get github.com/prometheus/node_exporter
Go into the newly downloaded package and build it:
$> cd ~/src/go/github.com/prometheus/node_exporter $> make
That should build you a
node_exporter binary in the same directory. Next you'll need a
systemd service to actually run that binary. Add one to the same
/lib/systemd/system/ path called
#/lib/systemd/system/node-exporter.service [Unit] Description=Node Exporter [Service] ExecStart=/home/user/src/go/github.com/prometheus/node_exporter/node_exporter [Install] WantedBy=multi-user.target
Fire it up and make sure it is enabled on boot:
$> sudo systemctl start node-exporter $> sudo systemctl enable node-exporter
To make sure it's running try navigating to
http://router.home:9100. You should get served a page with a bunch of metrics and their values. This is what we need Prometheus to start scraping. Once you've got the page loading properly add another static config to your
... scrape_configs: - job_name: 'prometheus' static_configs: - targets: ['router.home:9090'] - job_name: 'router' static_configs: - targets: ['router.home:9100']
Restart Prometheus to pick up the new config:
$> docker restart prometheus
Now go check out your targets page in the Prometheus console again. You should see a new target for
router.home:9100 and it should be up. If it is then Prometheus is successfully pulling stats from the exporter. Woot woot. Now let's make some pretty graphs.
The steps for getting Grafana going are fairly similar to the steps for Prometheus: make directories for the data, write a config, and run the container. So to begin let's make the data directories. Grafana needs two directories, one for data and one for the configs:
$> mkdir -p ~/data/grafana/data $> mkdir ~/data/grafana/config
For the configuration the default values will be fine for most things. You will want to configure the server name though so the links point to the right places. I've setup my DNS with a domain called
monitoring.home that I will use to point to the instance, but as long as it points to the right place it doesn't matter if you use a domain or just the IP address. Create a config called
grafana.ini in the
~/data/grafana/config directory with these lines:
; grafana.ini instance_name = monitoring.home [server] domain = monitoring.home root_url = http://monitoring.home:3000
The config is pretty self explanatory. Just change the values to match your setup. If you want to set up email based alerting then add another section with your email server settings:
[smtp] enabled = true host = <your email server> user = <your email username> password = <your email password> from_address = [email protected]
OK, it should be ready to go. Fire up a container like so:
docker run -d \ -p 3000:3000 \ --name=grafana \ -v ~/data/grafana/data:/var/lib/grafana \ -v ~/data/grafana/config:/etc/grafana \ braingamer/grafana-arm
Again, if you aren't using an ARM based system then don't use the ARM based Docker image. Navigate to
http://monitoring.home:3000 (or whatever you called yours) to make sure it is working. You should see a Grafana login page where you can use the default user
admin and default password
admin to get in. If you can login successfully then move on to creating a
systemd service for it.
Stop the container so
systemd can take over control:
$> docker stop grafana
And create another standard Docker
systemd service as
# /etc/systemd/system/grafana.service [Unit] Description=Grafana container Requires=docker.service After=docker.service [Service] Restart=always ExecStart=/usr/bin/docker start -a grafana ExecStop=/usr/bin/docker stop grafana [Install] WantedBy=default.target
Start the service and enable it at boot:
$> sudo systemctl start grafana $> sudo systemctl enable grafana
That should take care of keeping Grafana going. You'll still need to set it up though. Login from your browser again with the default user and password. Follow the on boarding steps to set up Prometheus as a data source and for the love of god change the default password. From there you can create a dashboard out of your Prometheus metrics, or just do what I did and steal one from someone else. There are a bunch of dashboard templates available at https://grafana.com/dashboards and you can filter by Prometheus/node exporter support. The one I selected though was called Node Exporter Server Metrics. Download the template JSON and import it into Grafana as a new dashboard. If all goes well it should find all the metrics and give you a pretty dashboard like this:
Ain't it a beaut. And with that your complete monitoring solution should be good to go. There are a bunch more things you can do with it if you have the patience. I've put mine behind an Nginx proxy so you don't always have to type in the port number in your browser. It also adds TLS and basic auth so I'm the only one that can get into it. You can always add more exporters to Prometheus and more dashboards and alerts to Grafana too. I'll leave that up to you though if it is something you care about.
And with that the router build is finished!