Homelab Alerts And What's Actually Worth Logging at Home
I run a very large homelab, with many servers and services. Some are public facing, which means that attackers are able to exploit any unpatched vulnerabilities which may linger. It's important to be ready for that eventuality, and to be able to identify when it happens. By setting up robust monitoring and logging, you're able to have better oversight of your systems. I've had to learn this lesson the hard way, as I've had many systems probed and compromised over the years. I'm here to tell you that you should setup robust monitoring, logging, and alerting so that you have the data you need to mitigate the risks involved with running a homelab.
While you may think it's ideal to setup a robust logging solution and capture everything, consider if that's effective for your setup. Capturing all data in say, a SIEM solution, seems like a nice idea until you're sitting on a huge pile of data taking up valuable disk space. While I do log data, I keep things pretty simple regarding logging. The more important thing for me is strong monitoring and alerting.
So what do I monitor? Many things, starting with system uptime. I use Uptime Kuma to monitor the availability of many of my services. It's highly configurable to query for availability, and it alerts me if there are any outages. Sometimes, attackers interrupt your services, and being alerted when this happens is ideal. Uptime Kuma can be configured to send notifications in many ways, and we will go more into notifications a little later.
First, let's talk about resource monitoring. In addition to availability monitoring via Uptime Kuma, I monitor my systems resources with the help of Beszel, a robust resource monitoring and logging solution. I run Beszel agents on all of my assets, which logs process usage, CPU, GPU, Memory, Disk, Network etc. This data is extremely valuable, since you can do trend analysis and behavior analysis based on each servers unique parameters. I can't tell you how many times I've caught malware simply by being alerted that my CPU usage or CPU temperature exceeded normal thresholds. A lot of malware comes in the form of cryptominers, which often ramp system resources. This can be detected if you're effectively monitoring for high utilization. Keeping an eye on your system metrics can be an effective way to detect abnormalities.
One thing I like to do is run system health checks via cron jobs and scripts. I have these checking key things on my systems to ensure healthy parameters. If my ZFS pools have issues, if my docker swarm is bugged, if watchtower detects out of date containers- many different things can trigger an alert which informs me of the anomaly. You aren't going to have the time to check the health of your systems daily, so why not outsource that work to automation?
Logging is also important, and to be honest, it's the one thing I believe I need to improve within my homelab. Currently, I don't use any centralized logging solution. All the machines capture their own logs and data, and I leave it to myself and my AI agents to parse. For a homelab, it's up to you to determine what's necessary here. For me, the default logging capabilities of the apps and services I host have been sufficient for diagnosing and evaluating what is going on with the systems. If a system has malware or is being probed or attacked, it's vital to know all you can about that activity. In my case, NGINX access logs and system logs have been extremely valuable. I may have to level up my logging game going forward, as I can't deny how important they are.
If you've setup proper monitoring, logging, and alerting, let's talk about notifications. You need to actually be alerted if there's something worth addressing, so it's critical to setup ways for you to receive these messages. I like to use a few techniques.
- Email Notifications
- Push Notifications
- Admin Dashboard
Let's go over each briefly.
Emails are useful, especially if you want detailed health reports or more in-depth logs. I have several emails that I review each day, detailing all sorts of information about my homelab. My health check scripts have pass / fail conditions for many things like the state of my ZFS storage pool, resource availability, docker container image update availability, etc. It's an effective way to capture an overall picture of your setup without doing all the checks yourself. It helps me make sure my system is operating within normal parameters, and saves me a ton of time.
Push Notifications. This is where I can receive immediate notifications. I use Gotify for this, which paired with Gotify clients, alerts me via toast notifications on the computers I use, or via notifications on my phone. An audible alert can be heard, and this is especially useful for Uptime Kuma or Beszel alerts. I'm made aware of any outages or discrepancies with my resource monitors immediately.
My Admin Homepage or Admin Dashboard is another tool. I created a dashboard which pulls data from Uptime Kuma, Beszel, and other sources and displays them in real time in one place. On this dashboard, I have links to all my admin pages and services. This way, I can get at a glance the health status of my systems as I'm navigating the different admin pages and functions in my homelab.
If you're a one person homelab, self-hoster, or tinkerer, it's important to work smart. Taking the time to build out strong monitoring, logging, and alerting can save you countless hours of time later. It's what makes self-hosting possible- smart automations that make it so you don't have to be everywhere at once. Tools built to keep an eye on things while you're not looking are a critical piece of the homelab, and take it from me- it's not worth waiting till you're attacked before doing so. It pays off quickly whenever something happens. While you may not need a SIEM, strong signals you trust, dashboards, and alerts can make all the difference.