The way of using/deploying software, especially on servers has changed over the years. In days gone by things were installed directly on the operating system, this is referred to as a bare metal install. This type often leads to various types of conflicts and nightmares as things change, trying to find what caused the resulting breakage. The industry moved to virtual machines, these were full on stand alone OSs making them bloated requiring more storage and CPU power to use in exchange for fewer conflicts and easier maintenance. To address the bloat containers came about. These are essentially trimmed down virtual machines. One of the other advantages is that it doesn’t matter the OS (Mac, Windows, Linux), what does matter is architecture x86, Arm 64, the Pi5s are Arm64.
To get that stability, ease of deployment and maintenance there’s a small trade off, isolation. Depending on how things are created containers can’t talk to each other or to the underlying OS. In general this isolation is a good thing, but there are circumstances when it can get in the way, monitoring and logging fall into the later, but I’m getting ahead of myself.
The Pi shop is using Docker containers for all but three services. The first two bare metal installs are smb (samba) for making file shares on the local network so i can copy files from any of my devices or computers on the lan; and, Grafana alloy, so the OS & host level logs can be sent to a Docker container for evaluation and display. Docker is the third bare metal install that creates the foundation for all the containers to run in/on.
Docker on Linux is a CLI (command line interface) UI, i.e. terminal window; not the most friendly. I use Portainer, a web based UI to administer my containers and stacks. A stack is a collection of containers, usually put together so they can work together (that isolation thing).
With that foundation it is fairly easy to add, delete, modify and try different containers and prearranged stacks to get the services(software) I wanted for an always on, available anywhere, system. Since this whole project started as a way to display photos my first interest was photo viewers. There are several to choose from. I currently use 3. I point this out to emphasis that you can tailor your system to your wants or needs.
The three photo display programs in use are Pi Gallery2 as a memory lane with family photos going back over 100 years. Immich that has replaced Google Photos and catches the images and videos from my phone and tablet. The third is Modula gallery providing the gallery viewer on this website.
Besides the photo display capabilities, as long as I had a domain name I might as well use it, so WordPress was a natural for making this website. WordPress is not the only CMS player; I’m hearing interesting things about Statamic and may look into it to replace WP, if I were starting from scratch I’d probably start with Statamic.
I like to take notes for thoughts, articles, quotes, etc. I tried Nextcloud, which works fine, but was overkill for my needs and had features I wouldn’t use. I’ve settled on Joplin Server on a Pi and the Joplin app on all devices and OSs to have my notes available wherever and from whatever device I’m using. I’m typing this now from my PC and will review it later on my tablet.
A good media server is also a first install for most home systems. I chose Jellyfin as my primary. Cloudflare’s terms of service forbid video streaming so I tend to minimize watching videos on the road through Cloudflare, this is where Tailscale and tailnet shine. While streaming video thru the tunnel is verboten, linking to YouTube for video & embedding them is okay, expect to see more of that here.
For just listening to tunes, Jellyfin is a bit heavy so I also use Navidrome as my tunes machine. Besides my own music I can add streams from radio stations and other services.
Ever had the situation you needed a piece of data that was buried on a piece of paper in a file cabinet or in a file you couldn’t get to? That’s the hole Paperless-ngx fills. Scanned docs, text files, spread sheets, docs, pdf, etc. drop them on the input folder and shortly there after they have been scanned OCR’d, indexed and ready to be searched and retrieved. Contract number, last payment date, account number all easily found just when you need it wherever you are.
Home Assistant is a late addition to the Pi shop and provides home automation and home security features to bring your house into the connected, always on world. Convenience at your fingertips, but, convenience comes with a cost; another learning experience and way to spend money invest in the house.
Those are the ‘above the water line’ services. Down below there are several other services to keep the Pis fresh, updated and working smoothly. That takes monitoring the metrics and the ability to see the logs. Each Pi has a common monitoring and logging stack and a few other solo containers to help with the admin and maintenance.
Trying to get metrics (cpu utilization, cpu temp, drive space, etc) or logs runs into the boundary layer between the bare metal and the container. Dealing with that adds a wrinkle, being on a Pi adds another wrinkle or two; and a recent change in how memory groups and the Pi kernel are exposed to the rest of the world changed. At the time most of this was unknown, add to that an AI that is a task driven problem solver that can modify a configuration to get out of any situation; it thinks. I spent 3 weeks chasing a unicorn at the end of the rainbow. I still have not arrived but am very close. Metrics work as desired, host OS logs are working and the Docker/container logs need more tweaking; but the right combination of tools appear to be in place and configured correctly. Putting alloy on bare metal rather than a container appears to be one of the last pieces.
Homepage provides a home page dashboard for all the services in the Pi shop. I added Glances to provide some at a glance widgets for the status of each Pi. Watchtower is a container whose sole job is to keep other containers up to date, along with some cron jobs (chronology/timed), helps keep the Pi and it’s software up to date automatically, these too run on each Pi
To get a deep dive look at the internals of the Pis took a complex stack of containers to collect the metrics and logs from the host Pi & the containers running on it.
The scrapers are smartcrl exporter for metrics on the NVMe drive, temps, space, usage, etc. Node exporter gets the metrics from the host Pi. cAdvisor provides metrics from all the running containers. with Alloy, mentioned above handling the logs the pulse of the system can be monitored. Each of the exporting services deliver their goods to the next layer for organization and storage. Loki is the software for the logs and Prometheus is the software for the metrics; they feed their output to Grafana for graphing and display to make pretty graphs and dashboards. If you look closely you will see it used all over the web.
The magic tunnels of privacy and security are provided by a cloudflare container on one machine along with a proxy manager container. The cloudflare container provides the endpoint for the tunnel. Adding nginx, the proxy manager allowed me to remove any knowledge (local ips and ports) of my system from Cloudflare. Now instead of Cloudflare doing that final routing the proxy manger, under my control determines what is accepted and where to send it, everything else gets ignored. Thing of it this way Cloudflare provides me a high end venue, the proxy manager is the muscled bouncer at the door, only the finest visitors that behave themselves are allowed. Welcome.
Tailscale it the final piece to complete the puzzle. each computer & device we own has tailscale installed. With the key to unite them all you form your own tailnet, a secure encrypted tunnel, point to point between any two of your devices/computers. This is the avenue used for admin and maintenance on the servers when on the road, or to watch video when outside the lan. I think of it as the service entrance where the maintenance engineer has keys to everything.