I’ve got your exact setup:

• running immich (and other services with valuable data) at home (on OMV!)
• backup server at family member’s house
• regular backups from here to there through Wireguard tunnel with Borgbakup

AMA!

I don’t want a compromised device on the family member’s network to compromise mine

If with “mine” you mean the pi Zero, I wouldn’t worry too much about that. Just configure the pi’s firewall to drop all traffic from the LAN (they have no business with this device anyway), except any incoming traffic from the router on the Wireguard port.

Immich is entirely free and open-source software and got a grant from the FUTO Foundation, which is fiercely pro privacy. As such, it does not engage in any data sharing. Development is financed through the grant and purchases of the server licence (I got one; it’s ridiculously good software).

Ah! I use virt-manager + qemu on desktop Linux, mainly to try things and and contain dodgy software (Windows). I didn’t know it was commonly used on headless servers. Thanks for pointing that out!

I’m self-hosting immich (with ML-based facial recognition, semantic search and OCR) and psiTTSa, an offline text-to-speech engine.

(Prob not the answer you were looking for, but a reminder that “AI” is more than just LLMs.)

Thanks for your detailed reply and expertise! I really appreciate it!

First question you have to ask yourself is so what?

You’re right to call out my lack of threat modeling. I am aware that the “cost” of what I’m planning here likely outweighs the actual risks. That’s a deliberate choice though: one of my objectives in homelabbing is to “learn how the big guys do it”, i.e. learning and implementing, bit by bit, networking and security best practices.

That said,

• all of my drives are (LUKS) encrypted (although this will only help against evil maid attacks physically stealing my disks, since data is only encrypted at rest, not while the server is in operation)
• I do have some CPU monitoring in place
• I have applied the principle of least privilege (as far as I can see at least)
• I have set up an off-site backup server running (although I need to improve automation and deduplication; I’ll probably go with Borg here)

Thank you also for pointing me towards OWASP Docker cheat sheet! I didn’t know this resource yet, although I seem to have found others that said the same thing. I already have in place:

• no docker socket exposed to services
• user set for every service that didn’t give me a headache doing that (some assume root and won’t accept setting a user)
• username remapping is active
• all containers have their caps dropped and selectively only put those back in which they need
• I expose almost no container ports, and only those necessary; instead, communication with the containers is handled on my reverse proxy’s network (i.e. via the internal docker network)
• privilege escalation is disabled

I have played around with rootless docker (again, stubborn containers assuming root prevented me from going all-in) and considered Podman, although just putting containers in a VM looks like a more frictionless route to me.

All things considered, your advice seems to be:

1. to first consider whether I actually need all this security. I really don’t, yet I want to learn how to set it up anyway.
2. to first explore and all other hardening avenues. I have, to a certain degree, and beyond that run into brick walls.

If I might exhaust a little more of your time, I’d be curious how you judge the next step I’ve suggested: splitting duties between separate servers, thus segmenting public from private physically, or going the virtual route (again, by splitting into a public and a private VM and separate them through strict software firewalling).

I still use separate user accounts for different purposes with heavily restrictive permissions.

Good call, that’s an even more effective first line of defense! (I have, however, already implemented it. ;)

Thanks for your input. It is correct that a lot of ISPs/consumer grade hardware is poorly maintained. Mine is a rare exception, however: I’m running an AVM FritzBox, which are somewhat famous for still receiving updates even after 10+ years on the market. They have only recently pushed a security update my way, so I’m good enough on that front - for now.

I shall consider a firewall in the future though.

I hope part of your hardening SSH is changing the port number!

I used to do that, but then realised that doing so will do very little. Scanning all possible ports is a matter of seconds, and SSH (unlike Wireguard) will signal to a potential attacker that it’s listening on port X instead of just remaining silent. In other words: in my eyes, it’s security by obscurity and will have very little effect.

What I have done to harden SSH is:

• set up pubkey authentication and disable password authentication
• disable root login
• not expose my SSH port publicly; it is accessible only from my LAN and through Wireguard

I have also tried out IP whitelisting (which may harden my setup against attacks from the LAN), but found it may be overkill for now. I might turn it back on just for shits and giggles.

Before doing anything else , consider if you actually need to expose anything to the internet at all. Running a VPN would let you reduce your exposed service to 1 port ( wireguard is UDP based so it’s harder to test if the port is open too ). A VPN like NetBird / tailscale / Cloudflare warp don’t need you to open any ports at all

Good advice! Here’s my setup in more detail:

• a good 90% of my services are “private”, i.e. they’re only accessible through Wireguard or from the LAN. I use local DNS + a reverse proxy to get “virtual” URLs + proper SSL termination within the LAN. Also, all administrative tools (e.g. SSH) are private (Wireguard or LAN only); none of them are publicly available.
• some of my services (e.g. Nextcloud, Wireguard, RustDesk) are public so as to allow me to use collaboration and file sharing tools with others. For this I do expose some ports publicly.

As you can see, I have mostly realised what you said. However, I don’t see myself setting up VPN connections for each and every user of my services and train them how to use them; link shares via Nextcloud or tech support via RustDesk need to be frictionless, so taking these services private isn’t an option.

Yet if one of these public services were to be compromised and root access gained, I do see the risk of a (motivated) attacker to attempt lateral movement to get at the more sensitive services that I’m keeping local for a reason.

This sounds a lot like my second option, except yours is a little more custom-made. What do you use to set up and run your VMs?

I wouldn’t stress to much. Your risk level is probably very low.

I’m not, no worries! :) You are correct to say that what I’m trying to do is likely above my threat model. Yet, I’m also in this for the learning experience, which includes picking up best practices and networking/security concepts that go a little bit above what I actually need.

Also…

Moving laterally requires a human (or agent these days)

you’re bringing up the emerging threat of agentic attacks that allows for automating lateral movement. While infiltrating some rando’s network probably doesn’t warrant burning through several thousands of expensive tokens now, odds are this cost will decrease over time, thus increasing the likelihood of opportunistic automated attacks. If and when that happens, I’d like to be prepared.

I’ve done my research on that, and apparently LXCs share the host kernel, meaning: if the LXC is compromised, so is the host system. So if I go with Proxmox, I’ll probably swallow the (somewhat) larger overhead of running a full VM.