Many of the posts I read here are about Docker. Is anybody using Kubernetes to manage their self hosted stuff? For those who’ve tried it and went back to Docker, why?
I’m doing my 3rd rebuild of a K8s cluster after learning things that I’ve done wrong and wanted to start fresh, but when enhancing my Docker setup and deciding between K8s and Docker Swarm, I decided on K8s for the learning opportunities and how it could help me at work.
What’s your story?
You must log in or register to comment.
Kubernetes is useful if you have gone full cattle over pets. And that is very uncommon in home setups. If you only own one or two small machines you cannot destroy infra easily in a “cattle” way, and the bloatware that comes with Kubernetes doesn’t help you neither.
In homelabs and home servers the pros of Kubernetes are not very useful: high availability, auto-scaling, gitops integrations, etc: Why would you need autoscaling and HA for a SFTP used only by you? Instead you write a docker-compose.yml and call it a day.
The one exception to this is if you’re using your homelab to learn kubernetes.
That was the only time I used K8s and k3s on my homelab.
And for anything that I do want to set up in a HA/cattle kind of way, I use Docker Swarm, as it feels like a more comfortable extension of docker compose.
This mostly, I haven’t seen a compelling reason to leave my docker setup.
I run k3s and all my stuff runs in it no need to deal with docker anymore.
Kubernetes is awesome for self hosting, but tbh is superpower isn’t multi-node/scalability/clustering shenanigans, it’s that because every bit of configuration is just an object in the API, you can really easily version control everything - charts and config in git, tools like Helm make applying changes super easy, use Renovate to do automatic updates, use your CI tool of choice to deploy on commit, leverage your hobby into a DevOps role, profit
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
Fewer Letters More Letters DNS Domain Name Service/System Git Popular version control system, primarily for code HA Home Assistant automation software ~ High Availability HTTP Hypertext Transfer Protocol, the Web LXC Linux Containers NAS Network-Attached Storage SSD Solid State Drive mass storage SSH Secure Shell for remote terminal access VPN Virtual Private Network VPS Virtual Private Server (opposed to shared hosting) k8s Kubernetes container management package nginx Popular HTTP server
11 acronyms in this thread; the most compressed thread commented on today has 11 acronyms.
[Thread #82 for this sub, first seen 26th Aug 2023, 23:55] [FAQ] [Full list] [Contact] [Source code]
Seems a bit overkill for a personal use selfhosting set-up.
Personally, I don’t need anything that requires multiple replicas and loadbalencers.
Do people who have homelabs actually need them? Or is it just for learning?
I find mine useful as both a learning process and as a thing need. I don’t like using cloud services where possible so I can set things up to replace having to rely on those such as next loud for storage, plex and some *arr servers for media etc. And I think once you put the hardware and power costs vs what I’d pay for all the subs (particularly cloud storage costs) it comes out cheaper at least with hardware I’m using.
Yes, those are all great uses of it. But could all still be achieved with docker containers running on some machines at home, right?
Have you ever had a situation where features provided by kubernetes (like replicas, load balancers, etc) came in handy?
I’m not criticizing, I’m genuinely curious if there’s a use-case for kubernetes for personal self-hosting (besides learning).
A lot of people thought this was the case for VMs and docker as well, and now it seems to be the norm.
A lot of people thought this was the case for VMs and docker as well, and now it seems to be the norm.
Yes, but docker does provide features that are useful at the level of a hobbyist self-hosting a few services for personal use (e.g. reproducibility). I like using docker and ansible to set up my systems, as I can painlessly reproduce everything or migrate to a different VPS in a few minutes.
But kubernetes seems overkill. None of my services have enough traffic to justify replicas, I’m the only user.
Besides learning (which is a valid reason), I don’t see why one would bother setting it up at home. Unless there’s a very specific use-case I’m missing.
I was looking into converting my docker services into a cluster to get high availability and to learn it for work, but while investigating it, I read that kubernetes is actually meant for scalability and just a single service per cluster.
Also read that docker swarm is actually what is recommended for my homelab use case. So I’m right now on my way to convert everything to docker stacks. What do you think?
I’m not sure what you mean by that.
It provides high availability if you have multiple nodes and pods.
Also what do you mean by single service per cluster? Because that’s not the idea at all.
Of course high availability always requires multiple nodes.
Its just that while choosing how to set up my cluster I looked up several options (proxmox, swarm, kubernetes…) and I noticed that kubernetes is generally meant for bigger deployments.
I only need a single replica for each of my containers and they can all run on a single node, so kubernetes is overkill just to get high availability For my use case
I use k8s at work and have built a k8s cluster in my homelab… but I did not like it. I tore it down, and currently using podman, and don’t think I would go back to k8s (though I would definitely use docker as an alternative to podman and would probably even recommend it over podman for beginners even though I’ve settled on podman for myself).
- K8s itself is quite resource-consuming, especially on ram. My homelab is built on old/junk hardware from retired workstations. I don’t want the kubelet itself sucking up half my ram. Things like k3s help with this considerably, but that’s not quite precisely k8s either. If I’m going to start trimming off the parts of k8s I don’t need, I end up going all the way to single-node podman/docker… not the halfway point that is k3s.
- If you don’t use hostNetworking, the k8s model of traffic routes only with the cluster except for egress is all pure overhead. It’s totally necessary with you have a thousand engineers slinging services around your cluster, but there’s no benefit to this level fo rigor in service management in a homelab. Here again, the networking in podman/docker is more straightforward and maps better to the stuff I want to do in my homelab.
- Podman accepts a subset of k8s resource-yaml as a docker-compose-like config interface. This lets me use my familiarity with k8s configs iny podman setup.
Overall, the simplicity and lightweight resource consumption of podman/docker are are what I value at home. The extra layers of abstraction and constraints k8s employs are valuable at work, where we have a lot of machines and alot of people that must coordinate effectively… but I don’t have those problems at home and the overhead (compute overhead, conceptual overhead, and config-overhesd) of k8s’ solutions to them is annoying there.
Kubernetes is great if you run lots of services and/or already use kubernetes at work. I use it all the time and I’ve learned a lot on my personal cluster that I’ve taken to work to improve their systems. If you’re used to managing infra already then it’s not that much more work, and it’s great to be able to shutdown a server for maintenance and not have to worry about more than a brief blip on your home services.
I manage like 200 servers in Google cloud k8s but I don’t think I’d do that for home use. The core purpose is to manage multiple servers and assign processes between them, auto scaling, cluster internal network - running docker containers for single instance apps for personal use doesn’t require this kind of complexity
My NAS software has a docker thing just built into it. I can upload or specify a package and it just runs it on the local hardware. If you have a Linux shell, I guess all you really have to do is run
dockerdto start the daemon, make sure your network config allows connections, and upload your docker containers to it for runningThe Lemmy instance I’m speaking from right now is running in my k8s cluster.
Love is a strong word, but kubernetes is definitely interesting. I’m finishing up a migration of my homelab from a docker host running in a VM managed with Portainer to one smaller VM and three refurbished lenovo mini PCs running Rancher. It hasn’t been an easy road, but I chose to go with Rancher and k3s since it seemed to handle my usecase better than Portainer and Docker Swarm could. I can’t pass up those cheap mini PCs
Does rancher connect the pcs together? I have like 3 mini pcs sitting around, and I’ve always wanted to kinda combine them somehow
Like being able to combine cpu power or something. Idk if this is possible without getting a mobo with multiple cpu slots, but if it is. I’d love to learn!
Yeah, Kubernetes is designed to run in a cluster so you can pool processing power and memory from multiple devices. I banged my head against the wall for hours trying to figure out how to set up a cluster by hand, but then discovered if you install Rancher in a regular docker container it can handle all that for you
No shit. So you’re saying I can hook up like three mini pcs and make a mega at home server!? I gotta look into this. Did you follow a guide or anything you think is good enough or is as easy as a Google?
My recommendation is to look into k3sup and Rancher. I had a lot of trouble trying to install rancher in a docker container and migrating to a cluster after, and k3sup makes it really easy to set up a k3s cluster without having to configure everything manually
You can accomplish the same task with docker swarm, but I figured it would be better to learn something that wasn’t abandonware
I haven’t dug into the storage side yet since I have a separate NAS, but it will probably be beneficial to set up something like Ceph, GlusterFS, or Longhorn if you don’t have one
Oh I just realized this is for kubernates. Unraid is all dockers. Can a docker swarm also pool resources?
Yep, similar concept. Not sure how well unraid will handle the swarm behavior but I can imagine there’s someone out there who has tried it before
