Hi all, i am quite an old fart, so i just recently got excited about self hosting an AI, some LLM…
What i want to do is:
- chat with it
- eventually integrate it into other services, where needed
I read about OLLAMA, but it’s all unclear to me.
Where do i start, preferably with containers (but “bare metal”) is also fine?
(i already have a linux server rig with all the good stuff on it, from immich to forjeio to the arrs and more, reverse proxy, Wireguard and the works, i am looking for input on AI/LLM, what to self host and such, not general selfhosting hints)
If you like videos, I’d highly recommend techno Tims video on how to do this. Its what I used when building mine. Link
100% agree. TechnoTim is quite good. Also take a look at NetworkChuck. But be aware, these two will send you down rabbit holes of self-hosting ideas. Awesome rabbit holes, but rabbit holes nonetheless. I’ve spent weeks playing with stuff they’ve suggested. N8n and MCP is my latest obsession.
There’s another community for this: !localllama@sh.itjust.works
Though we mostly discuss the news and specific questions there, beginner questions are a bit more rare.I think you already got a lot of good answers here, LMStudio, OpenWebUI, LocalAI…
I’d like to add KoboldCpp that’s kind of made for gaming/dialogue, but it can do everything. And from my experience it’s very easy to set up and bundles everything into one program.One of these projects might be of interest to you:
Do note that CPU inference is quite a lot slower than GPU or the well known SAAS providers. I currently like the quantized deepseek models as the best balance between quality of replies and inference time when not using GPU.
Indeed, other than being able to get the model running, having decent hardware is the next most important part.
3060 12gb is probably cheapest card to get, 3090 or other 24gb card if you can get it
You can host ollama and open-webui on container. If you want to wire search you can connect open-webui to playwright (also container) and searxng (also container) and llm will search the web for answers
Openwebui is awesome and allows u to use it as an api for all the models u have it hooked up to. Can point it at ollama or any openai api compatible endpoint (like open routers)
LM Studio is a beast.
