AI on your own servers, where your data has to stay

Local AI means running a language model on hardware you control, in your office or on a dedicated server, instead of sending requests to a provider. The main reason is control over data: where a mandate, a contract or a regulator rules out external processing, local operation is often the only way to use AI at all. Law firms, tax advisers, medical practices and suppliers with strict confidentiality clauses ask about it most.

Local operation is a trade-off, not a badge. You gain control over where data goes. In return you carry hardware, updates and monitoring, and for some tasks the quality of a smaller open model may not reach the largest cloud models. Whether there is a gap depends on the task. Sorting, extracting fields and drafting standard replies are good candidates for smaller models, but only a comparison on your own cases shows the quality. Run that comparison as described in our article on your own AI benchmark.

Between your own server and a global cloud service lies a middle route: models through AWS Bedrock or Azure OpenAI in EU regions, under a data processing agreement. In typical EU deployments we use exactly that. Two details matter. The location of the resource you create is not always the location where a request is processed: Bedrock's geographic profiles keep processing within a geography such as the EU, but can route requests between regions of that geography. And Microsoft's EU Data Boundary covers the EU and the EFTA states, not only EU member states. Our article Local AI or EU cloud? compares the options with a data protection checklist and a cost example.

A mix is common: a local model for the most sensitive data, an EU cloud model for the rest. The workflow decides per case which route a document takes.

Three things decide the size of the hardware: the model and its quantisation, the context length you need and how many requests run at the same time. The parameter count alone does not tell you the memory you need, and graphics memory is not interchangeable with system memory. Our article on hardware for local AI explains the sizing and proposes an acceptance test for concurrency, because no vendor figure tells you how many of your users one machine serves.

The software is the smaller decision. Ollama and LM Studio make it easy to try models. For several users and steady load, a production server such as vLLM is the usual choice, with authentication and access rules in front of it; an API key alone is not complete protection. Our guide to Ollama in a company covers setup, model selection and licences.

Local does not automatically mean private. Ollama's documentation states that it does not see prompts or data when you run locally, but Ollama can also use cloud models and web search; these can be switched off with the setting OLLAMA_NO_CLOUD=1. Applications connected to the model may send data elsewhere on their own. So check the configuration, the connected applications and the network rules, and record the result.

Open models also come with different licences. The runtime's licence does not cover the model you download, and licences can change between model generations. Check the licence of each model for commercial use before it runs in your company.

1. Name the task and the data rules that apply to it.
2. Decide which data must stay local and which may go to an EU region.
3. Test two or three candidate models on twenty or more of your own cases.
4. Size the hardware from your measured load, or start on a rented GPU server in an EU data centre.
5. Agree updates, monitoring, access and a test run after every model change before go-live.

Our local AI page describes how a first project with us runs.