Skip to content

What is Ollama?

Ollama is free open-source software that lets you run AI language models on your own computer or server, without the cloud and without costs per question.

In short

Ollama makes local AI simple. You install it on a computer or server, choose an open model (such as Gemma, Llama, Qwen or Mistral) and start that model on your own machine. Applications talk to Ollama via a simple web API on the machine itself, similar to the APIs of the big AI providers. With fully local processing, the input and the answers stay in your own environment.

How does it work

  • Download models with a single command, in a compact format (GGUF) that also fits on ordinary hardware.
  • Run on the graphics card or Apple Silicon, for speed. On a Mac with plenty of memory, large models also run well.
  • Several models side by side, where the available memory decides which models can stay loaded at the same time.

What do you need

Memory determines which models fit. A small model of a few billion parameters runs on an ordinary laptop. Models of twenty to thirty billion parameters require a workstation or server with plenty of memory. Besides the model itself, allow room for the context: the longer the texts you pass in, the more memory you need.

What you can use it for

  • An internal assistant that answers questions about your own documents.
  • Summarising, translating or checking texts.
  • Extracting data from documents.
  • As the engine behind your own AI application, for example with model escalation between small and large models.

What to watch out for

  • Match the model to the task. A small model is fast, but more likely to make things up.
  • Check the licence of each model.
  • Set up management properly: which models are loaded, how long they stay in memory, and who may use the API.

Would you like advice on Ollama? Request a free advice conversation.

Updated on

AI

Personal data in the AI cloud: who is watching?

A conversation with an AI service in the cloud is not private by default. A case in Canada shows what a provider can see and share. What does that mean under the GDPR, in the United Kingdom and in Switzerland?