Skip to content

Training an open AI model further: LoRA and quantisation

With LoRA you teach an open AI model your own field without retraining it from scratch, and with quantisation you make it small enough to run on your own hardware.

In brief

Open AI models are free to use and can run on your own hardware. They know a lot, but not everything about your field. Two techniques make them suitable for your own use: LoRA to train them further, and quantisation to make them compact.

Fine-tuning and LoRA

Fine-tuning means training an existing model further on your own examples. Updating the whole model takes a lot of computing power. LoRA (Low-Rank Adaptation) leaves the base model untouched and trains only a small addition, often less than one per cent of its size. You can merge that addition into the model later.

What that means in practice: fine-tuning a model with seven billion parameters can be done on one powerful workstation in a few hours.

Quantisation

A model consists of billions of numbers. Quantisation stores them with less precision, for example 4 bits instead of 16. That makes the model file smaller. What it does to speed and quality you can measure on your own hardware and with your own test questions. A 14 GB model then fits into about 4.5 GB. Common formats are GGUF (used by Ollama and llama.cpp, among others) with variants such as Q4_K_M.

Fine-tuning or RAG?

  • RAG (looking up documents and passing them along) for knowledge that changes: rates, customers, procedures.
  • Fine-tuning for skills and fixed formats: a programming language, a writing style, a fixed answer format.

The combination is often strongest. For example, we are training a small model to write ABAP, while current knowledge about the SAP system comes in through look-ups.

What to watch out for

  • Good examples matter more than many examples. Check them, preferably automatically.
  • Measure before and after with a fixed set of test questions.
  • Check the licence of the base model: not every open model may be used commercially.

Would you like advice on fine-tuning AI models? Request a free advice conversation.

Updated on

AI

Personal data in the AI cloud: who is watching?

A conversation with an AI service in the cloud is not private by default. A case in Canada shows what a provider can see and share. What does that mean under the GDPR, in the United Kingdom and in Switzerland?