In brief
Open AI models are free to use and can run on your own hardware. They know a lot, but not everything about your field. Two techniques make them suitable for your own use: LoRA to train them further, and quantisation to make them compact.
Fine-tuning and LoRA
Fine-tuning means training an existing model further on your own examples. Updating the whole model takes a lot of computing power. LoRA (Low-Rank Adaptation) leaves the base model untouched and trains only a small addition, often less than one per cent of its size. You can merge that addition into the model later.
What that means in practice: fine-tuning a model with seven billion parameters can be done on one powerful workstation in a few hours.
Quantisation
A model consists of billions of numbers. Quantisation stores them with less precision, for example 4 bits instead of 16. That makes the model file smaller. What it does to speed and quality you can measure on your own hardware and with your own test questions. A 14 GB model then fits into about 4.5 GB. Common formats are GGUF (used by Ollama and llama.cpp, among others) with variants such as Q4_K_M.
Fine-tuning or RAG?
- RAG (looking up documents and passing them along) for knowledge that changes: rates, customers, procedures.
- Fine-tuning for skills and fixed formats: a programming language, a writing style, a fixed answer format.
The combination is often strongest. For example, we are training a small model to write ABAP, while current knowledge about the SAP system comes in through look-ups.
What to watch out for
- Good examples matter more than many examples. Check them, preferably automatically.
- Measure before and after with a fixed set of test questions.
- Check the licence of the base model: not every open model may be used commercially.
Would you like advice on fine-tuning AI models? Request a free advice conversation.
Updated on