llama.cpp — Beginner’s Guide for Mac

llama.cpp runs AI models locally on your computer. This guide covers installation, choosing a model, browser chat, and using the API.

1. Install llama.cpp

If you don’t have Homebrew, follow the instructions at brew.sh.

Install llama.cpp:

brew install llama.cpp

Verify the installation:

llama-server --version

To update an existing installation:

brew update
brew upgrade llama.cpp

Reference: Installation documentation.

2. Find and choose a model

Go to Hugging Face Models and search for a model name plus GGUF, the file format used by llama.cpp. Prefer the original model publisher’s upload when available.

On a model page:

1. Read the model card for capabilities, supported context length, and usage instructions.
2. Open Files and versions.
3. Choose a .gguf file.
4. Download it into ~/models, or copy its download link to use with curl.

For this walkthrough, use Qwen’s official Qwen3-0.6B-GGUF model. It’s a small download suitable for testing the setup, though its answer quality is limited compared with larger models.

Its files page contains Qwen3-0.6B-Q8_0.gguf, approximately 639 MB to download. Running it requires additional memory for conversation context and computation; total memory usage depends on your server settings.

Understanding model filenames

Part	Meaning
Qwen3	Model family
0.6B	Approximately 600 million parameters
Q8_0	An 8-bit quantization format
.gguf	File format

Why choose a quantized model?

Quantization stores model weights with lower precision, reducing download size and memory requirements, with a possible reduction in answer quality.

GGUF itself is a file format; it can also contain higher-precision weights.

When a repository offers several versions, these are useful starting points:

Version	Typical reason to choose it
Q4_K_M	A practical balance of memory use and quality
Q5_K_M / Q6_K	Preserve more precision if memory allows
Q8_0	Preserve still more precision, with larger files
F16 / BF16	Higher-precision weights with substantially higher memory requirements

These comparisons apply to versions of the same model. A higher quantization number doesn’t automatically make one model better than another.

For our small example, Q8_0 is already a manageable download and is available from the original publisher. It isn’t a universal recommendation to download every model in Q8.

References: GGUF documentation and llama.cpp quantization documentation.

Download the model

Create a model directory:

mkdir -p ~/models

Find the download URL on Hugging Face:

1. Open the model page.
2. Open Files and versions (the tab may be labeled Files).
3. Click Qwen3-0.6B-Q8_0.gguf to open its file page.
4. Choose Copy download link, or right-click Download and choose Copy Link Address / Copy Link in your browser.
5. Paste that link as the final argument in the curl command below, keeping it inside quotes.

The browser address for the file page contains /blob/main/. The download link contains /resolve/main/, which retrieves the file itself. Copy the download link rather than the browser address; otherwise, you may save an HTML page instead of the model.

For this file, the download URL is:

https://huggingface.co/Qwen/Qwen3-0.6B-GGUF/resolve/main/Qwen3-0.6B-Q8_0.gguf

URL part	Meaning
https://huggingface.co	Hosting website
Qwen	Publisher’s account
Qwen3-0.6B-GGUF	Model repository
resolve	Retrieve the file itself
main	Repository branch
Qwen3-0.6B-Q8_0.gguf	File to download

The copied link may also end in ?download=true; you can keep that suffix. Quote the complete URL when using it in Terminal.

You can instead click Download and save the file into ~/models through your browser. To download from Terminal, use:

curl -fL --retry 3 \
  -o ~/models/Qwen3-0.6B-Q8_0.gguf \
  "https://huggingface.co/Qwen/Qwen3-0.6B-GGUF/resolve/main/Qwen3-0.6B-Q8_0.gguf"

The options mean:

• -f: fail on HTTP errors.
• -L: follow redirects.
• --retry 3: retry eligible failures.
• -o: specify the destination file.

3. Choose how to serve models

Serve one local model

-m is short for --model. It points to a model file already on your computer:

llama-server -m ~/models/Qwen3-0.6B-Q8_0.gguf

Download and serve one model from Hugging Face

-hf selects a model from a Hugging Face repository, downloading it into llama.cpp’s cache if needed:

llama-server -hf Qwen/Qwen3-0.6B-GGUF:Q8_0

These downloads aren’t automatically placed in ~/models.

Offer a model picker

Router mode lets one server offer a collection of models at the same website and API address. You select a model in the browser picker, or specify its ID in an API request. The router loads the selected model when needed and forwards your request to it.

For example, if your model folder contains two supported GGUF models, you can select either one without stopping the server and starting it again with a different -m argument. Both can be available to select without both being loaded into memory at once; the command in section 4 limits loaded models to one with --models-max 1.

To start in router mode, omit -m and -hf and point the server at your model folder:

llama-server --models-dir ~/models

--models-dir tells the router where to find models. It doesn’t download them.

Without that option, router mode uses the llama.cpp cache. Restart the server after adding model files.

Reference: Multiple-model documentation.

4. Run the server for personal use

llama-server \
  --models-dir ~/models \
  --models-max 1 \
  --jinja \
  --host 127.0.0.1 \
  --port 8080 \
  -c 8192 \
  -np 1 \
  -ngl auto \
  --sleep-idle-seconds 300

Option	Purpose
--models-dir ~/models	Find models in your model directory
--models-max 1	Limit simultaneously loaded models to one
--jinja	Enable Jinja chat templates
--host 127.0.0.1	Accept connections from this Mac
--port 8080	Set the server’s port
-c 8192	Configure an 8,192-token context
-np 1	Use one concurrent generation slot
-ngl auto	Let llama.cpp choose GPU layer offloading
--sleep-idle-seconds 300	Release model memory after five minutes without tasks

Leave Terminal open while using the server. Press Ctrl+C to stop it.

Why configure idle sleep?

Without idle sleep, unused models can remain loaded in memory.

With --sleep-idle-seconds 300, the model and its context cache unload after five minutes without tasks. A new request reloads the model, so the next response takes longer to start.

This setting controls model memory, not whether macOS sleeps.

Reference: Idle sleep documentation.

Why choose an 8,192-token context?

The supported context window is model-dependent. There are two separate settings to understand:

• The model’s supported context: check its model card.
• Your configured context: how much capacity you give the local server.

For example, Qwen3-0.6B advertises a context length of 32,768 tokens.

Our command uses:

-c 8192 -np 1

This configures an 8,192-token context with one generation slot. The budget includes the system prompt, conversation history, current message, and generated response.

8,192 is a starter choice, not a model requirement. It provides room for ordinary chats while using less context-cache memory than a larger setting.

Increase it for longer documents or conversations if your model and available memory support that. Increasing the setting doesn’t make the model smarter, and exceeding its supported context doesn’t automatically extend its capabilities.

5. Open the website

Visit http://localhost:8080.

Choose a model and start chatting. Initial loading—and reloading after idle sleep—can take time.

If your model is missing, check your folder:

ls -lh ~/models

6. Use the API

No API key is required for the setup in this guide. llama-server defaults to no API authentication, and our command does not configure a key. These requests go to your own Mac at localhost:8080; they do not require an OpenAI account or OpenAI API key.

Authentication can be enabled with --api-key, --api-key-file, or their corresponding environment variables. If you configure authentication, clients must supply a matching key. If an app requires an API-key field even though your server has no authentication configured, enter a placeholder such as local.

List available model IDs:

curl http://localhost:8080/v1/models

Copy an exact id from the response into MODEL_ID_HERE:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MODEL_ID_HERE",
    "messages": [
      {
        "role": "user",
        "content": "Explain why the sky is blue in two sentences."
      }
    ],
    "stream": false
  }'

The answer appears in choices[0].message.content.

For apps supporting an OpenAI-compatible connection, use:

Setting	Value
Base URL	http://localhost:8080/v1
Model	Exact ID from /v1/models
API key	Not required by this configuration; use local if the app requires a value

Reference: Server API documentation.

7. Common setup pitfalls

Pitfall	What to check or change
Models stay in memory while unused	Add --sleep-idle-seconds 300; the next request will reload the model
Only one model is available	Stop the server started with -m or -hf. Put multiple supported GGUF models in ~/models, then run llama-server --models-dir ~/models --models-max 1. This lets the same server offer a browser model picker and load whichever model you select. See 3. Choose how to serve models.
Downloaded model doesn’t appear	Check whether it’s in ~/models or the separate download cache; restart after adding files
Model fits on disk but runs out of memory	Running also needs context cache and working memory; leave room for macOS and other apps
Context uses too much memory	Start with a smaller -c setting and increase when needed
Server won’t start	Check the log; another server may already be using port 8080
Background job prints a PID but the website fails	Check the log; a PID confirms launch, not successful startup
A new model fails to load	Update llama.cpp and check support for the model’s architecture
Commands have unfamiliar flags	Run llama-server --help; online examples may target another release

Appendix: Optional shell aliases

These optional shortcuts save you from typing the full server command each time. The main guide works without them.

Open your shell configuration:

nano ~/.zshrc

Add these lines, or replace the existing aliases if they’re already present:

alias localmodels='llama-server --models-dir ~/models --models-max 1 --jinja --host 127.0.0.1 --port 8080 -c 8192 -np 1 -ngl auto --sleep-idle-seconds 300'

alias localmodels-bg='nohup llama-server --models-dir ~/models --models-max 1 --jinja --host 127.0.0.1 --port 8080 -c 8192 -np 1 -ngl auto --sleep-idle-seconds 300 > ~/Library/Logs/llama-server.log 2>&1 & disown'

Save with Ctrl+O, press Enter, then exit with Ctrl+X.

Create the log directory and load your shortcuts:

mkdir -p ~/Library/Logs
source ~/.zshrc

Start in the foreground:

localmodels

Or start in the background:

localmodels-bg

Choose one; both use port 8080.

Background mode survives closing Terminal but does not automatically restart after a reboot.

How the background command works

The llama-server options are the same as in section 4. The surrounding parts are shell commands and operators that control how the server runs:

Part	Meaning
alias localmodels-bg='...'	Create a shortcut named localmodels-bg. The quoted command runs when you type that name.
nohup	Make the server ignore the hangup signal commonly sent when its terminal session closes. It does not put the server in the background by itself.
> ~/Library/Logs/llama-server.log	Send standard output to this log file instead of Terminal. Create the file if needed, or overwrite its existing contents on each launch. Use >> instead to append.
2>&1	Send standard error (file descriptor 2) to the same destination as standard output (file descriptor 1), so both go into the log.
&	Run the preceding command in the background, returning your Terminal prompt immediately.
disown	Remove the background job from the current shell’s job table. The process continues running independently of shell job management.

The order of the redirections matters: > ... 2>&1 sends standard output to the file first, then points standard error to that same destination.

Together, these let you keep using or close Terminal while the server runs and writes its output to a log. They do not prevent your Mac from sleeping or create a service that restarts automatically if it crashes.

Check background startup

tail -40 ~/Library/Logs/llama-server.log

A printed process ID confirms that the shell launched a job, not that the server successfully started.

Stop the background server

Find the process listening on port 8080:

lsof -nP -iTCP:8080 -sTCP:LISTEN

Then run the following, replacing PID with that server’s process ID:

kill PID