llama.cpp — Beginner’s Guide for Mac

Run an AI model on your Mac, chat with it in your browser, and connect to it from other apps.

This guide uses llama.cpp’s built-in Hugging Face downloads. Pick a model, run one command, and let llama.cpp manage the files.

1. Install llama.cpp

If you don’t have Homebrew, install it from brew.sh.

In Terminal, run:

brew install llama.cpp

Check the installation:

llama version

If you already have llama.cpp installed, update it:

brew update
brew upgrade llama.cpp

This guide uses the llama serve and llama cli commands shown in the official quick start. Older examples may use the standalone executables llama-server and llama-cli.

2. Download and start a model

Run:

llama serve -hf Qwen/Qwen3-0.6B-GGUF:Q8_0

The first run downloads the model and starts the server. Later runs reuse the cached download.

We’re using Qwen3-0.6B, a small model for trying the setup. Expect limited answer quality compared with larger models.

Here’s what the command means:

Part	Meaning
llama serve	Start a server with browser chat and an API
-hf	Get a model from Hugging Face
Qwen/Qwen3-0.6B-GGUF	The model repository
:Q8_0	Choose its Q8_0 quantization

Wait for the download and model loading to finish. Leave Terminal open while using the server.

3. Open the chat

Visit http://localhost:8080 in your browser and send a message.

To stop the server, press Ctrl+C in Terminal. To start it again, run the same command from step 2.

You now have a local model running on your Mac.

Where are the model files?

Current versions use the Hugging Face cache by default:

~/.cache/huggingface/hub/

You don’t need to create this folder or refer to the files inside it. Continue using the model’s Hugging Face name in your commands.

The cache has its own directory structure. Existing environment settings can change its location; older llama.cpp releases used different defaults. See the cache implementation.

To list models llama.cpp recognizes in its cache:

llama serve --cache-list

Choose another model

Browse Hugging Face for models in GGUF format that llama.cpp supports. Check the model card for hardware requirements and usage instructions.

The command format is:

llama serve -hf PUBLISHER/REPOSITORY:QUANTIZATION

Replace those placeholders with values from the model’s repository. Stop your current server with Ctrl+C before starting another on the same port.

Quantization reduces the size of model weights by storing them at lower precision. Within the same model family and size:

Quantization	Typical tradeoff
Q4_K_M	Smaller download and lower memory use
Q5_K_M / Q6_K	More precision, with larger files
Q8_0	Still more precision, with larger files

Check which versions the repository actually provides. Running a model also needs memory for conversation context and computation, so its download size is only part of its memory requirement.

See llama.cpp’s model documentation for supported download workflows.

Optional: chat in Terminal

If you prefer Terminal to a browser, stop the server and run:

llama cli -hf Qwen/Qwen3-0.6B-GGUF:Q8_0

This uses the same cached model. You don’t need to download a separate copy for Terminal chat.

Optional: use the model through an API

The same server also provides an API. You can send it requests from your own code or connect software that supports an OpenAI-compatible server.

Start the server from step 2. To connect compatible software, use these settings:

Setting	Value
Base URL	http://localhost:8080/v1
Model	An ID returned by /v1/models
API key	This setup has no key configured; use local if the app requires a value

To get the model ID, open a second Terminal tab and run:

curl http://localhost:8080/v1/models

Copy an id from the response into MODEL_ID_HERE below:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MODEL_ID_HERE",
    "messages": [
      {"role": "user", "content": "Explain why the sky is blue in two sentences."}
    ],
    "stream": false
  }'

Find the answer in choices[0].message.content. These requests go to your Mac. See the server API documentation for more endpoints and authentication options.

Optional: adjust memory and idle behavior

After the basic setup works, you can restart the server with explicit settings:

llama serve -hf Qwen/Qwen3-0.6B-GGUF:Q8_0 \
  --host 127.0.0.1 \
  --port 8080 \
  -c 8192 \
  -np 1 \
  --sleep-idle-seconds 300

Option	Purpose
--host 127.0.0.1	Listen on this Mac’s loopback interface
--port 8080	Use port 8080
-c 8192	Set an 8,192-token context
-np 1	Use one concurrent generation slot
--sleep-idle-seconds 300	Unload the model after five idle minutes

Context includes your messages and the model’s responses. A larger context generally needs more memory; stay within the model’s supported context length.

Idle sleep releases model memory while leaving the server running. The next request reloads the model, so it takes longer to start responding. This setting does not control macOS sleep.

Optional: store downloads on another drive

Keep the default cache unless you need another location—for example, an external SSD with more space.

Choose one of the following locations. Add only one export LLAMA_CACHE=... line to ~/.zshrc.

External drive: replace YourDrive with your mounted drive’s name.

export LLAMA_CACHE="/Volumes/YourDrive/llama-cache"

Or, a folder in your home directory:

export LLAMA_CACHE="$HOME/models"

Open a new Terminal window and use the same -hf commands. llama.cpp will use your chosen location for its managed cache, with its own subdirectories.

Changing this setting does not move existing downloads. Models absent from the new location will need to be downloaded there. If you chose an external drive, keep it mounted whenever you use this cache. Cache configuration

Troubleshooting

Problem	What to check
llama is not found	Update your Homebrew installation of llama.cpp and open a new Terminal window.
Browser cannot connect	Wait for loading to finish and check Terminal for errors. Keep the server running.
Port 8080 is already in use	Stop the previous server, or add --port 8081 and open http://localhost:8081.
Download fails	Check your connection and the repository name. Private or gated models may require authentication.
Model runs out of memory	Try a smaller model or quantization, or reduce the configured context.
-m reports “No such file or directory”	-m expects an existing file. Use this guide’s -hf command to download and select the model automatically.
A flag is unrecognized	Update llama.cpp and check llama serve --help for your installed version.

For all available server options:

llama serve --help