How to Run AI Models on Your Computer with Ollama: A Beginner’s Guide

Run AI Models Locally: What It Actually Means

When you use ChatGPT or Gemini, your questions travel over the internet to a company’s servers, get processed there, and the answer comes back. Running AI locally means the AI model lives on your own computer. You download it once, and from then on you can chat with it, ask it to write code, or summarize documents — with nothing leaving your machine.

Ollama is the most popular free tool for doing this. It is open-source (MIT licensed), handles downloading and running models for you, and works on Windows, Mac, and Linux. Once a model is downloaded, it runs fully offline, there are no per-message charges, and according to Ollama’s own FAQ, “we don’t see your prompts or data when you run locally.”

This guide shows you how to run AI models locally, taking you from zero to chatting with your own private AI model in about 15 minutes.

Prerequisites

Before you start, check that you have the following:

• A computer running Windows 10 22H2 or newer, macOS 14 (Sonoma) or newer, or a 64-bit Linux distribution
• An internet connection for the initial download (the model itself runs offline afterward)
• At least 8 GB of RAM (16 GB is more comfortable) — AI models live in memory
• Free disk space: a small starter model needs about 2 GB; larger ones need much more
• Basic comfort with a terminal or command prompt (you only need to copy and paste a few commands)

Step 1: Install Ollama

Ollama’s official installer is the safest way to get it. Do not download Ollama from random third-party sites — always use the official source.

On Windows

Download OllamaSetup.exe from the official download page at ollama.com/download. Run it and follow the prompts — the default install does not need administrator rights and installs into your user folder. Alternatively, install from PowerShell with this command:

irm https://ollama.com/install.ps1 | iex

After installation, Ollama keeps running in the background. Open PowerShell and check it with:

ollama --version

On macOS

Download the macOS app from ollama.com/download, or install from the Terminal with the official script:

curl -fsSL https://ollama.com/install.sh | sh

When prompted, allow the app to create its command-line link, then verify with:

ollama --version

On Linux

Run the official install script in your terminal:

curl -fsSL https://ollama.com/install.sh | sh

If the Ollama server is not running afterward, start it with:

ollama serve

Then verify the installation in another terminal window:

ollama --version

Step 2: Download and Run Your First Model

Ollama’s model library at ollama.com/library contains hundreds of open models. For your first run, pick a small one. Llama 3.2 (3B) is a good starting point — it is only about 2 GB, so it downloads quickly and runs on an ordinary laptop.

Type this command:

ollama run llama3.2

The first time you run it, Ollama downloads the model (this takes a few minutes depending on your connection). Then a chat prompt appears right in your terminal. Type a question, press Enter, and the answer streams back word by word.

To leave the chat session, type:

/bye

If you want a more capable model and your computer has at least 8 GB of graphics memory, try Gemma 4 E2B (about a 7.2 GB download):

ollama run gemma4:e2b

Step 3: Useful Commands to Know

Once you are comfortable chatting, these commands cover almost everything you will need day to day when you run AI models locally.

Command	What it does
ollama run llama3.2	Download (first time) and chat with a model
ollama pull llama3.2	Download a model without starting a chat
ollama list	Show the models already on your computer
ollama rm llama3.2	Delete a model to free up disk space
ollama --version	Check your installed Ollama version
ollama run llama3.2 "Summarize this:"	Ask one question without opening a chat session

Step 4: Choosing the Right Model Size

The biggest beginner mistake when people first run AI models locally is downloading a model that is too large for the computer.

Model size	Example	Download size	Needs
Small (1–3B parameters)	llama3.2, qwen3:0.6b	~0.5–2 GB	Any modern laptop
Medium (7–8B parameters)	gemma4:e2b	~7 GB	8 GB+ VRAM or unified memory
Large (20B+ parameters)	gpt-oss:20b	~14 GB	Powerful GPU, 16 GB+ VRAM
Very large (70B+)	llama3.3:70b	~40 GB+	Workstation-class hardware

Start small. A 3B model running smoothly on your laptop is far more useful than a 70B model that crawls at one word per second. You can always pull a bigger model later.

Step 5: Use the Local API in Your Own Projects

Ollama also runs a local web API at http://localhost:11434, so your own programs can talk to the model. This API needs no key for ordinary local use. Here is a quick test with curl:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Why is the sky blue?",
  "stream": false
}'

The API is compatible with the OpenAI format too, which means many existing apps and code examples work with a one-line change — just point them at your local address instead of the cloud.

Using the API from Python

If you write Python, the requests library is all you need to talk to your local model from a script:

import requests

response = requests.post(
    "http://localhost:11434/api/generate",
    json={
        "model": "llama3.2",
        "prompt": "Give me three tips for better sleep.",
        "stream": False,
    },
)
print(response.json()["response"])

Run this while Ollama is open and the model is downloaded, and you get an answer with no account, no key, and no cloud. This is how developers build private tools — document helpers, study apps, or automation — on top of a local model.

Troubleshooting

Problems are rare when you run AI models locally, but here are fixes for the common ones.

“ollama: command not found”

On Windows and Mac, the installer should add Ollama to your command path automatically — try closing and reopening your terminal first. On macOS, make sure you allowed the app to create its command-line link during setup. On Linux, check that /usr/local/bin is in your PATH.

The model is painfully slow

This almost always means the model is too big for your hardware. Run ollama rm on the large model and switch to a smaller one like llama3.2:1b. Closing other heavy programs also helps, since the model competes with them for memory.

Download fails or is very slow

Model downloads can be several gigabytes. If the download keeps failing, try ollama pull again — it resumes rather than restarting. Avoid mobile hotspots for large models.

Ollama won’t start on Linux

Run ollama serve manually in a terminal. If you want it to start automatically, Ollama’s documentation includes instructions for setting it up as a systemd service.

The model “forgets” things or gives odd answers

Small models have shorter memories and weaker reasoning than the big cloud models. Break your task into smaller steps, give clear instructions, and don’t expect a 1B model to write a novel. Match the task to the model’s size.

Security Warnings

A few important safety notes to follow when you run AI models locally:

• Download models only from Ollama’s official library (ollama.com/library) or other sources you fully trust. A model file is executable code — never pull one from a random link someone sent you.
• Local does not mean automatically private. Ollama itself does not see your prompts when you run locally, but if you install a third-party chat interface, connect a web-search plugin, or paste the output into a cloud service, your data can leave the computer through those.
• Keep API keys out of prompts. If you ask a local model to help with code, never paste real API keys, passwords, or tokens into the chat — treat model conversations like any other place you would not type a password.
• Verify anything the model tells you about your system. A local model can confidently give wrong advice about commands that delete files. Read a command before you run it.

5 Things to Try With Your Local Model

Once your first model is running, here are some practical things to try. They all work offline, with your data staying on your computer.

1. Summarize a long document

Paste a long article or a meeting transcript into the chat and ask: “Summarize this in 5 bullet points.” Because nothing leaves your machine, this is ideal for work documents you would never paste into a cloud chatbot.

2. Draft emails and messages

Ask the model to draft a reply in a specific tone: “Write a polite email declining this meeting invitation.” Edit the result yourself — treat it as a first draft, not a final answer.

3. Learn a new skill

Use the model as a patient tutor: “Explain how photosynthesis works like I am 12 years old, and quiz me at the end.” Small models handle teaching surprisingly well.

4. Brainstorm ideas

Stuck on a project name, a gift idea, or a weekend plan? Ask for 10 options. The privacy of a local model makes it a safe place for half-formed ideas you would not type into a public service.

5. Get coding help

Ask for small code snippets or explanations: “Write a Python function that removes duplicates from a list, and explain each line.” Larger models do this noticeably better, so this is a good reason to graduate to an 8B model once you are comfortable.

Ollama’s Desktop App

If you prefer not to live in the terminal, Ollama also ships a desktop application with a clean chat interface. On Windows and macOS, the app runs in the background after installation and includes a model picker — you can download and switch models from a dropdown instead of typing commands. Everything underneath is the same local engine, so the privacy properties do not change. The terminal commands in this guide still work alongside the app, and many people use both: the app for casual chats, the terminal for scripting.

Local Models vs Cloud Models

One thing that confuses beginners: Ollama now also offers cloud models, whose names end in :cloud (for example, gemma4:cloud). These run on Ollama’s servers, not your computer. They let you use bigger models on weaker hardware, but they require an Ollama account — and your prompts leave your computer. If privacy is your reason for running AI locally, stick to the regular model names without the :cloud suffix. You can also disable cloud features entirely with the OLLAMA_NO_CLOUD=1 setting or the app’s airplane mode.

Summary

To run AI models locally, install Ollama from the official source, run ollama run llama3.2, and you are chatting with a private model that needs no internet and charges nothing. Start with a small model that fits your hardware, learn the basic commands, and only then explore bigger models or the local API. Your prompts stay on your machine — and that is the whole point.

For this week’s biggest AI release, read our breakdown of Google’s Gemini 4 Carbon — and for the policy side of AI, see our guide to Japan’s AI regulation framework.

Sources

• Ollama — official site and download page
• Ollama on GitHub — official repository and README
• AMD Developer — Getting Started with Ollama