AI, LLM

Setting Up Local LLM

Ollama Installation

• Update NVIDIA Driver Version

  • As of July 2026, Ollama requires NVIDIA 551.61 or newer drivers if you have an NVIDIA card

  • In cmd: nvidia-smi or Start > NVIDIA Control Panel > System Information (bottom left corner)

  • Update Windows: Start > Settings > Windows Update

  • Update driver version: Download and install from Manual Driver Search

    • Updating driver will fail if you have GTX 10 series on Windows 11 25H2. They have conflict

      • If that’s your case, download NVIDIA GeForce Hotfix Driver 581.94 (NVIDIA’s hotfix for 24H2/25H2)

      • Install the driver only, not NVIDIA App, as Ollama only needs the driver, and there could be conflict again

    • Example (failed):

      • Product Category: GeForce

      • Product Series: GeForce 10 Series

      • Product: GeForce GTX 1080

      • Operating System: Windows 11

      • Find > Download Studio version

        • Game Ready is the latest fastest version, while Studio version is a recent stable version

• Download and Install Ollama for Windows

• Configure Model Location (Optional)

  • By default, Ollama pulls models into C:\Users\scott\.ollama\models\blobs, not an issue if you have huge C drive

  • To change the model location, set system env variable OLLAMA_MODELS to, say D:\OllamaModels

    • System, not user. System env variables are under user

    • Delete C:\Users\scott\.ollama\models\ if it already exists

  • Run ollama pull tinyllama in cmd to test if it’s pulled into D drive. This is a tiny model

Ollama Commands

• Run a model

  • ollama run qwen2.5-coder:7b

  • If it’s the first time you use a specific model, it takes a few minutes to pull it

  • When you see the >>> prompt just say something like “code me a python quick sort”

  • To quit: /exit, /bye or Ctrl-D

• Pull model, but not run

  • ollama pull qwen2.5-coder:7b

  • ollama pull qwen2.5-coder:1.5b

  • ollama pull tinnyllama:latest

• Delete downloaded models

  • ollama list: Check existing models

  • ollama rm <YOUR_LLM>

• Full list of Ollama commands: ollama --help

• Quit Ollama: Right click on the bottom right corner ollama icon in windows

• Fix seed of a model: ollama create qwen7b-fixed -f ./Modelfile with the below Modelfile

FROM qwen2.5-coder:7b

# Fix random seed
PARAMETER temperature 0
PARAMETER top_p 0.01
PARAMETER seed 42

# Forcing LaTeX format
SYSTEM """
You are an expert quantitative developer.
CRITICAL FORMATTING RULE:
1. You MUST use single dollar signs for inline math equations, like $e = mc^2$.
2. You MUST use double dollar signs for standalone block math equations, like $$E = mc^2$$.
3. NEVER use \( \) or \[ \] for math equations under any circumstances. If you use them, the system will crash. Always convert them to $ or $$.
"""

Some Open-Weight LLMs

• Search models on the Ollama Library

• Coding: SetneufPT/Qwen3.6-27B-CODER-MTP_Q4_105k_24GB-GPU

• Math, logical reasoning: deepseek-r1:70b

  • Find the ones tagged thinking

• 27B models require 24 GB RAM

• 70B models require 64 GB RAM

• DO NOT USE cloud models like glm-5.2:cloud

  • They are not open-weight, not running locally

  • Only paper/architecture is published

Ollama – Behind the Scenes

• Ollama is an server running on http://localhost:11434/ which accepts REST API

• Example endpoints:

  • http://localhost:11434/api/version GET

  • http://localhost:11434/api/generate POST

  • http://localhost:11434/api/chat POST

• VS Code extensions can package user’s prompt, send as a payload with rest call to Ollama, and get response

Continue

• Install VS Code extension: Continue - open-source AI code agent

• Right click > Move To > Secondary Side Bar (optional)

• Open settings > Configs > click on Main Config

  • It will open C:\Users\scott\.continue\config.yaml

  • Copy and paste the config (see below)

• Without capabilities: [] in the config, the agent will spit out tool-calling JSON, not code

• systemMessage is for AI, not users

• Optional Project Rules (.continue/rules/)

  • Continue parses .md rule files automatically at conversation initialization

name: Local Config
version: 0.0.1
schema: v1
models:
  - name: Qwen 7B Coder
    provider: ollama
    model: qwen2.5-coder:7b
    # apiBase: "http://10.0.0.100"  # required for local network deployment
    roles:
      - chat
      - edit
    capabilities: []
    systemMessage: "You are an expert developer. Respond ONLY with raw code blocks or direct text explanations. NEVER output tool-calling JSON or wrap your entire response in a JSON object."
  - name: Qwen 1.5B Autocomplete
    provider: ollama
    model: qwen2.5-coder:1.5b
    # apiBase: "http://10.0.0.100"  # required for local network deployment
    roles:
      - autocomplete

Local Network Deployment

• Mac Studio server farm

  • Mac has CPU and GPU share a single, coherent memory pool (Unified Memory Architecture, UMA)

  • Apple Mac Studio with M3 Ultra, which comes with 96, 256 or 516 GB RAMs

  • 4 server machines are enough for 25 people

• Server machine setup

  • Each machine has its own Ollama running, exposed through NGINX

  • NGINX set up on two of the 4: Central server and backup server

  • On each machine, run launchctl setenv OLLAMA_HOST "0.0.0.0:11434" and reboot ```yaml # nginx.conf for both central and backup server, mirrored & managed by Keepalived

upstream ollama_pool { least_conn; # send requests to the least busy server server 10.0.0.51:11434 max_fails=2 fail_timeout=30s; server 10.0.0.52:11434 max_fails=2 fail_timeout=30s; server 10.0.0.53:11434 max_fails=2 fail_timeout=30s; server 127.0.0.1:11434 max_fails=2 fail_timeout=30s; # this machine keepalive 64; }

server { listen 80; server_name llm-gateway.local;

location / {
    proxy_pass http://ollama_pool;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host localhost; # tell Ollama the request is from Host Header so that Ollama won't reject it

    proxy_buffering off;             # So that token stream sends out real time
    proxy_read_timeout 600s;         # It takes time if output text is a lot

    # access control
    allow 192.168.1.0/24;  # only we can use it 192.168.1.1 - 192.168.1.254
    deny all;              # all others get 403 Forbidden
}

}

### Local Network Deployment: Single-Node Starter Setup

* Simpler, no NGINX needed, optimal for ~5 Users
* One single Apple Mac Studio acts as a centralized API server over the secure local network/VPN
    * Install Ollama
    * Run `launchctl setenv OLLAMA_HOST "0.0.0.0:11434"` and reboot
    * macOS might have firewall, in which case allow all incoming connections to Ollama
* Optionally set up [Open WebUI](https://github.com/open-webui/open-webui) on Mac Studio for teammates to chat on the browser, not IDE
    * Install [Docker Desktop](https://www.docker.com/products/docker-desktop/) and run
```bash
docker run -d \
  -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  -v open-webui:/app/backend/data \
  --name open-webui \
  --restart always \
  ghcr.io/open-webui/open-webui:main

VS Code Copilot BYOK (Agent Mode Not Working)

• Install VS Code extension: Ollama

• Once done there will be a ollama section in copilot chat

• Switch model (using llama3.1 as example):

  • ollama pull llama3.1

  • Ctrl + Shift + P > Ollama: Refresh Models

• All failed to edit the files in repo:

  • qwen2.5-coder:7b gives me JSON again, not code

  • llama3.1 gives cli commands and ask me to run it myself

Model Capacity

• qwen2.5-coder:7b

  • 可以做：

    • 單一檔案解釋

    • 為一個 class 寫 unittests

    • review one function

    • Documentation, comment

  • 做不到：

    • 理解大型 repo 架構

    • 跨 modules 追蹤 control flow

    • 大規模 refactor

    • 理解 domain-specific abstractions

• VRAM requirement

  • 我的 8 GB VRAM 跑 qwen 7b 就差不多全滿了

  • VRAM 裡要裝

    • 0.5 GB \(\times\) # parameters (billions)（用 4-bit quantization 粗估）

    • KV cache

    • runtime buffers

    • macOS

    • ollama

    • code index

    • context window

  • 48 GB 可以跑 35B 但輸入大型 code context 還是會因為 KV cache 太大卡住

  • MoE model: 雖然參數多但每次 inference 只 active 一小部份

  • 96 GB 可以跑 70B

• What coding assistants (like continue) do:

  • Index the entire repo

  • Find files with embeddings, keyword search or symbol search

  • 選出片段交給 LLM

  • 必要時讓 agent 再搜尋更多檔案

• 參數越多越聰明？

  • 2020 - 2023: 7B << 13B << 34B << 70B << GPT-4

  • 2026: quality = traning data + architecture + context handling + tool use + parameter cout

    • 32B 專門訓練 coding 在 coding 上的表現勝過 70B generic model

  • 70B 的強項：

    • 長推理

    • 幻覺少

    • knowledgable

    • 多步驟 planning

    • coding style

    • 大型 repo 可以理解 architecture

  • 14B 90% capability vs 70B 95% capability 但很慢又貴很多，觀眾會選 14B

• 目前（July 2026）Open weight model 導不出 SABR-Cheyette

任務	好的 7B	好的 14B	好的 30–35B	雲端旗艦模型
一般 coding	70	82	90	100
大型 C++ 專案	55	75	88	100
Numerical methods	50	70	85	100
Quant research	35	55	75–85	100

What To Test

• 同一個 repo，讓 14B, 32B, 70B

  • 找出某個 pricing function 的完整 call chain

  • 修改一個 interface，影響 6 個檔案

  • 幫一個陌生 module 加 unit tests

  • 找出一個跨 module 的 bug

Context Window

• Context Window = Input Tokens + Output Tokens 上限

• 誰決定了 Context Window？

  • 模型定上限

  • Inference engine (Ollama) 決定配置多少

  • IDE extension (Continue)

• Ollama 接受 OLLAMA_KV_CACHE_TYPE=q8_0 or q4_0 讓 context window 變大，但會有些微品質損失

• IDE extension (Continue) 送：

  • 你的 prompt (say 20 tokens)，current file, imports, related files, gid idff, symbols，加起來 18000 token

• 對大型 codebase 更不能依賴超長。不要想著把整個 repository 塞進 100k token context，而是

  • 建立 repository index

  • 用 symbol search、dependency graph 或 embeddings 找相關部分

  • 每次只送必要的檔案和介面

  • 需要時讓 agent 再取下一批 context

Retrieval-Augmented Generation (RAG)

• 找出最相關的資訊給模型

  • Global search keywords，抓 dependency (include, import)，再 Global search keyword dependency 裡的 keywords，如此重覆

  • Language Server, Go to Definition, Find References, IDE features

  • Embedding：把每個 file 向量化

  • Agent（July 2026 目前最強）

    • 第一輪找 interface，看到它呼叫 PricingEngine

    • 第二輪送 PricingEngine，看到它呼叫 VolSurface

    • 第三輪送 VolSurface

    • 每一輪不需要很大的 context window

• 只要抓的都是需要的，context window 不用很大也能給出很好的答案

• Retrieval 是做在 IDE extension（Continue）上的，model 只負責看著 input tokens 吐出 output tokens

• 現代 agent 系統裡，Retrieval 和模型是合作的

  • 模型思考需要哪些資訊 > tool calling > Retrieval 回傳結果 > 模型看完說，我還需要 VolSurface.cpp > 再做一次 Retrieval

  • Retrieval 負責找資料，模型決定要找什麼，怎麼用找到的資料

• AI coding agent 能力 = agent + retrieval + model

Agent

• Agent 是一個 workflow，orchestrator

• Agent = Retrieval + File Reader + Terminal + Git + Compiler + Browser + Model

• Agent 會控管整個流程，例如如果模型連續十次要求讀同一個檔案，Agent 很可能會偵測到異常並停止，而不是無限照做

• 現代 AI coding stack：Prompt > Agent > LLM > Tool Calls > Retrieval > Model 再推理 > 完成任務

Agent Example

• User prompt: please fix the compile error

• Agent sends

You are an autonomous coding assistant.

You have these tools:

1. search_code(...)
2. read_file(...)
3. edit_file(...)
4. run_tests(...)
5. run_terminal(...)

Decide the next action.

• Model returns tool calling

{
  "tool": "search_code",
  "query": "PricingEngine"
}

• Agent runs search_code("PricingEngine") and got PricingEnging.cpp and PricingEnging.h

• Agent asks again

Search results:

PricingEngine.cpp

PricingEngine.h

What's next?

• Model says

{
    "tool":"read_file",
    "path":"PricingEngine.cpp"
}

模型記憶機制

• 當長對話變長，不送整個歷史而是搜尋關鍵字，summarize 上下文，以 summary 代替整個歷史。這也是一種 Retrieval

Version 2

• 不是 Can local AI run large code bases?

• 是 Can an open-source coding stack support large code bases?

Stack	Large Codebase Navigation	Cross-file Edit	Unit Test	Build Fix	Speed
Continue + Ollama	?	?	?	?	?
Cline + Ollama	?	?	?	?	?
Roo Code + Ollama	?	?	?	?	?

• Continue 是第一波真正做出完整 open-source AI coding assistant 生態系的代表

能力	Cursor（商業）	Open Source 現況	是否已接近
本地模型支援	v	v（Ollama 等）	幾乎追上
Codebase Retrieval	v	v（Continue/Cline 等）	接近，但仍需實測
Agent	v	v	接近，但成熟度不一
Tab Completion	* 自研模型	有，但品質差異較大	尚有差距
多 Agent 協作	v	開始出現	仍在快速演進
整體使用體驗	非常成熟	工具整合程度不一	還有差距

• Open Source 能不能以較低成本、較高隱私，提供足夠好的工程生產力？