{ "cells": [ { "cell_type": "markdown", "id": "ce74b798-76f3-4985-826f-7e176d8324db", "metadata": {}, "source": [ "# AI, LLM" ] }, { "attachments": {}, "cell_type": "markdown", "id": "1daa3318-8a9b-449b-812c-27c19f5f4358", "metadata": {}, "source": [ "## Setting Up Local LLM\n", "\n", "### Ollama Installation\n", "\n", "* Update NVIDIA Driver Version\n", " * As of July 2026, Ollama [requires](https://docs.ollama.com/windows#system-requirements) NVIDIA 551.61 or newer drivers if you have an NVIDIA card\n", " * In `cmd`: `nvidia-smi` or Start > NVIDIA Control Panel > System Information (bottom left corner)\n", " * Update Windows: Start > Settings > Windows Update\n", " * Update driver version: Download and install from [Manual Driver Search](https://www.nvidia.com/en-us/software/nvidia-app/)\n", " * **Updating driver will fail if you have GTX 10 series on Windows 11 25H2.** They have conflict\n", " * If that's your case, download NVIDIA GeForce Hotfix Driver 581.94 (NVIDIA's hotfix for 24H2/25H2)\n", " * Install the driver only, not NVIDIA App, as Ollama only needs the driver, and there could be conflict again\n", " * Example (failed):\n", " * Product Category: GeForce\n", " * Product Series: GeForce 10 Series\n", " * Product: GeForce GTX 1080\n", " * Operating System: Windows 11\n", " * Find > Download Studio version\n", " * Game Ready is the latest fastest version, while Studio version is a recent stable version\n", "* Download and Install [Ollama for Windows](https://ollama.com/download/windows) \n", "* Configure Model Location (Optional)\n", " * By default, Ollama pulls models into `C:\\Users\\scott\\.ollama\\models\\blobs`, not an issue if you have huge C drive\n", " * To change the model location, set **system** env variable `OLLAMA_MODELS` to, say `D:\\OllamaModels`\n", " * System, not user. System env variables are under user\n", " * Delete `C:\\Users\\scott\\.ollama\\models\\` if it already exists\n", " * Run `ollama pull tinyllama` in `cmd` to test if it's pulled into D drive. This is a tiny model\n", "\n", "### Ollama Commands\n", "\n", "* Run a model\n", " * `ollama run qwen2.5-coder:7b`\n", " * If it's the first time you use a specific model, it takes a few minutes to pull it\n", " * When you see the `>>>` prompt just say something like \"code me a python quick sort\"\n", " * To quit: `/exit`, `/bye` or `Ctrl-D`\n", "* Pull model, but not run\n", " * `ollama pull qwen2.5-coder:7b`\n", " * `ollama pull qwen2.5-coder:1.5b`\n", " * `ollama pull tinnyllama:latest`\n", "* Delete downloaded models\n", " * `ollama list`: Check existing models\n", " * `ollama rm `\n", "* Full list of Ollama commands: `ollama --help`\n", "* Quit Ollama: Right click on the bottom right corner ollama icon in windows\n", "* Fix seed of a model: `ollama create qwen7b-fixed -f ./Modelfile` with the below Modelfile\n", "\n", "```python\n", "FROM qwen2.5-coder:7b\n", "\n", "# Fix random seed\n", "PARAMETER temperature 0\n", "PARAMETER top_p 0.01\n", "PARAMETER seed 42\n", "\n", "# Forcing LaTeX format\n", "SYSTEM \"\"\"\n", "You are an expert quantitative developer.\n", "CRITICAL FORMATTING RULE: \n", "1. You MUST use single dollar signs for inline math equations, like $e = mc^2$.\n", "2. You MUST use double dollar signs for standalone block math equations, like $$E = mc^2$$.\n", "3. NEVER use \\( \\) or \\[ \\] for math equations under any circumstances. If you use them, the system will crash. Always convert them to $ or $$.\n", "\"\"\"\n", "```\n", "\n", "### Some Open-Weight LLMs\n", "\n", "* Search models on the [Ollama Library](https://ollama.com/library?sort=newest)\n", "* Coding: `SetneufPT/Qwen3.6-27B-CODER-MTP_Q4_105k_24GB-GPU`\n", "* Math, logical reasoning: `deepseek-r1:70b`\n", " * Find the ones tagged `thinking`\n", "* 27B models require 24 GB RAM\n", "* 70B models require 64 GB RAM\n", "* **DO NOT USE** cloud models like `glm-5.2:cloud`\n", " * They are not open-weight, not running locally\n", " * Only paper/architecture is published\n", "\n", "### Ollama -- Behind the Scenes\n", "\n", "* Ollama is an server running on http://localhost:11434/ which accepts REST API\n", "* Example endpoints:\n", " * http://localhost:11434/api/version GET\n", " * http://localhost:11434/api/generate POST\n", " * http://localhost:11434/api/chat POST\n", "* VS Code extensions can package user's prompt, send as a payload with rest call to Ollama, and get response\n", "\n", "### Continue\n", "\n", "* Install VS Code extension: **Continue - open-source AI code agent**\n", "* Right click > Move To > Secondary Side Bar (optional)\n", "* Open settings > Configs > click on Main Config\n", " * It will open `C:\\Users\\scott\\.continue\\config.yaml`\n", " * Copy and paste the config (see below)\n", "* Without `capabilities: []` in the config, the agent will spit out tool-calling JSON, not code\n", "* `systemMessage` is for AI, not users\n", "* Optional Project Rules (`.continue/rules/`)\n", " * Continue parses `.md` rule files automatically at conversation initialization\n", "\n", "```yaml\n", "name: Local Config\n", "version: 0.0.1\n", "schema: v1\n", "models:\n", " - name: Qwen 7B Coder\n", " provider: ollama\n", " model: qwen2.5-coder:7b\n", " # apiBase: \"http://10.0.0.100\" # required for local network deployment\n", " roles:\n", " - chat\n", " - edit\n", " capabilities: []\n", " systemMessage: \"You are an expert developer. Respond ONLY with raw code blocks or direct text explanations. NEVER output tool-calling JSON or wrap your entire response in a JSON object.\"\n", " - name: Qwen 1.5B Autocomplete\n", " provider: ollama\n", " model: qwen2.5-coder:1.5b\n", " # apiBase: \"http://10.0.0.100\" # required for local network deployment\n", " roles:\n", " - autocomplete\n", "```\n", "\n", "### Local Network Deployment\n", "\n", "* Mac Studio server farm\n", " * Mac has CPU and GPU share a single, coherent memory pool (Unified Memory Architecture, UMA)\n", " * **Apple Mac Studio with M3 Ultra**, which comes with 96, 256 or 516 GB RAMs\n", " * 4 server machines are enough for 25 people\n", "* Server machine setup\n", " * Each machine has its own Ollama running, exposed through NGINX\n", " * NGINX set up on two of the 4: Central server and backup server\n", " * On each machine, run `launchctl setenv OLLAMA_HOST \"0.0.0.0:11434\"` and reboot\n", "```yaml\n", "# nginx.conf for both central and backup server, mirrored & managed by Keepalived\n", "\n", "upstream ollama_pool {\n", " least_conn; # send requests to the least busy server\n", " server 10.0.0.51:11434 max_fails=2 fail_timeout=30s;\n", " server 10.0.0.52:11434 max_fails=2 fail_timeout=30s;\n", " server 10.0.0.53:11434 max_fails=2 fail_timeout=30s;\n", " server 127.0.0.1:11434 max_fails=2 fail_timeout=30s; # this machine\n", " keepalive 64;\n", "}\n", "\n", "server {\n", " listen 80;\n", " server_name llm-gateway.local;\n", "\n", " location / {\n", " proxy_pass http://ollama_pool;\n", " proxy_http_version 1.1;\n", " proxy_set_header Connection \"\";\n", " proxy_set_header Host localhost; # tell Ollama the request is from Host Header so that Ollama won't reject it\n", " \n", " proxy_buffering off; # So that token stream sends out real time\n", " proxy_read_timeout 600s; # It takes time if output text is a lot\n", "\n", " # access control\n", " allow 192.168.1.0/24; # only we can use it 192.168.1.1 - 192.168.1.254\n", " deny all; # all others get 403 Forbidden\n", " }\n", "}\n", "``` \n", "\n", "### Local Network Deployment: Single-Node Starter Setup\n", "\n", "* Simpler, no NGINX needed, optimal for ~5 Users\n", "* One single Apple Mac Studio acts as a centralized API server over the secure local network/VPN\n", " * Install Ollama\n", " * Run `launchctl setenv OLLAMA_HOST \"0.0.0.0:11434\"` and reboot\n", " * macOS might have firewall, in which case allow all incoming connections to Ollama\n", "* Optionally set up [Open WebUI](https://github.com/open-webui/open-webui) on Mac Studio for teammates to chat on the browser, not IDE\n", " * Install [Docker Desktop](https://www.docker.com/products/docker-desktop/) and run\n", "```bash\n", "docker run -d \\\n", " -p 3000:8080 \\\n", " --add-host=host.docker.internal:host-gateway \\\n", " -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \\\n", " -v open-webui:/app/backend/data \\\n", " --name open-webui \\\n", " --restart always \\\n", " ghcr.io/open-webui/open-webui:main\n", "```\n", "\n", "### VS Code Copilot BYOK (Agent Mode Not Working)\n", "* Install VS Code extension: Ollama\n", "* Once done there will be a ollama section in copilot chat\n", "* Switch model (using llama3.1 as example):\n", " * `ollama pull llama3.1`\n", " * Ctrl + Shift + P > Ollama: Refresh Models\n", "* All failed to edit the files in repo: \n", " * `qwen2.5-coder:7b` gives me JSON again, not code\n", " * `llama3.1` gives cli commands and ask me to run it myself" ] }, { "cell_type": "markdown", "id": "896a9224-ad7f-45c0-9c40-2fb8ac1ab6d8", "metadata": {}, "source": [ "## Model Capacity\n", "\n", "* `qwen2.5-coder:7b`\n", " * 可以做：\n", " * 單一檔案解釋\n", " * 為一個 class 寫 unittests\n", " * review one function\n", " * Documentation, comment\n", " * 做不到：\n", " * 理解大型 repo 架構\n", " * 跨 modules 追蹤 control flow\n", " * 大規模 refactor\n", " * 理解 domain-specific abstractions\n", "* VRAM requirement\n", " * 我的 8 GB VRAM 跑 qwen 7b 就差不多全滿了\n", " * VRAM 裡要裝\n", " * 0.5 GB $\\times$ # parameters (billions)（用 4-bit quantization 粗估）\n", " * KV cache\n", " * runtime buffers\n", " * macOS\n", " * ollama\n", " * code index\n", " * context window\n", " * 48 GB 可以跑 35B 但輸入大型 code context 還是會因為 KV cache 太大卡住\n", " * MoE model: 雖然參數多但每次 inference 只 active 一小部份\n", " * 96 GB 可以跑 70B\n", "* What coding assistants (like continue) do: \n", " * Index the entire repo\n", " * Find files with embeddings, keyword search or symbol search\n", " * 選出片段交給 LLM\n", " * 必要時讓 agent 再搜尋更多檔案\n", "* 參數越多越聰明？\n", " * 2020 - 2023: 7B << 13B << 34B << 70B << GPT-4\n", " * 2026: quality = traning data + architecture + context handling + tool use + parameter cout\n", " * 32B 專門訓練 coding 在 coding 上的表現勝過 70B generic model\n", " * 70B 的強項：\n", " * 長推理\n", " * 幻覺少\n", " * knowledgable\n", " * 多步驟 planning\n", " * coding style\n", " * 大型 repo 可以理解 architecture\n", " * 14B 90% capability vs 70B 95% capability 但很慢又貴很多，觀眾會選 14B\n", "* 目前（July 2026）Open weight model 導不出 SABR-Cheyette\n", "\n", "\n", "| 任務 | 好的 7B | 好的 14B | 好的 30–35B | 雲端旗艦模型 |\n", "| ----------------- | ----: | -----: | --------: | -----: |\n", "| 一般 coding | 70 | 82 | 90 | 100 |\n", "| 大型 C++ 專案 | 55 | 75 | 88 | 100 |\n", "| Numerical methods | 50 | 70 | 85 | 100 |\n", "| Quant research | 35 | 55 | 75–85 | 100 |\n" ] }, { "cell_type": "markdown", "id": "efafffc5-54d9-4544-bae4-2bc08059601b", "metadata": {}, "source": [ "## What To Test\n", "\n", "* 同一個 repo，讓 14B, 32B, 70B\n", " * 找出某個 pricing function 的完整 call chain\n", " * 修改一個 interface，影響 6 個檔案\n", " * 幫一個陌生 module 加 unit tests\n", " * 找出一個跨 module 的 bug" ] }, { "cell_type": "markdown", "id": "8b25e000-0ebf-47a5-870a-62ec54b363fc", "metadata": {}, "source": [ "## Context Window\n", "\n", "* Context Window = Input Tokens + Output Tokens 上限\n", "* 誰決定了 Context Window？\n", " * 模型定上限\n", " * Inference engine (Ollama) 決定配置多少\n", " * IDE extension (Continue)\n", "* Ollama 接受 `OLLAMA_KV_CACHE_TYPE=q8_0` or `q4_0` 讓 context window 變大，但會有些微品質損失\n", "* IDE extension (Continue) 送：\n", " * 你的 prompt (say 20 tokens)，current file, imports, related files, gid idff, symbols，加起來 18000 token\n", "* 對大型 codebase 更不能依賴超長。不要想著把整個 repository 塞進 100k token context，而是\n", " * 建立 repository index\n", " * 用 symbol search、dependency graph 或 embeddings 找相關部分\n", " * 每次只送必要的檔案和介面\n", " * 需要時讓 agent 再取下一批 context" ] }, { "cell_type": "markdown", "id": "956fe6c0-af34-4238-a7d3-936cf4e74ed3", "metadata": {}, "source": [ "## Retrieval-Augmented Generation (RAG)\n", "\n", "* 找出最相關的資訊給模型\n", " * Global search keywords，抓 dependency (include, import)，再 Global search keyword dependency 裡的 keywords，如此重覆\n", " * Language Server, Go to Definition, Find References, IDE features\n", " * Embedding：把每個 file 向量化\n", " * Agent（July 2026 目前最強）\n", " * 第一輪找 interface，看到它呼叫 PricingEngine\n", " * 第二輪送 PricingEngine，看到它呼叫 VolSurface\n", " * 第三輪送 VolSurface\n", " * 每一輪不需要很大的 context window\n", "* 只要抓的都是需要的，context window 不用很大也能給出很好的答案\n", "* Retrieval 是做在 IDE extension（Continue）上的，model 只負責看著 input tokens 吐出 output tokens\n", "* 現代 agent 系統裡，Retrieval 和模型是合作的\n", " * 模型思考需要哪些資訊 > tool calling > Retrieval 回傳結果 > 模型看完說，我還需要 VolSurface.cpp > 再做一次 Retrieval\n", " * Retrieval 負責找資料，模型決定要找什麼，怎麼用找到的資料\n", "* AI coding agent 能力 = agent + retrieval + model" ] }, { "cell_type": "markdown", "id": "ecd692dd-fa69-4eda-86d1-717ef5522ed5", "metadata": {}, "source": [ "## Agent\n", "\n", "* Agent 是一個 workflow，orchestrator\n", "* Agent = Retrieval + File Reader + Terminal + Git + Compiler + Browser + Model\n", "* Agent 會控管整個流程，例如如果模型連續十次要求讀同一個檔案，Agent 很可能會偵測到異常並停止，而不是無限照做\n", "* 現代 AI coding stack：Prompt > Agent > LLM > Tool Calls > Retrieval > Model 再推理 > 完成任務\n", "\n", "### Agent Example \n", "\n", "* User prompt: please fix the compile error\n", "* Agent sends\n", "```\n", "You are an autonomous coding assistant.\n", "\n", "You have these tools:\n", "\n", "1. search_code(...)\n", "2. read_file(...)\n", "3. edit_file(...)\n", "4. run_tests(...)\n", "5. run_terminal(...)\n", "\n", "Decide the next action.\n", "```\n", "* Model returns tool calling\n", "```json\n", "{\n", " \"tool\": \"search_code\",\n", " \"query\": \"PricingEngine\"\n", "}\n", "```\n", "* Agent runs `search_code(\"PricingEngine\")` and got `PricingEnging.cpp` and `PricingEnging.h`\n", "* Agent asks again\n", "```\n", "Search results:\n", "\n", "PricingEngine.cpp\n", "\n", "PricingEngine.h\n", "\n", "What's next?\n", "```\n", "* Model says\n", "```json\n", "{\n", " \"tool\":\"read_file\",\n", " \"path\":\"PricingEngine.cpp\"\n", "}\n", "```" ] }, { "cell_type": "markdown", "id": "50a64534-a0aa-4be9-829a-78d08800b228", "metadata": {}, "source": [ "## 模型記憶機制\n", "\n", "* 當長對話變長，不送整個歷史而是搜尋關鍵字，summarize 上下文，以 summary 代替整個歷史。這也是一種 Retrieval " ] }, { "cell_type": "markdown", "id": "2ede2da8-fe50-4538-9817-14c1aa234843", "metadata": {}, "source": [ "## Version 2\n", "\n", "* 不是 Can local AI run large code bases?\n", "* 是 Can an open-source coding stack support large code bases?\n", "\n", "| Stack | Large Codebase Navigation | Cross-file Edit | Unit Test | Build Fix | Speed |\n", "| ----------------- | ------------------------- | --------------- | --------- | --------- | ----- |\n", "| Continue + Ollama | ? | ? | ? | ? | ? |\n", "| Cline + Ollama | ? | ? | ? | ? | ? |\n", "| Roo Code + Ollama | ? | ? | ? | ? | ? |\n", "\n", "* Continue 是第一波真正做出完整 open-source AI coding assistant 生態系的代表\n", "\n", "| 能力 | Cursor（商業） | Open Source 現況 | 是否已接近 |\n", "| ------------------ | ---------- | ------------------- | --------- |\n", "| 本地模型支援 | v | v（Ollama 等） | 幾乎追上 |\n", "| Codebase Retrieval | v | v（Continue/Cline 等） | 接近，但仍需實測 |\n", "| Agent | v | v | 接近，但成熟度不一 |\n", "| Tab Completion | * 自研模型 | 有，但品質差異較大 | 尚有差距 |\n", "| 多 Agent 協作 | v | 開始出現 | 仍在快速演進 |\n", "| 整體使用體驗 | 非常成熟 | 工具整合程度不一 | 還有差距 |\n", "\n", "* Open Source 能不能以較低成本、較高隱私，提供足夠好的工程生產力？" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.4" } }, "nbformat": 4, "nbformat_minor": 5 }