Running Kimi K3 on a M1 Max Uh oh! There was an error while | Coderz Club

Running Kimi K3 on a M1 Max

Uh oh! There was an error while loading. Please reload this page. gavamedia / deltafin Public Notifications You must be signed in to change notification settings Fork 7 Star 109 mainBranchesTagsGo to fileCodeOpen more actions menuFolders and filesNameNameLast commit messageLast commit dateLatest commit History9 Commits9 Commitsresearch/studyresearch/study toolstools .gitignore.gitignore BRAINSTORM-SPEED.mdBRAINSTORM-SPEED.md LICENSELICENSE PLAN.mdPLAN.md README.mdREADME.md View all filesRepository files navigation ____ _ _ __ _ | _ \ ___| | |_ __ _ / _(_)_ __ | | | |/ _ \ | __/ _` | |_| | '_ \ | |_| | __/ | || (_| | _| | | | | |____/ \___|_|\__\__,_|_| |_|_| |_| An experiment in running Kimi K3 (2.8T parameters) on one Apple Silicon Mac Deltafin is a small research project that runs a Mixture-of-Experts model far larger than the machine it sits on. It is not fast — about 16 seconds per token on our M1 Max — but it is exact, reproducible, and it works on a 64 GB laptop. Newer chips and more RAM make it faster automatically. Install Three commands, then you're generating. The only real decision is step 3. # 1. environment (Python 3.12+, and Xcode CLT for clang) python3 -m venv venv ./venv/bin/pip install torch numpy safetensors tiktoken ml_dtypes blobfile \ "transformers==4.56.2" einops tokenizers # 2. build the fused MXFP4 kernel clang -O3 -mcpu=native -shared -DNO_MAIN -o tools/libmxfp4gemv.dylib tools/fused_gemv.c # 3. download the model (see the two modes below) ./venv/bin/python tools/setup_k3.py --full The two modes --full (recommended) --stream Disk needed ~1.7 TB ~215 GB Download time 5–10 hours, resumable ~30 minutes Speed afterwards ~60–76 s/token, every prompt ~3+ min/token for anything not already cached Network at inference none constant Every token reads 16 experts × 92 layers = 25.8 GB of expert data. From local disk that's about 4 seconds; over the network it's minutes. That single fact is the whole difference between the two columns. Run setup_k3.py with no flag and it picks --full when the disk allows, otherwise falls back to streaming and tells you exactly how much space you'd need to free. Starting with streaming and upgrading later Streaming is a fine way to try Deltafin without committing 1.7 TB. Whenever you want the speed, one command finishes the job — no reinstall, no reconfiguration, and it picks up whatever is already cached: ./venv/bin/python tools/fetch_experts_all.py # resumable, run anytime ./venv/bin/python tools/fetch_experts_all.py --dry-run # just show the numbers ./venv/bin/python tools/fetch_experts_all.py --layers 1-40 # partial is fine too Deltafin prints a reminder at startup — both for the CLI and the API server — whenever it's still in streaming mode, showing how much of the pool is local and what finishing would cost. Optional: int8 spine Halves per-token I/O for the non-expert weights, with no meaningful quality change in our checks. Takes a few minutes: ./venv/bin/python tools/convert_spine_int8.py Usage # ask a question; generates until the model finishes its answer ./venv/bin/python tools/kimi_run.py --chat --prompt "What are the three largest moons of Saturn?" # raw completion (no chat template); runs until you press Ctrl-C, or cap it ./venv/bin/python tools/kimi_run.py --prompt "The capital of France is" --max-new 16 Tokens print as they are generated, so you always see the text as it comes. Ctrl-C stops cleanly at any point and prints the result so far; --max-new N caps the length. One honest warning: K3 thinks before it answers, and at about a token per minute a full chat answer can take a while — watching it stream is part of the experience. Router selections are logged to router_trace.jsonl if you want to study K3's routing behaviour. OpenAI-compatible server Deltafin can serve the standard OpenAI API, so chat interfaces, the openai SDK and coding agents can use it by changing a base URL: ./venv/bin/python tools/serve_openai.py --port 8000 curl http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' \ -d '{"model": "deltafin-kimi-k3", "messages": [{"role": "user", "content": "Hello!"}]}' from openai import OpenAI client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="none") r = client.chat.completions.create( model="deltafin-kimi-k3", messages=[{"role": "user", "content": "Hello!"}]) print(r.choices[0].message.content) # the answer print(r.choices[0].message.reasoning_content) # K3's thinking, when present /v1/chat/completions, /v1/completions and /v1/models are implemented, and streaming ("stream": true) works. Most tools that read OPENAI_BASE_URL and OPENAI_API_KEY will work by pointing those at the server. Please read these caveats before pointing anything automated at it: Time. Answers arrive when they arrive — set your client's timeouts to hours, not seconds. Omitting max_tokens lets the model finish its answer (recommended); raw completions, which never end on their own, default to 256. Operat