In development

Local inference for coding agents, designed for Apple M5 Max.

tokspur is a native inference engine, still in development, for developers who run coding agents on their own Mac. It is written for this project in C++, Objective-C++ and Metal, and its aim is less time to a correct, tested code change, through long sessions, with memory left for your other work. The only target so far is an Apple M5 Max with a 40-core GPU and 128 GiB of unified memory.

Explore the design Development status

02 Coding focus

Correct, sooner

A coding agent works in turns: it reads files, proposes an edit, runs the tests, reads the output and tries again. tokspur is designed around that loop. The aims below are development goals, and they stay labelled that way until each one is implemented and measured.

  1. Development goal

    Responsive turns

    Each turn should start streaming quickly and hold its pace late into a long session.

    Approach Keep one model loaded and admit one interactive request at a time from a bounded queue. A cancelled request releases the GPU before the next one starts.

  2. Development goal

    Long context, reused

    Agents often go back: they edit an earlier tool result, retry a step or branch the conversation. Those turns should not pay to process the whole conversation again.

    Approach Save model state at message boundaries and restore the newest state before the change, so only the changed part is processed.

  3. Development goal

    Memory for your work

    The model should leave enough unified memory for macOS, Xcode, a browser and your builds.

    Approach Keep one memory ledger for weights, caches and state under a fixed ceiling, initially 96 GiB on a 128 GiB Mac. Under memory pressure, drop optional caches first, then refuse new work with a clear error rather than swap.

03 Runtime design

Propose, verify, commit

The tokspur engine is one native executable, built with Apple’s toolchain, that reads existing model weights and calls no other inference engine. Parts of it are finished, the complete model runs experimentally and the rest is still to be written, so every part below carries its status.

Engine layout

  1. Coding agent

    An agent in your editor or terminal, calling a local API

  2. tokspur engine one native executable

  3. Apple M5 Max

    40-core GPU, 128 GiB unified memory, Metal and Apple system frameworks

Comparison control not shipped

Built on other inference engines, it runs on the target Mac as a measurement baseline, apart from the engine.

  1. 1Full model runs, experimentalNot yet qualified

    Owned native engine

    Each layer, from reading model files to the GPU kernels, can be tuned for one chip and one memory budget, and the whole path can be audited.

    The engine is C++, Objective-C++ and Metal, built with Apple’s toolchain, and the current binary links only Apple system libraries. Implemented: a bounds-checked model file reader, the tokenizer with its own Unicode normalisation, the chat renderer, Metal operators for the whole text model and sampling, each checked against independent references. The complete model now runs on short test cases.

  2. 2Appending worksEdits and branches planned

    Context reuse

    Going back in a conversation, to edit a tool result or try another branch, should cost little more than the part that changed.

    Appending a tool result already reuses the committed prefix without processing it again. For edits and branches, the engine will save model state at message and tool boundaries, keep a few recent states within a byte budget and restore the newest one before the change, so only the tokens after that point are processed.

  3. 3Planned

    Verified speculation

    Several tokens can be accepted in one step, and every token is checked by the main model before it is kept.

    A lookup into text already in the conversation, or a small draft model, proposes a short run of tokens. The main model checks the run in one pass, keeps the tokens it agrees with and supplies its own at the first disagreement. A measured policy picks the cheaper source each round, or plain decoding when proposals do not pay.

The comparison control is built on Splash (native core) and oMLX (management and API layer). It is not part of the tokspur engine, and neither project endorses tokspur.

Illustration

One verification step (planned)

  1. 1Reuse. Earlier context is restored from saved state.
  2. 2Propose. The draft model suggests four tokens.
  3. 3Verify. The main model checks all four in one pass.
  4. 4Commit. Three are kept and the main model replaces the fourth.
  • Reused
  • Proposed
  • Verified
  • Rejected
  • Main model token
  • Commit point
Illustration of planned engine behaviour. The tokens and timing are invented and are not measurements.

04 Measurement

Correct first

Performance is the point of tokspur, and it only counts when the work is right. Every result will report these four measures, in this order.

Benchmarks pending tokspur has no qualified performance results yet. Development timings from engine experiments and diagnostics from the comparison control guide the work, but they are not qualified, so this page leaves them out.

  1. 1

    Correct task completion

    Coding tasks finished with their tests passing. The agent cannot edit the tests, and failed attempts count as failures.

    Pending

  2. 2

    Time to a correct result

    Wall time from the first request to a correct, tested change, including tool and build time.

    Pending

  3. 3

    First-token latency

    Time from a request to the first streamed token, measured cold and with reused context.

    Pending

  4. 4

    Memory use

    Peak unified memory used by the engine, and any swap growth during the run.

    Pending

Release targets

What the first release has to show. None of these targets has been met yet.

  • At least 10 per cent lower median time to a correct result than the local setup it replaces, beyond run-to-run noise.
  • At least 25 per cent lower first-token latency after a conversation is edited or branched, with no more than 5 per cent added p95 latency elsewhere.
  • No new memory pressure or swap within the chosen memory budget.
  • Comparisons with registered competing runtimes, with losses and trade-offs published.

Result record

Each published result will carry these fields, so a number can be traced to the machine, model and work that produced it.

Runs will use matched settings in alternating order, keep failed and truncated attempts in the record, and state the trade-offs beside any gain.

Hardware
Chip, GPU cores, unified memory, macOS version and power mode
Pending
Model
Name, revision and quantisation
Pending
Configuration
tokspur version, context size, proposal source and memory ceiling
Pending
Workload
Task set, attempts per task, input sizes and cache state
Pending
Correctness
Tasks completed correctly, with every failure listed
Pending
Timing
Time to a correct result and first-token latency, as medians with intervals
Pending
Memory
Peak engine memory and swap growth
Pending

05 Project status

Progress

The complete model runs experimentally on the engine’s own code. Tuning has started, the later stages are planned, and none of them has a date.

Current stage

Full model, experimental

The engine’s own code now runs the complete text model on the target Mac, with its own tokenizer, chat renderer, Metal operators and sampling, checked against independent references on short test cases. It is not yet ready for coding work: correct results on real coding tasks, the local API and performance qualification are open. A comparison control built on other engines runs separately as a measurement baseline. There is no public release, and tokspur benchmarks are pending.

Checked

  1. Done

    Plan and comparison control

    Engine brief written, and a runnable comparison control with a recorded baseline.

  2. Done

    Native foundations

    Model reader, tokenizer, chat renderer, text protocol and Metal operators, each checked against independent references.

  3. Current

    Run the full model

    The complete text model runs experimentally with sampling. Correct results on real coding tasks and a local API come next.

  4. Started

    M5 Max tuning

    Opt-in kernel experiments guided by profiling, kept out of the defaults until qualified.

  5. Planned

    Context reuse and speculation

    Saved state for edited and branched conversations, prompt lookup and measured draft selection.

  6. Planned

    Qualification

    Coding task comparisons, registered competitor runs and operational tests against the release targets.

  7. Planned

    Package and release

    A standalone package with a full dependency audit.

Repository and documents

The repository and its design documents are private for now. Their links appear here once each destination is public.

  • RepositoryPrivate for now
  • Implementation planPrivate for now
  • Evidence summaryPrivate for now