Local AI

Fitting a model into a browser tab

GrowingEnglish1 min read

Open in the cabin →

Sample note, replace me.

Part one argued for running the model on the device. The first practical obstacle is size: the weights have to travel over the network and then fit in memory next to everything else the tab is doing.

Quantisation

Weights are usually trained as 32-bit floats. Storing them with fewer bits makes the file smaller at some cost in quality. The numbers below are illustrative for a model with one billion parameters:

PrecisionBits per weightApproximate size
float32324 GB
float16162 GB
8-bit81 GB
4-bit40.5 GB

Four bits is a common sweet spot for language models; below that, quality tends to fall off quickly.

Ask the device first

Never assume a GPU. Check, then choose a model, then fall back:

async function pickBackend(): Promise<'webgpu' | 'wasm'> {
  const gpu = (navigator as any).gpu;
  if (!gpu) return 'wasm';
  const adapter = await gpu.requestAdapter();
  return adapter ? 'webgpu' : 'wasm';
}

On the WebAssembly path, pick a smaller model rather than making the user wait.

A budget, written down

Decide the budget before choosing the model, not after:

  • download: what a visitor will tolerate on their connection;
  • memory: well under what a mid-range phone can spare;
  • time to first result: seconds, not minutes.

If the model does not fit the budget, the model changes. The budget does not.

Next: Shipping a local model to real people.

Tags: on-device, quantisation, webgpu

Linked from

  • Shipping a local model to real people

    Caching the weights, showing honest progress and keeping the page alive while the model thinks. Part three of three.

    SeedlingEnglish1 min readA local model in the browser · 3

  • Why run the model on the device

    Running a model on the user's own device removes the server, the bill and the privacy question in one move. Part one of three.

    EvergreenEnglish1 min readA local model in the browser · 1