Local AI

Shipping a local model to real people

SeedlingEnglish1 min read

Open in the cabin →

Sample note, replace me.

A demo works on the author's laptop. A product works on a stranger's phone, on hotel Wi-Fi, the second time they visit. This last part of the series that began with Why run the model on the device is a checklist for that gap.

Download once

Store the weights in the Cache API so the second visit is instant:

async function fetchCached(url: string): Promise<Response> {
  const cache = await caches.open('models-v1');
  const hit = await cache.match(url);
  if (hit) return hit;
  const res = await fetch(url);
  if (res.ok) await cache.put(url, res.clone());
  return res;
}

Put a version in the cache name. When the model changes, delete the old cache instead of leaving a gigabyte behind.

Be honest about the wait

Show the size before the download starts and let the user decide. Show real progress in megabytes, not a spinner. If the device cannot run the model, say so plainly and offer the smaller one from part two.

Stay off the main thread

Run inference in a Web Worker. A page that freezes while it thinks feels broken, even when it is fast. Stream partial results back so there is something to read within the first second.

Still to write

  • Measuring quality after quantisation without a test lab.
  • What to do when the tab is in the background and the browser reclaims the GPU.

Tags: on-device, caching, ux

Linked from

  • Fitting a model into a browser tab

    Quantisation, a memory budget and a fallback path. Part two of three.

    GrowingEnglish1 min readA local model in the browser · 2

  • Why run the model on the device

    Running a model on the user's own device removes the server, the bill and the privacy question in one move. Part one of three.

    EvergreenEnglish1 min readA local model in the browser · 1