Local AI

Why run the model on the device

Evergreenupdated English1 min read

Open in the cabin →

Sample note, replace me.

Most AI features are a text box wired to somebody else's data centre. That is a fine place to start, and a strange place to stop. A laptop or a phone bought in the last few years can run a useful model by itself.

What you gain

  • Privacy by construction. The text never leaves the device, so there is nothing to promise and nothing to leak.
  • No bill per request. The user brings the hardware. A hobby project can stay free for ever.
  • It works offline. On a train, on a plane, in a cabin.
  • Latency you control. No queue, no cold start on someone else's cluster.

What you give up

The largest models do not fit, and the ones that do are less capable. The first visit includes a large download. Devices differ wildly, so the same page can be instant on one machine and unusable on another.

When it is the right choice

Local models shine at narrow, well-defined jobs: transcribing speech, embedding text for search, classifying, summarising a page the user is already looking at. They struggle when the task needs broad knowledge or long reasoning.

A good test: would the feature still be worth having if it were only ninety percent as good, but private and free? If so, run it locally.

Next: Fitting a model into a browser tab looks at getting a model small enough to download, and Shipping a local model to real people at the unglamorous parts of putting it in front of people. There is a shorter Turkish take in Cihazda çalışan yapay zekâ.

Tags: on-device, privacy, webgpu

Linked from

  • Cihazda çalışan yapay zekâ

    Model kendi cihazınızda çalıştığında veriniz sizde kalır, fatura gelmez ve internet kesilse bile iş görür.

    BüyüyorTürkçe1 dk okuma

  • Shipping a local model to real people

    Caching the weights, showing honest progress and keeping the page alive while the model thinks. Part three of three.

    SeedlingEnglish1 min readA local model in the browser · 3

  • Fitting a model into a browser tab

    Quantisation, a memory budget and a fallback path. Part two of three.

    GrowingEnglish1 min readA local model in the browser · 2