GerbilGerbil

Vision

Gerbil can see. Run Vision Language Models locally to describe, analyze, and answer questions about images — entirely on your device, with no image ever leaving it.

Quick Start

Vision runs natively on the WebGPUEngine using Qwen3.5-0.8B's own built-in ViT tower. Build the engine with enableVision: true, decode an image to RGB pixels, and call describeImage:

vision-basic.ts
01import { WebGPUEngine } from "@tryhamster/gerbil/gpu";
02

03// The vision tower (ViT, ~192 MB) ships inside the Qwen3.5 4-bit repo.
04const engine = await WebGPUEngine.create({
05  repo: "mlx-community/Qwen3.5-0.8B-4bit",
06  enableVision: true,
07});
08

09// describeImage takes pre-decoded RGB pixels (see "Image Input" below).
10const result = await engine.describeImage(image, "What's in this image?");
11

12console.log(result.text);
13// → "A golden retriever playing fetch in a sunny park..."
14

15engine.destroy();

Vision Models

Currently supported vision-capable models:

ModelParametersContextFeaturesSize
qwen3.5-0.8b0.8B (own ViT)262K tokensVision + Reasoning~404MB (4-bit, +192MB ViT)
Note: Vision runs natively on the WebGPUEngine using the compact mlx-community/Qwen3.5-0.8B-4bit repo (~404MB, Qwen3.5's own ViT included) — see React Hooks below. The same qwen3.5-0.8b model runs on-device in Node.

Image Input

describeImage accepts pre-decoded RGB pixels (HWC, 0..255), so you decode any <img>, File, or URL via a canvas first:

image-input.ts
// describeImage accepts either decoded pixels...
type PixelImage = {
  pixels: Float32Array | Uint8ClampedArray | Uint8Array;
  width: number;
  height: number;
};


// ...or pre-extracted ViT patches (advanced):
type PatchImage = {
  patches: Float32Array;
  gridTHW: [number, number, number];
};


// Decode an <img>/File/URL to RGB pixels via a canvas:
async function decodeToPixels(src: string): Promise<PixelImage> {
  const img = new Image();
  img.crossOrigin = "anonymous";
  await new Promise((res, rej) => { img.onload = res; img.onerror = rej; img.src = src; });
  const canvas = document.createElement("canvas");
  canvas.width = img.naturalWidth;
  canvas.height = img.naturalHeight;
  const ctx = canvas.getContext("2d")!;
  ctx.drawImage(img, 0, 0);
  const { data } = ctx.getImageData(0, 0, canvas.width, canvas.height);
  const rgb = new Uint8ClampedArray(canvas.width * canvas.height * 3);
  for (let i = 0, j = 0; i < data.length; i += 4, j += 3) {
    rgb[j] = data[i]; rgb[j + 1] = data[i + 1]; rgb[j + 2] = data[i + 2];
  }
  return { pixels: rgb, width: canvas.width, height: canvas.height };
}

Capability Detection

Check whether a vision encoder was built with the hasVision flag:

capability-detection.ts
const engine = await WebGPUEngine.create({
  repo: "mlx-community/Qwen3.5-0.8B-4bit",
  enableVision: true,
});


if (engine.hasVision) {
  const { text } = await engine.describeImage(image, "Describe this");
} else {
  // Engine was created without enableVision — describeImage() would throw.
  const { text } = await engine.generate("Describe what you know about...");
}

Vision Encoder Only

To get raw image-embedding tokens (e.g. for retrieval) without decoding text, call encodeImage with ViT patches and a temporal/height/width grid:

encode-image.ts
const { embeds, rows, dim } = await engine.encodeImage(patches, gridTHW);
// dim === 1024 (Qwen3.5's merged image-embedding dimension)

React Hooks

In the browser, vision runs natively on WebGPU via the WebGPUEngine. Decode the image to RGB pixels and call describeImage:

native-vision.ts
01import { WebGPUEngine } from "@tryhamster/gerbil/gpu";
02

03// The vision tower (ViT) ships inside the lean Qwen3.5 4-bit repo.
04const engine = await WebGPUEngine.create({
05  repo: "mlx-community/Qwen3.5-0.8B-4bit",
06  enableVision: true,
07});
08

09// Decode any <img>/File/URL to RGB pixels (HWC, 0..255) via a canvas.
10async function decodeToPixels(src: string) {
11  const img = new Image();
12  img.crossOrigin = "anonymous";
13  await new Promise((res, rej) => {
14    img.onload = res; img.onerror = rej; img.src = src;
15  });
16  const canvas = document.createElement("canvas");
17  canvas.width = img.naturalWidth;
18  canvas.height = img.naturalHeight;
19  const ctx = canvas.getContext("2d")!;
20  ctx.drawImage(img, 0, 0);
21  const { data } = ctx.getImageData(0, 0, canvas.width, canvas.height);
22  const rgb = new Uint8ClampedArray(canvas.width * canvas.height * 3);
23  for (let i = 0, j = 0; i < data.length; i += 4, j += 3) {
24    rgb[j] = data[i]; rgb[j + 1] = data[i + 1]; rgb[j + 2] = data[i + 2];
25  }
26  return { pixels: rgb, width: canvas.width, height: canvas.height };
27}
28

29const image = await decodeToPixels("/skelly.png");
30const { text } = await engine.describeImage(image, "What's in this image?");
31console.log(text);

Performance Tips

Image Size

  • —Larger images take longer to process
  • —Consider resizing to 512×512 to 1024×1024 for optimal performance
  • —Models cache in IndexedDB (browser) after first download

Expected Performance

MetricValue
Vision model load time~2s (cached)
Image processing~0.5s
Generation speed70-100+ tok/s in-browser (desktop WebGPU)
Memory usage~4GB (model + KV cache)

Troubleshooting

"Engine was not created with enableVision"

describeImage and encodeImage throw unless the engine was built with enableVision: true. The ViT tower adds ~192 MB, so it is only built on request.

Slow image processing

  • —ViT attention memory scales with patch-count², so downscale large photos before decoding (the longest side ≤ 448–1024 px works well)
  • —On WebKit/iOS, lower maxVisionPatches (default 1024 on WebKit, 4096 desktop)
  • —The native engine is WebGPU-only — there is no CPU fallback. Confirm a WebGPU adapter is available up front

Image not loading

  • —Check the URL is accessible (CORS may block some URLs)
  • —For local files, ensure the path is absolute
  • —Base64 data URIs must include the mime type prefix: data:image/png;base64,...

API Reference

describeImage()

describeImage.ts
// Image-in → text-out. Throws unless created with { enableVision: true }.
describeImage(
  image:
    | { pixels: Float32Array | Uint8ClampedArray | Uint8Array; width: number; height: number }
    | { patches: Float32Array; gridTHW: [number, number, number] },
  prompt?: string,            // default "Describe this image."
  options?: GenerateOptions & { imageProcessor?: ImageProcessorConfig }
): Promise<GenerateResult>

encodeImage()

encodeImage.ts
// Vision encoder only (patches → merged image-embedding tokens).
encodeImage(
  patches: Float32Array,
  gridTHW: [number, number, number]
): Promise<{ embeds: Float32Array; rows: number; dim: number }>
// dim === 1024 for Qwen3.5

hasVision

hasVision.ts
// true if the engine was created with { enableVision: true }
get hasVision(): boolean

WebGPUEngineOptions (vision fields)

WebGPUEngineOptions.ts
interface WebGPUEngineOptions {
  repo: string;              // e.g. "mlx-community/Qwen3.5-0.8B-4bit"


  /** Build the Qwen3.5 ViT tower (~192 MB) so describeImage() works. */
  enableVision?: boolean;


  /** Max ViT patches. Default 4096 desktop, 1024 on WebKit/iOS. */
  maxVisionPatches?: number;
}