Swarms (MoTA)

A swarm (MoTA, Mixture of Tuned Adapters) is many tiny specialists answering at the same time on one small model. Each specialist is a LoRA adapter, a small add-on file tuned for one job, and Gerbil runs them all together in a single batched pass over one shared base model. One model in GPU memory, a few megabytes per specialist, and every specialist gives the same answers it would give running alone (verified token-exact).

createSwarm(): members, plan, reduce

The highest-level primitive. Declare named members (LoRA adapters on one shared base, or null for the bare base), an optional plan that fans one input out into per-member sub-tasks, and an optional reduce that joins the results. swarm.run() decodes every sub-task concurrently in one batched pass, so the fan-out costs about the slowest member, not the sum:

01import { WebGPUEngine } from "@tryhamster/gerbil/gpu";
02
03const engine = await WebGPUEngine.create({
04  repo: "mlx-community/Qwen3.5-0.8B-4bit", // the shared base, loads ONCE
05});
06
07const swarm = await engine.createSwarm({
08  members: {
09    classifier: "hf:acme/ticket-classifier-lora",
10    redactor: "hf:acme/pii-redactor-lora",
11    responder: null, // bare base, no adapter
12  },
13  // Optional: input -> sub-tasks. Default: broadcast the input to every member.
14  plan: (ticket, members) =>
15    members.map((member) => ({ member, prompt: ticket })),
16  // Optional: joined results -> answer. Default: member-keyed map of results.
17  reduce: (results) =>
18    Object.fromEntries(results.map(({ member, result }) => [member, result.text])),
19});
20
21// One ticket -> all specialists, ONE batched pass
22const record = await swarm.run(ticket, { sampling: { temperature: 0 } });
23
24// Or route one prompt to one member (still batches with concurrent work)
25const reply = await swarm.generate("responder", ticket, { maxTokens: 96 });

Each sub-task can carry per-member options (system prompt, token budget) via the plan's options field. A runnable end-to-end example, a four-specialist support-triage swarm, ships with the engine as scripts/engine/example-support-swarm.mjs.

createScheduler(): continuous batching

Underneath the swarm sits a continuous-batching scheduler. Submit any number of requests with independent prompts, budgets, and adapters; the scheduler multiplexes them over the batch lanes with dynamic admission and immediate lane retirement. No fixed batch width:

01const scheduler = engine.createScheduler();
02
03// Submit as many requests as you like, whenever you like.
04const a = scheduler.submit("Route this ticket: my invoice is wrong", {
05  adapter: "support-router", // registered adapter name; omit for bare base
06  maxTokens: 32,
07});
08const b = scheduler.submit("Draft a reply for: where is my order?", {
09  systemPrompt: "You are a friendly support agent.",
10  maxTokens: 128,
11  onToken: (piece) => process.stdout.write(piece), // streaming
12});
13
14const [routed, reply] = await Promise.all([a, b]);
15// Each result includes text, tokensGenerated, finishReason,
16// and serving timings: ttftMs, e2eMs.

Per-request output is token-exact against a single-sequence generate() call with the same adapter. Concurrency changes throughput, never output.

registerAdapter(name, source)

Registers a named LoRA adapter for batched decode. The adapter's low-rank factors are fetched and packed into shared GPU factor buffers once; after that, any number of lanes can decode with it concurrently by name. Registering the same name twice is a no-op. createSwarm calls this for you for every member.

01import { WebGPUEngine } from "@tryhamster/gerbil/gpu";
02
03const engine = await WebGPUEngine.create({
04  repo: "mlx-community/Qwen3.5-0.8B-4bit", // the shared base
05});
06
07// Fetch + pack factors into shared GPU buffers. Once per adapter.
08await engine.registerAdapter("support-router", "hf:acme/support-router-lora");
09await engine.registerAdapter("pii-redactor", "hf:acme/pii-redactor-lora");
10
11engine.getRegisteredAdapters(); // ["support-router", "pii-redactor"]

• ·Sources are the same as the rest of the adapter system: a Hugging Face repo (hf:owner/repo), an https URL, a local path (file:/abs/dir), or a Gerbil Tune output. Private repos honor the engine's hfToken.
• ·Auto-enables batching. The first call switches the engine onto the batch path if it isn't on yet. No flags needed.
• ·Distinct from the runtime overlay. loadAdapter hot-swaps one adapter onto the warm base for single-sequence generation (one specialist at a time, serialized). registerAdapter is the concurrent path: many specialists resident at once, selected per request.

Per-lane adapters on generateBatch

For a fixed set of prompts, generateBatch decodes N prompts in lockstep: one batched dispatch stream per step, one token per lane. The adapters option assigns each lane a registered adapter by name; null lanes decode the bare base. Mixed lanes, with different adapters and no-adapter lanes in the same batch, are the intended shape.

01const results = await engine.generateBatch(
02  [
03    "Route this ticket: my invoice is wrong and I was double charged",
04    "Draft a reply for: where is my order?",
05    "Redact PII: John Smith, john@acme.com, +1 555 0100, ...",
06    "Summarize: ...",
07  ],
08  {
09    // adapters[i] selects lane i's specialist; null = bare base.
10    adapters: ["support-router", null, "pii-redactor", null],
11    maxTokens: 256,
12    sampling: { temperature: 0 }, // batched decode is greedy-only today
13  },
14);
15
16// results[i] is a normal GenerateResult for lane i.

The correctness contract is the load-bearing part: a lane running adapter A is token-exact against single-sequence generation with adapter A applied. Lanes never interact; the base GEMM stays shared across the batch and each lane adds only its own gathered low-rank correction. Concurrency changes throughput, never output.

Defaults & kill switches

Batching is on by default: calling any batch-shaped API (createSwarm, createScheduler, generateBatch, registerAdapter) is the opt-in. Environment variables exist only to override or disable:

• ·GERBIL_BATCH=N overrides the lane count (default: 4, or the prompt count for generateBatch).
• ·GERBIL_BATCH=0 or GERBIL_PAGED_KV=0 disable the batch path entirely.
• ·GERBIL_BATCH_LORA=0 disables the per-lane adapter machinery (batching still works, bare base only).

Current limits

• ·Node (Dawn) only. Batched decode ships on the Node backend today. The kernels are the same WGSL the browser runs, but browser batch widths (especially phone-class WebKit) are unmeasured. Browser support is coming.
• ·Greedy-only. Pass sampling: { temperature: 0 } (or omit it). Sampled batched decode comes later.
• ·Fixed width on generateBatch. generateBatch sizes the batch to its first call's prompt count and then requires that many prompts; use maxTokensPerRow for per-lane budgets. The scheduler has no such restriction, submit any number of requests.
• ·Text models only. Multimodal models, PLE models, and engines with a runtime-overlay adapter (loadAdapter) are rejected on the batch path for now.
• ·Specialists are pre-tuned. There is no per-request training, no gradients in the serving path, and no “learns as it goes”. At request time the engine only selects among resident adapters.

What's coming

• ·Request groups. submitGroup / cancelGroup group many lanes into one logical task: joint cancellation, per-task stats, and (later) shared-prefix KV reuse across a fan-out.
• ·Sampled batched decode. Temperature and top-k/top-p sampling on the batch path.
• ·Browser batching. The same swarm APIs on the in-browser engine.

When to reach for it

Swarms fit decomposable tasks with production shape: classification cascades and intent routing, multi-aspect extraction, map-reduce over documents, ensemble voting, and agentic fan-out. They are not a substitute for a frontier model on open-ended novel reasoning: a 0.8B base with rank-16 adapters doesn't become GPT-class because thirty-two of it run at once. For tasks that decompose, though, tuned specialists on one device are faster, cheaper, private, and offline.

To build your first specialist, start with Gerbil Tune. Every Tune job outputs an adapter on a shared base, which is exactly the artifact registerAdapter takes. For single-adapter usage without batching, see Adapters; for engine sharing and memory budgeting, see Concurrency & Memory.