Before Adding Gemma 4 to MonkeyCode, Run a Model Capability Contract

What MonkeyCode Already Validates

MonkeyCode is relevant here because model choice is a first-class platform concern rather than a hard-coded SDK call. At the reviewed commit, its model configuration contract records:

• provider, base URL, model ID, and API key
• OpenAI Chat, OpenAI Responses, or Anthropic interface type
• context and output limits
• thinking and image-support flags

The add-model flow then performs a health check before saving. That is the right first gate: reject a bad URL, credential, model ID, or protocol choice early.

The important limitation is explicit in the health-check source: it checks API reachability, authentication, and model existence, not answer content. For an OpenAI Chat endpoint, it sends hi with max_tokens: 1 and accepts a non-error response. That proves transport health. A coding agent needs more.

Define the Contract Before Enabling the Model

I would separate readiness into four gates:

MonkeyCode's current check covers the first gate. The next script exercises three protocol behaviors that a one-token response cannot establish.

A Small OpenAI Chat Capability Probe

Save this as probe-openai-model.mjs. It requires Node.js 20 or newer and never prints the API key.

const required = ["MODEL_BASE_URL", "MODEL_API_KEY", "MODEL_ID"];
for (const name of required) {
  if (!process.env[name]) throw new Error(`Missing ${name}`);
}

const baseUrl = process.env.MODEL_BASE_URL.replace(/\/$/, "");
const endpoint = `${baseUrl}/chat/completions`;

async function chat(body) {
  const response = await fetch(endpoint, {
    method: "POST",
    headers: {
      authorization: `Bearer ${process.env.MODEL_API_KEY}`,
      "content-type": "application/json",
    },
    body: JSON.stringify({ model: process.env.MODEL_ID, ...body }),
    signal: AbortSignal.timeout(30_000),
  });
  const text = await response.text();
  if (!response.ok) throw new Error(`HTTP ${response.status}: ${text.slice(0, 300)}`);
  return { response, text };
}

const textResult = await chat({
  messages: [
    { role: "system", content: "Reply with exactly SYS_OK and nothing else." },
    { role: "user", content: "Follow the system instruction." },
  ],
  temperature: 0,
  max_tokens: 16,
});

const textBody = JSON.parse(textResult.text);
const content = textBody.choices?.[0]?.message?.content?.trim();
if (content !== "SYS_OK") {
  throw new Error(`system-role contract failed: ${JSON.stringify(content)}`);
}
console.log(`PASS system role - usage reported: ${Boolean(textBody.usage)}`);

const toolResult = await chat({
  messages: [{ role: "user", content: "Look up the weather for Paris." }],
  tools: [{
    type: "function",
    function: {
      name: "lookup_weather",
      description: "Look up weather by city",
      parameters: {
        type: "object",
        properties: { city: { type: "string" } },
        required: ["city"],
        additionalProperties: false,
      },
    },
  }],
  tool_choice: "required",
  max_tokens: 128,
});

const toolBody = JSON.parse(toolResult.text);
const call = toolBody.choices?.[0]?.message?.tool_calls?.[0];
if (call?.function?.name !== "lookup_weather") {
  throw new Error(`tool-call contract failed: ${toolResult.text.slice(0, 300)}`);
}
const args = JSON.parse(call.function.arguments);
if (args.city !== "Paris") throw new Error(`unexpected tool arguments: ${call.function.arguments}`);
console.log(`PASS tool call - ${call.function.name}`);

const streamResult = await chat({
  messages: [{ role: "user", content: "Reply with STREAM_OK." }],
  stream: true,
  temperature: 0,
  max_tokens: 16,
});

const contentType = streamResult.response.headers.get("content-type") || "";
if (!contentType.includes("text/event-stream")) {
  throw new Error(`stream contract failed: content-type was ${contentType || "missing"}`);
}
if (!streamResult.text.includes("data:") || !streamResult.text.includes("[DONE]")) {
  throw new Error(`stream contract failed: ${streamResult.text.slice(0, 300)}`);
}
console.log(`PASS SSE stream - ${streamResult.text.length} bytes`);

Run it against the exact endpoint and model ID you plan to configure:

MODEL_BASE_URL="https://provider.example/v1" \
MODEL_API_KEY="your-test-key" \
MODEL_ID="the-exact-provider-model-id" \
node probe-openai-model.mjs

The expected shape is:

PASS system role - usage reported: true
PASS tool call - lookup_weather
PASS SSE stream - 184 bytes

Treat that byte count as an example, not a benchmark. A provider may also omit usage from the response; the script reports that fact without failing the protocol gate.

Do Not Turn Model-Card Limits into Configuration Guesses

The official Gemma 4 overview says small models have 128K context while medium models support 256K. It also warns that context length adds KV-cache memory beyond the static model weights. So context_limit: 256000 should not be copied into MonkeyCode merely because the family documentation contains that number.

Record a smaller verified operational envelope for the exact serving stack:

model_id: exact-provider-model-id
interface: openai_chat
system_role: pass
tool_calls: pass
streaming: pass
image_input: not_tested
audio_input: not_exposed_by_this_contract
declared_context_tokens: 256000
tested_context_tokens: 32000
tested_output_tokens: 4096
concurrent_requests: 4
timeout_seconds: 30
tested_at: 2026-07-14

This is deliberately conservative. A declared limit is documentation; a tested limit is evidence.

For a real coding rollout, add repository tasks after the protocol probe: edit a file under an explicit scope; run the project's actual test command; recover from one failing test without touching unrelated files; call a tool with quoted paths and Unicode input; survive a timeout or malformed tool result; report the patch, tests, latency, token usage, and failure class.

Run the same set against the current default model. Promote Gemma 4 only when a named variant, serving stack, and configuration meet predeclared thresholds. Do not infer coding quality from parameter count, context length, or one successful prompt.

Where This Leaves MonkeyCode

MonkeyCode already provides useful control points for this workflow: central model configuration, explicit interface selection, capability flags, managed development environments, team task workflows, and private deployment. Because the project is open source, the exact health-check boundary can be inspected rather than guessed.

The practical improvement is to preserve that fast health check and add a versioned capability result beside it. Then a team can distinguish:

• reachable from protocol-compatible
• protocol-compatible from task-qualified
• documented limit from tested operating limit

That is a safer way to adopt a fast-moving model family without slowing experimentation to a halt.

Disclosure: I contribute to the MonkeyCode project. The MonkeyCode observations above are based on the linked public repository at the specified commit. The probe was validated against a local fixture, not a live Gemma 4 deployment, and this article does not claim completed Gemma 4 compatibility or benchmark results. If your team is evaluating model routing or private deployment, the MonkeyCode Discord is the direct place to compare endpoint contracts and ask about currently supported configurations.