Update model sizes for github-copilot#810
Conversation
|
I just realized the screenshot from vscode shows input and output sizes but all copilot models are defined in terms of context size and output size (no input) Don't really know if it's correct feel free to close this PR if it's a totally wrong approach |
|
There is a concept of input actually |
|
Look at openai gpt-5 variant models (from openai provider) |
|
Here is the models from copilot |
|
The input and output values from the json don't match the ones showed in vscode. The gemini models have 109k input size in vscode and 128k in the json I'm not entirely sure which ones are correct. I've added context sizes from the json but kept input and output sizes from vscode |
Add context size, sync input and output sizes with vscode
|
I think there is a token counting bug in OC, going to be doing some investigation/fixes there and then merge this prolly |
| [limit] | ||
| context = 128_000 | ||
| output = 16_000 | ||
| context = 200_000 |
There was a problem hiding this comment.
where is this 200k coming from?
In their models api I see:
"max_context_window_tokens": 144000,
"max_non_streaming_output_tokens": 16000,
"max_output_tokens": 32000,
"max_prompt_tokens": 128000,
So I would expect:
context: 144k
input: 128k
output: 32k
There was a problem hiding this comment.
Same goes for a lot of the other values in here, I can update them all myself but im wondering the logic here maybe im missing soemthing
There was a problem hiding this comment.
I trusted the json in the comment above. If you have a more trustworthy source or a way to update these values programmatically I think it’s better to use it instead
It would also help with opus 4.6 fast and gpt 5.3 codex (even though the rollout is paused)
There was a problem hiding this comment.
Nevermind the gpt 5.3 part, it’s now generally available:
https://github.blog/changelog/2026-02-09-gpt-5-3-codex-is-now-generally-available-for-github-copilot/
There was a problem hiding this comment.
I took a peak inside the vscode extension and it doesn't seem to have context size only "maxInputTokens" and "maxOutputTokens" fields. I can post the json of models available to my account if needed but if you have a more reliable way you should use it instead
There was a problem hiding this comment.
Ex:
{
"capabilities": {
"family": "claude-haiku-4.5",
"limits": {
"max_context_window_tokens": 144000,
"max_non_streaming_output_tokens": 16000,
"max_output_tokens": 32000,
"max_prompt_tokens": 128000,
"vision": {
"max_prompt_image_size": 3145728,
"max_prompt_images": 5,
"supported_media_types": [
"image/jpeg",
"image/png",
"image/webp"
]
}
},
"object": "model_capabilities",
"supports": {
"max_thinking_budget": 32000,
"min_thinking_budget": 1024,
"parallel_tool_calls": true,
"streaming": true,
"tool_calls": true,
"vision": true
},
"tokenizer": "o200k_base",
"type": "chat"
},
"id": "claude-haiku-4.5",
"model_picker_category": "versatile",
"model_picker_enabled": true,
"name": "Claude Haiku 4.5",
"object": "model",
"policy": {
"state": "enabled",
"terms": "Enable access to the latest Claude Haiku 4.5 model from Anthropic. [Learn more about how GitHub Copilot serves Claude Haiku 4.5](https://gh.io/copilot-anthropic)."
},
"preview": false,
"supported_endpoints": [
"/chat/completions"
],
"vendor": "Anthropic",
"version": "claude-haiku-4.5"
},
There was a problem hiding this comment.
Here's what i have for haiku 4.5 inside vscode:
{
"metadata": {
"extension": {
"value": "GitHub.copilot-chat",
"_lower": "github.copilot-chat"
},
"id": "claude-haiku-4.5",
"vendor": "copilot",
"name": "Claude Haiku 4.5",
"family": "claude-haiku-4.5",
"tooltip": "Fastest and most compact Claude model. Ideal for quick responses and simple tasks. Rate is counted at 0.33x.",
"version": "claude-haiku-4.5",
"multiplier": "0.33x",
"maxInputTokens": 127805,
"maxOutputTokens": 32000,
"auth": {
"providerLabel": "GitHub Copilot Chat",
"accountLabel": "nulladdict"
},
"isDefaultForLocation": {
"panel": false,
"terminal": false,
"notebook": false,
"editor": false
},
"isUserSelectable": true,
"modelPickerCategory": {
"label": "Premium Models",
"order": 1
},
"capabilities": {
"vision": true,
"toolCalling": true,
"agentMode": true
}
},
"identifier": "copilot/claude-haiku-4.5"
}Input tokens seems kinda weird to me (it matches the ui, but still)
It seems close enough so I think you should just use whatever the api gives you
There was a problem hiding this comment.
Meanwhile copilot-cli shows something different
There was a problem hiding this comment.
I also took a look inside copilot cli and scraped models from there. Here's the haiku definition:
{
"billing": { "is_premium": true, "multiplier": 0.33 },
"capabilities": {
"family": "claude-haiku-4.5",
"limits": {
"max_context_window_tokens": 144000,
"max_non_streaming_output_tokens": 16000,
"max_output_tokens": 32000,
"max_prompt_tokens": 128000,
"vision": {
"max_prompt_image_size": 3145728,
"max_prompt_images": 5,
"supported_media_types": ["image/jpeg", "image/png", "image/webp"]
}
},
"object": "model_capabilities",
"supports": {
"max_thinking_budget": 32000,
"min_thinking_budget": 1024,
"parallel_tool_calls": true,
"streaming": true,
"tool_calls": true,
"vision": true
},
"tokenizer": "o200k_base",
"type": "chat"
},
"id": "claude-haiku-4.5",
"is_chat_default": false,
"is_chat_fallback": false,
"model_picker_category": "versatile",
"model_picker_enabled": true,
"name": "Claude Haiku 4.5",
"object": "model",
"policy": {
"state": "enabled",
"terms": "Enable access to the latest Claude Haiku 4.5 model from Anthropic. [Learn more about how GitHub Copilot serves Claude Haiku 4.5](https://gh.io/copilot-anthropic)."
},
"preview": false,
"supported_endpoints": ["/chat/completions"],
"vendor": "Anthropic",
"version": "claude-haiku-4.5"
},So I think the api can be trusted over whatever vscode is doing
There was a problem hiding this comment.
Here are so more interesting findings:
{
"family": "claude-opus-4.6-fast",
"limits": {
"max_context_window_tokens": 200000,
"max_non_streaming_output_tokens": 16000,
"max_output_tokens": 64000,
"max_prompt_tokens": 128000,
"vision": {
"max_prompt_image_size": 3145728,
"max_prompt_images": 1,
"supported_media_types": ["image/jpeg", "image/png", "image/webp"]
}
},
}{
"family": "claude-opus-4.6",
"limits": {
"max_context_window_tokens": 144000,
"max_non_streaming_output_tokens": 16000,
"max_output_tokens": 64000,
"max_prompt_tokens": 128000,
"vision": {
"max_prompt_image_size": 3145728,
"max_prompt_images": 1,
"supported_media_types": ["image/jpeg", "image/png", "image/webp"]
}
},
}So regular opus 4.6 has 144k context window and fast has the full 200k
|
I'll close this. @rekram1-node please use values from the api. Would also be cool if you add opus 4.6 fast with it and gpt 5.3 codex if it's available (it's not yet for me) Here's the simplest script to get all the models that I could make (requires copilot-cli): import { CopilotClient } from "@github/copilot-sdk";
const client = new CopilotClient();
await client.start();
console.log(JSON.stringify(await client.listModels(), null, 2));Edit: seems like it's missing opus 4.6 for me (not enabled) |
|
will do, thanks for helping me figure this out |
3 participants
A bunch of model sizes are wrong. I fixed them
The sizes are also approximate (some are kinda weird and don't end with zeros, e.g. 127805 input size for gpt 5.2). I didn't fix that