Computer interface silently drops modifier keys (keys param) from API response

[FuturizeRush]

Description

The OpenAI API's computer tool actions (click, double_click, drag, move, scroll) support an optional keys parameter — described as "The keys being held while clicking" (e.g., Shift+click for multi-select, Ctrl+click for new tab).

The agents SDK's Computer and AsyncComputer abstract classes do not accept this parameter, and the action dispatch in tool_actions.py does not extract or pass it. Modifier keys from the model response are silently dropped.

Reproduction

import asyncio
from openai.types.responses.response_computer_tool_call import (
    ActionClick, ResponseComputerToolCall
)
from agents.run_internal.run_loop import ComputerAction

class LoggingComputer:
    # ... (implements Computer interface)
    def click(self, x, y, button):
        print(f"click({x}, {y}, {button})")  # no keys param

async def test():
    computer = LoggingComputer()
    action = ActionClick(type="click", x=500, y=300, button="left", keys=["shift"])
    tool_call = ResponseComputerToolCall(
        id="c1", type="computer_call", action=action,
        call_id="c1", pending_safety_checks=[], status="completed",
    )
    await ComputerAction._execute_action_and_capture(computer, tool_call)
    # Output: click(500, 300, left)
    # keys=["shift"] is silently dropped

asyncio.run(test())

All 5 action types are affected: click, double_click, drag, move, scroll.

Why this matters

• keys on click is semantically different from keypress — it means "hold these keys while clicking", not "press and release"
• Shift+click (multi-select), Ctrl+click (new tab), Alt+drag are standard desktop operations
• The API type definitions (ActionClick.keys, ActionDoubleClick.keys, etc.) are part of the GA computer tool, not beta
• Silent data loss with no warning or error

Suggested fix

Add keys: list[str] | None = None as a keyword argument to the affected methods on Computer and AsyncComputer:

# Before
def click(self, x: int, y: int, button: str) -> None: ...

# After
def click(self, x: int, y: int, button: str, *, keys: list[str] | None = None) -> None: ...

This is backward compatible — existing subclasses that don't accept keys will continue to work at runtime (Python does not enforce signature matching on abstract method overrides). The dispatch in tool_actions.py would need to extract action.keys and pass it through.

Happy to submit a PR if this direction makes sense.

[FuturizeRush]

Correction on backward compatibility:

I want to clarify my earlier claim. Adding keys to the abstract class signature alone is not enough — if tool_actions.py starts passing keys=["shift"] to an existing subclass whose click(self, x, y, button) doesn't accept it, it will raise TypeError.

The dispatch layer would need to handle this gracefully, e.g.:

import inspect

sig = inspect.signature(computer.click)
if "keys" in sig.parameters:
    computer.click(x, y, button, keys=action.keys)
else:
    computer.click(x, y, button)
    if action.keys:
        logger.warning("Computer.click() does not accept 'keys' — modifier keys dropped")

This way:

• New implementations that add keys get the full data
• Old implementations keep working with a deprecation warning
• No silent data loss in either case

Apologies for the incomplete analysis in the original post.