Computer use

A user asks for something

"Book the cheapest flight", "fill in this form", "export the report as PDF".

The agent boots a desktop sandbox

With a resolution, DPI and timeout.

It takes a screenshot

The current state of the screen goes to a vision model.

The model picks the next action

Click here, type this, scroll down.

The SDK executes it — and the loop repeats

Until the model says the task is done.

Install#

JavaScript & TypeScriptPython

npm i @e2b/desktop

pip install e2b-desktop

Start a desktop and watch it#

TypeScriptPython

import { Sandbox } from '@e2b/desktop'
 
const sandbox = await Sandbox.create({
  resolution: [1024, 720],
  dpi: 96,
  timeoutMs: 300_000,
})
 
await sandbox.stream.start()
console.log('View desktop at:', sandbox.stream.getUrl())

from e2b_desktop import Sandbox
 
sandbox = Sandbox.create(resolution=(1024, 720), dpi=96, timeout=300)
 
sandbox.stream.start()
print("View desktop at:", sandbox.stream.get_url())

The agent loop#

TypeScriptPython

while (true) {
  const screenshot = await sandbox.screenshot()          // Buffer
  const action = await getNextActionFromLLM(screenshot)  // your model call
  if (!action) break
 
  switch (action.type) {
    case 'click':    await sandbox.leftClick(action.x, action.y); break
    case 'type':     await sandbox.write(action.text); break
    case 'keypress': await sandbox.press(action.keys); break
    case 'scroll':   await sandbox.scroll(action.scrollY < 0 ? 'up' : 'down', Math.abs(action.scrollY)); break
    case 'drag':     await sandbox.drag([action.startX, action.startY], [action.endX, action.endY]); break
  }
}
 
await sandbox.kill()

while True:
    screenshot = sandbox.screenshot()                  # bytes
    action = get_next_action_from_llm(screenshot)      # your model call
    if not action:
        break
 
    if action.type == "click":
        sandbox.left_click(action.x, action.y)
    elif action.type == "type":
        sandbox.write(action.text)
    elif action.type == "keypress":
        sandbox.press(action.keys)
    elif action.type == "scroll":
        sandbox.scroll("up" if action.scroll_y < 0 else "down", abs(action.scroll_y))
    elif action.type == "drag":
        sandbox.drag([action.start_x, action.start_y], [action.end_x, action.end_y])
 
sandbox.kill()

Every control#

Kind	JavaScript	Python
Mouse	`leftClick`, `rightClick`, `doubleClick`, `middleClick`, `moveMouse`, `drag`	`left_click`, `right_click`, `double_click`, `middle_click`, `move_mouse`, `drag`
Keyboard	`write(text)`, `press(key)`	`write(text)`, `press(key)`
Scroll	`scroll('down', 3)`	`scroll("down", 3)`
Screen	`screenshot()` → Buffer	`screenshot()` → bytes
Shell	`commands.run('ls -la /home')`	`commands.run("ls -la /home")`

E2B Surf is an open-source reference implementation of this loop.

Next#

Desktop template

Build your own desktop image

Analyze data with AI

The code-only version of the same loop

Updated Sep 23, 2026