38 · How agents get files, pictures, plots and diagrams

The rule: every file the person attaches reaches the model or agent as the file itself. Ensemble does not decide beforehand what a model can read, does not read files for it and runs no OCR. A file is left out only when what Ensemble or the provider can carry is exceeded, or when the provider refuses it, and then the model is told which file and why. This applies in Ensemble chat, in Ensemble Node and on a task an agent runs. File content is always passed as information, never as instructions.

1. What each surface sends

In chat, hub_list_files finds files by name, summary or text when the person asks for one in words ("the Q3 deck"), each with its summary; it tells the model to ask which file is meant when several match. hub_view_file opens a file by id: the model is given the file itself. Choosing a saved file never uploads it again.

2. Models with a key and included models

files/model-input.ts attaches every file: a PNG, JPEG, GIF or WebP picture (recognised from its bytes) as an image part, anything else (PDF, Office, text, other pictures, audio, video…) as a file part with its media type. The model's provider decides what it reads.

What one request carries. Up to 20 attachments and 32 MB of them together (MODEL_ATTACHMENTS_PER_REQUEST, MODEL_ATTACH_TOTAL_BYTES): what Ensemble itself can carry, including a desktop or CLI included-model turn (64 MB of base64, routes/included.ts). A file past that is named with the tool that opens it. In a running conversation the newest 20 attachments within 32 MB are kept and older ones become a note (keepRecentAttachments), so a long run does not resend them every turn.

When the provider refuses. If a request with attachments is refused (an answer worded about pictures or files, or any 400, 413 or 415), it goes again with only the attachments verified for that model, then with none (attachmentsRefused, narrowAttachments in lib/vision.ts); the turn says so (a status line in chat, an engine note on a task) and the model is told which files were removed. A text file's own contents stand in for its file part. What is verified:

Every included deployment was checked on Foundry with files whose answer is only inside the file: each read a text PDF, a scanned (image-only) PDF, a .docx, a .pptx, a .xlsx, a CSV, a Markdown file, a JSON file and a PNG, and answered correctly. An archive (.zip) is refused with "Invalid file data… valid file MIME type".

Credits. An included-model reservation counts a picture as 8,000 input tokens and a file as its size in bytes, at most 150,000 tokens, plus 3,000 (runtime/included.ts, included.py). The call is settled on the tokens Azure reports (33).

Transport. hub-api writes user content as OpenAI chat parts: {type:"image_url"} and {type:"file", file:{filename, file_data}}. Chat-compatible providers take them as they are; runtime/content-parts.ts (and apps/agent-runtime/ensemble_agent/content_parts.py, its Python mirror) rewrites them for the OpenAI Responses API (input_image, input_file) and the Anthropic Messages API (image blocks; text files as text document blocks, every other file as a base64 document block with its own media type; joined to the tool-result turn they follow). Requests to GitHub Copilot that carry a picture add Copilot-Vision-Request: true. A file a tool opened (hub_view_file, view_attachment) follows all of that step's tool results as a user message, narrowed the same way after a refusal (partsAtStage).

History. Chat stores only the file mentions. A replayed earlier message names its files as links, and the model opens them again if it needs to. File bytes go only into the model request: they are not stored with the conversation, the tool call or the run trace.

3. A person's own subscription agent in chat

When the chat model is one of the person's own subscriptions, the prompt names the attached files ("These come with this message as files…") and runSubscriptionTurn (subscriptions/turn.ts) delivers every one to the agent's computer. The computer downloads each (GET /api/devices/self/assistant-turns/:id/files/:fileId, only the turn's own files, only while the turn runs) into the turn's files/ folder (subscriptions/turn-files.ts), and each goes inside the agent's message in the agent's own input form:

A file is saved only, not put inside the message, where putting it there would fail the whole turn (§7). The message names it, says it is saved in the turn's files/ folder and why it is not inside the message. In a chat, Ask Ensemble or a routine the agent keeps its own tools (30 §8), so it can open a saved file itself (a Word file, a deck, a scan) with whatever it has on the computer. The exact rules: 30 §8. A computer whose Ensemble is older (assistantTurns 1) is not sent the files; the agent is told that updating Ensemble on that computer fixes it (25).

4. Ensemble's extracted text is not sent to models

Files extracts text from PDFs with a text layer, Word, PowerPoint, spreadsheets, CSV, Markdown and code (32 §7) for search, previews and page conversion in the Hub. Ensemble does not give that text to the models and agents it runs: EXTRACTED_TEXT_TO_MODELS in files/extracted-text-policy.ts is off. Behind that switch, kept for a later decision:

• the start of each attached file's text in chat, Ensemble Node and task briefs, and in the assistant's open-page context and a paired computer's task brief (filesSection);
• Ensemble's text sent with a file to a person's own agent (TurnFile.text, notGiven in subscriptions/turn-files.ts), and to a computer too old to receive files (runSubscriptionTurn fallback);
• the read tools: hub_read_file (assistant/tools/files.ts), read_attachment (workspace/tools.ts), and uploaded files in hub_search_context and hub_read_context (assistant/tools/context.ts).

A file's summary is not that text: a model made it from the file itself when the file was added (32 §4). Models are given it wherever files are listed for them (hub_list_files, the files section of a brief or message), to tell which file is which without opening it, and a file's name and summary are searchable everywhere (context/graph/search.ts). The Context Bridge is the person's own key reading their Files: ensemble_read gives a file's details and summary, and ensemble_get_file gives Ensemble's text of it when asked and the file itself (bridge/files.ts, CONTEXT_BRIDGE §5).

5. Safety

• Access. Every read goes through the space's own Files (filesByIds, fileOriginal filter by space and deletedAt). A Node message can carry only the sender's own files; a node reads only its owner's. A computer fetches only the files of a turn it is running. Choosing a file never widens access.
• Untrusted content. Files, pictures, captions and plot data arrive marked as information, in the user turn, never in the system prompt. Tools and permissions do not change with what a file says.
• Logs. Requests carrying files are not logged; stored tool calls keep the file id and name only.

6. Design choices

7. Files outside the message

Each case is named to the model when it happens. Closing one is per agent (for example a document-reading MCP server, or the agent's own read tools limited to the turn's files/ folder), not Ensemble reading the file.

Not built: audio and video understanding; version pinning (a file replaced between attaching and reading is read as it is now); the "open in app" handoff (routes/handoffs.ts) passes the task's title and description, not its files.

8. Tests