Sensors
Each sensor watches one macOS signal and turns it into a typed event.
A filled dot means the sensor is on by default.
Subscribe
Every event carries kind, eventID, timestamp and displayID.
const ausm = createRuntime({
textInput: { enableNativeTextInput: true }, // off by default
});
ausm.onTextSelectionEvent((e) => {
if (e.selectionStage === 'END') explain(e.selectedText);
});
await ausm.initialize();
Materials
A material is one thing you look at: a tab, a file, a thread. What happens while it's in front collects on it.
Each cube is one MaterialEvent. Click a card to bring that material to the front.
const ausm = createRuntime({
enableScreenStreaming: true,
materialManager: { enabled: true },
});
ausm.onMaterialEvent((e) => {
if (e.materialEventType === 'MATERIAL_FOCUSED') show(e.materialId, e.title);
});
One title in one app, one id
Switch tabs, or open the same tab in a second window.
title
bundle_id com.apple.Safari
window_id 4182
material_id …
mat_ + sha256(title + "\0" + bundle_id)[0:12]
Open in new window
The window id changes. The material id doesn't.
Frames are kept only when the window changes
Each capture is compared to the last kept frame by perceptual hash.
Captured on appear, focus, title change, unfocus, every 2 s
Kept per material 200 frames, oldest dropped
Needs enableScreenStreaming
Apps add their own fields
Extractors read the window event and attach appMetadata.
Browsers url https://dl.acm.org/doi/…
domain dl.acm.org
Editors file /src/runtime.ts
filename runtime.ts
extension ts
More apps add an extractor in
material-manager/
metadata-extractors.ts
Materials
Memory
Where materials are kept: one SQLite file on the Mac.
Path materialManager.dbPath
Default :memory:, gone on quit
On disk owner-only folder (0700)
Pause writes freezeMemory()
Recent work weighs more
Focus time decays with a half-life. getMaterialWeights(days) ranks materials by it.
Half-life 7 days
How much one hour of focus still counts, by age.
Query
const [top] = await ausm.getMaterials(20);
const frame = await ausm.getMaterialLatestFrame(top.materialId);
const ocr = await ausm.getMaterialContent(top.materialId); // { text, lineCount, … } with OCR on
const rank = await ausm.getMaterialWeights(7);
Materials
Material Chat
An optional local HTTP service over memory: search your materials, or ask Claude about them.
Request
Reads
Returns
Leaves the Mac
A filled dot means the call goes to Claude.
Turn it on
The runtime starts the service as a child process. Don't run the material-chat bin yourself.
const ausm = createRuntime({
enableScreenStreaming: true,
materialManager: { enabled: true, dbPath: 'ausm.db' }, // a real file, not :memory:
memory: { ocrEnabled: true }, // search reads OCR text
materialChat: { enabled: true },
});
await ausm.initialize();
const base = ausm.getHttpUrl(); // http://127.0.0.1:8788
const { hits } = await fetch(base + '/memory/search?q=context').then((r) => r.json());
Listens on 127.0.0.1:8788, change with http.port
Ask and chat need ANTHROPIC_API_KEY
Answers requests with no Origin, localhost and file:// pages
Other origins list them in allowedOrigins
Search rank text match + recency + frames kept
Web tools off without BRAVE_API_KEY
Detectors
A detector is a plain-language rule. A model checks it against the focused window and your recent events.
Focused windowlatest kept frame
Recent eventslast 40
Group detectorsby sources + events
One model callper group
onDetectorEventafter cooldown
onTaskEventvia TaskTracker
Write a detector
A folder with a DETECTOR.md.
--- name: stuck-on-error description: Notices a repeated error event: stuck trigger: interval interval: 30000 cooldown: 30000 sources: [screen, events] events: [copy-paste, text-input] isolated: false --- Fire when the same error stays on screen and the user has copied or searched it. Summarise the error in under 12 words.
createRuntime({ detectors: { extraDirs: ['./detectors'] } });
ausm.onDetectorEvent((e) => { if (e.eventType === 'stuck') offerHelp(e.summary); });
Detectors
Tasks
rapid-task is the detector that ships with AUSM. It names what you're doing, and TaskTracker keeps that name on one id.
Same work = label words overlap ≥ 70% and description ≥ 60%, compared with the task before.
ausm.onTaskEvent((e) => {
if (e.status === 'TASK_DETECTED') startTimer(e.taskId, e.taskLabel);
});
Status detected · updated · completed
Runs on new activity, in its own model call
History not stored, keep the events yourself
Detectors
Models
Detectors need a vision model that returns JSON. Without one the pipeline stays off and everything else still works.
What one call sends
System promptfixed
Current taskif any
Detector rulesthis group
Recent events≤ 40
Window frameJPEG, downsized
ResponseJSON
Only the focused window's frame is sent, never the whole screen.
Pick a provider
Cerebras
Vertex
Local or OpenAI-compatible
Output JSON only, temperature 0
No key set pipeline stays off, no error
Model override TASK_DETECTOR_MODEL