Vision

rust/visiond/ ingests camera streams, records synchronized frames, and supports replay with the shared URDF. It is designed so downstream code can see which profile and timestamp domain produced a frame.

Frame contract

Each capture record should identify the device, requested and active profile, timestamp domain, sequence number, calibration revision, and dropped-frame or transport warnings. A consumer must not silently substitute a profile or calibration.

Robot, camera and simulator coordinates

The legacy robot-world JSON field world_from_base means world_from_URDF_root: the solver’s FK already includes the fixed arm mounts. Use scripts/lib/robot_world.py:root_from_world to invert it, and require the registration’s calibration_id to match the camera data being transformed. The calibrated camera pose in the robot scene is root_from_world @ world_from_camera. A consumer expressed in the follower arm base additionally applies base_from_root, derived from the URDF.

root and rig_center are the arm-pair midpoint. With +X forward through the fixed cameras and +Z up, the robot’s left base is at Y=+0.2675 m and its right base is at Y=-0.2675 m. Fixed frame and nominal camera mounts are relative to rig_center. Never move the frame to compensate for a registration that pairs one wrist’s markers with the other arm’s encoders. During teleoperation the joint vectors can be nearly identical, so a low reprojection error alone cannot establish the physical arm association. Such registrations and their derived surfaces need replacement, even if one arm’s FK is numerically unchanged.

Fixed camera bodies in Rerun follow those measured optical poses. The URDF supplies the body-to-optical geometry, not their measured rig mounting poses. The calibration producer owns world/calibration/camera_models; arm producers omit these fixed bodies, avoiding competing static transforms. Bodies without a matching calibration are omitted. Articulated wrist camera geometry remains on its arm’s joint chain.

The cockpit uses --robot-world (or the current local registration) to place world-frame tracking axes in the URDF root, or --camera-world to draw them as published, in the camera world. An absent or mismatched registration makes the robot-root tracking geometry unavailable. Raw camera previews remain in sensor axes. Python FK refuses unknown links instead of substituting the root origin, and resolves mimic joints from their driven joint, multiplier and offset. The left laser attachment rides the opposite carriage and depends on this mapping.

Developing the pipeline

cd rust/visiond
cargo test
cargo build --release

Use fixture streams or recorded data for tests. Keep live addresses, camera identifiers, credentials, and calibration snapshots in private acceptance records rather than in this page.

Calibration

The five-camera overhead calibration pipeline (tatbot calib capture phases, retained candidates, preview and apply) was removed in 2026-09 with the PoE scene cameras it measured. Tool-tip calibration on the demo rig is the ROS 2 stack’s station probe (tatbot ros calib run).

On the demo stack the D555 is the only fixed camera, so its colour optical frame is the world. tatbot ros calib overhead --arm <arm> measures an arm in it through the wrist D405, and is the registration the stack adopts:

  • one D555 frame with the arm at rest, the wrist’s holds over the palette’s roof tag, and its straight-down views of the fiducial tile on the holder;

  • both targets’ corners placed in the arm base by the wrist (forward kinematics and the URDF’s chain to the D405, the tile’s corners by bundle adjustment with the D405’s depth as a prior), fitted to where the D555 sees them, together with the wrist’s depth of the scene on the D555’s own;

  • each region (the station, the holder) left out and predicted by the other: one region alone misses the other by about ten millimetres, so both are measured. A tile the D555 cannot read (the 12 mm forearm tile) has the holder’s views aimed at the page the overhead tracks, and the holder rests on the wrist’s depth there and on that tracked page’s place: about 0.4-0.5° from the silicone tile’s registration, where the cube’s is about 2°, and the print within about 5 mm of where the wrist finds it. The silicone tile on the holder remains the better registration.

The wrist-tag registration below, tatbot ros register --arm <arm>, places the camera by the cube on the carriage instead. The cube gives under the tool’s weight and its stickers stand millimetres off their measured layout, and its fit came out tilted about 2° against the wrist camera and the measured tool, which reads the palette station about 8 mm low; tatbot ros setup runs it as the coarse first step, and ros calib overhead replaces it. It measures an arm as follows. The ROS stack drives the arm through 27 holds over its side of the table: the tool points down, turned and tilted so that the wrist tag triplet shows the camera different faces. The D555 owner captures each hold the arm reached: its measured tool within 5 mm and 2° of the hold. Every wrist-tag corner seen, placed in the arm base by forward kinematics and the measured tag layout, is one correspondence of a single camera-to-base transform. The corners are fitted to each tag’s edges, since the tags span about 23 px: OpenCV’s cornerSubPix leaves most of their corners on whole pixels, and its AprilTag refinement places them 0.44 px off in x and y. The transform is solved by PnP, and sightings far off the fit are dropped. The camera bundle holds the D555’s colour entry and the aligned depth entry at identity, with the optics the camera reports. A later run reuses the bundle while the camera’s profile, lens model, coefficients and serial are unchanged, since thermal focal drift is not a new camera.

--adopt installs a result only when it passes its gates: tags seen from at least 12 distinct poses (holds within 3 cm and 15° of each other count once), every tag seen from 3 of them, a corner residual median of at most 2.5 px (p95 6 px), and no single view moving the arm more than 5 mm or 0.5° when left out. Holds that repeat one pose fit that view closely and leave the arm anywhere else unmeasured. The owner then restarts so its frames carry the bundle. A new bundle is a new world: the arm is registered again in it.

A rigid transform cannot take up an error in the chain it sees the tags through. The rest is modelled in report.json’s chain, two terms each scored on holds it was not given:

  • the tag set’s seat on its parent frame (the target’s parent_frame);

  • offsets on joints 1–4, with the tags at FK(q + dq). Joint 0 is the camera’s yaw and joint 5 the seat’s roll, so neither is fitted.

tatbot ros chain <run-id>... pools several registrations of one arm and scores each model on the runs it was not fitted to. Neither gates nor adopts anything. The WXAI follower’s layout, solved on 2026-09-19 by the retired PoE cameras, sits 6.4, 6.8 and 2.3 mm (x, y, z of right/gripper_left) and 1.8° off its seat on the four registrations of 2026-09-29. The seat takes a left-out run’s corner median from 1.47 px to 1.01, and to 0.77 px with joint offsets of +23, +21, −24 and +1 mrad. The seat alone is written as wrist-layout.json, the wrist solve that scripts/vision/export_wrist_tags.py <file> --write --target <target> adopts. Adopting it moves every registration solved through the old layout, so the arm registers again.

With --prior <run-id>, the holds are planned from that run’s camera. Each candidate is solved from the pen-down joints over it. It must hold upright as an inspection view does, and stand clear of the arm’s own bodies. Of these, the holds are chosen greedily for the information they give the camera, the seat and joints 1–4 (D-optimal). The run prints the 1-sigma the plan predicts. For the WXAI follower, 40 holds at heights 0.10–0.22 m predict 1.5–2.2 mrad at 0.8 px. The holds of 2026-09-29 predicted 2.3–3.9.

Stock wrist camera geometry

Each wrist D405 sits in a stock Trossen case. Its chain from the case to its <prefix>_color_optical_frame and <prefix>_depth_optical_frame matches the Trossen WidowX AI model. Each case hangs from its arm’s last joint on fixed joints alone. On the WXAI follower’s wrist (urdf/tatbot.urdf) the case sits on link_6 at the original bracket’s pose, right/realsense_*, and a second case sits under it, its mount joint rolled 180° about link_6’s x axis, as right/realsense_lower_*. On the xArm (urdf/xarm7.urdf) both cases ride link7 in the dual-D405 bracket’s carriers, across the pen from each other, named after their cameras: xarm/realsense2_* and xarm/realsense1_*. These poses are nominal CAD geometry, not a measured correction. Each case is its own collision mesh and each camera link carries a box for the body and a cylinder where the cable leaves the case, so the arm’s self-collision guard and the station clearance checks see the cameras.

The camera registry binds wrist_2 (D405-2, the session’s) and wrist_1 (D405-1) to the xArm, each naming its own optical frames (two cameras on one arm must). The WXAI follower carries none since 2026-11-08, and the blue arm is the teleop leader alone and is not in the model. Moving an arm changes camera pose through that arm’s measured joint angles; it does not change the fixed mounting transform. Use color optical coordinates for depth explicitly aligned to color. Registering either view in the scene still needs exposure-time joint observations and the robot-to-scene transform.

The lower bracket’s mount joint is measured, not CAD: its pose to the upper camera from both cameras’ reads of one print (internal/evidence/wrist_pair.py), 1.76° and 1.96 mm off the stock bracket rolled 180°. The page filter reads it beside the upper camera at every hover and lift. An image kept from before it returned to the right wrist (it was the left arm’s wrist_left, and before that an earlier right lower camera) keeps its original geometry.

Live viewer

Every live workflow (tatbot live cockpit, which subscribes to what the camera owners publish on the bus, and the ROS 2 stack’s Rerun bridge) streams into one fleet Rerun viewer (a headless server on the rerun-server node; operator nodes attach windows) rather than opening a viewer of its own:

tatbot viewer start            # on the server node: gRPC server on :9876 (+ web :9090)
tatbot viewer open             # attach a capped window on this node (--memory-limit 4GB --fullscreen on the station)
tatbot viewer status           # server, window, version, and the web viewer's ?url= address
tatbot viewer reset            # send the fixed blueprint again: every window drops its own edits
tatbot viewer view file.rrd    # stream a recording (.rrd, .wxtl, or a teleop run dir) into the fleet viewer
tatbot viewer install          # on the server node (sudo): the server unit and its load monitor
tatbot viewer station-install  # on the station (sudo): its load monitor

viewer open runs the window as the transient user unit tatbot-viewer-window: the viewer drops its oldest rows at --memory-limit (default 1GB), and the unit’s MemoryMax, the cap plus 1 GiB, is the hard ceiling. A window that already runs is left alone, so the station’s launcher can call it every time; --restart closes it first, to open it with a new cap or on a new path. A Rerun window never reconnects to a fleet viewer server that restarted, so the unit keeps it on its server: after 15 s without a connection, or without the server answering, it closes the window and opens it again once the server answers, at the same address and behind the window that has the focus. The new window receives the server’s buffer like any other, which is small just after a restart. A window that closes by itself ends the unit, except the fullscreen demo page, which comes back; tatbot viewer stop closes it for good. Once the window maps it is maximized, or with --fullscreen made fullscreen (no title bar, no GNOME top bar) and the focus handed back to the window that held it, so the station’s demo page opens behind its agent terminal and the operator alt-tabs to it. Both are one EWMH request to the X11 window manager, no key typed; a window already in the state is not asked again, and nothing happens off X11. Each window keeps its own copy of the blueprint, and a tap or click edits it. The eye beside a pane’s maximize button hides that pane: an empty title strip is left where it was, and while a pane is hidden Rerun 0.36 saves the whole layout again on every frame. The station’s page shows no blueprint panel, so nothing on screen brings the pane back. tatbot viewer reset sends the fixed blueprint again, and every attached window adopts it: hidden panes, a dragged eye and dragged bounds return to the fixed ones, without restarting the hub or a window. viewer status prints the browser form, http://<server>:9090/?url=rerun%2Bhttp%3A%2F%2F<server>%3A9876%2Fproxy, with the server’s rig address from the fleet map; it needs nothing installed on the viewing machine.

A window pulls the heaviest stream on the rig, and the server may share its rig port with an arm controller, whose command packets then wait behind each depth chunk. A node with a direct cable to the server dials the server’s address on that cable instead: the server’s links entry in the fleet map, keyed by that node’s role ({"links": {"station": "<address>"}}). The cable is used only while the kernel routes its address out another interface than the rig address and the server answers there within a second; otherwise the node dials the rig address, so a dead cable never costs the window. Producers on that node (its load monitor) follow the same rule. viewer status says which path the window rides (direct link to the station over <interface> or rig LAN over <interface>, with the reason the cable was passed over), and on the server lists each client by the path it arrived on. A window keeps the path it opened on: viewer open --restart moves it.

viewer install and viewer station-install refuse with exit 3 unless rerun --version on that node reports the fleet’s 0.36.0, and name the fix (uv tool install 'rerun-sdk==0.36.0'). Each installs tatbot-sysmon@, which logs the host’s CPU and memory %, its hottest CPU/SoC thermal zone (deg C) and the station’s GPU % and temperature at 1 Hz under system/ros/ and system/station/: display only, about 0.1% of a core. Each source’s legend is its host’s platform in the fleet map, else its role. The station’s monitor also reads each host whose load_via names the station’s role (the e-stop relay’s board, which has no checkout) into system/estop-relay/: one ssh session that reads /proc/stat and the thermal zones every 2 s with shell builtins and a sleep, so nothing is installed, started or written there. The ros node’s monitor adds system/robot/: the hottest motor of the arm the drawing stack drives (the controller’s rotor temperatures in the newest record of the stack’s flight log) and that stack’s controller process’s CPU in % of one core. It reads both from the newest ros-stack run’s files, never from the stack’s graph or its control loop. The Demo tab plots cpu_percent and temp_c of these four sources, each in one colour in both panes: CPU % on a fixed 0-100 scale with the one legend above its lines, beside TEMP, both over the last ten minutes. The blueprint names only the entities. Each legend and colour is a static row the monitor logs, so a blueprint sent from a newer tatbot-visiond needs the monitors from the same revision, or older monitors show their own labels in Rerun’s per-entity colours. On the ros node, tatbot-cockpit.service is the standing bus subscriber (visiond subscribe --recording-id live-cameras --duration-seconds 0 --camera-world), installed by a fleet deploy of that node between draws. It shows the D555’s depth out to --overhead-depth-max-m (0.85 m: the table and what stands on it, not the room behind; 0 shows all of it). tatbot rig check reports all of these under part station, never under pink drawing.

live cockpit --duration 30 runs a bounded check. It opens no camera: it subscribes to the frames the owners already publish on the bus. The operator needs the fleet viewer and the bus router, but no camera-LAN address. It streams into a recording of its own (Cockpit HH:MM), and every window on the fleet viewer switches to a recording it has not seen, the station’s demo page included, as viewer view does on purpose: while the station shows the demo, the standing tatbot-cockpit.service is the camera feed, and an ad hoc cockpit is not run. It refuses --recording-id live-cameras, whose overlays the standing cockpit places in camera world.

viewer view streams from the node that holds the file: a path that does not exist where you typed it is looked for on the arm node, where flight logs live. An unavailable fleet viewer returns exit 5 with a recovery hint. It never opens a local window implicitly; viewer open is the explicit extra-window command.

Every source carries a name in the viewer’s list: workflow recordings are named from their id (Cockpit 20:15, Sweep 15:02, Replay <file> HH:MM, UTC like the time panel), and the standing recording the blueprint is sent into Rig preview. A producer that joins a recording sends the same name, derived from the id, so it never flips between producers.

Producers share one application id, join a session by recording id, stamp the capture_time wall-clock timeline, and log under fixed entity prefixes (cameras/, robot/, world/, teleop/, surface/, draw/, session/, system/). One blueprint definition in rust/visiond always presents the fixed Demo / Session / Telemetry / Calibration top-level tab bar, in that order, with no nested tab stacks. Demo comes first because the Rust blueprint cannot select a tab: it is the portrait drawing page (status, the 3D scene in the camera world, the design map beside the two upright wrist RGBs, the D555 RGB with the markers the trackers read on it and the tags no tracker follows, which the cockpit reads there itself once a second (subscribe --fiducials: the inventory’s palette target, and the xArm’s cube xarm_cube while its layout is nominal, each tag outlined green and labelled with its id), beside CPU and temperatures). The rerun-server installs it once per server generation, and viewer reset sends it again. Workflows publish data only, and neither workflow startup nor recording selection resets the operator’s selected tab. Every viewer and producer is memory- and rate-capped; scripts/check rerun-caps refuses a launcher or producer that is not. Python producers use scripts/lib/tatbot_rerun.py.

tatbot live cockpit --display-quality normal caps the subscriber at 2 Hz; --display-quality showcase raises that cap to 5 Hz, and --fps overrides either. Showcase does not turn reduced upstream bus derivatives into full-resolution sensor data. These controls affect display copies only, never capture, tracking, policy, journal, or sensor rates.

Beneath that cap (subscribe --max-fps, default 5 Hz) each stream has its own rate: the overhead colour at 5 Hz (--overhead-color-hz), its depth at 1 Hz (--overhead-depth-hz), each wrist tap’s colour at 2 Hz (--wrist-hz). The hub streams every row on to the station over a link the arm controllers’ packets may share, so no image the cockpit logs exceeds 60 KiB (its chunk stays within 64 KiB, about 0.5 ms at 1 Gbps), and images go out one per 20 ms tick. Colour is re-encoded as a 640x360 JPEG of quality 70 (a lower quality, then half the size, only for a frame that would not fit), scaled up or down to fill 640x360 keeping its shape, so a smaller wrist preview (480x270 on a USB 2.1 link) fills its pane like the rest; the depth is decimated to 320x180 by the nearest valid range in each block (zero stays invalid) and logged RVL-compressed, lossless (EncodedDepthImage, the Overhead depth pane). The Demo scene draws that depth instead as an RGB cloud (cloud/<band>, depth_cloud.rs): every second return each way on its pixel-centre ray under the same Pinhole, coloured with the mean of the colour pixels it stands for in the newest colour image and a tenth of a cool blue tint, each point 0.6 of the gap to its neighbours at its range so the cloud reads as a surface, in four rows of at most 3,600 points so each stays within the row cap: ~106 KB a second on the rig’s table (2026-10-19), at most 230 KB. The Scene leaves the depth image out, so a window shows the RGB cloud only once it holds a blueprint sent since (tatbot viewer reset). The wrist tap’s depth is not logged: the bridge draws the wrist cloud the same way, with a warm amber tint, so the two read apart where they overlap. The Demo tab has a pane per wrist camera, D405-1 (RealSense 1) and D405-2 (RealSense 2, the session’s), each turned upright by its camera’s display rotation (one’s mount is the other’s turned 180°). Each keeps a draw’s last frame under its age label; a cockpit that starts covers that frame, which it cannot vouch for, with a black one, clears its label and shows “no frame yet” until that camera’s first frame. The print’s markers each wrist reading decodes in a frame (a gauge hold’s, a curved page’s locate view’s, the tracker’s hover and stroke-end frames) are drawn on that frame: each marker’s outline, green when the reading’s accepted fit kept it, amber when its fit kept it but was not accepted, red when read and left out, its id at its centre, and a line naming the reading under the age label. They come from the session’s wrist tap as one tatbot.wrist-markers/1 envelope on tatbot/vision/d405/<camera>/markers, stamped as their frame, a few hundred bytes each, about 0.6 s after it (the reading’s own time); the cockpit logs them at that stamp in the pixels it logs the frame at, and the camera’s next frame hides them. A flat page’s locate views read a rectified page image and are not drawn. At the defaults the cockpit streams ~0.4 MB/s, ~0.5 while a draw’s wrist tap sends; it streamed ~2.6 MB/s in rows up to ~470 KB before 2026-10-18. --camera-world draws trackd’s and stencild’s poses at identity in the camera world, the ROS world the Demo tab shows. --output FILE.rrd also writes an offline capture, and --connect may then be left out. --duration-seconds 0 runs until SIGINT or SIGTERM. A malformed sample is counted, reported on stderr at most every 10 s and skipped; it never ends the subscriber.

Evidence limits

Timestamp alignment is not geometric calibration, and a synchronized recording is not proof of safe robot behavior. State those limits in every published result.

Tracker memory

The tracker service allows up to eight malloc arenas for its parallel camera detectors. A single arena serialized the allocation-heavy planar initializer; this setting changes allocation contention, not detection or pose-solver math. Camera-owner allocation settings are independent and unchanged.

Experimental surface replay

tatbot vision surface replay --synthetic --max-frames 300 --rerun evaluates sparse OpenCV tracks against known image motion, brightness changes, blank texture, occlusion, timestamp gaps and ambiguous repeating texture. --width 1920 --height 1080 exercises larger frames. Reports and optional RRDs go into the surface-replay run log.

For retained camera pixels, use --recording /path/to/camera/frames.jsonl --roi X Y W H. The reader verifies native payload checksums and reuses the visiond pixel decoder, including JPEG evidence. It requires normalized capture timestamps and one sensor per index. It never opens a camera or robot connection. Tracks keep their initial identities; default loss stays latched until a new replay.

Reports separate active tracking from initialization/loss and input decoding. Optical flow operates on a moving crop around surviving points. CPU cost is measured offline with one OpenCV thread; synthetic image generation and optional Rerun logging contribute to total process cost. Neither throughput nor image-space error establishes live latency, 3D accuracy, material identity through occlusion, or performance beside the camera services. Repeated texture, background features and gradual drift can still produce incorrect tracks.

Add --recover to evaluate bounded blur handling and reference-only reacquisition. This experimental mode withholds coordinates when image sharpness falls or too few reference tracks survive. It retains the last accepted image and point IDs, retries at most 10 times per second, and requires two successful reference matches before reporting reacquired. It never detects replacement features. A reference older than 750 ms ends recovery; the original 150 ms input-gap check still applies. Large turns, repeating textures, prolonged occlusion and nonrigid changes remain unqualified. Recovery does not establish physical material identity or grant motion authority.

The synthetic suite includes short blur and occlusion cases. Reports include wall and CPU processing percentiles separately for each state, so cheap paused or lost frames cannot hide the cost of recovery attempts.

--match orb or --match sift adds a periodic advisory keyframe search. ORB is the cheaper comparison; SIFT provides a second rotation matcher. Both reuse OpenCV, retain the initial surface patch, and search without assuming cylinder geometry. Each attempt searches one region: a square around the last candidate (initially the selected ROI), then a larger square after a miss, then the full image. After three misses the cycle repeats; a candidate recenters the next local search. This center is an advisory search hint, not a verified surface position. Matches must pass bidirectional descriptor-ratio checks, robust local image alignment, spatial coverage and projection checks. Output is a candidate polygon only: it never reactivates tracking, reseeds point IDs, or certifies physical material identity. A homography checks local image consistency; it is not a curved-surface model.

Search images are capped at 1280 pixels on the long edge and extraction requests at most 1200 scene features, with separate quotas in a 4-by-3 image grid to limit background competition. Reference patches retain native detail up to the same edge cap. Local search squares span 1.5 times the candidate’s longest bounding-box edge; expanded squares span 3 times that edge. Local matches cannot rule out identical texture outside the searched region. Attempts are at least 500 ms apart and are spaced further using measured CPU cost to target 10% of one core over source time. This is an offline workload target, not a hard live CPU quota. Matching remains disabled by default. The rotation suite measures candidate-corner error separately from tracked-point error; Rerun records matching attempts at their source timestamps.

--match-keyframes (requires --match and --recover) experiments with at most three appearance references, retaining the initial reference permanently. It reuses the existing flow observations; it opens no cameras and computes no additional optical-flow pyramids. Admission requires clear tracked points, 80% of original IDs (at least 24), broad geometric support, and agreement within 2 pixels between flow and a direct descriptor match to the initial reference. Descriptors near the projected patch boundary are excluded. Each admitted reference maps back to the original patch coordinates; candidates cannot admit other candidates or resume robot tracking.

Admission and search share one measured CPU allowance. Searches rotate through references from newest to oldest, one per attempt, while each reference expands its own search window. Cheap unchanged-view checks may run every 50 ms; replay images remain capped at 5 Hz. Reports include the first tracking interruption, peak reference count, and bank decision reasons. This is an opt-in image-space experiment: agreement on repeating texture does not establish physical material identity, and nonrigid surface motion remains unqualified.

The scale_turn synthetic case gradually changes scale and orientation before a larger jump, with a single visible image copy. Candidate-corner errors and flow-point errors are reported separately; the repeating-texture alias case continues to document the material-identity limitation.

Wrist camera ownership

Each D405 in the vision registry declares a physical arm and an owner_role. The owner role resolves through the node registry and must have exactly one owner. It describes the USB host, independently of which computer controls the arm or which teleop role that arm takes. Device names and serials remain stable; view roles distinguish the cameras. Both views are on the xArm: wrist_2, D405-2, the session’s, and wrist_1, D405-1 (on the WXAI follower’s wrist from 2026-10-23, and before that the left arm’s wrist_left). Both are owned by the ROS stack’s node, which opens them directly; no visiond D405 service runs for them.

The D405 service resolves its node’s cameras and passes explicit, repeatable --sensor arguments to capture-realsense-all --group d405. Unknown names, duplicate selections and names outside the selected group refuse before any camera is opened. Node ownership and device selection use the same deployed source snapshot. The service template gets its node identity from deployment, so the same template runs on whichever node owns a wrist camera. Each owner keeps its local /tmp/tatbot-d405-frames.sock and local LeRobot camera handoff; starting a recording never stops another node’s camera service.

A frame timeout — no frame, error or finish event from any camera worker for 15 s — is answered by the owner itself before it is answered by systemd. The owner stops the pipeline of every device it runs, puts each through a librealsense hardware reset, reopens it as after any transport failure, and publishes one tatbot.capture-health/1 record per device with event: reset on its capture topic (tatbot/vision/d405/capture/<owner-node>; tatbot/vision/overhead-depth/capture for the fixed camera) naming the last error that worker saw. The reset is issued by the device’s own worker, so it reaches a device whose pipeline is up and delivering nothing (the SDK’s frame wait timing out); a worker that never returns from a librealsense call (pipeline start or stop) issues none, and the owner’s exit says so. A second timeout inside one minute exits the process so systemd restarts the unit; a reset that restored frames for a minute earns the next timeout its own reset. A device that does not re-enumerate after its reset is an error storm (one reopen attempt every 250 ms), not a timeout: the owner neither exits nor resets again until the device returns. The reset re-enumerates that one USB device and nothing else on its hub.

Live frames retain physical_arm, capture_owner_role and device serial metadata. Camera stream topics remain tatbot/vision/d405/<sensor>; raw wrist snapshot queries now require tatbot/vision/d405/capture/<owner-node>. There is no unqualified wrist snapshot endpoint that might return the other arm’s view. The fixed overhead endpoint remains unchanged.

LeRobot recording selects only the receiving physical arm’s local cameras and supports a single view. Rollout compares the resulting RGB/depth keys with the checkpoint before acquiring motion authority; a two-view checkpoint refuses a one-view configuration. Single-camera reconstruction and the moved camera’s new extrinsic require separate implementation and calibration; this capture split does not qualify either one. Clock alignment between hosts is also a separate measurement, not implied by successful local capture.

Use tatbot live cockpit to view both capture owners.

Fixed overhead depth camera

The fixed D555 is a separate overhead-depth RealSense capture group. Its owner publishes aligned color and depth at 5 Hz on tatbot/vision/overhead-depth/*, answers complete RGB-D snapshots on tatbot/vision/overhead-depth/capture, and exposes its full-rate local stream at /tmp/tatbot-d555-frames.sock. It is deliberately separate from the two moving wrist D405s: opening, synchronizing, or restarting one group must not take ownership of the other.

The cockpit shows the overhead colour and depth and the wrist taps’ colour as display copies within its row cap (above). Valid-min decimation preserves the depth scale and near edges. The overhead colour and depth each carry a Pinhole from the owner’s full-resolution intrinsics scaled to the logged size, so the depth becomes a cloud in the camera world; the overhead 2D panes are rooted at those pinholes. Full capture requests retain the original RGB-D frames. Join an existing recording with tatbot live cockpit --recording-id <id>. Raw camera-world wrist tracking is not overlaid on the robot model without a measured registration between those frames.

The installed profile is colour 1280x720 and depth 640x360, both at 30 Hz, depth preset 0, laser power 150 and a measured depth scale of 0.001 m per Z16 unit; librealsense aligns the depth to the colour, so the aligned depth is 1280x720 too. The colour’s 0.86 px/mm on the page decodes skin-stencil markers (an 18 mm marker’s 3 mm cells at ~2.6 px; 12 mm markers, ~1.7 px a cell, only upsampled twice: the skin stencil), where 640x360’s 0.43 px/mm decoded none. The two streams together held 30 sets a second with no drop over a 30 s test on the camera node. The depth stays at 640x360: in the installed view it gave substantially better valid-pixel coverage and repeatability than 896x504 at 30 Hz or 1280x720 at 15 Hz. Preset 0 gave the best stable-pixel coverage of presets 0-4. Laser 150 retained essentially the same coverage as 240 or 360 while using less projector power. These are measured starting settings for this scene, not universal D555 recommendations or drawing tolerances.

A 200-frame A/B/A check in the installed central 260x220 ROI compared the D555 while the wrist-D405 owner was active, stopped, then active again. The off and post-restore samples were effectively identical: 72.103% valid depth, 69.617% versus 69.638% pixels with at least 90% support, 2.965 mm temporal robust-sigma p95, 4.000 mm per-pair delta p95, and 24.42 versus 24.51 delivered frames/s. The pre-stop coverage was consistent at 72.046% valid and 69.587% stable. No central-ROI interference attributable to active wrist capture was measurable in this unattended scene. This does not qualify whole-surface coverage, other arm/projector geometries, lighting changes or physical depth accuracy.

The Ethernet transport requires a gigabit link, DDS discovery and an MTU of 9000 from the camera owner through the PoE switch. The camera sends only 9000-byte frames, so the owner’s NIC must accept them; the owner’s other routes can stay at 1500, since librealsense caps what the host sends at 1470 bytes. The manifest marks the camera transport = "dds", and its owner states the camera-LAN address discovery is bound to (--dds-address; the launcher passes the owner’s lan from config/nodes.json). A DDS camera’s context enables DDS for that address only and leaves USB enumeration out, so it never touches a wrist camera; USB cameras keep the SDK’s default context. The distribution’s librealsense package has no DDS. A node that owns a DDS camera provisions the SDK built with DDS as static archives under ~/.local/share/tatbot/toolchain/librealsense-dds (with a realsense2.pc naming every archive), and release builds there link it (fleet_toolchain::realsense_dds). The SDK’s file-format archive carries a pre-0.6 xxhash whose XXH32 symbols collide with the one the Rust lz4-sys crate links, and whose state layout differs. The provisioned librealsense-file.a therefore has those symbols renamed with an rs2_ prefix (objcopy --redefine-syms), and a PROVENANCE.md beside the archives records it. The capture backend maps the device hardware clock onto host Unix time, records the original device frame number separately, suppresses repeated-timestamp DDS framesets, and assigns a strictly increasing evidence sequence so a reconnect or repeated device frame number cannot overwrite a retained payload.

The active bundle’s two D555 entries hold the D555’s native colour optical frame, solved with its native intrinsics and distortion fixed so the solve agrees with the pixel grid the owner aligns depth to. The strict companion overhead_depth_depth entry carries the same optical geometry, because the owner aligns Z16 into color before publishing.

Once both D555 entries exist in the active bundle, its owner stamps that bundle and trackd reads the D555’s stamped sets for each pose update. If the bundle lacks either entry, the D555 stays available for camera-frame evidence and the tracker does not subscribe to it. This makes calibration membership explicit: an unstamped frame cannot become a world-frame tracking observation.

stencild is the same kind of subscriber for printed stencils: one observer for the fleet beside the overhead camera’s owner, reading its socket under the same stamped bundle. Each turn it hands the newest overhead RGB-D set to the observer; the D555 is its one view. The wrist camera’s readings belong to the drawing session.

Only a fiducial-tile print is placed, by its markers (fiducials.stencil.rgbd_pose). Each one found in the colour has each corner lifted along its ray to the depth about its own pixel: the mean of the middle half of a 15 x 15 px square (about 7 x 7 of the D555’s own depth pixels), taken over pixels whose mirror through the corner is valid too, so the arm over one side does not shift it. No surface is fitted to the depth, under a marker or across the sheet. A read is kept when it spans its printed side within 15%, and the most reads, one per code, are kept whose corners all lie no farther apart than printed and no nearer than their chord on the tightest skin the layout goes on, within 4 mm (Tile.curve_radius_m: the forearm tile a 35 mm forearm, the silicone tile 100 mm, its mount reading 146-191 mm): a false read stands a pitch or more from where the others put it. Of equal groups the one nearest its printed distances is kept, never the first by code, so two true reads and a stray beside them keep the true pair. The published pose is the layout laid unstretched round a cylinder through the kept corners (fiducials.stencil.bent: any axis in the print’s plane, convex to the camera, no tighter than 0.75 of the layout’s skin radius): its origin the print’s centre on that skin, its z the skin’s normal there. Markers along one row or column, or fewer than three, cannot show a bend, and the pose is then the flat layout’s rigid fit, the chord plane of the markers in view, whose tilt moves with whatever part the arm leaves visible (a synthetic 40 mm forearm, half hidden, turned it 22°). On a live forearm (2026-10-20) the chord plane tilted 16.7° median and 25.3° p95; the bent layout read a 59 mm skin and put the centre’s normal at 14.3° median and 20.0° p95, the centre and turn within 0.2 mm and 0.3° of the chord plane’s. The drawing session reads the pose for the page’s centre and turn and for its first gauge holds and wrist locate views; it judges a moved sheet by the kept markers’ rays and measures a curved skin at its gauge holds. The view that reads the most markers anchors the pose. Fewer than two agreeing markers is no pose for that turn (the arm over the sheet hides most of them), published lost with the reason.

Several sheets of one print can stand in the view: other skin mounts and spares on the table, the page of the rig beside it. They read the same codes, so nothing in a read says which sheet it lies on. Once the largest group is kept, the reads left are grouped the same way and each further group is placed too. The target’s pose is the largest group’s, and that is whichever sheet reads the most this turn, so with two sheets of about the same size it changes from turn to turn. The others ride in support.copies, each a pose, its sigmas and a support of its own. Which sheet is the page is decided by where it stands, and only a reader that knows where its arm is can say: the ROS bridge follows the sheet chosen (--sheet-at) or set up, and never hands the page to another (ros/README.md, section 4.3). The viewer draws the target as published. A small-marker tile keeps no read window on a turn that read copies, since the window would hide all but one.

The translation sigma is the markers’ residual floored at 2 mm, and the rotation sigma a fixed 1.5°: the depth and registration bias the markers cannot see. An installed reference that is not a tile, or fails to load, is published lost with the reason under the pattern its manifest names and never stops the other prints. The service publishes each print’s pose as tatbot.target-pose/1 on tatbot/tracking/target/<pattern_id> with a support block: the anchor camera, its marker count, ids and residual, both sigmas, the page corners (corners_m, the pose’s flat outline the viewer draws), the kept markers’ corners (markers.ids[k] names anchor_points_m[4k:4k+4]), the print’s other sheets in view (copies), the bundle id, and motion_authority: false.

Its configured period is one second; processing_p95_ms and capture_to_publication_p95_ms report the achieved rate and age. Per-turn capture scratch uses the service’s managed memory directory; durable estimates stay in its run log. References are installed under the log root (stencils/references/<name>/tracking.json beside its stencil.png and the tile’s tile.json) and rescanned by mtime, so installing one never restarts the service and none installed leaves it idle. --max-age-ms (3 s) gates the input only: a turn whose selected exposure is older than that, or in the future, is skipped before the estimator runs and counted as a stale set. A measured result is published as measured however long its turn took, stamped with its capture time, so a consumer reads its age from the sample. A late pose is never turned into lost: the ROS bridge holds the last measured pose through a lost, which would be older still. The observer decides nothing about motion. Each turn logs capture-write, estimator, publication, and observer-stage timings with cumulative per-camera socket and selected frame counts, sequence gaps and ring overwrites. The service metrics also expose the stale-set count and the last capture-to-publication age. The shared Rerun Session view includes each measured target’s live page outline under /world/tracking/target/<pattern_id> beside the calibrated robot. A lost sample clears that outline.

With matching D555 camera and robot-world bundles, consumers deproject aligned Z16 using the native color rays and the paired depth-to-color transform. Stencil tracking uses this conversion, including the distortion convention and the native-depth-Z to color-Z calculation. The robot-world JSON key world_from_base is historical: it maps the URDF root into calibration world, including the fixed arm mounts. Thus camera points use inverse(world_from_base) @ world_from_camera, with no additional arm-base translation. Fixed RGB observations and calibrated viewer frustums use this same convention. The installed-scene input range is 0.3-1.5 m. A stale owner, payload digest change, profile change, calibration mismatch, or missing RGB-D member refuses the capture rather than silently falling back.

Runtime optics and the bound model. The camera does not report one set of optics for the life of a process. librealsense’s thermal compensation re-reads the device calibration table as the ASIC temperature moves (the D4xx host monitor polls every 2 s and acts on 2 °C steps), and the D555 pushes each such update over DDS as a calibration-changed notification that replaces the stream profile’s intrinsics in place: same profile id, same extrinsics. The SDK align block, by contrast, freezes the aligned profile’s intrinsics when it is created. The owner therefore reads the model before every alignment, rebuilds the align block when the model changes, withholds an exposure whose aligned profile disagrees with it, and publishes the model it aligned with in both intrinsics and alignment_calibration. Consumers keep that frame-internal match exact: rays come from the frame’s own intrinsics and must equal the alignment model’s colour optics to the digit, which is what guarantees depth is deprojected on the grid the owner aligned to. Recorded evidence replays under the same rule and involves no bundle.

The bundle binds one snapshot of those optics, so a live model can differ from the bound one by the thermal drift alone. Observed on the installed D555 at 640x360: two frames from one owner process 77 minutes apart on 2026-09-16 reported fx 323.392 / fy 322.901 and then fx 323.216 / fy 322.726 (0.17 px, 5.4e-4 of the focal length) with the principal point and distortion coefficients unchanged; the cold camera two days earlier reported fx 321.988 before warming to 323.129. The scan and stencil-tracking paths report ray drift against the bound bundle but use each frame’s aligned intrinsics for deprojection; thermal focal drift does not hold a session. Image dimensions, the lens model and its coefficients, the stream profile, and the device serial (when recorded in the bundle) must still match. A calibration sweep requires one unchanged native model across its shots.

The D555 grants no motion authority until a measured camera-to-arm registration is adopted (tatbot ros register, above). Timestamp alignment is not that extrinsic. Even then a stencil page’s overhead pose is a prior: the drawing stack re-measures the page with the wrist camera and its touches before it draws. The present mounting distance also places part of the working surface near the lower edge of the camera’s preferred range, so placement and full-surface coverage must be rechecked during that calibration.

Offline RGB-D registration benchmark

tatbot vision surface rgbd --synthetic --rerun compares Open3D point-to-plane and colored ICP on known 3D motion, missing depth, absent overlap, timestamp gaps, and an intentionally ambiguous uniform cylinder. These sampled 3D controls are not sensor simulations or evidence of physical accuracy.

For existing visiond evidence:

tatbot vision surface rgbd --recording /path/to/evidence --sensor wrist \
  --roi 100 80 240 200 --depth-range 0.1 0.4 --max-frames 300 --rerun

For an operator-confirmed stationary scene, replace --rerun with --stationary-depth to measure depth coverage and repeatability without ICP. The report separates missing depth from temporal jitter: robust per-pixel sigma uses only pixels valid in at least 90% of evaluated frames. Flagged frames count against coverage and break consecutive comparisons, as do gaps over 150 ms. This measures repeatability, not absolute accuracy; object or camera motion contaminates the result. Cropped depth storage is limited to 64 MiB; reduce the ROI or frame count if that limit is exceeded.

Choose ROI and depth bounds for the actual working patch; these example bounds are not operating limits. The input must contain wrist_color/frames.jsonl and wrist_depth/frames.jsonl, aligned RGB/Z16 payloads, normalized timestamps, active intrinsics and measured depth units for registration. Stationary-depth mode needs the aligned stream identity, stable profiles and measured depth units, but neither Open3D nor camera rays, so it also runs on the aarch64 camera owner. Payload hashes are checked. No camera is opened. Nonzero inverse Brown-Conrady distortion reuses the drawing mapper’s cached rays at its supported 640×480 profile: a numpy port of the RealSense SDK’s deprojection, pinned by a checked-in table of the SDK’s own answers, so no SDK wheel is needed. Unsupported distortion models or dimensions are refused, never approximated as pinhole. Older captures missing this metadata cannot establish calibrated results.

RealSense records also retain alignment_calibration: the native depth and color intrinsics and the SDK’s depth-to-color extrinsics used for that exposure. Rotation uses the SDK’s column-major layout and translation is in metres. The SDK maps samples onto color pixels while copying their native depth-axis Z16 values. This metadata makes that distinction explicit; it does not alter captured pixels or apply a fitted depth offset.

The shared rgbd_geometry implementation uses that transform to intersect each color ray with its measured native-depth Z plane. Wrist and overhead scans, the stencil observer’s markers, the post-draw inspection’s views and recordings carrying this metadata use the resulting color-frame coordinates. Median captures retain and freeze the alignment model for the whole batch. Geometry without the native alignment metadata remains historical evidence; it cannot establish a new calibrated live surface. Calibration adoption is tatbot ros calib overhead --adopt and requires a consistent camera-to-robot registration.

Run logs contain per-pair transforms, bidirectional overlap, geometric residual, separate read/preparation/solver/output timings, CPU consumption and optional 5 Hz point-cloud RRD output through the shared writer. True material-point and rotation errors, and false acceptance/rejection counts, are available for the known-motion controls. Real recordings have no ground truth and are explicitly unlabelled. A low ICP residual can still accompany an incorrect pose, as the symmetry control demonstrates. Default thresholds are experimental comparison settings, not qualified drawing tolerances.

Registration starts from identity for each consecutive pair. It does not chain poses, recover persistent material identities, compensate wrist motion into robot coordinates, or model deformation. Segmentation and camera-to-robot calibration are separate requirements. Numerical thread pools are limited to one thread; timing includes both comparison methods and is not a hard real-time guarantee.

Optional --ground-truth FILE accepts JSON with schema tatbot.surface-rgbd-ground-truth/1, a provenance description of independent measurement, and frames containing timestamp_ns plus rigid 4x4 object_to_camera matrices in meters. Every evaluated timestamp must be covered. This enables physical-point RMS and rotation error for recordings; the benchmark cannot verify the accuracy of supplied measurements. Do not use ICP outputs as its own ground truth. Open3D is an optional offline dependency, pinned by the CLI; its tests skip explicitly when unavailable in the general test environment.

--image-check measures RGB optical flow once per pair, filters forward/backward and photometric inconsistencies, and compares the corresponding measured 3D points with each registration transform. It reports residuals and unavailable checks separately. These are correlated sensor measurements, not independent physical ground truth, and include depth noise and nearest-pixel quantization. They do not grant motion authority or change the existing candidate gates. The check retains only the current/previous images and reuses cached SDK rays.

--method colored runs only colored registration; the default compares both methods. --max-points 5000 reduces the deterministic input sample cap from 20000. Compare consistency and timing on the same recording before choosing a budget; fewer points are not automatically equivalent accuracy. The output records method selection, point cap and consistency-check settings.

tatbot vision depth compare compares depth filter settings offline on a retained raw capture burst.