Vision
Root image, codec, capture, media, viewer, and Vulkan Video values plus FnImage and FnDetection.
Examples
Vision
Functions
Classes
0 members
codec config parsed from av1C (AV1 ISO-BMFF §2.3): the configOBUs blob, which carries the sequence-header OBU out-of-band. AV1 MP4 samples are OBU temporal units that omit the sequence header, so we prepend this to every keyframe temporal unit (analogous to SPS/PPS for H.264/H.265).
Source0 members
codec config parsed from avcC (H.264): SPS + PPS as Annex-B bytes, ready to prepend to the first IDR packet. Also exposes the NAL length field width so readNextPacket can rewrite length-prefix → start codes.
Source15 members
Explicit completion and release boundary for producer-owned images and mapped ring slots still referenced by GPU consumers.
acquire and upload the newest available frame. Returns false when the camera has not produced another frame yet.
0 members
0 members
10 members
Host-visible creation path for sidecars, tests, and CPU integrations. Native GPU postprocess will use the same record layout and consumer API.
0 members
ISO-BMFF fragment defaults. `moov/mvex/trex` establishes this state; later `moof/traf/trun` boxes use it without retaining media payloads.
Source0 members
0 members
0 members
0 members
0 members
0 members
codec config parsed from hvcC (H.265): VPS + SPS + PPS as Annex-B bytes
Source0 members
Lightweight NAL unit descriptor — payload references back into the caller's bitstream buffer (no copy). For Annex-B framing the payload begins at the NAL header byte (skip the 00 00 00 01 / 00 00 01 prefix).
Source0 members
0 members
11 members
Stops the producer thread, completes exact frame-consumer dependencies, and releases portal, PipeWire, and vulkan resources.
Non-blocking. Returns true when a newer frame than the previous poll() is published through the common video-frame contract.
release a frame returned by poll(). The no-token overload declares that no asynchronous consumer remains.
0 members
0 members
Container information (probed from file header)
Source0 members
36 members
hardware video decoder session. Wraps VkVideoSessionKHR + DPB management.
21 members
Video stream demuxer
Explicit resource-release boundary. Reports local file close failures; pending packets are discarded and no container data is manufactured.
Returns SPS+PPS concatenated as Annex-B (empty if not parsed). Caller prepends to the first IDR packet to bootstrap the decoder.
index of the next sample readNextPacket() will read (0-based). After decoding sample N, this equals N+1. Used by Video for step-back.
0 members
Network/live-source policy. local files ignore reconnect and jitter fields. Live transports use the same bounded packet/reconnect policy as file-backed demuxers. Protocol support is capability-gated by OA rather than inherited from a process-global third-party media registry.
Source0 members
0 members
0 members
11 members
hardware video encoder session — wraps VkVideoSessionKHR for encoding.
Explicit completion and resource-release boundary. pending encoded output is discarded; call flush() first when the packets are required.
convert a buffer-backed RGBA texture and encode one frame through this session's owning engine. pending compute producers are completed before the encoder snapshots the source allocation.
0 members
Timestamped frame shared by decode, capture, processing, presentation and encode. resource handles are non-owning and remain valid for the lifetime contract of their producer.
4 members
Video frame resource pool (reuse decoded frames).
Source12 members
0 members
39 members
Explicit completion and resource-release boundary. Waits exact retained frame-consumer events before releasing decoder-owned image storage.
convert the current decoder-owned frame to an ML tensor. conversion lives on VideoPlayer rather than VideoFrame because the decoder session owns the image views, YCbCr metadata, synchronization and conversion resources.
Unified viewer transport timebase. Public timeline APIs use microseconds; container-specific ticks remain private to VideoPlayer::seek.
isDone() is permanently false in looping mode. in non-looping mode, flips to true once the underlying stream's EOS has been observed and the last decoded frame consumed.
Record the compute submission that most recently sampled currentFrame(). The session will not recycle that RGBA image until inConsumed completes. frames advanced without being rendered remain immediately reusable.
Advances by exactly one decoded frame (ignores playing_, ignores the wall-clock accumulator). Use tick() for time-paced playback. advance to the next display-order frame. open() presents the first frame, so the initial currentFrame() is immediately usable. next() reports decode, demux and synchronization failures instead of hiding them.
Read back the current converted RGBA frame. This is intended for CPU reference overlays and diagnostics; realtime inference should consume currentFrame() directly on the GPU.
Seek in the container video-track timebase. Ordinary seeks flush decoder state without destroying reusable RGBA allocations; genuine stream-format generation changes recreate the decoder.
Select an exact display-order frame. The caller owns play/pause policy; timestamp rounding remains confined to time-based seekUs().
Present the previous display-order frame. Cache-resident history does no decode work; a miss seeks to the preceding keyframe and rebuilds a bounded presentation window.
Scrub by a signed number of display-order frames. Positive movement advances normally; negative movement uses retained presentation history first and replays through the reorder path only on a cache miss.
0 members
0 members
0 members
14 members
0 members
3 members
Synchronous frame transcoder. The source packet is decoded on the vulkan video queue, converted to RGBA on compute, then submitted to the vulkan encoder without a CPU pixel round-trip. Container demux/mux remains the caller's responsibility.
Source0 members
Enums
Constant · Replicate · Reflect · Reflect101 · Wrap
Auto · Jpeg · Png · Webp · Bmp · Tga
Nearest · Bilinear · Bicubic
NV12 · RGB8 · RGBA8 · BF16 · F32
Hidden · Embedded
MonitorOrWindow · Monitor · Window
Bit8 · Bit10 · Bit12
Monochrome · Yuv420 · Yuv422 · Yuv444
H264 · H265 · AV1 · VP9
codec-standard profile. values are codec-qualified so a VideoProfile is self-describing without exposing vulkan std Video enums.
Unspecified · H264Baseline · H264Main · H264High · H264High444Predictive · H265Main · H265Main10 · H265MainStillPicture · H265FormatRangeExtensions · H265ScreenContentCodingExtensions · Av1Main · Av1High · Av1Professional · Vp9Profile0 · Vp9Profile1 · Vp9Profile2 · Vp9Profile3
Progressive · InterlacedInterleavedLines · InterlacedSeparatePlanes
ConstantQp · Cbr · Vbr
Decoded-frame resource path — selected once at decoder create() based on device capabilities + queue topology.
CoincidentFastStaging · DistinctComputeConvert · CoincidentComputeStaging · DirectCoincidentSampling · Unavailable
Auto · BT709 · BT2020
Functions
Binary mask counts [true-positive, false-positive, false-negative, true-negative]. Inputs are equally-shaped UInt8 masks.
Pairwise IoU for FP32 center-x/center-y/width/height boxes. inA [N,4], inB [M,4] -> FP32 [N,M]. Coordinates may be normalized or pixel-valued, but widths and heights must be non-negative.
Classification confusion matrix. rows are target classes and columns are predicted classes: Int32 [N], Int32 [N] -> UInt32 [C,C].
Dataset-level object-detection evaluation. Predictions are FP32 boxes [P,4], FP32 scores [P], Int32 classes [P] and Int32 image IDs [P]. targets are FP32 boxes [G,4], Int32 classes [G] and Int32 image IDs [G]. IoU thresholds are FP32 [T]. Matching is greedy by descending score, class-aware and constrained to the same image. classes without targets do not contribute to mAP. All outputs and internal scratch stay GPU-resident.
Multiclass semantic-segmentation metrics over flattened Int32 label matrices. labels outside [0,classCount) are ignored by the confusion accumulator, which provides the ordinary ignore-index behavior.
Deterministic class-aware NMS. boxes are FP32 [N,4] cx/cy/w/h, scores FP32 [N], and classes Int32 [N]. The implementation is GPU-resident and records as one graph node; it never sorts or compacts on the host.
Functions
adaptiveThresholdGaussian: Local Gaussian adaptive binary threshold.
adaptiveThresholdMean: Local mean adaptive binary threshold.
alphaBlend: Scalar alpha blend of equal-shaped images.
averageBlur: Normalized box blur lowered through the generic 2D filter.
bilateralFilter: Edge-preserving bilateral neighborhood filter.
brightnessContrast: Affine brightness and contrast adjustment.
centerCrop: Centered crop lowered through Crop.
channelReorder: Reorder one to four NCHW channels.
clamp: Clamp image values to a scalar interval.
colorTwist: Apply a shared 3x4 RGB color transform matrix.
composite: Per-pixel masked composite with one or C mask channels.
convolve2d: Same-shape NCHW 2D cross-correlation with an odd shared kernel and explicit border semantics.
crop: Crop NCHW image: out = A[y:y+h, x:x+w].
Stateless still-image codec boundaries. Decode uploads one normalized Float32 NCHW image through the active engine. Encode and saveFile are explicit synchronous host boundaries which complete and read back their semantic image input.
dilate: Rectangular NCHW dilation using the shared border contract.
erase: Fill a rectangular region with a scalar value.
erode: Rectangular NCHW erosion using the shared border contract.
flip: Flip NCHW image horizontally or vertically.
gammaContrast: Gamma and gain adjustment.
gaussianBlur: Gaussian blur filter with configurable kernel size and sigma.
gaussianNoise: Add deterministic GPU-native Philox Gaussian noise.
grayscale: Rec.709 RGB to single-channel grayscale conversion.
inRange: Inclusive scalar range mask.
invert: Invert values around a configurable maximum.
laplacian: Signed four-neighbour discrete Laplacian lowered through the generic 2D filter.
medianBlur: Odd-window median filter with explicit border semantics.
morphologyBlackHat: Morphological closing minus input.
morphologyClose: Dilation followed by erosion through the reusable morphology primitive.
morphologyGradient: Difference between dilation and erosion.
morphologyOpen: Erosion followed by dilation through the reusable morphology primitive.
morphologyTopHat: Input minus morphological opening.
normalize: Apply per-channel normalization to an NCHW image tensor.
pad: Asymmetric NCHW padding with the shared border contract.
posterize: Quantize an intensity interval to a fixed number of levels.
remap: Absolute-coordinate NCHW remapping with nearest or bilinear sampling.
resize: Resize an NCHW image using nearest or bilinear interpolation.
rotate: Rotate NCHW image by 90/180/270 degrees.
saltPepperNoise: Apply deterministic GPU-native salt-and-pepper noise.
Synchronous host sink for one exact packed RGBA8 image. This is the encoder boundary used by image-backed render sessions after their own explicit readback/completion protocol has produced host bytes.
Synchronous file sink for a device-local packed RGBA8 texture. pending work in the engine's matching private recorder completes before readback. The output codec is inferred from .png, .jpg/.jpeg, .bmp, or .tga.
scharr: Signed 3x3 Scharr x or y derivative lowered through the generic 2D filter.
Blend an Int32 semantic label map over an FP32 NCHW RGB/RGBA image. mask is [N,H,W] or [N,1,H,W], palette is FP32 [K,3], and invalid labels leave the source pixel unchanged.
separableConvolve2d: Same-shape NCHW separable cross-correlation with odd horizontal and vertical kernels.
sharpen: Parameterized 3x3 sharpening lowered through Convolve2d.
sobel: Signed 3x3 Sobel x or y derivative lowered through the generic 2D filter.
solarize: Invert values above a threshold.
thresholdBinary: Global binary threshold.
thresholdBinaryInv: Global inverse binary threshold.
thresholdToZero: Set values at or below a threshold to zero.
thresholdToZeroInv: Set values above a threshold to zero.
thresholdTruncate: Truncate values above a threshold.
unsharpMask: Gaussian unsharp masking composed from graph-native primitives.
warpAffine: Inverse 2x3 affine warp with explicit interpolation and border semantics.
warpPerspective: Inverse 3x3 perspective warp with explicit interpolation and border semantics.
Ui
Functions
Classes
0 members
0 members
4 members
─── CvFrame ────────────────────────────────────────────────────────────────
0 members
─── CvOverlay ──────────────────────────────────────────────────────────────
Source8 members
0 members
0 members
10 members
0 members
0 members
One source-anchored glyph quad. anchor is normalized to the source image; offsets and dimensions are compose-target pixels so labels remain readable while the image camera pans and zooms.
Source11 members
upload caller-provided planes. inPlanes and inDtypes must have the same length (1–4). Each span is W*H * sizeof(dtype) bytes. Synchronous.
Decode any stb_image-supported file to planar U8. 1-channel files → gray (channelCount=1). 3-channel files → RGB (channelCount=3). 4-channel files → RGBA (channelCount=4). HDR (.hdr) → planar F32. synchronous (transfer completes on return).
11 members
─── InputSystem ────────────────────────────────────────────────────────────
YAML schema: {version: 1, bindings: [{action, key, modifiers, context, allow_repeat}]}. Loading is transactional for syntax and schema errors. Without yaml-cpp these calls report Unimplemented, never success.
register the reusable Realm tool bindings. This installs named actions with no callbacks; the application owns every operation they trigger: F12 — screenshot ctrl+R — start/stop recording Space — toggle camera ctrl+Z — undo (node graph) ctrl+Y — redo ctrl+S — save F — fit-to-view (node canvas)
0 members
─── KeyAction ──────────────────────────────────────────────────────────────
Source1 members
─── KeyBinding ─────────────────────────────────────────────────────────────
Source14 members
0 members
0 members
0 members
─── Caller-borrowed hit-test records ────────────────────────────────────────
Source0 members
0 members
─── TextLayout ───────────────────────────────────────────────────────────── CPU-side layout: strictly decodes UTF-8, selects explicit per-codepoint font fallback, and uses HarfBuzz for OpenType substitution, positioning, and source-byte clusters over OA's embedded left-to-right coverage. Paragraph BiDi and right-to-left script coverage are intentionally outside this API.
Source8 members
─── TextAtlas ────────────────────────────────────────────────────────────── Manages the device-side coverage atlas and the glyph metadata table.
Selects the nearest generated physical-pixel strike. Returns nullptr for unmapped codepoints or an invalid font/size.
2 members
0 members
0 members
Immutable generated artifact handed from a training/evaluation producer to the viewer. The texture allocation is shared; the timeline token prevents presentation until the producing GPU batch has completed. Replacing a pending frame drops it without ever blocking the training owner.
Source0 members
13 members
Inherits public ViewerLiveSource
Source74 members
─── Ui ──────────────────────────────────────────────────────────────────────
valid after widgets are submitted and until the next beginFrame. This is a flat, caller-borrowed platform-adapter snapshot, not an OA-owned desktop or document tree. bounds are ancestor-clipped physical pixels.
The viewport is required only by top-layer popup/tooltip placement. Viewer supplies its live compose extent. content scale converts the init style's logical geometry into physical pixels; event coordinates and explicit PixelRect values remain physical. Non-finite/non-positive scale is a frame error.
beginPopup/endPopup render through the same deferred UI compositor, but their commands are submitted after every base-layer command. Popup calls must use the same parent scope as openPopup.
Explicit rows hug measured item widths and advance with layout.gap. Column items otherwise stretch across the panel's padded inner width.
Scroll panels retain only interaction/offset state under their stable ID. content coordinates are shifted and every nested draw/hit-test is clipped to the viewport. inContentHeight includes any layout padding. The caller owns content data; endScrollPanel must close the matching begin call.
Borrows the Viewer-owned atlas and allocates completion-tracked dynamic glyph slots. Bind once before the first frame; the atlas must outlive Ui.
Explicit renderer completion and release boundary. Waits every exact frame completion accepted by markFrameSubmitted before releasing pipelines, descriptor slots, and transient upload storage.
Selection is caller-owned. Empty lists, invalid selection indices and invalid popup metrics are reported through recordRender.
One reusable viewport/chart/canvas grid. The default decimal hierarchy is 10-unit minor, 100-unit major, 1000-unit super-major, plus black axes and a subtle theme-derived vertical background gradient.
Host values → heatmap through the same bounded frame-safe upload ring as PlotLine. Intended for compact metric tables and recorded oa::plot figures.
Matrix convenience overload. rows/cols and value type are inferred when omitted, and matrix byte offsets are honored for views.
texture values are engine-checked before recording. Buffer-backed shared leases are retained through the exact markFrameSubmitted completion.
Planar path: blitPlanar.slang handles per-channel dtype conversion + sRGB. inPlanes must remain at a stable address through markFrameSubmitted; that call attaches the exact consumer event used by its non-blocking RAII path.
source stage/access describe the producer's last write made visible to this sampled read. Defaults preserve compute storage-image producers; graphics render targets must pass their color-attachment scopes.
Called once after init, before the first frame. inComposeImageView: vkImageView (as void*) of the compose storage image (set=1, slot 0).
Single-line UTF-8 editor. Returns true only when inOutText changes. The caller owns the value; the field retains bounded local edit history and IME composition state. It supports single/double/triple-click caret/word/all selection, shift and word navigation, scalar-safe deletion, clipboard, ctrl/Cmd+Z redo variants, Home/End, horizontal scrolling, visible native pre-edit and committed UTF-8 input. An external value replacement rebases local undo history; document-level undo remains application policy.
label does not soft-wrap; text wraps to the current panel's inner width. Both honor explicit newlines, require an active panel and a bound text atlas, and report misuse through recordRender instead of dropping text.
Marks transient resources sampled by this frame. Plot buffers are recycled only after the exact engine-owned graphics completion is reached.
draw one adaptive world-space canvas grid in one compute dispatch. Canvas screen coordinates are local to inRect; colors follow the current style.
at most one interactive popup owns pointer/keyboard input. The no-anchor overload anchors to the most recently submitted interactive item. The explicit overload supports context menus and application-defined anchors.
CPU explicit-X/Y metric arrays -> one GPU polyline dispatch. The shader handles arbitrary segment direction without one dispatch per segment.
Finite fractions are clamped to [0, 1]. An empty overlay renders a rounded integer percentage; an explicit overlay is centered without soft wrapping.
Draws a passive compact label/value row. Empty values leave the returned Value rectangle available for a caller-owned editor control.
draw filled or outlined axis-aligned rectangles directly into the GPU compose image. Rectangles must already be clipped to its extent.
draw normalized rectangle records from a bindless GPU buffer in one dispatch. inDstRect maps source-image coordinates into the compose image.
Sliders clamp the referenced value into [min, max] and return true only when it changes. Focused sliders accept arrow-key adjustment (1% of the range, shift=10x, ctrl=0.1x); pointer drags remain captured outside bounds.
Divides an explicit rectangle without beginning either child panel. The caller owns and may persist inOutRatio; it is the first region's fraction of the extent remaining after handleSize. Pointer drags retain capture outside the handle. Focused row splits accept Left/Right, and focused column splits accept Up/Down; shift/ctrl use the shared coarse/fine scale.
Draws one caller-owned tab sequence. Selection and overflow position live in inOutState; close and reorder are returned as requests so OA never owns documents or mutates their order. Left/Right/Home/End navigate the focused sequence and ctrl/Cmd+W requests closure of a closable focused tab.
Returns and clears a focused editor's pending copy/cut payload so the platform layer can publish it without introducing SDL into Ui.
Positioned GPU text primitive for plots, overlays, and other fixed-layout drawing. It does not participate in panel flow; inRect is an absolute compose-space clip/alignment rectangle. Stretch alignment is invalid.
Explicit transport timeline. inOutFraction is normalized to [0, 1]; returns true when pointer scrubbing changes it.
Attaches to the immediately preceding interactive item. tooltip geometry is passive and shares the top-layer command route with popups.
One row of a caller-flattened hierarchy. The caller owns open/selection state and applies open when openChanged is returned. Pointer activation on the disclosure toggles only; activation elsewhere selects. Focused tree rows use Up/Down to select among prior rendered rows, Left/Right to close or open, and Return/Space to activate. Explicit rectangles make the row directly compatible with virtualRows.
Returns the visible half-open row range for the active scroll panel. rows start after layout.padding.top and use inRowHeight + inRowGap stride. Overscan is expressed in rows and is clamped to the item count.
0 members
3 members
0 members
0 members
1 members
─── UiInputState ──────────────────────────────────────────────────────────── Stable per-frame snapshot updated by Ui::beginFrame before widget dispatch.
Source0 members
0 members
0 members
─── Widget config structs ────────────────────────────────────────────────────
Source0 members
0 members
0 members
0 members
0 members
0 members
0 members
0 members
5 members
0 members
0 members
0 members
0 members
0 members
0 members
0 members
0 members
24 members
Windowed or headless application session for inspecting OA images, video, audio and live GPU output. The Viewer owns UI, input, window and presenter state. It either creates one engine for run() or borrows the caller's engine for run(oa::Engine & ); it never creates a second device runtime in that path.
Unified blocking preview front door. Paths are decoder-probed as image, video, then audio and may either own an application engine or borrow the caller's presentation-capable engine. Direct GPU values always borrow the caller's engine and retain the exact completion contract of show().
run the windowed session against an existing presentation-capable engine. The viewer borrows the engine and closes only its presenter/window state.
Headless sink for a ready buffer-backed texture. Image-backed renderer frames use their owning session's saveTo/consumeReadback contract.
Compatibility spelling for blocking one-shot display sinks. Matrix and image overloads record the RGBA8 conversion into the engine's matching private recorder, submit it once, and pass its exact completion into presentation. New code uses preview() so file-backed and direct values share one discoverable verb.
Examples
0 members
0 members
10 members
Non-owning live producer attached to the one Viewer application lifecycle. It may render domain-specific overlays, but it never owns the window, swapchain, input pump or presentation submission.
24 members
Viewport — Passive view description for rendering Declares: camera, target, viewport, and view-specific settings. Does NOT: handle input, own windows, manage swapchains, or animate.
═══════════════════════════════════════════════════════════════════════ clear values ═══════════════════════════════════════════════════════════════════════
═══════════════════════════════════════════════════════════════════════ layer filtering ═══════════════════════════════════════════════════════════════════════
═══════════════════════════════════════════════════════════════════════ Render target (required) ═══════════════════════════════════════════════════════════════════════
2 members
ViewportDesc — viewport rectangle and depth range.
Source3 members
─── WorldAabb ──────────────────────────────────────────────────────────────
SourceEnums
U8 · U16 · F32 · BF16
Global · NodeCanvas · TextInput · Timeline
═════════════════════════════════════════════════════════════════════════════ RenderLayer — layer/filter flags for selective rendering ═════════════════════════════════════════════════════════════════════════════
None · Default · Ui · Overlay · Background · Debug · All
None · Focus · Activate · Toggle · Increment · Decrement · SetValue · Close
─── Accessibility export ───────────────────────────────────────────────
Button · Checkbox · Slider · TextField · ComboBox · MenuItem · Tab · TreeItem · Splitter · Timeline
None · Disabled · Focused · Checked · Selected · Expanded · Editable · HasPopup
Start · Center · End · Stretch
Previous · Next
─── UiEventType ─────────────────────────────────────────────────────────────
None · MouseMove · MouseDown · MouseUp · MouseScroll · KeyDown · KeyUp · KeyChar · WindowResize · WindowClose · WindowFocus · WindowBlur · Pinch · TextEditing
─── UiKey ─────────────────────────────────────────────────────────────────── SDL3 scancode subset. values deliberately match SDL_Scancode to allow direct cast; do not rely on the numeric values in switch logic.
Unknown · A · B · C · D · E · F · G · H · I · J · K · L · M · N · O · P · Q · R · S · T · U · V · W · X · Y · Z · Num1 · Num2 · Num3 · Num4 · Num5 · Num6 · Num7 · Num8 · Num9 · Num0 · Return · Escape · Backspace · Tab · Space · Minus · Equals · Comma · Period · Slash · Home · Delete · End · Right · Left · Down · Up · F1 · F2 · F3 · F4 · F5 · F6 · F7 · F8 · F9 · F10 · F11 · F12 · KpEnter · Kp1 · Kp2 · Kp3 · Kp4 · Kp5 · Kp6 · Kp7 · Kp8 · Kp9 · Kp0
─── UiPinchPhase ────────────────────────────────────────────────────────────
None · Begin · Update · End
─── UiScrollGesture ─────────────────────────────────────────────────────── Classified once in the input layer (SDL wheel heuristics). Navigation maps each gesture to its own sensitivity — wheel zoom, touchpad pan, pinch zoom.
None · MouseWheel · TouchpadPan · PinchScroll
Fill · Fixed · Hug
LeftToRight · BottomToTop
Waveform · Spectrum · Mel
Auto · Image · Video · Audio · Live
Image2D · Scene3D · Matrix · Video · CameraStream