Fallback Capture — Feasibility & Architecture Blueprint

Status: V1 IMPLEMENTED (2026-08-18) — in user-testing iteration. See "Implementation status & open items" at the bottom. Goal: Let users generate a fallback image for a creative directly in the builder, by navigating/scrubbing the creative to a chosen state (flow step, video position) and capturing a screenshot of the preview iframe.

---

Background

Ad servers require a static fallback image for environments where the interactive creative cannot run. Today users must produce this manually outside the platform. The idea: since the builder already renders the creative live at its exact dimensions, let users pick the best-looking state (often the end state with CTA button and text visible, or a specific video frame) and capture it in one click.

Two complementary user flows:

• A) Capture current state — user interacts with the preview normally (clicks through the flow), then captures what they see.
• B) Timeline scrubbing — a timeline viewer lets the user step/scrub through flow steps and video time, then capture. Branching flows are handled by letting the timeline represent the path taken, with the choice at each branch point selectable.

A plain video thumbnail is NOT sufficient: creatives layer blocks (text, buttons, graphics) on top of video, so the fallback must be a composite screenshot of the whole creative.

Current state of the codebase (verified 2026-08-18)

No fallback system exists. No data model, API field, upload UI, or tag emission for a fallback image anywhere in AF/CE/Backend. Nearest patterns:

• AR poster upload (Application-Frontend/.../configs/ArConfiguration.vue) — closest upload-UI pattern
• DOOH/dooh-feasibility.md (lines ~180-207, 284-286) already proposes a "Static Fallback Creative" as future work

Key existing building blocks:

Capability	Where	Notes
Same-origin preview iframe	Application-Frontend/src/pages/Chatbots/components/Visuals/Preview/LocalBuildPreview.vue (iframe src /preview-frame.html), src/preview-iframe.ts	Second Vite entry, same origin. Parent dispatches creative-data-update CustomEvent directly into frame.contentDocument. Full DOM access from parent.
Flow step-through bridge	Creative-Engine/src/devtools/flowBridge.ts → window.__cavaiDevtools	getFlowInfo(), getStepComponents(stepId), stepForward(), stepTo(nextStepId), toggleExpand(). Gated on DataStore.devMode, which the builder preview enables (preview-iframe.ts sets devMode: true).
Existing step UI (devtools)	Application-Frontend/src/components/DevTools/FlowControls.vue	Prev/next/jump, browseStep() (inspect without executing), executeStep(nextStepId) with stabilization polling. The scrubber is largely a productization of this.
Flow graph → step mapping	Application-Frontend/src/pages/Chatbots/components/Flow/utils/DataHelper.ts (mapStepsAndIDs)	BFS from startingComponent over operatorList + linkList; steps enumerable, branches = Question → multiple Answers, each with own nextStepId.
Engine flow execution	Creative-Engine/src/logic-system/processors/conversationFlow.ts	DataStore.componentData.components is FlowComponentInterface[][] indexed by step ID.
Video element	Creative-Engine/src/components/creative/CreativeVideoBlock/CreativeVideoBlock.vue	MP4 mode: native <video> with https://delivery-6.cavai.com/{streamId}/original. Same-origin iframe → parent can set currentTime directly for video scrubbing.
Server-side video frames	Bunny CDN: /{streamId}/preview.webp; videoThumbnailUrl() in Application-Frontend/src/utils/delivery.ts	Proves server-side frame extraction exists (single server-picked frame today).
Asset upload/storage	Asset library, Application-Backend (sharp for raster ops)	Reuse for storing the captured fallback.

What does NOT work (and why we avoid it):

• html2canvas (already a dependency, used in Reports export) re-renders the DOM rather than capturing real pixels: it cannot rasterize <video> or <model-viewer>/WebGL at all, and cross-origin CDN images without CORS headers taint the canvas (delivery-6.cavai.com has no confirmed CORS headers; <video>/<img> carry no crossorigin attribute).
• No headless-browser/screenshot infra exists in any repo (backend is AdonisJS with only sharp). Out of scope for this feature; only relevant later for bulk generation without a user in the builder.

Chosen capture approach: Screen Capture API + Region Capture

Real-pixel capture of the builder's preview iframe, sidestepping all CORS/taint issues:

1. navigator.mediaDevices.getDisplayMedia({ preferCurrentTab: true }) — one browser permission prompt ("Share this tab"). Captures the composited pixels: video frames, WebGL, HTML blocks, everything.
2. Region Capture (CropTarget.fromElement(iframeElement), Chrome 104+) crops the stream to exactly the preview iframe.
3. Grab a single frame from the stream (e.g. ImageCapture or draw the track to a canvas — this canvas is NOT tainted because capture is permission-gated), export as PNG/JPEG.
4. Upload as an asset / store on the creative.

Constraints & mitigations:

• Permission prompt per capture — keep the stream alive for the session so the prompt appears once.
• Chrome-only Region Capture — acceptable for the builder (internal tool). Fallback for other browsers: crop by iframe bounding rect × devicePixelRatio (fragile; defer).
• Resolution = displayed pixels — force preview zoom to 100% before capture so a 300×250 creative captures at its true size (retina gives 2x, a bonus). Check how zoom is applied (zoomlevel in buildCreative settings) and neutralize during capture.
• Occlusion: Region Capture captures the tab's composited output for that element region — verify nothing (tooltips, overlays, cursors) overlaps the iframe during capture; hide builder overlays first.

Timeline viewer (scrubbing)

Productize FlowControls.vue into a user-facing timeline docked with the preview:

• Flow axis: enumerate steps from getFlowInfo().totalSteps + getStepComponents(i). Timeline shows the current path: linear segments between branch points; at a Question step, render the answer choices — picking one calls stepTo(answer.nextStepId). Going backwards = reset (creative-data-update with _resetState) + replay the recorded choice sequence up to the target step.
• Video axis: when the current step contains a video block, show a time scrubber; parent sets videoEl.currentTime directly (same-origin). Pause playback while scrubbing.
• Determinism caveat: JS/CSS/ShowHide/Delay operators make pixels path-dependent — fine, because we scrub the live engine (state accumulates correctly along the replayed path). Time-based operators (Delay) may need a "settle" wait, mirroring executeStep()'s stabilization polling.

Data model & consumption (needs a decision)

• Store captured image as an asset linked to the creative (e.g. fallbackImageUrl on the creative, or a tagged asset in the asset library).
• Open question: where is the fallback consumed? Options: (a) downloadable asset in the Delivery page for manual ad-server upload (likely the actual need), (b) emitted in the tag (TagGenerator.ts currently emits no backup image). Start with (a).
• Multiple captures per creative should be possible (user picks/replaces).

Implementation phases

Phase 0 — Spike — RESULTS (2026-08-18, tested by Nicolay)

Spike lives as a temporary DevTools panel (FallbackCapture.vue, branch add-fallback-capture). Verified:

• Region Capture works ("Region crop: active" in user's Chromium browser). Capture is pixel-perfect, iframe-only. A 300×600 creative captured at 1050×2100 (3.5× from devicePixelRatio × browser zoom) — downscale to exact ad size on save.
• Flow stepping + branch choices work in the live preview via the flow bridge; reset via iframe reload works. Gotcha found: the browser keeps the same contentWindow proxy across iframe reloads, so prop watchers never fire — remount step-UI with a :key bumped on preview-iframe-ready.
• Manual-crop fallback implemented (crops iframe rect from full-tab frame) for browsers without Region Capture — not yet exercised in test.
• Permission UX decisions: the native share dialog cannot be styled/bypassed. Product version should AUTO-STOP the stream after each capture (fallback capture is a rare, deliberate action → one prompt per capture is fine, no lingering "sharing this tab" indicator, no Stop-sharing button needed). Keep-alive session only makes sense for rapid repeat captures (spike behavior).
• Video seek + capture: VERIFIED — captured frame matches the preview's video frame pixel-perfectly.

Phase 0 — Spike (validate the two risky primitives)

1. In the builder, run getDisplayMedia + CropTarget.fromElement on the preview iframe and export a PNG. Verify: video frame included, correct crop, correct dimensions at 100% zoom.
2. Drive __cavaiDevtools.stepTo() from a scratch button and confirm reliable navigation incl. reset + replay for backwards scrubbing.
3. Set currentTime on the preview's video element from the parent and confirm the frame renders before capture (wait for seeked event).

Phase 1 — Capture button (flow A)

• "Capture fallback" entry point in the product UI — NOT DevTools (the spike lives there temporarily). Likely home: the Delivery tab or the publish flow, where users already think about tags and export. Exact placement TBD with Nicolay.
• Capture flow: hide overlays → force 100% zoom → capture → preview result → save as asset.
• Storage: creative-level field + backend endpoint; download from Delivery page.

Phase 2 — Timeline viewer (flow B)

• Timeline component (path-based flow scrubbing + branch choice picking + video time scrubber), integrated with the capture button.

Phase 3 — Later / optional

• Cross-browser crop fallback; multiple-size batch capture; server-side headless rendering for bulk generation (see DOOH doc).

Future ideas spawned by this work (Nicolay, 2026-08-18)

The same-origin video control proven in the spike (setting currentTime from the parent) opens the door to bigger video tooling — not part of this feature, but worth separate issues later:

• In-builder video editor — crop and trim videos directly in the builder (visual crop rect + in/out trim points), instead of requiring pre-edited uploads. Server-side transcode (Bunny) would apply the edits.
• Play/stop video control in the flow — dedicated flow operators (or Library Script helpers) to play, pause, and seek the video block at specific flow steps, e.g. "play video when this step is reached, pause at choice".

Repos & conventions

• Changes land in Application-Frontend (timeline UI, capture, storage plumbing), Creative-Engine (possible flowBridge extensions, e.g. reset/replay helper, video seek helper), Application-Backend (fallback asset field/endpoint).
• Follow .claude/rules/coding-conventions.md in each repo (AF: i18n via $t, InputSelect, inputLocked propagation, sectionSettings entries; CE: JSS styling only, DataStore state, #include annotations, manual lint-save before commit).

Key files reference

• Creative-Engine/src/devtools/flowBridge.ts — step-through bridge
• Creative-Engine/src/logic-system/processors/conversationFlow.ts — flow execution
• Application-Frontend/src/components/DevTools/FlowControls.vue — existing step UI to productize
• Application-Frontend/src/pages/Chatbots/components/Visuals/Preview/LocalBuildPreview.vue + src/preview-iframe.ts — preview iframe + data channel
• Application-Frontend/src/pages/Chatbots/components/Flow/utils/DataHelper.ts — graph → step mapping
• Creative-Engine/src/components/creative/CreativeVideoBlock/CreativeVideoBlock.vue — video element
• Application-Frontend/src/utils/TagGenerator.ts — tag emission (if fallback ever goes into the tag)
• Cavai-Documentation/src/DocumentationTexts/DOOH/dooh-feasibility.md — prior fallback proposal

---

Implementation status & open items (2026-08-18, end of day)

V1 implemented via subagent-driven execution of the implementation plan — 7 tasks, per-task independent reviews, final whole-branch review, plus two user-test fix rounds. Branch add-fallback-capture (AF, ~13 commits) + add-fallback-capture (AB, 1 commit). Draft PRs open; NOT merged.

What works (user-verified)

• Delivery tab "Fallback image" section (empty + has-fallback states) opens the capture dialog without form-submit side effects
• Dialog renders its own centered LocalBuildPreview instance (unique iframe name; multi-instance guardrails in LocalBuildPreview)
• Timeline renders step nodes with current-step highlight; capture button enables after best-effort go-to-end
• Region Capture pipeline (from the verified spike) shared via src/utils/fallbackCapture.ts (unit-tested crop/scale/naming math)
• Backend skip_optimization flag keeps fallback uploads as real PNG (no WebP conversion)

Evening user-test iteration log (2026-08-18/19, all fixes pushed)

• Round 1: TheButton defaults type=submit → section buttons submitted the Delivery form (tab jump); runaway goToEnd on auto-advancing/looping creative left busy stuck; preview layout illusion of "two boxes". Fixed.
• Round 2: steps beyond branch points unreachable → BFS pathfinding over the step graph (findPath, 11 unit tests); branch chips never showed because currentStep went stale without a live poll → added 800ms poll. Commit 3bda7eea. USER-VERIFIED working.
• Round 3 (polish): two-pane stage (live preview | captured frame, equal size), rail-based step scrubber, single footer action bar. Commit c48c21114.
• Round 4: capture bled dialog UI into the image — stage CLIPPED the iframe while Region Capture crops the element's full viewport box (clipped region shows composited dialog content). Redesigned per Nicolay: modal fits viewport, DARK canvas (builder-like) instead of checkerboard, preview ALWAYS scale-to-fit (no clipping possible). Commit e0775c149. USER-VERIFIED clean capture.
• Round 5 (in flight at session end): captured image looked soft — (a) result pane displayed the already-downscaled file upscaled on retina → show raw captured canvas, downscale only on save; (b) capture happened at fitted scale → capture-mode expansion to native scale 1 (overflow-visible + z-index raise + scrollIntoView) when the creative fits the viewport. Commit Capture at native scale and show full-res result preview — VERIFY IN BROWSER next session.

Open items for next session

1. Verify round 5 quality fix in browser (capture sharpness at 100% scale; no bleed regression).
2. Choices UX rethink — Nicolay: "mulig vi bør bygge litt annerledes for choices". Current model: BFS auto-resolves branch choices on step click; chips as override. Consider making choices first-class in the timeline (branching lanes / per-branch picker).
3. Auto-advancing/looping creatives: goToEnd is best-effort with loop/restart guards; user captures the live creative. True state-freeze needs a CE flow-pause (flowBridge extension) — proposed v1.1.
4. Master creatives: should the section hide for masters (TagCode does)? Product decision pending.
5. End-to-end backend verification: PNG (not WebP) asset lands via skip_optimization (AB branch must run), persistence round-trip, Download file.
6. Deferred minors are logged in the SDD ledger: Application-Frontend/.superpowers/sdd/2026-08-18-fallback-capture/progress.md (gitignored, local to the original machine — key ones: double-toast on upload failure, resetPreview timeout polish, ConfirmDialog double-gating).

Key implementation files

• AF src/pages/Chatbots/components/Delivery/FallbackSection.vue — Delivery card
• AF src/pages/Chatbots/components/Delivery/FallbackCaptureDialog.vue — capture modal (own preview, capture/save flow)
• AF src/pages/Chatbots/components/Delivery/FlowTimeline.vue — timeline (pathfinding, chips, video scrubber)
• AF src/utils/fallbackCapture.ts (+ __tests__) — pure capture/crop/downscale/naming
• AF src/pages/Chatbots/components/Visuals/Preview/LocalBuildPreview.vue — iframeName prop + scoped ready-listener
• AB app/Validators/StoreAssetValidator.ts + app/Controllers/Http/AssetLibraryController.ts — skip_optimization
• Storage model: asset upload + creativeProperties.fallbackImage = { assetId, url, width, height, capturedAt } (no new endpoints)