AI Assistant System

The AI Assistant is an in-builder chat interface that lets users modify creatives through natural language. It uses Claude (Anthropic) with agentic tool use to interpret user intent and apply changes to the creative JSON blob.

Architecture Overview

User (chat panel)
    |
    v
Frontend (AIAssistantPanel.vue)
    |  POST /ai-assistant/chat
    v
Backend Controller (AiAssistantController.ts)
    |  Fetches creative, authorizes, parses blob
    v
AI Service (AiAssistantService.ts)
    |
    +-- System prompt (principles + creative structure)
    +-- Tool definitions (patch_block, add_block, etc.)
    +-- Agentic loop:
    |     1. Send messages + tools to Claude
    |     2. Claude returns tool_use blocks
    |     3. Validate input (validateToolInput)
    |     4. Execute tool (executeTool) -- mutates blob in-place
    |     5. Return tool results to Claude
    |     6. Repeat until Claude stops calling tools (max 12 iterations)
    |
    v
Response: { message, actions, affectedBlocks, usage, clearHistory? }
    |
    v
Frontend stores message + navigable block badges
    |  If blobUpdated: emits 'creative-updated' to reload preview
    |  If clearHistory: clears all chat messages (fresh start)
    v
Builder re-renders with updated creative

Key Files

File	Repo	Role
app/Services/AiAssistantService.ts	AB	Core service: system prompt, tools, validators, executor, agentic loop
app/Controllers/Http/AiAssistantController.ts	AB	HTTP endpoint: validation, auth, blob parsing, response formatting
src/services/aiAssistantService.ts	AF	Axios API client
src/store/modules/aiAssistant.ts	AF	Vuex module: messages, loading, token usage, cost estimation
src/components/AIAssistant/AIAssistantPanel.vue	AF	Chat UI: messages, input, affected block badges, reset button

Data Flow

Request

1. User types a message in AIAssistantPanel.vue
2. Component dispatches aiSendMessage Vuex action with { message, creativeId }
3. Store action:
   • Adds user message to state.messages
   • Builds history from state.messages excluding the just-added message (slice 0 to -1), filtered of error messages, last 10
   • Calls aiAssistantService.sendMessage(message, creativeId, history)
   • The current message is sent both as message (top-level) and excluded from history to avoid duplication
4. Backend controller:
   • Validates payload (message, creativeId, optional history)
   • Fetches creative from DB, authorizes via bouncer
   • Parses creativeBlob JSON
   • Calls AiAssistantService.chat(blob, message, history)

Processing (Agentic Loop)

1. Service builds system prompt with creative structure summary
2. Sends to Claude with tool definitions
3. Claude responds with text and/or tool_use blocks
4. For each tool call:
   • Validate input (e.g., reject <script> in HTML blocks, reject deleting baseProperties)
   • Execute tool (mutate blob in-place)
   • Track action result and affected blocks
5. Tool results are sent back to Claude for the next iteration
6. Loop ends when Claude stops calling tools or max iterations (12) reached

Response

1. Service returns { message, actions, affectedBlocks, usage, clearHistory? }
2. Controller checks if any action succeeded -- if so, saves updated blob to DB
3. Frontend receives response:
   • Adds assistant message with affected block badges
   • Accumulates token usage for cost display
   • If blobUpdated, emits creative-updated so the builder reloads
   • If clearHistory, clears all messages and shows only the new response (prevents old context from contaminating new requests after reset)

Tools

High-level block tools

Tool	Purpose
set_background	Set the creative background color. Handles backgroundSettings plumbing internally
add_text	Add a text block with styling. Auto-contrasts text color with background
add_button	Add a button block with label and colors
add_group	Create a flexbox group for organizing blocks
add_script	Add a Tag operator with JavaScript code. Sets branding label on the operator
add_css_effect	Add a CSS-only HTML block (0x0, invisible) for animations/effects

High-level flow tools

Tool	Purpose
add_statement	Add a statement/question to the flow. after param chains operators
add_answer	Add an answer choice linked to a statement. Multiple answers fan vertically
add_delay	Wait for a specified time (milliseconds) before continuing to the next operator. Use for timed sequences
add_showhide	Show or hide blocks at a point in the flow. Targets use abbreviation format (t1, b1, g1)
patch_showhide	Add or update targets on an existing show/hide operator. Enables simultaneous show+hide actions without creating a new operator

Low-level tools

Tool	Purpose
patch_block	Update raw properties on an existing block via dot-notation. Deep-merges objects
remove_block	Delete a block (top-level or sub-block)
inspect_block	Read-only: returns block properties (filters verbose defaults)
reset_creative	Reset to clean state: remove all blocks and operators, reset base background

Removed tools (replaced by high-level tools)

• add_block -- replaced by add_text, add_button, add_group, add_css_effect
• add_operator -- replaced by add_script, add_statement, add_answer, add_showhide
• patch_operator -- not needed with high-level tools
• remove_operator -- not needed; reset_creative handles cleanup

Chain-Linking

When operators are added to the flow, they automatically link to the tail of the existing chain instead of all linking from the starting operator. This means three operators A, B, C chain as A -> B -> C (not A -> B, A -> C).

The findChainTail function finds the last operator with no outgoing links and links the new operator there. This is critical for sequential flows like timed reveals (showhide -> delay -> showhide) where each step must follow the previous one.

When after is specified (e.g., on add_answer), the operator links to that specific operator instead of the chain tail.

Quiz Layout

The offsetOperatorPosition function positions operators in a quiz-friendly layout:

• Statements go horizontally: each new statement is placed to the right of all existing operators
• Answers fan vertically to the right of their parent statement, with 170px vertical spacing between answers
• Generic operators chain horizontally to the right

This produces a clean layout like: S1 -> [A1, A2, A3] -> S2 -> [A4, A5, A6]

Intent Detection

Simple reset commands ("slett alt", "delete everything", "reset", etc.) bypass Claude entirely and execute reset_creative directly. This is 100% reliable, costs 0 tokens, and handles the most common failure mode (Haiku skipping the reset tool call).

Pattern: /^\s*(slett alt|delete everything|start over|reset|...)\s*[.!?]*\s*$/i

Auto-Contrast

When add_text is called without an explicit color, the service checks the base background luminance and auto-sets text to white (#FFFFFFFF) on dark backgrounds or black (#000000FF) on light backgrounds.

Color Normalization

All color inputs are normalized to 8-character hex with alpha: normalizeHexColor("1a1a1a") -> "#1A1A1AFF". Handles 3, 6, and 8 character formats with or without # prefix.

Operator Branding

AI-created operators get branding labels and colors to visually identify what the AI added in the flow UI:

Operator type	Color	Label source
Script (tag)	#7C3AED (purple)	displayName
Statement	#2563EB (blue)	Indexed: "Sporsmaal 1", "Sporsmaal 2"
Answer	#16A34A (green)	Indexed: "Svar 1", "Svar 2"
Show/Hide	#D97706 (amber)	displayName

Statements and answers use indexed labels instead of content text to avoid redundancy (the content is already visible in the operator body).

Validators

Validators run before tool execution and reject invalid input with an error message that Claude can use to self-correct.

Validator	Rule	Rationale
No <script> in HTML blocks	Rejects add_css_effect and patch_block with <script> in html content	JS must go in Tag operators via add_script
Cannot delete baseProperties	Rejects remove_block for baseProperties	Base block is required, use patch_block to modify

Validators are intentionally lightweight -- they enforce platform constraints, not design preferences. The system prompt handles design guidance.

System Prompt Design

The system prompt is concise (~300 tokens) and covers:

• Block tools: what each tool does in one line
• Flow tools: statement, answer, showhide, patch_showhide
• Flow patterns: quiz, reveal, chaining, simultaneous show/hide via patch_showhide
• Principles: minimal changes, inspect before patch, hex colors
• Scripts: DOM-only manipulation, CSS class abbreviations, IIFE wrapping

The prompt does NOT prescribe specific animation durations, design styles, or include lengthy examples. It stays general-purpose and token-efficient.

Block Defaults

Block defaults in the service mirror the frontend's defaults.ts. They provide complete default property sets when creating new blocks, so the AI only needs to specify overrides.

Types with defaults: textProperties, buttonProperties, graphicProperties, htmlProperties, blockGroupProperties.

Operator defaults: statement, answer, showHide, tag (with full flow JSON structure including inputs/outputs). All share a common base via operatorBase() helper.

Creative Structure Summary

Before each conversation, the service generates a text summary of the creative's current state:

## Blocks
- textProperties-1 [textProperties] "Heading": "Welcome"
- graphicProperties-1 [graphicProperties] "Logo"
- blockGroupProperties-1 [blockGroupProperties] "Content Group"
  - textProperties-1 [textProperties]: "Subtitle"

## Flow operators
- statement_0 [statement] "Sporsmaal 1": "What is your favorite color?"
- answer_0 [answer] "Svar 1": "Red"
- answer_1 [answer] "Svar 2": "Blue"
- tag_0 [tag] "Countdown Script"
- showHide_0 [showHide] "Show Results"

## Flow connections
- statement_0 -> answer_0
- statement_0 -> answer_1
- answer_0 -> tag_0

This gives Claude context about what exists without sending the full blob. Operator text content is included as a preview.

CSS Class Name Mapping

The engine renders each block with a short CSS class:

Block ID	CSS class
textProperties-1	.t1
buttonProperties-2	.b2
graphicProperties-1	.g1
htmlProperties-1	.h1
blockGroupProperties-1	.gp1

Pattern: first letter(s) of type + number. CSS-only HTML blocks use these classes with #creative-container prefix for specificity.

Token Usage and Cost Tracking

The frontend tracks cumulative token usage across the chat session:

• Input and output tokens accumulated per request
• Cost estimated using Haiku 4.5 pricing ($0.80/M input, $4.00/M output)
• Displayed as dollar amount in the panel header with color-coded progress bar
• Reset when user clicks the "new chat" button

Affected Blocks

Each tool execution can produce an AffectedBlock with { blockId, displayName, action } where action is "created", "modified", or "removed". These are:

• Deduplicated (same blockId only shown once)
• Displayed as clickable badges below the assistant's response
• Color-coded: created=purple, modified=green, removed=red (with strikethrough)
• Clicking navigates to that block in the builder sidebar

Message Construction (Backend)

The backend service builds the Claude API messages array like this:

const messages: Anthropic.MessageParam[] = [
  ...history.map(msg => ({ role: msg.role, content: msg.content })),
  { role: 'user', content: message },  // current message appended last
]

The system prompt is built once per request:

const systemPrompt = `${SYSTEM_PROMPT}\n\n## Current creative structure\n${structureSummary}`

Where structureSummary is generated from the CURRENT blob fetched from DB (not from history). This means Claude always sees the actual state of the creative, regardless of what history says.

Important: The structure summary is built once at the start and NOT updated between agentic loop iterations. If Claude calls tools that change the structure, subsequent iterations still see the original summary. This is acceptable because tool results confirm success/failure, and the loop rarely needs more than 1-2 iterations.

Agentic Loop Details

while (iteration < MAX_ITERATIONS) {
  const response = await client.messages.create({ system, tools, messages })

  // If no tool calls -> extract text, return
  if (toolUseBlocks.length === 0) return result

  // Execute each tool call
  for (const toolUse of toolUseBlocks) {
    const action = executeTool(blob, toolUse.name, toolUse.input)
    // Tool results: { success: true } or { success: false, error: "..." }
  }

  // Append assistant response + tool results to messages
  messages.push({ role: 'assistant', content: response.content })
  messages.push({ role: 'user', content: toolResults })

  // If stop_reason is 'end_turn', Claude is done
  if (response.stop_reason === 'end_turn') return result
}

Key behaviors:

• Claude may return BOTH text and tool_use blocks in a single response
• When stop_reason is tool_use, the loop continues (Claude expects tool results)
• When stop_reason is end_turn, the loop returns even if tools were used (they've been executed)
• reset_creative sets action.success = true but does NOT set affectedBlock (no badge shown in UI)
• When reset_creative succeeds, the response includes clearHistory: true so the frontend clears all messages

Resolved Issues

"Slett alt" (delete everything)

Problem: Haiku sometimes responded with text claiming it reset the creative without actually calling the tool.

Solution: Backend-side intent detection. Simple reset messages are matched by regex and execute reset_creative directly, bypassing Claude entirely. 0 tokens, 100% reliable.

Duplicate message bug

The store action was adding the current user message to state.messages before building history, causing the same message to be sent twice. Fixed by slicing history to exclude the last message.

White text on white background

add_text now auto-contrasts text color with the base background using hexLuminance().

Link format mismatch

Links must use { fromOperator, toOperator } (not { from, to }). The frontend's DataHelper.mapStepsAndIDs expects these exact property names.

History contamination after reset

After reset_creative, old conversation history caused the AI to reference blocks that no longer exist. Fixed by returning clearHistory: true from the backend when reset is used, causing the frontend to clear all messages and start fresh.

Operators all linking to starting operator

Three operators A, B, C were linking as A -> B, A -> C instead of A -> B -> C. Fixed with findChainTail that finds the last operator with no outgoing links and chains from there.

Known Limitation: Haiku Hallucinating Tool Calls

Haiku sometimes claims it performed an action (e.g. "Fjernet! HTML-blokken er slettet") without actually calling the tool. This is visible when:

• No affected block badge appears in the response
• The block/operator is still present after the response

This affects remove_block and patch_block most -- the AI says it did the thing but skipped the tool call. reset_creative is solved with intent detection, but per-block operations are harder to detect deterministically.

Why Haiku and Determinism

We use Claude Haiku 4.5 for cost efficiency. Haiku is fast and cheap, but less reliable at following complex instructions compared to Sonnet or Opus. This makes determinism doubly important:

1. Reliability: High-level tools with built-in defaults eliminate entire categories of mistakes. Instead of asking Haiku to construct a complete operator JSON, we give it add_statement with a text param and handle all the plumbing.

2. Cost: Deterministic tools mean fewer iterations, fewer tokens, lower cost. A quiz that would take Sonnet 2-3 iterations with low-level tools takes Haiku 5-6 iterations with high-level tools -- but the per-token cost is so much lower that it still wins.

3. Upgradability: If we later offer a premium tier with Sonnet, the same high-level tools work even better. The determinism doesn't hurt a smarter model, it just makes it faster.

Known Issue: Flow Refresh

After the AI modifies flow operators, the flow view in the builder does not always reflect the changes immediately. Users need to refresh the flow panel to see updates. This should be fixed by emitting a flow-specific refresh event alongside creative-updated.

Future Work

Engine Capabilities Not Yet Taught

The engine supports far more than what the AI assistant currently knows about. Based on a scan of the Creative Engine codebase, here's what we need to add:

Block types missing from AI tools:

• graphicProperties -- add_graphic tool for images (background image URL, sizing, border)
• videoProperties -- add_video tool (streamId, autoplay, layout mode, pauseOutOfView)
• formProperties -- add_form tool with sub-block inputs (text, email, dropdown, radio, checkbox, phone, etc.)
• conversationProperties -- add_conversation tool (speed, typing animation, sender icon, scroll behavior)
• sliderProperties -- slider/carousel with slides, swipe, auto-scroll
• arProperties -- AR block (iOS/Android model URLs, placement)

Operator types missing from AI tools:

• change -- dynamically update block content at runtime (changeImage, changeText, changeUrl, changeVideo)
• redirect -- navigate to URL
• css -- inject CSS rules into the page at a flow step
• reset -- clear all flow state and visual overrides
• restart -- full reset + restart from first operator with fade transition
• payment -- payment flow integration (placeholder)

Animation system:

• Blocks support animations with effects: fade, slideX, slideY, scale, rotate, blur, skew
• Triggers: appear, hover, click
• Config: duration, delay, loop, iterationCount, direction, easing
• Keyframes: arbitrary percent-based keyframes per effect
• The AI should be able to add animations to blocks via patch_block or a dedicated add_animation tool

Rich text:

• Text and button blocks support ProseMirror JSON rich text (bold, italic, underline, strike, fontSize, fontWeight, color)
• The engine converts ProseMirror JSON to HTML via jsonToHtml()
• HTML sanitization whitelists: strong, em, u, s, p, br, span with font-size/font-weight/color
• The AI should understand how to set richText properties

CSS capabilities per block:

• Background (color, gradient with stops, image)
• Border (per-side width, style, color)
• Box shadow (multiple, with x/y/blur/spread/color/inset)
• Filters (blur, brightness, contrast, grayscale, saturate, hueRotate, invert, sepia)
• Backdrop filter (same set, applied to background)
• Text shadow (x, y, blur, color)
• Transform (rotate, scale, translateX, translateY)
• Custom CSS string per block
• Custom fonts (URL + font-family)

Button/form states:

• Buttons have hover, active states with separate styling and transitions
• Form inputs have hover, focus, disabled states
• State transitions: duration, easing, delay

Script capabilities:

• Full DOM access, no sandboxing
• Global functions: changeImage, changeText, changeUrl, changeVideo, showElement, hideElement, resetCreative, restart
• Feed functions for slider: changeFeedFilter, changeFeedParam, changeFeedSortingKey, changeFeedSortingOrder, changeFeedUrl

Example Solutions to Teach

Common patterns people need that we could provide as examples or deterministic tools:

• Clip-path animations: CSS clip-path with keyframes for reveal effects
• Quiz with scoring: JavaScript that tracks correct answers, stores score, shows result at end
• Countdown timer: JavaScript countdown to a date with live display
• Parallax scrolling: CSS/JS scroll-based animation effects
• Typewriter effect: Character-by-character text reveal with cursor
• Hover card flip: CSS 3D transform on hover
• Staggered entrance: Multiple blocks appearing one after another with delays
• Progress bar: Animated fill bar for quiz progress or loading

Improvements Planned

• Rich text on flow operators: When the rich-text-formatting branch merges, update add_statement and add_answer to accept rich text params (bold, italic, fontSize, fontWeight, color). The engine already supports ProseMirror JSON on messageProperties/choiceProperties, so the AI tools just need to pass it through. Could also be done via patch_block on the operator's text properties.
• Confirm-before-execute: Proposals that users approve before execution (already implemented for single actions and choices)
• Undo/redo: Snapshot blob before each AI modification for rollback
• Asset library integration: list_assets tool to let AI use uploaded brand assets
• Flow refresh: Emit flow-specific refresh event so the flow view updates without manual refresh
• Smarter intent detection: Match "fjern html", "slett css-blokken" etc. against existing blocks for deterministic per-block deletion
• Sonnet premium tier: Option to use a more capable model for complex tasks, same tools