Logs Parser

Source: /Logs-Parser/

Overview

Node.js service (single dependency: @duckdb/node-api) that processes raw Bunny CDN access logs into aggregated Parquet files consumed by the Backend.

Repo Structure

index.js               -- Pipeline orchestration (4 stages)
src/
  Bunny.js             -- Stage 1: Fetch raw CDN logs from Bunny API
  Aggregator.js        -- Stages 3 & 4: Session + hourly aggregation (DuckDB SQL)
  helpers/
    Decoder.js         -- Stage 2: Filter & decode base64 analytics payloads
    DateHelpers.js     -- Date range utility
logfiles/bunny/        -- Working directory for intermediate files

Deployed as Docker container (Node 18), built and pushed via GitHub Actions to GHCR.

Pipeline (4 Stages)

Stage 1: Bunny.pullRawData()

Fetches yesterday's CDN access logs from https://logging.bunnycdn.com/{MM-DD-YY}/{pull_zone_id}.log.

Raw CSV has 12 pipe-delimited columns: cache_status, status_code, timestamp, bytes_sent, pull_zone_id, remote_ip, referer_url, url, edge_location, user_agent, unique_request_id, country_code.

Converts to Parquet via DuckDB.

Stage 2: Decoder.decodeFromParquet()

Streams through raw Parquet, filtering for rows where url contains analytics/v1.gif.

Extracts the encoded_object query parameter, base64-decodes it, JSON-parses it.

Filters out: events with inTesting: true or missing creativeId.

Outputs: NDJSON with all decoded event fields + timestamp (hour-bucketed from CDN timestamp) + country_code.

Stage 3: Aggregator.aggregateSessions()

Groups decoded events by session id using DuckDB SQL.

Key detail: Uses MAX() for cumulative fields because the engine sends running totals in each event (e.g., creativeActions increases over the session). SUM() is used for count-type events (like flow steps).

Output: one row per session.

Stage 4: Aggregator.aggregateHourlyBuckets()

Aggregates sessions into hourly buckets grouped by creative_id, creative_group_id, campaign_id, dsp, domain.

Output: hourly_summary.parquet (hive-partitioned by date).

Metric Aggregation Details

Session-Level Aggregation (Stage 3)

Engine Field	Session Column	SQL
creativeActions	creative_actions	MAX(creativeActions)
clicks	clicks	MAX(clicks)
secondsToActive	seconds_to_active	MAX(secondsToActive)
secondsInView	seconds_in_view	MAX(secondsInView)
clickedLink	link_clicks	SUM(CASE WHEN clickedLink THEN 1 ELSE 0 END)
viewable	viewable	MAX(viewable)
reachedEnd	reached_end	MAX(reachedEnd)
header/background	header_clicks, background_clicks	SUM(CASE WHEN messageType='count' AND objectId=-1 THEN 1 ELSE 0 END)
pageUrl	domain	Extracted by stripping scheme/port/path/www

Hourly Aggregation (Stage 4)

Report Metric	SQL
impressions	COUNT(*) (one per session)
interacted	SUM(CASE WHEN seconds_to_active > 0 THEN 1 ELSE 0 END)
started	SUM(creative_actions > 0)
continued	SUM(creative_actions > 1)
creative_actions	SUM(creative_actions)
link_clicks	SUM(link_clicks)
viewable_impressions	SUM(viewable)
seconds_in_view	SUM(seconds_in_view)
seconds_to_active	SUM(seconds_to_active)
header_clicks	SUM(CASE WHEN header_clicks > 0 THEN 1 ELSE 0 END) (sessions, not total clicks)
background_clicks	SUM(CASE WHEN background_clicks > 0 THEN 1 ELSE 0 END) (sessions, not total clicks)
reached_end	SUM(reached_end)

Important: interacted Definition

interacted counts sessions where seconds_to_active > 0, which means the user's first click fired (setting creativeClickedOnce and sending secondsToActive). This is entirely independent of numUserActions/creativeActions.

Known Gaps (Repo vs Deployed)

The Application-Backend expects some columns the committed Logs-Parser code doesn't produce:

Column	Status
clicks	Field exists in decoded events but Aggregator SQL may not aggregate it
device_type	Not parsed (TODO in Decoder about user_agent parsing)
seconds_total_active	Not aggregated
custom_metrics (MAP)	messageType: 'metric' events not processed

This suggests the deployed version has evolved beyond what is committed.