Building StoryKit: An AI Video Pipeline from Text Idea to 9:16 MP4

A short-form video on TikTok looks effortless: 30 seconds, four cuts, a voiceover, some lofi music. Behind the scenes it’s a chain of expensive operations — script breakdown, image generation, image-to-video, text-to-speech, then an FFmpeg merge that has to scale every clip to 1080×1920 and ducked-mix the music under a voice track.

StoryKit is my single-user MVP that runs that whole chain from one text prompt. This post walks through the actual wiring: the choices that worked, the bugs that took hours to find, and the cache I added to make iteration cheap.

The Stack At a Glance

Monorepo with pnpm workspaces. Two apps: apps/api and apps/web. No shared package — the OpenAPI pipeline is the contract.

Pipeline Shape

┌──────────────┐  POST /projects        ┌─────────────────┐
│ User pastes  │ ─────────────────────▶ │ Project created │
│ a 2k-char    │                        │ in Postgres     │
│ video idea   │                        └────────┬────────┘
└──────────────┘                                 │
                                       POST /:id/breakdown
                                                 ▼
                                       ┌─────────────────┐
                                       │ BullMQ:         │
                                       │ breakdown queue │ ── Gemini 2.5 Flash
                                       └────────┬────────┘    → 3-6 scenes JSON
                                                ▼
                              ┌──────────────────────────────────┐
                              │ For each scene, in parallel:     │
                              │                                  │
                              │  image queue → fal.ai Flux dev   │
                              │  video queue → fal.ai Hailuo i2v │
                              │  audio queue → ElevenLabs TTS    │
                              └────────────────┬─────────────────┘
                                               ▼
                                       POST /:id/render
                                               ▼
                                       ┌─────────────────┐
                                       │ merge queue:    │
                                       │ FFmpeg combine  │
                                       │ + music mix     │
                                       └────────┬────────┘
                                                ▼
                                       Cloudflare R2: final-{ts}.mp4

Each arrow is a typed mutation in the FE. Each queue is one processor file on the BE. The SSE event stream pushes status changes to the FE so the storyboard updates without polling.

Contracts First: OpenAPI → Orval

The first thing I locked down was the contract. Every controller endpoint declares an explicit operationId:

@Post(':id/breakdown')
@HttpCode(HttpStatus.ACCEPTED)
@ApiOperation({ operationId: 'enqueueProjectBreakdown', summary: 'Run idea → scenes breakdown' })
@ApiResponse({ status: 202, type: EnqueueJobResponseDto })
enqueueBreakdown(@Param('id') id: string) {
  return this.pipeline.enqueueBreakdown(id);
}

DTOs are createZodDto(...) classes:

export const CreateProjectSchema = z.object({
  title: z.string().min(1).max(200),
  idea: z.string().min(1).max(IDEA_MAX_LENGTH),
  aspectRatio: z.string().max(10).optional(),
  style: z.string().max(200).optional(),
});

export class CreateProjectDto extends createZodDto(CreateProjectSchema) {}

nestjs-zod’s cleanupOpenApiDoc() lifts those Zod schemas into the generated OpenAPI document. Then:

pnpm --filter @storykit/api openapi:export    # → apps/api/openapi.json
pnpm --filter @storykit/web generate:api      # → apps/web/src/api/generated/

The FE gets a fully typed useCreateProject() hook plus the Zod schema for form validation, both derived from the same source. When I add a field on the BE, the FE either picks it up automatically or stops compiling. No drift.

The Date fields needed a small trick: JSON Schema can’t represent JS Date, so response DTOs use z.iso.datetime(). The wire shape is always ISO strings; the FE coerces back to Date only where it matters.

BullMQ Orchestration

Five queues: breakdown, image, video, audio, merge. Each processor is a NestJS class extending WorkerHost:

@Processor(QUEUES.IMAGE, { concurrency: 4 })
export class ImageProcessor extends WorkerHost {
  async process(job: Job<{ sceneId: string }>): Promise<void> {
    // ... call fal.ai, upload to R2, update Prisma, emit SSE event
  }
}

I learned three things the hard way.

1. jobId is your dedup primitive

Every job is scoped to a deterministic id like image-{sceneId}. Concurrent clicks on “Regenerate” can’t double-charge — BullMQ silently returns the in-flight job’s handle on the second add().

But that helpful behavior turns hostile when a job ends in the failed set (which you keep with removeOnFail: 50 for debugging). The jobId stays taken. Every retry click silently no-ops until the failed entry rolls out of the retention window.

The fix is a small guard:

export async function guardJobId(queue: Queue, jobId: string) {
  const existing = await queue.getJob(jobId);
  if (!existing) return 'fresh';
  const state = await existing.getState();
  if (state === 'failed') {
    await existing.remove();
    return 'cleaned_failed';
  }
  return 'in_flight';
}

Call it before every queue.add(). Bulk enqueue uses the batched variant and reports an extra deduped: number field so the FE can show “Generating 4 images (1 already in progress)” instead of overcounting.

2. node --watch will eat your FFmpeg child

This one cost me a real afternoon. I’d kick off a render, the job would mark itself active in BullMQ, and then nothing. Twenty minutes later, still active, no FFmpeg process anywhere. BullMQ’s stalled-check eventually re-queued and the cycle restarted.

The watcher process kept restarting the API on each save. The merge processor’s FFmpeg child got SIGKILL’d along with the parent. BullMQ never saw a clean failure, so the job sat as active with a stale lock.

Two fixes, both necessary:

// main.ts
app.enableShutdownHooks();

// merge.processor.ts
async onModuleDestroy() {
  if (this.currentAbort) this.currentAbort.abort();
  await this.worker.close();
}

Combined with passing an AbortSignal into every fluent-ffmpeg call (so cmd.kill('SIGTERM') fires on abort), restarts now end with a clean failure and BullMQ retries cleanly.

3. fluent-ffmpeg doesn’t auto-start

This is the bug that made me question my career choices. The merge processor logged “All assets downloaded” and then sat forever. No FFmpeg in ps. No error. Just a Promise that would never resolve.

The minimal reproducer:

const cmd = ffmpeg().input('in.mp4').outputOptions(['-c copy']).output('out.mp4');
await new Promise((resolve, reject) => {
  cmd.on('end', resolve).on('error', reject);
});
// Promise never resolves. Ever.

.output(path) does not start the command. You either need .save(path) (sugar for .output().run()) or an explicit cmd.run(). The fluent API’s chaining tricks your eyes — cmd.on('end', resolve) looks like it should be enough.

Two more sibling gotchas in the same file:

// WRONG — inputOptions attaches to the LAST added input.
// '-stream_loop -1' ends up on the VIDEO input, not the music.
ffmpeg()
  .input(videoPath)
  .inputOptions(['-stream_loop -1'])
  .input(musicPath)

// WRONG — inputOptions before any input is a no-op.
// On run(): "No input specified".
ffmpeg()
  .inputOptions(['-f concat', '-safe 0'])
  .input(listPath)

Always: input() first, inputOptions() second, attached to the same input.

Live Progress with SSE

The merge phase takes 30-60 seconds on Apple Silicon. The user shouldn’t be staring at a spinner.

I used Server-Sent Events because they’re the right tool: one-way, long-lived, auto-reconnect for free. RxJS Subject per-projectId on the BE, vanilla EventSource on the FE.

// modules/events/.../rxjs-event-bus.service.ts
publish(projectId: string, event: EventPayload): void {
  this.subjects.get(projectId)?.next(event);
}

// modules/events/.../events.controller.ts (sketch)
@Sse(':id/events')
stream(@Param('id') id: string) {
  return this.bus.observe(id).pipe(
    map((event) => ({ data: JSON.stringify(event) })),
  );
}

On the FE, one hand-rolled hook owns the EventSource:

useEffect(() => {
  const source = new EventSource(`/api/projects/${projectId}/events`);
  source.addEventListener('message', (event) => {
    const data = JSON.parse(event.data);
    if (data.type === 'render.progress') handlersRef.current.onRenderProgress?.(data.percent);
    // ... etc
  });
  return () => source.close();
}, [projectId]);

The SSE endpoint is @ApiExcludeEndpoint() — text/event-stream doesn’t map to a typed hook, so Orval skips it and the FE keeps the hand-rolled subscriber. Everything else flows through the generated client.

Cloudflare R2: The Checksum-Mode Trap

R2 is great. Egress-free. S3-compatible. Until the day every signed URL started returning 403.

The setup looked normal:

this.client = new S3Client({
  region: 'auto',
  endpoint: this.config.get('R2_ENDPOINT'),
  credentials: { ... },
});

return s3GetSignedUrl(this.client, new GetObjectCommand({ Bucket, Key }), { expiresIn });

s3:GetObject worked when called directly through the SDK. The signed URL it returned, however, was rejected by R2 with HTTP/1.1 403 Forbidden.

The culprit is a behavior change introduced in @aws-sdk/client-s3 v3.729: it now injects x-amz-checksum-mode=ENABLED into the pre-signed URL by default. R2 doesn’t recognize the header, treats it as an unsigned parameter, and rejects the signature.

The fix is two config flags on the S3 client:

this.client = new S3Client({
  region: 'auto',
  endpoint: this.config.get('R2_ENDPOINT'),
  credentials: { ... },
  // R2 doesn't honor the SigV4 integrity headers AWS SDK v3.729+ injects.
  requestChecksumCalculation: 'WHEN_REQUIRED',
  responseChecksumValidation: 'WHEN_REQUIRED',
});

Signed URLs after this change have no checksum-mode query parameter, R2 happily serves them, and the browser plays the video in a <video> tag with proper CORS headers. Bonus: every other signed URL (per-scene image, per-scene clip, per-scene voiceover, music preview) was silently affected too — they all started working the same day.

The FFmpeg Merge

For a 4-scene project, the merge does roughly:

1. Download 8-9 R2 objects (clips + voiceovers + optional music) in parallel.
2. Normalize each clip: scale to 1080×1920, replace its audio with the voiceover, re-encode H.264 + AAC.
3. Concat the normalized clips with -c copy (cheap — same params).
4. Mix the background music under the voice track via amix=duration=first, music ducked to 20% by default.
5. Upload final MP4 to R2.

Per-scene normalize is by far the most expensive step. On an M-series Mac each 6s clip takes 1-1.5s; on a slower box it’s 5-15s.

That cost gets paid again every time the user tweaks the music track. Which they will — that’s the part most users iterate on.

Caching the Concat Artifact

After every successful full render, I now upload the post-concat / pre-mix artifact to R2 at library/concat/{projectId}-{fingerprint}.mp4 and stash four columns on Project:

lastRenderFingerprint String?
lastConcatKey         String?
lastRenderMusicKey    String?
lastRenderMusicVolume Float?

The fingerprint is a SHA-256 over per-scene (order:videoKey:audioKey) — anything that affects the concat output, excluding music:

export function computeRenderFingerprint(scenes: FingerprintScene[]): string {
  const sorted = [...scenes].toSorted((a, b) => a.order - b.order);
  const payload = sorted
    .map((s) => `${s.order}:${s.videoKey ?? ''}:${s.audioKey ?? ''}`)
    .join('|');
  return createHash('sha256').update(payload).digest('hex');
}

The merge processor now branches into three paths:

The fast-remix path looks like this:

async runFastRemix(projectId, lastConcatKey, musicKey, musicVolume, tempDir, signal) {
  try {
    await this.transfer.downloadToFile(lastConcatKey, concatPath);
  } catch (err) {
    throw new FastPathCacheMissError(err.message);  // → caller falls back to full path
  }

  if (musicKey) {
    await this.transfer.downloadToFile(musicKey, musicPath);
    await mixMusic(concatPath, musicPath, musicVolume, finalLocalPath, onProgress, signal);
  } else {
    await rename(concatPath, finalLocalPath);  // detach case
  }

  await this.transfer.uploadFile(finalLocalPath, finalKey, 'video/mp4');
  // ... persist new music snapshot
}

If the cached concat is missing on R2 (someone deleted it, lifecycle policy ate it, whatever), the typed FastPathCacheMissError triggers the full path automatically. The user sees no failure, just a slightly slower render.

The HTTP layer mirrors the same three paths:

@Post(':id/render')
@HttpCode(HttpStatus.ACCEPTED)
async enqueueRender(@Param('id') id: string, @Res({ passthrough: true }) reply: FastifyReply) {
  const result = await this.pipeline.enqueueRender(id);
  if (result.mode === 'cached') reply.status(HttpStatus.OK);
  return result;
}

A queued job returns 202 with mode: 'queued'; a cache hit returns 200 with mode: 'cached' and finalVideoUrl. The FE handler skips the progress UI on cached and navigates straight to the preview.

Cost Guardrails

Every Render Final click is free at the bottom but a fal.ai image gen is $0.025 and a Hailuo clip is $0.20. A 6-scene video is roughly $1.40 end-to-end. Without guardrails I’d burn $30/day testing.

A few layers:

• Confirm dialog with cost preview on every bulk endpoint: “Generate 4 videos at ~$0.80 total?”
• Per-scene regen cap (SCENE_REGEN_LIMIT = 8, soft confirm at 5): forces the user to either edit the prompt or accept the result.
• jobId dedup: even if the user double-clicks, BullMQ returns the in-flight handle.
• No-op short-circuit on render: identical state returns the cached final video without re-encoding.

Together they take “I’ll just click again” from “$1.40 wasted” to “0¢ wasted”.

What I’d Do Differently

A few things I’d rethink if I started over today:

• Status as a string, not a Prisma enum. I went with strings (‘draft’, ‘generating’, ‘ready’, ‘failed’) so migrations stay cheap during MVP. Now that the states are stable, an enum would be safer and shorter.
• No shared packages/. Tempting in a monorepo. The OpenAPI pipeline replaces any need for shared TypeScript types between BE and FE. One less moving part.
• Routes are thin shells. Each TanStack file-based route is ≤40 lines — pulls useParams/useSearch and forwards to a <DomainPage> component in features/. Made the router opt-out underscore convention (projects.$projectId_.preview.tsx) catch me once when a preview page silently rendered storyboard content because the parent route didn’t have <Outlet />. Lesson: prefer flat routes unless you actually want a shared layout.
• Cache lifecycle policy. The concat cache and orphan music keys accumulate forever. R2 lifecycle rules are the right home for that — I just haven’t written them yet.

Wrap-up

StoryKit is roughly 6k LOC of API + 5k LOC of FE. The interesting code is concentrated in three places: the merge processor (FFmpeg orchestration + cache), the pipeline service (cost guards + queue dedup), and the OpenAPI/Orval contract pipeline (typesafety end-to-end).

The pieces that paid off best:

• OpenAPI as the contract — zero manual TypeScript interfaces between BE and FE.
• BullMQ jobId discipline — every queue operation is idempotent under retry.
• Per-project concat cache — music iteration went from “I’ll wait a minute” to “I’ll just toggle that”.
• enableShutdownHooks() — restarts no longer leave jobs stalled.

The pieces I keep refactoring:

• Status string drift between BE and FE.
• The four-field cache snapshot has surfaced one subtle bug already (volume sticking after detach) and probably has more.
• FFmpeg progress reporting through SSE flickers when the pipeline switches paths mid-job.

Good enough for one person making 30-second videos. The architecture would need real auth, rate limits, and quota tracking before it survives more than a single user — but those are problems I’m happy to defer.