Live Rotoscoping

A Next.js app that uploads video to blob storage and calls fal.ai SAM endpoints for image/video segmentation.

Overview

This project provides a UI for:

• uploading a video file, then running full-video segmentation
• using the webcam for “near real-time” per-frame segmentation (sampling every 5th frame)

Segmentation work is performed by fal.ai models, and uploads are stored via Vercel Blob.

Core Concepts

• Upload blob URL app/api/upload-video/route.ts accepts multipart upload and writes the bytes to blob storage via @vercel/blob.

• Frame segmentation (image) app/api/segment-frame/route.ts uploads a JPEG frame to fal storage and calls fal-ai/sam-3/image.

• Video segmentation (SAM2 vs SAM3) app/api/segment-video/route.ts chooses:

  • fal-ai/sam-3/video when model === "sam3"
  • fal-ai/sam2/video otherwise

  and returns the fal result payload.

• Client loop + overlay rendering app/page.tsx:

  • draws video frames into a canvas
  • periodically POSTs frames to /api/segment-frame
  • draws a semi-transparent overlay (if data.image.url is returned) and bounding boxes (from data.masks[].box).

Technology Stack

• Root: Next.js (App Router), React, TypeScript, Tailwind CSS
• Segmentation backend: fal.ai serverless client (@fal-ai/serverless-client)
• Storage: Vercel Blob (@vercel/blob)

Getting Started

pnpm install
pnpm dev

Scripts

build: next build
dev: next dev
lint: eslint .
start: next start

Repository Layout

• app/page.tsx: UI (upload/webcam + canvas overlay)
• app/api/upload-video/route.ts: blob upload
• app/api/segment-frame/route.ts: per-frame segmentation
• app/api/segment-video/route.ts: full-video segmentation

Configuration

This repo expects credentials at runtime for:

• process.env.FAL_KEY (used by fal.ai route handlers)
• whatever environment configuration @vercel/blob requires for server-side put().

Notes

• Keep this README factual and repository-grounded.