← All work
Computer vision · AI

Rotoscope

AI-powered image segmentation and layer extraction — dual-model detection (Florence-2 + GPT-4o), SAM-3 masks, depth-based ordering, context-aware inpainting.

View example ↗

AI-powered image segmentation and layer extraction tool built with Next.js. Upload any image and automatically detect objects, generate precise segmentation masks, separate layers, and export individual objects with transparent backgrounds.

Features

  • Dual-Model Object Detection: Florence-2 (fast detection) + GPT-4o Vision (semantic understanding) with intelligent deduplication
    • Graceful Degradation: Falls back to Florence-2 only if GPT-4o is unavailable
  • Depth-Based Ordering: Automatic spatial analysis determines front-to-back object arrangement for optimal removal workflow
  • Precise Segmentation: SAM-3 (Segment Anything Model 3) generates pixel-perfect masks
  • Context-Aware Inpainting: AI-powered background fill considers depth relationships and scene semantics
  • Manual Object Addition: Draw bounding boxes to add objects the AI missed
  • Layer Separation: Extract objects from backgrounds with intelligent inpainting
  • Interactive Editing: Resize, reposition, rename, and delete detected objects
  • Layer Panel: Photoshop-style layer management with depth indicators, visibility toggles, and thumbnails
  • Color-Coded System: Consistent colors across bounding boxes, masks, and layer indicators
  • Separated Layer Visualization: Dotted outlines show where objects have been removed while maintaining mask visibility
  • Export Options: PNG/JPEG/WebP with transparent, white, or black backgrounds
  • Shadow Generation: Add realistic drop shadows to exported objects
  • Batch Export: Download all objects as a ZIP file

How It Works

``` ┌─────────────────────────────────────────────────────────────────┐ │ Upload │ │ Drag-drop, file picker, or paste from clipboard │ └─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐ │ Object Detection │ │ Florence-2 (fast detection) + GPT-4o (semantic understanding) │ │ Semantic deduplication with word matching & IoU merge │ │ Depth analysis for front-to-back ordering (0-10 scale) │ └─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐ │ Segmentation │ │ SAM-3 generates pixel-level masks per object │ │ Objects sorted by depth (frontmost first) │ └─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐ │ Layer Extraction │ │ Separate objects Context-aware inpainting Composite │ │ Prompts consider depth, neighbors, and scene context │ └─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐ │ Export │ │ Individual cutouts, full composite, or batch ZIP download │ └─────────────────────────────────────────────────────────────────┘ ```

Getting Started

Prerequisites

You'll need API keys from:

  • fal.ai: For Florence-2, SAM-3, and inpainting (Get API key) - Required
  • Vercel AI Gateway: GPT-4o vision analysis is available by default when deployed to Vercel with AI Gateway credits
  • OpenAI (optional): For local development or when AI Gateway is unavailable (Get API key)

Note: The app works perfectly with just the FAL_KEY. GPT-4o vision adds semantic understanding and depth analysis but is optional. If GPT-4o calls fail or time out, the app automatically falls back to Florence-2 detection.

Installation

  1. Clone and install dependencies: ```bash npm install ```

  2. Set up environment variables: ```bash

Copy the example file

cp .env.local.example .env.local

Edit .env.local and add your API keys:

FAL_KEY=your_fal_ai_api_key_here

OPENAI_API_KEY=your_openai_api_key_here (optional, for local dev only)

```

  1. Run the development server: ```bash npm run dev ```

  2. Open the app: Visit http://localhost:3000

Environment Variables

```env

fal.ai API Key (required)

Used for: Florence-2 detection, SAM-3 segmentation, image storage, inpainting

FAL_KEY=your_fal_ai_api_key_here

OpenAI API Key (optional)

- On Vercel: Uses AI Gateway by default (no key needed if you have gateway credits)

- Locally: Set this to use direct OpenAI API and bypass the gateway

- If not set: App uses AI Gateway, falls back gracefully if unavailable

OPENAI_API_KEY=your_openai_api_key_here ```

Keyboard Shortcuts

Key Action
Escape Deselect / Exit modes
Delete / Backspace Remove selected object
Ctrl/Cmd + Z Undo
Ctrl/Cmd + Shift + Z Redo

Technical Architecture

Core Principles

Rotoscope is built on 14 foundational principles organized into 6 categories:

  1. History & State: Consistent undo/redo with updater functions to prevent stale closures
  2. Identity: Stable object IDs, layers as derived views (not source of truth)
  3. Inpainting: Consistent mask format, stable background plate strategy with mask dilation
  4. Rotoscoping: Complete extract-to-layer pipeline with optional shadow handling
  5. Stages & Tools: Clear processing stages with proper gating
  6. Coordinates: All bounds stored as percentages, converted to pixels only at API boundaries

See AGENT_INSTRUCTIONS.md for complete implementation details.

Coordinate System

All coordinates are stored as percentages (0-100) of the original image dimensions. This ensures:

  • Consistent rendering across different viewport sizes
  • Resolution-independent state management
  • Simplified undo/redo operations

Conversion to pixels happens only at API boundaries (Florence output storage, storage SAM input).

Background Plate Strategy

For reliable layer extraction and inpainting:

  • Cutouts: Always use originalImageUrl || imageUrl (stable base)
  • Sequential inpainting: Build upon inpaintedBackgroundUrl || imageUrl
  • Mask dilation: Expand masks by 5px before cutout extraction to prevent edge halos

Project Structure

``` ├── app/ │ ├── page.tsx # Main page │ ├── actions.ts # Server actions for AI orchestration │ └── api/ │ ├── upload/ # Image upload to fal.ai storage │ ├── segment/ # SAM-3 segmentation │ ├── segment-point/ # Point-based segmentation │ └── inpaint/ # Object removal & background fill │ ├── lib/detection/ │ ├── florence.ts # Florence-2 object detection │ ├── gpt.ts # GPT-4o vision analysis │ ├── vision.ts # Vision API wrapper with error handling │ ├── merge.ts # Result merging │ └── schemas.ts # Type definitions │ ├── components/ │ ├── image-uploader.tsx # Main component (source of truth for all state) │ └── image-uploader/ │ ├── components/ # UI components │ │ ├── image-canvas.tsx # Image + mask rendering │ │ ├── layer-panel.tsx # Layer management UI │ │ ├── object-bounding-box.tsx # Interactive bounding boxes │ │ └── ... │ ├── constants.ts # Color palette, KISS_CUT_EXPANSION_PX │ └── utils/ # Export, mask processing (dilateMask) │ ├── hooks/ # Zoom/pan, keyboard, history (updater functions) │ └── docs/ # Full documentation ├── README.md # Documentation index ├── getting-started.md # Installation & setup ├── user-guide.md # Usage instructions ├── architecture.md # Technical architecture ├── api-reference.md # API documentation ├── development.md # Contributing guide └── troubleshooting.md # Common issues ```

Tech Stack

  • Framework: Next.js 16 (App Router)
  • UI: React 19, Tailwind CSS v4, shadcn/ui
  • AI Models:
    • Florence-2 (fal.ai) - Fast object detection
    • GPT-4o (OpenAI) - Semantic understanding
    • SAM-3 (fal.ai) - Precise segmentation
    • Inpainting models (fal.ai) - Background fill
  • Storage: fal.ai storage

Documentation

See the docs folder for comprehensive documentation including architecture diagrams, API reference, and troubleshooting guides.

For AI agents: See AGENT_INSTRUCTIONS.md for the 14 architecture principles and implementation checklist.

License

MIT