CanDo

Take a photo of gym equipment and get a workout built for it. A Rust serverless function (running on Vercel's Rust runtime / Fluid Compute) forwards the image to Gemini 2.5 and returns a structured workout; a Vite + React SPA handles the upload. Both deploy together as a single Vercel project.

Architecture

React SPA (Vite, static)  ──raw image bytes──▶  api/analyze.rs (Rust fn)  ──▶  Gemini 2.5
        ▲                                                                         │
        └──────────────────── structured workout JSON ◀──────────────────────────┘

.
├── api/analyze.rs    # Vercel Rust function  POST /api/analyze  (Gemini  workout)
├── api/detect.rs     # Vercel Rust function  POST /api/detect   (fal Florence-2  boxes)
├── core/lib.rs       # shared GeminiClient + FalClient
├── Cargo.toml        # [lib] core + [[bin]] analyze, detect
├── src/              # React app (upload auto-runs outline + workout in parallel)
├── index.html, vite.config.ts, package.json, tsconfig.json
├── vercel.json       # framework: vite, function maxDuration
└── .vercelignore

The SPA pings both functions with a GET on load to warm them (trigger the cold start early), so the first real request is fast.

The function constrains Gemini's output with a JSON responseSchema, so the frontend always receives well-typed equipment + workout data. The browser posts the raw image bytes (with the file's Content-Type), which the function reads directly from the request body.

Deploy to Vercel

1. Enable the Rust runtime for your account/project (one-time):

   npx vercel plugins add vercel/vercel-plugin

2. Set the API key (get one at https://aistudio.google.com/apikey):

   vercel env add GEMINI_API_KEY        # paste key; add to Production/Preview/Development

3. Deploy — either push to the connected GitHub repo, or:

   vercel deploy --prod

Vercel auto-detects Vite (builds dist/) and compiles the Rust function in api/ separately.

Local development

Use vercel dev, which serves the Vite app and the Rust function on the same origin (so /api/analyze just works):

npm install
vercel dev

vercel dev reads GEMINI_API_KEY from your linked project's env or a local .env.local.

API

POST /api/analyze — body is the raw image bytes; Content-Type is the image mime type (e.g. image/png). Returns:

{
  "equipment": [{ "name": "Kettlebell", "confidence": 0.95 }],
  "workout": {
    "title": "...",
    "summary": "...",
    "exercises": [
      { "name": "...", "equipment": "...", "sets": 3, "reps": "8-12", "rest_seconds": 60, "notes": "..." }
    ]
  }
}

POST /api/detect — body is the raw image bytes; Content-Type is the image mime type. Gemini first identifies the equipment present (deduped — one per item), then SAM 3 segments each name (text-promptable, max_masks: 1, in parallel). The grayscale mask's boundary is traced server-side into a crisp contour PNG (opaque edge stroke, transparent elsewhere) that the frontend tints per object via CSS mask-image. An optional X-Detect-Prompt header (period-separated phrases) skips Gemini and segments those directly. Returns:

{
  "objects": [
    {
      "label": "treadmill",
      "score": 0.94,
      "box": [0.50, 0.64, 0.77, 0.64],
      "mask": "data:image/png;base64,..."
    }
  ]
}

box is normalized [cx, cy, w, h] (0..1); mask is a full-image contour PNG (opaque edge stroke) the frontend tints per object.

Configuration