Context
Stills are easy. Ops is the product.
Diffusion made a single image cheap. A pipeline that keeps a face, style, and tone across photos, 3–4 second clips, and accompanying text is not. Project Lucia is Tobto’s internal PoC for that ops layer: node workflows you can reason about, local prep, cloud bursts for the heavy frames — not a one-click demo that dies on the second batch.
Role
R&D pipeline, not a shipped SaaS.
This is a proof of concept: ComfyUI graphs for image and video, a local LLM assist for prompt expansion, and RunPod pods for batch renders when a 24 GB card spends an hour on a few seconds of motion. The point was to learn the control surface — what stays local, what bursts to a pod, how a character stays recognizable without hand-fixing every face.
Pipeline
Two tracks, same sandbox.
ComfyUI as the workbench — models, LoRAs, ControlNet, masks, upscalers chained like LEGO. Track one is generative: scene and character from prompt only; keep the first pass simple, CFG around five, denoise at one, then layer LoRA for style and detailing passes. Track two starts from a reference: lower denoise, ControlNet for pose and depth, masks for surgical edits. Micro-workflows for face, hands, and skin — not one monster graph that nobody can debug.
Text
LLM as a prompt co-pilot, not a narrator bot.
A local model via Ollama describes the generated frame and expands the original prompt — sharper scene language, fewer silent contradictions between what you asked and what the pixels did. Text sits beside the image pipeline: captions, scene notes, and prompt revisions that feed the next pass. Same character spine, three media outputs.
Scale
Prep local. Burst on a pod.
A few seconds of video at modest resolution can take half an hour on a single 24 GB card. The loop that worked: build and test the workflow locally, then rent a ready ComfyUI pod (RunPod and similar) only for heavy batches — roughly a couple of dollars an hour plus cheap storage. Same graph, hardware that finishes a clip in one to two minutes. Math beats owning a rack when you are still proving the pipeline.
Consistency
Train a small LoRA. Bootstrap the rest.
Recurring character without fixing every face: pick a base (Flux, Pony, or similar), a handful of good frames, captions, and train a LoRA. Cheat-code: use the same pipeline to expand the dataset across locations, poses, and light — then retrain so the model remembers features instead of inventing a cousin each run. Control stays around the high nineties; hands and reflections still ask for micro-fixes.
Outcome
A PoC that proves the ops, not a brand launch.
Lucia showed a workable spine: stills in minutes, video when the pod is warm, text that rides the same frame description loop, and a character that survives more than one lucky seed. It is research you can reuse on client work — content systems, campaign packs, prototype worlds — without pretending a hobby graph is already a product.
Tone
Sandbox discipline over magic buttons.
ComfyUI is not a one-click generator. You mix models, learn CFG and denoise, keep graphs small enough to trust. Lucia’s lesson for Tobto: generative media is an engineering surface — queues, cost, consistency — the same as any other delivery pipe.
Story
- 01
One image is cheap; a consistent photo / video / text loop is the real problem.
- 02
ComfyUI workflows: generative track and reference track with ControlNet and LoRA.
- 03
Local LLM expands prompts from the frame — text sits in the same spine.
- 04
Prep locally; burst video on RunPod when a single GPU becomes the bottleneck.
- 05
Small LoRA + bootstrap dataset keeps the character recognizable across batches.
Notes
Technical notes on MediumInternal PoC — ComfyUI, pods, and the boring ops that make generative media usable.