Summary

ComfyUI’s founding bet was a powerful workflow interface rather than another simplified interface. Comfy discovered Stable Diffusion in October 2022 without having written a line of PyTorch, started the project on January 1, 2023, and released it on January 16, guided by the contrarian brief: “Everyone’s trying to make an easy-to-use interface; let’s make a hard-to-use interface.”
Its breakout distribution was operational: ComfyUI became the practical way for many people to run SDXL on regular GPUs. SDXL’s code preceded its restricted 0.9 checkpoint, that checkpoint leaked, and Comfy rapidly fixed user issues while AUTOMATIC1111’s implementation remained too inefficient for many users. Many people had little choice but to use ComfyUI.
Open-image-model adoption depends on material improvement and release momentum. Comfy says FLUX is probably the best overall, while SD 3.5’s 2.5B and 8B models may be more creative. Stable Cascade, which Comfy considered a step up from SDXL, was held in red-teaming for about three months and was eclipsed by the SD3 announcement one week after release.
ComfyUI’s local execution advantage is its memory-aware orchestration across hardware configurations. It estimates each sampling job’s memory needs and evicts as little as possible, navigating the “fine line” before Windows’ NVIDIA driver spills into RAM and performance collapses; AMD remains workable on Linux but problematic on Windows pending proper ROCm and PyTorch ROCm support.
The customization stack is becoming a compatibility layer across model generations, though shared components matter more than model branding. Textual inversions trained for SD 1.5 can still be used with SDXL and SD3 through compatible text encoders, but weaken as conditioning is diluted; LoRAs remain portable and inference-efficient because their low-rank weight differences are applied before sampling, after which execution runs at normal speed.
Easy custom-node creation has turned ComfyUI from an image interface into a general workflow platform, creating ecosystem growth and duplicated functionality. Users have connected it to Krita, built a Wolfenstein-like game whose textures are generated from a workflow, and added ingestion nodes such as YouTube downloaders; duplicated custom-node functionality is something the core project is trying to address.
The commercial path preserves local execution, at least on the open-source side, while adding packaged usability, cloud inference, and possible enterprise products. A version 1 release is intended to bring an easier packaged experience to Windows and hopefully Mac, but timing remains only “soon”; Comfy welcomes third-party businesses using ComfyUI because their usage expands the ecosystem.

Deep dive

1. ComfyUI began as an escape hatch for experiments AUTOMATIC1111 could not express

Comfy discovered Stable Diffusion in October 2022 with a background in basic software and automation, no image-generation expertise, and “not a line of PyTorch” written. Getting “addicted to generating images” supplied the motivation to learn the stack from the workflow outward.
The first constraint was the high-res fix: generate at low resolution, upscale, then refine. Comfy wanted different samplers, step counts, settings, and eventually a different model on the second pass; modifying AUTOMATIC1111 became harder than building a separate interface. He began coding January 1, 2023 and published ComfyUI on January 16.
Area conditioning supplied the first adoption spark: assign different prompts to regions, run each region and the whole image through every diffusion step, then average the results. A similarly structured MultiDiffusion paper appeared roughly a month later. Different models can participate only when they share a latent space—SD 1.5 variants can mix, while SDXL and SD 1.5 cannot.

2. Model winners need both a visible leap and uncontested release momentum

Comfy’s current model call is qualified: FLUX is probably the best overall, particularly when consistency matters. SD 3.5 offers a 2.5B model and an 8B model, is smaller than FLUX, and might be preferable when the goal is something more creative rather than predictably consistent.
The hosts proposed that the community abandons each generation wholesale; Comfy argued the opposite. Users move only for a significant improvement: SD 1.5 retains substantial usage, while SD2 was largely ignored because it did not improve enough. Most workflows remain structurally portable, though prompts may need complete rewriting.
Stable Cascade is Comfy’s cautionary example of release timing overwhelming technical merit. The model was ready about three months before release but remained stuck in red-teaming; when it finally launched, SD3 was announced one week later. Momentum was stolen, and little was built on top of it. The hosts also noted that the “Cascade” name made it seem like a branch off the main development tree; Comfy identified its authors as the Würstchen researchers, who left Stability right after the release.
Evaluation remains aesthetic and informal: users mostly generate samples and decide, “I think this image is nice.” Comfy’s warning is that capability and taste diverge—“most of the images on the internet are ugly,” so a smart, consistent model trained indiscriminately may still fail to produce the look artists want.

3. Conditioning techniques work by exploiting shared representations—and fail where those representations deepen

Textual inversion effectively trains a new word: prompts become tokens, tokens become vectors, and training finds a vector that represents the supplied images. The host characterized it as surprisingly sample-efficient, but Comfy says the community has mostly stopped using it, even though Stability’s internal T5-XXL experiments reportedly worked well.
Compatibility follows shared encoders. An SD 1.5 textual inversion can still be used with SDXL because one of SDXL’s two encoders is the same CLIP-L encoder, but its effect is weaker; SD3 has three encoders, diluting it further. Comfy suggests a T5-XXL inversion might work across models sharing T5-XXL, while explicitly leaving that as something needing testing.
Standard CLIP accepts 77 tokens; LongCLIP extends that to 256. The older long-prompt hack simply splits text into 77-token chunks, encodes each, and puts the results together at the end. Prompt weighting similarly interpolates between empty-prompt and supplied-prompt vectors, but the trick works with shallow CLIP-L and “doesn’t work at all” with deeper T5-XXL; language-based prompting may compensate.
LoRA instead represents the difference between base and fine-tuned weights through two low-rank matrices. Those small, portable matrices are applied to model weights before sampling, producing only a brief startup delay and then normal inference speed. LoCon, LoHa, and related variants are alternative ways to represent that weight difference.

4. Local performance depends on separating the interface from a memory-aware execution engine

Comfy deliberately optimized for power: asynchronous queues, selective re-execution after part of a graph changes, and local-first memory management. When building the frontend, he found the JavaScript library LiteGraph, which already supplied the node interface, and plugged it into the backend rather than spending time developing the frontend. On Gradio, he said its coupling of interface and backend logic is useful for quick ML demos but makes long-lived software harder to maintain.
Memory management remains “the biggest pain in the ass.” Loading and unloading every model was acceptable while models were small; a roughly 10GB model on a 4090 can take seconds each time. ComfyUI now estimates the next sampling operation’s memory requirement and removes the minimum amount of already-loaded model state needed to make room.
The dangerous boundary is overcommitting NVIDIA memory on Windows: the driver begins paging into system RAM, so a job technically continues while becoming extremely slow. AMD works on Linux, albeit slower than an equivalently priced NVIDIA GPU by Comfy’s account, but Windows support remains poor until ROCm and its PyTorch build work properly there.

5. Extensibility has pushed ComfyUI beyond images and into general multimodal pipelines

Core controls are exposed at several difficulty levels: four sampler nodes run from easiest to most advanced, with the regular node providing basic settings and the advanced node exposing individual components and settings. Comfy advises lowering diffusion steps until output quality deteriorates; CFG acts roughly like image contrast by multiplying the positive-minus-negative prediction. At CFG 1.0, the negative prompt is unused and sampling becomes twice as fast.
Custom nodes made ComfyUI a backend for Krita, data-ingestion tools, and experiments such as a Wolfenstein-like game whose generated texture was a workflow input. Before the official registry, ComfyUI Manager’s maintainer searched GitHub daily and added nodes manually; the registry is intended to reduce that work. Many custom-node packs share similar functionality, which the project is trying to address by bringing some capabilities into core.
Video exposes a meaningful architectural divide. Stable Video Diffusion took SD 2, added temporal attention, and—like AnimateDiff—still uses a 2D latent space and compresses only spatially. Comfy reserves “true video models” for systems such as Mochi, with 3D latents that allow movement through space and a temporal VAE that also compresses through time. He implemented Mochi because it appeared strongest, while describing CogVideoX as another decent open model.

6. SDXL adoption bridged the solo project to a local-first business model

The area-conditioning Reddit post appeared at the end of January or beginning of February 2023; an Olivio Sarikas video in March brought ComfyUI’s first real burst of attention. Stability AI hired Comfy in June 2023 because it wanted to experiment with chaining SDXL’s base and refiner models. The refiner had been trained on lower timesteps, and chaining the two increased quality in testing, though users have since mostly forgotten it.
SDXL’s unusual rollout accelerated ComfyUI. Code was released before the model checkpoint, 0.9 access was given to people who signed up but limited to educational email addresses, and the checkpoint then leaked. ComfyUI was the only way many people could easily run SDXL, whereas AUTOMATIC1111’s quick implementation was so inefficient that regular GPUs could not run it effectively—turning backend efficiency into the project’s growth engine.
The core backend was still primarily Comfy’s work at the time of the conversation, with most team effort redirected toward the historically neglected frontend. Hiring was underway, with more backend help expected. The planned version 1 is a packaged, easier-to-install release for Windows and hopefully Mac, but Comfy would commit only to “soon.”
Comfy says it will continue making local model execution the best option, at least on the open-source side, supplemented by cloud inference and potentially enterprise offerings. He welcomes startups building around ComfyUI because their usage enlarges the ecosystem even without direct contributions. Text already works through custom nodes, but native support is lower priority unless a strong open text-diffusion model appears.

2025 New-Year Conversation: AI’s Pivotal Year, the First Year of Agents | A Conversation with ZhenFund’s 戴雨森 Why We Remain Extremely Bullish on AI in 2025 | AI Year-End Review