Builder Story: dotsimulate x Daydream

Building StreamDiffusionTD Operator - a Real-Time Generative Video Operator for TouchDesigner, Powered by the Daydream API

Creator: Lyell Hintz (@dotsimulate)
Operator: StreamDiffusionTD
Backends Supported: Local + Daydream (Livepeer)

“Most of the things I do are not necessarily things that I have done before… not that I’m breaking new ground every time, but I’m attracted to things that aren’t laid out in a schema. I like to create my own endings.” — Lyell Hintz

Overview

StreamDiffusionTD is a TouchDesigner operator that connects real-time inputs like audio, sensors, and camera feeds to StreamDiffusion, enabling live generative visuals controlled in real time. With the Daydream API, it adds remote inference capabilities on top of the existing local GPU inference and unlocks more flexibility for users.

Built by Lyell Hintz, a technical artist and TouchDesigner developer, the operator is used in live shows, installations, and experimental workflows.

Why It Was Built

Lyell began working on the operator a few hours after StreamDiffusion was released on GitHub. He wanted to use it in TouchDesigner - a powerful tool for real time interactive content creation.

“TouchDesigner is the only place this could be controlled from… it can hook into everything else.”

From the start, he avoided creating a “black box.” The operator exposes core parameters like prompt, seed, and ControlNet weights, allowing users to adjust values and see results immediately.

Key Features

• Real-time video generation
• Prompt and seed morphing
• Dynamic ControlNet weighting
• Live input support: audio, sensors, camera
• Local GPU and Daydream backend options
• Instant visual feedback in TouchDesigner

Daydream API Integration

StreamDiffusionTD works with the Daydream API, which allows the operator to run on a remote GPU backend. This eliminates the major barrier of requiring a high-end PC with an NVIDIA RTX 4090 to run StreamDiffusion at professional quality, unlocking the flexibility to run it from any location, on any device form factor.

Just drop in your API key and hit “Start Stream.” The backend handles orchestration, model hosting, and frame delivery, so builders can stay focused on their creative and technical workflows.

Setup takes less than 1 minute and once installed, the configuration is remembered for future use.Daydream’s API brings new features to StreamDiffusion:

• Multi-controlnet: Mixing different controlnets for better artistic control
• IPAdapter: Use images as powerful style guides
• TensorRT: Better frame rate for smooth video output

Daydream is adding support for more real time video generation models, and developers can request features, suggest improvements, or build on top of the API itself. It aligns with the values of open tooling and community-led infrastructure.

How Artists can use StreamDiffusionTD in TouchDesigner

• Audio-reactive visuals for concerts
• Camera-driven generative visuals
• Real-time visuals for LED walls and stages
• TouchDesigner automation workflows

Because it's built inside TouchDesigner, the operator can be extended using Python, MIDI, OSC, or any other input TouchDesigner supports.

Current State

The operator is live and ready to use, with active development underway for new features and improved performance. It’s a great time to jump in, explore, and help shape what comes next.

Try it Yourself

Operator Access: patreon.com/dotsimulate
Community and Support: discord.gg/daydreamlive
API Keys can be requested in Daydream Discord