On Making HEARSAY

I have wanted to build something like this for twenty years. Interactive fiction where the characters actually listen. Conversations that become literature. The problem was always resources. A project like this would have required a team—programmers, voice actors, animators, writers—and funding to support them, and years of development, and people willing to believe in something that didn't exist yet.

I didn't have any of that. So the idea stayed an idea.

Generative AI changed what one person can attempt. Not because it does the work for me, but because it lets me work at a scale I couldn't reach alone. I have spent thousands of hours on this project over the past several months. The codebase runs to around 15,000 lines across frontend, backend, and documentation. The character prompts alone are thousands of words each. This is not prompt-and-accept. This is not a vending machine.

What I actually do: I write, I design, I direct. The models generate raw material—images, sounds, video, dialogue possibilities—and I edit, composite, rework, and reject until the output matches what I'm trying to make. When a character's voice isn't right, I rewrite the constraints and try again. When a generated image is close but wrong, I paint over it, cut it apart, combine it with something else. The iteration is constant.

I think of the models as collaborators, not tools. We consult each other. We debug each other. I praise them when something lands and tease them when it doesn't. This is a working relationship, not a button I press.

The characters in HEARSAY have bounded autonomy. I wrote their voices, their knowledge, their secrets, their relationships. The AI performs within those constraints, improvising in ways I didn't script. That emergence is the point. I am the author of the possibility space. The AI and the user together author each instance.

On the ethics of training data

I know the models learned from work by artists who didn't consent. I don't have a clean answer. What I can say is that this project has not displaced anyone. I would not have hired a team to build this—I couldn't have. Without these tools, HEARSAY would still be an idea I mentioned at parties. The choice was not "human artists or AI." The choice was "this exists or it doesn't."

If the project moves to commercial release, I would replace generated assets with work by human artists and musicians. The generative approach let me build a proof of concept in months instead of years. It is scaffolding. The structure it supports—the characters, the system, the narrative architecture—that is mine.

I am working in a moment when the implications of these tools are unresolved. I have tried to use them thoughtfully. I do not claim to have all the answers. I claim only to have made something that could not exist without both human authorship and machine collaboration, and to have done the work to make it mine.

— Caleb Weintraub