Use Streams for Real-Time Data

What this skill does

This skill enables workflows to stream data to clients in real-time using `dbos.WriteStream`, `dbos.CloseStream`, and `dbos.ReadStream`/`dbos.ReadStreamAsync`. Instead of accumulating results and returning them only when the entire workflow completes, it allows incremental output as data is processed. This is particularly useful for delivering live progress updates or streaming large outputs, such as results from LLMs, without waiting for full completion.

Streaming workflows maintain append-only, immutable streams identified by unique keys. Writes happen exactly once per item, and streams automatically close when the workflow ends, ensuring reliable and consistent real-time communication with clients.

Who it's for

This skill is valuable for performance marketers running complex data pipelines who need immediate feedback on batch processing steps. Growth leads handling multi-step user segmentation or campaign attribution can benefit from streaming partial results to dashboards in real time. SEO and PPC operators managing large-scale keyword or ad performance analyses can use streaming to monitor ongoing results and adjust bids or targets without waiting for full workflow completion.

Agencies delivering client reports or integrations that require live data updates will also find this skill essential for improving responsiveness and decision speed.

Key workflows

First, you define a workflow step that processes items individually, invoking `dbos.RunAsStep` for each unit of work. As each result becomes available, you call `dbos.WriteStream` to append it to a named stream. Once all items are processed, you explicitly close the stream with `dbos.CloseStream` to signal completion.

On the client side, you consume the stream either synchronously via `dbos.ReadStream`, which blocks until the stream closes, or asynchronously via `dbos.ReadStreamAsync`, which returns a channel emitting streamed values as they arrive. This allows you to build interfaces or monitoring tools that reflect workflow progress or output dynamically.

Common questions

Can I use multiple streams within a single workflow? Yes, workflows can have multiple uniquely keyed streams operating independently for different data channels. What happens if a workflow terminates before calling `dbos.CloseStream`? Streams are automatically closed when the workflow ends to prevent indefinite blocking readers. Is streaming guaranteed to deliver each write exactly once? Yes, stream writes are exactly-once and append-only, ensuring no duplication or data loss in the stream.

How to use in Metaflow

Attach this skill to any agent task that runs workflows requiring incremental or real-time data output. When enabled, your agents can write streaming results with `dbos.WriteStream` and clients can read them live using the `ReadStream` or `ReadStreamAsync` methods. Expect improved responsiveness for workflows that produce large or stepwise outputs, and smoother integration with dashboards or downstream processes. For details on setup and usage, see our internal documentation on workflow streaming.

For broader context, see our roundup of claude marketing skills, and read Claude Code reporting workflows for marketing agencies for related setup guidance.