Streaming

The Streaming API lets you provision message queue clusters for high-throughput data streaming to OLCF compute resources.

Required Permission: data-streaming

Note

The Python and Go examples on this page use gRPC client packages that are not yet publicly available. ORNL-internal users may request access by contacting the S3M team: olcf-s3m@email.ornl.gov. The REST API (via curl or otherwise) is available to all users.

Available Backends

RabbitMQ →

Full-featured message broker with queues and bindings

S3M - Streaming: RabbitMQ

Redis →

High-performance Redis-compatible streams and pub/sub

S3M - Streaming: Redis

Cluster Lifecycle

All streaming clusters are created with a 7-day lifetime countdown.

Any token for a cluster’s project can be used to reset the countdown at any time, using the RabbitMQ or Redis API’s extension endpoint.

When the countdown reaches 0, clusters are deleted and cannot be restored.

List Available Backends

Discover available streaming backends and their resource options.

GET /olcf/v1alpha/streaming/list_backends

curl

curl -H @.env \
    https://s3m.olcf.ornl.gov/olcf/v1alpha/streaming/list_backends

Python

from s3m_apis_betterproto.streaming.v1alpha import ListBackendsRequest

client = factory.create_client(StreamingStub)
backends = await client.list_backends(ListBackendsRequest())

Go

import (
    "context"
    streamingpb "s3m.olcf.ornl.gov/apis/streaming/v1alpha"
)

client := streamingpb.NewStreamingClient(conn)
backends, err := client.ListBackends(context.Background(), &streamingpb.ListBackendsRequest{})

Python (requests)

import os
import requests

S3M_BASE_PATH = "https://s3m.olcf.ornl.gov/olcf/v1alpha/streaming"
S3M_TOKEN = os.getenv("S3M_TOKEN")

headers = {
    "Authorization": S3M_TOKEN,
}

response = requests.get(
    f"{S3M_BASE_PATH}/list_backends",
    headers=headers,
)

if response.ok:
    streaming_response = response.json()
    print(streaming_response)

else:
    raise ValueError("Request to S3M failed")