aiagent.org aiagent.org Ray
Distributed AI Compute & Serving

Ray & Ray Serve

Scale AI agent workloads from a single laptop to thousands of GPU nodes. Learn Ray Core (Tasks & Actors), the zero-copy Plasma Shared Memory store, and Ray Serve for multi-model LLM pipelines and batch embedding generation.

1 Core Architectural Concepts

Ray Tasks (Stateless)

Functions decorated with @ray.remote run asynchronously across cluster nodes without blocking the caller, returning future ObjectRefs.

Ray Actors (Stateful)

Stateful classes running on dedicated workers. Perfect for managing dedicated GPU model sessions, persistent scratchpads, or stateful agent supervisors.

Plasma Object Store

In-memory shared memory engine using Apache Arrow. Multiple worker processes read huge NumPy embedding matrices simultaneously via zero-copy deserialization.

Ray Serve Deployments

Framework-agnostic model serving layer providing dynamic request batching, replica autoscaling, fractional GPU allocation, and streaming HTTP responses.

2 Production Python: Parallel Embedding Generation

Distributed Ray Actor Pool for GPU Batch Ingestion

# distributed_embedder.py
import ray
@ray.remote(num_gpus=0.5)
class EmbeddingWorker:
def __init__(self):
# Model loaded once in GPU memory
self.model = load_model("all-MiniLM-L6-v2")
def embed_batch(self, chunks: list[str]) -> list[list[float]]:
return self.model.encode(chunks).tolist()
# Distribute across 8 worker replicas in parallel
ray.init()
pool = [EmbeddingWorker.remote() for _ in range(8)]

3 FAQ

How does Ray differ from Celery or Kubernetes? →
Celery is a task queue designed for slow I/O tasks with message brokers (RabbitMQ/Redis). Kubernetes schedules container processes. Ray is an in-memory distributed compute runtime built specifically for machine learning with microsecond scheduling overhead, shared memory (Plasma), and native fractional GPU scheduling.