# shove

> Type-safe async pub/sub for Rust on RabbitMQ, AWS SNS+SQS, NATS JetStream, Apache Kafka, or in-process.

## Docs

- [Getting Started](/getting-started): You can run a real `shove` publisher and consumer in 60 seconds against the **in-process backend** — no Docker, no AWS credentials, no config. This page is a copy-paste walkthrough.
- [Reference](/reference): The full API reference for `shove` is auto-generated rustdoc on **[docs.rs/shove](https://docs.rs/shove)**.
- [Backend Ops Notes](/ops/backends): Setup is covered by the per-backend overview pages. This page covers what tends to bite in production: which knobs to set before you go live, which surprises to anticipate, and which monitoring to put in place. One section per backend.
- [Performance Tuning](/ops/performance): Most `shove` performance questions resolve to four levers: prefetch count, worker count, the concurrent-processing flag, and (negatively) transactional mode. Knowing which lever to pull first — and which to ignore — is what this page is for. Numbers below are from a single-node setup and intended as a starting baseline; always measure on your own hardware and workload.
- [Audit Logging](/guides/audit): Compliance requirements, fraud investigation, and debugging all share the same need: a durable record of what happened, when, and what the outcome was. `shove` provides a transparent wrapper — `Audited\<H, A\>` — that captures this for every handler invocation, regardless of backend. The wrapper is a drop-in: it implements `MessageHandler\<T\>` with the same associated `Context` type as the inner handler, so it fits anywhere a handler fits.
- [Exactly-Once (RabbitMQ)](/guides/exactly-once): Charging a payment card twice, sending a confirmation email to a customer twice, triggering an external API that has no idempotency tokens — these are the cases where at-least-once delivery causes real harm and exactly-once delivery matters. `shove` is at-least-once by default on every backend. RabbitMQ has an **opt-in transactional mode** that makes routing decisions atomic, eliminating the publish-then-ack race that produces duplicates. It is slower — expect roughly 10–15× lower throughput per channel compared to the default confirm-mode consumer — so reserve it for handlers with irreversible side effects where duplicates cause real harm.
- [Consumer Groups & Autoscaling](/guides/groups): When your topic fills up faster than one consumer can drain it, you add consumers. When traffic drops off at night and you don't need ten idle workers, the autoscaler removes them. `broker.consumer_group()` is `shove`'s answer to both: a coordinated, min/max-bounded group of consumers that the autoscaler can grow or shrink based on queue depth. Available on RabbitMQ, NATS JetStream, Apache Kafka, Redis Streams, and InMemory.
- [Liveness Probes](/guides/liveness): Kubernetes liveness and readiness probes need a single, fast, bounded answer to "can this process still talk to the broker?". `shove` exposes `Broker<B>::ping` for exactly that question: one bounded RPC against the cluster, returning `Ok(())` iff it completes within the deadline. Probe policy — retries, failure thresholds, reconnect — belongs to the caller, so `ping` itself does none of those things.
- [Observability](/guides/observability): When something goes wrong — a handler is slow, a topic is backing up, a retry loop is burning through the budget — you need to know where to look. `shove` integrates with the standard Rust `tracing` ecosystem. Every interesting event — handler invocation outcomes, retry routing, DLQ routing, group scaling, autoscaler decisions, connection errors — is emitted as a structured `tracing` event with named fields. Add a subscriber and you have a full observability trail without any instrumentation code in your handlers.
- [Retries, Hold Queues & DLQs](/guides/retries): Transient failures recover. Permanent failures don't. `shove` gives you the vocabulary to express the difference: **hold queues** park a message for a configurable delay before redelivering it, giving downstream services time to recover; a **DLQ** catches messages that have exhausted all retries or are permanently invalid, so they don't disappear silently but instead wait for human inspection and replay. Together they turn a binary ack/reject model into a resilient, observable pipeline.
- [Sequenced Topics](/guides/sequenced): Some messages depend on each other. A ledger entry assumes the previous balance was applied. A state-machine transition assumes the entity is in the right prior state. An audit record references the preceding record's hash. When processing message N assumes message N−1 has already been applied, you need `shove`'s sequenced delivery: messages with the same key arrive one at a time, in strict publish order.
- [Shutdown & Exit Codes](/guides/shutdown): Rolling deploys, Kubernetes preStop hooks, systemd `ExecStop`, CI pipelines — all of them send a signal and expect the process to finish cleanly before they terminate it. `shove` ships a shutdown contract that fits all of these: stop accepting new work, finish in-flight handlers, and surface the result as a process exit code that operators can read and act on.
- [The Broker<B> Pattern](/concepts/broker): The central value of `shove` is portability: write your topic definitions, handler logic, and consumer registrations once — then run the same code against an in-process broker in tests and a real broker in production. Swapping backends means changing one marker type and one config struct. Nothing else changes.
- [Codecs](/concepts/codecs): A codec controls how a topic's messages cross the wire. Every topic carries a `Codec` associated type that decides how `T::Message` is encoded on `publish` and decoded on `consume`. The default is JSON, which means existing code keeps working without any changes; opting into a different encoding is a one-line edit on the topic definition.
- [Handlers & Context](/concepts/handlers): A handler is your business logic. It receives a message, does work — writes a record, calls an API, updates state — and returns an `Outcome` that tells the consumer what to do next. The library calls `handle()` for each delivered message and routes the message based on the returned `Outcome`. Handlers are parameterized on a `Topic`, which prevents accidentally sharing a handler between two topics that happen to carry the same message type.
- [Outcomes & Delivery](/concepts/outcomes): Handlers in `shove` do not write to queues directly. They return one of four `Outcome` values after processing a message, and the consumer routes the message accordingly. The `Outcome` is the only mechanism a handler has to influence what happens next.
- [Topics & Topology](/concepts/topics): A topic is how you express a domain event as a type. When you define `OrderSettlement`, you're declaring that these events flow through a specific queue, carry a specific message shape, and obey specific retry rules. Publishers, consumer registrations, and topology declarations all reference the topic; queue names and message types are derived from it.
- [Integrate with your existing stack](/backends/choosing): `shove` sits on top of the broker you already run. Drop it into your code, point it at your broker, and your handlers get retries, ordering, consumer groups, and audit logging without rewriting the rest of your service.
- [In-process](/backends/inmemory): The fastest way to get started with `shove` and the right choice for tests. No Docker, no AWS, no broker process — the broker lives in process RAM. Suitable for unit tests, integration tests, and single-process applications where you want the full pub/sub API without external infrastructure.
- [Apache Kafka](/backends/kafka): If you already have a Kafka cluster or need streaming semantics — replayable log, partition-key ordering, consumer-lag autoscaling — this backend fits naturally. Best for high-throughput pipelines where log retention and replay matter.
- [NATS JetStream](/backends/nats): If NATS is already in your stack, this backend gives you everything you need: subject-routed messaging with per-key ordering, JetStream-backed durability, coordinated pull consumers, and pending-message autoscaling. Fast, simple, and lighter to operate than AMQP brokers.
- [RabbitMQ](/backends/rabbitmq): If you already run RabbitMQ, this backend slots in with zero infrastructure changes. It covers the full feature surface: hold-queue retry topologies, dead-letter routing, coordinated consumer groups with queue-depth autoscaling, per-key FIFO delivery via consistent-hash exchange routing, and optional AMQP transactional mode for exactly-once routing on handlers with irreversible side effects.
- [Redis / Valkey Streams](/backends/redis): If Redis or Valkey is already in your stack, this backend gives you stream-based messaging with FNV-1a shard routing for per-key ordering, consumer groups, hold-queue retries, and queue-depth autoscaling — without adding another broker to your infrastructure.
- [AWS SNS + SQS](/backends/sqs): If you're already on AWS and don't want to run a broker, this is your path. SNS fans messages out to SQS queues; shove handles polling, retries, DLQ routing, and consumer supervision. No broker process to provision or maintain — pay AWS for what you use.
- [SQS — Audited Consumer](/backends/sqs/examples/audited): Use this when you want to confirm that trace IDs survive the SQS hold-queue round-trip. The same UUID appears on both the initial `Outcome::Retry` delivery and the successful second attempt — exactly the property you need when correlating retries in a payment audit log or compliance record. The `AuditHandler` API is identical across backends.
- [SQS — Autoscaler](/backends/sqs/examples/autoscaler): Use this when you want to see queue-depth-based autoscaling on SQS end-to-end. Sixty tasks are published at once; with one slow consumer the queue fills faster than it drains, triggering scale-up to five consumers. The autoscaler then scales back once the queue is empty. Run this before production to tune the hysteresis and cooldown settings for your traffic pattern.
- [SQS — Basic Pubsub](/backends/sqs/examples/basic): Use this as your starting point for SQS integration: it covers the three non-sequenced topology shapes (minimal queue, DLQ, escalating retry backoff), all three SQS-compatible `Outcome` variants, and all three publish variants against a LocalStack backend. The SQS counterpart to the [RabbitMQ basic example](/backends/rabbitmq/examples/basic) — same programming model, different infrastructure.
- [SQS — Concurrent Pubsub](/backends/sqs/examples/concurrent): Use this when sizing the prefetch and concurrency configuration for SQS I/O-bound handlers. The example shows the throughput improvement from enabling concurrent processing, and highlights the SQS-specific constraint: `ReceiveMessage` returns at most 10 messages per call, so the maximum useful prefetch is lower than for AMQP backends.
- [SQS — Consumer Groups](/backends/sqs/examples/consumer-groups): Use this when you want to observe how a pool of SQS poll workers drains a queue over time. Because SQS has no broker-side coordinated-group primitive, the registry spawns independent consumers; the example shows how to monitor `messages_ready` and `messages_not_visible` as the backlog drains — useful for sizing your worker pool before adding the autoscaler.
- [SQS — Sequenced Pubsub](/backends/sqs/examples/sequenced): Use this when you need to verify sequenced delivery semantics on SQS: `SequenceFailure::Skip` lets a sequence continue after one bad message; `SequenceFailure::FailAll` quarantines the rest. The example asserts exact totals for both policies, giving you a concrete end-to-end test you can run against LocalStack before deploying to AWS.
- [SQS — Stress](/backends/sqs/examples/stress): Use this to measure SQS throughput on your workload profile and compare it against the [InMemory baseline](/backends/inmemory/examples/stress) and [RabbitMQ results](/backends/rabbitmq/examples/stress) to quantify the HTTP round-trip cost. Each consumer is an independent poll worker, reflecting SQS's architecture — the numbers here set realistic expectations before sizing a production SQS deployment.
- [Redis — Audited Consumer](/backends/redis/examples/audited): Wraps a payment handler in `MessageHandlerExt::audited` and pipes audit records through a simple stdout `AuditHandler` impl. Every delivery emits one structured audit line with the payload, outcome, duration, and trace ID — exactly what the [Audit Logging guide](/guides/audit) describes, applied to the Redis backend.
- [Redis — Autoscaler](/backends/redis/examples/autoscaler): Wires the Redis backend into the autoscaler. A consumer group with `min=1, max=5` starts with a single worker; the autoscaler polls the group's consumer lag (messages awaiting delivery, as reported by Redis — already-processed history does not count as backlog) and spawns workers up to `max` while a burst of forty messages is pending, then drains back down to `min` as the backlog clears. The driver thread prints periodic stats so you can watch the scale decisions.
- [Redis — Basic](/backends/redis/examples/basic): A minimal Redis Streams publish/consume round-trip. Three `OrderPaid` messages are published to a topic with a hold queue and DLQ, then consumed by a single-worker consumer group until Ctrl-C.
- [Redis — Consumer Groups](/backends/redis/examples/consumer-groups): Registers a single coordinated consumer group with `min_consumers=2, max_consumers=4`, publishes a batch of twenty work items, and lets the group drain them on Ctrl-C (or a 30 s timeout). Demonstrates the `RedisConsumerGroupConfig` builder shape that mirrors every other coordinated-group backend in shove.
- [Redis — Sequenced](/backends/redis/examples/sequenced): Demonstrates per-key ordering with `SequencedTopic` and `register_fifo`. Interleaved ledger entries for three accounts are published; entries for the same account are always consumed in strict publish order, regardless of how many shard consumer tasks run.
- [Redis — Stress](/backends/redis/examples/stress): Throughput benchmark for the Redis backend. Publishes a configurable burst of messages on a non-sequenced topic and consumes them through a coordinated consumer group with concurrent processing. Useful for sanity-checking client tuning (`prefetch_count`, `max_consumers`, `response_timeout`) against your Redis or Valkey deployment before rolling production traffic.
- [RabbitMQ — Audited Consumer](/backends/rabbitmq/examples/audited): Use this when you want to verify that trace IDs are preserved across retries on RabbitMQ. The handler returns `Outcome::Retry` on the first delivery; both delivery attempts emit an audit record with the same UUID — proving that trace identity survives the hold-queue round-trip. The pattern directly applies to payment processing, compliance logging, and any workflow where correlating retries to the original delivery matters.
- [RabbitMQ — Basic Pubsub](/backends/rabbitmq/examples/basic): Use this as your starting point for RabbitMQ integration: it covers every non-sequenced topology shape (minimal queue, DLQ, escalating retry backoff, deferred delivery), all four `Outcome` variants, and all three publish variants against a real broker. The canonical reference for how [topic definitions](/concepts/topics) and [handler outcomes](/concepts/outcomes) map to RabbitMQ.
- [RabbitMQ — Concurrent Pubsub](/backends/rabbitmq/examples/concurrent): Use this when deciding whether to enable concurrent processing for I/O-bound handlers. A 100 ms handler processes 20 messages sequentially in ~2 s; the same handler with `with_concurrent_processing` processes them in ~0.2 s — a 10x speedup visible in the printed summary. The example also confirms that acknowledgements remain in-order regardless of which mode is active.
- [RabbitMQ — Consumer Groups](/backends/rabbitmq/examples/consumer-groups): Use this when you want to see the RabbitMQ autoscaler in action: start at one consumer, watch the count climb as queue depth rises, and watch it fall once the queue drains. The example uses the one-line `enable_autoscaling` API, showing how `shove` adapts consumer count to actual load with the autoscaler lifecycle folded into `run_until_timeout`.
- [RabbitMQ — Exactly-Once](/backends/rabbitmq/examples/exactly-once): Use this when you need to eliminate duplicate deliveries for handlers with irreversible side effects — payment charges, outbound notifications, external API calls without idempotency tokens. The example shows how a single `with_exactly_once()` call wraps every routing decision in an AMQP transaction, covering all three routing paths: ack, retry-then-ack, and reject-to-DLQ.
- [RabbitMQ — Sequenced Pubsub](/backends/rabbitmq/examples/sequenced): Use this when you need to decide between `SequenceFailure::Skip` and `SequenceFailure::FailAll` for your ordering policy. Ledger entries for two accounts are published, one entry is deliberately rejected, and the example asserts the resulting totals — making the behavioral difference between the two policies concrete and verifiable against a real RabbitMQ broker.
- [RabbitMQ — Stress](/backends/rabbitmq/examples/stress): Use this to measure RabbitMQ throughput on your own hardware and compare it against the [InMemory baseline](/backends/inmemory/examples/stress) to quantify the AMQP round-trip cost. The harness sweeps handler profiles and consumer counts, making it straightforward to find the prefetch and concurrency configuration that maximises throughput for your workload.
- [NATS — Audited Consumer](/backends/nats/examples/audited): Use this when you need to confirm that audit trace IDs survive hold-queue retries on NATS. The handler returns `Outcome::Retry` on first delivery; both attempts emit an audit record with the same UUID, demonstrating that the `trace_id` propagates correctly through JetStream's header mechanism.
- [NATS — Basic](/backends/nats/examples/basic): Use this as your starting point for NATS JetStream integration. Three `OrderCreated` events are published and consumed via a coordinated consumer group, showing how `shove`'s generic `Broker\<Nats\>` API maps to JetStream streams and durable consumers with hold queues and DLQ.
- [NATS — Sequenced](/backends/nats/examples/sequenced): Use this when you need per-user (or per-entity) FIFO ordering on NATS. Events for two users are published in interleaved order; each user's events arrive strictly in sequence while the two users are processed concurrently across different shards. Shows the `NatsConsumer::run_fifo` path for sequenced delivery.
- [NATS — Stress](/backends/nats/examples/stress): Use this to measure NATS JetStream throughput and compare it against the [InMemory baseline](/backends/inmemory/examples/stress) and [RabbitMQ results](/backends/rabbitmq/examples/stress). The numbers reveal JetStream's persistence cost versus in-process, and the protocol efficiency difference versus AMQP — useful data before choosing a backend or sizing a deployment.
- [Kafka — Audited Consumer](/backends/kafka/examples/audited): Use this when you need to confirm that audit trace IDs survive Kafka hold-queue retries. The handler returns `Outcome::Retry` on first delivery; both delivery attempts emit a JSON audit record with the same UUID, demonstrating that the `trace_id` propagates correctly through Kafka record headers across the hold-queue topic round-trip.
- [Kafka — Basic](/backends/kafka/examples/basic): Use this as your starting point for Kafka integration. Three `OrderCreated` messages are published and consumed via a coordinated consumer group, showing how `shove`'s generic `Broker\<Kafka\>` API maps to rdkafka producers, consumer groups, and partition assignment — including how hold queues are implemented as separate Kafka topics.
- [Kafka — MSK IAM](/backends/kafka/examples/msk_iam): Round-trips a single `Ping` message against an IAM-enabled Amazon MSK cluster, using `SASL_SSL` + `OAUTHBEARER` with auto-refreshed presigned tokens. Run this against a real MSK cluster you have access to — it's the canonical end-to-end check for the `kafka-msk-iam` feature.
- [Kafka — Schema Registry](/backends/kafka/examples/schema-registry): Configure a Kafka consumer to decode [Confluent Schema Registry](https://docs.confluent.io/platform/current/schema-registry/index.html)-framed messages, and a publisher to produce them. On each consumed message the consumer strips the wire frame (magic byte `0x00` + big-endian schema id), resolves the schema from the registry (cached in memory), validates the schema subject against the accepted set, then delegates to the topic's `Codec` for the actual payload decode. On the produce side, an opt-in publisher wraps each encoded payload in the same frame — see [Producing SR-framed messages](#producing-sr-framed-messages).
- [Kafka — Sequenced](/backends/kafka/examples/sequenced): Use this when you need per-entity FIFO ordering on Kafka. Events for two users are published in interleaved order; each user's events arrive strictly in sequence. The example also highlights the key Kafka constraint: partition count caps the maximum number of active consumers, so partition count must be planned before topic creation.
- [Kafka — Stress](/backends/kafka/examples/stress): Use this to measure Kafka throughput on your workload and compare it against the [InMemory baseline](/backends/inmemory/examples/stress) and [NATS results](/backends/nats/examples/stress). The numbers quantify Kafka's write-ahead log cost relative to lighter backends — useful data when choosing between Kafka and NATS, or when sizing partitions and consumer counts for a production deployment.
- [InMemory — Audited Consumer](/backends/inmemory/examples/audited): Use this when you want to add audit logging — trace IDs, outcomes, handler duration — to an existing handler without changing its logic. The `Audited` decorator wraps any `MessageHandler` transparently; this example shows the wiring with no external services required.
- [InMemory — Basic](/backends/inmemory/examples/basic): Use this when you want to see the full publish/consume cycle with no external services: declare a topic, publish five messages, handle them, and exit cleanly. The simplest end-to-end proof that your topic definition and handler are wired correctly.
- [InMemory — Consumer Groups](/backends/inmemory/examples/consumer-groups): Use this when you want to see how consumer groups scale workers to match queue depth. One hundred messages are published and processed by three to six workers in parallel, each simulating 5 ms of I/O — a practical demonstration of the min/max worker range and prefetch configuration, all in-process.
- [InMemory — Sequenced](/backends/inmemory/examples/sequenced): Use this when you want to verify per-key ordering before wiring a real broker. Messages for three accounts are published in interleaved order; each account's entries arrive in strict sequence, proving that ledger-style FIFO delivery works correctly. No external infrastructure required.
- [InMemory — Stress](/backends/inmemory/examples/stress): Use this to establish your framework performance baseline before adding a real broker. Everything runs in-process, so results here isolate `shove`'s scheduling and dispatch overhead from network and broker costs — the ceiling against which all durable backends are measured.