Kinesis vs SQS vs MSK: Streams or Queues

Kinesis vs SQS vs MSK in one line: SQS is a queue, where a consumer deletes each message it has processed and nobody sees it again, while Kinesis Data Streams and MSK are logs, where records stay for a retention period and every consumer reads them at its own position, able to go back. MSK is managed Apache Kafka; Kinesis is the AWS-native log with AWS APIs. Pick queue or log first; the choice between the two logs is mostly ecosystem. Everything below comes from the AWS documentation listed at the end, as of October 2026.

Service	Model	After a consumer is done
SQS	queue; consumers compete for messages	it deletes the message; no replay
Kinesis Data Streams	sharded log; each consumer has a position	records stay until retention ends
MSK	Apache Kafka topics, partitions and offsets	records stay per the topic's retention

The problem it solves

"Put a queue between them" is the default way to decouple two services. It stops being enough when:

• Several independent consumers need every event, such as billing, analytics and a fraud model.
• Order matters per key, such as all events of one order in sequence.
• You need to reprocess history after fixing a bug, or start a new consumer that reads last week.

A queue answers "who does this job?". A log answers "what happened, in what order?". The three services split along that line.

The intuition

SQS: work, done once. A consumer receives a message, which is hidden from other consumers for the visibility timeout, 30 seconds by default and up to 12 hours. When the work succeeds, the consumer deletes it. If it does not, the message becomes visible again and someone retries. Messages are kept for 4 days by default, from 60 seconds up to 14 days. Several consumers on one queue split the messages between them; giving three services every message takes three queues behind an SNS topic, as in SQS vs SNS vs EventBridge. Ordering per group needs a FIFO queue; see SQS Standard vs FIFO.

Kinesis Data Streams: a log cut into shards. A stream is a set of shards. Each record has a partition key, a sequence number and a data blob of up to 1 MB. The partition key is hashed with MD5, and the hash picks the shard, so one key's records stay together in one shard, in order. A shard is the unit of capacity: up to 1,000 records or 1 MB per second of writes, and 2 MB per second of reads. Records stay for 24 hours by default, extendable up to 8,760 hours, that is 365 days, at extra charge. Many applications can read the same stream independently, each tracking its own position; the Kinesis Client Library keeps those checkpoints in DynamoDB tables. Consumers share a shard's 2 MB per second unless they register for enhanced fan-out, which pushes each registered consumer its own 2 MB per second per shard.

MSK: managed Apache Kafka. MSK runs the control plane, creating, updating and deleting clusters and replacing failed brokers, while your applications use Kafka's own data-plane operations to produce and consume. It runs open-source Kafka versions, so existing Kafka applications, tools and plugins work without code changes. The model is Kafka's: topics split into partitions, consumer groups and committed offsets, which map almost one to one onto streams, shards and checkpoints. MSK Serverless removes cluster sizing, and MSK Connect runs Kafka Connect connectors.

Watch it run

The lesson animation is the queue side of the story. An order service publishes one event to an SNS topic, which pushes a copy into one SQS queue per team. Shipping is down, so its copy waits; a received message is hidden for the visibility timeout and deleted when done; a poison message moves to the dead-letter queue after maxReceiveCount receives. Notice that fan-out needed one queue per team, where a log gives every consumer the whole history from one stream.

The code

A toy model, not SQS or Kinesis, built on the lesson's queue model: a Queue with receive, visibility timeout and delete, and a Log that appends by hashed partition key and keeps one position per consumer:

import hashlib

class Queue:
    """Toy model of an SQS queue: a received message is hidden for the
    visibility timeout; a deleted one is gone. Consumers compete."""
    def __init__(self, visibility=30):
        self.visibility, self.messages, self.next_id = visibility, {}, 0

    def send(self, body):
        self.messages[self.next_id] = {"body": body, "visible_at": 0}
        self.next_id += 1

    def receive(self, now, max_n=2):
        batch = []
        for mid, m in self.messages.items():
            if m["visible_at"] <= now and len(batch) < max_n:
                m["visible_at"] = now + self.visibility
                batch.append((mid, m["body"]))
        return batch

    def delete(self, mid):
        self.messages.pop(mid, None)

class Log:
    """Toy model of a Kinesis stream or Kafka topic: records are appended to
    the shard whose hash range holds MD5(partition key); every consumer keeps
    its own position in each shard."""
    def __init__(self, shards=2):
        self.shards, self.checkpoints = [[] for _ in range(shards)], {}

    def put(self, key, value):
        h = int(hashlib.md5(key.encode()).hexdigest(), 16)      # 128-bit
        self.shards[h * len(self.shards) >> 128].append((key, value))

    def read(self, consumer, shard, from_start=False):
        pos = 0 if from_start else self.checkpoints.get((consumer, shard), 0)
        return self.shards[shard][pos:]

    def checkpoint(self, consumer, shard, upto):
        self.checkpoints[(consumer, shard)] = upto

events = [("order-1", "created"), ("order-2", "created"), ("order-1", "paid"),
          ("order-3", "created"), ("order-2", "paid"), ("order-1", "shipped")]

Two consumers on the queue split the six events, a crash before delete brings messages back after the timeout, and once everything is deleted a new consumer finds nothing:

# Queue: billing and analytics poll the same queue and split the work.
q = Queue(visibility=30)
for key, what in events:
    q.send(key + ":" + what)
billing = q.receive(now=0)
analytics = q.receive(now=0)
print("billing  ", [b for _, b in billing])
print("analytics", [b for _, b in analytics])
for mid, _ in analytics:
    q.delete(mid)                       # analytics finishes and deletes
# billing crashes before deleting: its messages come back after the timeout
print("at t=31  ", [b for _, b in q.receive(now=31, max_n=10)])
for mid in list(q.messages):
    q.delete(mid)
print("new consumer sees", q.receive(now=60))
# billing   ['order-1:created', 'order-2:created']
# analytics ['order-1:paid', 'order-3:created']
# at t=31   ['order-1:created', 'order-2:created', 'order-2:paid', 'order-1:shipped']
#            (billing's two are back, plus the two nobody had taken)
# new consumer sees []

The same events in a two-shard log. Both consumers see all six, one key's events stay in order in one shard, a restart resumes from the checkpoint, and a brand-new consumer replays from the start:

# Log: both consumers read every record, each at its own position.
log = Log(shards=2)
for key, what in events:
    log.put(key, what)
for shard in range(2):
    print("shard", shard, log.shards[shard])
seen = {c: sum(len(log.read(c, s)) for s in range(2)) for c in ("billing", "analytics")}
print(seen)

# analytics processes shard 0 up to record 2, checkpoints, then restarts
log.checkpoint("analytics", 0, 2)
print("analytics resumes at", log.read("analytics", 0))
print("audit replays from 0", log.read("audit", 0, from_start=True))
# shard 0 [('order-1', 'created'), ('order-2', 'created'), ('order-1', 'paid'), ('order-2', 'paid'), ('order-1', 'shipped')]
# shard 1 [('order-3', 'created')]
# {'billing': 6, 'analytics': 6}
# analytics resumes at [('order-1', 'paid'), ('order-2', 'paid'), ('order-1', 'shipped')]
# audit replays from 0 [('order-1', 'created'), ('order-2', 'created'), ('order-1', 'paid'), ('order-2', 'paid'), ('order-1', 'shipped')]

Per-key order checked on 5,000 events for 50 keys across four shards, read by three consumers in random shard order. The uneven shard sizes are the other lesson: with few keys, hashing does not spread load evenly.

import random

# Per-key order check: 5,000 events for 50 keys on 4 shards. Every consumer,
# reading shard by shard, must see each key's events in the order produced.
random.seed(11)
big, produced = Log(shards=4), {}
for n in range(5000):
    key = "order-%d" % random.randrange(50)
    big.put(key, n)
    produced.setdefault(key, []).append(n)
ok = True
for consumer in ("billing", "analytics", "audit"):
    got = {}
    for shard in random.sample(range(4), 4):          # any shard order
        for key, n in big.read(consumer, shard):
            got.setdefault(key, []).append(n)
    ok &= got == produced
print(ok, [len(s) for s in big.shards])
# True [1779, 1064, 772, 1385]

The complexity

The costs differ in kind:

• SQS scales without shards to plan. The work is per message: receive, process, delete. Fan-out to many services means one copy per queue.
• Kinesis capacity is the number of shards in provisioned mode, or managed for you in on-demand mode. One hot partition key is capped by one shard's write limit. Extra consumers cost read throughput unless they use enhanced fan-out.
• MSK capacity is brokers and partitions you size, or MSK Serverless, with Kafka's full feature set and its operational concepts.

Where it goes wrong

• Using SQS when two services need every event. They will split the messages. Use one queue each, or a log.
• Expecting replay from SQS. A deleted message is gone. Keep raw events in a log or in storage if you may reprocess.
• A low-cardinality partition key. A few keys mean a few hot shards, exactly as the check above shows.
• Checkpointing before processing. A crash between the two skips records. Checkpoint after the work, and make the work idempotent, because a restart can read a record twice.

When it shows up in interviews

Event pipelines, clickstreams and ad-click counting all hit this choice; the interviewer listens for queue versus log, per-key ordering and consumer restarts. Delivery guarantees around it are in message queues explained and store-and-forward vs fire-and-forget; the queue side is in the SQS vs SNS vs EventBridge lesson.

How to say it in an interview

"First, queue or log. If each task should be done once by one of a pool of workers, SQS: a received message is hidden for the visibility timeout and deleted on success, with a dead-letter queue for repeated failures. If several consumers need every event, in order per key, with replay, a log. On AWS that is Kinesis Data Streams, where the partition key picks a shard, the shard is the unit of throughput and ordering, and each application checkpoints its own position within the retention period. If the team already runs Kafka, or needs Kafka Connect and Kafka's tooling, MSK gives the same model as managed Kafka."

Frequently asked questions

What is the difference between Kinesis and SQS?

SQS is a queue: a consumer receives a message, processes it and deletes it, and several consumers split the messages between them. Kinesis Data Streams is a sharded log: records stay for the retention period, every application reads all of them at its own position, and records with the same partition key stay in order.

When should I use MSK instead of Kinesis?

Choose MSK when you already use Apache Kafka or need its ecosystem, such as Kafka Connect, Kafka clients and tooling, since it runs open-source Kafka without code changes. Choose Kinesis Data Streams for an AWS-native stream with AWS APIs and no brokers to size, especially in on-demand mode.

Can SQS replay messages?

No. Once a consumer deletes a message, it is gone, and undeleted messages expire after the retention period, at most 14 days. For replay, use a log such as Kinesis Data Streams or Kafka on MSK, where consumers can read again from an earlier position within retention.

Sources

• Amazon SQS visibility timeout — AWS Documentation
• Amazon SQS message quotas — AWS Documentation
• Amazon Kinesis Data Streams terminology and concepts — AWS Documentation
• Develop enhanced fan-out consumers with dedicated throughput — AWS Documentation
• What is Amazon MSK? — AWS Documentation