Skip to main content

How to migrate immutable stream properties

Use whena stream property that JetStream will not let you change has to change anyway.

You getblue-green stream recreation, what it costs, and when it refuses to run.

When is migration needed?​

Most stream config changes are mutable: the transport applies them with a plain update on startup. Migration is needed only for immutable properties that NATS locks after creation. The full property classification lists them.

PropertyExample changeRequires migration
storageFile -> MemoryYes
retentionWorkqueue -> LimitsNot allowed: controlled by the transport
max_age, num_replicas, etc.Any valueNo: mutable, updated automatically

How to enable​

import { StorageType } from '@nats-io/jetstream';
import { JetstreamModule } from '@horizon-republic/nestjs-jetstream';

JetstreamModule.forRoot({
  name: 'orders',
  servers: ['nats://localhost:4222'],
  provisioning: { allowDestructiveMigration: true },
  events: {
    stream: { storage: StorageType.Memory },
  },
});

Without the flag, the transport logs a warning, applies only the mutable part of the change and keeps the existing stream. Once every pod runs the new config, remove the flag: it is needed only for the migration itself.

How it works​

The transport recreates the stream blue-green via NATS stream sourcing, a server-side copy:

Phase 1/4  Quiesce: remove the original's subjects
           (new publishes are rejected loudly instead of being acked
            into a stream that is about to be deleted)

Phase 2/4  Create backup stream ← sourcing ← original
           (server-side copy; completion is tracked via source lag)

Phase 3/4  Delete original, create it again with the new config

Phase 4/4  Original ← sourcing ← backup
           -> drain -> delete backup -> detach source

The stream keeps its name. Each pod's startup or self-healing recreates the consumers afterwards.

A producer either gets an ack (the message survives the migration) or a rejection it can retry, never an ack for a message that later disappears.

Backup stream as a distributed lock​

The backup stream ({stream}__migration_backup) holds the messages between delete and recreate, and also acts as a lock: other pods' self-healing sees it and waits instead of recreating consumers.

What happens during migration​

To publishers​

From the quiesce (Phase 1) until the restore completes, publishes to the migrating stream are rejected: no stream matches the subject, so producers get "no stream matches subject" or no-responders errors. Both client.emit() and client.send() reject, and the caller can retry.

The window lasts as long as the copy. That is deliberate: a rejected publish can be retried, while an ack into a stream about to be deleted is silent data loss. For a migration with no rejections at all, pause publishers during a maintenance window.

To consumers on other pods (rolling updates)​

Deleting the stream breaks other pods' consumers. Self-healing retries with backoff, sees the __migration_backup stream and does not recreate the consumer until the backup is gone. Then the next retry creates it and consumption resumes.

Waiting keeps old pods from overwriting a newer pod's consumer config, and keeps consumers from taking workqueue messages while they are being sourced back.

To the migrating pod itself​

Its startup blocks until all phases complete, then it creates consumers and starts processing.

Duration​

Sourcing runs on the server, so no message crosses the network to the application, and the copy is bounded by the server's disk or memory. Duration grows with message count and size. No measured figures are published: time a migration of a realistic stream on your own hardware before a rolling update.

Error handling​

  • Failure before the original is deleted. The quiesce is rolled back (subjects restored), the backup is removed, and an error is thrown: the original stream is intact and serving traffic.
  • Failure after the original is deleted. The backup is the only copy of the data and is always preserved. Restoration resumes automatically on the next application startup.
  • Sourcing timeout (30 s). Same as above: the backup is preserved and the restore resumes on the next startup.
  • Process killed mid-migration. Detected on the next startup: a stranded backup is restored into the stream (recreating it first if the crash happened between delete and create), then cleaned up.
  • Instances migrating concurrently (rolling deploy). Backups carry a start timestamp. An instance that finds a backup younger than 90 s waits for that migration (up to 60 s) instead of interfering; older backups are treated as leftovers and recovered.

Manual streams are never migrated​

Streams in ManagementMode.Manual are never created, updated or migrated, whatever provisioning.allowDestructiveMigration says. With mixed ownership the flag applies to the Auto-managed streams only, and combined with a global Manual it logs a warning at boot. See Bring Your Own Infrastructure.

Limitations​

  • retention is not migratable. The transport controls it (Workqueue for event, command and schedule streams, Limits for broadcast, ordered and DLQ). A mismatch throws on startup.
  • The publisher window is inherent. NATS has no atomic stream rename or swap, so publishes reject from the quiesce until the restore.
  • provisioning.allowDestructiveMigration is one flag for every Auto-managed stream of the service. Any of them with an immutable conflict is migrated.
  • The shared broadcast stream is never migrated. Recreating broadcast-stream would delete every other service's durable consumers and replay retained history to them, so an immutable conflict on it throws instead. Coordinate that change by hand.
  • The schedule stream is never migrated. NATS refuses to source into a stream with allow_msg_schedules, so an immutable conflict on {service}__microservice_sched-stream throws. Revert the change, or recreate the stream by hand: the next boot reconciles the schedules declared in decorators, and dynamic schedules have to be created again.
  • Manual streams are never migrated. See the section above.

See also​