RDS to KV Online Migration

Problem

The metadata control plane needed to migrate its persistence layer from RDS to KV for better scalability, but the service was a critical dependency for object access — any downtime or data loss was unacceptable.

My Role

Designed the rollout strategy, implemented the migration wrapper and dual-write logic, built the consistency verification and repair flow, and participated in rollout and rollback planning.

Architecture / Approach

Designed a 6-stage online migration: (1) introduce migration wrapper with RDS as sole backend, (2) enable dual-write from RDS to KV, (3) backfill historical data with a migration tool, (4) switch primary to KV with RDS as secondary, (5) run consistency verification and repair using updatedAt timestamps, (6) cut over to KV-only. Each stage supported rollback to RDS.

• Migration wrapper abstracting the storage layer for dynamic traffic switching
• Dual-write strategy during transition with configurable write ordering
• Backfill tool for copying existing records from RDS to KV
• Consistency verification with timestamp-based repair mechanism
• Rollback capability maintained at every migration phase

Architecture Diagrams

flowchart LR
    S0["Stage 0\nRDS Only"]
    S1["Stage 1\nMigration Wrapper\nRead/Write: RDS"]
    S2["Stage 2\nDual Write\nWrite: RDS → KV\nRead: RDS"]
    S3["Stage 3\nBackfill\nExisting Data"]
    S4["Stage 4\nKV Primary\nWrite: KV → RDS\nRead: KV"]
    S5["Stage 5\nConsistency\nVerification"]
    S6["Stage 6\nKV Only"]
    S0 --> S1 --> S2 --> S3 --> S4 --> S5 --> S6
    R["Rollback to RDS"]
    S1 -. rollback .-> R
    S2 -. rollback .-> R
    S4 -. rollback .-> R
    S5 -. rollback .-> R

Key Decisions

• Favored migration safety over rollout simplicity — staged approach over big-bang cutover
• Maintained rollback capability at every phase to reduce risk
• Used dynamic configuration for traffic switching rather than code deployments

Challenges

• Ensuring correctness during zero-downtime migration
• Preventing data inconsistency between RDS and KV during dual-write
• Designing safe rollback under partial rollout failure

Result

Successfully migrated the metadata backend online with zero downtime, preserving rollback safety throughout the migration and reducing migration risk without service disruption.

What I Learned

Gained deep experience in online migration patterns — dual-write, staged rollout, consistency verification, and the importance of rollback-safe design in mission-critical systems.