RDS to KV Online Migration

Problem

The metadata control plane needed to migrate from RDS to KV for two reasons: aligning with the ToB storage product stack for consistency and lower operational complexity, and because looking up bucket metadata by bucket name was a natural fit for KV storage. However, the service was a critical dependency for object access — any downtime or data loss was unacceptable.

My Role

Led the system design and implementation, partnered with SRE on the production rollout, and monitored system health throughout the migration.

Architecture / Approach

Designed a 6-stage online migration: (1) introduce migration wrapper with RDS as sole backend, (2) enable dual-write from RDS to KV, (3) backfill historical data with a migration tool, (4) switch primary to KV with RDS as secondary, (5) run consistency verification and repair using updatedAt timestamps, (6) cut over to KV-only. Each stage supported rollback to RDS.

• Migration wrapper abstracting the storage layer for dynamic traffic switching
• Dual-write strategy during transition with configurable write ordering
• Backfill tool for copying existing records from RDS to KV
• Consistency verification with timestamp-based repair mechanism
• Rollback capability maintained at every migration phase

Architecture Diagrams

flowchart LR
    S0["Stage 0\nRDS Only"]
    S1["Stage 1\nMigration Wrapper\nRead/Write: RDS"]
    S2["Stage 2\nDual Write\nWrite: RDS → KV\nRead: RDS"]
    S3["Stage 3\nBackfill\nExisting Data"]
    S4["Stage 4\nKV Primary\nWrite: KV → RDS\nRead: KV"]
    S5["Stage 5\nConsistency\nVerification"]
    S6["Stage 6\nKV Only"]
    S0 --> S1 --> S2 --> S3 --> S4 --> S5 --> S6
    R["Rollback to RDS"]
    S1 -. rollback .-> R
    S2 -. rollback .-> R
    S4 -. rollback .-> R
    S5 -. rollback .-> R

Key Decisions

• Favored migration safety over rollout simplicity — staged approach over big-bang cutover
• Maintained rollback capability at every phase to reduce risk
• Used dynamic configuration for traffic switching rather than code deployments

Challenges

• Ensuring correctness during zero-downtime migration
• Preventing data inconsistency between RDS and KV during dual-write
• Designing safe rollback under partial rollout failure

Result

Successfully completed the online metadata backend migration on clusters across dozens of regions with zero downtime, preserving rollback safety throughout the migration and reducing migration risk without service disruption.

What I Learned

Gained deep experience in online migration patterns — dual-write, staged rollout, consistency verification, and the importance of rollback-safe design in mission-critical systems.