TOS Metadata Control Plane

Problem

Object storage metadata was a critical dependency for every read/write request. The workload was extremely read-heavy (peaking at ~160k reads/s per region) and write-light (<1k writes/s). The system needed to serve metadata at scale across dozens of virtual regions (US, Southeast Asia, China, Europe) while tolerating backend failures gracefully.

My Role

System owner responsible for the metadata control plane — designing cache logic, credential security, authentication integration, and leading the backend migration from RDS to KV.

Architecture / Approach

Built a cache-first serving strategy with multiple layers of backend protection. Cache warmup on startup prevented cold-start stampedes during large-scale restarts. TTL jitter randomized expiration times to prevent cache avalanche. Singleflight coalesced concurrent requests for the same key to prevent cache breakdown on hot-key expiry. Negative caching (caching 404 responses) prevented cache penetration from queries on non-existent buckets. Async refresh combined with singleflight kept hot entries fresh without blocking callers. Graceful degradation extended cache timestamps and returned stale data when the underlying DB/KV experienced failures.

• Cache-first serving with stale-tolerant refresh and configurable TTL tiers
• Cache warmup on startup to prevent cold-start stampede when instances restart at scale
• TTL jitter to randomize expiration times and prevent cache avalanche (mass simultaneous expiry)
• Singleflight to coalesce concurrent requests for the same key, preventing cache breakdown on hot-key expiry
• Negative caching (cache 404 responses) to prevent cache penetration from queries on non-existent buckets
• Async refresh for warm-but-aging entries, sync load for cold misses
• Graceful degradation: extend cache timestamps and return stale data on backend errors

Architecture Diagrams

flowchart LR
    Gateway[Storage Gateway API]
    subgraph Meta[Metadata Control Plane]
        Cache[Local Memory Cache]
        Logic[Metadata APIs / Business Logic]
        Auth[ZTI Authentication]
        Encrypt[Credential Encryption]
        Validate[Validation]
        Cache --> Logic
        Logic --> Auth
        Logic --> Encrypt
        Logic --> Validate
    end
    Gateway --> Cache
    Logic --> KV[(KV Store)]
    Logic -. legacy / migration .-> SQL[(SQL / RDS)]

flowchart TD
    A[Request Metadata] --> B{Cache Exists?}
    B -- No --> C[SingleFlight Sync Load from DB/KV]
    C --> D[Update Cache]
    D --> E[Return Metadata]
    B -- Yes --> F{Age < TTL/2?}
    F -- Yes --> E
    F -- No --> G{Age < TTL?}
    G -- Yes --> H[Return Cached Immediately]
    H --> I[Trigger Async Refresh]
    I --> J[SingleFlight Async Load]
    J --> D
    G -- No --> K[SingleFlight Sync Refresh]
    K --> D
    C -. DB/KV Error .-> M[Return Internal Error]
    J -. DB/KV Error .-> L[Extend Cache / Return Old]
    K -. DB/KV Error .-> L

Key Decisions

• Prioritized availability over consistency — when the DB was down, reads continued serving stale cached data while writes failed; this was acceptable because the workload was overwhelmingly read-heavy and metadata staleness was tolerable
• Metadata writes were infrequent (<1k/s) and not latency-sensitive, making write failures during DB outages an acceptable trade-off for uninterrupted read availability
• Optimized backend protection under high read traffic rather than optimizing write paths

Result

Maintained and improved a business-critical metadata control plane handling ~160k reads/s per region across dozens of virtual regions globally, serving multiple large-scale product teams with high availability.

What I Learned

Deepened understanding of cache design trade-offs at scale — defending against the classic cache failure modes (avalanche, breakdown, penetration), knowing when to serve stale data, and how to build degradation strategies that keep services available during partial outages.