TOS Metadata Control Plane

Problem

Object storage metadata was a critical dependency for every read/write request. The workload was extremely read-heavy (peaking at ~160k reads/s per region) and write-light (<1k writes/s). The system needed to serve metadata at scale across dozens of virtual regions (US, Southeast Asia, China, Europe) while tolerating backend failures gracefully.

My Role

System owner responsible for the metadata control plane — designing cache logic, credential security, authentication integration, and leading the backend migration from RDS to KV.

Architecture / Approach

Built a cache-first serving strategy with stale-tolerant refresh. Used async refresh combined with singleflight to protect the backend store under high read traffic. Implemented graceful degradation so the service could continue serving cached metadata when the underlying DB/KV experienced failures.

• Cache-first serving with stale-tolerant refresh and configurable TTL tiers
• Async refresh + singleflight to coalesce concurrent cache misses
• Graceful degradation: extend cache timestamps and return stale data on backend errors
• Sync load for cold misses, async refresh for warm-but-aging entries

Architecture Diagrams

flowchart LR
    Gateway[Storage Gateway API]
    subgraph Meta[Metadata Control Plane]
        Cache[Local Memory Cache]
        Logic[Metadata APIs / Business Logic]
        Auth[ZTI Authentication]
        Encrypt[Credential Encryption]
        Validate[Validation]
        Cache --> Logic
        Logic --> Auth
        Logic --> Encrypt
        Logic --> Validate
    end
    Gateway --> Cache
    Logic --> KV[(KV Store)]
    Logic -. legacy / migration .-> SQL[(SQL / RDS)]

flowchart TD
    A[Request Metadata] --> B{Cache Exists?}
    B -- No --> C[SingleFlight Sync Load from DB/KV]
    C --> D[Update Cache]
    D --> E[Return Metadata]
    B -- Yes --> F{Age < TTL/2?}
    F -- Yes --> E
    F -- No --> G{Age < TTL?}
    G -- Yes --> H[Return Cached Immediately]
    H --> I[Trigger Async Refresh]
    I --> J[SingleFlight Async Load]
    J --> D
    G -- No --> K[SingleFlight Sync Refresh]
    K --> D
    C -. DB/KV Error .-> M[Return Internal Error]
    J -. DB/KV Error .-> L[Extend Cache / Return Old]
    K -. DB/KV Error .-> L

Key Decisions

• Favored availability and resilience over strict metadata freshness
• Metadata writes were infrequent and not latency-sensitive, enabling aggressive caching
• Optimized backend protection under high read traffic rather than optimizing write paths

Result

Maintained and improved a business-critical metadata control plane handling ~160k reads/s per region across dozens of virtual regions globally, serving multiple large-scale product teams with high availability.

What I Learned

Deepened understanding of cache design trade-offs at scale — when to serve stale data, how to protect backends with singleflight and async refresh, and how to build degradation strategies that keep services available during partial outages.