Quota Automation & Traffic Governance

Problem

• Capacity & Operations
  • Excess allocation: evolving workloads left many buckets with more quota than needed; total allocated quota exceeded safe physical cluster capacity.
  • Manual overhead: frequent user quota requests created a growing operational burden for the TOS team.
• Asymmetric Risks
  • Scale-down: variable traffic and critical workloads made underestimated peaks dangerous — automated reductions could disrupt production traffic.
  • Scale-up: aggressive increases could worsen cluster oversubscription. Safe automation mattered more than maximum coverage.

My Role

• Ownership: led service evolution, design, safety policies, production rollout, and customer onboarding across versions.
• Implementation: built prediction validation, policy evaluation, guardrails, execution, notifications, and tidal throttling.
• Model boundary: another internal team supplied the prediction model, based on 21-day historical data.

Architecture / Approach

• Daily pipeline: consume 21-day traffic predictions → validate against recent traffic → evaluate asymmetric scaling policies → execute bounded quota changes.
• Progressive Rollout
  • V1: validate the architecture on internal test buckets, including my own.
  • V2: introduce production scale-down in batches, starting with lower-criticality workloads. For each batch: create a dedicated communication group with bucket owners, send group announcements explaining the rollout, provide detailed user-facing documentation, and offer a whitelist opt-out before enabling auto-adjustment.
  • V3: extend to controlled scale-up with stricter thresholds.

• Prediction validation: skip buckets with missing or stale predictions, or predictions below the previous 7-day traffic maximum; accept less reclamation to protect valid traffic.
• Gradual scale-down: reduce quota by at most 5% per day, keeping changes meaningful and observable while limiting impact and allowing monitoring and customer feedback.
• Tiered scale-up: auto-approve increases ≤20% for small-traffic buckets; larger increases or larger buckets require human review.
• Whitelist exemption: owners can opt workloads out of automatic adjustment.
• Tidal throttling: temporarily lower high-throughput bucket quotas during configurable, predictable peak windows to address acute pressure beyond long-term governance; alert on restoration failures.
• Notifications: send bucket owners a Lark message for every quota change, with before/after values.
• Observability: track bucket-level decisions, adjustments, and errors.

Architecture Diagrams

flowchart TD
    Pred[21-Day Traffic Predictions\nfrom upstream team]
    Filter1[Filter Whitelisted Buckets]
    Validate{Prediction ≥\n7-Day Max Traffic?}
    Skip1[Skip Bucket]
    Decision{Scale Down\nor Scale Up?}
    Down[Gradual Scale Down\nMax 1x/day, ≤5% reduction]
    Up{Bucket Size\nTier?}
    SmallCheck{Increase\n≤ 20%?}
    SmallAuto[Auto Approve]
    SmallReview[Human Review]
    LargeReview[Human Review]
    Apply[Apply Quota Change]
    Metrics[Record Metrics]
    Notify[Lark Notification\nold → new quota]
    Pred --> Filter1 --> Validate
    Validate -- no --> Skip1
    Validate -- yes --> Decision
    Decision -- scale down --> Down --> Apply
    Decision -- scale up --> Up
    Up -- small bucket --> SmallCheck
    Up -- large bucket --> LargeReview --> Apply
    SmallCheck -- yes --> SmallAuto --> Apply
    SmallCheck -- no --> SmallReview --> Apply
    Apply --> Metrics --> Notify

flowchart LR
    V1["V1\nInternal Validation\nOwn test buckets"]
    V2["V2\nProduction Scale-Down\nBatch onboarding\nCustomer communication"]
    V3["V3\nControlled Scale-Up\nStricter thresholds\nHuman review gates"]
    V1 --> V2 --> V3

flowchart TD
    subgraph Long["Long-Term: Quota Automation"]
        QA[Daily Scheduled Job]
        QA --> Align[Gradually align allocated\nquota with actual demand]
    end
    subgraph Short["Short-Term: Tidal Throttling"]
        Peak[Peak Hour Trigger]
        Peak --> Restrict[Temporarily restrict\nhigh-throughput buckets]
        Restrict --> Restore[Restore after peak]
    end
    Long --- Short
    Risk[Platform Capacity Risk]
    Long --> Risk
    Short --> Risk

Key Decisions

• Conservative inputs: treat predictions as signals, not ground truth; validate against observed traffic to avoid anomalous forecasts.
• Asymmetric protection: gradual, rate-limited scale-down protects customers; tiered scale-up review protects the platform.
• Bounded impact: combine daily limits, bucket-size tiers, manual review, whitelist exemptions, and progressive rollout.
• Selective automation: automate low-risk changes with understood impact; keep high-risk changes under human review.
• Operational readiness: make customer communication, documentation, exemptions, and onboarding part of production rollout.

Challenges

• Safety boundaries: deciding what to automate under uncertain demand and imperfect predictions was harder than implementation.
• Organizational readiness: contact owners, explain governance, prepare documentation, and handle exemptions before changing customer quotas.
• Coverage vs. safety: aggressive reclamation risks legitimate traffic; excessive caution leaves oversubscription unresolved.
• Failure handling: skip individual execution failures for daily re-evaluation rather than retry; alert for manual intervention if tidal quota restoration fails.

Result

• Scale: onboarded ~60k buckets across CN (~40k), SG (~10k), and US (~10k).
• Capacity: reduced unnecessary allocation and progressively mitigated storage-cluster oversubscription risk.
• Operations: replaced largely manual, reactive governance with continuous automation, conservative validation, gradual scale-down, bounded scale-up, and human review and override.
• Production safeguards: combined progressive rollout, bucket-level observability, customer-facing processes, and tidal throttling for layered peak-hour protection.

What I Learned

• Asymmetric safety: scale-down must protect users; scale-up must protect platform capacity.
• Conservative automation: uncertain predictions favor no action over unsafe action, with impact constrained at multiple levels.
• People matter: technical readiness can precede organizational and customer readiness.

Future Direction

flowchart TD
    subgraph Server["Governance Server · Policy Evaluation"]
        Pred[Fetch Prediction Data] --> Validate[Validate Data]
        Validate --> Eligible[Whitelist Filtering & Gradual-Rollout Checks]
        Eligible --> Compute[Calculate Target Quota]
        Compute --> Event[Create Bucket Adjustment Event]
    end
    Event -->|Publish durably| Kafka[(Kafka)]
    subgraph Workers["Worker Consumer Group · Independent Scaling"]
        W1[Worker 1]
        W2[Worker 2]
        WN[Worker N]
    end
    Kafka --> W1 & W2 & WN
    W1 & W2 & WN --> Ticket[Create Ticket · Preserve Approval Gates]
    Ticket --> Message[Send Lark Message]
    Ticket -. Failed operation .-> Retry[Bounded Retry / Dead-Letter Queue]
    Message -. Failed operation .-> Retry

• Proposed change: evolve the synchronous pipeline by retaining validation and quota decisions in the server, and moving ticket creation and notifications to Kafka workers.
• Benefits
  • Failure isolation: slow or failed ticketing and messaging requests would no longer block policy evaluation; Kafka would buffer pending work.
  • Independent scaling: scale workers with bounded concurrency to protect downstream services.
• Tradeoffs
  • Consistency and operations: completion becomes eventually consistent, with additional Kafka operational overhead.
  • Recovery: require idempotency and per-step progress to prevent duplicate tickets or messages, bounded retries, a dead-letter queue, and lag monitoring.
  • Decision safety: preserve per-bucket ordering, check for stale quota decisions before execution, and retain existing approval gates.