Credentials Encryption

Problem

Sensitive credentials (AK/SK) were stored in plaintext in the metadata persistence layer. Security hardening was required to reduce the blast radius of potential database exposure.

My Role

Designed and implemented the credential encryption layer, including the rollout strategy for backward-compatible encryption adoption.

Architecture / Approach

Introduced a credential security layer that encrypts credentials before persistence and conditionally decrypts on authorized read paths. Rolled out with a feature switch: decryption logic was always enabled for backward compatibility, encryption was enabled for new writes, and historical plaintext records were backfilled via a migration tool.

• Encrypt credentials before persistence, conditional decryption on authorized read paths
• Feature switch for gradual encryption rollout
• Always-on decryption for backward compatibility with existing plaintext records
• Migration tool to backfill encryption for historical credentials

Architecture Diagrams

flowchart LR
    Caller[Admin / Gateway / Authorized Service]
    Caller --> Meta[Metadata Service]
    Meta --> Policy[Credential Security Layer]
    Policy --> Enc[Encrypt Before Persistence]
    Policy --> Dec[Conditional Decryption]
    Enc --> KV[(KV Store)]
    Dec --> Caller2[Authorized Caller]

Key Decisions

• Introduced encryption without requiring a coordinated service-wide cutover
• Maintained backward compatibility for existing plaintext records during rollout
• Used feature switches to control rollout pace and enable rollback

Challenges

• Introducing encryption without service downtime
• Maintaining backward compatibility for existing plaintext records
• Avoiding read/write disruption during rollout

Result

Reduced blast radius of database exposure, improved metadata security posture, and achieved zero-downtime rollout for credential protection.

What I Learned

Learned the value of backward-compatible security rollouts — applying encryption incrementally with always-on decryption avoids the fragility of big-bang security migrations.