Certificate and Secret Lifecycle at Scale

Overview

Expired certificates and secrets are among the most avoidable causes of outages. This page covers three pieces of lifecycle work: a shared TLS certificate renewal that also changed the certificate's filename, a host-key rollout to every environment, and a repeatable way to triage expiring app-registration secrets.

The Challenge

• The renewed certificate changed type, and with it the filename the pods mount.
• That filename appeared in Helm values and in app settings stored in Key Vault. Both had to change together, or pods would look for a file that wasn't there.
• Each environment had its own certificate, vault and change approval.

Architecture

Implementation

1. Scope tightly: change only the target environments' values, on a dedicated branch.
2. Update settings in bulk: one script updated the filename across dozens of app-settings secrets in a single, controlled pass.
3. Respect approvals: when the change was approved for one environment only, the other environment's values were held back until its own change request.
4. Host keys: created the expected SFTP host-key secret in every environment and verified the pods. That caught one environment that pointed at a different SFTP provider and needed that provider's published fingerprint.
5. Expiring secrets: triaged app-registration secret-expiry alerts by checking sign-in activity, pipeline usage and Graph, then recommending rotate, delete or suppress.

$secrets = Get-AzKeyVaultSecret -VaultName $vault | Where-Object Name -like '*-appsettings'
foreach ($s in $secrets) {
    $v = Get-AzKeyVaultSecret -VaultName $vault -Name $s.Name -AsPlainText
    Set-Content "backup/$($s.Name).json" $v                      # back up first
    $new = $v -replace 'old-cert-name\.pfx', 'new-cert-name.pfx'
    if ($new -ne $v) {
        Set-AzKeyVaultSecret -VaultName $vault -Name $s.Name `
            -SecretValue (ConvertTo-SecureString $new -AsPlainText -Force) | Out-Null
    }
}

Key Outcomes

• Dozens of service configurations updated consistently in one pass, ahead of expiry.
• Host-key secret in every environment, with an environment-specific mismatch caught before production.
• Change control held: nothing released outside its approval scope.

Lessons Learned

• Sequence settings and mounts together. If pods restart between the two changes, they look for a file that isn't mounted yet.
• Back up before bulk edits. Vault versioning helps, but an explicit backup makes rollback a single command.
• A single trusted fingerprint is a time bomb. When the provider rotates its key, the app breaks, so support two.

Conclusion

Rotating a certificate is easy. Rotating it across dozens of services, several approval scopes and every environment without an outage takes sequencing and discipline.

Technologies Used