Drill 1: Couple of vault instances down

  1. log into each downed instance, unseal the vault

  2. Make sure we can still read the values from vault from command line/curl

  3. Check consul and make sure the vault instance back online

Drill 2: The whole vault cluster is down and unable to restart

  1. retrieve the last stable AMI, or rerun packer to generate one

  2. Use tf to generate the new vault ASG, but with the same consul tags as the prvevious one

  3. login into each instance in the new cluster, unseal the instance, and verify that the instance is healthy on consul

  4. use tf to destroy the old vault cluster’s ASG

Drill 3: Restore vault data stored on consul

  1. take a snapshot of current consul cluster

  2. create a new consul cluster with new tags

  3. restore the consul cluster from the snapshot, and deploy a new vault cluster on top of the new consul cluster

  4. unseal the vault with original keys and verify that read is successful