I had a #patch accepted to an Ansible role I use in my Vagrant environment github.com/mprahl/ansible-role-lets-encrypt-route-53/pull/14
permalinkI promised a postmortem: hyperbo.la/w/secrets-in-parameter-store-postmortem
permalinkI was hard down for ~5min last night while rolling out secrets in parameter store. 0.149.0, 0.149.1, 0.149.2, and 0.149.3 were bad releases #fail. 0.149.4 is stable: github.com/hyperbola/hyperbola/compare/v0.148.0...v0.149.4. Postmortem pending.
permalinkAdd in some manual #terraform state edits and deleting things in the #aws console and we're recovered #fail #win
permalinkA red herring was adding a call to dd in the cleanup script to zero the free space on the disk. I thought I was somehow filling the disk and not reclaiming space.
permalinkRemoving just the dev package with #ansible #provisioning had no effect in local env, but combined with the apt purge in the #packer script, uninstalled mysqlclient
permalinkRoot cause: My app depends on the python package mysqlclient, which in turn depends on the libmysqlclient system package. I was pulling this in transitively via the dev package required for building.
permalinkUndeployable manifested as healthz returning 502 when adding a new instance to the ALB, marking it as unhealthy and timing out #terraform
permalinkI recently started cleaning up after myself during #provisioning. There are two steps to this: #ansible uninstalls build deps and #packer executes a cleanup script that does an apt purge and autoremove
permalinkGot into an undeployable state due to differences in #provisioning between local and prod environments #fail github.com/hyperbola/hyperbola/commit/a914992
permalink