And proof that this setup is both good and bad – CI failed because I forgot to set the publish timestamp in the post YAML on the previous post. But CI ALSO failed because I forgot to run gazelle. #win #fail #bazel #hypstatic

I took a hacksaw to my homebrew installation. I'm not sure how this happened but the brewfile I've been carrying around at one point had the entire dependency graph in it rather than the explicitly installed deps ... which made everything on my new Mac an explicitly installed dep. I had hundreds of libs installed for no reason. Burned everything down and started from scratch. #fail #win github.com/lopopolo/dotfiles/blob/fb9af6bd19df2149dba5a63a8db…

yikes and pagintated navigation for archive pages had month and year mixed up in the link. #hypstatic #fail

ooof. While hacking on the lifestream part of the static site generator, uncovered that previous and next buttons for traversing lifestream permalinks had their polarity swapped. That bug has been there for over a year. #hypstatic #fail

Dependency management in Python makes me sad. #fail #python #hypstatic

I've lost my ability to develop locally since I don't want to install VirtualBox on my new laptop. #fail #hypstatic

I run an outdated version of Django. #django #fail #hypstatic

The terraform config for this project is on 0.12 and I have no desire to update it to 0.13. #fail #terraform #automation #hypstatic

#vscode and LLDB in VSCode were SO amazing to use. Within a couple of iterations, I was able to track down the use-after-free: github.com/artichoke/artichoke/pull/674 #fail #win #artichoke

After 50 PRs, I was able to put GH-442 to bed and remove the Rc wrapper from the Artichoke state github.com/artichoke/artichoke/pull/670 #win This refactor took 4 months #fail #rust #artichoke

It mostly works and is reasonably speedy, although while iterating, a template error in the lifesteam post rendering code caused ~1000 templates to fail compilation. I let the build peg the CPU for 20 minutes before giving up #fail

I made the mistake of exporting the database first AND not documenting how I transformed it to YAML, so I've been hesitant to post here #fail

hyperbola was down for 26 hours because the CA cert bundle on its backing RDS instance was not rotated before the 2015 bundle expired #fail #outage #aws

Nothing like a little build breakage to start the weekend github.com/artichoke/artichoke/pull/417 #artichoke #fail

Reported a critical bug in rustfmt #fail #github #patch github.com/rust-lang/rustfmt/issues/3770

Also added many more converter implementations with macros. There are 988 TryConvert implementations. #win #fail #artichoke

Led to reporting this #mruby bug: github.com/mruby/mruby/issues/4684 #github #fail although I was able to work around it #artichoke

The converters are central to #artichoke. Changing this core abstraction was painful: 84 files changed, 2000 lines added, 2800 lines removed. #fail #git

This was a nasty PR github.com/artichoke/artichoke/pull/242 #artichoke #fail

I'm trying to improve Regexp performance in #artichoke by using the regex crate instead of oniguruma in some cases. It turns out not to be faster in all cases #patch #rust #fail github.com/rust-lang/regex/issues/604

This was hard to track down: emscripten linker error with a particular function signature #rust #wasm #patch #github #fail github.com/emscripten-core/emscripten/issues/9135

I forked onig and hope to add support for the wasm-unknown-unknown build target. For #artichoke though, it's too soon. Once mruby-sys is pulled into #Rust I'll try again #ruby #mruby #fail #patch #Wasm

Neither mruby-sys nor onig can build with wasm-unknown-unknown or wasm-wasi, so I was stuck with wasm-unknown-emscripten. The linker on mruby-sys would dead code eliminate artichoke_backend::Artichoke #fail #Wasm #artichoke #mruby

With renewed focus of building my own #Ruby instead of extending #mruby, I put in some effort to make mruby an implementation detail of #artichoke. Lots of refactoring with sed. #fail

CactusRef is still an incredibly unsafe crate, but at least it aborts if it detects a use-after-free #fail #artichoke

That memory leak turned out to be quite the yak shave. setjmp/longjmp from C leaving #rust memory in inconsistent state #fail github.com/artichoke/ferrocarril/pull/168

#git bisect is why it's important that your repo be buildable at every commit. Currently fixing a massive memory leak in ferrocarril #fail

Despite all I talk about #aws #ec2, I cannot for the life of me type isntances correctly #fail

here's another Clippy bug #rust #fail github.com/rust-lang/rust-clippy/issues/4143

You can tell which parts of ferrocarril are new to me based on how many consecutive commits are dedicated to a feature. By this measure, the new (and hard) things were sys crates, #ci, and TryFrom traits #rust #win #fail

About 6 weeks ago I reached out to two literary agents to see if they'd help me publish a book. I got an explicit rejection last week! A little bit #fail, but also definitely progress #win 😊

I generated FFI bindings for mruby with bindgen! I wrote a #c extension library for wrapping mruby macros! #win I wrote (a lot of) unsafe #rust code. Clippy helps a lot to make sure I'm not doing anything stupid. I have caused quite a few #segfault #fail

I overflowed the compiler (which didn't crash it) and crashed the linter with an ICE #win #fail

I found an ICE (internal compiler error) in clippy in #rust nightly #fail github.com/rust-lang/rust/issues/60067

looks like I didn't finalize the deploy in January so I had a few extra AMIs kicking around that I was needlessly paying for #aws #cost #fail #automation

The true motivation for getting me to mess with hyperbola today was an instance retirement notification from AWS. I had to muck with poetry to cycle my ASG #fail #aws

Uninstalling python 3.7.1 from pyenv broke my local poetry venv for hyperbola which prevented me from managing my deps #fail #python

Yikes haven't done a deploy since January 6. I've definitely been ignoring all of the USNs about Linux kernel vulnerabilities. #fail Django 2.2 release is what got me to get to upgrading #django #win

Deck making for an internal tech talk called "How to Implement the Chromecast Protocol in Rust When You Don't Know Rust" #rust futures and impl trait 😭 #fail

The most expensive part of hyperbola's #aws infrastructure is the SSM PrivateLink endpoint in 3 AZs #fail #cost

Just accidentally truncated my .bash_history. Restored from backup but the latest was 55 days ago. #fail

I was hard down for ~5min last night while rolling out secrets in parameter store. 0.149.0, 0.149.1, 0.149.2, and 0.149.3 were bad releases #fail. 0.149.4 is stable: github.com/hyperbola/hyperbola/compare/v0.148.0...v0.149.4. Postmortem pending.

The computers did exactly what I told them to do 😕 #fail

Add in some manual #terraform state edits and deleting things in the #aws console and we're recovered #fail #win

The cleanup script didn't error because my set flags were in the shebang but #packer was invoking the script via bash instead of directly #fail So many yaks.

This change was introduced in 0.146.0 but did not manifest due to a bug in the cleanup script. I was not passing -y to apt autoremove, which caused the command to abort and end the script with an error. #fail

Got into an undeployable state due to differences in #provisioning between local and prod environments #fail github.com/hyperbola/hyperbola/commit/a914992

I accidentally skipped v0.139.0 today because prettier barfed during cutting the release and I forgot to reset my git tree. I guess I forgot to run prettier on my whole repo when I enabled it. #fail

Sometimes using the #AWS cost and usage reports is just not fun, mostly due to the myriad of columns being undocumented. #fail

Semantic versioning is a lie (looking at you #packer). My config stopped working because a key was deprecated between 1.1.x and 1.2.x. Somehow this prevented the config from validating. #fail

So it turns out I shouldn't have ignored that MySQL backtrace when printing the help text of my new django management command in dev. That's why it hung when building the AMI. #fail One line fix: github.com/hyperbola/hyperbola/commit/728f1d68

I have pinned dependencies everywhere _except_ my AMI build pipeline. Bitten by the packer 1.2.0 upgrade breaking the ansible-local provisioner. #fail

Found a lifestream bug during my Django 2.0 upgrade that was never exercised on the live site because I've never had more than 40 posts in a month #fail

Accidentally deleted my private subnet route tables in the process though, which broke S3 access for ~20 minutes. Could've been much worse #fail #outage

welp that didn't last long. CloudFlare only queries a subset of NS records to check for liveness and has determined that I no longer use CloudFlare. Working on purging them from #terraform and registrar now #fail

I accidentally created a CNAME (instead of an A record) for an IP today. Lots of confusing errors from nslookup, ssh, and host. Meanwhile dig appeared to resolve the record. #fail

I accomplished this migration with ~no downtime #win. I spun up the new infrastructure and then deployed new AMIs with updated service records. I did have ~2 minutes of 500s when I accidentally overwrote old mysql DNS record due to a bad copypasta #fail

#history throwback to the time that my wiki was spammed by a bot that turned all the pages into link spam for discount pharmaceuticals github.com/hyperbola/hyperbola/commit/1a3725b84bf82982945f68d… #fail

I skipped #django 1.11.4 and 1.11.5. Finally did an upgrade today to 1.11.6 #win. My dep upgrades for python, js, and ansible deps are too coarse grained. Pulled in ansible changes which blocked the deployment #fail

When sorting cards in a hand, I use multi-pass bubble sort #fail n is small i guess

#terraform is now a package manager. Great. #fail. For some reason plugin downloads hang if the download gets an IPV6 edge node in their CDN.

when reading from stdin, ansible-vault encrypt_string does not trim trailing newlines, forcing one to use echo -n. #ansible knows this is a sharp edge and documents it, but they should do the right thing and make it consistent with password files #fail

in the quest to thin out the top-level directory sprawl in hyperbola, I got rid of .secrets. required rotating ansible vault password and rekeying everything because my password had single quotes, double quotes, and backticks (of course) #fail

#fail do not try to install ansible in a virtualenv in packer. you need a compiler. you don't want a compiler

the hyperbola + hyperbola-tools binary star monorepos is a common failure state, so I am told #fail

One source of difficulty was getting django, s3, vpc endpoints, and security groups to work together to enable s3 access from my backends #fail

Discovered I had duplicate stanzas in my tfvars file. Removed the extraneous one. Broke EVERYTHING. #fail The worst was my admin IAM account losing access in the midst of a terraform apply. destroyed module.network, module.iam, and module.hyperbola-wiki

tried copying a 2TB Time Machine volume at the file system level using Finder drag and drop. #fail sadtimes. Was going on 6 hours and still resolving hard links. Currently attempting block-level copy with Disk Utility

I got a memory exhaustion pop up on my MBP earlier. I blame spotify, chrome, and the 2 VMs I carelessly had running. #fail

Media backups have been failing since I moved the site to the new server. I've been uploading empty, corrupted tarballs. #fail fix here: github.com/hyperbola/hyperbola/commit/859128 Thank you gitlab & national check your backups day

Hit the 5 duplicate certs per week rate limit from let's encrypt by repeatedly rebuilding my vagrant box #fail

Deployed hyperbola-wiki. Initial approach was to provision on boot with a user data script. This caused instances to take 15 minutes to come up (apt-get upgrade, compiling ruby) #fail

mucking around in settings.py I ended up doing a top-level import from debug_toolbar. Yay for staging. #fail github.com/hyperbola/hyperbola/commit/26c4e1d

I didn't notice this in dev because the front page of the lifestream is 20 of the same picture of Kendrick Lamar #fail

Minor crisis when running the mangle media script. Resume and lifestream models have auto_now=True (an updated timestamp) and I have been using them as creation timestamps. Apparently there is an auto_now_add param I should use. #fail

I need to make these out of band releases more painful so I migrate hyperbola to my new host #fail

nginx upgrade broke the init script. had to reboot to fix things. gunicorn failed to start after reboot due to log file perms #fail

hahahahaha #fail Daemons in macruby #ruby

"we successfully avoided a split brain scenario due to master/master replication" #mysql #fail

My ruby version on my mac stopped working because homebrew updated libicu. Had to recompile/upgrade #fail

Yelp's search stemming needs some work. I searched for "used" and it is highlighting "us" in the search results. #fail

When I see L7 I think of the band instead of the OSI Model #fail

Media directory backups have been failing since the 17th due to exceeding max attachment size. No alerts. #fail >>> SIZE=55759470 <<< 552 5.3.4 Message size exceeds fixed limit

It only took a debian VM and 4 hours of my time to reformat two new USB keys. #fail

Fixed a long outstanding bug in how the sidebar on the lifestream calculated monthly post counts #win Tickled by the upgrade to #django 1.8 which caused them to all display as 1. #fail

Wow! usability fail. The crontab command operates in 3 modes, two of which are edit (-e) and remove (-r). Lovely that e and r are right next to each other on a QWERTY keyboard. #fail

This oncall cycle, I observed my first real hardware failure. A machine critical to the deployment pipeline spontaneously shutdown and failed to reboot, problems with mounting RAID, etc. #fail

I want to live in a world where booting a machine doesn't cause config propagation to break #fail ghetto consensus algorithms

Just spent 5 minutes hopping through man pages trying to grok a script I wrote a while ago only to realize there were comments explaining exactly what was going on #fail #docs

... and I'm no longer dumping secrets to stdout due to `source` and `set -x` #12factor #fail

sudo lsof | grep libssl | cut -d" " -f1 | sort | uniq -c | sort -rn #fail mta.openssl.org/pipermail/openssl-announce/2015-March/000020.…

This is how you get a macbook to page like crazy #fail

OMG #ruby is using 19.22GB of memory executing a one liner #fail

Ran time ruby -e 'def is_prime n; ("1" * n) !~ /^1?$|^(11+?)\1+$/; end; is_prime 982_451_653' #fail keyboard events don't fire, gfx glitches, windows don't repaint, one core pegged

Rebooted hyperbola for the first time in a year due to the libc vuln. Minor panic ensued. Gunicorn failed to start due to log file perms, /var/run/gunicorn was wiped and had to be recreated #fail

I have an album with a non-square album art in #iTunes. The UI renders it 3 different ways #fail

Now I have #utf8 support in #MySQL. I thought I did hyperbo.la/lifestream/174 but that was a #fail. Bitten by the difference between utf8 and utf8mb4. Alas, #Django 500's; no emoji for me

I still don't have zero-downtime #MySQL upgrades. Lost a nine #fail

Just added CORS headers for web-fonts on my 2 assets domains through cloudflare ... whoa. This has apparently been broken for a while #fail #win #hyperbola

I do not understand Box Engineering's blanket aversion to a build step. So many generated artifacts checked in. #fail

Today I exceeded linux's max shebang length #fail it is 128 btw

heh, spoke too soon ... #fail Had forgotten to update #nginx config so when I deleted the legacy deployments, I lost static assets and media.

just did a dry run of a mostly automated hyperbola deployment. virtualenv, bash, git what fun :) Already staging and production have diverged. #fail next is to figure out how to bring it live

Oh no! just missed a deploy step #fail ... forgot to run pip install --upgrade ... new code was missing a dep

Today I learned more about sockets and ephemeral port exhaustion than I ever wanted to know. I normally take TCP as a given. #webscale #box #metadata #fail

it is a rare day that I don't get home and try to type my work password into my laptop #fail

I have no idea why, but every time my backup disk spins up on my laptop, my entire system hangs for a few seconds #fail

got another big #performance win by not minifying HTML in python not doing so shaved ~100ms off response time which means my server was spending more than 100ms of CPU time for the pleasure #fail

about ~20% of server time for lifestream is spent reversing URLs for hashtags :/ #performance #fail

howfuckedismydatabase.com hehe database problems. #fail #scale

wow link tags and auto RSS/Atom feed discovery ... welcome to the internet Ryan ... #fail ... I feel like #django is missing a template tag here.

so my resume link was broken for six months #fail stupid missing line in #nginx conf ... not enough exploratory testing during the refactor. What good is #staging for anyway?!

I reported this #github #outage #fail

Only minor drama when upgrading hyperbola to 12.04 LTS ... #mysql failed to cleanly upgrade to 5.5 and I forgot the correct URL for my staging environment #linux #fail

currently having a real bad time importing ffmpeg into xcode #fail #indexing

just installed 500MB of software for a scanner #wtf #fail

I remember a few years ago when my primary reason for staying on #firefox vs #Chrome was that I preferred how Firefox highlighted text on selection. #fail

I had a cronjob launching every minute that updated remotes on about 40 #git repos. I've apparently had ~50 git processes running all the time for about a year now. #fail

Just updated the firmware on my Air. First two reboots = #kernelpanic. 3rd reboot = no mouse or keyboard. 4th reboot, everything works? #fail

#versioncontrol, or the lack thereof. This was my first major software project. I was a junior in high school. I was making a symbolic math thing. #fail or the reasons any CS class should teach #git

#notetoself: Next time you need to run more than 3 tasks that take 10 minutes or more to run, invest in figuring out how to script it #fail #thesis2012

#lessonlearned: Don't create a billion Timer objects. One is probably enough. Managed to get #eclipse to fail at repainting itself because it was running about 700 threads #java #fail

I do not care whether your API is RESTful. Does it work and will it do what I want? #marketing #fail

all your heapspace are belong to me! #java #outofmemory #fail

github.com/lopopolo/dotfiles/blob/master/scripts/cron-update-… All of my problems with this script had to do with adding passphrases to my #ssh keys #fail

just discovered that once you're in cmd+TAB mode on OS X, cmd+` cycles backward. So much easier than cmd+shift+TAB #learnability? #fail #win

I don't ♥ manually managing character sets #fail Why doesn't #django ensure the #mysql db has #utf8 as its character set?

Cute ad on stackoverflow. It reads " Your Job" #adtargeted (apparently my db doesn't like unicode #fail)

That feeling when you go to empty your trash and you realize that there were over 14k files in it that account for 15GB of disk space. HOW DID THIS HAPPEN? #fail

So now that I got #airdrop working, I like it. It doesn't handle large transfers so well though. Transferring a gig of #music files has taken 15 minutes so far and I'm only 2/3 done. #fail and #win

Nokogiri::XML(file_bigger_than_1GB.xml) is a bad idea #fail

I dislike experiencing bugs in my day-to-day usage of software I paid for. Saints Row: The Third and iPhoto, I'm looking at you #fail

I don't understand why people think they need build scripts for their dotfiles. #fail Just keep your $HOME in #git: git init && git remote add ...

sweet, there's been a patch since may, but no need to put out a bugfix release or anything github.com/github/gollum/issues/147 #fail #git

this is a major #fail deadmemes.net/2010/10/19/fear-and-loathing-in-debianubuntu-or…

#hack to fix someone else's #PHP #fail foreach ($_POST as $key => $value) { $_POST[$key] = mysql_real_escape_string($value); }

apache, quit installing sites into my sites-enabled directory. You're taking my site offline #fail

I have iTunes autoname my files based of their tags. Apparently if a track starts with "(", it won't prepend the track number to the file name. #fail

* 4 * * * means every minute of the 4 o'clock hour #cron #fail

Google plus, you are no longer app tabified. slow #fail

Just did an ubuntu server upgrade from hardy #linux #fail

Why dynamic languages suck sometimes: I spent the last 45 minutes debugging a python script. I thought I had logic errors. What actually was wrong was that I renamed a variable and missed one. #fail

#backintheday, before I understood how the relational part of MySQL worked, I made 20 columns to hold metadata about a record. 20 columns wasn't enough. Foreign keys #ftw. learning #fail

So on this one server, my default shell was /bin/sh, i'm not a sudoer, root owns my home directory, and another user owns /home. #linux #fail

so in order to access Mail.app's preferences to make it not be the default mailto: handler, I first have to configure it #fail

the goes to operator (x --> 0) makes #C feel more like #ruby! #fail

Seriously, Office? You rm the backup from the disk when I close it instead of doing the sane thing and moving it to my trash? #fail #hard

#compiling #gcc from source to get an ELF cross-compiler on OSX. This is so #fail

Because kernel.org is down, I used airdrop to copy over a cached version of the git source and formula from another computer so I could install it with homebrew #fail

for the number of times i ssh'd into my iMac this sumer, you would think that I'd have just put my ssh key on it #fail #doingitthehardway

I was not able to access google plus for about a week. Figured they were just having issues because I was still able to access email fine. Ended up having to recover my account. #fail

So #rails caches column names across the database and generates SQL for them even if they are never used. Why does this make sense? #fail

My new favicon welcomes my site to 1996 #itsgreattobehere #fail

checked my Activity Monitor, had 2 #ruby processes each at 90% cpu which had been running for hours. trapping signals #fail

I need to get around to resetting up this server. I want to switch from svn to git and redo the whole apache config. why did I put the site in /var where I have to sudo all the time?? #linux #fail

rebased some already pushed commits to clean up my history and forced the push. I feel so dirty. #git #fail

lesson learned: never muck around with mysql's database directory; you will screw up #fail #linux

this site: now with 5 times fewer 500 errors #django #fail

for some reason every part of the #django admin worked except the root page. had to turn on debug mode to see I was improperly configured #fail

ruby's splat is just python's tuple (un)packing #win #fail

unintentional side effect of my server's firewall: I can't traceroute it. #fail

lol. edited the site live again while working on a new app. 500 errors abound. #django #fail

lesson learned: never edit a site live. #fail #django

Found out that my raid array not mounting a couple months ago was due to a failed drive. Can't recover it. I lost so much data. #linux #fail

I HATE FLEX SO MUCH #fail

Had a bad run-in with rm -rf. lost the sqlite database for the site. on the bright side, I'm using mysql now. #linux #fail