hyperbo.la :: lifestream
permalink

And proof that this setup is both good and bad – CI failed because I forgot to set the publish timestamp in the post YAML on the previous post. But CI ALSO failed because I forgot to run gazelle. #win #fail #bazel #hypstatic

permalink

I took a hacksaw to my homebrew installation. I'm not sure how this happened but the brewfile I've been carrying around at one point had the entire dependency graph in it rather than the explicitly installed deps ... which made everything on my new Mac an explicitly installed dep. I had hundreds of libs installed for no reason. Burned everything down and started from scratch. #fail #win https://github.com/lopopolo/dotfiles/blob/fb9af6bd19df2149dba5a63a8db161702bcbe5db/homebrew-packages/Brewfile.rooster

permalink

yikes and pagintated navigation for archive pages had month and year mixed up in the link. #hypstatic #fail

permalink

ooof. While hacking on the lifestream part of the static site generator, uncovered that previous and next buttons for traversing lifestream permalinks had their polarity swapped. That bug has been there for over a year. #hypstatic #fail

permalink

Dependency management in Python makes me sad. #fail #python #hypstatic

permalink

I've lost my ability to develop locally since I don't want to install VirtualBox on my new laptop. #fail #hypstatic

permalink

I run an outdated version of Django. #django #fail #hypstatic

permalink

The terraform config for this project is on 0.12 and I have no desire to update it to 0.13. #fail #terraform #automation #hypstatic

permalink

#vscode and LLDB in VSCode were SO amazing to use. Within a couple of iterations, I was able to track down the use-after-free: https://github.com/artichoke/artichoke/pull/674 #fail #win #artichoke

permalink

After 50 PRs, I was able to put GH-442 to bed and remove the Rc wrapper from the Artichoke state https://github.com/artichoke/artichoke/pull/670 #win This refactor took 4 months #fail #rust #artichoke

permalink

It mostly works and is reasonably speedy, although while iterating, a template error in the lifesteam post rendering code caused ~1000 templates to fail compilation. I let the build peg the CPU for 20 minutes before giving up #fail

permalink

I made the mistake of exporting the database first AND not documenting how I transformed it to YAML, so I've been hesitant to post here #fail

permalink

hyperbola was down for 26 hours because the CA cert bundle on its backing RDS instance was not rotated before the 2015 bundle expired #fail #outage #aws

permalink

Nothing like a little build breakage to start the weekend https://github.com/artichoke/artichoke/pull/417 #artichoke #fail

permalink

Reported a critical bug in rustfmt #fail #github #patch https://github.com/rust-lang/rustfmt/issues/3770

permalink

Also added many more converter implementations with macros. There are 988 TryConvert implementations. #win #fail #artichoke

permalink

Led to reporting this #mruby bug: https://github.com/mruby/mruby/issues/4684 #github #fail although I was able to work around it #artichoke

Photo for post 740.
permalink

The converters are central to #artichoke. Changing this core abstraction was painful: 84 files changed, 2000 lines added, 2800 lines removed. #fail #git

permalink

This was a nasty PR https://github.com/artichoke/artichoke/pull/242 #artichoke #fail

permalink

I'm trying to improve Regexp performance in #artichoke by using the regex crate instead of oniguruma in some cases. It turns out not to be faster in all cases #patch #rust #fail https://github.com/rust-lang/regex/issues/604

permalink

This was hard to track down: emscripten linker error with a particular function signature #rust #wasm #patch #github #fail https://github.com/emscripten-core/emscripten/issues/9135

permalink

I forked onig and hope to add support for the wasm-unknown-unknown build target. For #artichoke though, it's too soon. Once mruby-sys is pulled into #Rust I'll try again #ruby #mruby #fail #patch #Wasm

permalink

Neither mruby-sys nor onig can build with wasm-unknown-unknown or wasm-wasi, so I was stuck with wasm-unknown-emscripten. The linker on mruby-sys would dead code eliminate artichoke_backend::Artichoke #fail #Wasm #artichoke #mruby

permalink

With renewed focus of building my own #Ruby instead of extending #mruby, I put in some effort to make mruby an implementation detail of #artichoke. Lots of refactoring with sed. #fail

permalink

CactusRef is still an incredibly unsafe crate, but at least it aborts if it detects a use-after-free #fail #artichoke

permalink

That memory leak turned out to be quite the yak shave. setjmp/longjmp from C leaving #rust memory in inconsistent state #fail https://github.com/artichoke/ferrocarril/pull/168

Photo for post 705.
permalink

#git bisect is why it's important that your repo be buildable at every commit. Currently fixing a massive memory leak in ferrocarril #fail

permalink

Despite all I talk about #aws #ec2, I cannot for the life of me type isntances correctly #fail

permalink

here's another Clippy bug #rust #fail https://github.com/rust-lang/rust-clippy/issues/4143

permalink

You can tell which parts of ferrocarril are new to me based on how many consecutive commits are dedicated to a feature. By this measure, the new (and hard) things were sys crates, #ci, and TryFrom traits #rust #win #fail

permalink

About 6 weeks ago I reached out to two literary agents to see if they'd help me publish a book. I got an explicit rejection last week! A little bit #fail, but also definitely progress #win 😊

permalink

I generated FFI bindings for mruby with bindgen! I wrote a #c extension library for wrapping mruby macros! #win I wrote (a lot of) unsafe #rust code. Clippy helps a lot to make sure I'm not doing anything stupid. I have caused quite a few #segfault #fail

permalink

I overflowed the compiler (which didn't crash it) and crashed the linter with an ICE #win #fail

permalink

I found an ICE (internal compiler error) in clippy in #rust nightly #fail https://github.com/rust-lang/rust/issues/60067

permalink

looks like I didn't finalize the deploy in January so I had a few extra AMIs kicking around that I was needlessly paying for #aws #cost #fail #automation

permalink

The true motivation for getting me to mess with hyperbola today was an instance retirement notification from AWS. I had to muck with poetry to cycle my ASG #fail #aws

permalink

Uninstalling python 3.7.1 from pyenv broke my local poetry venv for hyperbola which prevented me from managing my deps #fail #python

permalink

Yikes haven't done a deploy since January 6. I've definitely been ignoring all of the USNs about Linux kernel vulnerabilities. #fail Django 2.2 release is what got me to get to upgrading #django #win

Photo for post 668.
permalink

Deck making for an internal tech talk called "How to Implement the Chromecast Protocol in Rust When You Don't Know Rust" #rust futures and impl trait 😭 #fail

permalink

The most expensive part of hyperbola's #aws infrastructure is the SSM PrivateLink endpoint in 3 AZs #fail #cost

permalink

Just accidentally truncated my .bash_history. Restored from backup but the latest was 55 days ago. #fail

permalink

I was hard down for ~5min last night while rolling out secrets in parameter store. 0.149.0, 0.149.1, 0.149.2, and 0.149.3 were bad releases #fail. 0.149.4 is stable: https://github.com/hyperbola/hyperbola/compare/v0.148.0...v0.149.4. Postmortem pending.

permalink

The computers did exactly what I told them to do 😕 #fail

permalink

Add in some manual #terraform state edits and deleting things in the #aws console and we're recovered #fail #win

permalink

The cleanup script didn't error because my set flags were in the shebang but #packer was invoking the script via bash instead of directly #fail So many yaks.

permalink

This change was introduced in 0.146.0 but did not manifest due to a bug in the cleanup script. I was not passing -y to apt autoremove, which caused the command to abort and end the script with an error. #fail

permalink

Got into an undeployable state due to differences in #provisioning between local and prod environments #fail https://github.com/hyperbola/hyperbola/commit/a914992

permalink

I accidentally skipped v0.139.0 today because prettier barfed during cutting the release and I forgot to reset my git tree. I guess I forgot to run prettier on my whole repo when I enabled it. #fail

permalink

Sometimes using the #AWS cost and usage reports is just not fun, mostly due to the myriad of columns being undocumented. #fail

permalink

Semantic versioning is a lie (looking at you #packer). My config stopped working because a key was deprecated between 1.1.x and 1.2.x. Somehow this prevented the config from validating. #fail

permalink

So it turns out I shouldn't have ignored that MySQL backtrace when printing the help text of my new django management command in dev. That's why it hung when building the AMI. #fail One line fix: https://github.com/hyperbola/hyperbola/commit/728f1d68

permalink

I have pinned dependencies everywhere _except_ my AMI build pipeline. Bitten by the packer 1.2.0 upgrade breaking the ansible-local provisioner. #fail

permalink

Found a lifestream bug during my Django 2.0 upgrade that was never exercised on the live site because I've never had more than 40 posts in a month #fail

permalink

Accidentally deleted my private subnet route tables in the process though, which broke S3 access for ~20 minutes. Could've been much worse #fail #outage

permalink

welp that didn't last long. CloudFlare only queries a subset of NS records to check for liveness and has determined that I no longer use CloudFlare. Working on purging them from #terraform and registrar now #fail

permalink

I accidentally created a CNAME (instead of an A record) for an IP today. Lots of confusing errors from nslookup, ssh, and host. Meanwhile dig appeared to resolve the record. #fail

permalink

I accomplished this migration with ~no downtime #win. I spun up the new infrastructure and then deployed new AMIs with updated service records. I did have ~2 minutes of 500s when I accidentally overwrote old mysql DNS record due to a bad copypasta #fail

permalink

#history throwback to the time that my wiki was spammed by a bot that turned all the pages into link spam for discount pharmaceuticals https://github.com/hyperbola/hyperbola/commit/1a3725b84bf82982945f68d14ff09ae7aa00d0ba #fail

permalink

I skipped #django 1.11.4 and 1.11.5. Finally did an upgrade today to 1.11.6 #win. My dep upgrades for python, js, and ansible deps are too coarse grained. Pulled in ansible changes which blocked the deployment #fail

permalink

When sorting cards in a hand, I use multi-pass bubble sort #fail n is small i guess

permalink

#terraform is now a package manager. Great. #fail. For some reason plugin downloads hang if the download gets an IPV6 edge node in their CDN.

permalink

when reading from stdin, ansible-vault encrypt_string does not trim trailing newlines, forcing one to use echo -n. #ansible knows this is a sharp edge and documents it, but they should do the right thing and make it consistent with password files #fail

permalink

in the quest to thin out the top-level directory sprawl in hyperbola, I got rid of .secrets. required rotating ansible vault password and rekeying everything because my password had single quotes, double quotes, and backticks (of course) #fail

permalink

#fail do not try to install ansible in a virtualenv in packer. you need a compiler. you don't want a compiler

permalink

the hyperbola + hyperbola-tools binary star monorepos is a common failure state, so I am told #fail

permalink

One source of difficulty was getting django, s3, vpc endpoints, and security groups to work together to enable s3 access from my backends #fail

permalink

Discovered I had duplicate stanzas in my tfvars file. Removed the extraneous one. Broke EVERYTHING. #fail The worst was my admin IAM account losing access in the midst of a terraform apply. destroyed module.network, module.iam, and module.hyperbola-wiki

permalink

tried copying a 2TB Time Machine volume at the file system level using Finder drag and drop. #fail sadtimes. Was going on 6 hours and still resolving hard links. Currently attempting block-level copy with Disk Utility

permalink

I got a memory exhaustion pop up on my MBP earlier. I blame spotify, chrome, and the 2 VMs I carelessly had running. #fail

permalink

Media backups have been failing since I moved the site to the new server. I've been uploading empty, corrupted tarballs. #fail fix here: https://github.com/hyperbola/hyperbola/commit/859128 Thank you gitlab & national check your backups day

permalink

Hit the 5 duplicate certs per week rate limit from let's encrypt by repeatedly rebuilding my vagrant box #fail

permalink

Deployed hyperbola-wiki. Initial approach was to provision on boot with a user data script. This caused instances to take 15 minutes to come up (apt-get upgrade, compiling ruby) #fail

permalink

mucking around in settings.py I ended up doing a top-level import from debug_toolbar. Yay for staging. #fail https://github.com/hyperbola/hyperbola/commit/26c4e1d

permalink

I didn't notice this in dev because the front page of the lifestream is 20 of the same picture of Kendrick Lamar #fail

permalink

Minor crisis when running the mangle media script. Resume and lifestream models have auto_now=True (an updated timestamp) and I have been using them as creation timestamps. Apparently there is an auto_now_add param I should use. #fail

permalink

I need to make these out of band releases more painful so I migrate hyperbola to my new host #fail

permalink

nginx upgrade broke the init script. had to reboot to fix things. gunicorn failed to start after reboot due to log file perms #fail

permalink

hahahahaha #fail Daemons in macruby #ruby

permalink

"we successfully avoided a split brain scenario due to master/master replication" #mysql #fail

permalink

My ruby version on my mac stopped working because homebrew updated libicu. Had to recompile/upgrade #fail

permalink

Yelp's search stemming needs some work. I searched for "used" and it is highlighting "us" in the search results. #fail

permalink

When I see L7 I think of the band instead of the OSI Model #fail

permalink

Media directory backups have been failing since the 17th due to exceeding max attachment size. No alerts. #fail >>> SIZE=55759470 <<< 552 5.3.4 Message size exceeds fixed limit

permalink

It only took a debian VM and 4 hours of my time to reformat two new USB keys. #fail

permalink

Fixed a long outstanding bug in how the sidebar on the lifestream calculated monthly post counts #win Tickled by the upgrade to #django 1.8 which caused them to all display as 1. #fail

permalink

Wow! usability fail. The crontab command operates in 3 modes, two of which are edit (-e) and remove (-r). Lovely that e and r are right next to each other on a QWERTY keyboard. #fail

permalink

This oncall cycle, I observed my first real hardware failure. A machine critical to the deployment pipeline spontaneously shutdown and failed to reboot, problems with mounting RAID, etc. #fail

permalink

I want to live in a world where booting a machine doesn't cause config propagation to break #fail ghetto consensus algorithms

permalink

Just spent 5 minutes hopping through man pages trying to grok a script I wrote a while ago only to realize there were comments explaining exactly what was going on #fail #docs

permalink

... and I'm no longer dumping secrets to stdout due to `source` and `set -x` #12factor #fail

permalink

sudo lsof | grep libssl | cut -d" " -f1 | sort | uniq -c | sort -rn #fail https://mta.openssl.org/pipermail/openssl-announce/2015-March/000020.html

Photo for post 330.
permalink

This is how you get a macbook to page like crazy #fail

permalink

OMG #ruby is using 19.22GB of memory executing a one liner #fail

permalink

Ran time ruby -e 'def is_prime n; ("1" * n) !~ /^1?$|^(11+?)\1+$/; end; is_prime 982_451_653' #fail keyboard events don't fire, gfx glitches, windows don't repaint, one core pegged

permalink

Rebooted hyperbola for the first time in a year due to the libc vuln. Minor panic ensued. Gunicorn failed to start due to log file perms, /var/run/gunicorn was wiped and had to be recreated #fail

Photo for post 322.
permalink

I have an album with a non-square album art in #iTunes. The UI renders it 3 different ways #fail

permalink

Now I have #utf8 support in #MySQL. I thought I did https://hyperbo.la/lifestream/174/ but that was a #fail. Bitten by the difference between utf8 and utf8mb4. Alas, #Django 500's; no emoji for me

permalink

I still don't have zero-downtime #MySQL upgrades. Lost a nine #fail

permalink

Just added CORS headers for web-fonts on my 2 assets domains through cloudflare ... whoa. This has apparently been broken for a while #fail #win #hyperbola

permalink

I do not understand Box Engineering's blanket aversion to a build step. So many generated artifacts checked in. #fail

permalink

Today I exceeded linux's max shebang length #fail it is 128 btw

permalink

heh, spoke too soon ... #fail Had forgotten to update #nginx config so when I deleted the legacy deployments, I lost static assets and media.

permalink

just did a dry run of a mostly automated hyperbola deployment. virtualenv, bash, git what fun :) Already staging and production have diverged. #fail next is to figure out how to bring it live

permalink

Oh no! just missed a deploy step #fail ... forgot to run pip install --upgrade ... new code was missing a dep

permalink

Today I learned more about sockets and ephemeral port exhaustion than I ever wanted to know. I normally take TCP as a given. #webscale #box #metadata #fail

permalink

it is a rare day that I don't get home and try to type my work password into my laptop #fail

permalink

I have no idea why, but every time my backup disk spins up on my laptop, my entire system hangs for a few seconds #fail

permalink

got another big #performance win by not minifying HTML in python not doing so shaved ~100ms off response time which means my server was spending more than 100ms of CPU time for the pleasure #fail

permalink

about ~20% of server time for lifestream is spent reversing URLs for hashtags :/ #performance #fail

permalink

https://howfuckedismydatabase.com/ hehe database problems. #fail #scale

permalink

wow link tags and auto RSS/Atom feed discovery ... welcome to the internet Ryan ... #fail ... I feel like #django is missing a template tag here.

permalink

so my resume link was broken for six months #fail stupid missing line in #nginx conf ... not enough exploratory testing during the refactor. What good is #staging for anyway?!

Photo for post 246.
permalink

I reported this #github #outage #fail

permalink

Only minor drama when upgrading hyperbola to 12.04 LTS ... #mysql failed to cleanly upgrade to 5.5 and I forgot the correct URL for my staging environment #linux #fail

permalink

currently having a real bad time importing ffmpeg into xcode #fail #indexing

permalink

just installed 500MB of software for a scanner #wtf #fail

permalink

I remember a few years ago when my primary reason for staying on #firefox vs #Chrome was that I preferred how Firefox highlighted text on selection. #fail

permalink

I had a cronjob launching every minute that updated remotes on about 40 #git repos. I've apparently had ~50 git processes running all the time for about a year now. #fail

permalink

Just updated the firmware on my Air. First two reboots = #kernelpanic. 3rd reboot = no mouse or keyboard. 4th reboot, everything works? #fail

Photo for post 199.
permalink

#versioncontrol, or the lack thereof. This was my first major software project. I was a junior in high school. I was making a symbolic math thing. #fail or the reasons any CS class should teach #git

permalink

#notetoself: Next time you need to run more than 3 tasks that take 10 minutes or more to run, invest in figuring out how to script it #fail #thesis2012

permalink

#lessonlearned: Don't create a billion Timer objects. One is probably enough. Managed to get #eclipse to fail at repainting itself because it was running about 700 threads #java #fail

permalink

I do not care whether your API is RESTful. Does it work and will it do what I want? #marketing #fail

permalink

all your heapspace are belong to me! #java #outofmemory #fail

permalink

https://github.com/lopopolo/dotfiles/blob/master/scripts/cron-update-remotes.bash All of my problems with this script had to do with adding passphrases to my #ssh keys #fail

permalink

just discovered that once you're in cmd+TAB mode on OS X, cmd+` cycles backward. So much easier than cmd+shift+TAB #learnability? #fail #win

permalink

I don't ♥ manually managing character sets #fail Why doesn't #django ensure the #mysql db has #utf8 as its character set?

Photo for post 173.
permalink

Cute ad on stackoverflow. It reads "<heart> Your Job" #adtargeted (apparently my db doesn't like unicode #fail)

permalink

That feeling when you go to empty your trash and you realize that there were over 14k files in it that account for 15GB of disk space. HOW DID THIS HAPPEN? #fail

permalink

So now that I got #airdrop working, I like it. It doesn't handle large transfers so well though. Transferring a gig of #music files has taken 15 minutes so far and I'm only 2/3 done. #fail and #win

permalink

Nokogiri::XML(file_bigger_than_1GB.xml) is a bad idea #fail

permalink

I dislike experiencing bugs in my day-to-day usage of software I paid for. Saints Row: The Third and iPhoto, I'm looking at you #fail

permalink

I don't understand why people think they need build scripts for their dotfiles. #fail Just keep your $HOME in #git: git init && git remote add ...

permalink

sweet, there's been a patch since may, but no need to put out a bugfix release or anything https://github.com/github/gollum/issues/147 #fail #git

permalink

#hack to fix someone else's #PHP #fail foreach ($_POST as $key => $value) { $_POST[$key] = mysql_real_escape_string($value); }

permalink

apache, quit installing sites into my sites-enabled directory. You're taking my site offline #fail

permalink

I have iTunes autoname my files based of their tags. Apparently if a track starts with "(", it won't prepend the track number to the file name. #fail

permalink

* 4 * * * means every minute of the 4 o'clock hour #cron #fail

permalink

Google plus, you are no longer app tabified. slow #fail

permalink

Just did an ubuntu server upgrade from hardy #linux #fail

permalink

Why dynamic languages suck sometimes: I spent the last 45 minutes debugging a python script. I thought I had logic errors. What actually was wrong was that I renamed a variable and missed one. #fail

permalink

#backintheday, before I understood how the relational part of MySQL worked, I made 20 columns to hold metadata about a record. 20 columns wasn't enough. Foreign keys #ftw. learning #fail

permalink

So on this one server, my default shell was /bin/sh, i'm not a sudoer, root owns my home directory, and another user owns /home. #linux #fail

permalink

so in order to access Mail.app's preferences to make it not be the default mailto: handler, I first have to configure it #fail

permalink

the goes to operator (x --> 0) makes #C feel more like #ruby! #fail

permalink

Seriously, Office? You rm the backup from the disk when I close it instead of doing the sane thing and moving it to my trash? #fail #hard

permalink

#compiling #gcc from source to get an ELF cross-compiler on OSX. This is so #fail

permalink

Because kernel.org is down, I used airdrop to copy over a cached version of the git source and formula from another computer so I could install it with homebrew #fail

permalink

for the number of times i ssh'd into my iMac this sumer, you would think that I'd have just put my ssh key on it #fail #doingitthehardway

permalink

I was not able to access google plus for about a week. Figured they were just having issues because I was still able to access email fine. Ended up having to recover my account. #fail

permalink

So #rails caches column names across the database and generates SQL for them even if they are never used. Why does this make sense? #fail

Photo for post 55.
permalink

My new favicon welcomes my site to 1996 #itsgreattobehere #fail

permalink

checked my Activity Monitor, had 2 #ruby processes each at 90% cpu which had been running for hours. trapping signals #fail

permalink

I need to get around to resetting up this server. I want to switch from svn to git and redo the whole apache config. why did I put the site in /var where I have to sudo all the time?? #linux #fail

Photo for post 43.
permalink

rebased some already pushed commits to clean up my history and forced the push. I feel so dirty. #git #fail

permalink

lesson learned: never muck around with mysql's database directory; you will screw up #fail #linux

permalink

this site: now with 5 times fewer 500 errors #django #fail

permalink

for some reason every part of the #django admin worked except the root page. had to turn on debug mode to see I was improperly configured #fail

permalink

ruby's splat is just python's tuple (un)packing #win #fail

permalink

unintentional side effect of my server's firewall: I can't traceroute it. #fail

permalink

lol. edited the site live again while working on a new app. 500 errors abound. #django #fail

permalink

lesson learned: never edit a site live. #fail #django

permalink

Found out that my raid array not mounting a couple months ago was due to a failed drive. Can't recover it. I lost so much data. #linux #fail

permalink

I HATE FLEX SO MUCH #fail

permalink

Had a bad run-in with rm -rf. lost the sqlite database for the site. on the bright side, I'm using mysql now. #linux #fail