hyperbo.la :: lifestream
permalink

Recently I've been on the "I should write a blog post" to "spend 2 hours updating dependencies across 4 language ecosystems in my blog's Bazel monorepo" pipeline. I should still write that blog post though #hypstatic #bazel

permalink

On Tuesday, a weekly CI run triggered a Miri failure for a crate I maintain which has some unsafe code. Two days later I had a fix validated, reviewed, and released plus a RustSec advisory published at https://rustsec.org/advisories/RUSTSEC-2023-0048.html. From advisory being published to the dependabot PR was about 16 hours. #rust #artichoke #security

permalink

And proof that this setup is both good and bad – CI failed because I forgot to set the publish timestamp in the post YAML on the previous post. But CI ALSO failed because I forgot to run gazelle. #win #fail #bazel #hypstatic

Photo for post 827.
permalink

My artisanally hand-crafted blog generator is an unholy mix of JavaScript, YAML, Markdown, assets distributed by npm package, Golang, Rust, and Python these days, glued together with Starlark in Bazel (deployed to GH Pages) #hypstatic #bazel

permalink

I recently published a demo Bazel workspace that shows off many of the rules and tools I use in the monorepo for this website. #hypstatic #bazel #github https://github.com/lopopolo/bazel_tools_demo

Photo for post 824.
permalink

Most frequently used commands, redux https://hyperbo.la/lifestream/560/

permalink

Just used exiftool and lots of zsh globbing to strip all metadata from JPEGs and PNGs used to build the static hyperbola site. #hypstatic

permalink

hyperbola/logo repository was merged into the hyperbola-static monorepo! There is only one repo used to build this site now! #hypstatic #bazel #monorepo

permalink

force a rebuild #hypstatic

permalink

Woo! hyperbola no longer requires webpack to build. Hundreds of dependencies removed! The build uses eslint for bundling. #webpack #hypstatic

permalink

I just removed all pagination from the #hypstatic lifestream section #win this design choice dates all the way back to the first implementation of lifestream in 2010 #history

permalink

I've been toying with the idea of removing all pagination from #hypstatic

permalink

There is no more #webpack in #artichoke. All web properties have been migrated to esbuild, eta template engine, the Node fs API, and custom build scripts. This is simple, this is good. #hypstatic is next but there is a lot to unwind. #win

permalink

with the lopopolo IAM user gone, the only ones left were the IAM users in the #artichoke and #hypstatic GitHub organizations used for terraform CI in the project-infrastructure repos. I used the new GitHub Actions OpenID Connect provider to wire up AWS identity federation following this guide – https://scalesec.com/blog/identity-federation-for-github-actions-on-aws/ #security #github #aws #win

permalink

With AWS SSO in place, I started using aws-vault locally for #terraform and AWS CLI. I learned about this tool at work. No IAM access keys with inline IAM policies! SSO + assume role for administrator access! #security #win

permalink

In #aws things, I migrated both hyperbola and #artichoke infra to separate AWS organizations and set them up with AWS Control Tower's account vending machine. Audit logs! AWS Config! SCPs! AWS Cost and Usage Reports! AWS SSO! #win #security #cost

permalink

I'm not sure how long ago this happened, but I got ownership of the @hyperbola handle on #github to match the npm namespace. Some js libs, old infrastructure for dynamic hyperbola, and #hypstatic live there now. I've added a 'project-infrastructure' repo like in Artichoke which does infra as code for the GitHub organization and #aws setup.

permalink

I took a hacksaw to my homebrew installation. I'm not sure how this happened but the brewfile I've been carrying around at one point had the entire dependency graph in it rather than the explicitly installed deps ... which made everything on my new Mac an explicitly installed dep. I had hundreds of libs installed for no reason. Burned everything down and started from scratch. #fail #win https://github.com/lopopolo/dotfiles/blob/fb9af6bd19df2149dba5a63a8db161702bcbe5db/homebrew-packages/Brewfile.rooster

permalink

macOS things – I tried out alacritty and tmux again but didn't switch away from iTerm2. I tried a bunch of fancy neovim LSP, floating pane, and lua config and then stripped all IDE bits out of vimrc to keep it just an editor. Finally enabled dark mode on my Mac (synced with the sunrise and sunset), switched to dark terminal and neovim color schemes. #win #vim

permalink

Another fun #rust update: `intaglio`, my string interning crate, recently got support for C strings (not yet released).

permalink

It's been a while since I've posted about #artichoke, but I recently landed a #rust implementation of #ruby's `String` class. #win https://github.com/artichoke/artichoke/pull/1222

permalink

yikes and pagintated navigation for archive pages had month and year mixed up in the link. #hypstatic #fail

permalink

ooof. While hacking on the lifestream part of the static site generator, uncovered that previous and next buttons for traversing lifestream permalinks had their polarity swapped. That bug has been there for over a year. #hypstatic #fail

permalink

I got a PR in artichoke suggesting to bump down the frequency of dependabot updates to monthly. I applied this change across all artichoke, hyperbola, and lopopolo repositories. It has been a big quality of life improvement and cuts down on a lot of churn in JS dependencies. Applying dep updates once a month is 👍 #win #artichoke

permalink

Yay! hyperbo.la is running Bootstrap v5. This upgrade was actually really small! Just some small changes to the Sass for customizing colors and updates to the navbar. #win

permalink

Most crates are no-std when possible, 100% documented, and I think pretty high quality. #patch #rust

permalink

focaccia and roe are UTF-8 case manipulation crates for implementing parts of `String` and `Symbol` in #artichoke. intaglio is Artichoke's symbol table, boba is an implementation of Bubble Babble Encoding which will be used in the `digest` stdlib package. #rust #patch

permalink

Published a few more #rust crates since I last checked in: boba, focaccia, intaglio, and roe. https://crates.io/crates/boba https://crates.io/crates/focaccia https://crates.io/crates/intaglio https://crates.io/crates/roe #patch

permalink

Since cutting hyperbo.la over to GitHub Pages and tearing down most of the AWS infra, monthly AWS costs have been about $1.25. The biggest two items have been ~$0.50 each for S3 and Route53. #aws #cost #hypstatic

permalink

This PR also took the opportunity to hand-optimize and hand-make retina variants of all images in the blog and lifestream. This eliminated most of the large asset warnings from #webpack #performance #hypstatic

permalink

Sharding the #webpack build meant `webpack-dev-server` was not longer an acceptable choice for a dev server. Replaced this with a #bash script and `python3 -m http.server`

permalink

There's some recursion depth error that causes #webpack to abort when using ~1500 `html-webpack-plugin` plugins that I worked around by sharding the webpack build.

permalink

In the process nukes 1000s of lines from package-lock.json #hypstatic

permalink

Just upgraded hyperbo.la's build pipeline to #webpack 5 #hypstatic

permalink

Correctness test that the lifestream handles rendering dates correctly when the week year and year are not the same. https://twitter.com/fasterthanlime/status/1342967204153876482

permalink

As of last night, all hyperbo.la AWS infrastructure is torn down. https://github.com/hyperbola/hyperbola/pull/111 #aws #terraform #devops #hypstatic

permalink

To make this change on dynamic hyperbola would have meant 5-10 Ansible deploys to a Vagrant VM. Iteration speed would have been terrible. #devops #a11y #lighthouse #hypstatic

permalink

The static site with local #webpack development environment is already paying off! Google Lighthouse accessibility score for hyperbo.la is now 100 with increased text contrast. #a11y #lighthouse #hypstatic

permalink

hyperbola is deployed to GitHub Pages now! Executed via DNS cutover with terraform. Now to destroy the old infrastructure in AWS. #github #hypstatic #aws #terraform #hypstatic

permalink

And just like that, dependabot is enabled for hyperbola-static, the WIP lifeastream branch is merged, and the CI is spruced up #hypstatic

permalink

There have been over 600 PRs in the last year of Artichoke. https://hyperbo.la/lifestream/737/ https://github.com/artichoke/artichoke/pull/847 #artichoke #git

permalink

This is a setup that has worked well for www.artichokeruby.org. Set dependabot to autoupdate deps and the "maintenance" has been updating content. Deploys are automatic, regression testing of dep upgrades is automatic. #automation #hypstatic

permalink

There is Ansible, Packer, Terraform, JavaScript, Webpack, Python, Django, Ubuntu, nginx, EC2, Route53, ALB, S3. I think I can simplify this to GitHub Actions, GitHub Pages, Node, Webpack, and statically configured DNS with my registrar. #hypstatic

permalink

Looking at the commit history for Python hyperbola over the last 3 years shows that it hasn't been developed, only maintained. It is done, but the current repo is expensive to maintain. #hypstatic

permalink

Switching to a static site built with webpack at least affords the opportunity to have an integration test that the site compiles. #automation #hypstatic

permalink

There is no automated build and no test suite. #automation #hypstatic

permalink

Dependency management in Python makes me sad. #fail #python #hypstatic

permalink

I've lost my ability to develop locally since I don't want to install VirtualBox on my new laptop. #fail #hypstatic

permalink

I run an outdated version of Django. #django #fail #hypstatic

permalink

The terraform config for this project is on 0.12 and I have no desire to update it to 0.13. #fail #terraform #automation #hypstatic

permalink

I will miss the ease of adding lifestream posts via the Django admin UI, but given that I'm posting elsewhere these days, I don't think I'll miss it too much #hypstatic

permalink

Remaining pieces are implementing hashtag browsing pages and the date archive browsing pages #hypstatic

permalink

Most of my updates these days are either in the Artichoke Discord or @artichokeruby Twitter #hypstatic

permalink

The AWS infra that the current site runs on is $50 a month I do not need to spend. #aws #cost #hypstatic

permalink

I've had a work in progress fork of hyperbola to migrate to a static site hosted on GitHub pages for over 6 months. I can't find the time to finish it. #hypstatic

permalink

More #artichoke updates since March: a marketing site at www.artichokeruby.org, prebuilt containers on Docker Hub https://hub.docker.com/r/artichokeruby/artichoke, and prebuilt nightlies https://github.com/artichoke/nightly/releases/latest #win

permalink

Also Windows Terminal and the Cascadia Code font were a dream to use. I replaced Inconsolata with Cascadia Code on my Mac iTerm2 #terminal #font #windows #win

permalink

#vscode and LLDB in VSCode were SO amazing to use. Within a couple of iterations, I was able to track down the use-after-free: https://github.com/artichoke/artichoke/pull/674 #fail #win #artichoke

permalink

After repeated attempts of replicating my windows CI setup in a developer VM, I sloppily installed software via web installers. With every VS component, rustup, RubyInstaller, LLVM, choco, and winflexbison, I was running with a reproducer #artichoke

permalink

Once GH-670 was merged, I had a segfault that only (and reliably) reproduced on one test in Windows #artichoke #rust #windows

permalink

After 50 PRs, I was able to put GH-442 to bed and remove the Rc wrapper from the Artichoke state https://github.com/artichoke/artichoke/pull/670 #win This refactor took 4 months #fail #rust #artichoke

Photo for post 768.
permalink

Nightly builds of Artichoke are available via ruby-build: rbenv install artichoke-dev #artichoke #win

permalink

The one dynamic bit I cannot truly replicate is shortlinks but I never used this feature #win

permalink

No more security updates, terraform, autoscaling group rolling deploys, packer and baking AMIs, Ansible deprecation warnings, RDS outages #win I can unsubscribe from nginx release notes listsrv lol

permalink

Big wins from switching to static: modifying content with a lightweight dev server, CI via GitHub Actions, automated deploys, better webpack integration and minification, site is free as in beer to operate #win

permalink

Partials tended to end up in the same places as before #win

permalink

Some bespoke filters I hand rolled before got pushed into dependencies, like hashtag parsing and linking moving to linkifyjs #win #yolo

permalink

There is about as much javascript as there is python in the dynamic, django-based hyperbola, which is to say not much #win

permalink

It mostly works and is reasonably speedy, although while iterating, a template error in the lifesteam post rendering code caused ~1000 templates to fail compilation. I let the build peg the CPU for 20 minutes before giving up #fail

permalink

The static site is based on webpack and webpack-html-plugin with a few preprocessor scripts that emit the templates #webpack #js

permalink

I made the mistake of exporting the database first AND not documenting how I transformed it to YAML, so I've been hesitant to post here #fail

permalink

for the past 2ish months I've been slowly working on porting hyperbola to a static site with the ultimate goal of winding down an #aws account

permalink

I published my first #rust crate (and package of any kind, ever)! rand_mt is a fork of an existing Mersenne Twister crate which adds compatibility with the latest rand_core and makes it no_std https://crates.io/crates/rand_mt #patch

permalink

I contributed a feature to #rust! Added an implementation to allow turning a &mut str into a String using the standard From trait #win #patch https://github.com/rust-lang/rust/pull/69661

permalink

Migrated all #artichoke CI to GitHub Actions #github #win Jobs and workflows are so much easier to reason about. And it's free 💪

permalink

I've been absent from the lifestream for a bit. I've been busy implementing!

permalink

hyperbola was down for 26 hours because the CA cert bundle on its backing RDS instance was not rotated before the 2015 bundle expired #fail #outage #aws

permalink

Pushing errors to the edge and forcing callers to handle them makes for better code #artichoke https://github.com/artichoke/artichoke/pull/434

permalink

Nothing like a little build breakage to start the weekend https://github.com/artichoke/artichoke/pull/417 #artichoke #fail

permalink

Lol following up on https://hyperbo.la/lifestream/743/ ... https://github.com/artichoke/artichoke/pull/348 disabled most converters to improve compilation times and https://github.com/artichoke/artichoke/pull/397 added them back behind a feature

permalink

#artichoke RubyConf recording here: https://www.youtube.com/watch?v=QMni48MBqFw

permalink

I missed a couple of months posting! I was working on my RubyConf talk about #artichoke. slides here: https://artichoke.github.io/rubyconf/2019/

permalink

Just explicitly shut down the frklft.tires site by doing a terraform destroy. Sad, but that code didn't need to live forever.

permalink

I added a feature to rust-bindgen to make a dependency optional to make the build lighter #github #patch https://github.com/rust-lang/rust-bindgen/pull/1615

permalink

Reported a critical bug in rustfmt #fail #github #patch https://github.com/rust-lang/rustfmt/issues/3770

permalink

Also added many more converter implementations with macros. There are 988 TryConvert implementations. #win #fail #artichoke

permalink

Led to reporting this #mruby bug: https://github.com/mruby/mruby/issues/4684 #github #fail although I was able to work around it #artichoke

permalink

This change added new functionality: Converting to and from byte and string slices #win #artichoke

Photo for post 740.
permalink

The converters are central to #artichoke. Changing this core abstraction was painful: 84 files changed, 2000 lines added, 2800 lines removed. #fail #git

permalink

The converters allow the runtime to convert between Rust and Ruby types. They are the glue that the entire Ruby Core and Ruby Standard Library implementation is built upon. #artichoke

permalink

I'm separating the core of #artichoke from the #mruby backend specific glue. This step involved reimplementing the converters with the new traits in core.

permalink

This was a nasty PR https://github.com/artichoke/artichoke/pull/242 #artichoke #fail

permalink

Some folks have been contributing #Ruby and #Rust code to #artichoke! #win #github

permalink

I added a macro to oniguruma to enable it to build on clang for #wasm targets #patch #github #artichoke https://github.com/kkos/oniguruma/pull/150

permalink

I reported an undocumented panic in #rust that was fixed today #patch #win https://github.com/rust-lang/rust/issues/63456

permalink

Reported a build issue on #mruby with gperf 3.1 #ci #build #patch #github #artichoke https://github.com/mruby/mruby/issues/4628

Photo for post 732.
permalink

#artichoke is the top trending repo on all of #github right now #win

permalink

I'm trying to improve Regexp performance in #artichoke by using the regex crate instead of oniguruma in some cases. It turns out not to be faster in all cases #patch #rust #fail https://github.com/rust-lang/regex/issues/604

permalink

This was hard to track down: emscripten linker error with a particular function signature #rust #wasm #patch #github #fail https://github.com/emscripten-core/emscripten/issues/9135

Photo for post 729.
permalink

I built an #artichoke #Wasm playground in the browser https://artichoke.github.io/artichoke/ #win

Photo for post 728.
permalink

That is a lot of code #artichoke #github #patch #win

Photo for post 727.
permalink

That is a lot of code #artichoke #github #patch #win

permalink

I forked onig and hope to add support for the wasm-unknown-unknown build target. For #artichoke though, it's too soon. Once mruby-sys is pulled into #Rust I'll try again #ruby #mruby #fail #patch #Wasm

permalink

Neither mruby-sys nor onig can build with wasm-unknown-unknown or wasm-wasi, so I was stuck with wasm-unknown-emscripten. The linker on mruby-sys would dead code eliminate artichoke_backend::Artichoke #fail #Wasm #artichoke #mruby

permalink

One of the goals is to have a #Wasm build target. This is what nerdsniped me this weekend. #artichoke

permalink

The biggest accomplishment was redoing the #artichoke README https://github.com/artichoke/artichoke#readme Thought about project goals and differentiators from MRI. Added milestones to track progress in achieving goals #win #ruby #artichoke

permalink

Being forced to set up the repository infra from scratch in cactusref was a good way to fix everything I disliked about my dev environment. CircleCI build and scripts/lint.sh got overhauled #win #artichoke #shell #bash #node #lint

permalink

With renewed focus of building my own #Ruby instead of extending #mruby, I put in some effort to make mruby an implementation detail of #artichoke. Lots of refactoring with sed. #fail

permalink

ferrocarril had different goals than artichoke: it wanted to run Rails. To do that, it had a lot of extra bits in it like gems, a Rack server, and sample apps. Nuked all of it #win #artichoke

permalink

Step 3: create the #artichoke repository where the core #ruby development will take place. #artichoke

permalink

Step 2: transfer ownership of ferrocarril to artichoke organization #artichoke #github. Not much to do here, updated some links, fixed some branding, got CircleCI building again #artichoke

permalink

CactusRef is still an incredibly unsafe crate, but at least it aborts if it detects a use-after-free #fail #artichoke

permalink

Extracting cactusref required completely rewriting #git history, updating docs, adding a build, adding benches, importing my blog post, updating links in the blog post #artichoke

permalink

Step 1: extract CactusRef from ferrocarril into its own repository https://github.com/artichoke/cactusref #artichoke

permalink

This weekend I did the migration, which took place in multiple parts #artichoke #github

permalink

ferrocarril is all grown up. I've moved development of my #ruby to a dedicated #GitHub organization. The project is called Artichoke https://github.com/artichoke/artichoke

permalink

Every toggle switch in a UI is secretly an Option<bool> #design

permalink

Matz likes my code omg omg omg #win #ruby #mruby #patch https://github.com/mruby/mruby/pull/4560#issuecomment-509572113

permalink

I’ve been uncovering a few bugs in mruby trying to get ruby/spec to at least run to completion. This bug report fixed an infinite loop that unblocked that #mruby #patch https://github.com/mruby/mruby/issues/4555

permalink

Reported a bug in layout generation for structure with flexible array members to #rust bindgen that was recently fixed #patch https://github.com/rust-lang/rust-bindgen/issues/1589

permalink

Reported an uninitialized memory access to #mruby upstream that was recently fixed #patch https://github.com/mruby/mruby/issues/4556

permalink

That memory leak turned out to be quite the yak shave. setjmp/longjmp from C leaving #rust memory in inconsistent state #fail https://github.com/artichoke/ferrocarril/pull/168

permalink

I had a PR accepted in mruby! #patch #ruby #mruby #win https://github.com/mruby/mruby/pull/4560

Photo for post 705.
permalink

#git bisect is why it's important that your repo be buildable at every commit. Currently fixing a massive memory leak in ferrocarril #fail

permalink

Optimized String#scan in ferrocarril to be comparable to CRuby performance #rust #ruby #mruby #performance https://github.com/artichoke/ferrocarril/pull/138

permalink

Added a feature (both a new capability and a compile time feature!) to rust-embed to allow it to embed sources relative to crate root which is useful in workspaces #rust #patch https://github.com/pyros2097/rust-embed/pull/67

permalink

Adding paragraph mode to String#each_line in mruby core #ruby #patch https://github.com/mruby/mruby/pull/4527

permalink

I've proposed a couple of PRs to speed up UTF-8 String handling in mruby https://github.com/mruby/mruby/pull/4529 https://github.com/mruby/mruby/pull/4531 #patch #ruby #github

permalink

ruby/spec is a set of specifications for the Ruby language shared across implementations. I had a fix accepted to the StringScanner spec! #patch #ruby https://github.com/ruby/spec/pull/671

permalink

I reported a compiler improvement for suggestion diagnostics #rust #patch https://github.com/rust-lang/rust/issues/61555

permalink

I have Regexp#match? https://github.com/artichoke/ferrocarril/pull/93 #win #rust #ruby

permalink

Filed a couple more issues I ran into while working on ferrocarril: https://github.com/SergioBenitez/Rocket/issues/1019 https://github.com/rust-onig/rust-onig/issues/102 #patch

permalink

It has been a really productive weekend. Merged 23 #github PRs for ferrocarril #win #rust #git https://github.com/artichoke/ferrocarril/pulls?utf8=%E2%9C%93&q=is%3Apr+created%3A2019-05-24..2019-05-27+is%3Aclosed+

permalink

Despite all I talk about #aws #ec2, I cannot for the life of me type isntances correctly #fail

permalink

here's another Clippy bug #rust #fail https://github.com/rust-lang/rust-clippy/issues/4143

permalink

Adding #ci to a project is encouraging such better code hygiene. Because I want to make sure the build stays green, I actually make PRs even if it is only me approving them. Seeing the code in the browser makes it easier to review than #git diff #win

permalink

In a related note, I now peg the nightly version in the rust-toolchain file to a specific date archive #ci #rust #win

permalink

My #ci builds of ferrocarril caught a bug in rustyline that made compilation fail on the latest nightly #rust #win #patch https://github.com/kkawakam/rustyline/issues/217

permalink

I reported a #segfault in mruby and spoke directly with @matz! #win https://github.com/mruby/mruby/issues/4460 #patch

Photo for post 688.
permalink

Excluding generated and vendored sources, ferrocarril is 8700 lines of #rust, #c, and #ruby. This is a lot of code. In very uncharacteristic fashion for a side project, a big chunk is tests #win https://github.com/artichoke/ferrocarril/commit/fe2febc

permalink

You can tell which parts of ferrocarril are new to me based on how many consecutive commits are dedicated to a feature. By this measure, the new (and hard) things were sys crates, #ci, and TryFrom traits #rust #win #fail

permalink

I tried to explain continuous integration to my Dad and he synthesized it as "spellcheck for code" which is pretty good tbh #win #ci

permalink

It only took 49 build jobs, but the ferrocarril build is passing on CircleCI https://circleci.com/gh/artichoke/ferrocarril/47 This is the first time I've ever set up #ci for a side project #win

permalink

ferrocarril is a new project I've been working on to embed #ruby on Rails in #rust https://github.com/artichoke/ferrocarril

permalink

The keyword "unsafe" appears 372 times in this codebase #rust

permalink

#patch got another PR accepted in the Ansible + ACME + Route53 role I use for the hyperbola dev env load balancer #win https://github.com/mprahl/ansible-role-lets-encrypt-route-53/pull/15

permalink

Using cargo:rerun-if-changed and cargo:rerun-if-env-changed to make most sys crate builds a noop. developer productivity #win #rust

permalink

About 6 weeks ago I reached out to two literary agents to see if they'd help me publish a book. I got an explicit rejection last week! A little bit #fail, but also definitely progress #win 😊

permalink

I just used #github to create a reproducer case for a panic in a #rust library #win https://github.com/lopopolo/rust-embed-panic https://github.com/pyros2097/rust-embed/issues/61 #patch

permalink

I generated FFI bindings for mruby with bindgen! I wrote a #c extension library for wrapping mruby macros! #win I wrote (a lot of) unsafe #rust code. Clippy helps a lot to make sure I'm not doing anything stupid. I have caused quite a few #segfault #fail

permalink

I implemented a #rust macro for the first time to implement some traits to convert between mruby values and Vec and Option containers #win

permalink

I overflowed the compiler (which didn't crash it) and crashed the linter with an ICE #win #fail

permalink

I've been working on creating an idiomatic, modern set of #rust bindings for mruby, an embedded #ruby interpreter. I am learning a ton and doing lots of code things I've never done before! #win

permalink

I found an ICE (internal compiler error) in clippy in #rust nightly #fail https://github.com/rust-lang/rust/issues/60067

permalink

looks like I didn't finalize the deploy in January so I had a few extra AMIs kicking around that I was needlessly paying for #aws #cost #fail #automation

permalink

The true motivation for getting me to mess with hyperbola today was an instance retirement notification from AWS. I had to muck with poetry to cycle my ASG #fail #aws

permalink

Uninstalling python 3.7.1 from pyenv broke my local poetry venv for hyperbola which prevented me from managing my deps #fail #python

permalink

Yikes haven't done a deploy since January 6. I've definitely been ignoring all of the USNs about Linux kernel vulnerabilities. #fail Django 2.2 release is what got me to get to upgrading #django #win

permalink

whoa throwback `chattr +i` to prevent puppet from overwriting files on a prod box that you were manually changing #devops

Photo for post 668.
permalink

Deck making for an internal tech talk called "How to Implement the Chromecast Protocol in Rust When You Don't Know Rust" #rust futures and impl trait 😭 #fail

permalink

When asking someone to do something via email, the fact that they’ve even opened it is a big hurdle. Make their life easy: in-line as much as you can, minimize clicks, make links easy to scan, be direct, have a tl;dr #communication #email

permalink

I'm speaking at Monitorama on June 4-6. My talk is titled The AWS Billing Machine and Optimizing Cloud Costs. #conference #aws #cost

permalink

I'm speaking at DevOpsDays Seattle on April 23. My talk is titled The AWS Billing Machine and Optimizing Cloud Costs. #conference #aws #cost

permalink

Larson Media Training 101: Answer the question you wish you were asked. #communication

permalink

Spaceships don’t come equipped with a rear view mirror 🚀 #win

permalink

I live in Seattle now. First week at the Stripe office was quiet due to the snow. Working on expanding Stripe into Latin America.

permalink

I have eliminated all explicit casts in punchtop #rust #win https://github.com/lopopolo/punchtop/commit/1e41ca3

permalink

More #rust safety: removed a cast to usize with the new usize::from_be_bytes converter https://github.com/lopopolo/punchtop/commit/2263c80 #win

permalink

#marketing is easier if you have metrics to support your impact. I like to always tie my metrics back to dollars.

permalink

One common hangup I've encountered when discussing #marketing my work with other engineers is the notion that the work was easy or trivial to implement. That's not the important part! What the org cares about is impact. Tell a good story!

permalink

To be good at #marketing yourself, you need to be good at communication, both written and verbal. Don't be afraid to talk about your work, either to your manager, your skip level, your users, an email list, or the company all hands meeting.

permalink

As an engineer, I care very much about #marketing myself, my team, and my work. I am of the impression that my work does not speak for itself, I have to speak for my work. Being high impact means making sure the org knows it, too.

permalink

The game that I'm working on is punchtop https://github.com/lopopolo/punchtop #win #rust #powerhour

permalink

Making a mental note that I've decided to go static with hyperbola. Not sure when this project will get done, but here's to 2019. I have dreams of a custom #webpack and #markdown generator, S3 and CloudFront for distribution.

permalink

I made a small #rust crate called stream-util for a game I'm working on. https://github.com/lopopolo/punchtop/tree/master/stream-util allows graceful drains of tokio mpsc channels or canceling a futures Stream. Clippy pedantic and fully documented #win

permalink

I implemented a #rust client for the #chromecast protocol for a game I'm working on. I collected a fairly exhaustive set of protocol documentation: https://github.com/lopopolo/punchtop/tree/master/cast-client

permalink

The most expensive part of hyperbola's #aws infrastructure is the SSM PrivateLink endpoint in 3 AZs #fail #cost

permalink

First #rust #patch accepted https://github.com/asomers/futures-locks/pull/10 closing out 2018 with a bang

permalink

Optimizing #react bundle ... cut bundle size by 72KB. Biggest wins are turning an image into CSS + an emoji, removing unused deps, enabling mini-css-extract-plugin in release builds #win #webpack index.html is 333 KB

permalink

Took 4.5 years but I've finally started to learn #rust: https://hyperbo.la/lifestream/298/. I'm implementing a power hour game that plays to a #chromecast with a #react webview UI.

permalink

Been using #ack for almost a decade. Installed #ripgrep today. Wow it's so fast! #win

permalink

I've now over-engineered an iTunes library integrity checker by turning a script into a module #win

permalink

Just accidentally truncated my .bash_history. Restored from backup but the latest was 55 days ago. #fail

permalink

I had a #patch accepted to an Ansible role I use in my Vagrant environment https://github.com/mprahl/ansible-role-lets-encrypt-route-53/pull/14

permalink

I promised a postmortem: https://hyperbo.la/w/secrets-in-parameter-store-postmortem/

permalink

I was hard down for ~5min last night while rolling out secrets in parameter store. 0.149.0, 0.149.1, 0.149.2, and 0.149.3 were bad releases #fail. 0.149.4 is stable: https://github.com/hyperbola/hyperbola/compare/v0.148.0...v0.149.4. Postmortem pending.

permalink

41-commit weekend 🤩Feel much better about my #webpack , #ansible, and #packer code #win

permalink

The computers did exactly what I told them to do 😕 #fail

permalink

Add in some manual #terraform state edits and deleting things in the #aws console and we're recovered #fail #win

permalink

At least I was able to repro in local #vagrant once I figured out what was going on. #win

permalink

A red herring was adding a call to dd in the cleanup script to zero the free space on the disk. I thought I was somehow filling the disk and not reclaiming space.

permalink

The cleanup script didn't error because my set flags were in the shebang but #packer was invoking the script via bash instead of directly #fail So many yaks.

permalink

This change was introduced in 0.146.0 but did not manifest due to a bug in the cleanup script. I was not passing -y to apt autoremove, which caused the command to abort and end the script with an error. #fail

permalink

Removing just the dev package with #ansible #provisioning had no effect in local env, but combined with the apt purge in the #packer script, uninstalled mysqlclient

permalink

Root cause: My app depends on the python package mysqlclient, which in turn depends on the libmysqlclient system package. I was pulling this in transitively via the dev package required for building.

permalink

Undeployable manifested as healthz returning 502 when adding a new instance to the ALB, marking it as unhealthy and timing out #terraform

permalink

I recently started cleaning up after myself during #provisioning. There are two steps to this: #ansible uninstalls build deps and #packer executes a cleanup script that does an apt purge and autoremove

permalink

Got into an undeployable state due to differences in #provisioning between local and prod environments #fail https://github.com/hyperbola/hyperbola/commit/a914992

permalink

I completed removing build artifacts from source control. Static assets dist and document-root dirs: replaced with running #webpack on host. Third-party #ansible roles: installed from galaxy via #packer and #vagrant at provision time. #win

permalink

code yellow 😕

permalink

That was a 35-commit weekend. New features and improved code quality. #win https://github.com/hyperbola/hyperbola/compare/918f6d0f..4c331eee

permalink

Significantly refactored my #webpack static asset pipeline today. No longer build assets locally and check them in. Build happens as part of #ansible provisioning. #win https://github.com/hyperbola/hyperbola/commit/eaf5b3197

permalink

I added code block and syntax highlighting to my new #blog. Planning on using it for an upcoming post about #terraform.

permalink

#django is a joy to use. Making my new blog app was so easy. Each view is only two lines of code! #win

permalink

I accidentally skipped v0.139.0 today because prettier barfed during cutting the release and I forgot to reset my git tree. I guess I forgot to run prettier on my whole repo when I enabled it. #fail

permalink

I made a post on how your org chart shows up in your infrastructure: https://hyperbo.la/w/aws-org-chart/ #blog

permalink

I made a post on Prosperity's partnership with Finance: https://hyperbo.la/w/engineering-finance-partnership/ #blog

permalink

Further #cost optimized my #AWS infra. Saved $3/month by turning off CloudWatch monitoring and making my ASG out of spot instances https://github.com/hyperbola/hyperbola/commit/ffa0e34 #win

permalink

I had a #patch accepted to gunicorn https://github.com/benoitc/gunicorn/pull/1889

permalink

Jumped to latest #django 2.0.x and then to latest 2.1.x today. No deprecation warnings, smooth sailing. As a plus, I turned on PYTHONWARNINGS for my app in systemd #win

permalink

#AWS suggested for us to use a snowball to ship our data to a new region. This is an amazing product, but yea no

permalink

That was easy! hyperbola running on t3s now. #aws #terraform #win

permalink

docker lessons: https://github.com/hyperbola/hyperbola/pull/94

permalink

I also pruned a lot of config from my 12factor env, favoring convention for things like DB connection strings. In doing so, migrated to one hostname for DB across all environments. /etc/hosts alias in Vagrant, network in Docker, private hosted zone in AWS

permalink

Finally deployed a 2 month old branch incorporating lessons learned from my Docker experiment into my VM-based deploys. Big change is building from local source instead of pulling in a tarball from GitHub. Iteration speed is improved #win

permalink

One hack I've been using to enhance my productivity in the face of KTLO work is to write lots of things down. Project plans, system diagrams, user interviews. It helps organize the work that I have done and keeps me motivated.

permalink

The past 4ish months I've been a team of one. Been a decent slog. It's been hard to be productive when all of the KTLO work falls on me. Getting a new teammate in a couple weeks. Excited. 🤩

permalink

I've been retraining myself to say "Hey Google" instead of "OK Google"

permalink

I published a post about AWS Reserved Instances today https://stripe.com/blog/aws-reserved-instances

permalink

Optimizing for #cost and complexity, #docker and ECS/EKS are not worth it for my 1 node crud app

permalink

Saved me $17 a month. My primary #AWS #cost is now my ALB. I'd replace it with an nginx if not for ACM making certs so easy

permalink

Modified RDS to be single-AZ #HA

permalink

Set desired backend ASG capacity to 1, but added autoscaling alarms to allow scaling up to 3 instances #HA

permalink

Realized that hyperbola doesn't need to be super #HA

permalink

Last week I tried to build hyperbola with #docker. I ended up with 5 containers to deploy a stage environment. https://github.com/hyperbola/hyperbola/pull/92

permalink

hyperbola is ready for #django 2.1. https://github.com/hyperbola/hyperbola/commit/d92a79f. That was easy #win

permalink

LuaLaTeX is amazing. My documents look better than they did with XeLaTeX. Less hyphenation, less inter-section spacing. fontspec, polyglossia, selnolig, nowidow, and impnattypo packages are dope too. #LaTeX #win

permalink

The Makefile for a project like this defines two versions of all the tasks. One runs locally on my mac and "recursively" calls make using `vagrant ssh` setting a make var to indicate the task is running in the VM #make #LaTeX

permalink

I've been using make and vagrant for self-contained builds in other projects. I've been working with #LaTeX a lot and a bionic vm with TexLive installed creates a repeatable build env, takes 15 minutes to set up, and doesn't pollute my desktop #win

permalink

Upgrades are painful, so upgrade frequently. hyperbola now running Ubuntu 18.04 Bionic. 2 build config changes and 1 removed ansible task. #win

permalink

Switched from community PPA to official nginx-provided binaries. That was ... really easy: https://github.com/hyperbola/hyperbola/commit/5461837 #win #ansible

permalink

I am surprised that #django autolinked the .tires domain. We live in the future.

permalink

for www.frklft.tires, I stopped using #terraform for managing the static content of the site. It now lives outside of my terraform code in a public directory, published explicitly with a make target #win

permalink

I launched https://www.frklft.tires/ last week. Built with S3, CloudFront, and Lambda.

permalink

My current understanding of this column is that it is the instance family if the usage type is eligible for RI instance size flexibility

permalink

What they neglect to mention is that DedicatedUsage types do not have an instance family associated with them

permalink

AWS says that product/instanceTypeFamily is "The instance family that is associated with the given usage. For example, t2 or m4."

permalink

Sometimes using the #AWS cost and usage reports is just not fun, mostly due to the myriad of columns being undocumented. #fail

permalink

https://hyperbo.la/lifestream/347/ was an anti-pattern which got fixed when I moved to pipenv

permalink

When all you have is a rock, everything looks like a hammer

permalink

This is a great article: https://rachelbythebay.com/w/2018/02/26/frame/

permalink

Last week I shipped secondary attribution for Jenkins builds. Email featured lyrics from Bills, Bills, Bills by Destiny's Child #win

permalink

pandas and jupyter notebooks are my new favorite hammers

permalink

Semantic versioning is a lie (looking at you #packer). My config stopped working because a key was deprecated between 1.1.x and 1.2.x. Somehow this prevented the config from validating. #fail

permalink

So it turns out I shouldn't have ignored that MySQL backtrace when printing the help text of my new django management command in dev. That's why it hung when building the AMI. #fail One line fix: https://github.com/hyperbola/hyperbola/commit/728f1d68

permalink

I have pinned dependencies everywhere _except_ my AMI build pipeline. Bitten by the packer 1.2.0 upgrade breaking the ansible-local provisioner. #fail

permalink

it looks like prod peaks at 75 req/s #performance

permalink

Not to mention Let's Encrypt to Amazon back to Let's Encrypt again (for vagrant) for certs

permalink

I've had significant churn on tools in the past year for building hyperbola: pip to pip-tools to pipenv. gulp to webpack. linode to AWS. bespoke deployment and scripts to ansible. All good, just exhasuting.

permalink

vagrant-lb PR: https://github.com/hyperbola/hyperbola/pull/86

permalink

Ansible, Let's Encrypt, nginx, and Vagrant have combined to make my local env look more like prod. Spun up a new ALB-like VM with HTTPS. less divergence #win

permalink

Starting 2018 off right by fixing pbr installation, deploying django 2.0.1, upgrading Pillow, and removing dep on django-localflavor #win

Photo for post 574.
permalink

Baby's first hadoop. Taken from long forgotten code written in 2005 #history

permalink

Removed some "bespokeness" from hyperbola: migrated from a shell script that exported env vars and exec'd to dotenv. Same concept, but now a standard mechanism #win

permalink

Put some investment into my #vagrant setup. MySQL running in a separate VM with automatically provisioned fixtures means I can go from zero to full replica of prod in 10 minutes. No more dep on laptop MySQL. #win

permalink

New in v0.122.0: python packaging actually works now. No .pth symlinks. No PYTHONPATH. setup.py and pbr install hyperbola directly into the venv. Finally. #win

permalink

hyperbola is now running #django 2.0. Added automigrate to hyperbola-app systemd unit, eliminating one of the last things I ever needed a bastion host for #win

permalink

Found a lifestream bug during my Django 2.0 upgrade that was never exercised on the live site because I've never had more than 40 posts in a month #fail

permalink

I suspect that NTP traffic was the trickle of traffic that traversed my recently decomissioned NAT

permalink

That was easy. Enabled #AWS Time Sync by adding one ansible role, reading some docs, and redeploying. #win

permalink

Accidentally deleted my private subnet route tables in the process though, which broke S3 access for ~20 minutes. Could've been much worse #fail #outage

permalink

Shaved another $35 (40%) off my #AWS bill by disabling the NAT on my app subnets. Yay immutable infrastructure and VPC endpoints #win

permalink

I made a thing! https://www.burnfastburnbright.com/ Bootstrap 4, route53 domains, and terraform made this really easy. went from 0 to 100 in about 1.5 hours. #win

permalink

New in v0.117.0: nuked time-elements webcomponents due to Firefox breakage (cut js payload by a factor of 4), infra improvements to resume handling #win

permalink

The v0.116.0 deploy was done using a spot instance with packer. A bigger instance for half the price #aws

permalink

New features in v0.116.0: bootstrap4, removed RSS and Atom feeds, 100% webpack frontend build, css purification improvements, and healthz middleware

Photo for post 560.
permalink

Most frequently used commands, redux https://hyperbo.la/lifestream/146/

permalink

LOL that was only six years ago ... don't let your dreams stay dreams: https://hyperbo.la/lifestream/51/ #aws

permalink

welp that didn't last long. CloudFlare only queries a subset of NS records to check for liveness and has determined that I no longer use CloudFlare. Working on purging them from #terraform and registrar now #fail

permalink

It is a good thing that I've automated things well enough that I don't need the bastion #win

permalink

Even more cost savings: dynamically provisioned bastion cloudformation stack #terraform #aws

permalink

More cost savings. RAM footprint of a hyperbola backend is 143MB. Switch from t2.micro to t2.nano #aws #win

permalink

I accidentally created a CNAME (instead of an A record) for an IP today. Lots of confusing errors from nslookup, ssh, and host. Meanwhile dig appeared to resolve the record. #fail

permalink

hyperbola: now with multi-homed DNS. AWS Route 53 and CloudFlare, made possible by terraform. (In the process upgraded hyperbo.la mail to a 2048-bit DKIM key) #win #redundancy #devops

permalink

I accomplished this migration with ~no downtime #win. I spun up the new infrastructure and then deployed new AMIs with updated service records. I did have ~2 minutes of 500s when I accidentally overwrote old mysql DNS record due to a bad copypasta #fail

permalink

Switch from 3 to 2 backend machines. 1 is enough to handle the load I get, so use the bare minimum for redundancy #aws

permalink

Removed dependency on redis by switching to a django-provided database-as-cache adapter. My redis cluster was used only for admin sessions and caching a sidebar on the lifestream page. Unnecessary overhead #aws

permalink

Switched DB instance type from db.t2.small to db.t2.micro. From running my linode I know that MySQL never used more than ~400MB of RAM so I knew this was safe. My database is tiny #aws

permalink

Switched from Aurora to a multi-az RDS instance. I don't need the complex topologies that aurora allows and it forced me to use an overprovisioned instance type #aws

permalink

Now that I've shown I can go all out with the most expensive #AWS components, today I exercised my cost efficiency and right sizing muscles. I cut my AWS bill in half with the following steps:

permalink

3 AZs I feel so alive #aws

permalink

Just bumped backend ASG from 1 -> 3 t2.micros. With this change now all parts of hyperbola (redis, mysql, backend, lb) are multi-AZ #win

permalink

mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -u root mysql was the magic incantation required to get lifestream archive views working locally

permalink

Today's shipped email featuring subtly modified lyrics from Kanye's Flashing Lights

permalink

The magic command to make homebrew work after uninstalling Xcode: sudo xcode-select -switch /Library/Developer/CommandLineTools #win

permalink

this is just one example of the times #git has saved my life #win

permalink

#history throwback to the time that my wiki was spammed by a bot that turned all the pages into link spam for discount pharmaceuticals https://github.com/hyperbola/hyperbola/commit/1a3725b84bf82982945f68d14ff09ae7aa00d0ba #fail

permalink

I've taken to opening shipped emails with playfully mutated rap lyrics. Super Nintendo. #win

permalink

Total upgrade time ~1 hour. Includes code changes, testing locally, cut release, deploy to vagrant, build AMI, terraform, smoke test #win

permalink

I skipped #django 1.11.4 and 1.11.5. Finally did an upgrade today to 1.11.6 #win. My dep upgrades for python, js, and ansible deps are too coarse grained. Pulled in ansible changes which blocked the deployment #fail

permalink

and no more chrome on iOS. Using safari there. The only value chrome brings is unified history and being logged in to my google account. I use neither of these features

permalink

I switched to Firefox Developer Edition today on my mac. It is pretty blazing fast. Looking forward to 58 making it to the dev channel in November

permalink

With @huff's help, I have discovered ETL's True Name. ETL is just indexes. Indexes are just caching. Caching is just named replication. Names are the only problem in computer science.

permalink

When sorting cards in a hand, I use multi-pass bubble sort #fail n is small i guess

permalink

#terraform is now a package manager. Great. #fail. For some reason plugin downloads hang if the download gets an IPV6 edge node in their CDN.

permalink

thinking of removing dependency on #cloudflare. currently only used for hyperbo.la DNS. Email records are the scary part. #terraform makes this mostly easy

permalink

I was first introduced to closures in summer '08 via #ruby blocks. This was probably the most magical moment I've ever experienced programming. That code was just another object you could pass around was amazing. Lisp would've been mind blowing. #history

permalink

when reading from stdin, ansible-vault encrypt_string does not trim trailing newlines, forcing one to use echo -n. #ansible knows this is a sharp edge and documents it, but they should do the right thing and make it consistent with password files #fail

permalink

in the quest to thin out the top-level directory sprawl in hyperbola, I got rid of .secrets. required rotating ansible vault password and rekeying everything because my password had single quotes, double quotes, and backticks (of course) #fail

permalink

#fail do not try to install ansible in a virtualenv in packer. you need a compiler. you don't want a compiler

permalink

the hyperbola + hyperbola-tools binary star monorepos is a common failure state, so I am told #fail

permalink

I just realized that by merging hyperbola-tools into hyperbola and converting my ad hoc ruby scripts to python, I moved to a monorepo and standardized on a language

permalink

6. addendum: since I already had a deploy script, it was straightforward to translate it to #ansible tasks

permalink

6. problem: bespoke deployment steps. deploy and cron run as lopopolo. deploy needs sudo password. solution: #ansible deploy_helper module

permalink

5. problem: environment names are inconsistent across tooling and deployment. solution: hyperbola_environment #ansible variable which parameterizes everything

permalink

4. addendum: I used #LetsEncrypt before migrating to AWS with a combination of cron, systemd timers, and dehydrated. ACM is easier, less error prone, and set-it-and-forget-it #win

permalink

4. problem: https is hard. solution: ACM + #terraform + ALB + CloudFront

permalink

3. problem: manual, bespoke server configuration. solution: #ansible, #vagrant, #packer, prebaked AMIs. immutable infrastructure

permalink

2. django-backup.py sucks. problems: email based, backup size is limited, unconfirmed delivery, plaintext transmission. solution: move media to versioned S3 bucket, daily RDS snapshots, logical backup json to S3

permalink

1. problem: python not running the latest 2.7.x release. solution: pyenv for local development and xenial (python3.5) + ansible + (future) deadsnakes ppa

permalink

I found a page in my wiki called hyperbola suckage 2015. This was a forward-looking list of gripes to fix as I moved from hyperbola1 to a new host #history

permalink

unexpectedly memory: I had to spoof my mac address to that of my xbox 360 so that I could register it on MIT's network #history

permalink

DNS is flipped and propagated! Live in #aws! #win

permalink

#aurora was much less forgiving with unique constraints than mysql. The schema for one of my utf8mb4 varchars had a max length of 255 and a unique index. It appears mysql silently truncates, whereas aurora complained the index was > 767 bytes. ~ #win

permalink

One source of difficulty was getting django, s3, vpc endpoints, and security groups to work together to enable s3 access from my backends #fail

permalink

my #terraform life became much easier by using name_prefix instead of name. name and name_prefix parameters were never interpolated. Instead, use interpolation in tags. In practice this means config can change without rebuilding the world #win

permalink

I initially went with the unclustered variants of elasticache and rds. Once I wrapped my head around the topology, #redis cluster mode and #aurora were much easier to work with in #terraform

permalink

buliding the #aws infra took about 30 commits, two #terraform destroys, and two terraform code rewrites. some fun bits in the following posts

permalink

The migration from a bespoke-bootstrapped, manual (scripted) deploy process to an #ansible playbook took about 20 commits. I deployed to a local #vagrant box first

Photo for post 507.
permalink

And with post 506, I can now say that hyperbola is hosted on #aws #win

permalink

2am hot take: all systems problems are best solved with pointers and routing. aka dependency and service injection

permalink

The #webpack configuration, on the other hand, was not fun to get set up. I particularly struggled fighting with the gulp plugin, which pins webpack 1.x

permalink

#webpack is amazing. I added the time-elements web components to the lifestream and contact pages. Only took an hour. Before this would have been next to impossible. #frontend #win

permalink

Rebuilt everything and modulo some zombie deposed resources, everything is good again

permalink

Discovered I had duplicate stanzas in my tfvars file. Removed the extraneous one. Broke EVERYTHING. #fail The worst was my admin IAM account losing access in the midst of a terraform apply. destroyed module.network, module.iam, and module.hyperbola-wiki

permalink

Converted wiki from ELB to ALB this morning ... took a couple of hours. modified #terraform config and updated #ansible ... also converted from Let's Encrypt to ACM. https://github.com/hyperbola/hyperbola-tools/commit/23fb9a7 #win

permalink

💯💯💯💯💯 = 500 posts #win

permalink

related, using pip-tools to manage and pin python dependencies has removed uncertainty from my deploy process. Upgrade packages only when I intend to #win #django 1.11.3 upgrade went super smoothly in part due to this

permalink

this is why #AWS

permalink

2 minutes of #downtime for libc security upgrade https://www.ubuntu.com/usn/usn-3323-1/

permalink

OMG just rewrote my #git PS1 which runs as part of my PROMPT_COMMAND. Now 20% faster both inside and outside a repo. terminal feels so much faster now #win #performance https://github.com/lopopolo/dotfiles/commit/2d5054c4ce

permalink

tried copying a 2TB Time Machine volume at the file system level using Finder drag and drop. #fail sadtimes. Was going on 6 hours and still resolving hard links. Currently attempting block-level copy with Disk Utility

permalink

I am living the dream with an n/ac-only network

permalink

Packer + Ansible + AWS + Terraform is really quite lovely #devops #win

permalink

I hope my next laptop has 32GB of RAM

permalink

I got a memory exhaustion pop up on my MBP earlier. I blame spotify, chrome, and the 2 VMs I carelessly had running. #fail

permalink

Migrated terraform state from a private github repo to a private, encrypted S3 bucket. State infra is bulkheaded from main app and protected with prevent_destroy lifecycle #win #terraform #aws

permalink

and the #monitoring even triggered when I took the box down. #win

permalink

All done. Downgraded from 8GB of RAM to a more reasonable 2GB of RAM. Went from 4 cores down to one though 😱 #devops #maintenance

Photo for post 487.
permalink

It looks like I am a *tad* overprovisioned. brb taking some #downtime to downsize my VM

permalink

Also, I have a google analytics snippet on the site. I have never once checked the analytics.

permalink

Moving the wiki to #AWS was a different beast. Everything is self-contained in the AMI.

permalink

sure an ELB + managed MySQL & redis + redundant frontends are "better" but I'm not sure I need the complexity

permalink

Not sure if it is worth the effort to migrate hyperbola to #AWS. Maintenance burden is pretty low. Log in maybe once a month, run an apt upgrade, and do a fresh deploy. Takes less time than a packer/terraform cycle would. #maintenance

permalink

Code that is organized in pipelines is easier to grok than code that is organized in trees #functionalprogramming

permalink

Upgraded to #django 1.11 ... woo LTS

permalink

Tested a #django release candidate (1.11RC1) for the first time. Site continues to work. No deprecation warnings. #win

permalink

Software shouldn't be "delightful." It should be invisible.

permalink

I would not be surprised if #AWS VPC networks were all running some franken overlay protocol

permalink

Today I learned about octopus merges #git https://marc.info/?l=linux-kernel&m=139033182525831

permalink

ELB for wiki does a basic healthcheck of nginx, but a backend can be down while the ELB thinks the node is healthy #monitoring

permalink

Added liveness monitoring of healthz endpoints on wiki and hyperbo.la today #monitoring #devops #win healthz goes all the way through to rack/django

permalink

Media backups have been failing since I moved the site to the new server. I've been uploading empty, corrupted tarballs. #fail fix here: https://github.com/hyperbola/hyperbola/commit/859128 Thank you gitlab & national check your backups day

permalink

Immutable infrastructure is so much better than mutable infrastructure. Much more satisfying to build a new image and redeploy than to upgrade nginx and mysql in place, taking #downtime.

permalink

https://www.youtube.com/watch?v=oyLBGkS5ICk

Photo for post 471.
permalink

Found this in the nginx logs of my wiki in ec2. hackers. #security

permalink

#django feature request and #patch accepted https://code.djangoproject.com/ticket/27541

permalink

Just freed up 350MB in my gmail by purging 35 emails. Did a search for size:5000000. removed emails with mp3s as attachments and event announcements with large images and pdfs #win

permalink

bastion is now in an ASG with an automatically bound (with user data) elastic IP. Yay fault-tolerant infra! #win #aws #terraform

permalink

Hit the 5 duplicate certs per week rate limit from let's encrypt by repeatedly rebuilding my vagrant box #fail

permalink

Yay refactoring! #python #django #win

permalink

Switched to building an ami with packer. Now an ASG roll takes 2 minutes #win #aws

permalink

Deployed hyperbola-wiki. Initial approach was to provision on boot with a user data script. This caused instances to take 15 minutes to come up (apt-get upgrade, compiling ruby) #fail

permalink

Provisioned a VPC, subnets in 3 AZs, and a bastion host #aws

permalink

Created an #aws account tonight

permalink

hyperbola-wiki is now fully deployed using #ansible. #vagrant made testing easy. #win

permalink

Added middleware to hyperbola app and wiki to inject a comment with the canonical hostname of the host that rendered the request. Debugging #win

permalink

Now have proper retina image support on lifestream and contact page. Generating @2x and @3x images. The flexibility of django-imagekit helped. This is a well-designed library. #django #ux

permalink

hyperbola, now with properly localized dates using window.Intl #win

permalink

Fourth most expensive query is selecting all lifestream items and pictures: a LEFT OUTER JOIN with ORDER BY DESC, LIMIT, and OFFSET. There is an index on the order by but MySQL prefers filesort. Table is only 500 rows though. #mysql

permalink

Most expensive queries on hyperbola are hashtag lookups because they use RLIKE. Top 3 queries are 4.3ms, 3.2ms, 2.1ms per query. #mysql

permalink

mucking around in settings.py I ended up doing a top-level import from debug_toolbar. Yay for staging. #fail https://github.com/hyperbola/hyperbola/commit/26c4e1d

permalink

diff of yesterday's backups and current backup confirm all that has been altered are filenames

permalink

I didn't notice this in dev because the front page of the lifestream is 20 of the same picture of Kendrick Lamar #fail

permalink

Fixing this required parsing a db dump (django fixtures much easier to read than mysqldump output) and manually running UPDATE queries in MySQL

permalink

Minor crisis when running the mangle media script. Resume and lifestream models have auto_now=True (an updated timestamp) and I have been using them as creation timestamps. Apparently there is an auto_now_add param I should use. #fail

permalink

just automated a spreadsheet (concatenating several columns) with filter #win filter(L6:L, L6:L<>"") vs manually updating ranges #automation

Photo for post 449.
permalink

DOMContentLoaded in less than 200ms on lifestream index #performance #cdn

permalink

Enabled cloudflare cdn on www.hyperbolacdn.com today. Took 4 clicks. #win

permalink

Some changes since moving to the new host: assets and user uploads served off of www.hyperbolacdn.com; TLS; frontend deps managed with bower

permalink

hyperbo.la staging and production are TLS enabled. Still have to turn on HSTS. #win #letsencrypt

permalink

❤️ Emoji: 🔥🔥🔥

permalink

Successfully migrated hyperbo.la and staging.hyperbo.la to a new host, hyperbola3. Ubuntu 16.04 and 8GB of RAM. Just shut down hyperbola1. #win

permalink

Did my first #aws prod deploy today #win

permalink

reifying #aws infrastructure with terraform is pretty cool. power is scary though. yay for sandboxes!

permalink

I need to make these out of band releases more painful so I migrate hyperbola to my new host #fail

permalink

I started a new job a couple weeks ago. I'm working at #Citadel.

permalink

#throwback #cruft just cleared out MIT client certs from 4 years ago that were hanging around in my keychain

permalink

Also found the samba password to the file server named supersecret that I built

permalink

All this discovered while clearing out my OS X Keychain

permalink

That was when hyperbo.la ran on a reclaimed Dell workstation in a network closet at MIT

permalink

I think I also used to host a redmine instance on dev.hyperbo.la back in the day ~2010

permalink

#throwback http://dev.hyperbo.la:80 hyperbo.la subversion repositories

permalink

successfully did an out of band release not of master while in the process of migrating hyperbola to a new host #win #deployment #automation

permalink

Forgot to blog about this: My printer ran out of toner a week before my move. Had the printer since 2010. RIP printer

permalink

With systemd I am one step closer to #12factor app bliss since all service logs get logged to syslog #win

permalink

made the #nginx config a little better, converted thin to bind to a domain socket, and deployed with a dedicated app user with deploy keys #win

permalink

In the process, I've added #TLS with Let's Encrypt #win

permalink

The past few days have been spent migrating my #gollum wiki from hyperbola1 (old server) to hyperbola2 (new server)

permalink

And a 4 years newer OS (Ubuntu 16.04 LTS) and 64-bit #win

permalink

Project new VPS is underway. For the same price as my current 1G box, I'm getting a 4G box with more cores #win

permalink

nginx upgrade broke the init script. had to reboot to fix things. gunicorn failed to start after reboot due to log file perms #fail

permalink

I love linters #js #css #python

permalink

hahahahaha #fail Daemons in macruby #ruby

permalink

I enjoy golfing my minified javascript. Just shaved 19 bytes off of the inlined date formatter js I include on the lifestream page #js #win

permalink

The only times I have ever run ant as a build tool were a couple of CS classes back at MIT—4 years ago. Even today, it remains my 8th most frequently used command on my mac

permalink

Filed a site issue JIRA on my last day. It was eventually closed as invalid. By the time this happened, though, I had lost systems access and couldn't discuss it further

permalink

Today was the day of linters. Added eslint, flake8 config, and isort to hyperbola

permalink

"we successfully avoided a split brain scenario due to master/master replication" #mysql #fail

permalink

hyperbola prod is now running #py3 what a time to be alive

permalink

#scala takes the #objc std lib approach to collections: One public interface/factory w/ many optimized implementations underneath

permalink

My ruby version on my mac stopped working because homebrew updated libicu. Had to recompile/upgrade #fail

permalink

Today I upgraded node to 4.3.2 (from 4.2.2), python to 3.5.1 (from 2.7.3), and django to 1.9.4 (from 1.8.8). What fun! #win

permalink

Hyperbola staging now running python 3.5.1 #py3

permalink

Someone forked my senior thesis ... what? https://github.com/nmoutana/sr

permalink

I use #node in my asset build pipeline. I compile js and css with yuglify. Today I productionized my node install. Upgraded to latest LTS and added docs for upgrading node

permalink

This new #django admin UI is BEAUTIFUL!

permalink

Finally got around to getting my #django 1.9 upgrade branch over the line. Didn't have django-pipeline pinned and 1.6 introduced a breaking change to settings. All done now.

permalink

Yelp's search stemming needs some work. I searched for "used" and it is highlighting "us" in the search results. #fail

permalink

When I see L7 I think of the band instead of the OSI Model #fail

permalink

<ENTER>~.

permalink

Reading about the LMAX Disruptor https://martinfowler.com/articles/lmax.html ... pretty cool. I've seen similar patterns (at lower optimization levels) at Box.

permalink

Caught an issue in staging today because staging is more like prod than dev is. Dev has pipeline disabled. Yay for release process. Yay for testing. #win

permalink

python -Wall is a neat trick. Cleaned up a bunch of #django deprecation warnings

permalink

Time to switch to using the #python SDK! In the meantime, no new media uploads

permalink

Media directory backups have been failing since the 17th due to exceeding max attachment size. No alerts. #fail >>> SIZE=55759470 <<< 552 5.3.4 Message size exceeds fixed limit

permalink

Switching from yui-compressor to yuglify saved me 1KB on bootstrap.js and 30 bytes on bootstrap+hyperbola css #performance

permalink

With this latest release, hyperbola has surpassed 500 commits! #win https://github.com/hyperbola/hyperbola/tree/2015-11-21-lifestream-template-refactor

permalink

lifestream template refactor didn't yield any #performance wins, but the code is much cleaner now. #win

permalink

Front page warm nginx latency is 12ms; contact page is 20ms #performance

Photo for post 396.
permalink

Current prod gunicorn time (measured at nginx) for the lifestream index page is 70ms with no partial caching and 40ms with the sidebar cached #performance

permalink

Tested rendering the lifestream index with all 3 interpreters. pypy is slower than python2.7 by 2x. python3.5 is faster than python2.7 by 2x. #performance

permalink

Local development lets me run cool experiments. I have python2.7, python3.5, and pypy virtualenvs.

permalink

Sped up the lifestream page by 30ms by caching the archive sidebar. More template speedups to be had by switching to python3 (str to unicode coercion in python2) #win #performance

permalink

Woo! I can run hyperbola on my mac. Local development! #win

permalink

Even better: precompiling app/hyperbola to pyc and getting to keep the deployment read-only. better #win

permalink

gunicorn workers get warm quicker now that I've allowed gunicorn write access to a deployment's app and virtualenv dirs #win

permalink

Root fs usage is down to 3.5G.

permalink

Finally started cleaning old deployments. After each deploy, my deploy script removes all but the latest 5 deployments for a given env. https://hyperbo.la/lifestream/352/ can rest peacefully.

permalink

90mbps upload to Box from the corporate network is pretty nice when backing up my photos (all 25GB of them).

permalink

OG status: https://github.com/hyperbola/hyperbola/tree/og-hyperbola-v1

Photo for post 385.
permalink

Did some archeology today to unearth an ancient version of hyperbo.la. Same color scheme as today. Yes, there was an assets domain. Had to do forensic analysis to reconstruct the index.html.

permalink

It continually amazes me at how much conflict is solved with "assume good intent."

permalink

Added a post-commit hook to my gollum wiki today to do a pull/push. Changes instantly synced to #github. No more stupid 5-minutely cron. #win

permalink

I feel like my #django site is futureproofed for at least the next two minor version upgrades now. Just converted a ton of render_to_response snippets to the new render hotness. Even feels faster.

permalink

Despite being stable, #django deprecates frequently. What they deprecate tends to have been the recommended way of doing it circa 1.4 ... feels like they are slowly rewriting everything.

permalink

15 commit night on hyperbola, 5 on hyperbola-tools, and 12 on my dotfiles. Mostly code and style cleanups, some correctness fixes.

permalink

Having a linter is so awesome. IntelliJ rocks. #win

permalink

Next major project is migrating to a new VM. 32-bit Ubuntu 12.04 is getting long in the tooth. I have eyes on 64-bit #Debian 8 and a shiny new #linode with 2GB of RAM

permalink

Last bits of major feature work were bootstrapifying several pages and adding automatic deployment (June 2014)

permalink

Just realized that hyperbo.la has been in maintenance mode for a while. Last few projects: upgrade to django 1.8, upgrade to django 1.7, upgrade vendored dependencies (bootstrap, retinajs).

permalink

#puppet is more fun when you know what you're doing.

permalink

I bought IntelliJ Ultimate and it has already been decently valuable: https://github.com/hyperbola/hyperbola/pull/28 ... fixing lots of warnings #win

permalink

It only took a debian VM and 4 hours of my time to reformat two new USB keys. #fail

permalink

Fixed a long outstanding bug in how the sidebar on the lifestream calculated monthly post counts #win Tickled by the upgrade to #django 1.8 which caused them to all display as 1. #fail

permalink

Finally upgraded to #django 1.8. Only an 8 commit adventure

permalink

Wondering why the response to some external asks isn't, "No, we're not ready yet."

permalink

Wow! usability fail. The crontab command operates in 3 modes, two of which are edit (-e) and remove (-r). Lovely that e and r are right next to each other on a QWERTY keyboard. #fail

permalink

This oncall cycle, I observed my first real hardware failure. A machine critical to the deployment pipeline spontaneously shutdown and failed to reboot, problems with mounting RAID, etc. #fail

permalink

We are loading the racks onto trucks and moving them to a real datacenter. Bye bye servers.

permalink

Responding "maybe" to a calendar invite is the equivalent of the SF "we should do lunch"

permalink

I want to live in a world where booting a machine doesn't cause config propagation to break #fail ghetto consensus algorithms

permalink

The only things that are truly unique in a relational DB are auto inc pks. Even uniques such as username may be duplicated across time.

permalink

I feel dirty doing pointer dereferences in #golang

permalink

One of the tools I've learned when writing #golang is how to coalesce errors behind an API boundary. Makes for cleaner code.

permalink

./unixtools gzcat ~/Desktop/f.gz.gz | ./unixtools gzip -d - ... having fun implementing parts of busybox using #golang. So far I've got gzcat, gzip, and yes

Photo for post 359.
permalink

#quora ahh yes ... database systems

Photo for post 358.
permalink

Services team get rekt #oncall pages from 9pm Friday to 8am Saturday ... something about the corp dc migration

permalink

So this is what a real oncall rotation feels like. So many things.

permalink

whoaaaaaaa ... an unsaved txt file from my work laptop showed up on my personal laptop. #cloud

permalink

This week at work I settled on a secret delivery mechanism that bootstraps with #kubernetes (or puppet if a service isn't containerized)

permalink

Just spent 5 minutes hopping through man pages trying to grok a script I wrote a while ago only to realize there were comments explaining exactly what was going on #fail #docs

permalink

took 1minute of downtime today for a #mysql 5.5 upgrade from ubuntu

permalink

still hanging around: 4GB of old staging installs and 2GB of old prod installs

permalink

Just did some disk usage cleanup on hyperbola. Had 4 old ruby installs hanging out (2GB), a 700MB gunicorn logfile that was being written to every second, and a very full apt cache (1GB)

permalink

even with a staging env, deploys still require a site check

permalink

I love Apple's "quit unexpectedly" euphemism for crash

permalink

... and I'm no longer dumping secrets to stdout due to `source` and `set -x` #12factor #fail

permalink

On the plus side my build script now dumps metadata about installed python packages to disk so I have one place to look instead of wondering

permalink

Finally subscribed to django-announce. I was running a super old version of 1.7 ... oops #django

permalink

Recently restructured my #dotfiles and added a Makefile so they don't live in $HOME (git clean terrifies me) #git

permalink

Playing with some JSON data and jQuery ... created a rudimentary VX points trip planner #javascript

permalink

I don't like structured identifiers because they presume data fits into a hierarchy

permalink

Borg paper was a fun read. Kubernetes is really exciting.

permalink

feast or famine on the hyperbola microblog

permalink

TIL: Selecting 2 cells and then dragging to repeat causes google sheets to infer the increment #win

permalink

#ruby Fixnum is a weird beast. A class that doesn't have new.

permalink

chrome has #emoji now. such web. #win

permalink

A good chunk of my qualms about #scala is complexity resulting from laziness tricks.

permalink

sudo lsof | grep libssl | cut -d" " -f1 | sort | uniq -c | sort -rn #fail https://mta.openssl.org/pipermail/openssl-announce/2015-March/000020.html

permalink

Healed my first master-master data drift today #devops #mysql

permalink

#bug fixed https://github.com/hyperbola/hyperbola/commit/01b34d with minor staging break caused by django-pipeline renaming their templatetags #django

permalink

#bug: page links do not appear on #hashtag pages

permalink

Upcoming Xen vuln was the forcing function to get my gunicorn upstart scripts able to work after a reboot. There is some nasty in them https://github.com/hyperbola/hyperbola-tools/commit/8e5c4 #bash

Photo for post 330.
permalink

This is how you get a macbook to page like crazy #fail

permalink

OMG #ruby is using 19.22GB of memory executing a one liner #fail

permalink

Ran time ruby -e 'def is_prime n; ("1" * n) !~ /^1?$|^(11+?)\1+$/; end; is_prime 982_451_653' #fail keyboard events don't fire, gfx glitches, windows don't repaint, one core pegged

permalink

Just spent the last 25 minutes installing homebrew packages on my work machine. My code is literally compiling. Took the time to do some scooter laps.

permalink

Rebooted hyperbola for the first time in a year due to the libc vuln. Minor panic ensued. Gunicorn failed to start due to log file perms, /var/run/gunicorn was wiped and had to be recreated #fail

permalink

I've been missing #ruby lately. I should make something.

permalink

(function _() {console.log('('+_+')()')})()

permalink

Answered a phone call on my computer for the first time. That was so cool. #win

Photo for post 322.
permalink

I have an album with a non-square album art in #iTunes. The UI renders it 3 different ways #fail

permalink

Upgrade to Bootstrap 3.3.0 went really smoothly. Everything just worked #win

permalink

#hyperbola is down to 200 lines of #CSS, half of which is used to render the logo and navbar

permalink

Finally bootstrap-ified the last bits of #hyperbola: the frontpage and the lifestream entry panel #win

permalink

Now I have #utf8 support in #MySQL. I thought I did https://hyperbo.la/lifestream/174/ but that was a #fail. Bitten by the difference between utf8 and utf8mb4. Alas, #Django 500's; no emoji for me

permalink

I still don't have zero-downtime #MySQL upgrades. Lost a nine #fail

permalink

Just added CORS headers for web-fonts on my 2 assets domains through cloudflare ... whoa. This has apparently been broken for a while #fail #win #hyperbola

permalink

I do not understand Box Engineering's blanket aversion to a build step. So many generated artifacts checked in. #fail

permalink

Started going through the Functional Programming in Scala book today #scala

permalink

why does any C program of sufficient complexity have its own allocator? Just use jemalloc?

permalink

TIL #nagios flips out when you reboot a host #devops

permalink

The admin site is slightly modernized in #django 1.7 ... looks nice #win

permalink

hyperbola, now with #django 1.7 ... I love how easy it has gotten to deploy this site.

permalink

Today I exceeded linux's max shebang length #fail it is 128 btw

permalink

I now understand the difference between teasing apart complexity versus simply moving it around

permalink

thinking of writing a toy in #rust or #golang

permalink

swift is the #scala to objective-c's java ... that switch statement looks an awful lot like scala's match construct

permalink

TIL that two trailing spaces + line break = br in the same p #markdown #win

permalink

lol at the #nginx empty_gif directive

permalink

heh, spoke too soon ... #fail Had forgotten to update #nginx config so when I deleted the legacy deployments, I lost static assets and media.

permalink

live site was just deployed automatically for the first time. Switched from legacy deployment to automated with no downtime! #win

permalink

Just did 2 fully automated deployments to staging! #win #automation

permalink

just did a dry run of a mostly automated hyperbola deployment. virtualenv, bash, git what fun :) Already staging and production have diverged. #fail next is to figure out how to bring it live

permalink

Oh no! just missed a deploy step #fail ... forgot to run pip install --upgrade ... new code was missing a dep

permalink

Woo! First 2 hyperbola releases with #git tags today #win. Also local development. dev + ops = #devops

permalink

PyCharm is neat. #python #django I found the code analysis to be useful.

permalink

Listened to a pretty intense #scala rant today: https://www.youtube.com/watch?v=uiJycy6dFSQ ... Highlights some of my gripes about the language, most notable implementation details leaking everywhere

permalink

Nothing like a little bit of #regex golf on a Saturday evening.

permalink

I just realized that when I first learned #java, there were no generics. Thanks Martin Odersky! #scala forever.

Photo for post 284.
permalink

Killed it at #Box Dev today #metadata

permalink

60ms server-side latency on the lifestream page now #performance

permalink

"It's Not Right but It's Okay" -- A poem by Whitney Houston about continually evolving software projects.

permalink

Today I learned more about sockets and ephemeral port exhaustion than I ever wanted to know. I normally take TCP as a given. #webscale #box #metadata #fail

permalink

Next #devops steps for hyperbola: Rebuild machine from scratch on 64-bit kernel; use #docker and dockerize all services: nginx, staging django, prod django, MySQL; real deploy step; #git tag releases

permalink

#ruby has object tainting and security levels. So cool! https://ruby-doc.org/core-2.1.1/Object.html#method-i-taint

permalink

#django sure as hell doesn't make it easy to get at the database backend ... but the fact that I can is still awesome. Enabled post count badges on lifestream.

permalink

Hyperbola, now with retina graphics! The logo and feed icons will come through as retina.

permalink

Lack of agency is the thing I fear most as an engineer. It's something I need to get better at dealing with.

permalink

Ways you can tell we have a major release coming up: Being in the office until 10:30 and 9:30 on consecutive nights.

permalink

Upgraded to Mavericks ... relinked all homebrew formulae like this: https://gist.github.com/lopopolo/9427762

permalink

Did my first service deployment today. 6 nodes in 3 datacenters #box #metadata #devops

permalink

Open-source #patch: https://github.com/fge/json-patch/pull/7

permalink

it is a rare day that I don't get home and try to type my work password into my laptop #fail

permalink

#metadata about to enter the War Room for 2 months ... #Box /dev brings deliverables. Let's go!

permalink

next project: convert #hyperbola to UTC, server, #django, and #mysql

permalink

I have no idea why, but every time my backup disk spins up on my laptop, my entire system hangs for a few seconds #fail

permalink

got another big #performance win by not minifying HTML in python not doing so shaved ~100ms off response time which means my server was spending more than 100ms of CPU time for the pleasure #fail

permalink

about ~20% of server time for lifestream is spent reversing URLs for hashtags :/ #performance #fail

permalink

Latency at nginx for hyperbola: frontpage: 25ms, contact: 40ms, lifestream: 200ms #performance

permalink

Switching gunicorn to a domain socket instead of a TCP port sped up the site by 2x #performance

permalink

Hyperbola, now with more html5-boilerplate and bootstrap ... it made CSS so much easier than I normally find it to be ... and the site is now mostly responsive and looks good on mobile

permalink

Spent the last week optimizing, re-styling, and significantly refactoring hyperbola. The lifestream page loads ~300ms faster and DOMReady is ~600ms faster #performance

permalink

Get optimized, HTML! Just added some middleware that minifies Django template HTML output #performance Google pagespeed really likes my site now (on Desktops) ... 94/100 #win

permalink

My wiki is getting filled up with #devops documentation for #hyperbola #win Runbooks are fun!

permalink

The last bits of the #hyperbola deployment are gone from my home directory ... finally productionized a cron to do a #database and media backup #devops

permalink

Most interesting learning from 2013: when it is appropriate to resist my engineer's temptation to build the perfect thing. #lean #failfast #box #metadata

permalink

I'm getting ad targeted for foundationDB on YouTube during my vacation ... are we using it yet? #box #metadata #database

permalink

Needed: #mobile optimized #django admin skin. Posting from my iPhone is less than ideal

permalink

https://howfuckedismydatabase.com/ hehe database problems. #fail #scale

permalink

wow link tags and auto RSS/Atom feed discovery ... welcome to the internet Ryan ... #fail ... I feel like #django is missing a template tag here.

permalink

#nginx reverse proxying to django with HTTP/1.1 ... wish this config option would be the default.

permalink

#PHP is just netcat ... the language of the web does not have a good HTTP abstraction. I'm envious of golang's HTTP package

permalink

woo! #hyperbola is now #pep8 and #flake8 compliant! Woooo linting! #python

permalink

And with about 30 minutes of downtime I've now got 2x the ram #devops

permalink

running alter tables in production is terrifying #mysql ... just converted from MyISAM to InnoDB and added a bunch of #indexes

permalink

#django 1.6 upgrade was a breeze. Just a single deprecation warning. #win ... took the opportunity to refactor some old code too :)

permalink

so my resume link was broken for six months #fail stupid missing line in #nginx conf ... not enough exploratory testing during the refactor. What good is #staging for anyway?!

Photo for post 246.
permalink

I reported this #github #outage #fail

permalink

Actors are overrated. So far, we've preferred to use future literals and ExecutorServices instead #scala #box #metadata

permalink

Finally added an upstart script for my gollum wiki #linux #ops One less thing to worry about when dealing with a reboot or upgrade

permalink

Only minor drama when upgrading hyperbola to 12.04 LTS ... #mysql failed to cleanly upgrade to 5.5 and I forgot the correct URL for my staging environment #linux #fail

permalink

Part of pushing #metadata to production meant we got to puppet our service across dev/staging/perfpod/production. #puppet is pretty cool #box #devops

permalink

6 months in the making, #box metadata made it to production today

permalink

The bare minimum I need on a server to feel comfortable is `set -o vi` #vim #bash

Photo for post 239.
permalink

On stage at #BoxWorks ... metadata launched #Box #gocloud

permalink

Now it's time to figure out what part of the stack is preventing #concurrency from exceeding 100 connections #jetty #linux

permalink

Throughput on the #box #metadata backend is limited by the rate at which our load driver can make requests. 2ms latency at the client. This thing is #fast #performance

permalink

Won best performance hack at the #Box #Hackathon last night #win

permalink

Rule number 1 of operating at #scale: Your assumptions are never universally true.

permalink

Woo: hyperbola, now with thumbnailing #performance

Photo for post 232.
permalink

Hello #nginx. Goodbye #apache

permalink

Upgraded #django from 1.4 alpha to 1.5.x stable tonight. It only took a couple hours. Mostly dealt with deprecated APIs and new template tag syntax

permalink

One of PHP's biggest problems is that it is too easy to create associative arrays. It discourages people from making proper objects. #php

permalink

mac #safari can attach to webviews running in the iOS simulator and open a web inspector. This is cool but ... what? #debugging

permalink

#databases Two things that amaze me: (1) idempotent writes, (2) idempotent and reorderable writes

permalink

git commit -m 'removed 2 chainz' #git #win

permalink

I am super excited to see the results of the move to the T3 #javascript architecture at #box

permalink

The Fundamental Theorem of Software Engineering: "We can solve any problem by introducing an extra level of indirection."

permalink

It makes me happy when I see people using If-None-Match sprinkled throughout the splunk logs at #box #win

permalink

currently having a real bad time importing ffmpeg into xcode #fail #indexing

permalink

just installed 500MB of software for a scanner #wtf #fail

permalink

I remember a few years ago when my primary reason for staying on #firefox vs #Chrome was that I preferred how Firefox highlighted text on selection. #fail

permalink

I have no idea why, but since I've been working at #box, I don't #git pull. I always fetch and merge --ff-only or rebase. Only time I ever use pull is via pull --rebase if I get ff-rejected on push

permalink

Participated in my first 24 hour hackathon last Thursday. It was awesome. Highlights: food trucks, caffeine, kreayshawn, and awesome hacks #box

permalink

Just forked, edited code, submitted a pull request, and got it merged all from the browser #win #git #github

permalink

#itunes 11 is beautiful

permalink

I had a cronjob launching every minute that updated remotes on about 40 #git repos. I've apparently had ~50 git processes running all the time for about a year now. #fail

permalink

I can't believe I ever liked #flex. To be fair though, #xcode and interface builder are basically the same thing

permalink

Running unit tests is the equivalent of waiting for code to compile.

permalink

Just updated the firmware on my Air. First two reboots = #kernelpanic. 3rd reboot = no mouse or keyboard. 4th reboot, everything works? #fail

permalink

9 months since the last reboot isn't too shabby. And now I have a 3.5 kernel!

permalink

#metaprogramming is awesome. In a new project, adding an API endpoint is as simple as defining a single constant

permalink

Changed my terminal and vim fontsize from 14pt to 24pt at work. zOMG so much better

permalink

alias jsonpp='python -mjson.tool' greatest alias ever for #json #api development

permalink

So I have some autocmds in my vimrc already, but tonight I wrote my own. It detects macruby in the shebang of a script and sets a var to tell syntastic to load a different syntax checker #vim

permalink

Just added #solarized and a #vim plugin (vrapper) to #eclipse. zOMG this is so awesome

permalink

#java gets #functional. Hooray chainable methods! https://datumedge.blogspot.com/2012/06/java-8-lambdas.html

permalink

Gollum finally updates! Woo! #ruby #rubygems

Photo for post 199.
permalink

#versioncontrol, or the lack thereof. This was my first major software project. I was a junior in high school. I was making a symbolic math thing. #fail or the reasons any CS class should teach #git

permalink

#Songza's use of high-res album art alone makes the experience feel better than #Pandora. Add in the fun way of selecting playlists and we have a winner

permalink

http://songza.com Not only is it awesome, but logging in doesn't stop my music from playing. sweet #ux

permalink

My wiki was spammed and turned into a link farm for discount pharmaceuticals. Good thing its backed by #git -- fixing it was trivial

permalink

google docs for #LaTeX https://www.sharelatex.com/

permalink

#notetoself: Next time you need to run more than 3 tasks that take 10 minutes or more to run, invest in figuring out how to script it #fail #thesis2012

permalink

A watched pot never boils. Neither does a long running task when you're not tailing the log file. #thesis2012

Photo for post 192.
permalink

Running experiments. It's been 9 hours now. #thesis2012

permalink

ssh and an html/js terminal that will run anywhere I have chrome because of native client? Yes please #win #awesome https://chrome.google.com/webstore/detail/pnhechapfaindjhompbnflcldabbghjo/details

permalink

T-minus 2.5 weeks #thesis2012 OMG I HAVE 70 EXPERIMENTS TO RUN,

permalink

screen wipes are the best things ever for both laptop screens and glasses #win #thingsineedtobuy

Photo for post 188.
permalink

I wanna give a big shoutout to the #java jvisualvm, without whom figuring out my perf problems would have been much harder

permalink

#lessonlearned: Don't create a billion Timer objects. One is probably enough. Managed to get #eclipse to fail at repainting itself because it was running about 700 threads #java #fail

permalink

I feel like serializing procs to strings, passing them over HTTP and evaling them is an abuse of #ruby, but it is awesome that I can #win

permalink

I do not care whether your API is RESTful. Does it work and will it do what I want? #marketing #fail

permalink

I remember when I first upgraded from notepad to scite when I first installed #ruby. You had to modify this conf file to get monospace all the time ... #vim is so much better #nostalgia

permalink

all your heapspace are belong to me! #java #outofmemory #fail

permalink

The number of posts on #hackernews about people reinventing/whining about #git never ceases to amaze me #fanboy

permalink

https://github.com/lopopolo/dotfiles/blob/master/scripts/cron-update-remotes.bash All of my problems with this script had to do with adding passphrases to my #ssh keys #fail

permalink

just discovered that once you're in cmd+TAB mode on OS X, cmd+` cycles backward. So much easier than cmd+shift+TAB #learnability? #fail #win

permalink

#ssh public key auth setup on hyperbola #womp #programmerresolutions

permalink

#golfed OR "why use ruby -e when there's uniq -c." It's 40 chars shorter than before: cat ~/.bash_history | cut -d" " -f1 | sort | uniq -c | sort -rn | head

permalink

Is it weird that my web server backups run at 4:04 AM? #lolz

permalink

I just saw a commercial for #IE9 with a dubstep music track. Um ... #browserwars?

permalink

You mean I have to implement hashCode()? I guess that means I'm doing some string concatenation #easyway #java

permalink

I don't ♥ manually managing character sets #fail Why doesn't #django ensure the #mysql db has #utf8 as its character set?

Photo for post 173.
permalink

Cute ad on stackoverflow. It reads "<heart> Your Job" #adtargeted (apparently my db doesn't like unicode #fail)

permalink

Today I learned that #vim and #emacs treat newlines as line terminators. This means files written by them always end in a newline. #learnsomethingnew

permalink

That feeling when you go to empty your trash and you realize that there were over 14k files in it that account for 15GB of disk space. HOW DID THIS HAPPEN? #fail

permalink

So now that I got #airdrop working, I like it. It doesn't handle large transfers so well though. Transferring a gig of #music files has taken 15 minutes so far and I'm only 2/3 done. #fail and #win

permalink

Linus Torvalds is a boss http://harmful.cat-v.org/software/c++/linus #c

permalink

I'd post about how #adtargeted I feel every time I see a New Relic ad, but it'd only make it worse

permalink

I hate writing unit tests #junit #java #dbclass

permalink

Math m = null; m.sin(0.5); is valid #java and computes the sin of 0.5. #learnsomethingnew Type resolution is cool.

permalink

the newest xcode download slimmed down a lot. less than 1.5 gigs #thingsthataregood

permalink

Laptop is using full disk encryption and #hyperbola now backs up the db and all of its media to dropbox #swag #programmerresolutions

permalink

Process for rendering pset: Type in vim using markdown > render to html and open in chrome > print to pdf #doingitthehardway

permalink

After a struggle to import data from Firefox, I've switched to Chrome #thingsthatarenew The only extensions I'm running are 3 greasemonkey scripts I can't live w/o

permalink

I've never had to implement a hash table before #spoiled by standard libraries

permalink

a productive day: did something I've been meaning to do for a while: a playlist queue for iTunes #ruby

permalink

A productive day: made my PS1 better and fixed a bug in the hashtag regex on hyperbola (again)

permalink

Something something something #git ... something something something complete

permalink

Nokogiri::XML(file_bigger_than_1GB.xml) is a bad idea #fail

permalink

ok so setting up 4 VMs all at once was a pain in the ass sr-master.xvm.mit.edu sr-worker-1.xvm.mit.edu sr-worker-2.xvm.mit.edu sr-worker-3.xvm.mit.edu #thesis2012

permalink

Thank goodness xml compresses so well

permalink

I dislike experiencing bugs in my day-to-day usage of software I paid for. Saints Row: The Third and iPhoto, I'm looking at you #fail

permalink

cat ~/.*h_h*|cut -d" " -f1|ruby -e'h=Hash.new 0;h[$_]+=1 while gets;h.each{|k,v|puts"#{v} "+k}'|sort -rn|head #golfed This snippet prints out my most frequently used commands

permalink

#vim modelines are the best. Just added filetype modelines to all of my wonkily named #bash config files. Syntax highlighting rules

permalink

I don't understand why people think they need build scripts for their dotfiles. #fail Just keep your $HOME in #git: git init && git remote add ...

permalink

This is the longest-lived and most consistently updated #blog I've ever had. Averaged a post every 2.5 days. I still post to #twitter more often. #socialmedia

Photo for post 146.
permalink

My most frequently used commands on my laptop

permalink

Just exported 1GB of article dumps from #wikipedia #thesis2012 is on

Photo for post 142.
permalink

you can see the xcode in my eyes ... #allidoiscode

permalink

Switched to solarized color scheme in terminal and #vim. Everything looks different, I love it though. I'll get used to it. Also, Inconsolata rocks.

permalink

lol my site looks like this http://twitter.github.com/bootstrap/examples/container-app.html #bootstrap

permalink

^(void) { NSLog(@"Closures are cool"); } https://spicedcocoa.wordpress.com/2010/06/16/working-with-blocks-in-objective-c/

permalink

history of version control. An entertaining read https://www.flourish.org/2011/12/astonishments-ten-in-the-history-of-version-control/ woo #git

permalink

so after hacking around in #objc all day, the syntax is starting to make sense. I actually kinda like it. #macruby is so awesome though.

permalink

https://johnkary.net/git-1-7-8-changes-for-the-everyday-developer/ #git grep is pretty cool, but I think i'll stick to #ack

permalink

sweet, there's been a patch since may, but no need to put out a bugfix release or anything https://github.com/github/gollum/issues/147 #fail #git

permalink

#firefox dims the favicons of inactive tabs. That's so nice.

permalink

printed over 600 pages in an athena cluster today #win

permalink

#hack to fix someone else's #PHP #fail foreach ($_POST as $key => $value) { $_POST[$key] = mysql_real_escape_string($value); }

permalink

what I've been working on: sandboxing #chrome extensions

permalink

apache, quit installing sites into my sites-enabled directory. You're taking my site offline #fail

Photo for post 126.
permalink

Best commit message ever #git #win

permalink

a CSS rule to make images sane in fluid gmail: html > body > img { max-width: 100%; max-height: 100%; }

permalink

I have iTunes autoname my files based of their tags. Apparently if a track starts with "(", it won't prepend the track number to the file name. #fail

permalink

#music curating has reached a new milestone. All albums in iTunes are complete. Next, get rid of all those 128 kbps tracks.

permalink

* 4 * * * means every minute of the 4 o'clock hour #cron #fail

permalink

speaking of proxies ... hyperbo.la, now with 100% more reverse proxying (apache to gollum) #gemsanity

permalink

I first used Charles this past summer. A very valuable tool for network observation and debugging https://www.charlesproxy.com/

permalink

Google plus, you are no longer app tabified. slow #fail

permalink

Just did an ubuntu server upgrade from hardy #linux #fail

permalink

Why dynamic languages suck sometimes: I spent the last 45 minutes debugging a python script. I thought I had logic errors. What actually was wrong was that I renamed a variable and missed one. #fail

permalink

when you're at pid 400000, it's probably time for a reboot #unix

permalink

I don't know why, but I am excited about home automation today. Z-Wave looks really cool

permalink

I have 3 spare computers lying around. What should I do with them? One idea is to turn them into a cluster with one node running a shared kernel for the entire cluster. #rpc #linux

permalink

6 doesn't byte 00110110 00100001 00111101 01100010 01111001 01110100

permalink

#backintheday, before I understood how the relational part of MySQL worked, I made 20 columns to hold metadata about a record. 20 columns wasn't enough. Foreign keys #ftw. learning #fail

permalink

So on this one server, my default shell was /bin/sh, i'm not a sudoer, root owns my home directory, and another user owns /home. #linux #fail

permalink

This article is so cool https://en.wikipedia.org/wiki/List_of_company_name_etymologies #wikipedia

permalink

so in order to access Mail.app's preferences to make it not be the default mailto: handler, I first have to configure it #fail

permalink

Way to steal my MEng thesis idea Twitter https://github.com/nathanmarz/storm/wiki/Rationale

permalink

the goes to operator (x --> 0) makes #C feel more like #ruby! #fail

permalink

Seriously, Office? You rm the backup from the disk when I close it instead of doing the sane thing and moving it to my trash? #fail #hard

permalink

why does iChat think it's ok to grow as large as it wants vertically? #no, I'll manage my windows by myself.

permalink

#timeboxed this gcc cross compiler nonsense. Using debathena in a VM is easier. #hammer

permalink

#compiling #gcc from source to get an ELF cross-compiler on OSX. This is so #fail

permalink

its amazing how much quicker a script runs when you don't create 60,000 Hashes and compile a regex 30k times #ruby

permalink

Because kernel.org is down, I used airdrop to copy over a cached version of the git source and formula from another computer so I could install it with homebrew #fail

permalink

bookmarking for later: useful defaults for OS X apps. https://github.com/mathiasbynens/dotfiles/blob/master/.osx

permalink

#UPnP stopped working on my router. Flashed it to fix.

permalink

I got to 100 #tweets a hell of a lot faster than I did here (still not there yet)

permalink

So I used #github socially to review someone's code and got flamed. That's the last time I do that.

permalink

Abiding by the principle of least surprise is, to me at least, least surprising.

Photo for post 90.
permalink

I remember when I wanted one of these #sonymylo

permalink

here's the #plan: 63 units, 6 classes, 3 of which are grad H. It's gonna be a bumpin term

permalink

https://stabyourself.net/mari0/ #mario + #portal = awesome

permalink

Just finished deploying my own #gollum wiki.

permalink

I'm going to miss #ruby's hash rocket http://blog.peepcode.com/tutorials/2011/rip-ruby-hash-rocket-syntax

permalink

for the number of times i ssh'd into my iMac this sumer, you would think that I'd have just put my ssh key on it #fail #doingitthehardway

permalink

https://developer.ibm.com/articles/wa-ruby/ #ruby on #android. This is pretty awesome, as is the name.

permalink

I was not able to access google plus for about a week. Figured they were just having issues because I was still able to access email fine. Ended up having to recover my account. #fail

permalink

It's been real California; can't wait to come back

permalink

So #rails caches column names across the database and generates SQL for them even if they are never used. Why does this make sense? #fail

permalink

Finished my internship at #ooyala. Had an awesome summer; learned so much.

permalink

Just demoed my Custom Events analytics project at #ooyala. It went well: "This is the future of freemium analytics."

permalink

RSS and Atom feeds now render posts the same way I render them on the lifestream page. Thanks to my one reader derenrich for the feature request.

permalink

Woo! my #django #patch got accepted. https://code.djangoproject.com/changeset/16609

permalink

this is way cooler than rvm: https://github.com/sstephenson/rbenv #ruby

permalink

next project: install rvm and homebrew

permalink

Got a #django bug accepted: https://code.djangoproject.com/ticket/16573

permalink

yet again, #vim is awesome: https://danielchoi.com/software/vitunes.html

permalink

#vim omni completion + #supertab = #vim #bliss. I'm never using an IDE again

permalink

And so I wait for the dns change to propagate ... soon, we'll be live here from NJ

permalink

Disabled #root login over ssh. Have no idea what the password is. This is the #ubuntu way.

permalink

I haven't posted about #music in a while: #songkick is the best. I'm going to so many more concerts now. I get too immersed in #turntable for it to be a regular thing.

permalink

#thisisa.pathological#testcase.#dealwithit

permalink

first post on the new server. It runs. I'm trying to make this #django thing leaner and less hacky. Still have a ways to go. Boy do I need to set up a dev env.

permalink

What a busy friday. Switched to iTerm. 256 colors is nice. Set up a git repo with all my dotfiles. Got real #serious with my #bash config.

permalink

got a new #server. It's in Newark. It's some lonely VM. It's time to get serious.

permalink

duplicity: https://duplicity.gitlab.io/ #backups

permalink

finally got around to giving my #vim config some love. MacVim is pretty cool.

permalink

#mashup #idea: quova + mixpanel. Give it the ability to scope by region/state/country/county

permalink

the hyperbola repos have been migrated to github. and so begins the death of dev.hyperbo.la

permalink

I avoid adding punctuation after my #hashtags because there's a #bug in my regex that parses them. #thingsishouldfix #django

permalink

After working with #rails for a bit now, I kinda like the #magic. DB interations seem so much more natural than they do in #django

Photo for post 55.
permalink

My new favicon welcomes my site to 1996 #itsgreattobehere #fail

permalink

oh #quora, you sly kitten, requiring me to sign up to view all the answers to a question

permalink

Next birthday present to myself: CrashPlan+. Offsite #backup of all of my media? Yes, please.

permalink

Thinking about migrating this site to EC2. Probably gonna stick with #django. http://dev.hyperbo.la will probably die; github does it way better. Moving all media assets to s3: maybe. ssl: probably not

permalink

checked my Activity Monitor, had 2 #ruby processes each at 90% cpu which had been running for hours. trapping signals #fail

permalink

open("|-"), you are a tricky beast. Successfully fork bombed myself 5 times before I sorted that out.

permalink

why `svn status` prints nothing when you have a clean working directory is beyond me. #git ftw

permalink

hyperbola, now with 500% more cache-control headers

permalink

I need to get around to resetting up this server. I want to switch from svn to git and redo the whole apache config. why did I put the site in /var where I have to sudo all the time?? #linux #fail

permalink

one day I'll set up a #django dev environment again. Until then, we'll do it live!

permalink

Finally fixed the css on my site to be less of an abomination. width: auto, you are wonderful

Photo for post 43.
permalink

rebased some already pushed commits to clean up my history and forced the push. I feel so dirty. #git #fail

permalink

wrote my first MapReduce job today!

permalink

have access to one of ESPN's private repos on github

permalink

finally broke down and got a github account. My username is lopopolo

permalink

Should I upgrade 10.04 LTS to 11.04? #linux

permalink

this weekend's project, get all my config files on github

permalink

vi mode is the best thing I have ever discovered about the command line

permalink

pushed my first bugfix today!!!!

permalink

lesson learned: never muck around with mysql's database directory; you will screw up #fail #linux

permalink

this site: now with 5 times fewer 500 errors #django #fail

permalink

for some reason every part of the #django admin worked except the root page. had to turn on debug mode to see I was improperly configured #fail

permalink

first day of work tomorrow #excited

permalink

I got accepted to the EECS MEng program!!! woooo!!

permalink

sysadmin ing is fun. hello beautiful browser based svn repo browsing (insurrection) and web-based svn admining (svn-admin) #linux #win

permalink

ruby's splat is just python's tuple (un)packing #win #fail

permalink

broing out is somehow making studying for algorithms not suck

permalink

its amazing how the time melts away when you're playing portal

permalink

unintentional side effect of my server's firewall: I can't traceroute it. #fail

permalink

it's amazing how the time melts away when ur mucking around on a server

permalink

I need a 512GB iPhone

permalink

killed my own email solution because I had no confidence it would work reliably. Rolling with Google Apps now

permalink

lol. edited the site live again while working on a new app. 500 errors abound. #django #fail

permalink

got my offer today. will be working for ooyala for 3 months this summer.

permalink

lesson learned: never edit a site live. #fail #django

permalink

#idea a new NFL stat - average number of plays to achieve a first down (both intra- and cross-drive). I think it'd be interesting to compare this to average drive lengths, points/drive, and defenses

permalink

Found out that my raid array not mounting a couple months ago was due to a failed drive. Can't recover it. I lost so much data. #linux #fail

permalink

relaxing during IAP. I've started working on a side project: a flash-based dj mixer that uses #music from youtube. #flash

permalink

I HATE FLEX SO MUCH #fail

permalink

This is why I like django's pluggable apps. I can get distracted and not finish a project, but the site still works. #django #distractions

permalink

Just set up three HD cable boxes and they look sweet. #tv

permalink

working on a sort of app store for projects i've done, a glorified portfolio

permalink

lots of code refactoring going on right now. One drawback of the ease and speed of #django is that it allows you not to think.

Photo for post 2.
permalink

props to a friend for giving me some tips on how to make my logo better. this is the tweaked version.

permalink

Had a bad run-in with rm -rf. lost the sqlite database for the site. on the bright side, I'm using mysql now. #linux #fail