hyperbola :: lifestream

Recently I've been on the "I should write a blog post" to "spend 2 hours updating dependencies across 4 language ecosystems in my blog's Bazel monorepo" pipeline. I should still write that blog post though #hypstatic #bazel

permalink

And proof that this setup is both good and bad – CI failed because I forgot to set the publish timestamp in the post YAML on the previous post. But CI ALSO failed because I forgot to run gazelle. #win #fail #bazel #hypstatic

permalink
Post image

My artisanally hand-crafted blog generator is an unholy mix of JavaScript, YAML, Markdown, assets distributed by npm package, Golang, Rust, and Python these days, glued together with Starlark in Bazel (deployed to GH Pages) #hypstatic #bazel

permalink

Just used exiftool and lots of zsh globbing to strip all metadata from JPEGs and PNGs used to build the static hyperbola site. #hypstatic

permalink

hyperbola/logo repository was merged into the hyperbola-static monorepo! There is only one repo used to build this site now! #hypstatic #bazel #monorepo

permalink

Woo! hyperbola no longer requires webpack to build. Hundreds of dependencies removed! The build uses eslint for bundling. #webpack #hypstatic

permalink

I just removed all pagination from the #hypstatic lifestream section #win this design choice dates all the way back to the first implementation of lifestream in 2010 #history

permalink

I've been toying with the idea of removing all pagination from #hypstatic

permalink

There is no more #webpack in #artichoke. All web properties have been migrated to esbuild, eta template engine, the Node fs API, and custom build scripts. This is simple, this is good. #hypstatic is next but there is a lot to unwind. #win

permalink

With AWS SSO in place, I started using aws-vault locally for #terraform and AWS CLI. I learned about this tool at work. No IAM access keys with inline IAM policies! SSO + assume role for administrator access! #security #win

permalink

In #aws things, I migrated both hyperbola and #artichoke infra to separate AWS organizations and set them up with AWS Control Tower's account vending machine. Audit logs! AWS Config! SCPs! AWS Cost and Usage Reports! AWS SSO! #win #security #cost

permalink

I'm not sure how long ago this happened, but I got ownership of the @hyperbola handle on #github to match the npm namespace. Some js libs, old infrastructure for dynamic hyperbola, and #hypstatic live there now. I've added a 'project-infrastructure' repo like in Artichoke which does infra as code for the GitHub organization and #aws setup.

permalink

I took a hacksaw to my homebrew installation. I'm not sure how this happened but the brewfile I've been carrying around at one point had the entire dependency graph in it rather than the explicitly installed deps ... which made everything on my new Mac an explicitly installed dep. I had hundreds of libs installed for no reason. Burned everything down and started from scratch. #fail #win github.com/lopopolo/dotfiles/blob/fb9af6bd19df2149dba5a63a8db161702bcbe5db/homebrew-packages/Brewfile.rooster

permalink

macOS things – I tried out alacritty and tmux again but didn't switch away from iTerm2. I tried a bunch of fancy neovim LSP, floating pane, and lua config and then stripped all IDE bits out of vimrc to keep it just an editor. Finally enabled dark mode on my Mac (synced with the sunrise and sunset), switched to dark terminal and neovim color schemes. #win #vim

permalink

Another fun #rust update: `intaglio`, my string interning crate, recently got support for C strings (not yet released).

permalink

yikes and pagintated navigation for archive pages had month and year mixed up in the link. #hypstatic #fail

permalink

ooof. While hacking on the lifestream part of the static site generator, uncovered that previous and next buttons for traversing lifestream permalinks had their polarity swapped. That bug has been there for over a year. #hypstatic #fail

permalink

I got a PR in artichoke suggesting to bump down the frequency of dependabot updates to monthly. I applied this change across all artichoke, hyperbola, and lopopolo repositories. It has been a big quality of life improvement and cuts down on a lot of churn in JS dependencies. Applying dep updates once a month is 👍 #win #artichoke

permalink

Yay! hyperbo.la is running Bootstrap v5. This upgrade was actually really small! Just some small changes to the Sass for customizing colors and updates to the navbar. #win

permalink

Most crates are no-std when possible, 100% documented, and I think pretty high quality. #patch #rust

permalink

focaccia and roe are UTF-8 case manipulation crates for implementing parts of `String` and `Symbol` in #artichoke. intaglio is Artichoke's symbol table, boba is an implementation of Bubble Babble Encoding which will be used in the `digest` stdlib package. #rust #patch

permalink

Since cutting hyperbo.la over to GitHub Pages and tearing down most of the AWS infra, monthly AWS costs have been about $1.25. The biggest two items have been ~$0.50 each for S3 and Route53. #aws #cost #hypstatic

permalink

This PR also took the opportunity to hand-optimize and hand-make retina variants of all images in the blog and lifestream. This eliminated most of the large asset warnings from #webpack #performance #hypstatic

permalink

Sharding the #webpack build meant `webpack-dev-server` was not longer an acceptable choice for a dev server. Replaced this with a #bash script and `python3 -m http.server`

permalink

There's some recursion depth error that causes #webpack to abort when using ~1500 `html-webpack-plugin` plugins that I worked around by sharding the webpack build.

permalink

In the process nukes 1000s of lines from package-lock.json #hypstatic

permalink

To make this change on dynamic hyperbola would have meant 5-10 Ansible deploys to a Vagrant VM. Iteration speed would have been terrible. #devops #a11y #lighthouse #hypstatic

permalink

And just like that, dependabot is enabled for hyperbola-static, the WIP lifeastream branch is merged, and the CI is spruced up #hypstatic

permalink

This is a setup that has worked well for artichokeruby.org. Set dependabot to autoupdate deps and the "maintenance" has been updating content. Deploys are automatic, regression testing of dep upgrades is automatic. #automation #hypstatic

permalink

There is Ansible, Packer, Terraform, JavaScript, Webpack, Python, Django, Ubuntu, nginx, EC2, Route53, ALB, S3. I think I can simplify this to GitHub Actions, GitHub Pages, Node, Webpack, and statically configured DNS with my registrar. #hypstatic

permalink

Looking at the commit history for Python hyperbola over the last 3 years shows that it hasn't been developed, only maintained. It is done, but the current repo is expensive to maintain. #hypstatic

permalink

Switching to a static site built with webpack at least affords the opportunity to have an integration test that the site compiles. #automation #hypstatic

permalink

I've lost my ability to develop locally since I don't want to install VirtualBox on my new laptop. #fail #hypstatic

permalink

I will miss the ease of adding lifestream posts via the Django admin UI, but given that I'm posting elsewhere these days, I don't think I'll miss it too much #hypstatic

permalink

Remaining pieces are implementing hashtag browsing pages and the date archive browsing pages #hypstatic

permalink

Most of my updates these days are either in the Artichoke Discord or @artichokeruby Twitter #hypstatic

permalink

The AWS infra that the current site runs on is $50 a month I do not need to spend. #aws #cost #hypstatic

permalink

I've had a work in progress fork of hyperbola to migrate to a static site hosted on GitHub pages for over 6 months. I can't find the time to finish it. #hypstatic

permalink

Also Windows Terminal and the Cascadia Code font were a dream to use. I replaced Inconsolata with Cascadia Code on my Mac iTerm2 #terminal #font #windows #win

permalink

After repeated attempts of replicating my windows CI setup in a developer VM, I sloppily installed software via web installers. With every VS component, rustup, RubyInstaller, LLVM, choco, and winflexbison, I was running with a reproducer #artichoke

permalink
Post image

Nightly builds of Artichoke are available via ruby-build: rbenv install artichoke-dev #artichoke #win

permalink

The one dynamic bit I cannot truly replicate is shortlinks but I never used this feature #win

permalink

No more security updates, terraform, autoscaling group rolling deploys, packer and baking AMIs, Ansible deprecation warnings, RDS outages #win I can unsubscribe from nginx release notes listsrv lol

permalink

Big wins from switching to static: modifying content with a lightweight dev server, CI via GitHub Actions, automated deploys, better webpack integration and minification, site is free as in beer to operate #win

permalink

Partials tended to end up in the same places as before #win

permalink

Some bespoke filters I hand rolled before got pushed into dependencies, like hashtag parsing and linking moving to linkifyjs #win #yolo

permalink

There is about as much javascript as there is python in the dynamic, django-based hyperbola, which is to say not much #win

permalink

It mostly works and is reasonably speedy, although while iterating, a template error in the lifesteam post rendering code caused ~1000 templates to fail compilation. I let the build peg the CPU for 20 minutes before giving up #fail

permalink

The static site is based on webpack and webpack-html-plugin with a few preprocessor scripts that emit the templates #webpack #js

permalink

I made the mistake of exporting the database first AND not documenting how I transformed it to YAML, so I've been hesitant to post here #fail

permalink

for the past 2ish months I've been slowly working on porting hyperbola to a static site with the ultimate goal of winding down an #aws account

permalink

I published my first #rust crate (and package of any kind, ever)! rand_mt is a fork of an existing Mersenne Twister crate which adds compatibility with the latest rand_core and makes it no_std crates.io/crates/rand_mt #patch

permalink

Migrated all #artichoke CI to GitHub Actions #github #win Jobs and workflows are so much easier to reason about. And it's free 💪

permalink

I've been absent from the lifestream for a bit. I've been busy implementing!

permalink

hyperbola was down for 26 hours because the CA cert bundle on its backing RDS instance was not rotated before the 2015 bundle expired #fail #outage #aws

permalink

Just explicitly shut down the frklft.tires site by doing a terraform destroy. Sad, but that code didn't need to live forever.

permalink

Also added many more converter implementations with macros. There are 988 TryConvert implementations. #win #fail #artichoke

permalink

This change added new functionality: Converting to and from byte and string slices #win #artichoke

permalink
Post image

The converters are central to #artichoke. Changing this core abstraction was painful: 84 files changed, 2000 lines added, 2800 lines removed. #fail #git

permalink

The converters allow the runtime to convert between Rust and Ruby types. They are the glue that the entire Ruby Core and Ruby Standard Library implementation is built upon. #artichoke

permalink

I'm separating the core of #artichoke from the #mruby backend specific glue. This step involved reimplementing the converters with the new traits in core.

permalink

Neither mruby-sys nor onig can build with wasm-unknown-unknown or wasm-wasi, so I was stuck with wasm-unknown-emscripten. The linker on mruby-sys would dead code eliminate artichoke_backend::Artichoke #fail #Wasm #artichoke #mruby

permalink

One of the goals is to have a #Wasm build target. This is what nerdsniped me this weekend. #artichoke

permalink

Being forced to set up the repository infra from scratch in cactusref was a good way to fix everything I disliked about my dev environment. CircleCI build and scripts/lint.sh got overhauled #win #artichoke #shell #bash #node #lint

permalink

With renewed focus of building my own #Ruby instead of extending #mruby, I put in some effort to make mruby an implementation detail of #artichoke. Lots of refactoring with sed. #fail

permalink

ferrocarril had different goals than artichoke: it wanted to run Rails. To do that, it had a lot of extra bits in it like gems, a Rack server, and sample apps. Nuked all of it #win #artichoke

permalink

Step 2: transfer ownership of ferrocarril to artichoke organization #artichoke #github. Not much to do here, updated some links, fixed some branding, got CircleCI building again #artichoke

permalink

CactusRef is still an incredibly unsafe crate, but at least it aborts if it detects a use-after-free #fail #artichoke

permalink

Extracting cactusref required completely rewriting #git history, updating docs, adding a build, adding benches, importing my blog post, updating links in the blog post #artichoke

permalink

Every toggle switch in a UI is secretly an Option<bool> #design

permalink
Post image

#git bisect is why it's important that your repo be buildable at every commit. Currently fixing a massive memory leak in ferrocarril #fail

permalink

Despite all I talk about #aws #ec2, I cannot for the life of me type isntances correctly #fail

permalink

Adding #ci to a project is encouraging such better code hygiene. Because I want to make sure the build stays green, I actually make PRs even if it is only me approving them. Seeing the code in the browser makes it easier to review than #git diff #win

permalink

In a related note, I now peg the nightly version in the rust-toolchain file to a specific date archive #ci #rust #win

permalink

You can tell which parts of ferrocarril are new to me based on how many consecutive commits are dedicated to a feature. By this measure, the new (and hard) things were sys crates, #ci, and TryFrom traits #rust #win #fail

permalink

I tried to explain continuous integration to my Dad and he synthesized it as "spellcheck for code" which is pretty good tbh #win #ci

permalink

The keyword "unsafe" appears 372 times in this codebase #rust

permalink

Using cargo:rerun-if-changed and cargo:rerun-if-env-changed to make most sys crate builds a noop. developer productivity #win #rust

permalink

About 6 weeks ago I reached out to two literary agents to see if they'd help me publish a book. I got an explicit rejection last week! A little bit #fail, but also definitely progress #win 😊

permalink

I generated FFI bindings for mruby with bindgen! I wrote a #c extension library for wrapping mruby macros! #win I wrote (a lot of) unsafe #rust code. Clippy helps a lot to make sure I'm not doing anything stupid. I have caused quite a few #segfault #fail

permalink

I implemented a #rust macro for the first time to implement some traits to convert between mruby values and Vec and Option containers #win

permalink

I overflowed the compiler (which didn't crash it) and crashed the linter with an ICE #win #fail

permalink

I've been working on creating an idiomatic, modern set of #rust bindings for mruby, an embedded #ruby interpreter. I am learning a ton and doing lots of code things I've never done before! #win

permalink

looks like I didn't finalize the deploy in January so I had a few extra AMIs kicking around that I was needlessly paying for #aws #cost #fail #automation

permalink

The true motivation for getting me to mess with hyperbola today was an instance retirement notification from AWS. I had to muck with poetry to cycle my ASG #fail #aws

permalink

Uninstalling python 3.7.1 from pyenv broke my local poetry venv for hyperbola which prevented me from managing my deps #fail #python

permalink

Yikes haven't done a deploy since January 6. I've definitely been ignoring all of the USNs about Linux kernel vulnerabilities. #fail Django 2.2 release is what got me to get to upgrading #django #win

permalink

whoa throwback `chattr +i` to prevent puppet from overwriting files on a prod box that you were manually changing #devops

permalink
Post image

Deck making for an internal tech talk called "How to Implement the Chromecast Protocol in Rust When You Don't Know Rust" #rust futures and impl trait 😭 #fail

permalink

When asking someone to do something via email, the fact that they’ve even opened it is a big hurdle. Make their life easy: in-line as much as you can, minimize clicks, make links easy to scan, be direct, have a tl;dr #communication #email

permalink

I'm speaking at Monitorama on June 4-6. My talk is titled The AWS Billing Machine and Optimizing Cloud Costs. #conference #aws #cost

permalink

I'm speaking at DevOpsDays Seattle on April 23. My talk is titled The AWS Billing Machine and Optimizing Cloud Costs. #conference #aws #cost

permalink

Larson Media Training 101: Answer the question you wish you were asked. #communication

permalink

Spaceships don’t come equipped with a rear view mirror 🚀 #win

permalink

I live in Seattle now. First week at the Stripe office was quiet due to the snow. Working on expanding Stripe into Latin America.

permalink

#marketing is easier if you have metrics to support your impact. I like to always tie my metrics back to dollars.

permalink

One common hangup I've encountered when discussing #marketing my work with other engineers is the notion that the work was easy or trivial to implement. That's not the important part! What the org cares about is impact. Tell a good story!

permalink

To be good at #marketing yourself, you need to be good at communication, both written and verbal. Don't be afraid to talk about your work, either to your manager, your skip level, your users, an email list, or the company all hands meeting.

permalink

As an engineer, I care very much about #marketing myself, my team, and my work. I am of the impression that my work does not speak for itself, I have to speak for my work. Being high impact means making sure the org knows it, too.

permalink

Making a mental note that I've decided to go static with hyperbola. Not sure when this project will get done, but here's to 2019. I have dreams of a custom #webpack and #markdown generator, S3 and CloudFront for distribution.

permalink

The most expensive part of hyperbola's #aws infrastructure is the SSM PrivateLink endpoint in 3 AZs #fail #cost

permalink

Optimizing #react bundle ... cut bundle size by 72KB. Biggest wins are turning an image into CSS + an emoji, removing unused deps, enabling mini-css-extract-plugin in release builds #win #webpack index.html is 333 KB

permalink

I've now over-engineered an iTunes library integrity checker by turning a script into a module #win

permalink

Just accidentally truncated my .bash_history. Restored from backup but the latest was 55 days ago. #fail

permalink

The computers did exactly what I told them to do 😕 #fail

permalink

At least I was able to repro in local #vagrant once I figured out what was going on. #win

permalink

A red herring was adding a call to dd in the cleanup script to zero the free space on the disk. I thought I was somehow filling the disk and not reclaiming space.

permalink

The cleanup script didn't error because my set flags were in the shebang but #packer was invoking the script via bash instead of directly #fail So many yaks.

permalink

This change was introduced in 0.146.0 but did not manifest due to a bug in the cleanup script. I was not passing -y to apt autoremove, which caused the command to abort and end the script with an error. #fail

permalink

Removing just the dev package with #ansible #provisioning had no effect in local env, but combined with the apt purge in the #packer script, uninstalled mysqlclient

permalink

Root cause: My app depends on the python package mysqlclient, which in turn depends on the libmysqlclient system package. I was pulling this in transitively via the dev package required for building.

permalink

Undeployable manifested as healthz returning 502 when adding a new instance to the ALB, marking it as unhealthy and timing out #terraform

permalink

I recently started cleaning up after myself during #provisioning. There are two steps to this: #ansible uninstalls build deps and #packer executes a cleanup script that does an apt purge and autoremove

permalink

I completed removing build artifacts from source control. Static assets dist and document-root dirs: replaced with running #webpack on host. Third-party #ansible roles: installed from galaxy via #packer and #vagrant at provision time. #win

permalink

code yellow 😕

permalink

I added code block and syntax highlighting to my new #blog. Planning on using it for an upcoming post about #terraform.

permalink

#django is a joy to use. Making my new blog app was so easy. Each view is only two lines of code! #win

permalink

I accidentally skipped v0.139.0 today because prettier barfed during cutting the release and I forgot to reset my git tree. I guess I forgot to run prettier on my whole repo when I enabled it. #fail

permalink

Jumped to latest #django 2.0.x and then to latest 2.1.x today. No deprecation warnings, smooth sailing. As a plus, I turned on PYTHONWARNINGS for my app in systemd #win

permalink

#AWS suggested for us to use a snowball to ship our data to a new region. This is an amazing product, but yea no

permalink

I also pruned a lot of config from my 12factor env, favoring convention for things like DB connection strings. In doing so, migrated to one hostname for DB across all environments. /etc/hosts alias in Vagrant, network in Docker, private hosted zone in AWS

permalink

Finally deployed a 2 month old branch incorporating lessons learned from my Docker experiment into my VM-based deploys. Big change is building from local source instead of pulling in a tarball from GitHub. Iteration speed is improved #win

permalink

One hack I've been using to enhance my productivity in the face of KTLO work is to write lots of things down. Project plans, system diagrams, user interviews. It helps organize the work that I have done and keeps me motivated.

permalink

The past 4ish months I've been a team of one. Been a decent slog. It's been hard to be productive when all of the KTLO work falls on me. Getting a new teammate in a couple weeks. Excited. 🤩

permalink

I've been retraining myself to say "Hey Google" instead of "OK Google"

permalink

Optimizing for #cost and complexity, #docker and ECS/EKS are not worth it for my 1 node crud app

permalink

Saved me $17 a month. My primary #AWS #cost is now my ALB. I'd replace it with an nginx if not for ACM making certs so easy

permalink

Modified RDS to be single-AZ #HA

permalink

Set desired backend ASG capacity to 1, but added autoscaling alarms to allow scaling up to 3 instances #HA

permalink

Realized that hyperbola doesn't need to be super #HA

permalink

LuaLaTeX is amazing. My documents look better than they did with XeLaTeX. Less hyphenation, less inter-section spacing. fontspec, polyglossia, selnolig, nowidow, and impnattypo packages are dope too. #LaTeX #win

permalink

The Makefile for a project like this defines two versions of all the tasks. One runs locally on my mac and "recursively" calls make using `vagrant ssh` setting a make var to indicate the task is running in the VM #make #LaTeX

permalink

I've been using make and vagrant for self-contained builds in other projects. I've been working with #LaTeX a lot and a bionic vm with TexLive installed creates a repeatable build env, takes 15 minutes to set up, and doesn't pollute my desktop #win

permalink

Upgrades are painful, so upgrade frequently. hyperbola now running Ubuntu 18.04 Bionic. 2 build config changes and 1 removed ansible task. #win

permalink

I am surprised that #django autolinked the .tires domain. We live in the future.

permalink

for frklft.tires, I stopped using #terraform for managing the static content of the site. It now lives outside of my terraform code in a public directory, published explicitly with a make target #win

permalink

I launched frklft.tires last week. Built with S3, CloudFront, and Lambda.

permalink

My current understanding of this column is that it is the instance family if the usage type is eligible for RI instance size flexibility

permalink

What they neglect to mention is that DedicatedUsage types do not have an instance family associated with them

permalink

AWS says that product/instanceTypeFamily is "The instance family that is associated with the given usage. For example, t2 or m4."

permalink

Sometimes using the #AWS cost and usage reports is just not fun, mostly due to the myriad of columns being undocumented. #fail

permalink

When all you have is a rock, everything looks like a hammer

permalink

Last week I shipped secondary attribution for Jenkins builds. Email featured lyrics from Bills, Bills, Bills by Destiny's Child #win

permalink

pandas and jupyter notebooks are my new favorite hammers

permalink

Semantic versioning is a lie (looking at you #packer). My config stopped working because a key was deprecated between 1.1.x and 1.2.x. Somehow this prevented the config from validating. #fail

permalink

I have pinned dependencies everywhere _except_ my AMI build pipeline. Bitten by the packer 1.2.0 upgrade breaking the ansible-local provisioner. #fail

permalink

Not to mention Let's Encrypt to Amazon back to Let's Encrypt again (for vagrant) for certs

permalink

I've had significant churn on tools in the past year for building hyperbola: pip to pip-tools to pipenv. gulp to webpack. linode to AWS. bespoke deployment and scripts to ansible. All good, just exhasuting.

permalink

Ansible, Let's Encrypt, nginx, and Vagrant have combined to make my local env look more like prod. Spun up a new ALB-like VM with HTTPS. less divergence #win

permalink

Starting 2018 off right by fixing pbr installation, deploying django 2.0.1, upgrading Pillow, and removing dep on django-localflavor #win

permalink
Post image

Baby's first hadoop. Taken from long forgotten code written in 2005 #history

permalink

Removed some "bespokeness" from hyperbola: migrated from a shell script that exported env vars and exec'd to dotenv. Same concept, but now a standard mechanism #win

permalink

Put some investment into my #vagrant setup. MySQL running in a separate VM with automatically provisioned fixtures means I can go from zero to full replica of prod in 10 minutes. No more dep on laptop MySQL. #win

permalink

New in v0.122.0: python packaging actually works now. No .pth symlinks. No PYTHONPATH. setup.py and pbr install hyperbola directly into the venv. Finally. #win

permalink

hyperbola is now running #django 2.0. Added automigrate to hyperbola-app systemd unit, eliminating one of the last things I ever needed a bastion host for #win

permalink

Found a lifestream bug during my Django 2.0 upgrade that was never exercised on the live site because I've never had more than 40 posts in a month #fail

permalink

I suspect that NTP traffic was the trickle of traffic that traversed my recently decomissioned NAT

permalink

That was easy. Enabled #AWS Time Sync by adding one ansible role, reading some docs, and redeploying. #win

permalink

Accidentally deleted my private subnet route tables in the process though, which broke S3 access for ~20 minutes. Could've been much worse #fail #outage

permalink

Shaved another $35 (40%) off my #AWS bill by disabling the NAT on my app subnets. Yay immutable infrastructure and VPC endpoints #win

permalink

I made a thing! burnfastburnbright.com Bootstrap 4, route53 domains, and terraform made this really easy. went from 0 to 100 in about 1.5 hours. #win

permalink

New in v0.117.0: nuked time-elements webcomponents due to Firefox breakage (cut js payload by a factor of 4), infra improvements to resume handling #win

permalink

The v0.116.0 deploy was done using a spot instance with packer. A bigger instance for half the price #aws

permalink

New features in v0.116.0: bootstrap4, removed RSS and Atom feeds, 100% webpack frontend build, css purification improvements, and healthz middleware

permalink

welp that didn't last long. CloudFlare only queries a subset of NS records to check for liveness and has determined that I no longer use CloudFlare. Working on purging them from #terraform and registrar now #fail

permalink

It is a good thing that I've automated things well enough that I don't need the bastion #win

permalink

Even more cost savings: dynamically provisioned bastion cloudformation stack #terraform #aws

permalink

More cost savings. RAM footprint of a hyperbola backend is 143MB. Switch from t2.micro to t2.nano #aws #win

permalink

I accidentally created a CNAME (instead of an A record) for an IP today. Lots of confusing errors from nslookup, ssh, and host. Meanwhile dig appeared to resolve the record. #fail

permalink

hyperbola: now with multi-homed DNS. AWS Route 53 and CloudFlare, made possible by terraform. (In the process upgraded hyperbo.la mail to a 2048-bit DKIM key) #win #redundancy #devops

permalink

I accomplished this migration with ~no downtime #win. I spun up the new infrastructure and then deployed new AMIs with updated service records. I did have ~2 minutes of 500s when I accidentally overwrote old mysql DNS record due to a bad copypasta #fail

permalink

Switch from 3 to 2 backend machines. 1 is enough to handle the load I get, so use the bare minimum for redundancy #aws

permalink

Removed dependency on redis by switching to a django-provided database-as-cache adapter. My redis cluster was used only for admin sessions and caching a sidebar on the lifestream page. Unnecessary overhead #aws

permalink

Switched DB instance type from db.t2.small to db.t2.micro. From running my linode I know that MySQL never used more than ~400MB of RAM so I knew this was safe. My database is tiny #aws

permalink

Switched from Aurora to a multi-az RDS instance. I don't need the complex topologies that aurora allows and it forced me to use an overprovisioned instance type #aws

permalink

Now that I've shown I can go all out with the most expensive #AWS components, today I exercised my cost efficiency and right sizing muscles. I cut my AWS bill in half with the following steps:

permalink

Just bumped backend ASG from 1 -> 3 t2.micros. With this change now all parts of hyperbola (redis, mysql, backend, lb) are multi-AZ #win

permalink

mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -u root mysql was the magic incantation required to get lifestream archive views working locally

permalink

Today's shipped email featuring subtly modified lyrics from Kanye's Flashing Lights

permalink

The magic command to make homebrew work after uninstalling Xcode: sudo xcode-select -switch /Library/Developer/CommandLineTools #win

permalink

this is just one example of the times #git has saved my life #win

permalink

I've taken to opening shipped emails with playfully mutated rap lyrics. Super Nintendo. #win

permalink

Total upgrade time ~1 hour. Includes code changes, testing locally, cut release, deploy to vagrant, build AMI, terraform, smoke test #win

permalink

I skipped #django 1.11.4 and 1.11.5. Finally did an upgrade today to 1.11.6 #win. My dep upgrades for python, js, and ansible deps are too coarse grained. Pulled in ansible changes which blocked the deployment #fail

permalink

and no more chrome on iOS. Using safari there. The only value chrome brings is unified history and being logged in to my google account. I use neither of these features

permalink

I switched to Firefox Developer Edition today on my mac. It is pretty blazing fast. Looking forward to 58 making it to the dev channel in November

permalink

With @huff's help, I have discovered ETL's True Name. ETL is just indexes. Indexes are just caching. Caching is just named replication. Names are the only problem in computer science.

permalink

When sorting cards in a hand, I use multi-pass bubble sort #fail n is small i guess

permalink

#terraform is now a package manager. Great. #fail. For some reason plugin downloads hang if the download gets an IPV6 edge node in their CDN.

permalink

I was first introduced to closures in summer '08 via #ruby blocks. This was probably the most magical moment I've ever experienced programming. That code was just another object you could pass around was amazing. Lisp would've been mind blowing. #history

permalink

when reading from stdin, ansible-vault encrypt_string does not trim trailing newlines, forcing one to use echo -n. #ansible knows this is a sharp edge and documents it, but they should do the right thing and make it consistent with password files #fail

permalink

in the quest to thin out the top-level directory sprawl in hyperbola, I got rid of .secrets. required rotating ansible vault password and rekeying everything because my password had single quotes, double quotes, and backticks (of course) #fail

permalink

#fail do not try to install ansible in a virtualenv in packer. you need a compiler. you don't want a compiler

permalink

the hyperbola + hyperbola-tools binary star monorepos is a common failure state, so I am told #fail

permalink

I just realized that by merging hyperbola-tools into hyperbola and converting my ad hoc ruby scripts to python, I moved to a monorepo and standardized on a language

permalink

6. addendum: since I already had a deploy script, it was straightforward to translate it to #ansible tasks

permalink

6. problem: bespoke deployment steps. deploy and cron run as lopopolo. deploy needs sudo password. solution: #ansible deploy_helper module

permalink

5. problem: environment names are inconsistent across tooling and deployment. solution: hyperbola_environment #ansible variable which parameterizes everything

permalink

4. addendum: I used #LetsEncrypt before migrating to AWS with a combination of cron, systemd timers, and dehydrated. ACM is easier, less error prone, and set-it-and-forget-it #win

permalink

4. problem: https is hard. solution: ACM + #terraform + ALB + CloudFront

permalink

3. problem: manual, bespoke server configuration. solution: #ansible, #vagrant, #packer, prebaked AMIs. immutable infrastructure

permalink

2. django-backup.py sucks. problems: email based, backup size is limited, unconfirmed delivery, plaintext transmission. solution: move media to versioned S3 bucket, daily RDS snapshots, logical backup json to S3

permalink

1. problem: python not running the latest 2.7.x release. solution: pyenv for local development and xenial (python3.5) + ansible + (future) deadsnakes ppa

permalink

I found a page in my wiki called hyperbola suckage 2015. This was a forward-looking list of gripes to fix as I moved from hyperbola1 to a new host #history

permalink

unexpectedly memory: I had to spoof my mac address to that of my xbox 360 so that I could register it on MIT's network #history

permalink

#aurora was much less forgiving with unique constraints than mysql. The schema for one of my utf8mb4 varchars had a max length of 255 and a unique index. It appears mysql silently truncates, whereas aurora complained the index was > 767 bytes. ~ #win

permalink

One source of difficulty was getting django, s3, vpc endpoints, and security groups to work together to enable s3 access from my backends #fail

permalink

my #terraform life became much easier by using name_prefix instead of name. name and name_prefix parameters were never interpolated. Instead, use interpolation in tags. In practice this means config can change without rebuilding the world #win

permalink

I initially went with the unclustered variants of elasticache and rds. Once I wrapped my head around the topology, #redis cluster mode and #aurora were much easier to work with in #terraform

permalink

buliding the #aws infra took about 30 commits, two #terraform destroys, and two terraform code rewrites. some fun bits in the following posts

permalink

The migration from a bespoke-bootstrapped, manual (scripted) deploy process to an #ansible playbook took about 20 commits. I deployed to a local #vagrant box first

permalink
Post image

And with post 506, I can now say that hyperbola is hosted on #aws #win

permalink

2am hot take: all systems problems are best solved with pointers and routing. aka dependency and service injection

permalink

The #webpack configuration, on the other hand, was not fun to get set up. I particularly struggled fighting with the gulp plugin, which pins webpack 1.x

permalink

#webpack is amazing. I added the time-elements web components to the lifestream and contact pages. Only took an hour. Before this would have been next to impossible. #frontend #win

permalink

Rebuilt everything and modulo some zombie deposed resources, everything is good again

permalink

Discovered I had duplicate stanzas in my tfvars file. Removed the extraneous one. Broke EVERYTHING. #fail The worst was my admin IAM account losing access in the midst of a terraform apply. destroyed module.network, module.iam, and module.hyperbola-wiki

permalink

💯💯💯💯💯 = 500 posts #win

permalink

related, using pip-tools to manage and pin python dependencies has removed uncertainty from my deploy process. Upgrade packages only when I intend to #win #django 1.11.3 upgrade went super smoothly in part due to this

permalink

tried copying a 2TB Time Machine volume at the file system level using Finder drag and drop. #fail sadtimes. Was going on 6 hours and still resolving hard links. Currently attempting block-level copy with Disk Utility

permalink

I am living the dream with an n/ac-only network

permalink

Packer + Ansible + AWS + Terraform is really quite lovely #devops #win

permalink

I hope my next laptop has 32GB of RAM

permalink

I got a memory exhaustion pop up on my MBP earlier. I blame spotify, chrome, and the 2 VMs I carelessly had running. #fail

permalink

Migrated terraform state from a private github repo to a private, encrypted S3 bucket. State infra is bulkheaded from main app and protected with prevent_destroy lifecycle #win #terraform #aws

permalink

All done. Downgraded from 8GB of RAM to a more reasonable 2GB of RAM. Went from 4 cores down to one though 😱 #devops #maintenance

permalink
Post image

It looks like I am a *tad* overprovisioned. brb taking some #downtime to downsize my VM

permalink

Also, I have a google analytics snippet on the site. I have never once checked the analytics.

permalink

Moving the wiki to #AWS was a different beast. Everything is self-contained in the AMI.

permalink

sure an ELB + managed MySQL & redis + redundant frontends are "better" but I'm not sure I need the complexity

permalink

Not sure if it is worth the effort to migrate hyperbola to #AWS. Maintenance burden is pretty low. Log in maybe once a month, run an apt upgrade, and do a fresh deploy. Takes less time than a packer/terraform cycle would. #maintenance

permalink

Tested a #django release candidate (1.11RC1) for the first time. Site continues to work. No deprecation warnings. #win

permalink

Software shouldn't be "delightful." It should be invisible.

permalink

I would not be surprised if #AWS VPC networks were all running some franken overlay protocol

permalink

ELB for wiki does a basic healthcheck of nginx, but a backend can be down while the ELB thinks the node is healthy #monitoring

permalink

Immutable infrastructure is so much better than mutable infrastructure. Much more satisfying to build a new image and redeploy than to upgrade nginx and mysql in place, taking #downtime.

permalink
Post image

Found this in the nginx logs of my wiki in ec2. hackers. #security

permalink

Just freed up 350MB in my gmail by purging 35 emails. Did a search for size:5000000. removed emails with mp3s as attachments and event announcements with large images and pdfs #win

permalink

bastion is now in an ASG with an automatically bound (with user data) elastic IP. Yay fault-tolerant infra! #win #aws #terraform

permalink

Hit the 5 duplicate certs per week rate limit from let's encrypt by repeatedly rebuilding my vagrant box #fail

permalink

Switched to building an ami with packer. Now an ASG roll takes 2 minutes #win #aws

permalink

Deployed hyperbola-wiki. Initial approach was to provision on boot with a user data script. This caused instances to take 15 minutes to come up (apt-get upgrade, compiling ruby) #fail

permalink

Provisioned a VPC, subnets in 3 AZs, and a bastion host #aws

permalink

Created an #aws account tonight

permalink

Added middleware to hyperbola app and wiki to inject a comment with the canonical hostname of the host that rendered the request. Debugging #win

permalink

Now have proper retina image support on lifestream and contact page. Generating @2x and @3x images. The flexibility of django-imagekit helped. This is a well-designed library. #django #ux

permalink

hyperbola, now with properly localized dates using window.Intl #win

permalink

Fourth most expensive query is selecting all lifestream items and pictures: a LEFT OUTER JOIN with ORDER BY DESC, LIMIT, and OFFSET. There is an index on the order by but MySQL prefers filesort. Table is only 500 rows though. #mysql

permalink

Most expensive queries on hyperbola are hashtag lookups because they use RLIKE. Top 3 queries are 4.3ms, 3.2ms, 2.1ms per query. #mysql

permalink

diff of yesterday's backups and current backup confirm all that has been altered are filenames

permalink

I didn't notice this in dev because the front page of the lifestream is 20 of the same picture of Kendrick Lamar #fail

permalink

Fixing this required parsing a db dump (django fixtures much easier to read than mysqldump output) and manually running UPDATE queries in MySQL

permalink

Minor crisis when running the mangle media script. Resume and lifestream models have auto_now=True (an updated timestamp) and I have been using them as creation timestamps. Apparently there is an auto_now_add param I should use. #fail

permalink

just automated a spreadsheet (concatenating several columns) with filter #win filter(L6:L, L6:L<>"") vs manually updating ranges #automation

permalink

Some changes since moving to the new host: assets and user uploads served off of hyperbolacdn.com; TLS; frontend deps managed with bower

permalink

❤️ Emoji: 🔥🔥🔥

permalink

reifying #aws infrastructure with terraform is pretty cool. power is scary though. yay for sandboxes!

permalink

I need to make these out of band releases more painful so I migrate hyperbola to my new host #fail

permalink

I started a new job a couple weeks ago. I'm working at #Citadel.

permalink

#throwback #cruft just cleared out MIT client certs from 4 years ago that were hanging around in my keychain

permalink

Also found the samba password to the file server named supersecret that I built

permalink

All this discovered while clearing out my OS X Keychain

permalink

That was when hyperbo.la ran on a reclaimed Dell workstation in a network closet at MIT

permalink

I think I also used to host a redmine instance on dev.hyperbo.la back in the day ~2010

permalink

Forgot to blog about this: My printer ran out of toner a week before my move. Had the printer since 2010. RIP printer

permalink

With systemd I am one step closer to #12factor app bliss since all service logs get logged to syslog #win

permalink

made the #nginx config a little better, converted thin to bind to a domain socket, and deployed with a dedicated app user with deploy keys #win

permalink

In the process, I've added #TLS with Let's Encrypt #win

permalink

The past few days have been spent migrating my #gollum wiki from hyperbola1 (old server) to hyperbola2 (new server)

permalink

And a 4 years newer OS (Ubuntu 16.04 LTS) and 64-bit #win

permalink

Project new VPS is underway. For the same price as my current 1G box, I'm getting a 4G box with more cores #win

permalink

nginx upgrade broke the init script. had to reboot to fix things. gunicorn failed to start after reboot due to log file perms #fail

permalink

I enjoy golfing my minified javascript. Just shaved 19 bytes off of the inlined date formatter js I include on the lifestream page #js #win

permalink

The only times I have ever run ant as a build tool were a couple of CS classes back at MIT—4 years ago. Even today, it remains my 8th most frequently used command on my mac

permalink

Filed a site issue JIRA on my last day. It was eventually closed as invalid. By the time this happened, though, I had lost systems access and couldn't discuss it further

permalink

Today was the day of linters. Added eslint, flake8 config, and isort to hyperbola

permalink

"we successfully avoided a split brain scenario due to master/master replication" #mysql #fail

permalink

hyperbola prod is now running #py3 what a time to be alive

permalink

#scala takes the #objc std lib approach to collections: One public interface/factory w/ many optimized implementations underneath

permalink

My ruby version on my mac stopped working because homebrew updated libicu. Had to recompile/upgrade #fail

permalink

Today I upgraded node to 4.3.2 (from 4.2.2), python to 3.5.1 (from 2.7.3), and django to 1.9.4 (from 1.8.8). What fun! #win

permalink

Hyperbola staging now running python 3.5.1 #py3

permalink

I use #node in my asset build pipeline. I compile js and css with yuglify. Today I productionized my node install. Upgraded to latest LTS and added docs for upgrading node

permalink

Finally got around to getting my #django 1.9 upgrade branch over the line. Didn't have django-pipeline pinned and 1.6 introduced a breaking change to settings. All done now.

permalink

Yelp's search stemming needs some work. I searched for "used" and it is highlighting "us" in the search results. #fail

permalink

When I see L7 I think of the band instead of the OSI Model #fail

permalink

Caught an issue in staging today because staging is more like prod than dev is. Dev has pipeline disabled. Yay for release process. Yay for testing. #win

permalink

python -Wall is a neat trick. Cleaned up a bunch of #django deprecation warnings

permalink

Time to switch to using the #python SDK! In the meantime, no new media uploads

permalink

Media directory backups have been failing since the 17th due to exceeding max attachment size. No alerts. #fail >>> SIZE=55759470 <<< 552 5.3.4 Message size exceeds fixed limit

permalink

Switching from yui-compressor to yuglify saved me 1KB on bootstrap.js and 30 bytes on bootstrap+hyperbola css #performance

permalink

lifestream template refactor didn't yield any #performance wins, but the code is much cleaner now. #win

permalink

Front page warm nginx latency is 12ms; contact page is 20ms #performance

permalink
Post image

Current prod gunicorn time (measured at nginx) for the lifestream index page is 70ms with no partial caching and 40ms with the sidebar cached #performance

permalink

Tested rendering the lifestream index with all 3 interpreters. pypy is slower than python2.7 by 2x. python3.5 is faster than python2.7 by 2x. #performance

permalink

Local development lets me run cool experiments. I have python2.7, python3.5, and pypy virtualenvs.

permalink

Sped up the lifestream page by 30ms by caching the archive sidebar. More template speedups to be had by switching to python3 (str to unicode coercion in python2) #win #performance

permalink

Woo! I can run hyperbola on my mac. Local development! #win

permalink

Even better: precompiling app/hyperbola to pyc and getting to keep the deployment read-only. better #win

permalink

gunicorn workers get warm quicker now that I've allowed gunicorn write access to a deployment's app and virtualenv dirs #win

permalink

Root fs usage is down to 3.5G.

permalink

Finally started cleaning old deployments. After each deploy, my deploy script removes all but the latest 5 deployments for a given env. hyperbo.la/lifestream/352 can rest peacefully.

permalink

90mbps upload to Box from the corporate network is pretty nice when backing up my photos (all 25GB of them).

permalink
Post image

Did some archeology today to unearth an ancient version of hyperbo.la. Same color scheme as today. Yes, there was an assets domain. Had to do forensic analysis to reconstruct the index.html.

permalink

It continually amazes me at how much conflict is solved with "assume good intent."

permalink

Added a post-commit hook to my gollum wiki today to do a pull/push. Changes instantly synced to #github. No more stupid 5-minutely cron. #win

permalink

I feel like my #django site is futureproofed for at least the next two minor version upgrades now. Just converted a ton of render_to_response snippets to the new render hotness. Even feels faster.

permalink

Despite being stable, #django deprecates frequently. What they deprecate tends to have been the recommended way of doing it circa 1.4 ... feels like they are slowly rewriting everything.

permalink

15 commit night on hyperbola, 5 on hyperbola-tools, and 12 on my dotfiles. Mostly code and style cleanups, some correctness fixes.

permalink

Having a linter is so awesome. IntelliJ rocks. #win

permalink

Next major project is migrating to a new VM. 32-bit Ubuntu 12.04 is getting long in the tooth. I have eyes on 64-bit #Debian 8 and a shiny new #linode with 2GB of RAM

permalink

Last bits of major feature work were bootstrapifying several pages and adding automatic deployment (June 2014)

permalink

Just realized that hyperbo.la has been in maintenance mode for a while. Last few projects: upgrade to django 1.8, upgrade to django 1.7, upgrade vendored dependencies (bootstrap, retinajs).

permalink

#puppet is more fun when you know what you're doing.

permalink

It only took a debian VM and 4 hours of my time to reformat two new USB keys. #fail

permalink

Fixed a long outstanding bug in how the sidebar on the lifestream calculated monthly post counts #win Tickled by the upgrade to #django 1.8 which caused them to all display as 1. #fail

permalink

Finally upgraded to #django 1.8. Only an 8 commit adventure

permalink

Wondering why the response to some external asks isn't, "No, we're not ready yet."

permalink

Wow! usability fail. The crontab command operates in 3 modes, two of which are edit (-e) and remove (-r). Lovely that e and r are right next to each other on a QWERTY keyboard. #fail

permalink

This oncall cycle, I observed my first real hardware failure. A machine critical to the deployment pipeline spontaneously shutdown and failed to reboot, problems with mounting RAID, etc. #fail

permalink

We are loading the racks onto trucks and moving them to a real datacenter. Bye bye servers.

permalink

Responding "maybe" to a calendar invite is the equivalent of the SF "we should do lunch"

permalink

I want to live in a world where booting a machine doesn't cause config propagation to break #fail ghetto consensus algorithms

permalink

The only things that are truly unique in a relational DB are auto inc pks. Even uniques such as username may be duplicated across time.

permalink

I feel dirty doing pointer dereferences in #golang

permalink

One of the tools I've learned when writing #golang is how to coalesce errors behind an API boundary. Makes for cleaner code.

permalink

./unixtools gzcat ~/Desktop/f.gz.gz | ./unixtools gzip -d - ... having fun implementing parts of busybox using #golang. So far I've got gzcat, gzip, and yes

permalink
Post image

Services team get rekt #oncall pages from 9pm Friday to 8am Saturday ... something about the corp dc migration

permalink

So this is what a real oncall rotation feels like. So many things.

permalink

whoaaaaaaa ... an unsaved txt file from my work laptop showed up on my personal laptop. #cloud

permalink

This week at work I settled on a secret delivery mechanism that bootstraps with #kubernetes (or puppet if a service isn't containerized)

permalink

Just spent 5 minutes hopping through man pages trying to grok a script I wrote a while ago only to realize there were comments explaining exactly what was going on #fail #docs

permalink

took 1minute of downtime today for a #mysql 5.5 upgrade from ubuntu

permalink

still hanging around: 4GB of old staging installs and 2GB of old prod installs

permalink

Just did some disk usage cleanup on hyperbola. Had 4 old ruby installs hanging out (2GB), a 700MB gunicorn logfile that was being written to every second, and a very full apt cache (1GB)

permalink

even with a staging env, deploys still require a site check

permalink

I love Apple's "quit unexpectedly" euphemism for crash

permalink

... and I'm no longer dumping secrets to stdout due to `source` and `set -x` #12factor #fail

permalink

On the plus side my build script now dumps metadata about installed python packages to disk so I have one place to look instead of wondering

permalink

Finally subscribed to django-announce. I was running a super old version of 1.7 ... oops #django

permalink

Recently restructured my #dotfiles and added a Makefile so they don't live in $HOME (git clean terrifies me) #git

permalink

Playing with some JSON data and jQuery ... created a rudimentary VX points trip planner #javascript

permalink

I don't like structured identifiers because they presume data fits into a hierarchy

permalink

Borg paper was a fun read. Kubernetes is really exciting.

permalink

feast or famine on the hyperbola microblog

permalink

TIL: Selecting 2 cells and then dragging to repeat causes google sheets to infer the increment #win

permalink

#ruby Fixnum is a weird beast. A class that doesn't have new.

permalink

A good chunk of my qualms about #scala is complexity resulting from laziness tricks.

permalink
Post image

This is how you get a macbook to page like crazy #fail

permalink

OMG #ruby is using 19.22GB of memory executing a one liner #fail

permalink

Ran time ruby -e 'def is_prime n; ("1" * n) !~ /^1?$|^(11+?)\1+$/; end; is_prime 982_451_653' #fail keyboard events don't fire, gfx glitches, windows don't repaint, one core pegged

permalink

Just spent the last 25 minutes installing homebrew packages on my work machine. My code is literally compiling. Took the time to do some scooter laps.

permalink

Rebooted hyperbola for the first time in a year due to the libc vuln. Minor panic ensued. Gunicorn failed to start due to log file perms, /var/run/gunicorn was wiped and had to be recreated #fail

permalink

I've been missing #ruby lately. I should make something.

permalink

(function _() {console.log('('+_+')()')})()

permalink

Answered a phone call on my computer for the first time. That was so cool. #win

permalink
Post image

I have an album with a non-square album art in #iTunes. The UI renders it 3 different ways #fail

permalink

Upgrade to Bootstrap 3.3.0 went really smoothly. Everything just worked #win

permalink

#hyperbola is down to 200 lines of #CSS, half of which is used to render the logo and navbar

permalink

Finally bootstrap-ified the last bits of #hyperbola: the frontpage and the lifestream entry panel #win

permalink

I still don't have zero-downtime #MySQL upgrades. Lost a nine #fail

permalink

Just added CORS headers for web-fonts on my 2 assets domains through cloudflare ... whoa. This has apparently been broken for a while #fail #win #hyperbola

permalink

I do not understand Box Engineering's blanket aversion to a build step. So many generated artifacts checked in. #fail

permalink

Started going through the Functional Programming in Scala book today #scala

permalink

why does any C program of sufficient complexity have its own allocator? Just use jemalloc?

permalink

The admin site is slightly modernized in #django 1.7 ... looks nice #win

permalink

hyperbola, now with #django 1.7 ... I love how easy it has gotten to deploy this site.

permalink

Today I exceeded linux's max shebang length #fail it is 128 btw

permalink

I now understand the difference between teasing apart complexity versus simply moving it around

permalink

swift is the #scala to objective-c's java ... that switch statement looks an awful lot like scala's match construct

permalink

heh, spoke too soon ... #fail Had forgotten to update #nginx config so when I deleted the legacy deployments, I lost static assets and media.

permalink

live site was just deployed automatically for the first time. Switched from legacy deployment to automated with no downtime! #win

permalink

just did a dry run of a mostly automated hyperbola deployment. virtualenv, bash, git what fun :) Already staging and production have diverged. #fail next is to figure out how to bring it live

permalink

Oh no! just missed a deploy step #fail ... forgot to run pip install --upgrade ... new code was missing a dep

permalink

Woo! First 2 hyperbola releases with #git tags today #win. Also local development. dev + ops = #devops

permalink

Nothing like a little bit of #regex golf on a Saturday evening.

permalink

I just realized that when I first learned #java, there were no generics. Thanks Martin Odersky! #scala forever.

permalink

"It's Not Right but It's Okay" -- A poem by Whitney Houston about continually evolving software projects.

permalink

Today I learned more about sockets and ephemeral port exhaustion than I ever wanted to know. I normally take TCP as a given. #webscale #box #metadata #fail

permalink

Next #devops steps for hyperbola: Rebuild machine from scratch on 64-bit kernel; use #docker and dockerize all services: nginx, staging django, prod django, MySQL; real deploy step; #git tag releases

permalink

#django sure as hell doesn't make it easy to get at the database backend ... but the fact that I can is still awesome. Enabled post count badges on lifestream.

permalink

Hyperbola, now with retina graphics! The logo and feed icons will come through as retina.

permalink

Lack of agency is the thing I fear most as an engineer. It's something I need to get better at dealing with.

permalink

Ways you can tell we have a major release coming up: Being in the office until 10:30 and 9:30 on consecutive nights.

permalink

it is a rare day that I don't get home and try to type my work password into my laptop #fail

permalink

#metadata about to enter the War Room for 2 months ... #Box /dev brings deliverables. Let's go!

permalink

I have no idea why, but every time my backup disk spins up on my laptop, my entire system hangs for a few seconds #fail

permalink

got another big #performance win by not minifying HTML in python not doing so shaved ~100ms off response time which means my server was spending more than 100ms of CPU time for the pleasure #fail

permalink

about ~20% of server time for lifestream is spent reversing URLs for hashtags :/ #performance #fail

permalink

Latency at nginx for hyperbola: frontpage: 25ms, contact: 40ms, lifestream: 200ms #performance

permalink

Switching gunicorn to a domain socket instead of a TCP port sped up the site by 2x #performance

permalink

Hyperbola, now with more html5-boilerplate and bootstrap ... it made CSS so much easier than I normally find it to be ... and the site is now mostly responsive and looks good on mobile

permalink

Spent the last week optimizing, re-styling, and significantly refactoring hyperbola. The lifestream page loads ~300ms faster and DOMReady is ~600ms faster #performance

permalink

Get optimized, HTML! Just added some middleware that minifies Django template HTML output #performance Google pagespeed really likes my site now (on Desktops) ... 94/100 #win

permalink

The last bits of the #hyperbola deployment are gone from my home directory ... finally productionized a cron to do a #database and media backup #devops

permalink

Most interesting learning from 2013: when it is appropriate to resist my engineer's temptation to build the perfect thing. #lean #failfast #box #metadata

permalink

I'm getting ad targeted for foundationDB on YouTube during my vacation ... are we using it yet? #box #metadata #database

permalink

Needed: #mobile optimized #django admin skin. Posting from my iPhone is less than ideal

permalink

wow link tags and auto RSS/Atom feed discovery ... welcome to the internet Ryan ... #fail ... I feel like #django is missing a template tag here.

permalink

#nginx reverse proxying to django with HTTP/1.1 ... wish this config option would be the default.

permalink

#PHP is just netcat ... the language of the web does not have a good HTTP abstraction. I'm envious of golang's HTTP package

permalink

And with about 30 minutes of downtime I've now got 2x the ram #devops

permalink

running alter tables in production is terrifying #mysql ... just converted from MyISAM to InnoDB and added a bunch of #indexes

permalink

#django 1.6 upgrade was a breeze. Just a single deprecation warning. #win ... took the opportunity to refactor some old code too :)

permalink

so my resume link was broken for six months #fail stupid missing line in #nginx conf ... not enough exploratory testing during the refactor. What good is #staging for anyway?!

permalink

Actors are overrated. So far, we've preferred to use future literals and ExecutorServices instead #scala #box #metadata

permalink

Finally added an upstart script for my gollum wiki #linux #ops One less thing to worry about when dealing with a reboot or upgrade

permalink

Only minor drama when upgrading hyperbola to 12.04 LTS ... #mysql failed to cleanly upgrade to 5.5 and I forgot the correct URL for my staging environment #linux #fail

permalink

Part of pushing #metadata to production meant we got to puppet our service across dev/staging/perfpod/production. #puppet is pretty cool #box #devops

permalink

6 months in the making, #box metadata made it to production today

permalink

The bare minimum I need on a server to feel comfortable is `set -o vi` #vim #bash

permalink

Throughput on the #box #metadata backend is limited by the rate at which our load driver can make requests. 2ms latency at the client. This thing is #fast #performance

permalink

Rule number 1 of operating at #scale: Your assumptions are never universally true.

permalink

Upgraded #django from 1.4 alpha to 1.5.x stable tonight. It only took a couple hours. Mostly dealt with deprecated APIs and new template tag syntax

permalink

One of PHP's biggest problems is that it is too easy to create associative arrays. It discourages people from making proper objects. #php

permalink

mac #safari can attach to webviews running in the iOS simulator and open a web inspector. This is cool but ... what? #debugging

permalink

#databases Two things that amaze me: (1) idempotent writes, (2) idempotent and reorderable writes

permalink

I am super excited to see the results of the move to the T3 #javascript architecture at #box

permalink

The Fundamental Theorem of Software Engineering: "We can solve any problem by introducing an extra level of indirection."

permalink

It makes me happy when I see people using If-None-Match sprinkled throughout the splunk logs at #box #win

permalink

just installed 500MB of software for a scanner #wtf #fail

permalink

I remember a few years ago when my primary reason for staying on #firefox vs #Chrome was that I preferred how Firefox highlighted text on selection. #fail

permalink

I have no idea why, but since I've been working at #box, I don't #git pull. I always fetch and merge --ff-only or rebase. Only time I ever use pull is via pull --rebase if I get ff-rejected on push

permalink

Participated in my first 24 hour hackathon last Thursday. It was awesome. Highlights: food trucks, caffeine, kreayshawn, and awesome hacks #box

permalink

Just forked, edited code, submitted a pull request, and got it merged all from the browser #win #git #github

permalink

I had a cronjob launching every minute that updated remotes on about 40 #git repos. I've apparently had ~50 git processes running all the time for about a year now. #fail

permalink

I can't believe I ever liked #flex. To be fair though, #xcode and interface builder are basically the same thing

permalink

Running unit tests is the equivalent of waiting for code to compile.

permalink

Just updated the firmware on my Air. First two reboots = #kernelpanic. 3rd reboot = no mouse or keyboard. 4th reboot, everything works? #fail

permalink

9 months since the last reboot isn't too shabby. And now I have a 3.5 kernel!

permalink

#metaprogramming is awesome. In a new project, adding an API endpoint is as simple as defining a single constant

permalink

Changed my terminal and vim fontsize from 14pt to 24pt at work. zOMG so much better

permalink

alias jsonpp='python -mjson.tool' greatest alias ever for #json #api development

permalink

So I have some autocmds in my vimrc already, but tonight I wrote my own. It detects macruby in the shebang of a script and sets a var to tell syntastic to load a different syntax checker #vim

permalink
Post image

#versioncontrol, or the lack thereof. This was my first major software project. I was a junior in high school. I was making a symbolic math thing. #fail or the reasons any CS class should teach #git

permalink

#Songza's use of high-res album art alone makes the experience feel better than #Pandora. Add in the fun way of selecting playlists and we have a winner

permalink

songza.com Not only is it awesome, but logging in doesn't stop my music from playing. sweet #ux

permalink

My wiki was spammed and turned into a link farm for discount pharmaceuticals. Good thing its backed by #git -- fixing it was trivial

permalink

#notetoself: Next time you need to run more than 3 tasks that take 10 minutes or more to run, invest in figuring out how to script it #fail #thesis2012

permalink

A watched pot never boils. Neither does a long running task when you're not tailing the log file. #thesis2012

permalink
Post image

I wanna give a big shoutout to the #java jvisualvm, without whom figuring out my perf problems would have been much harder

permalink

#lessonlearned: Don't create a billion Timer objects. One is probably enough. Managed to get #eclipse to fail at repainting itself because it was running about 700 threads #java #fail

permalink

I feel like serializing procs to strings, passing them over HTTP and evaling them is an abuse of #ruby, but it is awesome that I can #win

permalink

I do not care whether your API is RESTful. Does it work and will it do what I want? #marketing #fail

permalink

I remember when I first upgraded from notepad to scite when I first installed #ruby. You had to modify this conf file to get monospace all the time ... #vim is so much better #nostalgia

permalink

just discovered that once you're in cmd+TAB mode on OS X, cmd+` cycles backward. So much easier than cmd+shift+TAB #learnability? #fail #win

permalink

#golfed OR "why use ruby -e when there's uniq -c." It's 40 chars shorter than before: cat ~/.bash_history | cut -d" " -f1 | sort | uniq -c | sort -rn | head

permalink

Is it weird that my web server backups run at 4:04 AM? #lolz

permalink

You mean I have to implement hashCode()? I guess that means I'm doing some string concatenation #easyway #java

permalink
Post image

Cute ad on stackoverflow. It reads "<heart> Your Job" #adtargeted (apparently my db doesn't like unicode #fail)

permalink

That feeling when you go to empty your trash and you realize that there were over 14k files in it that account for 15GB of disk space. HOW DID THIS HAPPEN? #fail

permalink

So now that I got #airdrop working, I like it. It doesn't handle large transfers so well though. Transferring a gig of #music files has taken 15 minutes so far and I'm only 2/3 done. #fail and #win

permalink

I'd post about how #adtargeted I feel every time I see a New Relic ad, but it'd only make it worse

permalink

Process for rendering pset: Type in vim using markdown > render to html and open in chrome > print to pdf #doingitthehardway

permalink

After a struggle to import data from Firefox, I've switched to Chrome #thingsthatarenew The only extensions I'm running are 3 greasemonkey scripts I can't live w/o

permalink

I've never had to implement a hash table before #spoiled by standard libraries

permalink

a productive day: did something I've been meaning to do for a while: a playlist queue for iTunes #ruby

permalink

A productive day: made my PS1 better and fixed a bug in the hashtag regex on hyperbola (again)

permalink

Something something something #git ... something something something complete

permalink

Nokogiri::XML(file_bigger_than_1GB.xml) is a bad idea #fail

permalink

Thank goodness xml compresses so well

permalink

I dislike experiencing bugs in my day-to-day usage of software I paid for. Saints Row: The Third and iPhoto, I'm looking at you #fail

permalink

cat ~/.*h_h*|cut -d" " -f1|ruby -e'h=Hash.new 0;h[$_]+=1 while gets;h.each{|k,v|puts"#{v} "+k}'|sort -rn|head #golfed This snippet prints out my most frequently used commands

permalink

#vim modelines are the best. Just added filetype modelines to all of my wonkily named #bash config files. Syntax highlighting rules

permalink

I don't understand why people think they need build scripts for their dotfiles. #fail Just keep your $HOME in #git: git init && git remote add ...

permalink

This is the longest-lived and most consistently updated #blog I've ever had. Averaged a post every 2.5 days. I still post to #twitter more often. #socialmedia

permalink
Post image

My most frequently used commands on my laptop

permalink

Switched to solarized color scheme in terminal and #vim. Everything looks different, I love it though. I'll get used to it. Also, Inconsolata rocks.

permalink

so after hacking around in #objc all day, the syntax is starting to make sense. I actually kinda like it. #macruby is so awesome though.

permalink

#firefox dims the favicons of inactive tabs. That's so nice.

permalink

printed over 600 pages in an athena cluster today #win

permalink

#hack to fix someone else's #PHP #fail foreach ($_POST as $key => $value) { $_POST[$key] = mysql_real_escape_string($value); }

permalink

what I've been working on: sandboxing #chrome extensions

permalink

apache, quit installing sites into my sites-enabled directory. You're taking my site offline #fail

permalink

a CSS rule to make images sane in fluid gmail: html > body > img { max-width: 100%; max-height: 100%; }

permalink

I have iTunes autoname my files based of their tags. Apparently if a track starts with "(", it won't prepend the track number to the file name. #fail

permalink

#music curating has reached a new milestone. All albums in iTunes are complete. Next, get rid of all those 128 kbps tracks.

permalink

* 4 * * * means every minute of the 4 o'clock hour #cron #fail

permalink

I first used Charles this past summer. A very valuable tool for network observation and debugging charlesproxy.com

permalink

Google plus, you are no longer app tabified. slow #fail

permalink

Why dynamic languages suck sometimes: I spent the last 45 minutes debugging a python script. I thought I had logic errors. What actually was wrong was that I renamed a variable and missed one. #fail

permalink

when you're at pid 400000, it's probably time for a reboot #unix

permalink

I don't know why, but I am excited about home automation today. Z-Wave looks really cool

permalink

I have 3 spare computers lying around. What should I do with them? One idea is to turn them into a cluster with one node running a shared kernel for the entire cluster. #rpc #linux

permalink

6 doesn't byte 00110110 00100001 00111101 01100010 01111001 01110100

permalink

#backintheday, before I understood how the relational part of MySQL worked, I made 20 columns to hold metadata about a record. 20 columns wasn't enough. Foreign keys #ftw. learning #fail

permalink

So on this one server, my default shell was /bin/sh, i'm not a sudoer, root owns my home directory, and another user owns /home. #linux #fail

permalink

so in order to access Mail.app's preferences to make it not be the default mailto: handler, I first have to configure it #fail

permalink

Seriously, Office? You rm the backup from the disk when I close it instead of doing the sane thing and moving it to my trash? #fail #hard

permalink

why does iChat think it's ok to grow as large as it wants vertically? #no, I'll manage my windows by myself.

permalink

its amazing how much quicker a script runs when you don't create 60,000 Hashes and compile a regex 30k times #ruby

permalink

Because kernel.org is down, I used airdrop to copy over a cached version of the git source and formula from another computer so I could install it with homebrew #fail

permalink

#UPnP stopped working on my router. Flashed it to fix.

permalink

I got to 100 #tweets a hell of a lot faster than I did here (still not there yet)

permalink

So I used #github socially to review someone's code and got flamed. That's the last time I do that.

permalink

Abiding by the principle of least surprise is, to me at least, least surprising.

permalink

here's the #plan: 63 units, 6 classes, 3 of which are grad H. It's gonna be a bumpin term

permalink

for the number of times i ssh'd into my iMac this sumer, you would think that I'd have just put my ssh key on it #fail #doingitthehardway

permalink

I was not able to access google plus for about a week. Figured they were just having issues because I was still able to access email fine. Ended up having to recover my account. #fail

permalink

It's been real California; can't wait to come back

permalink

So #rails caches column names across the database and generates SQL for them even if they are never used. Why does this make sense? #fail

permalink

Finished my internship at #ooyala. Had an awesome summer; learned so much.

permalink

Just demoed my Custom Events analytics project at #ooyala. It went well: "This is the future of freemium analytics."

permalink

RSS and Atom feeds now render posts the same way I render them on the lifestream page. Thanks to my one reader derenrich for the feature request.

permalink

next project: install rvm and homebrew

permalink

And so I wait for the dns change to propagate ... soon, we'll be live here from NJ

permalink

Disabled #root login over ssh. Have no idea what the password is. This is the #ubuntu way.

permalink

I haven't posted about #music in a while: #songkick is the best. I'm going to so many more concerts now. I get too immersed in #turntable for it to be a regular thing.

permalink

first post on the new server. It runs. I'm trying to make this #django thing leaner and less hacky. Still have a ways to go. Boy do I need to set up a dev env.

permalink

What a busy friday. Switched to iTerm. 256 colors is nice. Set up a git repo with all my dotfiles. Got real #serious with my #bash config.

permalink

got a new #server. It's in Newark. It's some lonely VM. It's time to get serious.

permalink

finally got around to giving my #vim config some love. MacVim is pretty cool.

permalink

#mashup #idea: quova + mixpanel. Give it the ability to scope by region/state/country/county

permalink

the hyperbola repos have been migrated to github. and so begins the death of dev.hyperbo.la

permalink

After working with #rails for a bit now, I kinda like the #magic. DB interations seem so much more natural than they do in #django

permalink

oh #quora, you sly kitten, requiring me to sign up to view all the answers to a question

permalink

Next birthday present to myself: CrashPlan+. Offsite #backup of all of my media? Yes, please.

permalink

Thinking about migrating this site to EC2. Probably gonna stick with #django. dev.hyperbo.la will probably die; github does it way better. Moving all media assets to s3: maybe. ssl: probably not

permalink

checked my Activity Monitor, had 2 #ruby processes each at 90% cpu which had been running for hours. trapping signals #fail

permalink

open("|-"), you are a tricky beast. Successfully fork bombed myself 5 times before I sorted that out.

permalink

why `svn status` prints nothing when you have a clean working directory is beyond me. #git ftw

permalink

hyperbola, now with 500% more cache-control headers

permalink

I need to get around to resetting up this server. I want to switch from svn to git and redo the whole apache config. why did I put the site in /var where I have to sudo all the time?? #linux #fail

permalink

one day I'll set up a #django dev environment again. Until then, we'll do it live!

permalink

Finally fixed the css on my site to be less of an abomination. width: auto, you are wonderful

permalink
Post image

rebased some already pushed commits to clean up my history and forced the push. I feel so dirty. #git #fail

permalink

wrote my first MapReduce job today!

permalink

have access to one of ESPN's private repos on github

permalink

finally broke down and got a github account. My username is lopopolo

permalink

Should I upgrade 10.04 LTS to 11.04? #linux

permalink

this weekend's project, get all my config files on github

permalink

vi mode is the best thing I have ever discovered about the command line

permalink

pushed my first bugfix today!!!!

permalink

lesson learned: never muck around with mysql's database directory; you will screw up #fail #linux

permalink

for some reason every part of the #django admin worked except the root page. had to turn on debug mode to see I was improperly configured #fail

permalink

I got accepted to the EECS MEng program!!! woooo!!

permalink

sysadmin ing is fun. hello beautiful browser based svn repo browsing (insurrection) and web-based svn admining (svn-admin) #linux #win

permalink

broing out is somehow making studying for algorithms not suck

permalink

its amazing how the time melts away when you're playing portal

permalink

unintentional side effect of my server's firewall: I can't traceroute it. #fail

permalink

it's amazing how the time melts away when ur mucking around on a server

permalink

I need a 512GB iPhone

permalink

killed my own email solution because I had no confidence it would work reliably. Rolling with Google Apps now

permalink

lol. edited the site live again while working on a new app. 500 errors abound. #django #fail

permalink

got my offer today. will be working for ooyala for 3 months this summer.

permalink

#idea a new NFL stat - average number of plays to achieve a first down (both intra- and cross-drive). I think it'd be interesting to compare this to average drive lengths, points/drive, and defenses

permalink

Found out that my raid array not mounting a couple months ago was due to a failed drive. Can't recover it. I lost so much data. #linux #fail

permalink

relaxing during IAP. I've started working on a side project: a flash-based dj mixer that uses #music from youtube. #flash

permalink

This is why I like django's pluggable apps. I can get distracted and not finish a project, but the site still works. #django #distractions

permalink

Just set up three HD cable boxes and they look sweet. #tv

permalink

working on a sort of app store for projects i've done, a glorified portfolio

permalink

lots of code refactoring going on right now. One drawback of the ease and speed of #django is that it allows you not to think.

permalink
Post image

props to a friend for giving me some tips on how to make my logo better. this is the tweaked version.

permalink

Had a bad run-in with rm -rf. lost the sqlite database for the site. on the bright side, I'm using mysql now. #linux #fail

permalink