azurefire

Almost Awesome

2016-02-02T10:35:02+00:00

I feel like most things that I build fall into a state of perpetually being almost awesome.

If you give me a problem statement for a piece of software - even a vague one - I’ll usually quickly establish what I want to build and get excited about it. Designing systems is fun. There’s nothing quite like that feeling when something that I shipped makes a difference for someone, even just a small one, like removing a bit of friction from someone else’s day. I also do my best to ship features as soon as they’re remotely useful, to keep momentum going and to establish a good feedback flow.

The problem is that it’s much faster to envision a complex system than it is to actually write one; the brain always lags the fingers. The system in my head is inevitably more complete, less painful, and much cooler than the one that my users can interact with today. I never look at what I’ve shipped and think, “this is awesome, this works.” Instead, all I can ever see is the delta between what exists and what I want to exist.

It’s an unfortunate trait of my brain. It means I spend a lot of time feeling impatient with myself and frustrated with what I haven’t built. It’s a great way to have a lot of drive, but it’s also why I have to be very careful to do a healthy amount of non-software things in my life, to stave off stress and burnout.

Don’t Try to Fix People with Process

2015-10-01T00:00:00+00:00

Process can be a substantial source of friction on a team, but it doesn’t have to be. It’s a necessary and important component of building anything, but at its worst, it can feel like a series of arbitrary hoops to jump through before you can get anything done, or even be outright hostile. Keeping your process useful without ticking everyone off and killing velocity takes vigilance, care and attention.

The biggest pitfall that I’ve personally witnessed is trying to use your process to fix bad developers, and it’s poison to a working process. You’re in a conference room with a Word doc on the screen trying to hash out the steps that developers need to follow to make changes, and somehow the tone subtly shifts to “how do we keep these idiots from breaking everything.” It can happen even when the developers are crafting the process themselves or have heavy input; my speculation is that that’s some mix of lacking self-confidence or passive-aggressive sniping against teammates.

I’m exaggerating a little – if anyone actually says something like this, you’re already doomed. More realistically, symptoms of this include long checklists of peer reviews, voting systems, and lots of cross-validation and extra sign-offs. At some point, you need to admit to yourselves that it isn’t that you’re managing risk and preventing bugs from shipping, it’s that you don’t trust the rest of the team who are building the actual product, and trust is one of the most important ingredients in a functional team.

Some concrete things that you can do to counteract this:

Make mandatory steps voluntary when you can. Let a developer choose when a pull request needs extra review before it’s merged. You’ll be surprised how often your devs opt in to peer reviews, especially once they’ve seen it catch a few subtle bugs, and they’ll be greatful when they just want to fix a typo in the README.
Optimize for shipping, not for catching all bugs before they reach production at all costs. Like most decisions, there is a spectrum here, but it’s very, very easy to fall closer to the “preventing bugs” side than you need to be, and that too has a cost. No matter how many barriers and gates and checks you impose on your team, developers are human, and mistakes will make it through at some point. The more hurdles lie between a developer and their code being live in production, the longer those bugs will stay in production before fixes can be applied.

The real equation at work is:

(frequency of bugs shipped) * (damage from each) * (time to fix)

Taking more care can reduce the frequency of bugs shipped but only by increasing the time to fix. Sometimes that’s appropriate, but be mindful of the full picture. Note that giving developers more opportunities to use their own judgement on a task-by-task basis can also bring you closer to an optimum solution here.

Automate everything you can. Use continuous integration to catch test failures instead of just mandating that everyone run the test suite by hand. Link issues to pull requests and automatically cross-reference and close them when the code is actually merged or shipped. Pull reports from your ticketing system rather than make everyone email you what they’re doing all the time.

Each time you introduce a human into the workflow, you’re introducing a time delay, increasing social complexity, and adding a point of failure. A human being is far more likely to forget to run your integration tests than Travis is! The more work that you can push into infrastructure instead, the more accurately and expediently your process will be followed. Of course there are times when slowing you down is the point – peer reviews, for example – but there are many more opportunities for streamlining.

Essentially, automation lets you increase the amount of rigor you introduce, while keeping the prices of time and complexity low.
Revisit and refine often. Listen to your team, watch how you work together with a critical eye, and only keep the bits that are actually adding value for you. There’s a strong pull, as the one who writes the process, to add more process over time; fight the instinct and instead bias toward cutting as much of it as you can get away with.

The downside here is that changing too frequently can make it hard for everyone to keep up with that they’re supposed to be doing. I’d counter this by building process updates into the natural planning and shipping cadence so that everyone knows when to expect changes, and by keeping it lean enough to fit in a single page. You could even keep a workflow description in a CONTRIBUTING.md file alongside your source and propose changes with pull requests and take advantage of all of the existing notification and discussion machinery you use for source code to iterate on process, too.

Realistically, you should converge on something that works for your team and your project fairly quickly; I’d expect big changes that everyone has to relearn only after lots of people join or leave, or after major shifts in project direction, when things are chaotic anyway.
“Fix” developers with mentorship and communication instead. Process is a tool you can use for coordinating efforts, planning, and communicating outward; teaching, pair programming, and one-on-ones or just pinging someone in chat are the tools that you can use to deal with problems.

You’ll never be able to put together a magical flowchart that you can follow to turn a problem coworker into a helpful one. People are individuals and it’s the relationships that you share that help you all grow.

The bottom line is that it isn’t your process that’s creating software, with a group of engineers following it like automata to reach a predictable outcome. It’s a team of human beings using creativity and discipline and judgement, and because they’re writing the actual source, you’re already trusting them deeply. They all want to ship something that works, that they can be proud of. Remember you’re on the same side!

Cloudpipe: Compute on Demand

2015-01-30T00:00:00+00:00

When you want to execute a long-running or massively parallel process in a persistent way, you have a few options. You can launch a server, install a bunch of software, and stitch things together with cron. Or, you can install a Hadoop cluster, rewrite everything from the ground up to use map-reduce or Spark or another powerful concurrency framework, and orchestrate your job with that toolkit.

Those options all have their place, but sometimes, all you want to do is write some code and run it somewhere. No server spin-up, no system administration, no patches and security updates. Kyle Kelley and I spent a few weeks around the holidays putting together an open-source project called “Cloudpipe” to accomplish exactly this. Today, we have a proof of concept we can run that successfully executes jobs, and we’re hard at work filling in the blanks in the architecture and putting together a real deployment.

Using it

To start, we chose to target an existing client SDK, Python’s multyvac, which was nice for a few reasons:

It gave me a concrete spec to target while I was mapping out the endpoints for the API server. This saved us a lot of bikeshedding in the early days.
It let me start backend-first and test things with real code, just as a user would experience them. Otherwise, I would have had to use awkward curl scripts until we could design and put together a decent SDK.
In theory, people are already using the SDK. (If that search is any indication, though, we should have stuck with its predecessor, cloud, instead.)

As a result, the only client SDK that exists today is for Python (and Python 2, at that). That’s going to change, though - we’ve been planning to write our own client for Python and Shaunak Kashyap is already working on an SDK for node.js. We’re also likely to diverge more and more from multyvac as we move on.

Once you have an account running on a cluster somewhere, you can install multyvac with:

pip install multyvac

Then, you can submit functions as jobs:

import multyvac

# Connect to the cluster
multyvac.config.set_key(api_key='username',
                        api_secret_key='54321',
                        api_url='https://somewhere/v1')

def add(a, b):
  return a + b

# Submit the function as a job. Args and kwargs will be
# passed to it server-side.
jid = multyvac.submit(add, 3, 4)

# Use the job ID to get a Job object...
job = multyvac.get(jid)

# ... and use the Job object to get the result.
# This call will block until the result is ready.
result = job.get_result()

Multyvac uses some neat pickling tricks to send your function over the wire: you can actually send closures, or code that references external libraries, and it’ll work. Other languages’ SDKs may not be able to pull the same range of tricks, depending on how evil each runtime lets us be, but they’ll support the same basic idea.

One important thing to notice is that the way work is submitted to the API is completely language-agnostic: it’s a contract between the SDK and the base Docker image that’s invoked on the server. The Cloudpipe API speaks strictly in terms of POSIX things: command and arguments, environment variables, and standard input, output and error streams.

How it works

Behind that URL, Cloudpipe is built from a small set of parts, all written in Go.

cloudpipe/cloudpipe

The API server is an HTTPS server that hosts a JSON API. It’s a fairly thin layer that performs CRUD operations on job models persisted in a MongoDB instance.

We’re using Mongo because it’s convenient for us, by the way: storage operations are performed through an interface, so implementations built on other databases are possible and welcome.

The job runner is a separate goroutine within the same process as each API server. It polls a queue of submitted jobs in MongoDB and uses the go-dockerclient library to launch each one. The runner creates a container from the requested image, attaches to its stdin, stdout and stderr, and starts it. Any provided stdin is fed in to the process; stdout and stderr are collected and used to update the job’s model in Mongo as it executes.

In development mode, the container is launched on the same server that you’re running everything else on, but in any “real” setting user jobs should run on a different machine. We’re planning to use swarm as a way to manage submitting jobs to a cluster.

cloudpipe/auth-store

Under the hood, the API server talks to a pluggable authentication server over HTTPS on an internal network, secured with a closed network of TLS credentials – meaning, the authentication server only accepts connections from clients who offer certificates signed by an internal certificate authority that we (the cluster operators) control.

The authentication protocol consists of two calls. The reference authentication server, cloudpipe/auth-store, implements this with interally-stored usernames and passwords. It also offers an externally-facing API that can be used to create and manipulate accounts, but that’s all extra. We can also plug in an authentication backend that talks to Rackspace identity to use Rackspace credentials, for example, or implement one that speaks OAuth.

cloudpipe/runner-py2

A runner image is used to spawn a container that understands what the SDK is sending in and executes it as a non-root user. The Python one unpickles stdin and calls the resulting function, writing the result to a fixed location on the filesystem. Not bad for four lines of code.

Finally, all of these services are themselves running in Docker containers, because why not.

Here’s how it all fits together:

Into the Future

What’s happening next? To begin with, we’re working on a simple web UI to provide a friendlier way to get your API key, create user accounts, monitor your jobs and so on. Shaunak is working on our first non-Python client SDK, for Node.js, and I’d like to start laying the groundwork for a Ruby SDK Soon (tm).

I’m also interested in extending the base API to support:

Configurable sources of input and output. Run jobs to and from a Cloud Files container, or a database, or mount volumes into your container’s filesystem from a block volume.
Constructing derived images from the provided base images.
Job lifecycle management. Run jobs on a schedule. Save jobs and re-run them by name.

Additionally, we’re going to start writing some Ansible roles to help bootstrap servers as API hosts, authentication hosts, or user job hosts.

Want to get involved? Dive right in – Pull requests are always welcome!

Implicit Knowledge and Shaving Yaks

2014-09-01T00:00:00+00:00

Let’s say that you want to start contributing to an open-source project in a programming language that’s new to you. If it’s written in a modern language, it’ll be easy to get started, right? It’s a healthy project with an organized codebase and a solid base of tests, so all that you’ll need to do is clone it and start hacking. You’ll have a lot of real-world, idiomatic examples of source code to use as a reference, so between that and the tutorials you’ve just read, the language itself shouldn’t be a barrier for long, and that’s the hard part, isn’t it? Just look for the phrase “it’s easy” in the setup bit of someone’s README. I’m as guilty as anyone, here.

Of course this is a complete fiction. In reality, whatever your development environment du jour happens to be, there’s a daunting tower of auxiliary software and must-have utilities that you’ll need to install, configure, and learn before you can get started. (If you’re unusually lucky, there’ll be only one tower, and its pieces will have good SEO.) Each piece will make assumptions and follow conventions that are as invisible as air to those who live and breathe in these circles. Look carefully at any programming community and you’ll find pockets of tribal knowledge that few bother to document in a CONTRIBUTING.md, but that somehow define the elusive core of what constitutes a “sane” development environment.

Take Ruby for example. If you want to write Ruby you can just slap some code in an .rb file and run it, right? Well, not quite. Before you can start building something real, you’ll need to know:

Nobody actually uses the system Ruby, if you’re even on a system that ships with one. Any Ruby you get from your local package manager will be woefully out of date. These days it’ll at least be something in the 1.9.x line, although if you’re stuck on an older CentOS box you might get 1.8.7, which is a decade old and completely unsupported, even for security updates. Yikes. Instead, you’ll want to use a version manager of some kind, like rvm, or rbenv, or chruby. Which one is better, you ask? Hope you have time for a few hours of hunting around FAQs trying to understand what they all do well enough to understand the differences.
If you want to use anything but the standard library, you’ll need to learn about the package manager, RubyGems. Fortunately, you probably won’t need to dig too deep right away, since you’ll most likely be using other people’s gems for a while before you start writing your own. So you’ll learn how to install gems, and where to find new ones, and what to do when you see the dread message, “Building native extensions. This could take a while.” You’ll learn that you must use sudo if you didn’t follow the first point and you’re running a system install, but you can’t use sudo if you did, and that it’s a silent failure if you used it and you shouldn’t have. Maybe you’ll also learn to set up a .gemrc file so that it won’t spend minutes building documentation you’ll never see every time you install a package. Okay, great. Except…
Now you’ll learn that nobody actually uses gem on its own. Or rather, you’ll use it exactly once: to install bundler for dependency management. You’ll learn how to write a Gemfile, so other programmers can use the same set of dependencies that you use, even if they also work on other projects that have different versions of the same dependencies. You’ll learn about the Gemfile.lock file, and when you should commit it into source control, and when you shouldn’t. And you’ll learn the set of bundle commands that you need to know, to install everything you need for a certain project, and to use that project’s dependency set to run programs in that project.
All bets are off if you’re on Windows. Use pik maybe?

I’m picking on Ruby here because it’s what I’m most familiar with, but every language I’ve ever learned had its own unique ecosystem that newcomers need to learn before they can get to the point where they’re actually running tests and writing code. Python had a cavalcade of different package managers – just a year or two ago, you needed to use one, easy_install to install the other, pip – and a version split a mile wide between Python 2 and Python 3 that makes the answer to the question “which version do I install?” a noncommital “it depends.” There are eight ways to install Node.js on a *nix system, some of which involve things like chown’ing your /usr/local directories; the node frontpage only links a tarball. Go has its own very opinionated set of workspace directories and environment variables to work properly that you need to understand before you can build anything. C#, being steeped in Microsoft-monoculture, has a relatively consolidated toolbase, but not even Windows actually ships with the tools you need to build it, and its de facto package manager is another download you need to know to install. And so on, and so on.

We haven’t even gotten to more fundamental beginner issues like “using git” yet, or “picking an IDE.”

The astonishing thing is that we often don’t even think about any of this. Once you’ve been working in an ecosystem for a while and have internalized all of the latent background knowledge, all that you see is the code, and the rest fades out of the picture. If I was handed a brand new laptop tomorrow, I’d have all of this set up on it without a second thought. It’s oddly easy to claim that it’s simple to understand things that you already know.

What I’d like to see is a solution for making this implicit background knowledge explicit, to speed and guide the ritualistic yak shaving that every newcomer must encounter before they can be productive in a new environment. It’s hard because, as a developer, my first instinct is to write a tool to accomplish it; but ultimately, all that would accomplish is adding one more yak to the herd.

In the meantime, I’m going to at least try not to claim that it’s easy to get started.

One Month

2014-06-29T00:00:00+00:00

As of Saturday I’ve been a parent for one whole month. I’m pretty sure that makes me a seasoned expert.

Really, though, as babies go she’s been pretty easy on us. We’ve had some bad nights but she’s also already slept six or seven hours at a stretch. During the day she naps and she eats and she burbles. She fusses sometimes, but not too much. She loves to take walks, wide-eyed and quiet and staring, and she’s even starting to be marginally interested when her Dad reads to her.

I’m aware that there are half a million milestones and thresholds on the way that’ll make all of our lives challenging for a while. I’m not looking forward to the whole “teething” thing, for example. So far, though, this parenthood thing is pretty nice.

Fatherhood

2014-06-11T00:00:00+00:00

Our daughter was born eleven days ago. Right now, she’s napping happily in her bassinet, so I thought I’d take a moment to write some early impressions about parenthood. I haven’t been a father for long, which on one hand means that I haven’t had a chance yet to mess up in some hilarious and interesting ways, but on the other it does mean that I still remember what it’s like to not be one, so hopefully I’ll have an interesting perspective just the same.

We were very fortunate in having a complication-free pregnancy, for which I’m greatly thankful, and a complication-free birth as well. That’s not to say there wasn’t plenty of trauma, of course. I think we broke every bit of advice from the classes we’d taken during the course of labor, but… now we have a beautiful baby girl, in perfect health, and Jayne’s on her feet and recovering rapidly, so, hey.

In a bizarre way, I think I’d almost managed to forget (or repress) that we were having an actual, human child, even while we were in the delivery room. Then, at the end, labor was over and there was someone else there, looking at me. It’s… quite a moment. Also: for months, everyone tries to prepare you by telling you that newborns are ugly as sin. It’s true that right out of the womb they look fairly inhuman, because in that moment they’re caught half-assembled, in a way. Then she kind of popped together and her color filled out and our daughter was gorgeous within minutes of birth. Take that, other babies.

Somewhat selfishly, my greatest fear before she was born was dealing with the lack of sleep. It’s the first thing everyone warns you about in those “oh you’re about to be a parent” speeches, I think because it’s one of the few constants: newborns have tiny stomachs and are usually nocturnal until they adjust, hence the late-night antics. It’s also a relatively concrete thing for the good old anxiety brain spiders to latch on to in contrast to the more nebulous fears about things like “teaching someone how to be a compassionate human being.” This past week we’ve been getting anywhere from two or three to six hours a night, usually nonconsecutive, so it’s not a lie. The bizarre thing is that it just doesn’t seem to matter – and I’m a guy who’s always liked his sleep. Adrenaline carries you through the first few days, and after that your body just seems to adjust to it. I feel the fatigue in those moments when I do put my head down to catch a quick nap, but once I’m up I barely seem to notice it. I will say that, while we weren’t gifted with one of those precious few who sleep through the night the day you bring them home, our daughter has been quite lenient on us in this department. She does fuss, but only occasionally. Maybe we’ve been unreasonably lucky, but “hell week” wasn’t so bad for us.

Next week, I go back to work, and that brings with it a new set of challenges. With the way things have been going so far, though, I’m optimistic that we’ll work it out without too much of a crisis.