Growing into Production

In my entry on “Measure, Design, Build,” I talked about the prototyping process: how we get from data, users, and an interesting problem into a workable prototype. What’s the next step?

Optimally, this process becomes a step in organic growth: bringing in the additional skills that we need to make the new feature real, one step at a time.

Growing the SLO project

When Liz Fong-Jones and I created the Honeycomb SLO feature, for example, we kicked it off with three days of intense meetings in a Seattle co-working space. (By the way: if you ever have a chance to get to work with Liz? Absolutely worth it.) We looked at existing user feedback, finding limitations in the trigger product. We sketched possible UIs. We mocked up the core algorithms and equations — first in a spreadsheet, then a Python notebook. Those first steps got us far enough to learn about a lot of unstated assumptions in the algorithms. We were able to use the sheets to figure out how to model prediction, and to figure out what parameters the algorithm needed.

Then we started to build it into a product. We had three steps in mind:

1. Be able to run dogfood SLOs on Honeycomb data

2. Be able to hand-hold a few intrepid customers through SLOs

3. Any customer can create an SLO

The first thing we needed was a back-end engineer, who could start building out the query caching machinery. (This turned out to be rather hard! I wrote about some of our lessons — and how we spent $10,000 in a day.) I had enough enough coding prowess to start to build out the front-end, but anther engineer hopped in to to start wiring us into the production database. Liz put on her infra hat and started figuring out how we’d need to arrange databases and servers to support our expected user loads.

We now had four people on the SLO team. As we crossed the first threshold and had an internally-available tool. We encouraged the front- and back-end engineering teams at Honeycomb to incorporate SLOs into their practice, and they started adding SLO alerting to their on-call monitoring and handoff cycle. Liz stepped away, driving the growth of DevRel at Honeycomb.

The project continued to grow: a product manager to help manage internal usage and prioritize the growing todo lists. Design resources. Another front-end engineer to polish the graphs and get BubbleUp more fully incorporated. Step by step, we moved toward release, and the team grew to fulfill our needs. We brought in a Solutions Architect to help us coordinate Phase 2, hand-holding our first customers.

By now, the design was stable. I was less and less useful, as the engineers discussed query policies, API changes, and challenges incorporating various libraries. I transitioned into writing documentation, working with customers, and started up my next project.’

Growing into product

It feels like I’ve seen two different patterns for staffing projects. When you understand the scope of the project, it makes sense to assign a team at the start — say, “a designer, a PM, two devs” — and build out the feature with the team.

But other projects are harder to shape and scope. This pattern seems like a useful one for projects that still have some risks: we bring in resources on parts that we have either successfully de-risked, or where we need the expertise to build out the next step. When building SLOs, we couldn’t have used a full team at the start: there were too many dependencies to work out, too many pieces to put in place.

I love watching a project grow — and I love when I get a real expert on a topic outside my domain who can pick some early decision and sand off the sharp edges, turning it into smooth, well-operating code. And , always with some sadness, I love that moment where I realize the fledgeling product can now fly without me, and its time to start on the the next phase.