How we launched a smooth billing overhaul

Aha! has evolved significantly over the past several years. What began as a single-product offering is now a suite of world-class product development tools. To keep up with our growing userbase, we recently launched an overhaul of our billing system. We dedicated many months to rigorous planning, analyzing, and testing. After all this effort, I was shocked to discover the launch went... perfectly.

I found myself growing suspicious. We just changed an area with a high amount of essential complexity — surely we missed something. There must be an edge case, some sequence of events that would reveal an issue in our error tracking. What did I miss?

Instead, we were met with beautiful silence. My skepticism melted and hope fluttered in my stomach.

I've had my share of good and bad rollouts but this one stood out. There were numerous perfect moments during development. My teammates and I were in sync throughout, at times intuitively pushing up commits containing the precise code another had intended to write. Still, I wasn't prepared for how smoothly our system functioned. It was most profitable code I've ever written.

Leading up to the launch

A program had already run to reach out and familiarize customers with the coming changes in Q3. The date was set: January 1, 2022. We had approximately six months to implement and ship the Price Increase Program.

We considered our strategy options:

1. Get to work right away on the existing pieces of our system, updating them to handle price increases.
2. Build a new, larger-scale area in our system that could handle price increases, then adapt other parts of our system to use the new one.

Option 2 would save time overall since adapting each piece to handle price increases would be less work than reimplementing variations in each place. But it still felt risky. If we started with the piece of our system that communicated with our billing provider and got price increases working there, we could guarantee that was in place before the launch date. Investing upfront and adapting later meant pulling things together at the last minute and counting on time savings produced early.

Due to previous exposure to this domain, I knew option 2 would serve the business better — it would be worth the risk. I'm grateful to have been given the agency to pursue what I knew was best.

Discovering the environment

Coding is not a pursuit of complicated math. It is about striving to identify, understand, and organize the right pieces of the world in our minds. We then produce a corresponding set of patterns that represent these thoughts on a computer. Before diving into any code, you need to understand the environment this code was introduced to operate in. Otherwise it will be hard to know when to apply this advice.

Aha! hasn't increased prices since launching in 2013. The task for my team was to implement a 3% annual price increase. Simple, right? Just current_price * 1.03 and voilà!

Billing systems are endlessly complex. When working with such high essential complexity, it becomes vital to familiarize yourself with what you're contending with. I researched some helpful blog posts on separating billing from entitlements and why billing systems are a nightmare for engineers. In our case, we use a billing provider to tackle some of the most common concerns. Typical considerations include:

• Working with dates
• Prorating
• Upgrades and downgrades
• Corrections
• Pay in advance or arrears
• Dunning and retries
• Usage based
• Taxes and currencies
• Entitlements

Fortunately we only had to implement the first four above. The others lived in our billing provider just a network request away. In addition to our customers, there's also internal CRM tooling to consider. We need to generate quotes, track opportunities, and report on the financials that are core to monitoring and maintaining a profitable business.

Sequencing the work correctly

Once you understand your environment enough, where is the right place to start? To select this, you typically want to:

• Understand the pieces of the system and how they're connected.
• Identify the scale they operate on.
• Start with the riskiest piece first.

In our case:

The pieces of the system and how they're connected

I used this graphic as part of a presentation during the all-company meeting to explain what we were working on.

Identify the scale they operate on

Historically each piece maintained its own way of calculating the subset of billing-related data required. This has served us well for a long time. We started with Aha! Roadmaps, then Aha! Ideas, then Aha! Develop. Aha! Create had not been announced yet.

Price increases would impact our entire suite. Reimplementing the same logic in each area seemed like an awful lot of work. We began to consider an alternative approach. What if we consolidated the billing calculations into one set of objects that supported the full API surface of the rest of them and also handled price increases? Then the other areas would simply need a way to declare their data into this new, underlying format. We'd be able to leave our tables in place and incrementally roll out support across these areas in priority order. Thus, the idea of the "billing calculator" came into focus as we realized we'd require a slightly larger-scale abstraction.

Start with the riskiest piece first

One option was to start by rolling out this new billing calculator inside our opportunities. They contain a simpler surface area and only deal with more abstract revenue. It would be a smaller, isolated area to change in our system. However, we opted to focus on quotes first. Quotes encompass every possible transition a customer can make between our plans, deal with proration, and have much more fine-grained needs of a supporting API. This is opposed to records, which deal with more aggregate numbers. Quotes need to break down line items, where an opportunity only cares about the aggregate revenue value of a given record.

We dove into the most challenging area first. Design for the extremes and the middle will take care of itself.

Not technical debt: Wisdom, beauty, and age

At Aha! we think about code in two ways — wisdom and beauty. Code gains wisdom when it's exposed and adapted to real-world use. Code is beautiful when it is succinct, elegant, and extendable. As architect Christopher Alexander would say, it "has that property which cannot be named." Codebases also age naturally — code that once existed for a purpose may become irrelevant. When I initially reviewed the existing code, a lot of what appeared to be old age was actually wisdom. It just wasn't very beautiful.

An example of this is that we had two different internal structures representing our list plans in addition to our billing provider. Customers can purchase an Aha! Roadmaps subscription, then add-on an Aha! Ideas plan. One of the lists was to represent the values also present in our billing provider. Make sense.

But why the second whole list with a different set of values for the same items?

This second list existed so we could resolve future values we didn't know yet. In some interactions and forms, we needed a way to represent the idea of a plan that would be resolved to a real plan at a future date. We might not know which Roadmaps plan a customer would choose, but when they did choose one, we could associate the right Ideas add-on plan with it. This indicated the need for what has become a very useful part of our new Billing API surface: the ability to resolve the appropriate add-on plan for an existing standalone plan.

At first glance, this seemingly duplicated set of values might look like a surprising choice. But we made it because there were often situations in our internal tooling where we'd be able to massively simplify the UX by choosing a value that meant "fill this in with the right plan once we know what it is later." There were other places we computed prices in JavaScript. For these we could simply rearrange the UX to provide the same values from the Ruby on our servers.

The goal is differentiating wisdom from age and then finding ways to increase beauty. Such wisdom will be especially prevalent in billing systems.

When exploration is worth it

I had previous experience with our quotes but some challenges included:

• Our billing provider — Which API calls did we need to update the amount a customer was billed?
• The nature of the logic that would cause price increases.
• The API surface we truly needed to support.

Before changing any existing logic, my goal was to start building the billing objects we needed to handle the heavy lifting for quotes. I also needed to ensure that we updated our billing provider with the correct information on what to do in any given situation. This amounted to me oscillating between independent work and validating questions with the team. Billing behavior is intricate because many possible operations can occur. And it is stable because we generally want the exact same thing to occur in the same conditions over time. This meant unit testing would be valuable.

Abstractions with reach

The key property of code is how easy it is to change. So the right question is "what kind of change?" The kind of change you need to accommodate is determined by your environment. As you delineate different parts of your code and build connections between these parts, different types of change become easier and harder to introduce.

An abstraction is good when its underlying rules are simple, hard to vary, and the reach of situations it can grapple with is high. In our transition to a multi-product suite of tools, the environmental changes we need to accommodate are new products and new plans. I suspect this is the case for most billing systems. One of the complexities of our price increase program was around transitions. At given points in time we might need to either increase, retain, or reset the price levels for a given account.

This meant we needed the ability to compare any plan with any other given plan and determine which one was the "greater" plan. Then when deciding to increase the prices or reason for a plan change, all we have to do is ask if new_plan > old_plan.

When it comes to system design, consider the parts and how are they connected. Here is what a simplistic system composition that can accomplish comparisons looks like:

Each individual plan is stored as its own, fully connected piece of data. Any reasoning the system does is between different plans. If we store them in a big list ordered by ranking, all we need to do is compare positioning in the list. I initially took this approach.

While breaking this problem down and scoping it out with our product managers, my teammate Andrew Vit gently coached me down a pathway that lead to a different system composition. Instead of reasoning about each plan as its own independent and fully connected piece of data, what if our comparison logic reasoned about the pieces of a plan instead of the whole plan? What if we ranked the parts of a plan against each other, instead of just the whole plan?

Each plan consists of:

• Product
• Type (standalone or add-on)
• Plan level
• Period length

We currently have about 40 independent plans. If our transition logic operated against this list of "whole" plans then we'd be able to reason about our 40 plans. The objects we reasoned about were "strongly connected," meaning we could only reason about an entire plan and not pieces of that plan. However, if we added new plans, we'd have to update our implementation to account for each of the connections between plans.

Reality is not so discrete. Portions of our system can reason about "hypothetical" plans, and other areas need to deal with partial plan information. We want to be able to easily handle the addition of new products and new kinds of plans. Renegotiating the whole list for each addition is a lot of work.

What would the reach of our system be if instead of reasoning about whole plans, we subdivided what a plan was and reasoned about them at this finer scale? Here is what the system would look like if designed this way instead:

An order of magnitude more plans. Without going into all the periods and product subtrees (this is a simplified graphic), the number of plans the comparison logic can provably reason about is easily over 400. Instead of just being able to reason about the existing plans we have, an implementation approach oriented around the parts of a plan could reason about all other possible plans that could be made using these parts as well as being able to reason about "partial" plans (which is quite convenient for UI interactions). The tradeoff here is this piece of our system can't recall only the plans we actually use, but we don't need that here. We already have that elsewhere, which makes this a great tradeoff!

To implement this we leveraged the Comparable behavior of Ruby. This code requires minimal, additive changes to support new products and plans because all we need to update is what the parts of a plan are and how they relate.

module Billing
  module TransitionRankings
    include Comparable

    def <=>(other)
      return -1 if startup? && !other.startup?
      return 1 if !startup? && other.startup?

      by_type_rank(other) or
        by_product_rank(other) or
        by_period_length(other) or
        by_plan_rank(other) or
        0
    end
  end
end

Other elements of our system needn't care about comparisons between plans with all the behavior encapsulated in this one module.

Adding support for a whole new Aha! Create product means updating the rankings between products from:

    def product_rank
      return 3 if product == :roadmaps
      return 2 if product == :ideas
      return 1 if product == :develop

      0
    end

to:

    def product_rank
      return 4 if product == :roadmaps
      return 3 if product == :ideas
      return 2 if product == :develop
      return 1 if product == :create

      0
    end

Minimal code changes, maximal reach. Throughout the codebase areas that utilize this logic, a common piece of feedback we get during exploratory testing is:

I can't break it.

Do what makes sense for your environment

I'm going to give some seemingly contradictory advice. Adapt to your environment and don't blindly follow what you read in a blog post. I just said "don't hardcode your list of plans" and now I'm going to tell you: we hardcoded them and it was great. We also put all of our plans in a .yml file stored in our code base. These would all be read in a lazily instantiated hash (or map, for non-Ruby) as Plan objects accessible through the Billing::Plan.definitions class method.

    def self.definitions
      @definitions ||= YAML.load_file("config/billing_plans.yml").each_with_object({}) { |(id, defn), hash| hash[id] = Billing::Plan.new(id, defn) }
    end

This made sense for us because:

• These plans rarely change. Once used by a customer, the data representing a plan will never change.
• Storing them in our code allowed us to stop making a large number of network requests to retrieve data that doesn't change.
• It was useful to store extra pieces of information about our plans we couldn't easily specify in our billing system but was still valuable to know (like whether it was archived or fixed price).

By reading this data in from a .yml file into a collection of value objects, we're able to trivially change our mind on the source of data later in case we needed to start sourcing it from other systems again. My advice about not hardcoding any plan values is limited. The important element to focus on is each region of your system should operate on a single scale. The more popular way of articulating this notion is, "strong encapsulation." However, I don't find this advice actually helps me make good decisions on how to structure code because it's too ambiguous. Focusing on what the parts are, the scale they operate on, and how they're connected is much more useful.

Plans are value objects that operate on the scale of whole, independent plans. Plan comparison logic, by contrast, operates on the individual attributes of the given plans. Although each module is combined in the whole of the billing system, they make sense maintaining independent (or encapsulated) identities because they operate at different scales. The subdivision of the logic inside our comparison logic is what gives it so much reach.

Can we provide more reach at the larger scale of our value objects too? As long as we're providing linkage to entire plan objects — yes, it fits!

We were able to provide a single, larger-scale object that would let any part of our system easily go from some local value into the correct plan.

module Billing
  class Plan
    def self.from(model_or_val)
      case model_or_val
      when Billing::Plan
        model_or_val
      when String
        from_billing_code(model_or_val)
      when Integer
        from_enum_value(model_or_val)
      # etc
      end
    end
  end
end

This means places in our system that previously couldn't directly communicate could now reason together apples-to-apples. From there, a bevy of methods emerged, which made it easy to get from any plan you start with to any other desired plan.

    def self.relative_addon_plan(primary_plan, addon_sentinel)
    def self.switch_frequency(plan, desired_frequency)
    def self.for_product(products)
    # etc

The key is to reason with our Plan objects, rather than any specific hardcoded code or data, which was really a Plan by some other name.

Launch fundamentals

Now that we've dug into the environment and looked at some code written to represent it, let's talk about the process around how to arrive at the right™ code. This is the hard part. You have to commit to putting in the emotional labor to produce and then let go of your work. Then ensure the necessary communication between the elements of your team and organization is taking place. If you lack either of these efforts, you won't have a smooth launch.

How to produce perfect moments

When you're feeling internal friction and resistance while coding, it may mean there is something in your code you need to fix. Or maybe you don't understand your domain deeply enough and you need to go learn more. Great code comes from humility and error correction along these two dimensions.

There's a rare satisfaction that sometimes comes with writing code — when what you imagined is perfectly reflected on the screen and functions just as you planned. When your teammates are so in sync they are able to provide just the right collaboration and refinement needed to bring big ideas to life. This is how you produce perfect moments. The existence of these perfect moments is how you know you've correctly identified, organized, and represented your domain in code. New and useful applications of your code unfold as others discover what you have and bring their own perspective.

My trick to produce them involves investing, refining, and embracing.

Invest

Get to a point in your work where you're convinced the code simply can't be written any other way. Be honest with yourself about when this is, and once you've hit it, trust it. This is hard to do because you will need to endure periods of high uncertainty.

When grappling with quotes, I ended up producing two classes. Heavily tested and ready for use, they provided the API surface necessary to power our quotes. This meant they should theoretically support what we needed everywhere else, right? I held a small team meeting to walk through what these classes were and how they could be used.

As our efforts ran in parallel on different pieces, I began to get worried. Nobody was using the classes I'd taken the time to build out. Had I gotten it wrong? Had I wasted time and blocked others to produce enabling code they weren't even going to need?

I hopped on a call with a teammate a week later. They walked through the code they built out and showed me a couple of places they repeated a pattern:

 products = [
   # plan, number of seats, period length
   # plan, number of seats, period length
   # plan, number of seats, period length
 ]
 info = do_things_with_products(products)

They stored information about each product on an account and had areas of code that looked across all those areas to arrive at decisions. I started to get excited because we have classes for those. Our ensuing conversation turned into the descriptions at the top of the classes.

module Billing
  # = Billing Bundle
  #
  # The Billing Bundle houses and aggregates the concrete, instantiated values which describe an account
  # for a given period of time. It houses and aggregates Billing Sub Periods, which represent the individual
  # products, plans, seats and price levels an account may have.

  # = Billing Sub Period
  #
  # Billing Sub Period relates a product to a pricing strategy (a Billing::Plan,
  # Billing::Subscription, or other plan-like object) to a range of time.
  # Billing Sub Periods are contained inside a Billing Bundle to represent the
  # state of products an account might have at a given time.

Every other piece of our system related to billing can be declared as instances of these classes. I watched the light flip on in their eyes. What I had missed in my earlier presentation about these classes was an adequate dive into the parts of our system that made them necessary. Maybe the most expedient way wasn't a presentation at all. It was to let teammates get into the work and introduce the new classes the moment they were needed. These may not be independently discoverable, which is a helpful indication of how we could structure our code better.

Refine

The classes you build, the names you choose, and the logical patterns you employ are all necessary tools to scaffold your understanding. These are tools to help you on your journey but they are not the point of it. We must lean on these to gain deeper access and understanding. Consider all the alternatives and ensure you have the internal space to reflect.

Andrew suggested I approach the "transition rankings" of our plans differently. That argument I insisted must be required? Well, maybe it wasn't. Tweak by tweak, test by test, the logic gained flexibility, ease, and life.

This is the critical moment to listen. When you embrace the richness and detail of what the world tells you, when you choose to prioritize curiosity over behaviors of ego or insecurity, that is when it gets incredibly fun. And when your whole team does this together, you will really get moving.

Cherish the unfolding

Once teammates started using the classes, a bevy of improvements I couldn't have anticipated started making their way in. Convenient defaults. Better names. Here's one I absolutely loved:

module Billing
  class SubPeriod

    def change(**fields)
      SubPeriod.new(
        plan: fields.fetch(:plan, @plan),
        duration: fields.fetch(:duration, @duration),
        seats: fields.fetch(:seats, @seats),
        state: fields.fetch(:state, @state),
        price_level: fields.fetch(:price_level, @price_level),
        discount: fields.fetch(:discount, @discount)
      )
    end

SubPeriod#change returns a new instance of a sub period. In many places we are going from some existing account state to some slightly different version of it. This method makes the behavior of the code clear and obvious because you immediately understand the origin of each property in the new sub period.

Sometimes you'll get wonderful instance methods. Sometimes, if you're lucky, you'll get a pattern so powerful and well-applied you'll reforge the rest of the code in your domain in its shape.

The way of "#changes"

The pattern I'm about to describe saved us. Weeks before going live, scope changes came down the line. Between PM and engineering we realized, functionally, we will need to compute and update accounts at many more points in time than simply the ones we'd been targeting. We now had a deeper understanding of the various path dependencies. For one example, if account XYZ isn't updated at point B, then the system won't function properly at point C.

Increasing prices meant our internal system would now determine what a given product would cost and not plan information stored in the billing provider. Although the billing provider would actually charge and collect, our internal tooling now needed to become the control plane to cause this. To provide accurate prices to customers and the billing provider alike, our system needed to be able to run hypotheticals. We needed an extensive test surface and number of dry runs on this new behavior. That's when a beautiful pattern and language emerged that the rest of our code has since adopted:

1. Gather your inputs.
2. Compute your changes.
3. Compare those changes with the current state of the system.
4. If they're different, update the system to your desired state.

In our case we compute the price for a given plan based on an account's price_level. Price levels increment independently per product.

We can streamline our inputs to current + pending = incremented

• Current state of the account + Intended state of the account will result in = Resulting state of the account

The difference between the current state and the incremented state should be stored in a #changes hash containing the old value and the new value per attribute.

This language sidesteps the naming difficulties chronology can introduce. The names make sense whether you're talking about recomputing a transition that's occurred in the past, whether you're live and about to charge a customer an increased price, or you're running future hypotheticals while building a quote.

Storing values inside #changes facilitates easy testing, enables dry runs in production situations to see what would happen, and ensures a common structure for all the places in your application that need to compute transitions — real or hypothetical.

All the hallmarks of a good abstraction are present — a simple set of underlying rules that don't vary and are useful across a wide number of situations. This let us safely and deftly handle some large, last-minute changes to do exactly that.

Frame the best shapes

Shaping is how the work is defined. Framing is how the work is represented. While your implementation unfolds, certain insights about the whole effort will emerge. You will, as we did with the #changes pattern, find certain ideas that convey more complex patterns and ideas simply.

• "Our application is the control plane" — our internal administrative system is responsible for computing the correct price and controlling what value is inside our billing provider.
• "Plan is the source of truth" — the plan instances themselves are the units that contain the highly tested, trustable values as to what prices should be used.
• "Current + pending = next" — (discussed above).

With just these three frames, you can begin to make sense of and understand large portions of the system, or effectively guess where to look next.

Frame the ideas which define the shape of your system. These are mental affordances to build large amounts of useful context for future developers who will interact with your system.

Tune your communication

Most importantly a smooth launch requires open and ongoing communication from all teammates. For us this meant a dedicated Slack channel that received our highest level of priority and attention, short of incidents and security issues. We don't typically do standups, but for this we did. They began weekly and became more frequent as the date approached and our needs grew.

As the launch date crept nearer, we began feeling unsure that we would deliver on time. We discussed this concern freely and it changed the approaches we were taking. Sometimes nonessential scope was pulled out and moved later. Sometimes newly discovered essential scope was prioritized. We did this as we progressed through the implementation. Our team was honest about the work and our progress.

The temptation around communication processes is to try and ideate the perfect, unchanging version ahead of time. But when you do that, your process won't be adapted to your environment, so it will only work well if you're lucky. The key here is consistently thinking about how you communicate and choosing to change your behavior based on what's occurred. Simple is best.

Start a free trial today

Our suite of product development tools work seamlessly together to help teams turn raw concepts into valuable new capabilities — for customers and the business. Set strategy, spark creativity, crowdsource ideas, prioritize features, share roadmaps, manage releases, and plan development. Sign up for a free 30-day trial or join a live demo to see why more than 5,000 companies trust our software to build lovable products and be happy doing it.