While Booster mobile fuel delivery may seem as straightforward as bringing fuel from the yard to a fleet’s lot, all of that work is informed by data and the platform that the team has built over ten years of delivering to fleets across the country.
This week, we’re introducing a new series that lifts the hood on fuel tech and Booster’s platform. Each month, you can expect to see a deep dive into Booster’s technology from a member of our product and engineering team, to see how the team makes decisions, and to get a better understanding of how Booster approaches problem-solving and innovation.
Good technology is rarely designed in a single stroke. Product and engineering teams build the way anything durable gets built: in iterations, road testing something that works, learning from how it behaves in the real world, and evolving the platform and the code underpinning it with each new version. Each newly upgraded system carries the fingerprints of every decision made along the way. And with each evolution, the product team has the opportunity to evolve choices that were made earlier that need to change or be replaced with new cutting-edge tools. Booster works this way by design, which is also why the most useful engineering stories tend to be the ones told mid-stream, while the decisions are still open to challenge.
This week’s Under the Hood unpacks building in public and how Booster’s team settled on the right event bus, aka the system that carries messages between different parts of the company and the platform, for right now from our VP of Product & Technology Nick Tyshchenko.
How Booster Chose The Right Event Bus For The Moment
Booster’s product team fields a recurring question from their peers in fuel tech: Why move to Kafka instead of scaling up the database-backed queues the team already runs? At Booster, the honest answer is that the question doesn’t quite fit. The company never made a single decision about asynchronous messaging. It accumulated one, a piece at a time, over roughly a decade, and the useful work of recent months has been reckoning with what accumulated.
Now for the deeply technical bit, starting with scale, because scale is what makes the rest of the story make sense. Booster’s asynchronous messaging moves in the hundreds of events per second, not the hundreds of thousands that many companies handle. This was never a raw-throughput problem. It was a question of what shape the data actually wanted – and of who should be responsible for operating whatever carried it.
Booster runs a database-backed job scheduler from 2017 that still handles a couple of dozen job types on a single worker.
- It runs well over a hundred cron jobs per environment, each a small script.
- It runs a second database-backed queue, with leases and retries, that autoscales worker pods for the routing solver.
- It runs a claim-and-lease poller inside the main API for ops communications.
- And it runs a managed pub/sub bus with three dozen schema-versioned topics.
Each one of these was a locally reasonable choice for earlier iterations of the Booster platform. And like many tech solutions, none of these were designed as part of the whole platform that exists today because each was chosen at a moment in the platform-building process when they stood in isolation from other systems or offerings. As Booster has evolved with a decade’s worth of fuel delivery data and expanded customer offerings, the team has had to start making these choices deliberately rather than one at a time. The most recent of those choices was Kafka, not as a replacement for what already runs, but for one specific workload that the existing pieces genuinely couldn’t serve.
Now let’s unpack just how that decision came about. First, a few clarifications on how messages actually move between systems.
Message Queue vs. Event Log: A Distinction Worth Stating Precisely
The queue-versus-log distinction is real and worth stating precisely because the terms get used loosely. A message queue is built around delivery: a system puts a message out on the wire, another receives and responds to it, and once the message is processed, it is removed from the queue. That fits tasks: process this payment, send this notification, retry until it succeeds. Think of it like a to-do list.
A log is more like a security camera system, and is built around retention: new information is captured; it can be appended to records, ordered, kept as long as it is configured to be; it’s readable by any number of systems independently, each moving through the record at its own pace; and it’s replayable. Both are correct for different shapes of the problem of data management and usage. It was issues of governance and newer needs for log-style data management, specifically around fleet telemetry, that drove Booster toward current choices.
How Fleet Telemetry Data Influenced Booster’s Kafka Selection
As Booster has evolved its product over the years, data-handling needs have also evolved. Fuel-order updates can be handled with the to-do-list queue style, and that works fine. Only a few systems need that information, nobody needs to rewind it, so there’s no reason to change it.
With advancements in BoosterOS and their customer dashboard offerings, Booster needed the recording “log” style data system as well to handle fleet telemetry, which processes a steady stream of readings that driver phones and the onboard truck computers give off while vehicles are running, like location and engine activity. Fleet telemetry data lives with an outside company, and Booster runs an automated task on a set schedule that regularly pulls down updated data from their systems and copies it into Booster’s platform to feed the internal dashboards the team uses to monitor fleet and device health.
Data transfers like this have a price. The same data ends up billed twice: once by the vendor that collects it, and again as storage in Booster’s own analytics platform. Several of Booster’s own systems genuinely need the steady streams of data, and the ability to rewind and re-read it is valuable when something breaks. All of that is exactly what the recording-style system, a log, is built for. The next decision to make was managing the need for both log and queue style data systems with cost-effectiveness in mind.
Enter Kafka, a durable, rewindable log-style system, that many systems can read independently, that also happens to be flexible enough to act like a queue when necessary – which helps Booster blur lines. But first, how the product team set it up.
Self-Hosted vs. Managed Kafka: Optimizing for the Scarce Resource
Initially, the team prototyped on a self-hosted Kafka-protocol broker that keeps its log on object storage, which is the cheap, warehouse-style tier for holding large amounts of data, rather than replicated block storage, the pricier high-performance kind. The retention economics are far better that way: object storage runs roughly an order of magnitude cheaper per gigabyte-month, and since the whole point of a log is holding data for a long time, that gap compounds. Long retention was necessary to keep building an effective, data-informed platform for Booster’s customers. The product team built it, measured it, and ran the real engine in a local harness, then, before shipping, repointed to the cloud provider’s managed Kafka.
The product team didn’t make this decision because the self-hosted case was wrong. They made it because, at Booster’s current volume (comparatively modest next to the companies pushing hundreds of thousands of messages), the system was never going to be a dominant line on the bill. The storage and retention downstream dwarf it. Plus, self-hosting carries a standing cost that never lands on an invoice:
- Partition sizing or deciding how many pieces to split the log into,
- Retention policy or how long to keep data before it ages out,
- Cluster upgrades, meaning updating the software across every machine in the group without downtime,
- And being on call for a stateful distributed system because someone has to answer the call when something breaks at 3 a.m. while critical data is at risk.
All of it is real, ongoing work, and none of it is the product customers pay for. Self-hosting would have been defensible.
In the end, taking the managed route was cheaper in the currency that was actually scarce: engineering attention. The raw dollars on the hosting bill may even go up, but the trade buys back the time and focus of Booster’s most skilled people and points it at the product instead. Making this decision is just another demonstration of Booster’s own commitment to efficiency and time savings.
Where the Queue-vs-Log Dichotomy Blurs
When deciding on Kafka, the product team had two additional findings that are worth sharing.
First, the team assumed the order in which messages arrived would matter, and it turned out not to. Every message already carries its own timestamp inside it, and nothing in Booster’s system depends on the messages lining up in a particular sequence, so the team decided not to promise any fixed order at all. That freedom let them process messages in big batches instead of one step at a time, which pushed far more data through the system than they could have managed otherwise.
Second, Booster’s messaging system does double duty. Its other job is sending out notifications, and that job doesn’t really look like the “recording” style at all – it behaves like the to-do list. Tasks go out, the ones that fail are set aside and tried again later, and the ones that keep failing get moved to a separate holding area so they don’t gum up the line. Kafka handles that perfectly well, but it’s a good example of how the tidy split between “to-do list” and “recording” gets blurry the moment a real product is involved.
Right now, none of these decisions, from the choice of Kafka to the managed-host setup, are set in stone. They’re, as product teams say, in flight. Both services are in review, and the messaging system is not yet carrying production traffic. But the team is already learning from the process and the setup to continue evolving the technology in real time.
Practical Guidance for Fleet Engineering Teams
For teams earlier in this than Booster, the question worth putting ahead of queues-versus-streams is narrower: Does any specific workload actually have several independent readers that need to move through the same data at their own pace? If not, a streaming platform adds operating cost without buying anything back. If you have thoughts, our team would love to hear them in the comments below.
In the meantime, here are a few pieces of advice from our product team:
- Start managed. The operational overhead of a self-hosted stateful system rarely earns its keep at modest volume.
- Add a log when a specific workload has several genuinely independent readers and replay has operational value – not because the architecture diagram looks better with one.
- Instrument the publisher, not just the consumer, because a silent publish failure is the one a team finds last.
And a final question from our product team: How are your logistics and fleet operations handling this? For those of you running self-managed streaming infrastructure in logistics or fleet operations, what tipped the decision? Volume, retention requirements, cost, or simply that you already had the platform team? Our team would love to hear your thoughts. And we invite you to explore BoosterOS to learn about the fuel tech advances our product team is bringing to fuel distributors and our fleet clients daily. Our team is always growing for the right engineers who can help us evolve our platform and the way we deliver fuel.