When we decided to replace our third-party reporting stack, we knew precisely what experience we had to build: native reporting, faster answers, and a natural-language query experience that lets customers ask questions in plain English.
The nice thing about building first-party is that the first version does not have to be the last version.
To get to a first-party reporting experience we could trust, we had to solve three problems at the same time.
We needed a data ingestion pipeline that could move operational data fast enough for a customer-facing product.
We needed an analytics layer that could answer complicated questions without turning into a multi-year re-platform.
And we needed to use AI in a way that made the product easier to use without turning correctness, performance, or privacy into open questions.
None of those problems are novel nor challenging on their own. The challenge was solving all three at once, on a real timeline, inside a mature technology stack.
The Problem Beneath the Product
Justworks already had a Reporting and Analytics product. We partnered with a vendor that promised a seamless integration: we’d own the data, they’d own the experience, and all we had to do was build the pipeline to feed it.
Customers had access to the product and still were not using it in meaningful numbers. The issue was not adoption tactics or education but fit.
Why We Built the Ingestion Layer Ourselves
The decision to rebuild the platform led straight to the first technical challenge: getting operational data where it needed to go, fast enough for a customer-facing product.
We started where most teams would start, by looking at existing tools.
There is no shortage of ingestion and replication platforms that move data from operational systems into analytical ones. The problem was that every option solved only part of what we needed. Some worked well for certain database engines but not others. Some were built for batch extraction, not the event-driven or near-real-time patterns we cared about. We kept running into the same problem: half-solved.
Most of the tools we evaluated still would have left us building a meaningful chunk of the system, so we built the ingestion pipeline ourselves.
That gave us control over the two constraints that mattered most: source compatibility and latency. More importantly, it let us treat the pipeline as product infrastructure, not just analytics plumbing. We were not simply landing data somewhere else, we were powering customer-facing experiences with fresh, trustworthy data.
Building the Fastest Version of the “Wrong” Analytics Engine
Once data is moving, the next question is how to query it.
If we were starting from a blank page today, there are many modern OLAP systems (think Snowflake, BigQuery, RedShift) that do a very good job solving analytical query performance at scale. We needed an analytics engine quickly, and we needed one our team could reason about and operationalize without introducing a long adoption curve or a large new platform surface area. That led us to an unusual but pragmatic choice: we built an OLAP-cube-style query engine on top of MySQL.
On paper, that isn’t the most fashionable architecture but it was the right one for the moment. It let us move fast and it let us stay close to systems our engineers already understood.
This is one of those decisions that makes more sense when you understand the actual constraint. There is a tendency in technical storytelling to erase these kinds of choices afterward, as if every good system arrives fully modern from day one. Most do not. Sometimes the right decision is not to build the most future-proof architecture, but to quickly build an architecture that is safe, understandable, and good enough to carry real customer load.
The Hard Part of AI Was Restraint
Once the data and query layers were taking shape, the most visible product question arrived: how should customers ask for what they need?
The obvious answer in 2025 was to throw an LLM on top of the database and let it generate SQL. That path is easy to imagine because the demo is compelling. A user asks a question in natural language, the model translates it into a query, and the system returns an answer.
The problem is that “compelling” is not “production-ready.”
Customer reporting is not a place to be casual about correctness. The data is sensitive, the schemas are complex, and the questions often span many related entities. When a query touches dozens of tables and depends on business-specific definitions, there is no general guarantee that an LLM will produce the correct query every time. Even when the SQL parses, “syntactically valid” is not the same thing as “semantically right.”
We also had performance requirements. A system that sends a large schema context to a model, asks it to reason across it, and then waits for query generation to finish introduces latency exactly where the customer experiences it most directly. That did not align with the product we wanted to build.
And finally, we had a privacy requirement that was non-negotiable: customer data should not be exposed to the model as part of answering the question.
So what did we do?
The natural-language layer takes the user’s request and converts it into a structured representation that our analytics API understands. The model acts as a translator from human intent into a controlled schema, not as an analytical engine and not as a freeform database operator.
It gives us a tighter boundary around correctness because the downstream system still owns the actual query logic. It gives us better performance because the model’s job is narrower. And it gives us a stronger privacy posture because customer data does not need to be passed into the model for the translation step to work.
In our case, the point of AI wasn’t to replace the application logic. It was to reduce effort for the user while preserving the guarantees of the system underneath.
The customer still gets to describe what they want in natural language, the system still returns an answer.
Technical Progress Sometimes Looks Unfashionable Up Close
Looking back, what stands out most is how many of the right decisions were unglamorous.
We built our own ingestion pipeline because existing tools did not meet our mix of source coverage and latency needs. We built an OLAP-style engine on top of MySQL because it was the fastest path to something dependable, even if it was not the architecture a clean-sheet design would produce. We used AI conservatively, not because we lacked ambition, but because correctness, speed, and privacy mattered more than a flashy demo.
None of those decisions were the “default modern stack” answer, but all of them were grounded in the actual product we were trying to ship.
That is one of the recurring lessons in software development: the technically interesting choice is not always the newest tool or the cleanest diagram. Sometimes it is the tradeoff that lets a team move quickly without giving up control over the parts of the system that matter most.
For us, those parts were clear. Own the data path. Own the query semantics. Use AI where it lowers effort, not where it introduces uncertainty.
What We’d Do Differently, and What We’d Do the Same
The nice thing about building first-party is that the first version does not have to be the last version.
Now that the system is live, we can go back and clean up some of the choices we made to ship quickly. The analytics layer has improved a lot since the first version, but it still lives in the shape of those early constraints. We have a product that meets the need now. That gives us room to simplify the parts we had to build under deadline.
I would still optimize for owning the data flow end to end. I would still separate natural-language understanding from analytical execution. I would still treat freshness, privacy, and correctness as foundational design constraints.
The Deeper Lesson
The story of this project is not really that we built reporting features or a natural-language interface. It is that we built the technical foundations required to make those features trustworthy.
That meant resisting the temptation to accept half-solutions from the market. It meant making pragmatic architectural choices under time pressure. And it meant being intentional about where AI belonged in the system and where it did not.
Customers do not experience any of that directly. What they experience is a product where they can ask a question in plain English and get a reliable answer.
