Ask-Mo: How We Built a Knowledge Graph That Lets Engineers Query 40+ Services in Plain English
This blog was written by Mohit Keswani(opens in a new tab), Engineering Manager at Justworks.
Here at Justworks, we build software that helps small businesses run payroll, manage benefits, and stay compliant. These are hard things to get right, and mistakes can negatively impact people. Behind that product is a complex platform spanning dozens of services across 5 languages. When something breaks at 2 AM, the speed of resolution maps directly to how accurately a customer’s payroll is calculated, or how quickly their benefits enrollments are processed.To help reason with the breadth and depth of these business domains, we built a knowledge graph that maps how all of our services connect, along with an AI agent that lets our team interact with it in plain English. This post covers the problem that led us there and what changed.
The 2am Incident
A production database maxed out overnight. An engineer got paged, opened a dashboard, and saw the spike. It appeared to be the shared monolith database. Multiple services depended on it. But which one was flooding it?Three teams joined a bridge call and spent the next two hours tracing the issue across four codebases, along with a feature flag that had been toggled earlier that week. The root cause turned out to be a cron job in one service that hit the shared database without scoping its queries per-tenant. The unbounded queries simply overwhelmed our system.
Press enter or click to view image in full sizeEvery fact needed to solve that incident existed in the code. The job’s schedule was in a cron file. The database query lived in a repository three teams away from the one that owned the table. No single engineer could hold the map of tens of thousands of source files across dozens of services in their head, and no existing tool could trace a question like “which scheduled jobs query this database without customer scoping?” across these repositories.It wasn’t a tooling gap. It was a knowledge gap.
The Problem, and Our Approach
This problem isn’t unique to us. Distributed systems create a game of cat-and-mouse when something goes wrong. No single service owns the full picture. Every investigation turns into an archaeology dig across repositories, message queues, and shared databases, each owned by a different team, maybe written in a different language, documented to varying degrees. The more services you add, the wider the search space and the more coordination overhead just to understand what’s happening.Our architecture spans dozens of services in 5 languages, teams own their services but regularly need to understand what’s upstream, downstream, or sharing their infrastructure. Existing tools each solve a piece. Code search finds text matches but can’t follow how data flows across these repos. Service catalogs track ownership but not runtime dependencies. Observability tools show live traffic but can’t explain why a dependency exists. Architecture diagrams go stale within weeks of being drawn.What’s missing is connective intelligence: understanding not just what each service does, but how they relate, why they’re connected, and what happens when one changes.We decided to build an AI tool that holds the full mental model. It reads every codebase, discovers how services connect, and lets engineers interrogate the material.Our system operates as a three-stage pipeline, which the rest of this post walks through:
Ingest: Clone every repository, parse the code structurally, then use an LLM to extract higher-level understanding
Graph: Discover cross-service dependencies through multiple independent analyses, each looking for different signals
Query: Answer natural-language questions via an AI agent with access to the full knowledge graph
The key architectural decision was deterministic extraction first, LLM reasoning second. We don’t feed raw source files to an LLM and hope for the best. We first parse every file with language-aware tools to extract hard facts (function signatures, HTTP endpoints, message queue topics, database references) and only then use an LLM to synthesize understanding on top of those grounded facts.
Stage 1: How We Read 28,000+ Files Efficiently
The naive approach of feeding every source file to an LLM for analysis would produce unreliable results and waste enormous amounts of compute. Instead, we use a strict two-pass architecture designed to minimize unnecessary LLM calls while maximizing the quality of each one.Pass 1: AST (Abstract Syntax Tree) extraction, zero LLM cost: Language-aware parsers (tree-sitter) extract structural facts from all 28,000+ files across 9 language grammars. This gives us function signatures, HTTP route definitions, message queue producer/consumer declarations, database table references, and cross-file call chains. All deterministic, all cacheable. No LLM is involved.Pass 2: LLM summarization: Only after every repository completes the first pass does the LLM phase begin. This ordering is deliberate. When the LLM summarizes a service, it has the full cross-codebase picture available as grounding context. It sees which other services call this one’s endpoints, which message topics flow between them. That context reduces hallucination significantly because the LLM is synthesizing verified facts, not guessing.Reducing unnecessary LLM passesAt this scale, cost discipline comes from avoiding redundant work, not from making individual calls cheaper. Two strategies make this practical:Prompt caching: We structured our prompts so that stable extraction rules (~8,000 characters of instructions and output format) are separated from variable content (the actual source code). Marking the stable portion as cacheable means the LLM re-reads it from cache rather than reprocessing it on every call. This achieves roughly 90% input token cost reduction across batches.Delta ingestion: A full run processes all 28,000+ files in under 2 hours. But most weeks, only a fraction of code changes. Our delta ingestion detects changes at three levels: file-level git diffs, AST structural diffs (ignoring comment and formatting changes), and impact analysis of downstream artifacts. When fewer than 30% of files change, only affected artifacts regenerate. A typical weekly update completes in minutes because the system only re-processes what actually changed.
Stage 2: How We Know Services Are Connected
Knowing what each service does isn’t enough. The real value is in the connections, and no single analysis method catches them all.
We run five independent linking passes, each looking for a different signal:
Message queues:
Match producers and consumers sharing the same topics
HTTP:
Match client calls to server endpoint definitions
Database:
Detect shared table access from production config analysis
AST call chains:
Trace cross-service function call paths through the code
Text matching:
Service name mentions in code and documentation (fully deterministic, zero LLM)
Each pass has known blind spots. Message queue links are invisible in HTTP registries. Database sharing doesn’t show up in message queues. But the blind spots don’t overlap. When multiple independent passes agree that a connection exists, we can be highly confident it’s real.In practice, 80.8% of discovered edges are confirmed by multiple independent passes, with an average confidence score of 0.971 (on a 0–1 scale).
Stage 3: An Agent That Investigates, Not Just Searches
The query layer isn’t a search engine. It’s an AI agent that investigates questions the way an experienced engineer would: gathering evidence from multiple sources, forming hypotheses, and pursuing the most likely explanation.The agent has access to 13+ tools spanning the knowledge graph (code search, service info, dependency traversal, message topics), issue tracking, observability metrics, feature flags, version control history, and production data.There’s no prescribed investigation order. The agent decides what to look at based on what it learns. A simple “what does service X do?” might need 2–3 tool calls. A root-cause investigation might need 15, correlating code changes with metric spikes and issue tickets.
A concrete example
An engineer asks: “What writes to the payments ledger table without company scoping?”The agent:
Searches the knowledge graph for database references to the payments ledger table, finds 4 services referencing it
Checks service neighbors on each, discovers a cron job in a background processing service that writes to the table
Inspects the cron job’s source code, identifies a bulk INSERT with no WHERE company_id = ? clause
Checks what triggers the cron, finds it’s triggered by a daily message event with no company context in the payload
Synthesizes: “The nightly_reconciliation job in billing-workers writes to payments_ledger without company-specific scoping. The job queries all unreconciled payments globally.”
Total time: ~35 seconds.The same investigation previously required an engineer to grep across 4 repositories, trace the message topic through config files, and read the cron job source.The system is available both as a chat interface for interactive investigation and as an MCP server that can integrate directly into Claude Code.
Results
This system started as a hackathon prototype — a proof of concept to see if we could parse a few repositories and answer basic questions about service dependencies. The prototype used a single embedding model, no graph structure, and answered maybe 40% of queries usefully. But it proved the concept: engineers immediately started asking questions they’d never bothered to investigate before, because the cost of asking dropped from “schedule a meeting with three teams” to “type a sentence.”That signal — engineers voluntarily using an imperfect tool because the alternative was that painful — justified graduating this tool to production. Over several months, we evolved the system. Structured AST extraction replaced naive text ingestion. Multi-pass linking replaced single-model similarity. The agentic pipeline replaced simple retrieval. Each iteration was driven by specific failure modes we observed in real usage.
What changed in practice
Incident investigation: The 2am database incident took three teams over two hours, most of that time spent figuring out which services were involved, not fixing the actual problem. Now an engineer asks “which cron jobs query this database without customer scoping?” and gets a traced answer with specific file paths and call chains in under a minute.Onboarding: New engineers ask the system directly. “What services interact with our billing pipeline?” returns a traced dependency map with code-level specifics. Engineers ramp up on unfamiliar domains in days instead of weeks, without pulling seniors off their work.Architecture planning: “If we extract this module into its own service, what depends on it?” returns a complete impact analysis in minutes.
The multiplier effect
The real productivity gain isn’t per-question time savings. It’s the questions that never got asked before. When investigating a problem required assembling a multi-team call, engineers would often settle for a partial answer or a workaround. When the cost drops to a 30-second chat message instead of a 45-minute meeting, people ask more questions, catch more issues earlier, and make better-informed decisions.Catching gaps before they reach productionAn unexpected benefit: engineers are using the system to find gaps in cross-service logic before shipping code. When you can trace the full path of a request across multiple codebases — from API entry point through message consumers to database writes — missing error handling, unaccounted-for edge cases, and inconsistent assumptions between teams become visible.Engineers aren’t just shipping faster. They’re shipping with more confidence because they can see the full blast radius of their changes across service boundaries.
To be continued…
Want to understand more about the system and how we architected it? We’ll be publishing an in-depth follow-up soon.
Do you want to build products that help entrepreneurs and small businesses grow with confidence? We’re hiring across our Technology teams. Come build with us! Check out our Careers page.
