Introduction
In high-stakes domains like healthcare, important decisions rarely happen in a single leap. They unfold as a sequence of careful questions, each one narrowing the possibilities until the right course of action becomes clear. Clinical triage is a textbook example: to decide how urgently someone needs care, and of what kind, a clinician works through a structured set of questions, where each answer determines what to ask next. This logic is often formalized as a decision tree, a map of questions and conditional branches that encodes expert knowledge and ensures every person is assessed consistently, regardless of who (or what) is doing the assessing.
For decades, software has executed these trees with rigid, rule-based systems. They are reliable and predictable, but they are also brittle. Real people don't answer in tidy, pre-defined options. They volunteer extra information, answer two questions at once, hedge, or describe their symptoms in ways the system was never programmed to understand. Faced with this kind of nuance, rule-based systems tend to break the conversation rather than bend with it.
Large language models (LLMs) address this problem. They are remarkably good at interpreting messy, natural language and responding in a way that feels human. By handing the decision tree to an LLM and letting it do the navigating, we combine the flexibility of natural conversation with the structure of a clinical protocol.
The challenge is that these two qualities pull in opposite directions. A decision tree demands strict, deterministic adherence, at every step there is a correct next question, and taking the wrong branch in a triage flow isn't a stylistic slip, it's a safety risk. LLMs, by contrast, are flexible by nature, and that flexibility is exactly what makes them hard to keep on rails. Reconciling the two is the core problem Arbor was built to solve. This post distills the core ideas. The full technical report, with complete methodology and per model results, is available on arXiv.
The obvious approach, and why it breaks
The most common way to combine the two is also the most direct: paste the entire decision tree into the prompt and instruct the model to navigate it on its own, tracking where it is in the tree, evaluating the user's latest message against all the possible branches and writing a natural reply, all in a single inference call.
This works well enough in a demo. But as the decision tree grows to the size of a real clinical protocol (hundreds of nodes, thousands of branches), the single prompt approach runs into three problems that no amount of prompt-tuning fully fixes:
- The model stops seeing the whole tree. As prompts get longer, models reliably struggle to use information buried in the middle, a well-documented effect known as lost in the middle. Past a certain size, the tree simply won't fit in the context window at all. This is a perceptual limit, not a reasoning one because even a model that could reason perfectly would still fail to reliably retrieve the relevant branch from a very long prompt.
- The model is asked to do too many tasks at once. A single call has to simultaneously understand the tree's structure, remember where it is in the conversation, evaluate which branch the user's message satisfies, and write a warm, natural reply. Bundling these very different kinds of reasoning into one step is exactly the condition under which the LLM performance is known to degrade.
- When it goes wrong, you can't tell why. If the model lands on the wrong node, there's no way to know whether it misread the tree, mis-evaluated the condition, or simply lost track of the conversation. The whole process is one black box, which makes failures hard to diagnose and risky to fix.
These aren't quirks of a particular model, they are structural consequences of asking the same model call to do everything at once.
Enter Arbor
Arbor takes a different stance, instead of making the model better at navigating a giant prompt, it changes the problem so this approach is never required. Rather than handing the LLM the entire tree and asking it to find its way, Arbor decomposes navigation into small, node-level tasks and feeds the model only what it needs to make the next decision.
At each step, Arbor retrieves only the branches leaving the current node, asks the model to choose among that small, well-defined set, and then, in a separate step, asks it to write the reply. The model never has to hold the whole tree in memory, never has to juggle logic and language in the same call, and every decision it makes is isolated enough to inspect on its own.
This shift turns out to matter a great deal. Evaluated against the single prompt approach across 10 different foundation models on real clinical triage conversations, Arbor improved turn-level navigation accuracy by 29 percentage points, cut per-turn latency by 57%, and reduced per-turn cost by 14× on average. Just as striking, it made performance far less dependent on the underlying model as even mid-size open-weight models running inside Arbor matched or beat much larger, more expensive proprietary models running without it.
How Arbor works
Arbor is built around a single principle, decomposition. Instead of one model call doing everything, the work is split into specialized pieces, each with a narrow, well-defined job. That principle shows up in two places, an offline pipeline that turns a decision tree into a structure the system can query one node at a time, and a runtime agent that walks the tree turn by turn, deciding where to go and what to say.
Figure 1 Overview of the Arbor architecture. The processing phase (bottom) normalizes a raw decision tree into an edge-list database. At runtime, the evaluation phase (top left) dynamically retrieves the outgoing edges of the current node and evaluates transitions through iterative LLM calls until no further transition applies; the message generation phase (top right) then produces the user-facing reply.
Turning a tree into queryable data
Before Arbor can navigate a decision tree, it converts it into a form built for retrieval. Clinical decision trees can originate from heterogeneous sources, such as spreadsheets, configuration files, or legacy rule representations, each with its own structure. Rather than coupling the rest of the system to any one of them, a normalization step maps the source tree into a single canonical representation, an edge list, a flat set of transitions where each edge is one possible move from one node to another. Every edge carries what's needed to decide whether that transition should be taken, the question being asked, the answer or condition that triggers it, the source and target nodes, and any extra domain-specific context. Storing the tree this way separates control logic from domain knowledge, so clinical teams can refine a question or add a new path in the source definitions, and once the tree is re-validated and re-ingested the agent picks up the change.
Because a decision tree is only as safe as its structure, the pipeline validates its integrity before anything is indexed, confirming that every branch leads somewhere reachable and that no part of the workflow can trap the conversation in a dead end. Only once a tree passes all validations is it ingested into the retrieval system, ready for the agent to query one node at a time.
The runtime agent
At runtime, Arbor operates as a stateful agent that always knows exactly where it is in the tree. The current node is tracked explicitly and saved to a database, so the conversation can pick up exactly where it left off across turns. Every time the user sends a message, Arbor runs a two-step loop, first deciding where to go, then deciding what to say.
- Retrieval and transition evaluation. This is the decision-making core. Arbor retrieves only the current node's outgoing edges and hands that small, focused set to an LLM whose only job is to choose the right transition, weighing the candidate branches against the recent conversation history and any external context that can't be inferred from the dialogue alone, such as eligibility rules or risk flags. The model picks the branch the user's message satisfies, or outputs stay when nothing is satisfied yet and more information is needed. The step is iterative, so when a single message resolves several questions at once Arbor advances through each node in turn and re-evaluates, skipping questions that have effectively been answered rather than asking them one by one. It stops at a node it has to stay on, or at a terminal node with no outgoing edges left to evaluate.
- Message generation. Once Arbor knows which node it's on, a separate LLM call writes the actual reply. This call is dedicated purely to communication. It receives the content of the current node, the conversation so far, the relevant member context, and, crucially, the reasoning the evaluator produced in the first step. Passing that reasoning forward is what keeps the two steps aligned, since the model writing the message understands why the conversation is where it is, so its reply stays both natural and faithful to the decision that was just made.
Why separate evaluation from generation
Separating evaluation from generation fixes the "too many jobs at once" failure mode from earlier. Asking a single call to reason rigorously about logic and write warm, fluent prose tends to produce one at the expense of the other, the model follows the rules but sounds robotic, or it sounds great while quietly drifting off the correct path.
The separation also buys real operational flexibility.
- Independent tuning. The evaluation step can run cold and deterministic (low temperature) for consistent, correct decisions, while the generation step can run warmer for more natural language. Each can be configured on its own terms.
- Mix-and-match models. Because the steps are decoupled, they don't even have to use the same model. A stronger, pricier model can handle the high-stakes logic while a faster, cheaper one handles the conversation.
- Flexible responses. The generation step can adapt to the kind of node it's on, asking a question, presenting guidance, or closing out a flow, by selecting different prompts and drawing on node specific context.
The payoff is what the next section measures, whether decomposing the problem actually makes navigation more reliable, faster, and cheaper, and across which models.
Putting it to the test
A cleaner architecture is only worth something if it holds up under real conditions. To find out, we put Arbor head-to-head against the single prompt approach on real clinical triage conversations, using the same decision tree, the same conversation history, and the same underlying content for both. The only thing that differed was how each approach consumed the tree.
The setup
The evaluation draws on 20 recorded triage conversations between clinicians and patients moving through a deployed workflow, annotated turn by turn with the node the conversation was on before each patient message and the node it should land on afterwards. That gives a ground-truth path through the tree and 174 decision points in total, each one a moment where the system has to read the patient's message and choose the right next step. We treat any departure from the annotated path as an error, a deliberately strict bar that keeps the comparison consistent across models, even in ambiguous moments where a clarifying question might also be defensible.
The decision tree used has 449 nodes and 980 branches, reaches 19 levels deep, and, written out in full, takes up roughly 120,000 tokens. That is the entire structure the single prompt approach has to fit into its context and reason over at every single turn, while Arbor only ever looks at the handful of branches leaving the current node.
We ran both approaches across 10 models spanning the landscape, GPT5 (minimal, medium, and high reasoning effort), GPT4.1, Claude Sonnet 4.5, Gemini 3 Pro, Gemini 3 Flash, DeepSeek V3.1, and Qwen3 in 30B and 235B sizes, each repeated five times for stability. Within Arbor the same model handled both the evaluation and the generation step while we measured navigation accuracy, latency per turn, and cost per turn.
We measured three things that matter for actually deploying a system like this, navigation accuracy (how often it reaches the correct node), latency (how long a turn takes), and cost (the price of the tokens each turn consumes).
Navigation accuracy
Getting to the right node is fundamental, since in triage a wrong turn is a clinical safety issue. This is also where the two approaches diverge the most.
Figure 2 Turn-level navigation accuracy for each model under the single prompt baseline and under Arbor, averaged over five runs. Arbor lifts every model into a high accuracy band, while the baseline swings widely with model capability.
Under the single prompt baseline, accuracy has high variability and tracks raw model capability. The strongest reasoning models perform the best, with GPT5-high and GPT5-medium reaching around 83% and 81%, but that strength doesn't generalize. Claude Sonnet 4.5, a very capable model, manages only 43%, and the non-reasoning GPT4.1 lands at 52%. The open-weight models struggle hardest, with DeepSeek V3.1 at 38% and Qwen3 30B at just 15%. The pattern is clear, when the whole tree is included in the context window, whether a model copes at all depends almost entirely on how strong it is to begin with.
Arbor flattens that variance, every proprietary model converges close to 90%, including Claude Sonnet 4.5 at 91% and both Gemini models around 91%, all of them beating the best single prompt result. The same equalizing effect shows up in the open-weight models, with DeepSeek V3.1 jumping to 88% and Qwen3 235B to 91%. Even the small Qwen3 30B more than quadruples its score, from 15% to 67%. The single prompt approach makes accuracy a function of the model, while Arbor makes it a function of the architecture, so navigating the tree reliably no longer depends on reaching for the best model available.
Latency
Multi-step architectures introduce an obvious worry, that several sequential model calls will be slower than one. In practice, the opposite is usually true.
Figure 3 Average per-turn latency for each model. Arbor is faster than the single prompt baseline for most models and far more stable, while the baseline can spike dramatically when a model has to process the full tree.
The reason is that the single prompt baseline reads the entire 120,000-token tree on every turn, and that quickly outweighs the cost of a few small, focused calls. GPT4.1, for example, drops from 14.1 seconds per turn under the baseline to 5.1 seconds under Arbor. The effect is most dramatic for open-weight models forced to reason through the giant context, where DeepSeek V3.1 collapses from a punishing 81.5 seconds per turn to 5.9. The one exception is Claude Sonnet 4.5, which handles the large context efficiently and is actually faster in a single call (9.1 seconds) than across Arbor's multiple steps (17.6). The broader story is that big-context single prompts produce wildly unpredictable latency, while Arbor's bounded inputs keep response times stable and usually lower.
Cost
Cost is where decomposition pays off most plainly. Because the single prompt approach re-ingests the full tree at every turn, its token bill scales with the size of the tree, while Arbor's stays bounded no matter how large the workflow grows.
Figure 4 Per-turn cost for each model. Restricting the context to the active node and its branches yields an order-of-magnitude cost reduction across every model.
The gap is on average over an order of magnitude. GPT5-high, for instance, falls from $0.175 per turn under the baseline to $0.031 under Arbor, which makes even a top-tier reasoning model affordable to run at scale. And because the two steps are decoupled, the bill can drop further still by reserving an expensive model for the reasoning step while a cheaper one handles the conversation. As trees grow larger, this advantage only widens, since the baseline keeps paying for every extra branch and Arbor does not.
The headline numbers
Averaging across all 10 models gives a clean summary of the architecture's advantage, independent of any single model.
Three things stand out. Accuracy climbs by 29 points, and the standard deviation shrinks dramatically, which is the quantitative version of the story in Figure 2, the architecture doesn't just raise the average, it removes the wild swings between models. Latency drops by more than half, and cost falls by about fourteen times. Most importantly, these gains aren't confined to one class of model, they hold across proprietary and open-weight systems, across reasoning and non-reasoning ones, and across a wide range of sizes. By structuring how context is accessed and taking control flow out of the model's hands, Arbor lifts the floor for everyone and lets small, cheap models match or beat much larger ones left to navigate on their own.
Preserving conversational quality
There's a reasonable objection lurking behind all of this. By splitting the reasoning from the writing and feeding the model only a narrow slice of the tree, are we trading away the very thing that made LLMs attractive in the first place, their ability to sound human? A system that navigates perfectly but answers in stilted, robotic prose would be a poor trade in a setting where members are often anxious and how something is said can matter as much as what is said.
To check, we evaluated message quality on its own, holding navigation constant. When the two approaches take different paths through the tree, their replies are about different things, so comparing the text directly would be meaningless. We therefore looked only at turns where both approaches landed on the same correct node, which isolates writing quality from navigation. Both ran on GPT5-minimal, the strongest overall performer from the navigation experiments, and from the aligned turns we assembled 50 real triage cases. Each case produced two replies, one from the single prompt baseline and one from Arbor.
Three licensed physical therapists, with 4 to 18 years of clinical experience and specialties spanning women's health, vestibular rehabilitation, and orthopedics, judged every message. We used physical therapists because the triage flow is grounded in physical therapy and assessing whether a reply is correct, safe, and appropriate takes real clinical judgment. Each reviewer saw the full transcript, the member's context, and what the target node was supposed to accomplish, then did two things. First, they gave a binary accept or reject, where a message had to fulfil the node's intent, stay coherent with the conversation, and meet clinical safety standards to be accepted. Second, for accepted messages, they rated conversational naturalness on a 1-to-4 scale, from factually correct but robotic (1) up to engaging and indistinguishable from human writing (4).
The verdict was that decomposition costs nothing in quality. Acceptance was effectively tied, 97.3% for Arbor against 100% for the baseline, a gap too small to resolve with 50 cases, and the rare rejections came from minor annotation disagreements rather than unsafe guidance. Among accepted messages, average quality was 3.67 for Arbor and 3.62 for the baseline. A Wilcoxon signed-rank test confirmed the difference is not statistically significant (p = 0.455). The score distributions, shown in Figure 5, are nearly identical, with the bulk of messages from both approaches landing at the top of the scale.
Figure 5 Distribution of conversational quality scores (1-4) for accepted messages under each approach, rated by the clinical panel. The two distributions are nearly identical, with most messages scoring at the top end.
The takeaway is straightforward. When both approaches reason their way to the same correct node, clinicians rate their replies as equally natural. Arbor delivers its gains in accuracy, latency, and cost without sacrificing the warmth and fluency that make the conversation feel human, which is exactly the worry that the separated message-generation step was designed to put to rest.
What we learned
Building Arbor and evaluating it across ten different models surfaced a few lessons worth drawing out.
Constraining the decision space matters. The single prompt approach forces the model to search the entire tree at once, reasoning over a large, heterogeneous space where any node is a possible answer. Arbor exposes only the current node and its outgoing branches, which bounds the choice to a small, well defined set. A transition to an unrelated part of the tree isn't just discouraged by instructions, it's structurally impossible, because those branches were never put in front of the model.
Bounded context is what makes the approach scale. The baseline's cost and latency scale with the size of the tree, because the whole structure has to be reprocessed on every turn. As a clinical protocol grows, that overhead grows with it. Arbor is effectively invariant to tree size, since it only ever retrieves the immediate neighborhood of the current node, so its input stays small no matter how large the overall workflow becomes.
Localized structure is what makes it safe to operate. When a single opaque prompt takes a wrong turn, it's hard to tell whether it misread the tree, misjudged a condition, or lost the thread of the conversation, and fixing it means editing a giant prompt with unpredictable side effects. Arbor pins every decision to a specific node and branch, so a failure can be traced to one transition, inspected, and corrected in isolation, without risking unrelated parts of the workflow.
Conclusion
Deploying language models inside structured clinical workflows means satisfying two demands that usually pull against each other, the warmth and flexibility of natural conversation, and the strict, auditable adherence that a clinical protocol requires. The common instinct is to lean on ever-larger models and ever-longer prompts to bridge the gap. Arbor takes the opposite view: that the answer lies in the architecture rather than the size of the model.
By turning the decision tree into queryable data, retrieving only what's relevant at each step, and separating the act of choosing a path from the act of writing a reply, Arbor makes reliable navigation a property of the system instead of a gamble on raw model capability. Across ten models and real triage conversations, that shift delivered substantially higher accuracy, lower latency, and dramatically lower cost, while preserving the conversational quality clinicians expect.
Although we built Arbor for clinical triage, nothing about the approach is specific to healthcare. Any setting where a conversational agent must follow a defined procedure, from compliance and onboarding flows to technical support and structured intake, faces the same tension between flexibility and control.
Acknowledgments
Written by Luís Silva in collaboration with Diogo Gonçalves and Clara Matos, supported by the solution built by Luís Silva, Diogo Gonçalves, Miguel Constantino Cruz, Clara Matos and Luís Ungaro.
