7 Tree-Based Methods: Decision Trees and Random Forests
Suppose a platform team wants to automate a decision that an on-call engineer currently makes by eye: after a canary deployment runs for five minutes, should it roll back automatically or keep rolling out to the rest of the fleet?
The signals available are the payload size of the new build, the error rate observed in the canary window, the hour of day, and the number of downstream services the deployment depends on. A decision tree answers this kind of question by asking a sequence of yes-or-no questions about the data, each one splitting the remaining cases into two groups, until it reaches a final call.
7.1 Regression trees and recursive binary splitting
Consider first a simpler version of the problem: predicting a continuous number, the rollback duration in minutes, rather than a yes-or-no outcome. A regression tree builds its predictions by recursive binary splitting: at each step, it searches every possible predictor and every possible split point on that predictor. It then picks the single split that reduces the residual sum of squares (RSS) the most.
Before the formula: imagine guessing a stranger’s height with no clues, then guessing again after learning their age group. The second guess lands closer more often. RSS tallies how far off every guess is, squared so big misses count more. A split earns its place only if it shrinks that tally by enough.
\[\text{RSS} = \sum_{j=1}^{2}\sum_{i \in R_j}(y_i - \bar{y}_{R_j})^2\]
In other words, once a split divides the data into two regions \(R_1\) and \(R_2\), each region’s prediction is simply the average outcome of the training cases that fall into it. The split is scored by how much closer those regional averages get to the true values than a single overall average would.
The algorithm repeats this greedily: split the whole dataset once, then split each resulting region again, and so on, until a stopping rule (a minimum number of observations per region, typically) halts further splitting.
Working a small version of this same idea concretely: suppose eight historical rollbacks have durations of 2, 3, 3, 4, 9, 10, 11, and 12 minutes. A candidate split on canary error rate above versus at or below 3% sorts the first four into one region and the last four into the other.
Measured against the overall mean of 6.75 minutes, the unsplit RSS is 119.5. After the split, region \(R_1\)’s mean is 3.0 and region \(R_2\)’s mean is 10.5, and RSS drops to 7.0. That drop from 119.5 to 7.0 is the kind of signal recursive binary splitting looks for: the algorithm searches every predictor and every threshold, then keeps whichever candidate split shrinks RSS the most.
This is a greedy algorithm: at each step, it picks the best split available right then, without looking ahead to see whether a different split now would enable a better split later. That greediness is also what makes trees fast to fit, since the alternative, searching every possible sequence of splits jointly, is computationally out of reach for anything but a tiny dataset.
Recursive binary splitting never reconsiders a split once made. A split that looks mediocre now but would have opened up a much better split two levels down is never tried, since the algorithm only ever asks which split helps most right away.
7.2 Classification trees: Gini index, entropy, and error rate
The rollback decision itself is a yes-or-no outcome, which calls for a classification tree. The mechanics are the same recursive binary splitting, but the criterion for scoring a split changes, since RSS is not defined for a categorical outcome. Three criteria are used in practice.
The simplest, the classification error rate, scores a region by the fraction of training cases in it that do not belong to the majority class:
\[E = 1 - \max_k(\hat{p}_k)\]
where \(\hat{p}_k\) is the proportion of cases in the region belonging to class \(k\). In other words, if 8 of 10 canary deployments in a region eventually needed a rollback, the error rate is \(1 - 0.8 = 0.2\).
The trouble with this criterion is that it is not sensitive enough: two different splits can produce the same error rate while leaving one split’s regions much more skewed toward one class than the other. Skew is what a growing tree should reward, and the classification error rate cannot see it.
Picture sorting mixed red and blue marbles into two bins. A bin that ends up all one color is progress; a bin still near 50/50 means little was learned. The Gini index scores how mixed a bin still is, so a tree can pick the split that unmixes it the most. It fixes the error rate’s blind spot by measuring total variance across all \(K\) classes rather than just the majority class:
\[G = \sum_{k=1}^{K}\hat{p}_k(1 - \hat{p}_k)\]
Roughly speaking, the Gini index is small when a region is close to pure (nearly all one class) and large when a region is close to a 50/50 split. A split that produces two purer child regions is rewarded, even if the plain error rate would not have distinguished it from a worse split. Entropy (or cross-entropy) plays a similar role using a different formula:
\[D = -\sum_{k=1}^{K}\hat{p}_k \log \hat{p}_k\]
Like the Gini index, \(D\) is small when a region is close to pure and larger when its classes are more evenly mixed; it just reaches those values along a slightly different curve. In practice, the Gini index and entropy produce nearly identical trees. Either is a better splitting criterion than raw classification error, which is typically used only for pruning a tree once it has been grown, not for growing it.
Grow a classification tree with the Gini index or entropy, not the plain classification error rate. Save the error rate for pruning, where its coarser scale is standard practice rather than a liability.
Figure 7.1 shows why: the classification error rate is piecewise linear in \(\hat{p}\), while Gini and entropy are both curved, penalizing a region that sits at \(\hat{p} = 0.6\) or \(0.7\) more than a straight line would. Both criteria are convex functions of \(\hat{p}\), which rewards moving away from a 50/50 split even before a region becomes pure. That gradient is what lets a growing tree tell two similarly impure splits apart.
Working the rollback example concretely: suppose the canary-error-rate feature splits 40 historical deployments into two groups at a 2% threshold. Among the 12 deployments with a canary error rate above 2%, 10 needed a rollback and 2 did not (\(\hat{p}_{\text{rollback}} = 0.833\)); among the 28 deployments at or below 2%, 3 needed a rollback and 25 did not (\(\hat{p}_{\text{rollback}} = 0.107\)).
The Gini index for the first group is \(2 \times 0.833 \times 0.167 \approx 0.278\), and for the second group is \(2 \times 0.107 \times 0.893 \approx 0.191\).
Both are well below the Gini index of the unsplit dataset, \(2 \times 0.325 \times 0.675 \approx 0.439\) (13 of 40 deployments needed a rollback overall), which is the signal that tells the algorithm this split is worth making.
7.3 Trees, categorical predictors, and interactions
Before turning to overfitting, it is worth naming two things trees handle for free that Chapter 4’s linear and logistic regression models do not.
First, categorical predictors need no special treatment: a split on “deployment strategy is blue-green” versus “deployment strategy is rolling” is as natural to a tree as a split on canary error rate above or below 2%. A linear model, by contrast, would need that categorical field converted into dummy variables before it could be used at all.
Second, trees discover interactions automatically. Suppose the rollback risk from a high canary error rate is much worse specifically when the deployment also touches many downstream services, and mild otherwise.
A linear model only captures that if someone manually adds an interaction term (error rate times dependency count) to the formula. A tree finds it on its own: splitting first on dependency count and then on error rate within each resulting branch is the kind of structure recursive binary splitting is built to discover.
This does not make trees strictly better than linear models. Where the true relationship between predictors and outcome is close to linear, as the payload-size-to-latency relationship in Chapter 1 largely was, a linear model has the advantage. It fits the relationship with far fewer parameters and far less variance than a tree needs to approximate the same line out of a series of step functions.
Trees earn their keep specifically when the relationship is not additive and not linear, and when categorical and numeric predictors need to interact in ways nobody wants to hand-specify in advance.
7.4 Tree depth, overfitting, and cost-complexity pruning
Nothing stops recursive binary splitting on its own; left unchecked, it keeps growing a tree until its fit looks like the figure below.
Figure 7.2 shows the same underlying pattern fit by trees of increasing depth. A depth-1 tree barely captures the shape; a depth-8 tree chases every bump in the training noise. The tree that would generalize best sits somewhere in between, which is the tree size cost-complexity pruning is built to find.
Left to run to the end, recursive binary splitting continues until every region contains a single training observation, at which point the tree has memorized the training data perfectly and generalizes to nothing. A deep, fully grown tree is a high-variance model: change a handful of training examples, and the sequence of splits, especially near the top of the tree, can shift enough to produce a noticeably different final tree.
Think of a gardener who lets a hedge grow out fully, then trims back the branches that add mess rather than shape. The standard fix is cost-complexity pruning, sometimes called weakest-link pruning.
Resist the urge to stop splitting early with a fixed depth or leaf-count rule. Growing the tree out fully and pruning back with cross-validation almost always finds a better tree size than guessing a stopping point in advance, since an early stop can cut off a split that would have paid off further down.
Rather than stopping the tree early using an ad hoc rule (which risks stopping just before a split that would have paid off later), the standard approach grows the tree as large as reasonably possible, then prunes it back.
Pruning selects a sequence of subtrees indexed by a complexity parameter \(\alpha\) that penalizes the number of terminal nodes (each terminal node, also called a leaf, is one of the regions like \(R_1\) and \(R_2\) from the earlier regression-tree example).
Cross-validation (the previous chapter’s subject) then picks the value of \(\alpha\), and therefore the tree size, that minimizes estimated test error.
Figure 7.3 is that search made visible. Small \(\alpha\) barely penalizes extra leaves, so the tree stays close to its fully grown, high-variance size and cross-validated error stays close to its worst level. Past the marked \(\alpha\), the penalty starts removing splits the data supports, and error climbs, sharply once too few leaves are left to capture the underlying shape. The marked point is not a fixed rule of thumb; it comes from evaluating this particular tree on this particular dataset, which is why cross-validation, not a guessed leaf count, is what selects it.
7.5 Why a single tree has high variance
Consider a rollback-prediction tree built on one quarter’s worth of deployment data, and a second tree built on the next quarter’s data. Even if the underlying relationship between the features and rollbacks has not changed, the two trees will often disagree on which feature to split on first, and a disagreement near the root reshapes everything beneath it.
This instability is the central weakness of decision trees relative to nearly every other method in this book: they are easy to interpret and visualize, but a single tree’s predictions do not hold steady the way a linear model’s coefficients do when the training sample changes modestly.
Do not treat a single tree’s chosen splits, especially the root split, as a settled finding about which feature matters most. A different quarter of training data can move the whole structure.
Figure 7.4 makes the claim concrete rather than asserted. Each colored line is a tree of the same depth, fit on a random resample of the same 60 observations. The trees mostly agree on the broad shape, but disagree by a full unit or more on where each step happens and how tall it is, particularly in the busiest region of the data between \(x = 0\) and \(x = 2\).
Average enough of these disagreeing trees together, weighting each one equally, and the individual disagreements cancel out far more than any one tree’s error does, which is the entire mechanism bagging exploits next.
7.6 Bagging: bootstrap aggregation
This is the jellybean-jar trick: ask ten people to guess how many jellybeans are in a jar, and the average of all ten guesses usually lands closer to the true count than most individual guesses did, since the too-high and too-low guesses cancel out.
Bagging, short for bootstrap aggregation, addresses tree instability directly by exploiting a basic statistical fact: averaging a set of noisy, high-variance estimates reduces variance, since \(\text{Var}(\bar{X}) = \sigma^2/n\) for \(n\) independent estimates of the same quantity.
Bagging generates that set of estimates by drawing \(B\) bootstrap samples from the training data (each one a random sample of the same size, drawn with replacement) and fitting a full, unpruned decision tree to each bootstrap sample. It then averages the \(B\) trees’ predictions for regression, or takes a majority vote for classification.
For the rollback classifier, this means growing several hundred trees, each on a slightly different resample of the historical deployment data, and predicting rollback if a majority of those trees vote rollback. Individually, each tree still overfits its own bootstrap sample; the averaging is what recovers a stable, low-variance prediction from a collection of unstable, high-variance ones.
Grow each tree in a bagged ensemble deep and unpruned, on purpose. Averaging across trees is what controls overfitting here, not the pruning step a single tree relies on.
7.7 Random forests: decorrelating trees with random feature selection
Bagging alone has a limitation: if one feature (say, canary error rate) is by far the strongest predictor, nearly every bootstrapped tree will choose it for the first split. That means the \(B\) trees end up highly correlated with each other.
Averaging correlated estimates reduces variance by much less than averaging independent ones does, since correlation works directly against the \(\sigma^2/n\) variance reduction bagging relies on. Ten copies of the same guess average to that one guess no matter how many copies get thrown in; only guesses that miss in different directions buy the variance reduction bagging is built on.
A random forest, the method Leo Breiman formalized in 2001 (Breiman 2001), fixes this. At each split in each tree, it restricts the algorithm to choose only among a random subset of \(m\) predictors (typically \(m \approx \sqrt{p}\) for classification, where \(p\) is the total number of predictors) rather than all of them.
This forces the trees to occasionally split on a weaker predictor, such as service dependency count or hour of day, instead of always defaulting to canary error rate. That decorrelates the trees and lets averaging do more of its variance-reduction work. Bagging is the special case of a random forest where \(m = p\); the term “random forest” specifically refers to the version with \(m < p\).
Start with \(m \approx \sqrt{p}\) and leave it there. Random forests are unusually forgiving to tune, and hand-picking \(m\) rarely beats the default by much.
Figure 7.5 shows out-of-bag error (defined next) falling and then flattening as more trees are added to the forest. Note that adding more trees past that flattening point does not increase overfitting risk the way growing a single tree deeper does; it mainly costs training time. Random forests are far more forgiving to tune than a single pruned tree.
7.8 Out-of-bag error estimation
Each bootstrap sample used to grow a tree in the forest leaves out, on average, about a third of the training observations (a property of sampling with replacement: the probability any given observation is excluded from a bootstrap sample of the same size approaches \(1/e \approx 0.368\) as the sample size grows). Those left-out observations are called out-of-bag (OOB) for that particular tree.
Since the tree never saw them during training, predicting on them is a legitimate estimate of test error, computed for free as a byproduct of fitting the forest rather than requiring a held-out validation set or a cross-validation loop.
For the rollback forest, OOB error settles around 34% after a couple dozen trees. That number comes for free, without touching a separate validation set.
OOB error is only a fair stand-in for test error when bootstrap resampling matches how the model will see new data going forward. If deployment behavior drifts across quarters, an OOB estimate computed on an older training set can look fine while the forest is stale.
A 1-in-3 error rate is not something to fully automate on its own. It is cheap enough to recompute every time the model retrains, which makes it a useful early warning: a sudden jump in OOB error after a retrain is a signal worth investigating before trusting the forest’s rollback calls again.
7.9 Evaluating a classifier: precision, recall, and the ROC curve
The out-of-bag error above collapses the rollback classifier’s whole performance into one number, roughly 34% wrong. That single figure treats every mistake as the same size, but the two ways a classifier can be wrong are not equally costly. Missing a deployment that truly needed a rollback (a false negative) lets a bad release reach the rest of the fleet. Flagging a deployment that was fine (a false positive) delays a release for nothing. A metric that folds both into one error rate cannot tell a team which kind of mistake the model is making.
A confusion matrix keeps the two kinds of mistakes separate. It cross-tabulates every prediction against what happened, in a 2x2 grid: how many rollback deployments the model correctly flagged (true positives, TP), how many it missed (false negatives, FN), how many healthy deployments it wrongly flagged (false positives, FP), and how many it correctly left alone (true negatives, TN).
Suppose the rollback classifier is checked against 200 held-out canary deployments, of which 20 needed a rollback and 180 did not, a class split close to what a healthy deployment pipeline should look like. The classifier catches 16 of the 20 true rollbacks (TP = 16, FN = 4) and wrongly flags 18 of the 180 healthy deployments (FP = 18, TN = 162).
Precision asks: of every deployment the model flagged, how many needed a rollback?
\[\text{Precision} = \frac{TP}{TP + FP} = \frac{16}{16 + 18} = 0.47\]
Under half the flagged deployments turn out to need a rollback. Every flag costs an engineer’s time to review, so low precision means a team spends most of that review time on deployments that were never a problem.
Recall (also called sensitivity or the true positive rate) asks a different question: of every deployment that truly needed a rollback, how many did the model catch?
\[\text{Recall} = \frac{TP}{TP + FN} = \frac{16}{16 + 4} = 0.80\]
The model catches 80% of the deployments that needed a rollback and lets the remaining 20% through. Precision and recall pull in opposite directions: flagging more deployments to raise recall means flagging more borderline cases too, which drags precision down, and the reverse holds for tightening the flag to raise precision.
Chasing precision and recall separately never settles which threshold is best; a stakeholder still has to decide which mistake, a missed rollback or a wasted review, costs more. Precision and recall describe the trade-off. They do not resolve it.
F1 score summarizes the trade-off in one number, the harmonic mean of precision and recall:
\[F_1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} = 2 \times \frac{0.47 \times 0.80}{0.47 + 0.80} \approx 0.59\]
The harmonic mean punishes a lopsided score more than a plain average would: a model with precision 0.9 and recall 0.1 averages to 0.5 but F1-scores closer to 0.18, because a model that almost never flags anything is not a useful rollback classifier no matter how clean its few flags are.
Accuracy on this same held-out set is \((16 + 162) / 200 = 89\%\), which sounds close to good. A classifier that predicted “no rollback” for every single deployment, doing no work at all, would score \(180 / 200 = 90\%\): higher than the model that is catching four out of five true rollbacks. On an imbalanced dataset like this one, where most deployments never need a rollback, accuracy rewards the model for agreeing with the majority class and says almost nothing about whether it catches the rare, costly case. Precision, recall, and F1 do not carry that blind spot, since each one is computed against the minority class directly.
Figure 7.6 ties the two views together on a single held-out set. The classifier does not output “rollback” or “no rollback” directly; it outputs a predicted probability, and a threshold turns that probability into a decision. The rollback numbers worked by hand above used a threshold of 0.5, but nothing forces that choice. Drag the slider down toward 0.1 and the model flags almost every deployment with any elevated risk at all: the confusion matrix’s false positive count climbs, precision falls, and recall climbs toward 1.0. Drag it up toward 0.9 and the opposite happens: the model only flags deployments it is highly confident about, precision climbs, and recall falls as more true rollbacks slip through unflagged.
The third panel plots the same sweep a different way. A ROC curve (receiver operating characteristic, a name left over from its origin in World War II radar signal detection) plots the true positive rate against the false positive rate at every possible threshold, tracing out one continuous curve instead of a single point.
A model with no signal at all traces the diagonal line, since raising the threshold in a coin-flip model raises both rates by the same amount. A model worth deploying bows up and to the left of that diagonal, meaning it can raise its true positive rate while keeping the false positive rate low.
AUC (area under that curve) compresses the whole curve into one number between 0.5 (no better than a coin flip) and 1.0 (perfect separation). AUC has a plain-language reading that does not require picking a threshold at all: it is the probability that the model ranks a randomly chosen rollback deployment above a randomly chosen healthy one. An AUC of 0.85, for instance, means that roughly nine times out of ten, a true rollback deployment gets a higher predicted risk score than a healthy deployment picked at random.
Precision, recall, and F1 all depend on picking a threshold first, then judging the model on that one operating point. AUC judges the model’s ranking across every threshold at once, which is the right lens when a threshold has not been chosen yet, for instance while comparing candidate models before committing to a production cutoff. Once a threshold is fixed, for instance because engineering review capacity puts a hard cap on how many deployments can be flagged per day, precision, recall, and F1 at that specific threshold matter more than the model’s AUC across thresholds it will never operate at.
Putting the guidance in one place: reach for precision, recall, or F1 over plain accuracy the moment one class is rare relative to the other, which describes most of the production classification problems this book’s methods get applied to. Reach for AUC when comparing models before a threshold is set, or when the cost of a false positive and a false negative are close enough that no single threshold is an obviously correct default. Reach for a fixed-threshold metric once the threshold is locked in by a concrete operational constraint, since AUC alone cannot tell a team whether the threshold it ended up choosing is a good one.
7.10 Variable importance
A single decision tree is easy to read directly: trace the path from root to leaf. A forest of several hundred trees is not, which is the trade-off random forests make for their lower-variance predictions.
Variable importance recovers some of that lost interpretability. For each predictor, it measures the total decrease in RSS (regression) or Gini index (classification) produced by splits on that predictor, summed across every tree in the forest and averaged.
This measure is a rough ranking, not a precise decomposition of credit. A predictor with many candidate split thresholds gets more chances to turn up a locally good split than one with few distinct values, which can inflate its importance for reasons unrelated to how much it drives the outcome.
Two correlated predictors, meanwhile, split the credit for the same signal between them, which can make each one look weaker than either would look alone (Strobl et al. 2007).
Variable importance is a guide to where the forest is finding structure, not a certified attribution of cause. Treat a low score with suspicion if that predictor is correlated with a high-scoring one.
Figure 7.7 ranks the four rollback features by importance. Canary error rate dominates, which matches intuition.
But service dependency count carries more signal than hour of day, a finding worth acting on: a deployment touching many downstream services is meaningfully riskier at a given error rate than a deployment with few dependencies. That is the kind of interaction a human reviewer skimming a dashboard of raw error rates would likely miss.
7.11 Permutation importance, and where Gini and entropy importance mislead
Figure 7.8 adds one feature to the rollback dataset that never appeared in the worked examples above: a deployment batch ID, a number assigned in sequence to every deployment and otherwise unconnected to whether that deployment needed a rollback. The forest still reports a Gini importance score for it, 0.141, ahead of both dependency count (0.074) and hour of day (0.089). Entropy importance tells the same story: 0.150, again ahead of both. Permutation importance disagrees outright. It scores the batch ID at -0.006, the lowest score of all five features, trailing dependency count, hour of day, and even payload size: shuffling the batch ID costs the forest nothing, which is the tell that the forest never depended on it to predict well in the first place.
That disagreement is the point of the figure, and it is worth understanding before trusting either kind of importance score on a dataset with predictors of differing cardinality.
Permutation importance measures how much a fitted model’s accuracy drops when one feature’s column is shuffled at random on a held-out validation split, breaking that feature’s link to the outcome while leaving the fitted model and every other column untouched (Breiman 2001). A feature whose shuffled version costs the model nothing was carrying no predictive weight inside the model.
Gini and entropy importance measure something different: how much a predictor decreased impurity, summed across every split that used it, while the forest was being built. A predictor gets credit each time it wins a split, whether or not that split turns out to matter on data the forest has never seen.
The batch ID wins splits for a mechanical reason that has nothing to do with rollback risk: it has roughly 2,000 distinct values in a dataset with only a few thousand rows, so a tree searching for the best threshold on that column has thousands of candidate cut points to try, each one a fresh chance to carve out a small region that happens to look purer than it should by chance. Canary error rate is also continuous, but the split search finds its strongest signal early and keeps reusing it; dependency count and hour of day are lower-cardinality integers with far fewer candidate thresholds to search in the first place. Cardinality, not predictive value, is driving the gap.
Gini and entropy importance are biased toward high-cardinality and continuous predictors, a result formalized by the same source cited above for the correlated-predictor bias (Strobl et al. 2007). A field like a request ID, a timestamp cast to an integer, or a high-cardinality categorical code (zip code, SKU, session ID) can rank as an important predictor by Gini or entropy importance for no reason beyond how many places it gives the tree to search, not because it carries a signal that will hold up on new data.
Computing permutation importance takes more work than reading feature_importances_ off a fitted forest: it needs a held-out split the forest never trained on, and enough repeats of the shuffle-and-rescore loop (twenty, in the figure above) to average out the noise a single random shuffle introduces. sklearn.inspection.permutation_importance runs that loop directly and hands back both the mean importance and its spread across repeats.
The practical rule this figure earns: when predictors differ a lot in cardinality, which is common the moment a dataset includes anything like an ID field, a date broken into components, or a mix of binary flags and continuous measurements, reach for permutation importance before trusting a Gini or entropy ranking. Treat Gini and entropy importance as a first look, since the forest is computing them for free during training anyway, then confirm anything that ranking suggests is worth acting on against a permutation-based check before removing a feature or building a monitoring dashboard around it.
7.12 Decision boundaries: a tree, a forest, and logistic regression compared
Figure 7.9 answers a question the earlier sections in this chapter answered only in prose: what shape does the boundary between “rollback” and “no rollback” take, on the same data, once a tree, a forest, and a linear classifier each draw their own version of it?
A decision boundary is the line, or in a tree’s case the set of rectangular edges, separating the region of feature space where a classifier predicts one class from the region where it predicts the other. The single tree’s boundary is visibly blocky: a handful of straight, axis-aligned edges, each one a leftover from a single split on canary error rate or dependency count. The forest’s boundary traces roughly the same shape but with the corners softened, since it is an average over hundreds of slightly different trees, each with its own slightly different split thresholds. Logistic regression’s boundary is a single straight line, tilted, because it can only ever draw one line through two-dimensional feature space no matter how the data curves.
The gap between these boundaries matters for a concrete reason beyond aesthetics. If the relationship between error rate, dependency count, and rollback risk bends the way the tree and forest panels suggest, for instance if risk rises sharply once both error rate and dependency count cross their own thresholds together, a straight decision boundary will misclassify a describable band of deployments sitting on the wrong side of that line: the ones where a single linear combination of the two features cannot capture the joint effect. If the underlying relationship were close to linear instead, logistic regression would need far less training data to estimate that one line than a forest needs to approximate the same line out of many small rectangles stacked together, since the forest is not told the boundary is a line and has to discover its shape from splits.
Each boundary here is one plausible read of this simulated dataset: any of the three could sit closest to the rule that generated it. The value of the comparison is in seeing how differently each model class carves up the same feature space, not in picking a winner from the picture alone.
Building this figure only takes a fitted model and a grid: fit each of the three classifiers on the same two-column feature matrix, predict the probability of rollback across a fine grid spanning the observed range of both features, reshape those predictions back into the grid’s two-dimensional shape, and shade each cell by the predicted probability. Overlaying the deployments on top of each shaded panel shows how each boundary relates to where the data sits, rather than leaving the boundary as an abstract shape with no anchor to the observations that produced it.
7.13 When to reach for a single tree, a forest, or boosting
Four different tree-based tools have appeared across this chapter, and it is worth being explicit about when each earns its place, since defaulting to the fanciest available option is its own kind of mistake.
A single pruned tree is the right choice when the audience for the model needs to see the reasoning behind a prediction, not just trust a number. A compliance reviewer asking why a particular deployment was auto-rolled-back wants an answer like “canary error rate exceeded 2% and the deployment touched more than 6 downstream services,” which only a small, interpretable tree delivers directly. A 400-tree forest cannot be read that way at all, even though Figure 7.9 shows its predictions land close to the same shape a single tree draws.
A random forest is the right default for most production prediction problems where accuracy matters more than a fully traceable explanation. It trains fast, tunes with almost no effort (the number of trees and the number of features sampled per split are close to the only knobs), and gives OOB error and variable importance essentially for free.
The rollback classifier in this chapter is a clear case of this kind of problem: nobody needs to explain each individual prediction. They need the aggregate error rate low enough to trust the automation.
Gradient boosting, the subject of the next chapter, usually edges out a random forest on raw predictive accuracy on structured, tabular data like this one, at the cost of being more sensitive to hyperparameter choices and slower to reason about informally. It fits trees sequentially, each one correcting the errors of the ensemble built so far, rather than fitting many independent trees in parallel the way bagging and random forests do.
BART is worth the added compute specifically when a wrong prediction is costly enough that the team needs to know how much to trust each individual prediction, not just the aggregate error rate. An automated rollback system with no human review is a much stronger candidate for BART’s credible intervals, the per-prediction uncertainty ranges the next section works through, than a dashboard a human glances at before making the final call.
7.14 A Bayesian perspective
Picture asking a group of cautious forecasters for a prediction, where no single one is allowed to move the answer far from where the group started. Their small, careful adjustments add up to both a final guess and a sense of how much the group agrees, which is what BART delivers below.
Random forests give a point prediction and, through OOB error, a rough sense of how often the forest is wrong overall. Bayesian Additive Regression Trees (BART), introduced by Chipman, George, and McCulloch, go further and treat the sum of trees itself as the object of Bayesian inference rather than a black box averaged after the fact. That produces a full uncertainty interval around each individual prediction (Chipman, George, and McCulloch 2010).
That interval tends to be well calibrated in practice, but calibration is not automatic: it depends on the priors chosen for tree structure and leaf values and on the MCMC chain having converged, the kind of check Chapter 9 introduces for diagnosing a model fit by MCMC.
BART builds its prediction as a sum of many trees, typically 100 to 200, but constrains every individual tree to be a weak learner through the prior distribution placed on tree structure and on the value predicted at each leaf. The prior favors shallow trees with small leaf values, so no single tree is allowed to explain much of the outcome on its own; the trees only become expressive in aggregate, through their sum.
This is a meaningfully different mechanism from a random forest’s variance reduction through averaging many strong, overfit trees. BART instead reduces variance by keeping every individual tree weak from the start, then lets a Bayesian backfitting Markov chain Monte Carlo algorithm sample from the posterior distribution over the entire sum of trees.
In other words, instead of settling on one best-fitting sum of trees, the algorithm keeps track of many plausible sums of trees that all fit the training data reasonably well, each one weighted by how well it fits.
That posterior is the payoff. Instead of one point prediction per deployment, BART returns a distribution of predictions, one per MCMC sample, from which a full credible interval (the Bayesian analog of a confidence interval) follows directly: not just “this deployment has an 82% chance of needing a rollback” but a defensible range around that 82%, reflecting how much the training data supports it.
BART also gives a Bayesian analog to variable importance for free, since the posterior tracks how often each predictor gets used for a split across the sampled trees. A predictor used rarely across the posterior draws is one the data does not support leaning on.
A wide credible interval is information, not noise. It tells the team this specific prediction needs a human look before acting on it, even when the point estimate looks confident.
Figure 7.10 lines the two models up on the same eight held-out deployments. The point predictions mostly agree, but BART’s credible intervals are not the same width from deployment to deployment: the model is honest about which predictions rest on thin evidence and which ones it can stand behind.
The trade-off is computational. Fitting a random forest of several hundred trees is fast and trivially parallel across trees; fitting BART requires running an MCMC chain to convergence, which is slower and does not parallelize the same way, since each MCMC iteration depends on the last.
For the rollback classifier, a random forest retrained nightly on the latest deployment data is a reasonable production choice. BART is worth reaching for when the cost of an overconfident prediction, not just an incorrect one, is high enough to justify the extra compute, for instance when the rollback decision feeds directly into an automated action with no human in the loop.