Why XGBoost still matters

Every year someone announces that deep learning has finally conquered tabular data, and every year the models that actually ship in credit scoring, churn prediction, fraud detection, and demand forecasting are still gradient-boosted decision trees. This is not nostalgia. On heterogeneous tabular datasets — mixed numeric scales, categorical codes, skewed distributions, missing values, a few hundred informative columns — boosted trees remain the strongest practical baseline, and the practitioner consensus, backed by repeated benchmark studies, is that they win on this terrain more often than neural networks do. Trees need no normalization, ignore irrelevant features gracefully, treat missing values as first-class citizens, and train in minutes on commodity hardware.

XGBoost, introduced by Chen and Guestrin in their 2016 KDD paper, is the library that turned this model class into an industrial commodity. A decade on, it is still my first serious model on any new tabular problem. It delivers roughly 95% of the achievable accuracy for 5% of the effort, and — just as valuable — it tells you quickly whether the remaining 5% is worth anyone’s time. If you are tempted by a tabular transformer, fine: but make it beat a properly tuned XGBoost first. In my experience, it usually does not.

Quick start

Install from PyPI:

pip install xgboost

Here is a complete, runnable example with the scikit-learn-compatible XGBClassifier on the breast cancer dataset, including a validation split and early stopping. Note the XGBoost 2.x convention: early_stopping_rounds is a constructor argument, not a fit() argument.

import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

X, y = load_breast_cancer(return_X_y=True)
X_train, X_valid, y_train, y_valid = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

model = xgb.XGBClassifier(
    n_estimators=2000,          # a generous ceiling; early stopping finds the real number
    learning_rate=0.05,
    max_depth=5,
    subsample=0.8,
    colsample_bytree=0.8,
    eval_metric="auc",
    early_stopping_rounds=50,   # constructor argument in xgboost 2.x
    random_state=42,
)

model.fit(X_train, y_train, eval_set=[(X_valid, y_valid)], verbose=100)

print(f"Best iteration: {model.best_iteration}")
proba = model.predict_proba(X_valid)[:, 1]
print(f"Validation AUC: {roc_auc_score(y_valid, proba):.4f}")

That is the entire core workflow: set the tree budget high, let early stopping decide when to stop, and read your validation metric. Everything past this point is refinement.

The knobs that actually matter

XGBoost exposes dozens of parameters; five or six of them produce nearly all the gains. Tune these before touching anything exotic.

Early stopping done right

Early stopping is not a tuning trick — it is the correct way to choose the number of boosting rounds. Hand-picking n_estimators is guessing. The working pattern: set n_estimators to a ceiling of 2000–5000, set early_stopping_rounds somewhere between 10 and 50 depending on how noisy the validation metric is, and pass a held-out eval_set that the model never trains on. After fitting, model.best_iteration tells you where training stopped, and predict uses that best iteration automatically. The classic mistake is using your test set as the early-stopping set and then reporting metrics on it — that leaks. Keep three splits when you need an honest final number.

Learning rate vs n_estimators

These trade off directly: lower learning_rate means each tree contributes less, so you need more trees, and the ensemble generalizes better. My defaults: learning_rate=0.1 while iterating on features, then 0.05 or 0.03 for the final model, letting early stopping extend the tree count accordingly. Below 0.01, the extra training time almost never pays on typical business datasets. And never tune n_estimators by grid search — that is early stopping’s job.

Regularization (max_depth, min_child_weight, subsample, colsample_bytree)

max_depth is the dominant capacity control. The default of 6 is reasonable; use 3–5 on small or noisy data, 6–10 on large datasets with strong feature interactions. min_child_weight raises the threshold for making a split — push it to 3, 5, or 10 when validation curves show overfitting. subsample (row sampling per tree) and colsample_bytree (column sampling per tree) at 0.7–0.9 inject stochasticity that regularizes and speeds training; I set both to 0.8 almost reflexively. reg_alpha (L1) and reg_lambda (L2) on leaf weights exist too, but I reach for them last — the structural knobs above usually do the job.

GPU training (device=”cuda” / tree_method=”hist”)

Since XGBoost 2.0, GPU training is a one-line change: keep tree_method="hist" (the default histogram algorithm) and set device="cuda".

model = xgb.XGBClassifier(tree_method="hist", device="cuda", n_estimators=1000)

Above roughly a million rows, expect a 5–10x speedup. Below ~100k rows, transfer overhead means CPU hist is often just as fast, so do not assume the GPU always wins. The legacy gpu_hist value still functions but is deprecated; the device parameter is the current API.

Explainability with feature importance and SHAP

Built-in importances are the first sanity check. Prefer importance by gain over the default split-count weighting — frequently used is not the same as important:

import pandas as pd
from sklearn.datasets import load_breast_cancer

importances = pd.Series(
    model.feature_importances_,
    index=load_breast_cancer().feature_names,
).sort_values(ascending=False)
print(importances.head(10))

For anything shown to stakeholders or regulators, use SHAP. The shap package’s TreeExplainer works natively with XGBoost — shap.TreeExplainer(model) accepts a fitted XGBClassifier directly and computes exact Shapley values efficiently. SHAP gives per-prediction attributions rather than only a global ranking, which is the difference between “the model uses income” and “this applicant was declined mainly for these two reasons.” Gain importances are biased toward high-cardinality features; SHAP largely is not. Use both, trust SHAP.

When to use it

  • Heterogeneous tabular data with mixed numeric and categorical features — XGBoost’s home turf.
  • Medium-scale datasets, from a few thousand rows to tens of millions, where training time and iteration speed matter.
  • Strong baselines under deadline pressure — a defensible, evaluated, SHAP-explained model in an afternoon.
  • Data with missing values. Sparsity-aware split finding handles NaN natively, learning a default direction per split. No imputation pipeline required.
  • Regulated or audited settings where exact, fast tree SHAP beats the approximate attribution methods available for neural networks.
  • Ranking and survival problems — rank:ndcg, rank:pairwise, and AFT survival objectives ship in the box, which most competitors lack.

When not to use it

  • Text, images, audio. When signal lives in raw high-dimensional perceptual data, deep learning owns the problem. Trees cannot learn convolutional or sequential structure; the legitimate hybrid is feeding trees the embeddings.
  • Very small datasets — a few hundred rows. Regularized logistic regression is more stable and easier to defend. Boosting hundreds of trees on 200 rows is overfitting theater.
  • Strict latency or size budgets. A 1,500-tree ensemble is megabytes of model and real microseconds per score. For single-digit-microsecond serving or microcontroller deployment, a linear model wins outright.
  • Targets that must extrapolate. Trees predict constants outside the training range; if inference-time features drift past what training saw, XGBoost flatlines where a linear or neural model extrapolates.

Integration with IBM watsonx.ai

Enterprise AI platforms are marketed on their LLM capabilities, but most production value still flows through classical ML, and XGBoost is the workhorse of that side on IBM watsonx.ai. The platform’s Runtime (formerly Watson Machine Learning) treats XGBoost as a first-class deployable framework: train an XGBClassifier wherever you like, store it in a deployment space, and serve it behind a REST scoring endpoint with the same governance, monitoring, and access control applied to foundation-model assets. Your churn or credit-risk model lives in the same catalog as the LLM workflows instead of in a shadow stack.

The pattern I find most useful is the hybrid: the LLM handles language, the boosted trees handle calibrated prediction. A claims pipeline can call a watsonx.ai foundation model to extract structured fields from free-text descriptions — incident type, severity signals, inconsistency flags — then feed those fields plus conventional policy features into an XGBoost fraud scorer. The LLM is the feature extractor; XGBoost is the decision engine, because it is cheap to score, easy to calibrate, and exactly explainable. The connection sketch, with credentials strictly from environment variables:

import os
from ibm_watsonx_ai import APIClient, Credentials

client = APIClient(
    Credentials(
        url=os.environ["WATSONX_URL"],
        api_key=os.environ["WATSONX_APIKEY"],
    ),
    space_id=os.environ["WATSONX_SPACE_ID"],
)
# Store the fitted XGBoost model, deploy it, then score via
# client.deployments.score(deployment_id, {"input_data": [...]})

One model object, one online endpoint, full lineage in the governance layer.

Integration with IBM Watson Orchestrate

IBM watsonx Orchestrate works a level above the model: it chains skills and agents into automated business workflows, and XGBoost’s role there is as the scoring step inside a flow. A typical loan-origination automation: Orchestrate receives an application; a document skill extracts applicant data; a skill backed by a deployed XGBoost risk model — exposed as a REST endpoint from watsonx.ai Runtime and imported into Orchestrate via its OpenAPI specification — returns a default probability; a routing step auto-approves low-risk cases and queues high-risk ones for a human underwriter together with SHAP-based reason codes. The same shape fits invoice-fraud triage, churn-intervention campaigns, and supply-chain exception handling. The model never knows it is part of an agentic workflow; it just answers scoring calls quickly and deterministically, which is exactly what makes it a dependable component in one.

Alternatives compared

Library Training speed Categorical handling Ecosystem & deployment Best for
XGBoost Fast (hist), excellent GPU support Native (enable_categorical) but still maturing Largest: every cloud, sklearn, Spark, Dask Default choice; ranking/survival; peak accuracy with tuning
LightGBM Usually fastest on CPU, low memory Native, mature Strong, sklearn-compatible, industry standard Very large data, rapid iteration, forgiving defaults
CatBoost Slower training, fast inference Best in class (ordered target encoding) Good, somewhat smaller community Category-heavy data with minimal tuning
sklearn HistGradientBoosting Fast, CPU only Native (categorical_features) Zero extra dependencies Pure-sklearn stacks, prototypes, constrained environments
Neural nets (TabNet, FT-Transformer) Slow; needs GPU and tuning Learned embeddings, huge cardinality OK PyTorch ecosystem, heaviest ops burden Massive datasets, multimodal inputs, embedding reuse

My honest read after years with all five: accuracy differences between XGBoost, LightGBM, and CatBoost on a well-prepared dataset are usually within noise. What differs is the path to the result. LightGBM’s defaults are forgiving and its CPU training is noticeably faster on wide data, so it is my pick for fast experimentation at scale. CatBoost is the grab when categorical columns dominate — its ordered target encoding eliminates an entire class of preprocessing bugs.

XGBoost keeps the default slot because of everything around the model: documentation depth, deployment support on every major platform, exact tree SHAP, built-in ranking and survival objectives, and a decade of battle-tested stability. For a model that will live in production for three years, that maturity is worth more than half a point of log loss.

Tabular neural networks deserve a fair hearing rather than a dismissal. On very large datasets, or when you want embeddings shared with text or image inputs, FT-Transformer-style models can match or beat trees — at 10–100x the engineering cost. On the median business dataset, they lose. Run the boosted-tree baseline first and make the network earn its operational tax.

Limitations

Be clear-eyed about the rough edges. Memory: the histogram method is efficient, but hundreds of millions of rows push single-machine training toward swap, and while the external-memory and Dask/Spark modes work, they add genuine operational complexity — LightGBM tends to run lighter at the same scale. Hyperparameter sensitivity: XGBoost’s out-of-the-box settings leave more accuracy on the table than LightGBM’s defaults, so budget a real tuning pass. Categoricals: enable_categorical=True with pandas category dtypes works and keeps improving, but it is still less ergonomic and less accurate untuned than CatBoost’s treatment — for category-dominated data, preprocess carefully or switch libraries. And the universal tree caveats apply: no extrapolation beyond training ranges, and big ensembles carry nontrivial inference cost.

Final recommendation

If you work with tabular data, XGBoost belongs in your toolbox and should probably be your first serious model on any new problem. The recipe fits on an index card: generous n_estimators with early stopping on a clean validation split, learning_rate around 0.05, max_depth between 4 and 8, subsample and colsample_bytree at 0.8, device="cuda" once the data gets big, and SHAP plots ready before anyone asks why the model said no. Reach for LightGBM when iteration speed on huge data dominates, CatBoost when categoricals dominate, and a neural network only after it beats your boosted-tree baseline on your data. Most days the boring answer is the right one, and the boring answer is still XGBoost.

References

Powered by Jekyll & Minimal Mistakes.

About this article. This article was generated by the Best-of-the-Best autonomous AI digest and reviewed by Ruslan Magana Vsevolodovna. Package metadata was last checked on 11 June 2026. See the data leaderboard and the GitHub repository for sources.