Who Cited It

A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection

1995 · 10,743 citations · 10 from inside this corpus

Ron Kohavi

We review accuracy estimation methods and compare the two most common methods: crossvalidation and bootstrap. Recent experimental results on arti cial data and theoretical results in restricted settings have shown that for selecting a good classi er from a set of classiers (model selection), ten-fold cross-validation may be better than the more expensive leaveone-out cross-validation. We report on a largescale experiment| over half a million runs of C4. 5 and a Naive-Bayes algorithm| to estimate the e ects of di erent parameters on these algorithms on real-world datasets. For crossvalidation, we vary the number of folds and whether the folds are strati ed or not; for bootstrap, we vary the number of bootstrap samples. Our results indicate that for real-word datasets similar to ours, the best method to use for model selection is ten-fold strati ed cross validation, even if computation power allows using more folds.

A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection (1995)A Study of Cross-Validation a…Classification and Regression Trees. (1986)Classification and Regression…Stacked generalization (1992)Stacked generalizationWrappers for feature subset selection (1997)Wrappers for feature subset s…Survey of Clustering Algorithms (2005)Survey of Clustering Algorith…Bayesian Network Classifiers (1997)Bayesian Network ClassifiersSemi-Supervised Learning (2006)Semi-Supervised LearningAn Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Varia… (1999)An Empirical Comparison of Vo…MultiResUNet : Rethinking the U-Net architecture for multimodal biomedical image segmenta… (2019)MultiResUNet : Rethinking the…Sensitivity Analysis of k-Fold Cross Validation in Prediction Error Estimation (2009)Sensitivity Analysis of k-Fol…Deep Reinforcement Learning That Matters (2018)Deep Reinforcement Learning T…KEEL: a software tool to assess evolutionary algorithms for data mining problems (2008)KEEL: a software tool to asse…Automated Machine Learning (2019)Automated Machine Learning
12 of 12 neighbouring works in this corpus. Blue is what this paper cites; orange is what cites it, and a dashed line is one neighbour citing another. Only the largest labels are drawn — every node carries its full title on hover.
this paper works it cites works citing it node size = global citations · hover for the full title

Citation edge evidence

Each row is directed from the citing work to the cited work. Source status remains readable without the interactive graph.

What this paper cites, inside the corpus

PaperYearCited
Classification and Regression Trees.198621,044
Stacked generalization19927,777

What cites it, inside the corpus

PaperYearCited
Wrappers for feature subset selection19978,991
Survey of Clustering Algorithms20056,210
Bayesian Network Classifiers19974,754
Semi-Supervised Learning20064,334
An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Varia…19992,656
MultiResUNet : Rethinking the U-Net architecture for multimodal biomedical image segmenta…20192,334
Sensitivity Analysis of k-Fold Cross Validation in Prediction Error Estimation20091,897
Deep Reinforcement Learning That Matters20181,580
KEEL: a software tool to assess evolutionary algorithms for data mining problems20081,508
Automated Machine Learning20191,451

Links

Download BibTeX · Download RIS · DOI · Open access copy · OpenAlex record

Topics

Algorithms and Data CompressionComputer Science
Data Mining Algorithms and ApplicationsComputer Science
Machine Learning and Data ClassificationComputer Science

Is this record sound?

complete

Nothing in this record contradicts itself and no field we check is missing.

  • supports1 author record(s) attached.
  • supports14 reference(s) recorded.
  • neutralThe DOI carries no year to check against.
  • supportsA title is present.

Provenance

Everything above was read from one stored OpenAlex payload, fetched 2026-09-04T03:58:40+00:00.

sha256 7e3d99a592f7f61f…