Leave-one-out cross-validation (LOOCV) trains on every sample but one and tests on the one left out, repeating this n times. Stratified k-fold splits the data into k folds that each keep roughly the same class proportions as the full dataset. They answer different questions: LOOCV is the extreme case of k-fold where k equals n, while stratification is a way of building folds so rare classes are not lost from some of them. The version-specific details below come from scikit-learn 1.9.1 as of 2026-11-19.
How LOOCV works
The scikit-learn guide defines it plainly: each learning set is created by taking all samples except one, the test set being the sample left out, so n samples give n training sets and n test sets. Here is the guide's four-sample example, which prints the training and test indices for each split:
from sklearn.model_selection import LeaveOneOut
X = [1, 2, 3, 4]
loo = LeaveOneOut()
for train, test in loo.split(X):
print("%s %s" % (train, test))
# [1 2 3] [0]
# [0 2 3] [1]
# [0 1 3] [2]
# [0 1 2] [3]
Every sample is tested exactly once. The scikit-learn LeaveOneOut page adds that this method "can be very costly" because the number of test sets equals the number of samples, and recommends KFold, ShuffleSplit or StratifiedKFold for large datasets.
How stratified k-fold works
KFold splits samples into k groups of equal size where possible, trains on k โ 1 folds and tests on the remaining one. It ignores classes. StratifiedKFold changes how the folds are filled. The guide's example uses 50 samples, 45 of class 0 and 5 of class 1, with three splits. The counts of each class in training and test sets are:
| Splitter | Fold | Training (class 0, class 1) | Test (class 0, class 1) |
|---|---|---|---|
| StratifiedKFold | 1 | 30, 3 | 15, 2 |
| StratifiedKFold | 2 | 30, 3 | 15, 2 |
| StratifiedKFold | 3 | 30, 4 | 15, 1 |
| KFold | 1 | 28, 5 | 17, 0 |
| KFold | 2 | 28, 5 | 17, 0 |
| KFold | 3 | 34, 0 | 11, 5 |
Plain KFold produces a test fold with no class-1 samples, which makes a per-class metric meaningless there, and another that holds all five. The guide says StratifiedKFold preserves class ratios of roughly 1 in 10 in both train and test sets.
Cost compared with numbers
Take n = 1,000 samples. LOOCV fits 1,000 models, each trained on 999 samples. Ten-fold fits 10 models, each trained on 900 samples (the UW lecture gives the training size as (kโ1)/k ร n). So ten-fold is a hundred times cheaper in fits, and LOOCV trains on 99 more samples per fit.
Bias and variance: two sources, one picture
The sources agree that LOOCV costs more. They frame the accuracy trade-off differently:
- The UW lecture says LOO is "almost unbiased" because n โ 1 points are close to n, and k-fold is "more (pessimistically) biased" because it uses less data per fit, "usually k = 10."
- The scikit-learn guide says LOO often results in high variance as an estimator of test error, since the n models are virtually identical to each other. It also notes that if the learning curve is steep at the training size in question, 5 or 10-fold can overestimate generalization error.
Its recommendation: "most authors and empirical evidence suggest that 5 or 10-fold cross validation should be preferred to LOO."
A caution about stratification
The scikit-learn guide says stratified sampling was introduced "to workaround the aforementioned engineering problems rather than solve a statistical one," and warns that stratification can artificially shrink the spread of the metric across folds, so inter-fold variability no longer reflects the uncertainty in performance when classes are rare. Do not read a tight fold-to-fold spread as proof of a stable model.
Choosing between them
- Classification with imbalanced classes: use StratifiedKFold with 5 or 10 folds so every fold contains the rare class.
- Regression or balanced classes, moderate or large n: plain KFold with 5 or 10 folds is the documented default choice.
- Very small n where every training sample matters and fits are cheap: LOOCV is an option, with the variance caveat above. The scikit-learn page steers large datasets away from it, which leaves it for small ones.
Take this further with Oakgen Agent
Agent can explain the methods, write the scikit-learn code, run calculations in a Python sandbox, and create a downloadable .md or .docx summary.
Open this task in Oakgen Agent
The prompt is filled in for you. Nothing is sent until you press send.
Frequently asked questions
What is leave-one-out cross-validation (LOOCV)?
Each sample is used once as the test set while all other samples form the training set. The scikit-learn user guide says that for n samples there are n different training sets and n different test sets. scikit-learn's LeaveOneOut() is equivalent to KFold(n_splits=n) in terms of the splits it produces.
What does stratified k-fold do differently?
StratifiedKFold returns folds in which each set contains approximately the same percentage of samples of each target class as the complete set. In the scikit-learn guide's example with 45 samples of class 0 and 5 of class 1 and three splits, stratified folds preserve the class ratio of roughly 1 in 10 in both training and test sets, while plain KFold gives two test folds with no class-1 samples at all and one holding all five.
Is LOOCV better than 10-fold cross-validation?
Not by default. The scikit-learn guide says most authors and empirical evidence suggest that 5 or 10-fold cross-validation should be preferred to LOO, because LOO is more computationally expensive and often has high variance as an estimator of test error. The University of Washington CSE 446 lecture says LOO is almost unbiased but takes a lot of computational time, and that k-fold is faster, usually with k = 10, at the price of a pessimistic bias.
Can you stratify leave-one-out?
Stratification has nothing to preserve in LOOCV, because each test set holds a single sample, so there is no class mix to balance. This follows from the definition; the sources read for this guide do not discuss a stratified leave-one-out.
What are the default settings of StratifiedKFold in scikit-learn?
In the scikit-learn 1.9.1 documentation read on 2026-11-19, the signature is StratifiedKFold(n_splits=5, *, shuffle=False, random_state=None). The default n_splits changed from 3 to 5 in version 0.22, n_splits must be at least 2, and the test sets are generated so the smallest and largest differ by at most one sample.
Sources
- scikit-learn 1.9.1 user guide, Cross-validation: evaluating estimator performance (read 2026-11-19)
- scikit-learn 1.9.1 API reference, StratifiedKFold (read 2026-11-19)
- scikit-learn 1.9.1 API reference, LeaveOneOut (read 2026-11-19)
- University of Washington, CSE 446 Machine Learning, autumn 2025, lecture 4 slides (read 2026-11-19)


