Question

4.8 (Stratified sampling) Let D be a training set with only 10 examples, whose labels are 1,1,2,2,2,2,2,2,2,2, respectively. This dataset is both small in size and imbalanced. We need cross-validation during evaluation, and 2 -fold CV seems a good choice. (a) Write a program to randomly split this dataset into two subsets, with five examples in each subset. Repeat this random split 10 times. The histogram of class 1 examples in these two subsets can be (0,2) or (1,1)-one subset has zero (two) and the other has two (zero) class 1 examples, or every subset has exactly one class 1 example. In your 10 splits, how many times does (0,2) appear? (Note: This number can be different if you perform the experiments multiple times.) (b) What is the probability that (0,2) will appear in one random split of these 10 examples? (c) In your 2-fold CV evaluation, if the split's class 1 distribution in the two subsets is (0,2), how will it affect the evaluation? (d) One commonly used way to avoid this issue to use stratified sampling. In stratified sampling, we perform the train/test split for every class separately. Show that if stratified sampling is used, the distribution of class 1 examples will always be (1,1)

Answer 1

The question pertains to stratified sampling, which ensures balanced class distribution in subsets during cross-validation, preventing skewed results that could occur with random sampling. Using this method, a 2-fold CV will always have a balanced (1,1) distribution of class 1 examples in both subsets.

Step-by-step explanation:

The question deals with stratified sampling in the context of machine learning and data analysis. In this scenario, a dataset labeled with class 1 and class 2 is given, and the objective is to perform a 2-fold cross-validation (CV) using stratified sampling to maintain a balanced distribution of classes in each subset.

Stratified sampling ensures that both subsets of the data contain a proportional representation of each class. This is in contrast to random sampling, where the class distribution in the subsets could be imbalanced, resulting in potentially skewed evaluation results.

The probability of a (0,2) distribution during a random split might be lower with stratified sampling because this method specifically aims to prevent such imbalances by design - each class is represented proportionally in every subset created during a stratified sampling process.

When applying a 2-fold CV evaluation with stratified sampling, the distribution of class 1 examples will always be (1,1), meaning that each subset will have an equal number of class 1 examples. This representation ensures that the evaluation is fair and balanced, reflecting the true performance of the model across different subsets of the data.