If the null hypothesis of an A/B test is correct, the order of labels should not affect the differences in means between each group.
Why do we shuffle labels in an a/b test?
One of the reasons why we shuffle labels is to prevent confirmation bias. If the labels are not shuffled, researchers might unconsciously favor one group over the other, which can lead to biased results. Shuffling the labels helps to ensure that researchers treat each group equally and eliminates any potential for bias.
Another reason is to control confounding variables. Confounding variables are related to both the treatment variable and the outcome variable. If not controlled, they can lead to spurious results.