Puzzling outcomes in A/B testing
Interesting post and paper on AB goals and targets.
Few quick points on the subject
- Understand that long-term goals do not always align with short-term metrics.
- “Success” in your test may be just an instrumentation difference.
- Primacy effect occurs when you change the navigation on a web site, and experienced users may be less efficient until they get used to the new navigation, thus giving an inherent advantage to the Control. Conversely, when a new design or feature is introduced, some users will investigate the new feature, click everywhere, and thus introduce a “novelty” bias that dies quickly if the feature is not truly useful.
- Devil is in the details - generating numbers is easy, generating numbers you should trust is hard!
From MITSloan. Things to consider next time WFH or working remotely.
Its actually missing quite a few others - Pinterest, Fancy etc.
Primary school students with access to e-readers showed significant improvement in reading skills and in time spent reading, and the program is cost-effective. The theft rate was “near-zero,” but nearly half the e-readers broke.