An AB test sample size calculator. Calculations come through basic SciPy.stats methods, the web app is built with Streamlit and hosted on Heroku.
See the app live here.
See also my AB test significance calculator.
These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.
Clone/fork the repo onto your local machine.
It is then recommended to use a virtual environment to install the dependencies using the requirements.txt file.
pip install -r requirements.txt
With these installed, you simply need to run
streamlit run app.py
Alternatively, with Docker, use the following command and then navigate to localhost.
docker run -dp 80:8080 ryanfox212/samplesize
I utilised Streamit's Community Cloud service to host the app.
- Streamlit - The web application framework used and deployment service
- SciPy - For the statistical methods
- Seaborn - For vizualisations
The calculator asks for daily observations and conversions and the number of variants that you would like to test.
Optional inputs:
- business value
- significance level (1-alpha)
- statistical power
- maximum runtime
A notable exception from the possible inputs is minimum detectable effect that you hope to achieve through the change made. Instead, this sample size calculator outputs a range of MDEs based on how long you run the test.
Personally, I have found this form of output more useful when scoping out tests to understand the level of precision that might be achieved.
The number of visitors I need in my test to achieve the required sample by itself does not tell me very much. It is this number relative to my traffic levels that gives me the information I need.
Adding the business value of a conversion helps to understand what is the potential benefit/risk behind the change.
This project is licensed under the MIT License - see the LICENSE file for details.