What does the term reproducibility mean in reference to an experiment?
Why are the results from experiments random variables?
Why are the results form simulations random variables?
What two things must you do whenever you report the result of an experiment/simulation?
What do you need to do to demonstrate that a result has not been reproduced?
Whenever we do an experiment or simulation it is important that we give sufficient detail for others to reproduce our result. If what we have observed is important other scientists will want to redo what we did in order to confirm the conclusions that we arrived at.
In experiments: when you do an experiment you try to control all the factors that will affect the result. In practice, there will always be some factors that are not under your control though. There is thus a random component to the result you obtain because of these uncontrollable factors.
In simulations: many simulation methods involve generating (pseudo) random numbers. When reproducing simulation work it should be possible to (in theory) match the numbers obtained exactly by using the same random number seed. This is usually not desirable, however, and it is thus again necessary to assume that the result is a random variable.
To deal with random these random errors in simulation and experiments you:
An experiment has not been reproduced if there is sufficient evidence to show that the distribution of results obtained in the new experiment is different from the distribution obtained in the first experiment. Such judgements are made by doing a hypothesis test.
This idea of generating multiple results is discussed in this video:
https://www.youtube.com/embed/JBbu4aW3yvI
<aside> 📌 SUMMARY: When reporting the results from an experiment or simulation you must perform the experiment multiple times and characterize the distribution of results. This allows others to test if they have reproduced the result you obtained using hypothesis testing.
</aside>