Getting Started with Controlled Experiments
Topic Overview
Exemplary Controlled Experiments
In keeping with the strategy of learning by example, one of the best ways to learn to do controlled experiments is to look at controlled experiments performed by others.
For incredible examples of controlled experiments in education, I recommend Neil Heffernan’s work on ASSISTments as a researcher’s tool. A full list of controlled experiments completed with ASSISTments can be found here.
I would especially recommend looking at the following studies, both for their demonstration of the design of controlled experiments and for their interesting conclusions:
- Does Immediate Feedback While Doing Homework Improve Learning, from Paul Kehrer, Kim Kelly, and Neil Heffernan of Worcester Polytechnic Institute
- What Level of Tutor Interaction is Best?, from Leena Razzaq, Neil Heffernan, and Robert Lindeman of Worcester Polytechnic Institute
- Is it Better to Give Hints or Let Students Ask?, from Leena Razzaq of the University of Massachusetts-Amherst and Neil Heffernan of Worcester Polytechnic Institute
- Choice in Feedback Mediums, from Korinn Ostrow and Neil Heffernan of Worcester Polytechnic Institute
- Comparing discovery learning with direct instruction replicating (and extending) the famous Klahr and Nigam result, from Michael Sao Pedro, Janice Gobert, Neil Heffernan, and Joseph Beck of Worcester Polytechnic Institute
Controlled and Quasi-Experiments in Education
Controlled experiments are often hard to do in education. We can do controlled experiments in short lab studies, but real learning occurs over much longer time scales. However, over those longer time scales, it’s difficult to have the control necessary over the various lurking variables to perform true controlled experiments.
Here are a few sources on doing controlled experiments in education:
- Randomized Controlled Experiments in Education, from Adrien Bouguen and Marc Gurgand of the European Expert Network on Economics of Education
- Randomized Trials and Quasi-Experiments in Education Research, from Joshua Angrist of the National Bureau of Economic Research
- Identifying and Implementing Educational Practices Supported By Rigorous Evidence: A User Friendly Guide, from the National Center for Education Evaluation and Regional Assistance
- Quasi-Experiments in Schools: The Case for Historical Cohort Control Groups, from Tamara Walser of the University of North Carolina-Wilmington
- Experiments, from Research and Education
- Experiments, quasi-experiments, single-case research and internet-based experiments, from Research Methods in Education
- Controlled experiment replication in evaluation of e-learning system’s educational influence, from Ani Grubišic, Slavomir Stankov, Marko Rosic, and Branko Zitko of the University of Split
Scholarly Resources
For a little more information on the value, applicability, generalizability, and difficulties in the design of controlled experiment, see the following:
- What is evidence-based education?, from Philip Davies of the University of Oxford
- Random Versus Nonrandom Assignment in Controlled Experiments: Do You Get the Same Answer, from William Shadish and Kevin Ragsdale of the University of Memphis
- Practical guide to controlled experiments on the web: listen to your customers not to the hippo, from Ron Kohavi, Randal Henne, and Dan Sommerfield of Microsoft
- Controlled experiments on the web: survey and practical guide, from Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal Henne of Microsoft
- Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained, from Ron Kohavi, Alex Deng, Brian Frasca, Roger Longbotham, Toby Walker, and Ya Xu of Microsoft
- Seven Pitfalls to Avoid when Running, Controlled Experiments on the Web, from Thomas Crook, Brian Frasca, Ron Kohavi, and Roger Longbotham of Microsoft
- Lab experiments are a major source of knowledge in the social sciences, from Armin Falk and James Jeckman of the University of Munich
For more comprehensive information, see:
- Research Methods in Education (Chapter 12)
For demonstrations of good experimental design, we recommend using Google Scholar to search for controlled experiments in your chosen area. You’ll generally find several useful examples. Similarly, we’d recommend also searching similarly for quasi-experiments if you’re leaning in that direction. Or, see our own list of exemplary controlled experiments in education.
General Media
These sources would generally not be suitable for use in your assignments, but they may provide a useful general overview of the topic if you find yourself struggling with the more scholarly resources.
To get you started, here a few resources on what Controlled Experiments are and how to design them:
- Controlled Experiments in Technology and Physical Sciences, from the University of Cambridge
- Experimental Method, from Saul McLeod of SimplyPsychology
- Experimental Designs for Research, from Michelle Saint-Germain of California Stae University-Long Beach
- Experimental Research, from Boundless.com
- Experimental and Quasi-Experimental Designs for Research, from Donald Campbell of Syracuse University and Julian Stanley of Johns Hopkins University
- Experimental Research, from Education.com
- Experimental Design, from SocialResearchMethods.net