Data in Schools: Moneyball vs Trouble with the Curve
Statistics in schools have a controversial history. On one hand, statistics bring the promise of impartial decision-making and using the magic of numbers to see what works, and what doesn't. On the other hand, statistics can be manipulated or represent a "black box" which school leaders often find difficult to understand, leading educators to ignore data in favor of intuition.
One can think of these outcomes along a spectrum between two sports movies: Moneyball and Trouble with the Curve. In Moneyball, the heroes marshal the power of statistics to look at baseball players in a new way, ignoring biases like a player's "look" that scouts had traditionally focused on. By letting data be their guide, the Oakland A's revolutionized baseball. In contrast, Trouble with the Curve focuses on the unique sound that a well-thrown ball makes when it hits the glove. The lived experience of a baseball scout counts for so much more than the numbers thrown around by a computer.
These evaluation approaches represent two extremes, and like many things, the truth is probably somewhere in the middle. Data has an essential role in modern baseball, but it is most powerful when coupled with intuition and lived experience.
Data in schools is the same way. When relied on without question, data can generate significant controversy. When data is ignored, however, schools refuse themselves a powerful tool that could help improve student learning.
Data is most likely to be ignored in cases where black box problems occur; that is, when data is fed into a statistical process, results come out of the process, and nobody knows what happened in between. If we can solve black box problems, then we can begin to see the biases in our numbers and stop manipulation from occurring.
There are two critical ways to avoid black box problems:
First, if the statistical model can't be explained in plain English, results should be taken with a grain of salt. If you can't understand the basics of the models you use to help your decision-making, you put yourself at the mercy of the modelers rather than staying in charge of the process. Good statistical models, well-explained, can be understood at a fundamental level without advanced degrees in statistics. Their biases and assumptions should be clear.
Second, if the results of a model can't be confirmed using simple statistics, then you should be skeptical about the results. This is particularly true when the results of model are unusual or run counter to lived experience. Good statistical work connects your data to decision-making in ways that give you confidence.
Once you understand the process at hand, you can begin to check for manipulation or other data issues. When statistical outcomes of a process you understand give results that run sharply counter to your experience, you begin to worry about foul play. For example, if a school accountability system reports that a particular instructor adds no value in the classroom, despite observations showing excellent, effective teaching, then the data need a second look. Did something occur on test day? Were prior scores unusually high, so that it would be hard to add value? Do the results represent an unusually small sample? The statistical results offer a chance to dig deeper.
Statistics can become a powerful tool when yielded correctly. Data-drive strategy and near-real time accountability add real value to administrative decisions, especially when used in conjunction with the lived insights of experienced educators. When school leaders combine the power of Moneyball with Trouble with the Curve experience, they harness the best of both worlds to improve student learning.