On Facts

“Facts,” we know, have been the focus of public debate and discussion in recent months. It seems strange that such a simple word would be the cause of so much disagreement, but perhaps we should not be surprised.

Typically, we apply the word “fact” to that which is true. Implicit is that truth is objectivity, which means everyone agrees on the truth of a fact. If that definition of “fact” was accurate, then each individual having his or her own “facts” would not be possible. Because the disagreement is real (and politically quite powerful individuals are claiming the right to their own facts), I can only conclude the concept of “fact” as an objectively true statement is not accurate.

A more sophisticated concept of “fact” has been emerging for some time, and it is time for it to gain more widespread usage. Let’s take a closer look at “fact” and see if we can understand the disagreement over “facts.”

First, let’s begin by recognizing all “facts” are either supported by objective evidence or they are not. This leads to the counterintuitive, but correct, conclusion that there are “true facts” (those supported by objective observation) and “false facts” (those that are not). In the ideal world, and when science operates as it should, false facts are discarded and true facts are kept. True facts can become false facts when more accurate observations are made or are discovered.

Second, the observations used to support our facts are based on assumptions, and those assumptions can be correct or incorrect. Consider the question, “How much rain did I get at my house in the last 24 hours?” and my answer “one inch.” I can provide evidence from the rain gauge I keep attached to my garage to support my conclusion that my fact is true. Evidence that my gauge also collected rain that splashed off the roof or that my gauge is incorrectly calibrated can demonstrate that my fact is actually false.

So, the defining characteristics of “fact” is not truthfulness, but testability. “Facts” are those statements that can be demonstrated to be true (thus they can also be demonstrated as false). This more sophisticated definition of fact has unavoidable (and troubling for some) implications.

The quality of your facts depends on the quality of your evidence; they are inseparable. Even the facts we want desperately to be true may true out to be false; this is true even for those facts for which the falsifying evidence takes a long time to become obvious. Once evidence proves a fact to be false, it must be discarded and replaced with a new fact.

Once an individual comes to accept the truthfulness of a fact, he or she is likely to hold that truthfulness even in the face of growing evidence the fact is false. We can see this in those who hold any variety of supernatural beliefs, those who are deeply committed to a cause or a person, and those who hold other delusional or confabulatory beliefs. In these individuals, we see a tendency to valuable particular facts, but not facts in general. These individual hold their facts to be true in the face of indisputable evidence. Imagine the child with a frosting-cover face sitting in front of the pile of cupcake crumbs, some falling from his face as he denies eating the cake; he is an adorable and harmless example of these individuals.

It is an unfortunate reality that in many social groups, changing one’s mind, even in light of new and overwhelming evidence is perceived a s a weakness. The norm in these groups is to defend one’s first true fact, and abandoning it is a sign of weakness. In many cases, those whose first true fact will deny new evidence or will reinterpret both the old evidence and the new evidence to reaffirm the first true fact.

This of course leads to one final aspect of fact; all facts can be interpreted. Humans place facts in greater or lesser context and they assign different meanings to the same situations. And these are often presented as facts. Because these interpretations are based on values and meaning assigned by subjective humans, they cannot be facts which must be testable (and found true or false).

 

 

 

 

Being Data-Driven is Nothing to Brag About

Being Data-Driven is Nothing to Brag About

(c) 2016 Dr. Gary L. Ackerman

“Data-driven” has been the mantra of educators for the last generation. This mantra captures the practice of using students’ performance on tests to make instructional decisions. This model can be criticized for several reasons including the dubious reliability and validity of tests, the lack of control over variables, and incomplete and inappropriate analysis. My purpose here, however, is to criticize the “what works” focus that accompanies “data-driven decisions.”

Ostensibly, educators adopt a data-driven stance to create a sense of objectivity; they can reason, “I am taking these actions because, ‘it works’ to improve achievement.” The problem with this approach is that identifying “what works” is a superficial endeavor and it can be used on only very limited circumstances.

While designing physical systems, engineers can apply “what works” methods to improve their systems. Engineers can conceive and plan, build and test, then deploy their systems. At any point in the process they can change definitions of what “it works” means or abandon the project if “it works, but is too expensive” (or if other insurmountable problems arise). Ascertaining “what works” in educational settings is a far less controlled situation. Those who have tried to use others’ lessons plans and found the results disappointing have first-hand experience with this effect.

Understanding “Data-Driven” As a Scientific Endeavor

Humans have created two activities that are data-driven. In basic science, we use data to organize and understand nature so that we can support theories that allow us to predict and explain observations. In applied science, we gather data to understand how well our systems function.

Data-driven approaches to refine systems to build “what works” is the approach used by technologists who work in applied science. Vannevar Bush, a science advisor in President Franklin Roosevelt during and after World War II, placed basic and applied research as opposite ends of a continuum. Basic science was undertaken to make discoveries about the world, and applied science was undertaken to use and control those discoveries to develop tools useful to humans.

If we place data-driven education along this continuum, it must be considered an applied science as it is undertaken to build systems to instruct children. As it is typically undertaken, there is little attempt to understand why or how “it works,” as answering this questions are in the domain in basic science.

Continuum of basic to applied science as proposed by Vannever Bush
Figure 1. Continuum of basic to applied science as proposed by Vannever Bush

This is a very dissatisfying situation for educators (both those who claim to be data-driven and those who make no such claim). Fortunately, we can reconcile that dissatisfaction by recognizing that the basic to applied science continuum does not accurately describe the landscape of education.

Use-Based Research

In 1997, Donald Stokes, a professor of public policy at Princeton University, suggested the understanding that basic researchers seek and the use that applied researchers seek are different dimensions of the same endeavor, so research is not either basic or applied. According to Stokes, the continuum of science should be replaced with the matrix shown in figure 2.

The matrix created by placing the question “Do the researchers seek to develop or refine systems?” along the x-axis and “Do the researchers seek to make new discoveries?” along the y-axis creates four categories into which one can place a science-like activity:

  • Pure research is Bush’s basic research, and it is undertaken to satisfy curiosity, so the researchers are not motivated to create useful systems.
  • Technology development is Bush’s applied research, and researchers seek to develop useful systems, but they do not seek to make generalizations beyond those needed to build their systems.
  • Purposeful hobbies are undertaken for entertainment, and hobbyists are not motivated to share their systems they use or to make discoveries.
  • Use-based research is the label applied to endeavors in which the researchers seek to both develop new systems and make discoveries about the work.
Figure 2. Stokes' matrix of data activities
Figure 2. Stokes’ matrix of data activities

Stokes used the term “Pasteur’s quadrant” to capture the nature of work in the use-based research quadrant. He reasoned Pasteur’s work in microbiology had multiple purposes. As he developed methods of preventing disease (these are the  technologies he developed); Pasteur also sought to discover how and why the technologies worked, thus he established important details of microbial life.

Replacing Data-Driven Decisions

Educators who choose to adopt a more sophisticated approach to using data to drive decisions can adopt use-based research. This will require they begin to approach  data, its collection, and analysis in a more sophisticated manner. These educators will be faced with more work, but it is more interesting and more efficacious than the data-driven methods I typically observe. Use-based research necessitates educators:

  • Begin data projects and analysis with a question. The question cannot be “Which instruction is better?” It must be focused and precise: “Did the students who experienced intervention x perform better on y test?” They must also recognize that these questions can only be answered with large cohorts of students and using statistical methods. Further, these answers (like all answers supported with data) cannot be known with certainty.
  • Seek a theory to explain the results they find in the answers to their questions. While the “data-driven educator” may be satisfied with knowing “what worked,” the educator which uses data as a use-based researcher will seek to elucidate reasons and mechanisms, a theory, for “what worked.” This will leave them better prepared to developed and refine interventions for other settings and cohorts of students. This theory will also allow them to predict other observations that will confirm their theory.
  • Based on their predictions, seek other evidence to support their theories. This evidence cannot be the same measurements. If, for example, we accept the dubious conclusion that SBAC (or PARC tests) measure college and career readiness, then we should be able to devise other measures of career and college readiness and the instruction that affects those tests scores should be observed in other ways as well.
  • Use-based research will also cause educators to become more critical of the measures they use (including those they are mandated to use) and to better understand the reality that we must be active consumers and evaluators of the data that is collected about our students and the methods used to analyze it.

Reference

Stokes, D. E. (1997). Pasteur’s quadrant: basic science and technological innovation. Washington, D.C: Brookings Institution Press.

Download this post as a PDF file:

Being Data Driven is Nothing to Brag About