We hear lots of folks talking about how data-driven they are today, but it seems these folks are often woefully unaware of the fundamental principles of data collection, analysis and presentation. Specific principles I see missing in “data-driven” folks are:
- Understanding of the roll of calibration. Whenever measuring (collecting data), one checks the instrument is accurate by using it to measure known quantity. This is easy when measuring physical like calibrating a thermometer, but it is much more difficult when meaning how “smart” one is.
- Recognizing biases. All measurements are prone to bias and there are many. The most obvious is confirmation bias, in which one’s data aligns with what one expected to see, but many other types of bias influence out data collection, analysis, and presentation.
- Recognizing argumentation. All data is collected, analyzed, and presented in an attempt to convince the audience to agree with a conclusion. Critical consumption of data finds on questioning assumptions (including the unstated assumptions), questioning methods, and confirming conclusions based on independent analysis of the data and reasonableness of the argument,
- Tendency to hide dubious data. If one is unwilling to identify the source of their data and the details of their analysis, then a responsible audience member will assume the data are in fact not supportive of the conclusion.
- Nonexistence of authority. When making decisions based on data, it is the data and its quality that is arbiter of reasonableness. When one seeks to use authority as a proxy for data, assume the data do not support the conclusion.
- Questioning missing data. While boundaries must be set when collecting data, the decisions about where to set those bound and what to measure within the bonds are to be questioned as well.
If we are to claim to be data-driven in our decisions, then we are responsible to those affected by our decisions to be clear and critical in our use of data.