Reliability Issues in Performance Analysis

There are beliefs that have come to be accepted by many in performance analysis. This paper challenges some of these beliefs. The presence of precise operational definitions does not guarantee good reliability nor does their absence guarantee poor reliability. Intra-operator reliability studies cannot show a system to be objective. Some reliability statistics give values considered to reflect good reliability even when observations are not even looking at the same match! The value of a reliability statistic considered to be acceptable needs to be justified. Limited reliability can introduce variability into data that reduces the chance of finding a significant difference. Reliability is at least as important when performance analysis is used in coaching and judging contexts as when it is used for academic research. There is a trade off between reliability and the precision of measurement.