Hot Tea Class Methodology: Statistical Methodology for Eliminating Bias and Error

Hot Tea Class categories include Signature Famous Teas in the U.S. as well as general categories that enable lesser-known teas to be highlighted and distinguished in the marketplace.

Each submission into the class is evaluated blind and through organoleptic analysis of the following characteristics: dry leaf, brewed color, brewed aroma, brewed flavor, brewed mouth-feel and brewed harmony. An overall numerical value on a 100-point scale is then calculated based on the ratings of each characteristic above. First, second and third place winning teas are determined by rank. 

For each tea judged, an average score for that tea is calculated. Also for each tea judged, there is a “standard deviation” calculated for its average score. One way to think of standard deviation is as setting a range between which scores may be predicted to fall. The statistic can identify a range for each tea score into which 95% of all scores are expected. (For the statistically-minded, that will be explained in greater detail below.) NATC then checks each individual judge’s score to assure that it falls within this prediction range. If any judge’s scores fall outside of the 95% expectation range for a particular tea, whether unexpectedly high or low, it is eliminated.

Any such extreme scores are termed in the statistics world as outliers, and the 95% certainty range is similar to that used in social science research. It is a generally accepted cut-off point in research. The size of the range for each tea is variable – if all judges but one rate a tea within a few points of each other, the range size is small. If there is some disparity in scores assigned to a tea, then the range for that tea can be quite wide. The beauty of such statistical procedure is that it takes into account the overall variability between judges when establishing the predicted range, while identifying any individual score which falls considerably outside the norm.

These procedures are applied only to each judge’s total score, not to each of the sub-components (such as color, aroma, etc.) that make up the final score. By applying this criterion to each score submitted, any concern about favoritism toward a tea is addressed; but equally important any strong bias against a specific tea which falls outside of the 95% range of predictability (as established by the calculations done with all judges’ scores for a tea) is also identifi ed.