ADS-79D-HDBK Downloaded from https://www.everyspec.com going ―on-condition.‖ As more data is acquired on damaged components to add to the distributions of the damaged and undamaged data sets, this number will change. The example described provides a method to determine the minimum sample size required to evaluate the response of a CI to a specific fault. However, the example does not indicate if the CI responds with high confidence and reliability. To determine if the CI meets the 90% reliability metrics per equations 1 and 2, additional steps are required. The first step is to define an optimum threshold. This allows one to separate the data into the two conditions, faulted and unfaulted.
A curve, referred to as Receiver Operating Characteristic (ROC), can be used for defining this optimum threshold. 61 ROC curves are plots of the false alarm rate (probability of false alarm or false positive rate) on the horizontal axis ( x ) versus the hit rate (probability of detection-true alarm or true positive rate) on the vertical axis ( y ), providing a visual comparison of thresholds on a common scale. Since this is for demonstration purposed only, the data from helicopters plotted on the following figures will be used: I-2, I-3, I-4, I-5, I- 6, and I-8. The ROC curve for this dataset is shown in Figure I-15.
Figure I-15 also provides a visual illustration of how the ROC plot is obtained. The two distributions represent a no damage response and a damage response of a CI. The threshold line separates the graph into correct indication/TN-true negative (no damage—no indication), FN-false negative (damage present— no indication), false alarms/FP-false positive (no damage—indicated) and hits/TP-true positive (damage—indicated). The probability of detection would equal the area under the damage distribution curve to the right of the threshold line. The false alarm rate would equal the area of the no damage distribution to the right of the threshold line.
The optimum threshold for this CI, component and fault is 1.7. To determine the reliability of this system, the number of CI readings in the damaged dataset that exceed 1.7 is divided by the total readings. This determined the probability of detection equal to 92%. The false alarm rate was calculated by determining the number of readings in the undamaged dataset when the CI value exceeded 1.7. This determined the probability of correctly classifying an undamaged component as 88% or a false alarm rate of 12%. This CI does not meet the minimum demonstrated reliability of 90% for correctly classifying an undamaged component.
Note the number of samples required to assess the performance of the CI must also meet confidence and reliability metrics. I.4.3 Sample Size Method 3 I.4.3.1 Weibull Analysis 62 The sample size methodology in this section evaluates the distribution of CI data to see if the data may be incorporated into a Weibull distribution. For a CBM-monitored component, if the distribution of current CI values from all items is Weibull, then a method exists to establish 61 Dempsey, Paula, J.; Jonathon, A. Keller, Daniel R Wade, Signal Detection Theory Applied to Helicopter Transmission Diagnostic Thresholds , NASA/TM-2008-215262; AMRDEC PAO Control Number FN 3597, July 2008.
62 Abernethy, Robert B., The New Weibull Handbook – Reliability & Statistical Analysis for Predicting Life, Safety, Survivability, Risk, Cost, and Warranty Claims, Fourth Edition, 2000. 165