
Full text loading...
Category: Clinical Microbiology; Bacterial Pathogenesis
Molecular Method Verification, Page 1 of 2
< Previous page | Next page > /docserver/preview/fulltext/10.1128/9781555816834/9781555814977_Chap55-1.gif /docserver/preview/fulltext/10.1128/9781555816834/9781555814977_Chap55-2.gifAbstract:
This chapter talks about aspects of molecular method verification as well as the common statistical analyses that are used during this process. Stevenson et al. showed how polymorphisms in the probe target can affect the performance of an herpes simplex virus (HSV) real-time PCR assay. Such polymorphisms can result in decreased sensitivity of an assay, which is critical for a cerebrospinal fluid (CSF) specimen. In this study using the crossing threshold (CT) as a cutoff for assay sensitivity, it was estimated that as many as 15% of the HSV type 1 (HSV-1)-positive and 7% of the HSV-2-positive specimens would be missed when comparing two real-time HSV PCRs based on commercially available analyte specific reagents (ASRs). The author's clinical laboratory research is, in essence, translational research, in which their purpose is to describe, explain, predict outcomes, and control their systems. The chapter focuses on those aspects of biostatistics that are relevant to the verification and validation of new laboratory assays. Clinical Laboratory Improvement Act (CLIA) has clear guidelines on the verification and validation of laboratory-developed tests (LDTs) and other non-FDA-approved/ cleared assays. Choices of statistical methods and methods for drawing conclusions on the accuracy are less defined. Analysis of method verification data may require collaboration with a statistician or use of software specifically designed for CLIA method verification such as EP Evaluator.
Full text loading...
Depiction of a PCR efficiency calculation and graphical representation of PCR efficiency. Dose-response relationship for CT values and microbe density depicts a linear and inversely proportional relationship at high microbe concentrations. Linear regression shows 99% correlation with linearity to the highest dilutions tested; however, a visual check of CT values at 100 and 50 organisms per reaction depicts a breakdown in the precision of the data points at that concentration and may predict the microbial density at which assay linearity may begin to decline.
(A) Example of a ROC plot for an imaginary diagnostic test. The true-positive rate (sensitivity) is plotted against the false-positive rate (1—specificity) for every known threshold value. The dotted diagonal line shows the case when the test has no discriminatory power. A test that gives no false-positive or false-negative results has a curve that passes through the top left-hand corner (dotted line), i.e., the distributions of the test results in the two groups are completely separated, and the area under the concentration-time curve equals 1.0. (The test is perfect.) The double-headed arrow shows the effect of varying the threshold value. (B) Comparisons of ROC plots for different test methods; the top line has the highest discriminatory power.
Illustration of a 2 × 2 table to diagram terms commonly used to describe assay performance during diagnostic method assessment. Fill in the boxes according to agreement status and calculate the column totals (down, up) or row totals (right, left), as indicated in Fig. 2 B, to be used to calculate sensitivity, specificity, PPV, and NPV. Sensitivity = TP/(TP + FN) or a/ (a + c) × 100. Specificity = TN(FP + TN) or d/(d + b) × 100. PPV = TP/(TP + FP) or a/(a + b) × 100. NPV = TN/(TN + FP) or d/(d + c) × 100.
(Top) Example of a medical decision interval defined by biologic and assay variability. Per CLSI MM06-A, the medical decision interval for HIV-1 quantitative assay is 0.5 log or threefold. Therefore, if the biological variability is already known to be 0.3 log, as in the case of HIV-1, then in order to meet the medical decision requirements for HIV treatment protocols, the assay precision must possess an SD of ≤0.2 log across the quantitative dynamic range. If this is true, the %CV will vary across the range, as in this example, which shows a 12% CV at 50 copies and a 4% CV at 50,000 copies. (Bottom) Example of establishment of quantitative control limits by determining ranges based on assay tolerance limits, which are 0.2 log10 SD for a hepatitis B virus midrange control (n = 28 replicates). Note: for numerical values <10, two decimals are displayed; for values >10, no significant figures are displayed.
Example of calibration curves showing a standard curve with increased variability at the low end of microbial density (expected versus actual IU/milliliter, purchased standards [0–6.7 log IU/ml], real-time PCR, hepatitis C virus). The Poisson distribution characterizes the behavior of analyte and precision at the LLOD.
Example of Bland-Altman plot analysis to characterize the log difference between two assays with relationship to different genotypes.
Flow diagram depicting categories of results and options from a qualitative method verification. Describe data with descriptive statistics and 2×2 tables, and then analyze data with inferential statistics.
Graphing the distribution of data. One can use histograms or curves to display data distribution in the form of frequency distributions, which are important to assess whether or not the data have a normal distribution, as depicted in the top diagram. If the data do not have a normal distribution (e.g., bottom diagram), one cannot use parametric statistics, like the t test, to compare the means; one must use nonparametric statistics. One can use box and whisker plots and quantiles (described in Fig. 9 ) to help assess the midpoint of the data.
Descriptive midpoint measurements, depicted in a box and whiskers plot, displaying the major quantiles. Outliers have undue influence on the mean; therefore, the mean may be misleading, depending on the magnitude of the variation from the median. This image depicts an example of a box and whiskers plot created from the distribution of an internal positive control cycle threshold value (IPC Ct ) produced when a real-time PCR assay was performed in different sample matrices. Quantiles are values at which a specific percentage of results fall below the value of the quantile in the sample set.
The empirical rule. For data sets having a normal bell-shaped distribution, the following properties apply: about 68% of all values fall within 1 SD of the mean; about 95% of all values fall within 2 SD of the mean; and about 99.7% of all values fall within 3 SD of the mean.
Systematic error (bias) versus random error. A calculation can correct for a bias, but imprecision cannot be corrected by calculation. (Top left) Good precision with poor accuracy gives a biased result (the average is off center). (Top right) Poor precision and poor accuracy give a biased result (the average is off center). (Bottom left) Good precision and good accuracy give an unbiased result (the average is on center). (Bottom right) Poor precision with good accuracy gives an unbiased result (the average is on center).
A one-tailed t test in the positive direction. A one-tailed test determines whether a particular population parameter is larger than some predefined value. The t test uses one critical “t” value, defined from a table. If the sample mean falls outside the accepted area (95%), then the null is rejected (the area shaded in gray is the area for rejection of the null hypothesis).
(A) An x-y graph with regression analysis: in the simplest case, there are two variables, one explanatory (x) and one response variable (y), i.e., a change in x causes a change in y. It is always worth viewing your data (if possible) before performing regressions to get an idea as to the type of relationship (e.g., whether it is best described by a straight line or a curve). By looking at this scatter plot, it can be seen that variables x and y have a close relationship that may be reasonably represented by a straight line. This would be represented mathematically as y = a + bx + e, where a describes where the line crosses the y axis, b describes the slope of the line, and e is an error term that describes the variation of the real data above and below the line. Simple linear regression attempts to find a straight line that best fits the data, where the variation of the real data above and below the line is minimized. (B) Enlarged section of panel A, with residuals and fitted values shown. Assuming that variation in y is explained by variation of x, we can begin our regression. This output will tell us several things: the equation of the fitted line, formal information regarding the association of the variables, and how well the fitted line describes the data. There are two values: R 2 (the coefficient of determination), and r (the correlation coefficient). R 2 indicates the proportion of variance in one variable that can be explained by the variance in the other variable. If R 2 = 0.98, 98% of the change in y can be predicted by the changes in x. For the parameter, r is a measure of linear association (the degree of the relationship) for data with a normal distribution. For perfect positive linear correlations, r = 1.0; for a perfect negative linear correlation, r = –1.0 and the line slopes from the top right of the graph to the bottom left. Since not all relationships are linear, one must examine the data to assess if a straight line is the best fit for the data; in most laboratory evaluations, this will be the case. (C) The regression equation and its graphical depiction of the line of equality and the y intercept. The y intercept represents systematic bias of x to y quantitative comparison.
Test selection summary
Requirements for verification of molecular microbiology assays a
Overview of association methods