1887

Chapter 50 : Molecular Method Verification

MyBook is a cheap paperback edition of the original book and will be sold at uniform, low price.

Ebook: Choose a downloadable PDF or ePub file. Chapter is a downloadable PDF file. File must be downloaded within 48 hours of purchase

Buy this Chapter
Digital (?) $30.00

Preview this chapter:
Zoom in
Zoomout

Molecular Method Verification, Page 1 of 2

| /docserver/preview/fulltext/10.1128/9781555819071/9781555819088.ch50-1.gif /docserver/preview/fulltext/10.1128/9781555819071/9781555819088.ch50-2.gif

Abstract:

This chapter is intended to introduce the basic principles of method verification to laboratorians who have not yet been exposed to statistical concepts for method verification and provide a refresher for those who already use these analytic methods and concepts.

Citation: Wolk D, Marlowe E. 2016. Molecular Method Verification, p 721-744. In Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed), Molecular Microbiology. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50
Highlighted Text: Show | Hide
Loading full text...

Full text loading...

Figures

Image of FIGURE 1
FIGURE 1

Depiction of a PCR efficiency calculation and graphical representation of PCR efficiency. Dose-response relationship for values and microbe density depicts a linear and inversely proportional relationship at high microbe concentrations. Linear regression shows 99% correlation with linearity to the highest dilutions tested; however, a visual check of values at 100 and 50 organisms per reaction depicts a breakdown in the precision of the data points at that concentration and may predict the microbial density at which assay linearity may begin to decline.

Citation: Wolk D, Marlowe E. 2016. Molecular Method Verification, p 721-744. In Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed), Molecular Microbiology. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50
Permissions and Reprints Request Permissions
Download as Powerpoint
Image of FIGURE 2
FIGURE 2

(A) Example of an ROC plot for an imaginary diagnostic test. The true-positive rate (sensitivity) is plotted against the false-positive rate (1 – specificity) for every known threshold value. The dotted diagonal line shows the case when the test has no discriminatory power. A test that gives no false-positive or false-negative results has a curve that passes through the top left-hand corner (dotted line), i.e., the distributions of the test results in the two groups are completely separated, and the area under the concentration-time curve equals 1.0. (The test is perfect.) The double-headed arrow shows the effect of varying the threshold value. (B) Comparisons of ROC plots for different test methods; the top line has the highest discriminatory power.

Citation: Wolk D, Marlowe E. 2016. Molecular Method Verification, p 721-744. In Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed), Molecular Microbiology. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50
Permissions and Reprints Request Permissions
Download as Powerpoint
Image of FIGURE 3
FIGURE 3

Illustration of a 2 × 2 table to diagram terms commonly used to describe assay performance during diagnostic method assessment. Fill in the boxes according to agreement status and calculate the column totals (down, up) or row totals (right, left), as indicated in Fig. 2B , to be used to calculate sensitivity, specificity, PPV, and NPV. Sensitivity = TP/(TP + FN) or a/(a + c) × 100. Specificity = TN(FP + TN) or d/(d + b) × 100. PPV = TP/(TP + FP) or a/(a + b) × 100. NPV = TN/(TN + FP) or d/(d + c) × 100.

Citation: Wolk D, Marlowe E. 2016. Molecular Method Verification, p 721-744. In Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed), Molecular Microbiology. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50
Permissions and Reprints Request Permissions
Download as Powerpoint
Image of FIGURE 4
FIGURE 4

(A) Example of a medical decision interval defined by biologic and assay variability. Per CLSI MM06-A, the medical decision interval for HIV-1 quantitative assay is 0.5 log or 3-fold. Therefore, if the biological variability is already known to be 0.3 log, as in the case of HIV-1, then to meet the medical decision requirements for HIV treatment protocols, the assay precision must possess an SD of ≤0.2 log across the quantitative dynamic range. If this is true, the %CV will vary across the range, as in this example, which shows a 12% CV at 50 copies and a 4% CV at 50,000 copies. (B) Example of establishment of quantitative control limits by determining ranges based on assay tolerance limits, which are 0.2 log SD for a hepatitis B virus midrange control ( = 28 replicates). Note: for numerical values <10, two decimals are displayed; for values >10, no significant figures are displayed.

Citation: Wolk D, Marlowe E. 2016. Molecular Method Verification, p 721-744. In Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed), Molecular Microbiology. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50
Permissions and Reprints Request Permissions
Download as Powerpoint
Image of FIGURE 5
FIGURE 5

Example of calibration curves showing a standard curve with increased variability at the low end of microbial density (expected versus actual IU/milliliter, purchased standards [0–6.7 log IU/ml], real-time PCR, hepatitis C virus). The Poisson distribution characterizes the behavior of the analyte and precision at the LLOD.

Citation: Wolk D, Marlowe E. 2016. Molecular Method Verification, p 721-744. In Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed), Molecular Microbiology. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50
Permissions and Reprints Request Permissions
Download as Powerpoint
Image of FIGURE 13
FIGURE 13

A one-tailed test in the positive direction. A one-tailed test determines whether a particular population parameter is larger than some predefined value. The test uses one critical value, defined from a table. If the sample mean falls outside the accepted area (95%), then the null is rejected (the area shaded in gray is the area for rejection of the null hypothesis).

Citation: Wolk D, Marlowe E. 2016. Molecular Method Verification, p 721-744. In Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed), Molecular Microbiology. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50
Permissions and Reprints Request Permissions
Download as Powerpoint
Image of FIGURE 6
FIGURE 6

Example of Bland-Altman plot analysis to characterize the log difference between two assays with relationships to different genotypes.

Citation: Wolk D, Marlowe E. 2016. Molecular Method Verification, p 721-744. In Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed), Molecular Microbiology. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50
Permissions and Reprints Request Permissions
Download as Powerpoint
Image of FIGURE 7
FIGURE 7

Altman's nomogram for estimating sample size is based on predetermined selections for alpha, and the expected differences in proportions of attributes between new and old methods. It is a graphical method that can be found and described at the following website: http://www.statsref.com/HTML/index.html?sampling.html. To use the nomogram, mark the standardized difference on the left vertical scale and the statistical power you require on the right vertical scale, then draw a line between these points and read the sample size from the ladder section, depending on whether you chose an alpha of 0.05 or 0.10. In the figure, the point at which the blue line (0.6 proportional difference and 0.9 power) intersects the ladder-like sloping lines will provide the estimated sample size for significance levels 0.05 and 0.01, respectively. Reading from the scaled ladder, we can read = 120 and = 160 for values of 0.05 and 0.01, respectively. Thus, the total sample size required in each group you compare (e.g., new method and old method) could be 120/2 = 60 each for alpha 0.05.

Citation: Wolk D, Marlowe E. 2016. Molecular Method Verification, p 721-744. In Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed), Molecular Microbiology. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50
Permissions and Reprints Request Permissions
Download as Powerpoint
Image of FIGURE 8
FIGURE 8

Flow diagram depicting categories of results and options from a qualitative method verification. Describe data with descriptive statistics and 2 × 2 tables, and then analyze data with inferential statistics.

Citation: Wolk D, Marlowe E. 2016. Molecular Method Verification, p 721-744. In Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed), Molecular Microbiology. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50
Permissions and Reprints Request Permissions
Download as Powerpoint
Image of FIGURE 9
FIGURE 9

Graphing the distribution of data. One can use histograms or curves to display data distribution in the form of frequency distributions, which are important to assess whether or not the data have a normal distribution, as depicted in the top diagram. If the data do not have a normal distribution (e.g., bottom diagram), one cannot use parametric statistics, such as the test, to compare the means; one must use nonparametric statistics. One can use box-and-whisker plots and quantiles (described in Fig. 10 ) to help assess the midpoint of the data.

Citation: Wolk D, Marlowe E. 2016. Molecular Method Verification, p 721-744. In Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed), Molecular Microbiology. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50
Permissions and Reprints Request Permissions
Download as Powerpoint
Image of FIGURE 10
FIGURE 10

Descriptive midpoint measurements, depicted in a box-and-whiskers plot, displaying the major quantiles. Outliers have undue influence on the mean; therefore, the mean may be misleading, depending on the magnitude of the variation from the median.

Citation: Wolk D, Marlowe E. 2016. Molecular Method Verification, p 721-744. In Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed), Molecular Microbiology. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50
Permissions and Reprints Request Permissions
Download as Powerpoint
Image of FIGURE 11
FIGURE 11

The empirical rule. For data sets with a normal bell-shaped distribution, the following properties apply: about 68% of all values fall within 1 SD of the mean; about 95% of all values fall within 2 SD of the mean; and about 99.7% of all values fall within 3 SD of the mean.

Citation: Wolk D, Marlowe E. 2016. Molecular Method Verification, p 721-744. In Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed), Molecular Microbiology. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50
Permissions and Reprints Request Permissions
Download as Powerpoint
Image of FIGURE 12
FIGURE 12

Systematic error (bias) versus random error. A calculation can correct for a bias, but imprecision cannot be corrected by calculation. (Top left) Good precision with poor accuracy gives a biased result (the average is off center). (Top right) Poor precision and poor accuracy give a biased result (the average is off center). (Bottom left) Good precision and good accuracy give an unbiased result (the average is on center). (Bottom right) Poor precision with good accuracy gives an unbiased result (the average is on center).

Citation: Wolk D, Marlowe E. 2016. Molecular Method Verification, p 721-744. In Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed), Molecular Microbiology. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50
Permissions and Reprints Request Permissions
Download as Powerpoint
Image of FIGURE 14
FIGURE 14

(A) An graph with regression analysis: in the simplest case, there are two variables, one explanatory () and one response variable (), i.e., a change in causes a change in . It is always worth viewing your data (if possible) before performing regressions to get an idea of the type of relationship (e.g., whether it is best described by a straight line or a curve). By looking at this scatter plot, it can be seen that variables and have a close relationship that may be reasonably represented by a straight line. This would be represented mathematically as = + + , where describes where the line crosses the axis, describes the slope of the line, and is an error term that describes the variation of the real data above and below the line. Simple linear regression attempts to find a straight line that best fits the data, where the variation of the real data above and below the line is minimized. (B) Enlarged section of panel A, with residuals and fitted values shown. Assuming that variation in is explained by variation of , we can begin our regression. This output will tell us several things: the equation of the fitted line, formal information regarding the association of the variables, and how well the fitted line describes the data. There are two values: (the coefficient of determination) and (the correlation coefficient). indicates the of variance in one variable that can be explained by the variance in the other variable. If = 0.98, 98% of the change in can be predicted by the changes in . For the parameter, is a measure of linear association (the degree of the relationship) for data with a normal distribution. For perfect positive linear correlations, = 1.0; for a perfect negative linear correlation, = −1.0 and the line slopes from the top right of the graph to the bottom left. Since not all relationships are linear, one must examine the data to assess if a straight line is the best fit for the data; in most laboratory evaluations, this will be the case. (C) The regression equation and its graphical depiction of the line of equality and the intercept. The intercept represents systematic bias of to quantitative comparison.

Citation: Wolk D, Marlowe E. 2016. Molecular Method Verification, p 721-744. In Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed), Molecular Microbiology. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50
Permissions and Reprints Request Permissions
Download as Powerpoint

References

/content/book/10.1128/9781555819071.ch50
1. CDC. 2015. CLIA Regulations http://wwwn.cdc.gov/clia/Regulatory/.
2. U.S. Department of Health and Human Services, Food and Drug Administration. 2007. Guidance for Industry and FDA Staff - Commercially Distributed Analyte Specific Reagents (ASRs): Frequently Asked Questions. Food and Drug Administration, U.S. Department of Health and Human Services, Silver Spring, MD. http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm078423.htm
3. Jennings L, Van Deerlin VM, Gulley ML, College of American Pathologists Molecular Pathology Resource Committee. 2009. Recommended principles and practices for validating clinical molecular pathology tests. Arch Pathol Lab Med 133:743755.[PubMed]
4. Clark RB, Lewinski MA, Loeffelholz MJ, Tibbetts RJ,. 2009. Cumitech 31A: Method Verification and Validation Procedures in the Clinical Microbiology Laboratory. Coordinating ed, Sharp SE. ASM Press, Washington, DC.
5. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, Lijmer JG, Moher D, Rennie D, de Vet HC, Standards for Reporting of Diagnostic Accuracy. 2003. Toward complete and accurate reporting of studies of diagnostic accuracy. The STARD initiative. Am J Clin Pathol 119:1822.
6. Bossuyt PM, Reitsma JB, Standards for Reporting of Diagnostic Accuracy. 2003. The STARD initiative. Lancet 361:71.
7. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, Moher D, Rennie D, de Vet HC, Lijmer JG, Standards for Reporting of Diagnostic Accuracy. 2003. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Ann Intern Med 138:W1W12.
8. McQueen M, The STARD Initiative. 2003. Evidence-based laboratory medicine: addressing bias, generalisability and applicability in studies on diagnostic accuracy. The STARD initiative. Clin Chem Lab Med 41:1[CrossRef].[PubMed]
9. Birdsong GG. 2007. Testing the test: remediation is needed. Cancer 111:461464[CrossRef].[PubMed]
10. Dewitte K, Fierens C, Stöckl D, Thienpont LM. 2002. Application of the Bland-Altman plot for interpretation of method-comparison studies: a critical investigation of its practice. Clin Chem 48:799801, author reply 801–802.[PubMed]
11. Green TA, Black CA, Johnson RE. 2001. In defense of discrepant analysis. J Clin Epidemiol 54:210215[CrossRef].[PubMed]
12. Hadgu A. 1999. Discrepant analysis: a biased and an unscientific method for estimating test sensitivity and specificity. J Clin Epidemiol 52:12311237[CrossRef].[PubMed]
13. Ihorst G, Forster J, Petersen G, Werchau H, Rohwedder A, Schumacher M. 2007. The use of imperfect diagnostic tests had an impact on prevalence estimation. J Clin Epidemiol 60:902910[CrossRef].[PubMed]
14. Laposata M, Dighe A. 2007. “Pre-pre” and “post-post” analytical error: high-incidence patient safety hazards involving the clinical laboratory. Clin Chem Lab Med 45:712719[CrossRef].[PubMed]
15. Leeflang MM, Moons KG, Reitsma JB, Zwinderman AH. 2008. Bias in sensitivity and specificity caused by data-driven selection of optimal cutoff values: mechanisms, magnitude, and solutions. Clin Chem 54:729737[CrossRef].[PubMed]
16. McAdam AJ. 2000. Discrepant analysis: how can we test a test? J Clin Microbiol 38:20272029.[PubMed]
17. Sternberg M, McAdam AJ. 2001. Discrepant analysis is still at large. J Clin Microbiol 39:826827[CrossRef].[PubMed]
18. Westgard JO, Darcy T. 2004. The truth about quality: medical usefulness and analytical reliability of laboratory tests. Clin Chim Acta 346:311[CrossRef].[PubMed]
19. Westgard JO, Westgard SA. 2006. The quality of laboratory testing today: an assessment of sigma metrics for analytic quality using performance data from proficiency testing surveys and the CLIA criteria for acceptable performance. Am J Clin Pathol 125:343354[CrossRef].[PubMed]
20. Stevenson J, Hymas W, Hillyard D. 2005. Effect of sequence polymorphisms on performance of two real-time PCR assays for detection of herpes simplex virus. J Clin Microbiol 43:23912398[CrossRef].[PubMed]
21. Issa NC, Espy MJ, Uhl JR, Smith TF. 2005. Sequencing and resolution of amplified herpes simplex virus DNA with intermediate melting curves as genotype 1 or 2 by LightCycler PCR assay. J Clin Microbiol 43:18431845[CrossRef].[PubMed]
22. Wolff DJ, Heaney DL, Neuwald PD, Stellrecht KA, Press RD. 2009. Multi-Site PCR-based CMV viral load assessment-assays demonstrate linearity and precision, but lack numeric standardization: a report of the association for molecular pathology. J Mol Diagn 11:8792[CrossRef].[PubMed]
23. Clinical Laboratory Standards Institute. 2010. Quantitative Molecular Methods for Infectious Diseases; Approved Guideline—MM06-A2. CLSI 23(28).
24. Bland JM, Altman DG. 1986. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1:307310[CrossRef].[PubMed]
25. Barbeau JM, Goforth J, Caliendo AM, Nolte FS. 2004. Performance characteristics of a quantitative TaqMan hepatitis C virus RNA analyte-specific reagent. J Clin Microbiol 42:37393746[CrossRef].[PubMed]
26. Germer JJ, Qutub MO, Mandrekar JN, Mitchell PS, Yao JD. 2006. Quantification of hepatitis B virus (HBV) DNA with a TaqMan HBV analyte-specific reagent following sample processing with the MagNA pure LC instrument. J Clin Microbiol 44:14901494[CrossRef].[PubMed]
27. Sarrazin C, Gärtner BC, Sizmann D, Babiel R, Mihm U, Hofmann WP, von Wagner M, Zeuzem S. 2006. Comparison of conventional PCR with real-time PCR and branched DNA-based assays for hepatitis C virus RNA quantification and clinical significance for genotypes 1 to 5. J Clin Microbiol 44:729737[CrossRef].[PubMed]
28. Wolk DM, Hayden RT,. 2011. Quantitative molecular methods, p 83106. In Persing DH, Tenover C, Tang Y-W, Nolte FS, Hayden RT, van Belkum A (ed), Molecular Microbiology: Diagnostic Principles and Practice. ASM Press, Washington, DC.[CrossRef]
29. Liang SL, Lin MT, Hafez MJ, Gocke CD, Murphy KM, Sokoll LJ, Eshleman JR. 2008. Application of traditional clinical pathology quality control techniques to molecular pathology. J Mol Diagn 10:142146[CrossRef].[PubMed]
30. Westgard JO, Barry PL, Hunt MR, Groth T. 1981. A multi-rule Shewhart chart for quality control in clinical chemistry. Clin Chem 27:493501.[PubMed]
31. Glaser AN. 2001. High-Yield Biostatistics. Lippincott Williams, Philadelphia, PA.
32. Pagano M, Gauvreau K. 2000. Principles of Biostatistics. Duxbuty, Pacific Grove, CA.
33. Bachmann LM, Ter RG, Weber WE, Kessels AG. 2009. Multivariable adjustments counteract spectrum and test review bias in accuracy studies. J Clin Epidemiol 62:357361.
34. Rhoads DG. 2008. Lab Statistics Fun and Easy: A Practical Approach to Method Evaluation, 3rd ed, David G. Rhoads Associates, Kennett Square, PA.
35. Rhoads DG. 2008. EP Evaluator, Release 8 Users Manual. David G. Rhoads Associates, Kennett Square, PA.

Tables

Generic image for table
TABLE 1

Test selection summary: important CLSI protocols for test verification

Citation: Wolk D, Marlowe E. 2016. Molecular Method Verification, p 721-744. In Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed), Molecular Microbiology. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50
Generic image for table
TABLE 2

Requirements for verification of molecular microbiology assays

Citation: Wolk D, Marlowe E. 2016. Molecular Method Verification, p 721-744. In Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed), Molecular Microbiology. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50
Generic image for table
TABLE 3

Overview of association methods

Citation: Wolk D, Marlowe E. 2016. Molecular Method Verification, p 721-744. In Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed), Molecular Microbiology. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50

This is a required field
Please enter a valid email address
Please check the format of the address you have entered.
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error