Full text loading...

# Chapter 50 : Molecular Method Verification

^{1}, Elizabeth M. Marlowe

^{2}

Category: Clinical Microbiology

Molecular Method Verification, Page 1 of 2

< Previous page | Next page > /docserver/preview/fulltext/10.1128/9781555819071/9781555819088.ch50-1.gif /docserver/preview/fulltext/10.1128/9781555819071/9781555819088.ch50-2.gif**Abstract:**

This chapter is intended to introduce the basic principles of method verification to laboratorians who have not yet been exposed to statistical concepts for method verification and provide a refresher for those who already use these analytic methods and concepts.

**Citation:**

**Wolk D, Marlowe E.**2016. Molecular Method Verification, p 721-744.

*In*Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed),

*Molecular Microbiology*. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50

## Figures

**Citation:**

**Wolk D, Marlowe E.**2016. Molecular Method Verification, p 721-744.

*In*Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed),

*Molecular Microbiology*. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50

Depiction of a PCR efficiency calculation and graphical representation of PCR efficiency. Dose-response relationship for CT values and microbe density depicts a linear and inversely proportional relationship at high microbe concentrations. Linear regression shows 99% correlation with linearity to the highest dilutions tested; however, a visual check of CT values at 100 and 50 organisms per reaction depicts a breakdown in the precision of the data points at that concentration and may predict the microbial density at which assay linearity may begin to decline.

##### FIGURE 1

Depiction of a PCR efficiency calculation and graphical representation of PCR efficiency. Dose-response relationship for CT values and microbe density depicts a linear and inversely proportional relationship at high microbe concentrations. Linear regression shows 99% correlation with linearity to the highest dilutions tested; however, a visual check of CT values at 100 and 50 organisms per reaction depicts a breakdown in the precision of the data points at that concentration and may predict the microbial density at which assay linearity may begin to decline.

**Citation:**

**Wolk D, Marlowe E.**2016. Molecular Method Verification, p 721-744.

*In*Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed),

*Molecular Microbiology*. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50

(A) Example of an ROC plot for an imaginary diagnostic test. The true-positive rate (sensitivity) is plotted against the false-positive rate (1 – specificity) for every known threshold value. The dotted diagonal line shows the case when the test has no discriminatory power. A test that gives no false-positive or false-negative results has a curve that passes through the top left-hand corner (dotted line), i.e., the distributions of the test results in the two groups are completely separated, and the area under the concentration-time curve equals 1.0. (The test is perfect.) The double-headed arrow shows the effect of varying the threshold value. (B) Comparisons of ROC plots for different test methods; the top line has the highest discriminatory power.

##### FIGURE 2

(A) Example of an ROC plot for an imaginary diagnostic test. The true-positive rate (sensitivity) is plotted against the false-positive rate (1 – specificity) for every known threshold value. The dotted diagonal line shows the case when the test has no discriminatory power. A test that gives no false-positive or false-negative results has a curve that passes through the top left-hand corner (dotted line), i.e., the distributions of the test results in the two groups are completely separated, and the area under the concentration-time curve equals 1.0. (The test is perfect.) The double-headed arrow shows the effect of varying the threshold value. (B) Comparisons of ROC plots for different test methods; the top line has the highest discriminatory power.

**Citation:**

**Wolk D, Marlowe E.**2016. Molecular Method Verification, p 721-744.

*In*Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed),

*Molecular Microbiology*. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50

Illustration of a 2 × 2 table to diagram terms commonly used to describe assay performance during diagnostic method assessment. Fill in the boxes according to agreement status and calculate the column totals (down, up) or row totals (right, left), as indicated in Fig. 2B , to be used to calculate sensitivity, specificity, PPV, and NPV. Sensitivity = TP/(TP + FN) or a/(a + c) × 100. Specificity = TN(FP + TN) or d/(d + b) × 100. PPV = TP/(TP + FP) or a/(a + b) × 100. NPV = TN/(TN + FP) or d/(d + c) × 100.

##### FIGURE 3

Illustration of a 2 × 2 table to diagram terms commonly used to describe assay performance during diagnostic method assessment. Fill in the boxes according to agreement status and calculate the column totals (down, up) or row totals (right, left), as indicated in Fig. 2B , to be used to calculate sensitivity, specificity, PPV, and NPV. Sensitivity = TP/(TP + FN) or a/(a + c) × 100. Specificity = TN(FP + TN) or d/(d + b) × 100. PPV = TP/(TP + FP) or a/(a + b) × 100. NPV = TN/(TN + FP) or d/(d + c) × 100.

**Citation:**

**Wolk D, Marlowe E.**2016. Molecular Method Verification, p 721-744.

*In*Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed),

*Molecular Microbiology*. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50

(A) Example of a medical decision interval defined by biologic and assay variability. Per CLSI MM06-A, the medical decision interval for HIV-1 quantitative assay is 0.5 log or 3-fold. Therefore, if the biological variability is already known to be 0.3 log, as in the case of HIV-1, then to meet the medical decision requirements for HIV treatment protocols, the assay precision must possess an SD of ≤0.2 log across the quantitative dynamic range. If this is true, the %CV will vary across the range, as in this example, which shows a 12% CV at 50 copies and a 4% CV at 50,000 copies. (B) Example of establishment of quantitative control limits by determining ranges based on assay tolerance limits, which are 0.2 log10 SD for a hepatitis B virus midrange control (n = 28 replicates). Note: for numerical values <10, two decimals are displayed; for values >10, no significant figures are displayed.

##### FIGURE 4

(A) Example of a medical decision interval defined by biologic and assay variability. Per CLSI MM06-A, the medical decision interval for HIV-1 quantitative assay is 0.5 log or 3-fold. Therefore, if the biological variability is already known to be 0.3 log, as in the case of HIV-1, then to meet the medical decision requirements for HIV treatment protocols, the assay precision must possess an SD of ≤0.2 log across the quantitative dynamic range. If this is true, the %CV will vary across the range, as in this example, which shows a 12% CV at 50 copies and a 4% CV at 50,000 copies. (B) Example of establishment of quantitative control limits by determining ranges based on assay tolerance limits, which are 0.2 log10 SD for a hepatitis B virus midrange control (n = 28 replicates). Note: for numerical values <10, two decimals are displayed; for values >10, no significant figures are displayed.

**Citation:**

**Wolk D, Marlowe E.**2016. Molecular Method Verification, p 721-744.

*In*Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed),

*Molecular Microbiology*. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50

Example of calibration curves showing a standard curve with increased variability at the low end of microbial density (expected versus actual IU/milliliter, purchased standards [0–6.7 log IU/ml], real-time PCR, hepatitis C virus). The Poisson distribution characterizes the behavior of the analyte and precision at the LLOD.

##### FIGURE 5

Example of calibration curves showing a standard curve with increased variability at the low end of microbial density (expected versus actual IU/milliliter, purchased standards [0–6.7 log IU/ml], real-time PCR, hepatitis C virus). The Poisson distribution characterizes the behavior of the analyte and precision at the LLOD.

**Citation:**

**Wolk D, Marlowe E.**2016. Molecular Method Verification, p 721-744.

*In*Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed),

*Molecular Microbiology*. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50

A one-tailed t test in the positive direction. A one-tailed test determines whether a particular population parameter is larger than some predefined value. The t test uses one critical t value, defined from a table. If the sample mean falls outside the accepted area (95%), then the null is rejected (the area shaded in gray is the area for rejection of the null hypothesis).

##### FIGURE 13

A one-tailed t test in the positive direction. A one-tailed test determines whether a particular population parameter is larger than some predefined value. The t test uses one critical t value, defined from a table. If the sample mean falls outside the accepted area (95%), then the null is rejected (the area shaded in gray is the area for rejection of the null hypothesis).

**Citation:**

**Wolk D, Marlowe E.**2016. Molecular Method Verification, p 721-744.

*In*Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed),

*Molecular Microbiology*. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50

Example of Bland-Altman plot analysis to characterize the log difference between two assays with relationships to different genotypes.

##### FIGURE 6

Example of Bland-Altman plot analysis to characterize the log difference between two assays with relationships to different genotypes.

**Citation:**

**Wolk D, Marlowe E.**2016. Molecular Method Verification, p 721-744.

*In*Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed),

*Molecular Microbiology*. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50

Altman's nomogram for estimating sample size is based on predetermined selections for alpha, and the expected differences in proportions of attributes between new and old methods. It is a graphical method that can be found and described at the following website: http://www.statsref.com/HTML/index.html?sampling.html. To use the nomogram, mark the standardized difference on the left vertical scale and the statistical power you require on the right vertical scale, then draw a line between these points and read the sample size from the ladder section, depending on whether you chose an alpha of 0.05 or 0.10. In the figure, the point at which the blue line (0.6 proportional difference and 0.9 power) intersects the ladder-like sloping lines will provide the estimated sample size for significance levels 0.05 and 0.01, respectively. Reading from the scaled ladder, we can read N = 120 and N = 160 for P values of 0.05 and 0.01, respectively. Thus, the total sample size required in each group you compare (e.g., new method and old method) could be 120/2 = 60 each for alpha 0.05.

##### FIGURE 7

Altman's nomogram for estimating sample size is based on predetermined selections for alpha, and the expected differences in proportions of attributes between new and old methods. It is a graphical method that can be found and described at the following website: http://www.statsref.com/HTML/index.html?sampling.html. To use the nomogram, mark the standardized difference on the left vertical scale and the statistical power you require on the right vertical scale, then draw a line between these points and read the sample size from the ladder section, depending on whether you chose an alpha of 0.05 or 0.10. In the figure, the point at which the blue line (0.6 proportional difference and 0.9 power) intersects the ladder-like sloping lines will provide the estimated sample size for significance levels 0.05 and 0.01, respectively. Reading from the scaled ladder, we can read N = 120 and N = 160 for P values of 0.05 and 0.01, respectively. Thus, the total sample size required in each group you compare (e.g., new method and old method) could be 120/2 = 60 each for alpha 0.05.

**Citation:**

**Wolk D, Marlowe E.**2016. Molecular Method Verification, p 721-744.

*In*Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed),

*Molecular Microbiology*. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50

Flow diagram depicting categories of results and options from a qualitative method verification. Describe data with descriptive statistics and 2 × 2 tables, and then analyze data with inferential statistics.

##### FIGURE 8

Flow diagram depicting categories of results and options from a qualitative method verification. Describe data with descriptive statistics and 2 × 2 tables, and then analyze data with inferential statistics.

**Citation:**

**Wolk D, Marlowe E.**2016. Molecular Method Verification, p 721-744.

*In*Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed),

*Molecular Microbiology*. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50

Graphing the distribution of data. One can use histograms or curves to display data distribution in the form of frequency distributions, which are important to assess whether or not the data have a normal distribution, as depicted in the top diagram. If the data do not have a normal distribution (e.g., bottom diagram), one cannot use parametric statistics, such as the t test, to compare the means; one must use nonparametric statistics. One can use box-and-whisker plots and quantiles (described in Fig. 10 ) to help assess the midpoint of the data.

##### FIGURE 9

Graphing the distribution of data. One can use histograms or curves to display data distribution in the form of frequency distributions, which are important to assess whether or not the data have a normal distribution, as depicted in the top diagram. If the data do not have a normal distribution (e.g., bottom diagram), one cannot use parametric statistics, such as the t test, to compare the means; one must use nonparametric statistics. One can use box-and-whisker plots and quantiles (described in Fig. 10 ) to help assess the midpoint of the data.

**Citation:**

**Wolk D, Marlowe E.**2016. Molecular Method Verification, p 721-744.

*In*Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed),

*Molecular Microbiology*. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50

Descriptive midpoint measurements, depicted in a box-and-whiskers plot, displaying the major quantiles. Outliers have undue influence on the mean; therefore, the mean may be misleading, depending on the magnitude of the variation from the median.

##### FIGURE 10

Descriptive midpoint measurements, depicted in a box-and-whiskers plot, displaying the major quantiles. Outliers have undue influence on the mean; therefore, the mean may be misleading, depending on the magnitude of the variation from the median.

**Citation:**

**Wolk D, Marlowe E.**2016. Molecular Method Verification, p 721-744.

*In*Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed),

*Molecular Microbiology*. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50

The empirical rule. For data sets with a normal bell-shaped distribution, the following properties apply: about 68% of all values fall within 1 SD of the mean; about 95% of all values fall within 2 SD of the mean; and about 99.7% of all values fall within 3 SD of the mean.

##### FIGURE 11

The empirical rule. For data sets with a normal bell-shaped distribution, the following properties apply: about 68% of all values fall within 1 SD of the mean; about 95% of all values fall within 2 SD of the mean; and about 99.7% of all values fall within 3 SD of the mean.

**Citation:**

**Wolk D, Marlowe E.**2016. Molecular Method Verification, p 721-744.

*In*Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed),

*Molecular Microbiology*. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50

Systematic error (bias) versus random error. A calculation can correct for a bias, but imprecision cannot be corrected by calculation. (Top left) Good precision with poor accuracy gives a biased result (the average is off center). (Top right) Poor precision and poor accuracy give a biased result (the average is off center). (Bottom left) Good precision and good accuracy give an unbiased result (the average is on center). (Bottom right) Poor precision with good accuracy gives an unbiased result (the average is on center).

##### FIGURE 12

Systematic error (bias) versus random error. A calculation can correct for a bias, but imprecision cannot be corrected by calculation. (Top left) Good precision with poor accuracy gives a biased result (the average is off center). (Top right) Poor precision and poor accuracy give a biased result (the average is off center). (Bottom left) Good precision and good accuracy give an unbiased result (the average is on center). (Bottom right) Poor precision with good accuracy gives an unbiased result (the average is on center).

**Citation:**

**Wolk D, Marlowe E.**2016. Molecular Method Verification, p 721-744.

*In*Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed),

*Molecular Microbiology*. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50

(A) An x-y graph with regression analysis: in the simplest case, there are two variables, one explanatory (x) and one response variable (y), i.e., a change in x causes a change in y. It is always worth viewing your data (if possible) before performing regressions to get an idea of the type of relationship (e.g., whether it is best described by a straight line or a curve). By looking at this scatter plot, it can be seen that variables x and y have a close relationship that may be reasonably represented by a straight line. This would be represented mathematically as y = a + bx + e, where a describes where the line crosses the y axis, b describes the slope of the line, and e is an error term that describes the variation of the real data above and below the line. Simple linear regression attempts to find a straight line that best fits the data, where the variation of the real data above and below the line is minimized. (B) Enlarged section of panel A, with residuals and fitted values shown. Assuming that variation in y is explained by variation of x, we can begin our regression. This output will tell us several things: the equation of the fitted line, formal information regarding the association of the variables, and how well the fitted line describes the data. There are two values: R 2 (the coefficient of determination) and r (the correlation coefficient). R 2 indicates the proportion of variance in one variable that can be explained by the variance in the other variable. If R 2 = 0.98, 98% of the change in y can be predicted by the changes in x. For the parameter, r is a measure of linear association (the degree of the relationship) for data with a normal distribution. For perfect positive linear correlations, r = 1.0; for a perfect negative linear correlation, r = −1.0 and the line slopes from the top right of the graph to the bottom left. Since not all relationships are linear, one must examine the data to assess if a straight line is the best fit for the data; in most laboratory evaluations, this will be the case. (C) The regression equation and its graphical depiction of the line of equality and the y intercept. The y intercept represents systematic bias of x to y quantitative comparison.

##### FIGURE 14

(A) An x-y graph with regression analysis: in the simplest case, there are two variables, one explanatory (x) and one response variable (y), i.e., a change in x causes a change in y. It is always worth viewing your data (if possible) before performing regressions to get an idea of the type of relationship (e.g., whether it is best described by a straight line or a curve). By looking at this scatter plot, it can be seen that variables x and y have a close relationship that may be reasonably represented by a straight line. This would be represented mathematically as y = a + bx + e, where a describes where the line crosses the y axis, b describes the slope of the line, and e is an error term that describes the variation of the real data above and below the line. Simple linear regression attempts to find a straight line that best fits the data, where the variation of the real data above and below the line is minimized. (B) Enlarged section of panel A, with residuals and fitted values shown. Assuming that variation in y is explained by variation of x, we can begin our regression. This output will tell us several things: the equation of the fitted line, formal information regarding the association of the variables, and how well the fitted line describes the data. There are two values: R 2 (the coefficient of determination) and r (the correlation coefficient). R 2 indicates the proportion of variance in one variable that can be explained by the variance in the other variable. If R 2 = 0.98, 98% of the change in y can be predicted by the changes in x. For the parameter, r is a measure of linear association (the degree of the relationship) for data with a normal distribution. For perfect positive linear correlations, r = 1.0; for a perfect negative linear correlation, r = −1.0 and the line slopes from the top right of the graph to the bottom left. Since not all relationships are linear, one must examine the data to assess if a straight line is the best fit for the data; in most laboratory evaluations, this will be the case. (C) The regression equation and its graphical depiction of the line of equality and the y intercept. The y intercept represents systematic bias of x to y quantitative comparison.

**Citation:**

**Wolk D, Marlowe E.**2016. Molecular Method Verification, p 721-744.

*In*Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed),

*Molecular Microbiology*. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50

## References

## Tables

**Citation:**

**Wolk D, Marlowe E.**2016. Molecular Method Verification, p 721-744.

*In*Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed),

*Molecular Microbiology*. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50

Test selection summary: important CLSI protocols for test verification

##### TABLE 1

Test selection summary: important CLSI protocols for test verification

**Citation:**

**Wolk D, Marlowe E.**2016. Molecular Method Verification, p 721-744.

*In*Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed),

*Molecular Microbiology*. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50

Requirements for verification of molecular microbiology assays a

##### TABLE 2

Requirements for verification of molecular microbiology assays a

**Citation:**

**Wolk D, Marlowe E.**2016. Molecular Method Verification, p 721-744.

*In*Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed),

*Molecular Microbiology*. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50

Overview of association methods

##### TABLE 3

Overview of association methods

**Citation:**

**Wolk D, Marlowe E.**2016. Molecular Method Verification, p 721-744.

*In*Persing D, Tenover F, Hayden R, Ieven M, Miller M, Nolte F, Tang Y, van Belkum A (ed),

*Molecular Microbiology*. ASM Press, Washington, DC. doi: 10.1128/9781555819071.ch50