Predicting the sensitivity and specificity of published real-time PCR assays

  • Gordon H Lemmon1 and

    Affiliated with

    • Shea N Gardner2Email author

      Affiliated with

      Annals of Clinical Microbiology and Antimicrobials20087:18

      DOI: 10.1186/1476-0711-7-18

      Received: 17 June 2008

      Accepted: 25 September 2008

      Published: 25 September 2008

      Abstract

      Background

      In recent years real-time PCR has become a leading technique for nucleic acid detection and quantification. These assays have the potential to greatly enhance efficiency in the clinical laboratory. Choice of primer and probe sequences is critical for accurate diagnosis in the clinic, yet current primer/probe signature design strategies are limited, and signature evaluation methods are lacking.

      Methods

      We assessed the quality of a signature by predicting the number of true positive, false positive and false negative hits against all available public sequence data. We found real-time PCR signatures described in recent literature and used a BLAST search based approach to collect all hits to the primer-probe combinations that should be amplified by real-time PCR chemistry. We then compared our hits with the sequences in the NCBI taxonomy tree that the signature was designed to detect.

      Results

      We found that many published signatures have high specificity (almost no false positives) but low sensitivity (high false negative rate). Where high sensitivity is needed, we offer a revised methodology for signature design which may designate that multiple signatures are required to detect all sequenced strains. We use this methodology to produce new signatures that are predicted to have higher sensitivity and specificity.

      Conclusion

      We show that current methods for real-time PCR assay design have unacceptably low sensitivities for most clinical applications. Additionally, as new sequence data becomes available, old assays must be reassessed and redesigned. A standard protocol for both generating and assessing the quality of these assays is therefore of great value. Real-time PCR has the capacity to greatly improve clinical diagnostics. The improved assay design and evaluation methods presented herein will expedite adoption of this technique in the clinical lab.

      Background

      Real-time PCR assays are gaining popularity as a clinical tool for detecting and quantifying the presence of both viral and bacterial pathogens, as reviewed in [1]. Compared to traditional culturing methods used in identification, real-time PCR is fast and cost effective. In addition, it can be quantitative and sensitive, in some cases greatly exceeding the sensitivity for conventional testing methods. Commercially distributed kits are available for PCR-based pathogen diagnostics, and PCR is no longer thought of merely as confirmatory to culture. However real-time PCR assays are limited by the quality of the primers and probes chosen. These primers and probes must be sensitive enough to match all target organisms yet specific enough to exclude all others.

      A common approach to developing a primer/probe combination is by using commercial software such as PrimerExpress® (Applied Biosystems, Foster City, CA, USA). This software asks the user to upload a DNA sequence file, and then finds possible primer/probe sets that meet the assay criteria. Generally a researcher will provide as input a gene region conserved throughout the taxa that the assay is being designed to detect. The software then provides possible primer/probe sets. The researcher chooses a representative signature. If there are single nucleotide polymorphisms (SNPs) within the chosen conserved region, a signature with consensus primers and probes is often chosen. Next a BLAST [2] search is performed to ensure that the primers are not hitting other targets. Finally the signature is verified in vitro with laboratory strains.

      While this design approach may work acceptably well in the research laboratory, the clinical laboratory calls for a more thorough analysis to ensure detection of novel, diverse, and uncommon strains. These may appear, for example, as a result of spread by foreign travel or migration. Whole genome based automated signature design [3] presents a great improvement to the common method. However, in addition to better design strategies, methods for automated signature evaluation are needed. As additional sequence data becomes available, it is necessary to regularly reassess the predicted efficacy of a given signature. This analysis must include the predicted false negative and false positive rates for the developed signatures, and consider all available public sequence data.

      We have analyzed a number of real-time PCR assays found in the literature based on public sequence data. Herein we report how well these signatures performed, offer a revised approach to PCR assay design, and use this approach to produce new assays predicted to have higher sensitivity and specificity.

      Methods

      Literature search

      The literature was combed for recently published articles reporting real-time PCR assays for the clinical detection of bacterial and viral taxa. The primer and probe sequences were accumulated, with a preference for TaqMan assays. However, 3 intercalating dye assays were also selected. Papers reporting nucleotide sequences that could not easily be copied from an online source were avoided. In total, 112 signatures from 32 papers were analyzed.

      Database construction

      Local Oracle databases have been constructed from the complete genome sequence data available at NCBI Genbank, TIGR, EMBL, IMG (JGI), and Baylor HGSC. We used our "all_virus" and "all_bacteria" databases to find signature matches and predict false negatives and false positives. These databases were designed to contain only whole genomes and whole segments from segmented genomes. However, the heuristics used to separate whole genomes from partial sequences are not fail-proof due to inconsistency in sequence annotation within the public databases. Consequently many sequences in these databases may show up as false negatives when they are actually just a section or segment of a genome that is not expected to contain the signature, and we manually sorted these sequences into true or false negatives.

      BLAST search

      A freely available real time PCR analysis tool called TaqSim [4] was used to find public sequences that would match the primer/probe assay in question. TaqSim uses BLAST searches to find sequences that match both forward and reverse primers and probe. To be reported as a "hit" the primers and probe must match in the required orientations relative to one another and the primers must be in sufficiently close proximity. The forward/reverse primers may fall on either the plus or minus strand, so long as the orientation relative to one another is appropriate. There may not be mismatches at the 3' end of either primer. For each hit TaqSim calculates the primer and probe melting temperatures as bound to the candidate hit sequence (accounting for mismatches) based on reaction conditions (reagent concentrations and hybridization temperature), and returns sequences predicted to be amplified. Instead of replicating the various exact reaction conditions reported in each paper, very lenient settings were applied in all cases, essentially removing the screen for primer/probe vs. candidate hit Tm by setting this threshold to 0°K, and instead checking for specificity by requiring that hits have fewer than 3 mismatches per primer or probe.

      Signature/taxonomy comparison

      TaqSim's predicted sequence hits were compared with sequences listed under a given set of NCBI taxonomy tree nodes. For instance, if a signature was reported to detect Hepatitis B, then its set of TaqSim hits would be compared with the set of sequences under node 10407, corresponding to Hepatitis B virus. Sequences in both sets were considered true positives, sequences in the TaqSim output that were missing from the chosen taxonomy nodes were considered false positives, and sequences that were in the taxonomy tree but missing from the TaqSim output were considered false negatives. Test statistics such as specificity and sensitivity (power) were then calculated. In this paper we define sensitivity and specificity as follows:
      S e n s i t i v i t y = 1 f a l s e n e g a t i v e r a t e = 1 f a l s e n e g a t i v e s t r u e p o s i t i v e s + f a l s e n e g a t i v e s MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaem4uamLaemyzauMaemOBa4Maem4CamNaemyAaKMaemiDaqNaemyAaKMaemODayNaemyAaKMaemiDaqNaemyEaKNaeyypa0JaeGymaeJaeyOeI0IaemOzayMaemyyaeMaemiBaWMaem4CamNaemyzauMaeeiiaaIaemOBa4MaemyzauMaem4zaCMaemyyaeMaemiDaqNaemyAaKMaemODayNaemyzauMaeeiiaaIaemOCaiNaemyyaeMaemiDaqNaemyzauMaeyypa0JaeGymaeJaeyOeI0scfa4aaSaaaeaacqWGMbGzcqWGHbqycqWGSbaBcqWGZbWCcqWGLbqzcqqGGaaicqWGUbGBcqWGLbqzcqWGNbWzcqWGHbqycqWG0baDcqWGPbqAcqWG2bGDcqWGLbqzcqWGZbWCaeaacqWG0baDcqWGYbGCcqWG1bqDcqWGLbqzcqqGGaaicqWGWbaCcqWGVbWBcqWGZbWCcqWGPbqAcqWG0baDcqWGPbqAcqWG2bGDcqWGLbqzcqWGZbWCcqGHRaWkcqWGMbGzcqWGHbqycqWGSbaBcqWGZbWCcqWGLbqzcqqGGaaicqWGUbGBcqWGLbqzcqWGNbWzcqWGHbqycqWG0baDcqWGPbqAcqWG2bGDcqWGLbqzcqWGZbWCaaaaaa@9633@ http://static-content.springer.com/image/art%3A10.1186%2F1476-0711-7-18/MediaObjects/12941_2008_Article_134_Equa_HTML.gif
      S p e c i f i c i t y = 1 f a l s e p o s i t i v e r a t e = 1 f a l s e p o s i t i v e s t r u e n e g a t i v e s + f a l s e p o s i t i v e s MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaem4uamLaemiCaaNaemyzauMaem4yamMaemyAaKMaemOzayMaemyAaKMaem4yamMaemyAaKMaemiDaqNaemyEaKNaeyypa0JaeGymaeJaeyOeI0IaemOzayMaemyyaeMaemiBaWMaem4CamNaemyzauMaeeiiaaIaemiCaaNaem4Ba8Maem4CamNaemyAaKMaemiDaqNaemyAaKMaemODayNaemyzauMaeeiiaaIaemOCaiNaemyyaeMaemiDaqNaemyzauMaeyypa0JaeGymaeJaeyOeI0scfa4aaSaaaeaacqWGMbGzcqWGHbqycqWGSbaBcqWGZbWCcqWGLbqzcqqGGaaicqWGWbaCcqWGVbWBcqWGZbWCcqWGPbqAcqWG0baDcqWGPbqAcqWG2bGDcqWGLbqzcqWGZbWCaeaacqWG0baDcqWGYbGCcqWG1bqDcqWGLbqzcqqGGaaicqWGUbGBcqWGLbqzcqWGNbWzcqWGHbqycqWG0baDcqWGPbqAcqWG2bGDcqWGLbqzcqWGZbWCcqGHRaWkcqWGMbGzcqWGHbqycqWGSbaBcqWGZbWCcqWGLbqzcqqGGaaicqWGWbaCcqWGVbWBcqWGZbWCcqWGPbqAcqWG0baDcqWGPbqAcqWG2bGDcqWGLbqzcqWGZbWCaaaaaa@9655@ http://static-content.springer.com/image/art%3A10.1186%2F1476-0711-7-18/MediaObjects/12941_2008_Article_134_Equb_HTML.gif

      Taxonomy node selection

      The primary research articles were read carefully to determine what the authors had designed their primers/probes to detect. NCBI Taxonomy nodes were chosen to represent these target organisms. This was not a trivial task, since many articles lack clarity as to which taxa, specifically, their assay should detect. For instance, the cytomegalovirus assay did not detect all sequences in the cytomegalovirus genus (taxonomy node 10358), but rather all sequences in the human herpesvirus 5 species (taxonomy node 10359). None of the articles specified a taxonomy node for their signatures.

      Hand curating

      Perl scripting [5] was used to help compare BLAST hits and taxonomy node sequences, and count false negative, false positive and true positive sequence matches. However some sequences required hand sorting due to the wide array of sequence types and annotations. These often represented segmented genomes, in which case many of the would-be false negative sequences simply represent a different segment than that on which the signature lands, so we manually tabulated them as true negatives. They may also represent plasmids. In these situations, a careful review of the Genbank entry, and sometimes of the primary article cited by Genbank, was necessary to determine if the sequence of interest was truly a false negative.

      Although we attempt to include only complete genomes in our sequence database, because of inconsistencies in the annotation of sequence data some partial sequences nevertheless make it into our databases. Any of these partial CDS's documented as containing the target gene on which the signature was supposed to land were counted as false negatives, but those partial CDS's not documented to contain the target gene were eliminated from the false negative pool because it is possible the signature could land on the unsequenced section with the target gene. Our database also contains "glued fragments", which represent draft genomes "glued" together with hundreds of "N"s as a simple way to keep the separate contigs associated as part of the same genome. While we report false negatives from these draft genomes, it is possible that the signatures could land on gaps between the contigs, and that finished sequencing could result in re-classification as a true positive.

      Results

      Tables 1, 2, 3, 4, 5 summarize our analysis of various DNA signatures. Details of all true positive, false positive and false negative sequences are available from the authors. Note that these results are in silico results; no laboratory testing was performed for verification, so that by stating that an organism is "detected" we mean that this is our prediction based on sequence data. A few notes of interest concerning the data in the tables are described below:
      Table 1

      viral signature analyses.

      Reference Number

      Scope

      Year

      Target Organism/gene

      True Positives

      False Positives

      False Negatives

      Sensitivity

      [6]

      Strain

      2004

      Human coronavirus OC43

      5

      17

      0

      1.00

      [6]

      Strain

      2004

      Human coronavirus 229E

      1

      0

      0

      1.00

      [7]

      Strain

      2004

      Coxsackie virus B3

      8

      0

      2

      0.80

      [8]

      Strain

      2004

      Coxsackie virus B4

      3

      78

      0

      1.00

      [20]

      Serotype

      2007

      Influenza H5 strains

      26

      0

      402

      0.06

      [21]

      Species

      2004

      Influenza A MP segment

      238

      0

      3726

      0.06

      [21]

      Species

      2004

      Influenza B HA segment

      131

      0

      238

      0.36

      [10]

      Species

      2002

      Dengue virus

      0

      0

      185

      0.00

      [16]

      Serotype

      2006

      Dengue virus type 1

      0

      0

      47

      0.00

      [16]

      Serotype

      2006

      Dengue virus type 2

      0

      0

      57

      0.00

      [16]

      Serotype

      2006

      Dengue virus type 3

      2

      0

      68

      0.03

      [16]

      Serotype

      2006

      Dengue virus type 4

      0

      0

      11

      0.00

      [21]

      Species

      2004

      Adenovirus A

      1

      0

      0

      1.00

      [21]

      Species

      2004

      Adenovirus B

      17

      0

      3

      0.85

      [21]

      Species

      2004

      Adenovirus C

      6

      0

      0

      1.00

      [21]

      Species

      2004

      Adenovirus D

      6

      0

      0

      1.00

      [21]

      Species

      2004

      Adenovirus E

      12

      0

      0

      1.00

      [21]

      Species

      2004

      Adenovirus F

      2

      0

      0

      1.00

      [10]

      Species

      2002

      Lassa Virus

      1

      0

      14

      0.08

      [9]

      Species

      2004

      Ebola Sudan

      1

      0

      0

      1.00

      [9]

      Species

      2004

      Ebola Zaire

      5

      0

      0

      1.00

      [9]

      Species

      2004

      Marburg Virus

      13

      0

      5

      0.72

      [22]

      Species

      1996

      Hepatitis C virus

      102

      0

      33

      0.76

      [10]

      Species

      2002

      Rift Valley fever virus

      36

      0

      5

      0.88

      [23]

      Species

      2000

      West Nile Virus 3'NC

      71

      0

      13

      0.85

      [23]

      Species

      2000

      West Nile Virus- ENV

      71

      0

      13

      0.85

      [15]

      Species

      2007

      West Nile virus- RdRp

      49

      0

      35

      0.58

      [15]

      Species

      2007

      Japanese encephalitis virus

      0

      0

      43

      0.00

      [15]

      Species

      2007

      Yellow fever virus

      0

      0

      15

      0.00

      [15]

      Species

      2007

      St. Louis encephalitis virus

      0

      0

      2

      0.00

      [10]

      Species

      2002

      Yellow fever virus

      10

      0

      5

      0.67

      [24]

      Species

      2000

      Hepatitis B virus (1)

      942

      0

      56

      0.94

      [25]

      Species

      2000

      Hepatitis B virus (2)

      641

      0

      357

      0.64

      [26]

      Species

      2004

      Cytomegalovirus

      4

      0

      3

      0.57

      [21]

      Species

      2004

      Cytomegalovirus

      4

      0

      3

      0.57

      [21]

      Species

      2004

      Epstein-Barr (HHV 4)

      4

      0

      0

      1.00

      [27]

      Species

      2005

      Herpes Simplex Virus 1

      1

      0

      0

      1.00

      [21]

      Species

      2004

      Herpes Simplex Virus 1

      1

      0

      0

      1.00

      [27]

      Species

      2005

      Herpes Simplex Virus 2

      1

      0

      0

      1.00

      [21]

      Species

      2004

      Herpes Simplex Virus 2

      1

      0

      0

      1.00

      [28]

      Species

      1999

      Varicella-Zoster

      19

      0

      0

      1.00

      [21]

      Species

      2004

      Varicella-Zoster

      19

      0

      0

      1.00

      [21]

      Species

      2004

      Human herpesvirus 6

      3

      0

      0

      1.00

      [21]

      Species

      2004

      Human herpesvirus 7

      1

      0

      0

      1.00

      [21]

      Species

      2004

      Human herpesvirus 8

      2

      0

      0

      1.00

      [29]

      Species

      2005

      Mumps virus

      12

      0

      5

      0.71

      [30]

      Species

      2004

      Newcastle Disease irus

      23

      0

      15

      0.61

      [21]

      Species

      2004

      Parainfluenzaviruses-1

      1

      0

      0

      1.00

      [21]

      Species

      2004

      Parainfluenzaviruses-2

      4

      0

      1

      0.80

      [21]

      Species

      2004

      Parainfluenzaviruses-3

      2

      0

      0

      1.00

      [21]

      Species

      2004

      Respiratory syncytial virus

      7

      0

      3

      0.70

      [21]

      Species

      2004

      Human parvovirus B19

      3

      0

      0

      1.00

      [21]

      Genus

      2004

      Enteroviruses

      170

      0

      21

      0.89

      [21]

      Species

      2004

      JC polyomavirus

      378

      0

      2

      1.00

      [21]

      Species

      2004

      BK polyomavirus

      120

      378

      0

      1.00

      [31]

      Species

      2005

      HIV – type 1

      283

      1

      820

      0.26

      [10]

      Species

      2002

      Crimean-Congo hemorrhagic fever

      27

      0

      21

      0.56

      [32]

      Genus

      2001

      Enterovirus genus

      171

      0

      19

      0.90

      [10]

      Family

      2002

      Ebola and Marburg

      6

      0

      21

      0.22

      All but three are Taqman signatures. The three intercalating dye type assays are bolded.

      Table 2

      assorted bacterial DNA signature analyses.

       

      Scope

      Year

      Target Organism/gene

      True Positives

      False Positives

      False Negatives

      Sensitivity

      [33]

      Species

      2004

      Neisseria gonorrhoeae1

      1

      0

      0

      1.00

      [33]

      Species

      2004

      Neisseria gonorrhoeae2

      1

      0

      0

      1.00

      [34]

      Species

      2003

      Chlamydia pneumonia

      4

      0

      0

      1.00

      [35]

      Species

      2007

      Borrelia plasmid Ip54

      2

      0

      0

      1.00

      [12]

      Genus

      2004

      Bacteroides

      0

      0

      9

      0.00

      [12]

      Species

      2004

      Escherichia coli

      25

      11

      5

      0.83

      [36]

      Species

      2005

      Ehrlichia chaffeensis

      1

      0

      0

      1.00

      [12]

      Species

      2004

      Ehrlichia canis

      1

      0

      0

      1.00

      [11]

      Species

      2000

      Staphylococcus aureaus

      9

      2

      4

      0.69

      [12]

      Species

      2004

      Haemophilus influenzae

      0

      0

      15

      0.00

      [12]

      Species

      2004

      Pseudomonas aeruginosa

      0

      0

      7

      0.00

      [12]

      Genus

      2004

      Acinetobacter spp.

      2

      0

      0

      1.00

      [12]

      Family

      2004

      Enterobacteriaceae

      45

      0

      42

      0.52

      [12]

      Species

      2004

      Stenotrophomonas maltophilia

      0

      0

      1

      0.00

      Table 3

      Analysis of measles virus assays from Hummel, 2006 [13].

      Scope

      Target Gene

      True Positives

      False Positives

      False Negatives

      Sensitivity

      Species

      Measles F1

      15

      0

      2

      0.88

      Species

      Measles N1

      16

      0

      1

      0.94

      Species

      Measles F2

      15

      0

      2

      0.88

      Species

      Measles F4b

      16

      0

      1

      0.94

      Species

      Measles F3

      9

      0

      8

      0.53

      Species

      Measles N2

      10

      0

      7

      0.59

      Species

      Measles N3

      17

      0

      0

      1.00

      Species

      Measles H1

      10

      0

      7

      0.59

      Species

      Measles H2

      16

      0

      1

      0.94

      Species

      Measles H3

      9

      0

      8

      0.53

      Species

      Measles H4b

      14

      0

      3

      0.82

      Species

      Measles N4b

      11

      0

      6

      0.65

      Table 4

      Analysis of hepatitis assays from Gardner et al, 2003 [14].

      Scope

      Target Organism

      True Positives

      False Positives

      False Negatives

      Sensitivity

      Species

      Hepatitis A-1

      14

      0

      3

      0.82

      Species

      Hepatitis A-2

      14

      0

      3

      0.82

      Species

      All Hepatitis As

      14

      0

      3

      0.82

      Species

      Hepatitis B-1

      927

      2

      37

      0.96

      Species

      Hepatitis B-2

      799

      0

      165

      0.83

      Species

      All Hepatitis Bs

      959

      2

      5

      1.00

      Species

      Hepatitis C-1

      130

      0

      4

      0.97

      Species

      Hepatitis C-2

      76

      0

      58

      0.57

      Species

      All Hepatitis Cs

      133

      0

      1

      0.99

      Species

      Hepatitis E-1

      25

      0

      40

      0.39

      Species

      Hepatitis E-2

      11

      0

      54

      0.17

      Species

      Hepatitis E-3

      20

      0

      45

      0.31

      Species

      All Hepatitis Es

      55

      0

      10

      0.85

      Clustered signature results are bolded.

      Table 5

      Signatures predicted not to hit any target organisms

       

      Target Organism

      Year

      Analysis

      [15]

      Japanese encephalitis virus

      2007

      Too many mismatches in either forward or reverse primer. Several strains have 3 mismatches at 3' end of forward primer in addition to internal mismatches.

      [15]

      Yellow fever virus

      2007

      Reverse primer only has a blast hit to one strain (Angola71). Forward primer only has blast hits to 3 strains, and there are many mismatches (e.g. for Angola71, the 11 bases at the 3' end of the primer do not match).

      [15]

      Saint Louis encephalitis virus

      2007

      Too many mismatches in the reverse primer, with 3 mismatches at 3' end as well as at other locations.

      [16]

      Dengue virus 1

      2006

      Reverse primer does not have any BLAST hits to target.

      [16]

      Dengue virus 2

      2006

      Forward primer has 3 or 12 mismatches at 3' end for most strains, the probe has BLAST hits to only 7 of the 57 genomes available, and reverse primer only has a BLAST hit to 1 genome but there are 3 mismatches at the 3' end.

      [16]

      Dengue virus 4

      2006

      Too many mismatches in forward primer and in some cases the probe.

      [10]

      Dengue virus

      2002

      Too many mismatches in forward primer. However, they are at the 5' end, so assay could still work for some strains with 19 matches at the 3' end of the forward primer.

      [12]

      Pseudomonas aeruginosa

      2004

      No blast hits of probe to Pseudomonas aeruginosa

      [12]

      Bacteroides spp.

      2004

      Probe is not between or even in close proximity to the forward and reverse primers

      [12]

      Stenotrophomonas maltophilia

      2004

      No blast hits of probe to Stenotrophomonas maltophilia

      [12]

      Haemophilus influenzae

      2004

      Probe only matches in 17 of 22 bases, which is unlikely to give a strong signal, since probe is unlikely to bind prior to the primers as desired for real time TaqMan chemistry.

      Assorted viral signatures (table 1)

      The two human corona virus strains, 229E and OC43 are a frequent cause of the common cold [6]. A Taqman assay for 229E was predicted to perform perfectly, while an assay for OC43 turned up a number of false positives, all of which were animal corona viruses. A coxsackie B3 virus assay [7] performed well, but a coxsackie B4 assay [8] hit many other human coxsackie, echo, and entero viruses. Four out of 5 false negatives for a Marburg virus assay [9] were of the Lake Victoria variety. False negatives associated with a yellow fever signature [10] included Trinidad, French neurotropic, French viscerotropic, and vaccine strains. The Filoviridae (Ebola/Marburg) assay [10] detected only Ebola viruses.

      Assorted bacterial signatures (table 2)

      Staphylococcus aureus [11] and Enterobacteriaceae assays [12] had low sensitivity. An Escherichia coli assay [12] hit Shigella and Vibrio sequences.

      Measles signature set (table 3)

      Many of these signatures [13] had high sensitivity. Combining several of them into a multiplex assay would probably improve sensitivity further.

      Hepatitis signature set (table 4)

      These signatures were designed using a minimal set clustering approach [14]. While individual signatures have decent sensitivity, combining several signatures in one assay, as advocated in the publication greatly improved sensitivity. The signatures for Hepatitis A are currently undergoing laboratory screening by the FDA, and are performing well (G. Hartman, personal communication).

      Signatures that had no hits (table 5)

      Several reported signatures produced no predicted hits. These include assays for several flaviviruses [10, 15, 16], and 16S rRNA assays [12] for several bacteria. Examination of BLAST output showed that in these cases either a primer or internal oligo (probe) did not have BLAST hits to target, there were too many mismatches per primer or probe sequence above the threshold specified in our analyses, or there were mismatches at the 3' end of a primer relative to target. It is possible that if the sequences of the samples used in the laboratory differ from available genomic data, or if the PCR reaction conditions are performed at low stringency (e.g. low annealing temperatures or high salt concentrations) these assays could in fact work in the laboratory. However, according to the genomic data available, a better match of primers and probes to target is possible and is usually desired for high sensitivity detection.

      Improved signatures using minimal set clustering (additional file)

      Targeting a number of the organisms for which currently published signatures were predicted to perform poorly, as well as some for which additional signatures may be desired (even though published signatures may perform well), we generated new signatures using Minimal Set Clustering (MSC) according to methods previously described [14, 17]. MSC begins by removing non-unique regions from consideration as primers or probes from each of the target sequences relative to a database of non-target bacterial and viral sequences. The remaining unique regions of each target sequence are mined for all or many candidate signatures, without regard for conservation among other targets, yet satisfying user specifications for primer and probe length, Tm, GC%, avoidance of long homopolymer runs, and amplicon length. All candidate signatures are compared to all targets and clustered by the subset of targets they are predicted to detect. Signatures within a given cluster are equivalent, in that they are predicted to detect the same subset of targets, so by clustering we reduce the redundancy and size of the problem to finding a small set of signatures that detect all targets. Nevertheless, finding the optimal solution of the fewest clusters to detect all targets is an NP complete problem, so for large data sets we use a greedy algorithm to find a small number of clusters that together should pick up all targets. In cases where the target strains of a species are too diverse to be detected by a single signature that hits all target strains, the MSC software groups the targets into genetic clusters to enable signature generation per group. If there are single signatures that are sufficient to detect all targets, then each of those comprehensive signatures is reported. We have used this method to design signature sets for numerous viruses, including Influenza A HA serotypes, foot-and-mouth-disease virus, Norwalk, Crimean Congo hemorrhagic fever, Ebola, hepatitis A, and other divergent viruses. Many of these signatures have been tested by collaborators at Lawrence Livermore National Laboratory, US Centers for Disease Control (CDC), USDA, FDA, and elsewhere.

      In the supplementary table, we often provide more than one alternative signature to detect a given equivalence group of genomes to serve as a backup should a signature perform poorly in laboratory testing. Some of the signatures may have mismatches to some of their intended targets, although these mismatches are not predicted to reduce the Tm of primer/probe hybridizing to target below typical TaqMan reaction conditions. None of these computationally predicted signatures have been screened in the laboratory, as this is beyond the scope of this paper.

      The supplementary table contains signatures for the following organisms: Viruses: Dengue 1–4, human adenovirus A-F, canine distemper, coxsackie B4, cytomegalovirus, human herpes 1–4, 6–7, Japanese encephalitis, mumps, newcastle disease, Sendai, St. Louis encephalitis, human papillomavirus type 16, 71, human parainfluenza 1–3, JC, BK, and WU polyomavirus, West Nile, yellow fever, rabies, human respiratory syncytial, influenza A-B, influenza A serotypes H1, H2, H3, and H5, and HIV-1; Bacteria: Haemophilus influenzae, Mycobacterium tuberculosis, Staphylococcus aureus, Ehrlichia (genus), E. chaffeensis, E. ruminantium, Chlamydia trachomatis, Pseudomonas aeruginosa, Escherichia coli, and Neisseria meningitidis.

      Discussion

      As expected we found that false negatives were much more common than false positives. Though signatures are generally based on conserved gene regions, they often fail to take into account all of the variation within a target set of organisms. This may be because the signatures were developed using sequence data from a handful of strains, rather than a thorough study of all strains publicly available.

      These false negatives may also represent sequences that have become available since the publication of the given signature. Since new sequence data is made available at an ever increasing rate, there is great benefit in re-evaluating clinically used DNA signatures regularly. When new sequence data leads to false negative predictions for a signature, one of two explanations can be given. The new sequences either represent recently recognized variation that has been around since the time the signature was published, or new variation, the result of mutation and natural selection. In either case, an improved or additional signature should be designed.

      High false positive or false negative rates do not necessarily indicate a "bad" DNA assay. The quality of an assay must be considered in light of the milieu in which the testing will take place. In the clinical laboratory, a signature with high sensitivity but perhaps low specificity may be preferred over a test with lower sensitivity in cases where the putative pathogen requires immediate treatment or may spread quickly. The case of antibiotic resistant bacteria probably falls in this category. On the other hand, the nation's BASIS and BioWatch programs insists on zero false positives, so as to avoid public disturbances due to false alarms, while still aiming for zero false negatives [18].

      One must also consider the type of false negative and false positive results to determine their relevance. For instance, in this article an assay for human corona virus OC43 [6] was reported with a number of false positives to animal corona viruses. However, in the clinical lab in the US, need the clinician worry about the possibility of a false positive match to giraffe coronavirus in human sputum? What about such a match in a clinical lab in Africa? On the other hand, the echovirus sequences that the coxsackie B4 assay [8] can detect could produce misleading results in any clinical lab.

      The false negative and false positive rates presented in this study may vary substantially from those seen empirically. This is because the strains available in a laboratory may differ significantly from the sequence data available, or because the empirical protocol is more or less stringent than the sequence-based requirements we imposed, which allowed no more than 2 mismatches per primer or probe for detection.

      We believe that as more target sequences become available, our predicted false negative rates will tend to increase for a given published signature both as a result of better sampling of diversity and as a result of failure to detect newly evolved variants. It has been estimated that a minimum of 3–4 genomes are needed in order to computationally design TaqMan PCR signatures likely to detect most strains, with those isolates chosen for sequencing that have been selected to span gradients of geographic, phenotypic, and temporal variation [19]. Even more than 4 genomes are needed for particularly diverse organisms. Thus, older signatures may not perform as well as newly developed signatures from the most up-to-date sequence data. A future study of interest would be a longitudinal look at how these rates continue to change over time as additional sequences become available. This study could be performed retrospectively, since sequence submission dates are easily obtained from public databases.

      We also hypothesized that the wider the intended scope of a signature, the lower its sensitivity would tend to be. The point is illustrated loosely in our data tables. Twenty-six of the 28 signatures with less than 10 publicly available target sequences had sensitivities of 1 (i.e. zero false negatives), while signatures with 10 or more targets had an average sensitivity of 0.710116. However this approach only considers scope in the context of sequence data available.

      We tried to demonstrate the relationship between specificity and scope at a more fundamental level by grouping signatures by the taxonomic level of their target as shown in Figure 1. However the results are misleading. In virology, taxonomic level is not a good indicator of nucleotide diversity. For instance, there is more diversity in the influenza A species then there is in the entire Filoviridae family, which consists of only two known genera: Ebola-like viruses and Marburg viruses. A better approach might be to calculate nucleotide diversity as a function of phylogenetic branch length or shared k-mer clusters within a target taxonomy node.
      http://static-content.springer.com/image/art%3A10.1186%2F1476-0711-7-18/MediaObjects/12941_2008_Article_134_Fig1_HTML.jpg
      Figure 1

      Sensitivity by taxonomy level. Each colored diamond represents a real-time PCR assay examined in this paper. Black bars indicate the mean, grey bars indicate the median. Top and bottom of each box indicates 75th and 25th percentiles, and grey lines at whisker ends denote min and max values. The wide ranging sensitivities demonstrate both inconsistency in genetic diversity at a given taxonomy level, and inconsistency in signature design approaches.

      Finally, we averaged the sensitivities of microbes by genome type as shown in Table 6. Note that the ssRNA-RT category includes only HIV-1. This chart demonstrates that creating signatures with high sensitivity becomes more difficult for target organisms with high mutation rates.
      Table 6

      sensitivity by type.

       

      avg.

      std dev

      dsDNA

      0.97

      0.10

      dsDNA-RT

      0.79

      0.21

      Bacteria

      0.86

      0.86

      ssRNA

      0.71

      0.28

      ssRNA-RT

      0.26

       

      Average sensitivities of bacteria and of single stranded (ss), and double stranded (ds) viruses, with and without reverse transcriptase (RT). Results are consistent with high mutation rates in groups for which signatures have lower sensitivity. ssRNA-RT has no standard deviation as it refers only to HIV.

      Conclusion

      Current real-time PCR assay design approaches produce signatures with sensitivities generally too low for clinical use. We suggest that a rigorous approach involving false positive and false negative analysis should be the standard by which an initial assessment of signature quality is made. Signatures must also regularly be reassessed as sequence data becomes available. For targets with wide nucleotide diversity, it becomes necessary to develop a set of signatures, for which we suggest a minimal set clustering approach that may also include signatures with degenerate/inosine bases.

      Declarations

      Acknowledgements

      We thank Beth Vitalis, Jason Smith, and Tom Slezak for helpful discussion and for encouraging this work, and Kari Allmon for entering the references. We gratefully acknowledge financial support from the Intelligence Technology Innovation Center. Lawrence Livermore National Laboratory is operated by Lawrence Livermore National Security, LLC, for the U.S. Department of Energy, National Nuclear Security Administration under Contract DE-AC52-07NA27344.

      Authors’ Affiliations

      (1)
      Center for Structural Biology, Vanderbilt University
      (2)
      Pathogen Bioinformatics, Lawrence Livermore National Laboratory

      References

      1. Espy MJ, Uhl JR, Sloan LM, Buckwalter SP, Jones MF, Vetter EA, Yao JDC, Wengenack NL, Rosenblatt JE, Cockerill FR, Smith TF: Real-time PCR in clinical microbiology: Applications for routine laboratory testing. Clinical Microbiology Reviews. 2006, 19 (1): 165-256. 10.1128/CMR.19.1.165-256.2006PubMed CentralView ArticlePubMed
      2. Altschul SF, Gish W, Miller W, Myers EW, Lipman DJ: Basic local alignment search tool. J Mol Biol. 1990, 215: 403-410.View ArticlePubMed
      3. Phillipy AM, Mason JA, Ayanbule K, Sommer DD, Taviani E, Huq A, Colwell RR, Knight IT, Salzberg SL: Comprehensive DNA signature discovery and validation. PLoS Computational Biology. 2007, 18 (3): e98-10.1371/journal.pcbi.0030098.View Article
      4. TaqSim: TaqMan PCR Simulatorhttp://​staff.​vbi.​vt.​edu/​dyermd/​publications/​taqsim.​html
      5. The Perl Directoryhttp://​www.​perl.​org/​
      6. van Elden L, van Loon AM, van Alphen F, Hendriksen K, Hoepelman A, van Kraaij M, Oosterheert J, Schipper P, Schuurman R, Nijhuis M: Frequent Detection of Human Coronaviruses in Clinical Specimens from Patients with Respiratory Tract Infection by Use of a Novel Real-Time Reverse-Transcriptase Polymerase Chain Reaction. The Journal of Infectious Diseases. 2004, 189: x652-657. 10.1086/381207.View Article
      7. Padalko E, Nuyens D, De Palma A, Verbeken E, Aerts JL, De Clercq E, Carmeliet P, Neyts J: The Interferon Inducer Ampligen Markedly Protects Mice against Coxsackie B3 Virus-Induced Myocarditis. Antimicrobial Agents and Chemotherapy. 2004, 48 (1): 267-274. 10.1128/AAC.48.1.267-274.2004.PubMed CentralView ArticlePubMed
      8. Brilot F, Geenen V, Hober D, Stoddart CA: Coxsackievirus B4 Infection of Human Fetal Thymus Cells. Journal of Virology. 2004, 78 (18): 9854-9861. 10.1128/JVI.78.18.9854-9861.2004PubMed CentralView ArticlePubMed
      9. Weidmann M, Muhlberger E, Hufert FT: Rapid detection protocol for filoviruses. Journal of Clinical Virology. 2004, 30 (1): 94-99. 10.1016/j.jcv.2003.09.004View ArticlePubMed
      10. Drosten C, Gottig S, Schilling S, Asper M, Panning M, Schmitz H, Gunther S: Rapid Detection and Quantification of RNA of Ebola and Marburg Viruses, Lassa Virus, Crimean-Congo Hemorrhagic Fever Virus, Rift Valley Fever Virus, Dengue Virus, and Yellow Fever Virus by Real-Time Reverse Transcription-PCR. Journal of Clinical Microbiology. 2002, 40 (7): 2323-2330. 10.1128/JCM.40.7.2323-2330.2002PubMed CentralView ArticlePubMed
      11. Killgore GE, Holloway B, Tenover FC: A 5' Nuclease PCR (TaqMan) High-Throughput Assay for Detection of the mecA Gene in Staphylococci. Journal of Clinical Microbiology. 2000, 38 (7): 2516-2519.PubMed CentralPubMed
      12. Wellinghausen N, Wirths B, Franz AR, Karolyi L, Marre R, Reischl U: Algorithm for the identification of bacterial pathogens in positive blood cultures by real-time LightCycler polymerase chain reaction (PCR) with sequence-specific probes. Diagnostic Microbiology and Infectious Disease. 2004, 48 (4): 229-241. 10.1016/j.diagmicrobio.2003.11.005.View ArticlePubMed
      13. Hummel KB, Lowe L, Bellini WJ, Rota PA: Development of quantitative gene-specific real-time RT-PCR assays for the detection of measles virus in clinical specimens. Journal of Virological Methods. 2006, 132 (1–2): 166-173. 10.1016/j.jviromet.2005.10.006View ArticlePubMed
      14. Gardner SN, Kuczmarski TA, Vitalis EA, Slezak TR: Limitations of TaqMan PCR for detecting divergent viral pathogens illustrated by hepatitis A, B, C, and E viruses and human immunodeficiency virus. Journal of Clinical Microbiology. 2003, 2417-2427. 41
      15. Chao D, Davis BS, Chang GJ: Development of Mulitplex Real-Time Reverse Transcriptase PCR Assays for Detecting Eight Medically Important Flaviviruses in Mosquitoes. Journal of Clinical Microbiology. 2007, 45 (2): 584-589. 10.1128/JCM.00842-06PubMed CentralView ArticlePubMed
      16. Chien L, Liao T, Shu P, Huang J, Gubler DJ, Chang GJ: Development of Real-Time Reverse Transcriptase PCR Assays To Detect and Serotype Dengue Viruses. Journal of Clinical Microbiology. 2006, 44 (4): 1295-1304. 10.1128/JCM.44.4.1295-1304.2006PubMed CentralView ArticlePubMed
      17. Slezak TR, Kuczmarski T, Ott L, Torres C, Medeiros D, Smith J, Truitt B, Mulakken N, Lam M, Vitalis E, Zemla A, Zhou C, Gardner SN: Comparative genomics tools applied to bioterrorism defense. Briefings in Bioinformatics. 2003, 133-149. 4View ArticlePubMed
      18. Fitch JP, Gardner SN, Kuczmarski TA, Kurtz So, Myers R, Ott LL, Slezak TR, Vitalis EA, Zemla AT, McCready PM: Rapid Development of Nucleic Acid Diagnostics. Proceedings of the IEEE. 2002, 90 (11): 1708-1721. 10.1109/JPROC.2002.804680.View Article
      19. Gardner SN, Lam MW, Mulakken NJ, Torres CL, Smith JR, Slezak TR: Sequencing needs for viral diagnostics. Journal of Clinical Microbiology. 2004, 42: 5472-5476. 10.1128/JCM.42.12.5472-5476.2004PubMed CentralView ArticlePubMed
      20. Ellis JS, Smith JW, Braham S, Lock M, Barlow K, Zambon MC: Design and Validation of an H5 TaqMan Real-Time One-Step Reverse Transcription-PCR and Confirmatory Assays for Diagnosis and Verification of Influenza A Virus H5 Infections in Humans. Journal of Clinical Microbiology. 2007, 45 (5): 1535-1543. 10.1128/JCM.02007-06PubMed CentralView ArticlePubMed
      21. Watzinger F, Suda M, Preuner S, Baumgartinger R, Ebner K, Baskova L, Niesters HGM, Lawitschka A, Lion T: Real-Time Quantitative PCR Assays for Detection and Monitoring of Pathogenic Human Viruses in Immunosuppressed Pediatric Patients. Journal of Clinical Microbiology. 2004, 42 (11): 5189-5198. 10.1128/JCM.42.11.5189-5198.2004PubMed CentralView ArticlePubMed
      22. Morris T, Robertson B, Gallagher M: Rapid reverse transcription-PCR detection of hepatitis C virus RNA in serum by using the TaqMan fluorogenic detection system. Journal of Clinical Microbiology. 1996, 34 (12): 2933-2936.PubMed CentralPubMed
      23. Lanciotti RS, Kerst AJ, Nasci RS, Godsey MS, Mitchell CJ, Savage HM, Komar N, Panella NA, Allen BC, Volpe KE, Davis BS, Roehrig JT: Rapid Detection of West Nile Virus from Human Clinical Specimens, Field-Collected Mosquitoes, and Avian Samples by a TaqMan Reverse Transcriptase-PCR Assay. Journal of Clinical Microbiology. 2000, 38 (11): 4066-4071.PubMed CentralPubMed
      24. Pas SD, Fries E, De Man RA, Osterhaus A, Niesters H: Development of a Quantitative Real-Time Detection Assay for Hepatitis B Virus DNA and Comparison with Two Commercial Assays. Journal of Clinical Microbiology. 2000, 38 (8): 2897-2901.PubMed CentralPubMed
      25. Weinberger KM, Wiedenmann E, Bohm S, Jilg W: Sensitive and accurate quantitation of hepatitis B virus DNA using a kinetic fluorescence detection system (TagMan PCR). Journal of Virological Methods. 2000, 85 (1–2): 75-82. 10.1016/S0166-0934(99)00154-8View ArticlePubMed
      26. Piiparinen H, Hockerstedt K, Gronhagen-Riska C, Lautenschlager I: Comparison of two quantitative CMV PCR tests, Cobas Amplicor CMV Monitor and TaqMan assay, and pp65-antigenemia assay in the determination of viral loads from peripheral blood of organ transplant patients. Journal of Clinical Virology. 2004, 30 (3): 258-266.View ArticlePubMed
      27. Corey L, Huang M, Selke S, Wald A: Differentiation of herpes simplex virus types 1 and 2 in clinical samples by a real-time taqman PCR assay. Journal of Medical Virology. 2005, 76 (3): 350-355. 10.1002/jmv.20365View ArticlePubMed
      28. Hawrami K, Breuer J: Development of a flurogenic polymerase chain reaction assay (TaqMan) for the detection and quantitation of varicella zoster virus. Journal of Virological Methods. 1999, 79 (1): 33-40. 10.1016/S0166-0934(98)00176-1View ArticlePubMed
      29. Uchida K, Shinohara M, Shimada S, Segawa Y, Doi R, Gotoh A, Hondo R: Rapid and sensitive detection of mumps virus RNA directly from clinical samples by real-time PCR. Journal of Medical Virology. 2005, 75 (3): 470-474. 10.1002/jmv.20291View ArticlePubMed
      30. Wise MG, Suarez DL, Seal BS, Pedersen JC, Senne DA, King DJ, Kapczynski DR, Spackman E: Development of a Real-Time Reverse-Transcription PCR for Detection of Newcastle Disease Virus RNA in Clinical Samples. Journal of Clinical Microbiology. 2004, 42 (1): 329-338. 10.1128/JCM.42.1.329-338.2004PubMed CentralView ArticlePubMed
      31. Rouet F, Ekouevi DK, Chaix M, Burgard M, Inwoley A, D'Aquin Tony T, Danel C, Anglaret X, Leroy V, Msellati P, Dabis F, Rouzioux C: Transfer and Evaluation of an Automated, Low-Cost Real-Time Reverse Transcription-PCR Test for Diagnosis and Monitoring of Human Immunodeficiency Virus Type 1 Infection in a West African Resource-Limited Setting. Journal of Clinical Microbiology. 2005, 43 (6): 2709-2717. 10.1128/JCM.43.6.2709-2717.2005PubMed CentralView ArticlePubMed
      32. Verstrepen WA, Kuhn S, Kockx MM, Vyvere Van De ME, Mertens AH: Rapid Detection of Enterovirus RNA in Cerebrospinal Fluid Specimens with a Novel Single-Tube Real-Time Reverse Transcription-PCR Assay. Journal of Clinical Microbiology. 2001, 39 (11): 4093-4096. 10.1128/JCM.39.11.4093-4096.2001PubMed CentralView ArticlePubMed
      33. Giles J, Hardick J, Yuenger J, Dan M, Reich K, Zenilman J: Use of Applied Biosystems 7900HT Sequence Detection System and Taqman Assay for Detection of Quinolone-Resistant Neisseria gonorrhoeae. Journal of Clinical Microbiology. 2004, 42 (7): 3281-3283. 10.1128/JCM.42.7.3281-3283.2004PubMed CentralView ArticlePubMed
      34. Apfalter P, Barousch W, Nehr M, Makristathis A, Willinger B, Rotter M, Hirschl AM: Comparison of a New Quantitative ompA-Based Real-time PCR TaqMan Assay for Detection of Chlamydia pneumoniae DNA in Respiratory Specimens with Four Conventional PCR Assays. Journal of Clinical Microbiology. 2003, 41 (2): 592-600. 10.1128/JCM.41.2.592-600.2003PubMed CentralView ArticlePubMed
      35. Ivacic L, Reed KD, Mitchell PD, Ghebranious N: A LightCycler TaqMan assay for detection of Borrelia burgdorferi sensu lato in clinical samples. ScienceDirect – Diagnostic Microbiology and Infectious Disease. 2007, 57 (2): 137-143. 10.1016/j.diagmicrobio.2006.08.005.View Article
      36. Doyle CK, Labruna MB, Breitschwerdt EB, Tang Y, Corstvet RE, Hegarty BC, Bloch KC, Li P, Walker DH, McBride JW: Detection of Medically Important Ehrlichia by Quantitative Multicolor TaqMan Real-Time Polymerase Chain Reaction of the dsb Gene. Journal of Molecular Diagnostics. 2005, 7 (4): 504-514.PubMed CentralView ArticlePubMed

      Copyright

      © Lemmon and Gardner; licensee BioMed Central Ltd. 2008

      This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://​creativecommons.​org/​licenses/​by/​2.​0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

      Advertisement