var.roc(pROC)Spotfire S+ Documentation

Variance of a ROC curve

Description

These functions compute the variance of the AUC of a ROC curve.

Usage

var.roc(x, ...)
## S3 method for class 'auc':
var.roc(auc, ...)
## S3 method for class 'roc':
var.roc(roc, method=c("delong", "bootstrap", "obuchowski"), 
boot.n = 2000, boot.stratified = TRUE, reuse.auc=TRUE, 
progress = getOption("pROCProgress")$name, ...)
## S3 method for class 'smooth.roc':
var.roc(smooth.roc, ...)

Arguments

x, roc, smooth.roc, auc a “roc” object from the roc function, a “smooth.roc” object from the smooth.roc function or an “auc” object from the auc function.
method the method to use, either “delong”, “bootstrap” or “obuchowski”. The first letter is sufficient. If omitted, the appropriate method is selected as explained in details.
reuse.auc if TRUE (default) and the “roc” objects contain an “auc” field, re-use these specifications for the test. See details.
boot.n for method="bootstrap" only: the number of bootstrap replicates or permutations. Default: 2000.
boot.stratified for method="bootstrap" only: should the bootstrap be stratified (same number of cases/controls in each replicate than in the original sample) or not. Default: TRUE.
progress the name of progress bar to display. Typically “none”, “win”, “tk” or “text” (see the name argument to create_progress_bar for more information), but a list as returned by create_progress_bar is also accepted. See also the “Progress bars” section of this package's documentation.
... further arguments passed to or from other methods, especially arguments for var.roc when calling var, var.roc.auc and var.roc.smooth.roc. Arguments for auc (if reuse.auc=FALSE) and txtProgressBar (only char and style) if applicable.

Details

The var.roc function computes the variance of the AUC of a ROC curve. It is typically called with the roc object of interest. Three methods are available: “delong”, “bootstrap” and “obuchowski” (see “Computational details” section below).

The default is to use “delong” method except for with partial AUC and smoothed curves where “bootstrap” is employed. Using “delong” for partial AUC and smoothed ROCs is not supported (a warning is produced and “bootstrap” is employed instead). “obuchowski” can be used on partial AUCs only if they are defined as a region of specificity (AUC generated with partial.auc.region="specificity"). Requesting the variance to be computed with method="obuchowski" on an AUC with partial.auc.region="specificity" produces a warning and “bootstrap” is employed instead.

For smoothed ROC curves, smoothing is performed again at each bootstrap replicate with the parameters originally provided. If a density smoothing was performed with user-provided density.cases or density.controls the bootstrap cannot be performed and an error is issued.

var.default forces the usage of the splus::var function in the splus package, so that other code relying on var should continue to function normally.

Value

The numeric value of the variance.

AUC specification

var.roc needs a specification of the AUC to compute the variance of the AUC of the ROC curve. The specification is defined by:

  1. the “auc” field in the “roc” objects if reuse.auc is set to TRUE (default)
  2. passing the specification to auc with ... (arguments partial.auc, partial.auc.correct and partial.auc.focus). In this case, you must ensure either that the roc object do not contain an auc field (if you called roc with auc=FALSE), or set reuse.auc=FALSE.

If reuse.auc=FALSE the auc function will always be called with ... to determine the specification, even if the “roc” objects do contain an auc field.

As well if the “roc” objects do not contain an auc field, the auc function will always be called with ... to determine the specification.

Warning: if the roc object passed to roc.test contains an auc field and reuse.auc=TRUE, auc is not called and arguments such as partial.auc are silently ignored.

Computation details

With method="bootstrap", the processing is done as follow:

  1. boot.n bootstrap replicates are drawn from the data. If boot.stratified is TRUE, each replicate contains exactly the same number of controls and cases than the original sample, otherwise if FALSE the numbers can vary.
  2. for each bootstrap replicate, the AUC of the ROC curve is computed and stored.
  3. the variance of the resampled AUCs are computed and returned.

With method="delong", the processing is done as described in Hanley and Hajian-Tilaki (1997).

With method="obuchowski", the processing is done as described in Obuchowski and McClish (1997), Table 1 and Equation 4, p. 1530–1531. The computation of g for partial area under the ROC curve is modified as:

expr1 * (2 * pi * expr2) ^ (-1) * (-expr4) - A * B * expr1 * (2 * pi * expr2^3) ^ (-1/2) * expr3

.

Binormality assumption

The “obuchowski” method makes the assumption that the data is binormal. If the data shows a deviation from this assumption, it might help to normalize the data first (that is, before calling roc), for example with quantile normalization:

    norm.x <- qnorm(rank(x)/(length(x)+1))
    var(roc(response, norm.x, ...), ...)

“delong” and “bootstrap” methods make no such assumption.

Warnings

If method="delong" and the AUC specification specifies a partial AUC, the warning “Using DeLong for partial AUC is not supported. Using bootstrap test instead.” is issued. The method argument is ignored and “bootstrap” is used instead.

If method="delong" or method="obuchowski" and the ROC curve is smoothed, the warning “Using DeLong/Obuchowski for smoothed ROCs is not supported. Using bootstrap test instead.” is issued. The method argument is ignored and “bootstrap” is used instead.

When method="obuchowski" is requested on a partial AUC with method="obuchowski" produces the warning “Using Obuchowski for smoothed ROCs is not supported. Using bootstrap instead.” The method argument is ignored and “bootstrap” is used instead.

If boot.stratified=FALSE and the sample has a large imbalance between cases and controls, it could happen that one or more of the replicates contains no case or control observation, or that there are not enough points for smoothing, producing a NA area. The warning “NA value(s) produced during bootstrap were ignored.” will be issued and the observation will be ignored. If you have a large imbalance in your sample, it could be safer to keep boot.stratified=TRUE.

Errors

If density.cases and density.controls were provided for smoothing, the error “Cannot compute the covariance on ROC curves smoothed with density.controls and density.cases.” is issued.

References

Elisabeth R. DeLong, David M. DeLong and Daniel L. Clarke-Pearson (1988) ``Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach''. Biometrics 44, 837–845.

James A. Hanley and Karim O. Hajian-Tilaki (1997) ``Sampling variability of nonparametric estimates of the areas under receiver operating characteristic curves: An update''. Academic Radiology 4, 49–58. DOI: href{http://dx.doi.org/10.1016/S1076-6332(97)80161-4}{10.1016/S1076-6332(97)80161-4}.

Nancy A. Obuchowski, Donna K. McClish (1997). ``Sample size determination for diagnostic accurary studies involving binormal ROC curve indices''. Statistics in Medicine, 16(13), 1529–1542. DOI: href{http://dx.doi.org/10.1002/(SICI)1097-0258(19970715)16:13<1529::AID-SIM565>3.0.CO;2-H}{(SICI)1097-0258(19970715)16:13<1529::AID-SIM565>3.0.CO;2-H}.

See Also

roc, cov.roc

Examples

data(aSAH)

##  Basic example
roc1 <- roc(aSAH$outcome, aSAH$s100b)
roc2 <- roc(aSAH$outcome, aSAH$wfns)
var.roc(roc1)
var.roc(roc2)

# We could also write it in one line:
var.roc(roc(aSAH$outcome, aSAH$s100b))

### Not run: 
# The latter used Delong. To use bootstrap:
var.roc(roc1, method="bootstrap")
# Decrease boot.n for a faster execution
var.roc(roc1,method="bootstrap", boot.n=1000)
#
## End(Not run)

# To use obuchowski:
var.roc(roc1, method="obuchowski")

### Not run: 
# Variance of smoothed ROCs:
# Smoothing is re-done at each iteration, and execution is slow
var.roc(smooth(roc1))
#
## End(Not run)

# or from an AUC (no smoothing)
var.roc(auc(roc1))

## Test data from Hanley and Hajian-Tilaki, 1997
disease.present <- c("Yes", "No", "Yes", "No", "No", "Yes", "Yes", "No", "No", "Yes", "No", "No", "Yes", "No", "No")
field.strength.1 <- c(1, 2, 5, 1, 1, 1, 2, 1, 2, 2, 1, 1, 5, 1, 1)
field.strength.2 <- c(1, 1, 5, 1, 1, 1, 4, 1, 2, 2, 1, 1, 5, 1, 1)
roc3 <- roc(disease.present, field.strength.1)
roc4 <- roc(disease.present, field.strength.2)
# Assess the variance:
var.roc(roc3)
var.roc(roc4)

### Not run: 
# With bootstrap:
var.roc(roc3, method="bootstrap")
var.roc(roc4, method="bootstrap")
#
## End(Not run)


[Package pROC version 1.4.9 Index]