pROC: display and analyze ROC curves in R and S+

pROC is a set of tools to visualize, smooth and compare receiver operating characteristic (ROC curves). (Partial) area under the curve (AUC) can be compared with statistical tests based on U-statistics or bootstrap. Confidence intervals can be computed for (p)AUC or ROC curves.

More screenshots and examples…

Find us on:

If you use pROC in published research, please cite the following paper:

Xavier Robin, Natacha Turck, Alexandre Hainard, Natalia Tiberti, Frédérique Lisacek, Jean-Charles Sanchez and Markus Müller (2011). pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics, 12, p. 77. DOI: 10.1186/1471-2105-12-77.

Authors

Xavier Robin, Natacha Turck, Alexandre Hainard, Natalia Tiberti, Frédérique Lisacek, Jean-Charles Sanchez and Markus Müller

Contact

Xavier Robin

License

GPLv3

pROC comes in two flavours: a command line version for the R statistical software environment, and a graphical user interface (GUI) for S+.

install.packages("pROC")

library(pROC)

?pROC

install.packages("pROC")

update.packages() # will install updates to all your packages

install.pkgutils()
library(pkgutils)

install.packages("pROC_1.4.9_S_WIN386.zip") # Or for the source version:
install.packages("pROC_1.4.9_S_source.zip")

library(pROC)

Versions

1.7.2 (2014-04-05) (R, release notes)

Fixed bug where ci.coords with x="best" would fail if one or more resampled ROC curve had multiple "best" thresholds (thanks Berend Terluin for the report)

Fixed bug in ci.coords: passing more than one value in x now works

Fixed typo in documentation of direction argument to roc (thanks Le Kang for the report)

Add a warning when computing statistics of ROC curve with AUC = 1

Require latest version of Rcpp to avoid weird errors (thanks Tom Liptrot for the report)

1.7.1 (2014-02-20) (R, release notes)

Close SOCK cluster on Windows with parallel=TRUE

Fixed really use algorithm 1 when microbenchmark fails

1.7 (2014-02-19) (R, release notes)

Faster algorithm for DeLong roc.test, power.roc.test, ci.auc, var and cov function (no large matrix allocation)

Handling Math and Operations correctly on auc and ci objects (see ?groupGeneric.pROC)

The formula for roc.formula can now provide several predictors and a list of ROC curves will be returned

Fixed documentation of ci.coords with examples

Fixed binormal AUC computed with triangulation despite the claim in the documentation

Fixed unstated requirement on Rcpp >= 0.10.5

1.6.0.1 (2013-12-28) (R, release notes)

Removed erroneous error message displayed when predictors and responses were not vectors.

1.6 (2013-12-26) (R, release notes)

New power.roc.test function for sample size and power computations.

New cov and var functions supports new "obuchowski" method.

New ci.coords function to compute CI of arbitrary coords.

coords accepts new ret value "1-accuracy".

Introducing various algorithms to compute sensitivities and specificites, with a more vectorized code or Rcpp. See 'algorithm' in ?roc for more details on the trade-offs of the different methods.

Faster algorithm for DeLong roc.test, ci, var and cov function (thanks Kazuki Yoshida).

are.paired now also checks for identical levels.

Fixed a warning generated in the examples.

Fixed several bugs related with smooth.roc curves.

Additional input data sanity checks.

Now requires R ≥ 2.13 (in fact, since 1.5.1, thanks Emmanuel Curis for the report).

Progress bars now defaults to text on Macs where tcltk seems broken (thanks Gerard Smits for the report).

1.5.4 (2012-08-31) (R)

Running less smooth.roc examples with logcondens because they take too much time (requested by Uwe Ligges).

1.5.3 (2011-08-31) (R, release notes)

AUC specification was lost when roc.test, cov or var was passed an auc' object.

Correct computation of "accuracy" in coords (thanks to Kosuke Yoshihara for the report).

1.5.1 (2011-03-09) (R, release notes)

Faster loading of the package (thanks to Prof Brian Ripley and Glenn Lawyer for the report).

1.5 (2011-12-12) (R, release notes)

New cov and var functions.

coords accepts new ret values: "accuracy", "tn", "tp", "fn", "fp", "npv", "ppv", "1-specificity", "1-sensitivity", "1-npv", "1-ppv", "npe" and "ppe".

New legacy.axes argument to plot 1-specificity rather than specificity.

New axes argument to turn off the plotting of the axis.

New logcondens and logcondens.smooth (Univariate Log-Concave Density Estimation) smoothing methods.

New function has.partial.auc to determine if an AUC is full or partial.

New argument drop for coords.

auc and multiclass.auc objects now also have secondary class numeric.

Updated load call.

Delong's CI reversed in ROC curves with direction=">".

Delong's CI AUC returned values > 1 or < 0 in some rare cases.

Minor improvements in documentation.

1.4.9 (2014-01-04) (S+ only, release notes)

This is the last version of pROC for S+. It introduces some features of pROC 1.5 (covariance and variance functions, some bugfixes) and 1.6 (power tests) for R into S+, and will not be maintained any more.

Fixed custom output report.

New power.roc.test function for sample size and power computations.

New cov and var functions.

auc objects now also have secondary class numeric.

Delong's CI reversed in ROC curves with direction=">"

Delong's CI AUC returned values > 1 or < 0 in some rare cases

Delong's CI AUC returned NA values when AUC was 1. Now reporting 1-1 interval.

Minor improvements in documentation.

1.4.4 (2011-08-10) (R, S+, release notes)

Fixed alternative for one-tailed tests.

Removed COPYING file to fix a warning in r-devel.

1.4.3 (2011-03-18) (R, S+, release notes)

Updated citation.

1.4.2 (2011-03-03) (R, S+, release notes)

Fixed bootstrap roc.test generating NAs when smooth.rocs were used with reuse.auc=FALSE (thanks to Buddy for the report).

Documented a warning that was missing in roc.test.

Updated citation.

1.4.1 (2011-01-27) (R, S+, release notes)

Venkatraman's test for unpaired ROC curves.

1.4 (2011-01-21) (R, S+, release notes)

'smooth' does not apply on ordered factors anymore.

Multi-class AUC support (R only).

Can choose how best thresold is determined (best.method and best.weights in coords and print.thres.best.method and print.thres.best.weights in plot.roc).

Minor fixes in documentation (R and S+) and citation (S+ only).

print now prints the response instead of response and more informative data in htests.

Bootstrap with ci.auc consumes much less memory.

Unpaired bootstrap and DeLong's test.

Specificity and sensitivity tests (in roc.test).

1.3.2 (2010-08-24) (R, S+, release notes)

print.auc printed incorrect CI in plot.roc (thanks to Alexander B. Leichtle for the report).

Failed to detect local maximas in coords when 2 or less points were selected.

Don't consider ROC extremities (+-Inf at 1.0/0.0 SE<->SP) as local maximas.

1.3.1 (2010-08-18) (R, release notes)

Sensitivity and specificity were inverted in coords when results were reported as list.

Faster checks with \dontrun{} in roc.test.

1.3 (2010-08-13) (R, S+, release notes)

CI is not re-computed by default in smooth.roc. You can still turn it on with reuse.ci=TRUE.

New function are.paired.

Local maximas could be incorrectly detected in coords (and plot.roc) with predictors containing more than 2 levels.

New method venkatraman for roc.test.

MASS and tcltk packages are now only suggested instead of required (R only).

... not passed correctly in plot.ci.se with type="bars" resulting in an error (R only).

1.2.1 (2010-05-11) (R, S+, release notes)

Handle method arguments for smooth.roc and ci.auc separately in roc.default (R-only).

Added auc.polygon.* and max.auc.polygon.* arguments for polygon in plot.roc.

1.2 (2010-05-09) (R, release notes)

Added DeLong method in ci.auc (with a GUI, S+ only).

Return value of ci.auc does not contain an "aucs" item anymore.

Put most examples with bootstrap within \dontrun{} blocks for faster (but less useful) checks execution.

1.1 (2010-05-05) (R, S+, release notes)

Added lines.roc functions for ROC.

Added type argument for both lines.roc and plot.roc.

Added print.auc.col argument to plot.roc.

Fixed a warning in roc.test.default when the class of predictor1 had several elements.

Fixed an encoding failure during the checks on MacOS X (R only).

1.0.1 (2010-04-28) (R, release notes)

Reduced examples execution time. Added low boot.n in the slowest examples and reuse.auc and reuse.ci arguments in smooth.roc.roc.

1.0 (2010-04-27) (R, S+, release notes)

First public release.

Known bugs

Please report any bug you may encounter to Xavier Robin.

References

• James Carpenter and John Bithell (2000) “Bootstrap condence intervals: when, which, what? A practical guide for medical statisticians”. Statistics in Medicine 19, 1141–1164. DOI: 10.1002/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO;2-F; PMID: 10797513.
• Elisabeth R. DeLong, David M. DeLong and Daniel L. Clarke-Pearson (1988) “Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach”. Biometrics 44, 837–845. JSTOR: 2531595; PMID: 3203132.
• Lutz Duembgen, Kaspar Rufibach (2011) “logcondens: Computations Related to Univariate Log-Concave Density Estimation”. Journal of Statistical Software, 39, 1–28. URL: jstatsoft.org/v39/i06.
• Tom Fawcett (2006) “An introduction to ROC analysis”. Pattern Recognition Letters 27, 861–874. DOI: j.patrec.2005.10.010.
• James E. Hanley (1988) “The robustness of the “binormal” assumptions used in fitting ROC curves”. Medical Decision Making 8, 197–203. DOI: 10.1177/0272989X8800800308; PMID: 3398748.
• Donna Katzman McClish (1989) “Analyzing a Portion of the ROC Curve”. Medical Decision Making 9, 190–195. DOI: 10.1177/0272989X8900900307; PMID: 2668680.
• Nancy A. Obuchowski and Donna K. McClish (1997) “Sample size determination for diagnostic accurary studies involving binormal ROC curve indices”. Statistics in Medicine, 16, 1529–1542. DOI: 10.1002/(SICI)1097-0258(19970715)16:13<1529::AID-SIM565>3.0.CO;2-H.
• Nancy A. Obuchowski, Micharl L. Lieber and Frank H. Wians Jr. (2004) “ROC Curves in Clinical Chemistry: Uses, Misuses, and Possible Solutions”. Clinical Chemistry, 50, 1118–1125. DOI: 10.1373/clinchem.2004.031823.
• Margaret Pepe, Gary Longton and Holly Janes (2009) “Estimation and Comparison of Receiver Operating Characteristic Curves”. The Stata journal 9, 1. PMID: 20161343.
• Xavier Robin, Natacha Turck et al. (2011) “pROC: an open-source package for R and S+ to analyze and compare ROC curves”. BMC Bioinformatics, 12, 77. DOI: 10.1186/1471-2105-12-77.
• E. S. Venkatraman and Colin B. Begg (1996) “A distribution-free procedure for comparing receiver operating characteristic curves from a paired experiment”. Biometrika 83, 835–848. DOI: 10.1093/biomet/83.4.835
• E. S. Venkatraman (2000) “A Permutation Test to Compare Receiver Operating Characteristic Curves”. Biometrics 56, 1134–1138. DOI: 10.1111/j.0006-341X.2000.01134.x.
• Kelly H. Zou, W. J. Hall and David E. Shapiro (1997) “Smooth non-parametric receiver operating characteristic (ROC) curves for continuous diagnostic tests”. Statistics in Medicine 18, 2143–2156. DOI: 10.1002/(SICI)1097-0258(19971015)16:19<2143::AID-SIM655>3.0.CO;2-3; PMID: 9330425 .