pROC is a set of tools to visualize, smooth and compare receiver operating characteristic (ROC curves). (Partial) area under the curve (AUC) can be compared with statistical tests based on U-statistics or bootstrap. Confidence intervals can be computed for (p)AUC or ROC curves.

More screenshots and examples…

If you use pROC in published research, please cite the following paper:

Xavier Robin, Natacha Turck, Alexandre Hainard, Natalia Tiberti, Frédérique Lisacek, Jean-Charles Sanchez and Markus Müller (2011). pROC: an open-source package for R and S+ to analyze and compare ROC curves. *BMC Bioinformatics*, **12**, p. 77. DOI: 10.1186/1471-2105-12-77.

- Authors
- Xavier Robin, Natacha Turck, Alexandre Hainard, Natalia Tiberti, Frédérique Lisacek, Jean-Charles Sanchez and Markus Müller
- Contact
- Xavier Robin
- License
- GPLv3

pROC comes in two flavours: a command line version for the R statistical software environment, and a graphical user interface (GUI) for S+.

## R |
## S+ |
---|---|

This version is intended to be employed through the R command line. - Version: 1.7.2 (released on April 6th, 2014)
- Download pROC from CRAN
- Download R
- Follow pROC's development on GitHub
- Direct downloads:
## InstallationThere is no need to download the package. The installation can be done in one command directly from R: install.packages("pROC") The package must then be loaded with: library(pROC) To get help, enter the following in the R prompt: ?pROC ## UpdateAny of the following commands will update pROC if possible: install.packages("pROC") update.packages() # will install updates to all your packages |
This version is intended to be employed with the TIBCO Spotfire S+ program. It has a graphical user interface (GUI) for S+ 8.2 32bits for Windows, but the command line is also available. - Version: 1.4.9 (released on January 4th, 2014)
- Download from CSAN
- Get Spotfire S+ from TIBCO (proprietary)
- Direct downloads:
## Deprecation warningThe S+ version is not developed any longer (due to diverging code bases and apparent drop of support of S+ by TIBCO) but still contains the latest main features of pROC (especially power tests). The GUI is available only on the 32 bits version of S+ 8.2 for Windows (no Linux / Mac / 64 bits support). ## InstallationIn the S+ command prompt (in Windows, you can open it from S+ install.pkgutils() Save the installation file (binary for Windows, source for Linux) in your working directory and then type: install.packages("pROC_1.4.9_S_WIN386.zip") # Or for the source version: After loading from the library(pROC) More details about installation… ## UpdateRepeat the installation steps described above, except the |

- 1.7.2 (2014-04-05) (R, release notes)
- Fixed bug where
`ci.coords`

with`x="best"`

would fail if one or more resampled ROC curve had multiple "best" thresholds (thanks Berend Terluin for the report) - Fixed bug in
`ci.coords`

: passing more than one value in`x`

now works - Fixed typo in documentation of
`direction`

argument to`roc`

(thanks Le Kang for the report) - Add a warning when computing statistics of ROC curve with AUC = 1
- Require latest version of Rcpp to avoid weird errors (thanks Tom Liptrot for the report)
- 1.7.1 (2014-02-20) (R, release notes)
- Close SOCK cluster on Windows with parallel=TRUE
- Fixed really use algorithm 1 when microbenchmark fails
- 1.7 (2014-02-19) (R, release notes)
- Faster algorithm for DeLong
`roc.test`

,`power.roc.test`

,`ci.auc`

,`var`

and`cov`

function (no large matrix allocation) - Handling Math and Operations correctly on
`auc`

and`ci`

objects (see`?groupGeneric.pROC`

) - The
`formula`

for`roc.formula`

can now provide several predictors and a list of ROC curves will be returned - Fixed documentation of
`ci.coords`

with examples - Fixed binormal AUC computed with triangulation despite the claim in the documentation
- Fixed unstated requirement on Rcpp >= 0.10.5
- 1.6.0.1 (2013-12-28) (R, release notes)
- Removed erroneous error message displayed when predictors and responses were not vectors.
- 1.6 (2013-12-26) (R, release notes)
- New
`power.roc.test`

function for sample size and power computations. - New
`cov`

and`var`

functions supports new`"obuchowski"`

method. - New
`ci.coords`

function to compute CI of arbitrary coords. `coords`

accepts new`ret`

value "1-accuracy".- Introducing various algorithms to compute sensitivities and specificites, with a more vectorized code or Rcpp. See 'algorithm' in
`?roc`

for more details on the trade-offs of the different methods. - Faster algorithm for DeLong
`roc.test`

,`ci`

,`var`

and cov function (thanks Kazuki Yoshida). `are.paired`

now also checks for identical`levels`

.- Fixed a warning generated in the examples.
- Fixed several bugs related with
`smooth.roc`

curves. - Additional input data sanity checks.
- Now requires R ≥ 2.13 (in fact, since 1.5.1, thanks Emmanuel Curis for the report).
- Progress bars now defaults to text on Macs where
`tcltk`

seems broken (thanks Gerard Smits for the report). - 1.5.4 (2012-08-31) (R)
- Running less smooth.roc examples with logcondens because they take too much time (requested by Uwe Ligges).
- 1.5.3 (2011-08-31) (R, release notes)
- AUC specification was lost when
`roc.test`

,`cov`

or`var`

was passed an`auc`

' object. - Correct computation of "accuracy" in
`coords`

(thanks to Kosuke Yoshihara for the report). - 1.5.1 (2011-03-09) (R, release notes)
- Faster loading of the package (thanks to Prof Brian Ripley and Glenn Lawyer for the report).
- 1.5 (2011-12-12) (R, release notes)
- New
`cov`

and`var`

functions. `coords`

accepts new`ret`

values: "accuracy", "tn", "tp", "fn", "fp", "npv", "ppv", "1-specificity", "1-sensitivity", "1-npv", "1-ppv", "npe" and "ppe".- New
`legacy.axes`

argument to`plot`

1-specificity rather than specificity. - New
`axes`

argument to turn off the plotting of the axis. - New
`logcondens`

and`logcondens.smooth`

(Univariate Log-Concave Density Estimation) smoothing methods. - New function
`has.partial.auc`

to determine if an AUC is full or partial. - New argument
`drop`

for`coords`

. `auc`

and`multiclass.auc`

objects now also have secondary class`numeric`

.- Updated load call.
- Delong's CI reversed in ROC curves with
`direction=">"`

. - Delong's CI AUC returned values > 1 or < 0 in some rare cases.
- Minor improvements in documentation.
- 1.4.9 (2014-01-04) (S+ only, release notes)
- This is the last version of pROC for S+. It introduces some features of pROC 1.5 (covariance and variance functions, some bugfixes) and 1.6 (power tests) for R into S+, and will not be maintained any more.
- Fixed custom output report.
- New
`power.roc.test`

function for sample size and power computations. - New
`cov`

and`var`

functions. `auc`

objects now also have secondary class`numeric`

.- Delong's CI reversed in ROC curves with
`direction=">"`

- Delong's CI AUC returned values > 1 or < 0 in some rare cases
- Delong's CI AUC returned NA values when AUC was 1. Now reporting 1-1 interval.
- Minor improvements in documentation.
- 1.4.4 (2011-08-10) (R, S+, release notes)
- Fixed alternative for one-tailed tests.
- Removed COPYING file to fix a warning in r-devel.
- 1.4.3 (2011-03-18) (R, S+, release notes)
- Updated citation.
- 1.4.2 (2011-03-03) (R, S+, release notes)
- Fixed bootstrap
`roc.test`

generating NAs when`smooth.roc`

s were used with`reuse.auc=FALSE`

(thanks to Buddy for the report). - Documented a warning that was missing in
`roc.test`

. - Updated citation.
- 1.4.1 (2011-01-27) (R, S+, release notes)
- Venkatraman's test for unpaired ROC curves.
- 1.4 (2011-01-21) (R, S+, release notes)
- 'smooth' does not apply on
`ordered`

factors anymore. - Multi-class AUC support (R only).
- Can choose how
`best`

thresold is determined (`best.method`

and`best.weights`

in`coords`

and`print.thres.best.method`

and`print.thres.best.weights`

in`plot.roc`

). - Minor fixes in documentation (R and S+) and
`citation`

(S+ only). `print`

now prints the response instead of*response*and more informative data in`htest`

s.- Bootstrap with
`ci.auc`

consumes much less memory. - Unpaired bootstrap and DeLong's test.
- Specificity and sensitivity tests (in
`roc.test`

). - 1.3.2 (2010-08-24) (R, S+, release notes)
`print.auc`

printed incorrect CI in`plot.roc`

(thanks to Alexander B. Leichtle for the report).- Failed to detect local maximas in
`coords`

when 2 or less points were selected. - Don't consider ROC extremities (+-Inf at 1.0/0.0 SE<->SP) as local maximas.
- 1.3.1 (2010-08-18) (R, release notes)
- Sensitivity and specificity were inverted in coords when results were reported as list.
- Faster checks with
`\dontrun{}`

in`roc.test`

. - 1.3 (2010-08-13) (R, S+, release notes)
- CI is not re-computed by default in
`smooth.roc`

. You can still turn it on with`reuse.ci=TRUE`

. - New function
`are.paired`

. - Local maximas could be incorrectly detected in
`coords`

(and`plot.roc`

) with`predictor`

s containing more than 2 levels. - New method
`venkatraman`

for`roc.test`

. - MASS and tcltk packages are now only suggested instead of required (R only).
`...`

not passed correctly in`plot.ci.se`

with`type="bars"`

resulting in an error (R only).- 1.2.1 (2010-05-11) (R, S+, release notes)
- Handle
`method`

arguments for`smooth.roc`

and`ci.auc`

separately in`roc.default`

(R-only). - Added
`auc.polygon.*`

and`max.auc.polygon.*`

arguments for`polygon`

in`plot.roc`

. - 1.2 (2010-05-09) (R, release notes)
- Added DeLong method in
`ci.auc`

(with a GUI, S+ only). - Return value of ci.auc does not contain an
`"aucs"`

item anymore. - Put most examples with bootstrap within
`\dontrun{}`

blocks for faster (but less useful) checks execution. - 1.1 (2010-05-05) (R, S+, release notes)
- Added
`lines.roc`

functions for ROC. - Added
`type`

argument for both`lines.roc`

and`plot.roc`

. - Added
`print.auc.col`

argument to`plot.roc`

. - Fixed a warning in
`roc.test.default`

when the class of`predictor1`

had several elements. - Fixed an encoding failure during the checks on MacOS X (R only).
- 1.0.1 (2010-04-28) (R, release notes)
- Reduced examples execution time. Added low
`boot.n`

in the slowest examples and`reuse.auc`

and`reuse.ci`

arguments in`smooth.roc.roc`

. - 1.0 (2010-04-27) (R, S+, release notes)
- First public release.

Please report any bug you may encounter to Xavier Robin.

- James Carpenter and John Bithell (2000) “Bootstrap condence intervals: when, which, what? A practical guide for medical statisticians”.
*Statistics in Medicine***19**, 1141–1164. DOI: 10.1002/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO;2-F; PMID: 10797513. - Elisabeth R. DeLong, David M. DeLong and Daniel L. Clarke-Pearson (1988) “Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach”.
*Biometrics***44**, 837–845. JSTOR: 2531595; PMID: 3203132. - Lutz Duembgen, Kaspar Rufibach (2011) “logcondens: Computations Related to Univariate Log-Concave Density Estimation”.
*Journal of Statistical Software*,**39**, 1–28. URL: jstatsoft.org/v39/i06. - Tom Fawcett (2006) “An introduction to ROC analysis”.
*Pattern Recognition Letters***27**, 861–874. DOI: j.patrec.2005.10.010. - James E. Hanley (1988) “The robustness of the “binormal” assumptions used in fitting ROC curves”.
*Medical Decision Making***8**, 197–203. DOI: 10.1177/0272989X8800800308; PMID: 3398748. - Donna Katzman McClish (1989) “Analyzing a Portion of the ROC Curve”.
*Medical Decision Making***9**, 190–195. DOI: 10.1177/0272989X8900900307; PMID: 2668680. - Nancy A. Obuchowski and Donna K. McClish (1997) “Sample size determination for diagnostic accurary studies involving binormal ROC curve indices”.
*Statistics in Medicine*,**16**, 1529–1542. DOI: 10.1002/(SICI)1097-0258(19970715)16:13<1529::AID-SIM565>3.0.CO;2-H. - Nancy A. Obuchowski, Micharl L. Lieber and Frank H. Wians Jr. (2004) “ROC Curves in Clinical Chemistry: Uses, Misuses, and Possible Solutions”.
*Clinical Chemistry*,**50**, 1118–1125. DOI: 10.1373/clinchem.2004.031823. - Margaret Pepe, Gary Longton and Holly Janes (2009) “Estimation and Comparison of Receiver Operating Characteristic Curves”.
*The Stata journal***9**, 1. PMID: 20161343. - Xavier Robin, Natacha Turck
*et al.*(2011) “pROC: an open-source package for R and S+ to analyze and compare ROC curves”.*BMC Bioinformatics*,**12**, 77. DOI: 10.1186/1471-2105-12-77. - E. S. Venkatraman and Colin B. Begg (1996) “A distribution-free procedure for comparing receiver operating characteristic curves from a paired experiment”.
*Biometrika***83**, 835–848. DOI: 10.1093/biomet/83.4.835 - E. S. Venkatraman (2000) “A Permutation Test to Compare Receiver Operating Characteristic Curves”.
*Biometrics***56**, 1134–1138. DOI: 10.1111/j.0006-341X.2000.01134.x. - Kelly H. Zou, W. J. Hall and David E. Shapiro (1997) “Smooth non-parametric receiver operating characteristic (ROC) curves for continuous diagnostic tests”.
*Statistics in Medicine***18**, 2143–2156. DOI: 10.1002/(SICI)1097-0258(19971015)16:19<2143::AID-SIM655>3.0.CO;2-3; PMID: 9330425 .