pROC: display and analyze ROC curves in R and S+
pROC is a set of tools to visualize, smooth and compare receiver operating characteristic (ROC curves). (Partial) area under the curve (AUC) can be compared with statistical tests based on Ustatistics or bootstrap. Confidence intervals can be computed for (p)AUC or ROC curves.
More screenshots and examples…
If you use pROC in published research, please cite the following paper:
Xavier Robin, Natacha Turck, Alexandre Hainard, Natalia Tiberti, Frédérique Lisacek, JeanCharles Sanchez and Markus Müller (2011). pROC: an opensource package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics, 12, p. 77. DOI: 10.1186/147121051277. Authors
 Xavier Robin, Natacha Turck, Alexandre Hainard, Natalia Tiberti, Frédérique Lisacek, JeanCharles Sanchez and Markus Müller
 Contact
 Xavier Robin
 License
 GPLv3
Find us on:
pROC comes in two flavours: a command line version for the R statistical software environment, and a graphical user
interface (GUI) for S+.
R 
S+ 

This version is intended to be employed through the R command line.
InstallationThere is no need to download the package. The installation can be done in one command directly from R: install.packages("pROC") The package must then be loaded with: library(pROC) To get help, enter the following in the R prompt: ?pROC UpdateAny of the following commands will update pROC if possible: install.packages("pROC") update.packages() # will install updates to all your packages 
This version is intended to be employed with the TIBCO Spotfire S+ program. It has a graphical user interface (GUI) for S+ 8.2 32bits for Windows, but the command line is also available.
Deprecation warningThe S+ version is not developed any longer (due to diverging code bases and apparent drop of support of S+ by TIBCO) but still contains the latest main features of pROC (especially power tests). The GUI is available only on the 32 bits version of S+ 8.2 for Windows (no Linux / Mac / 64 bits support). InstallationIn the S+ command prompt (in Windows, you can open it from S+ Window menu, Commands Window item), type:
install.pkgutils() Save the installation file (binary for Windows, source for Linux) in your working directory and then type: install.packages("pROC_1.4.9_S_WIN386.zip") # Or for the source version: After loading from the File menu, item Load Library… , a new
ROC curves menu is available in the Statistics menu. You can
also load it with the command
library(pROC) More details about installation… UpdateRepeat the installation steps described above, except the install.pkgutils() command.
More details about update… 
Versions
 1.7.2 (20140405) (R, release notes)

Fixed bug where
ci.coords
withx="best"
would fail if one or more resampled ROC curve had multiple "best" thresholds (thanks Berend Terluin for the report) 
Fixed bug in
ci.coords
: passing more than one value inx
now works 
Fixed typo in documentation of
direction
argument toroc
(thanks Le Kang for the report)  Add a warning when computing statistics of ROC curve with AUC = 1
 Require latest version of Rcpp to avoid weird errors (thanks Tom Liptrot for the report)
 1.7.1 (20140220) (R, release notes)
 Close SOCK cluster on Windows with parallel=TRUE
 Fixed really use algorithm 1 when microbenchmark fails
 1.7 (20140219) (R, release notes)

Faster algorithm for DeLong
roc.test
,power.roc.test
,ci.auc
,var
andcov
function (no large matrix allocation) 
Handling Math and Operations correctly on
auc
andci
objects (see?groupGeneric.pROC
) 
The
formula
forroc.formula
can now provide several predictors and a list of ROC curves will be returned 
Fixed documentation of
ci.coords
with examples  Fixed binormal AUC computed with triangulation despite the claim in the documentation
 Fixed unstated requirement on Rcpp >= 0.10.5
 1.6.0.1 (20131228) (R, release notes)
 Removed erroneous error message displayed when predictors and responses were not vectors.
 1.6 (20131226) (R, release notes)

New
power.roc.test
function for sample size and power computations. 
New
cov
andvar
functions supports new"obuchowski"
method. 
New
ci.coords
function to compute CI of arbitrary coords. 
coords
accepts newret
value "1accuracy". 
Introducing various algorithms to compute sensitivities and specificites, with a more vectorized code or Rcpp. See 'algorithm' in
?roc
for more details on the tradeoffs of the different methods. 
Faster algorithm for DeLong
roc.test
,ci
,var
and cov function (thanks Kazuki Yoshida). 
are.paired
now also checks for identicallevels
.  Fixed a warning generated in the examples.

Fixed several bugs related with
smooth.roc
curves.  Additional input data sanity checks.
 Now requires R ≥ 2.13 (in fact, since 1.5.1, thanks Emmanuel Curis for the report).

Progress bars now defaults to text on Macs where
tcltk
seems broken (thanks Gerard Smits for the report).  1.5.4 (20120831) (R)
 Running less smooth.roc examples with logcondens because they take too much time (requested by Uwe Ligges).
 1.5.3 (20110831) (R, release notes)

AUC specification was lost when
roc.test
,cov
orvar
was passed anauc
' object. 
Correct computation of "accuracy" in
coords
(thanks to Kosuke Yoshihara for the report).  1.5.1 (20110309) (R, release notes)
 Faster loading of the package (thanks to Prof Brian Ripley and Glenn Lawyer for the report).
 1.5 (20111212) (R, release notes)

New
cov
andvar
functions. 
coords
accepts newret
values: "accuracy", "tn", "tp", "fn", "fp", "npv", "ppv", "1specificity", "1sensitivity", "1npv", "1ppv", "npe" and "ppe". 
New
legacy.axes
argument toplot
1specificity rather than specificity. 
New
axes
argument to turn off the plotting of the axis. 
New
logcondens
andlogcondens.smooth
(Univariate LogConcave Density Estimation) smoothing methods. 
New function
has.partial.auc
to determine if an AUC is full or partial. 
New argument
drop
forcoords
. 
auc
andmulticlass.auc
objects now also have secondary classnumeric
.  Updated load call.

Delong's CI reversed in ROC curves with
direction=">"
.  Delong's CI AUC returned values > 1 or < 0 in some rare cases.
 Minor improvements in documentation.
 1.4.9 (20140104) (S+ only, release notes)
 This is the last version of pROC for S+. It introduces some features of pROC 1.5 (covariance and variance functions, some bugfixes) and 1.6 (power tests) for R into S+, and will not be maintained any more.
 Fixed custom output report.

New
power.roc.test
function for sample size and power computations. 
New
cov
andvar
functions. 
auc
objects now also have secondary classnumeric
. 
Delong's CI reversed in ROC curves with
direction=">"
 Delong's CI AUC returned values > 1 or < 0 in some rare cases
 Delong's CI AUC returned NA values when AUC was 1. Now reporting 11 interval.
 Minor improvements in documentation.
 1.4.4 (20110810) (R, S+, release notes)
 Fixed alternative for onetailed tests.
 Removed COPYING file to fix a warning in rdevel.
 1.4.3 (20110318) (R, S+, release notes)
 Updated citation.
 1.4.2 (20110303) (R, S+, release notes)

Fixed bootstrap
roc.test
generating NAs whensmooth.roc
s were used withreuse.auc=FALSE
(thanks to Buddy for the report). 
Documented a warning that was missing in
roc.test
.  Updated citation.
 1.4.1 (20110127) (R, S+, release notes)
 Venkatraman's test for unpaired ROC curves.
 1.4 (20110121) (R, S+, release notes)

'smooth' does not apply on
ordered
factors anymore.  Multiclass AUC support (R only).

Can choose how
best
thresold is determined (best.method
andbest.weights
incoords
andprint.thres.best.method
andprint.thres.best.weights
inplot.roc
). 
Minor fixes in documentation (R and S+) and
citation
(S+ only). 
print
now prints the response instead of response and more informative data inhtest
s. 
Bootstrap with
ci.auc
consumes much less memory.  Unpaired bootstrap and DeLong's test.

Specificity and sensitivity tests (in
roc.test
).  1.3.2 (20100824) (R, S+, release notes)

print.auc
printed incorrect CI inplot.roc
(thanks to Alexander B. Leichtle for the report). 
Failed to detect local maximas in
coords
when 2 or less points were selected.  Don't consider ROC extremities (+Inf at 1.0/0.0 SE<>SP) as local maximas.
 1.3.1 (20100818) (R, release notes)
 Sensitivity and specificity were inverted in coords when results were reported as list.

Faster checks with
\dontrun{}
inroc.test
.  1.3 (20100813) (R, S+, release notes)

CI is not recomputed by default in
smooth.roc
. You can still turn it on withreuse.ci=TRUE
. 
New function
are.paired
. 
Local maximas could be incorrectly detected in
coords
(andplot.roc
) withpredictor
s containing more than 2 levels. 
New method
venkatraman
forroc.test
.  MASS and tcltk packages are now only suggested instead of required (R only).

...
not passed correctly inplot.ci.se
withtype="bars"
resulting in an error (R only).  1.2.1 (20100511) (R, S+, release notes)

Handle
method
arguments forsmooth.roc
andci.auc
separately inroc.default
(Ronly). 
Added
auc.polygon.*
andmax.auc.polygon.*
arguments forpolygon
inplot.roc
.  1.2 (20100509) (R, release notes)

Added DeLong method in
ci.auc
(with a GUI, S+ only). 
Return value of ci.auc does not contain an
"aucs"
item anymore. 
Put most examples with bootstrap within
\dontrun{}
blocks for faster (but less useful) checks execution.  1.1 (20100505) (R, S+, release notes)

Added
lines.roc
functions for ROC. 
Added
type
argument for bothlines.roc
andplot.roc
. 
Added
print.auc.col
argument toplot.roc
. 
Fixed a warning in
roc.test.default
when the class ofpredictor1
had several elements.  Fixed an encoding failure during the checks on MacOS X (R only).
 1.0.1 (20100428) (R, release notes)

Reduced examples execution time. Added low
boot.n
in the slowest examples andreuse.auc
andreuse.ci
arguments insmooth.roc.roc
.  1.0 (20100427) (R, S+, release notes)
 First public release.
Known bugs
Please report any bug you may encounter to Xavier Robin.
References
 James Carpenter and John Bithell (2000) “Bootstrap condence intervals: when, which, what? A practical guide for medical statisticians”. Statistics in Medicine 19, 1141–1164. DOI: 10.1002/(SICI)10970258(20000515)19:9<1141::AIDSIM479>3.0.CO;2F; PMID: 10797513.
 Elisabeth R. DeLong, David M. DeLong and Daniel L. ClarkePearson (1988) “Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach”. Biometrics 44, 837–845. JSTOR: 2531595; PMID: 3203132.
 Lutz Duembgen, Kaspar Rufibach (2011) “logcondens: Computations Related to Univariate LogConcave Density Estimation”. Journal of Statistical Software, 39, 1–28. URL: jstatsoft.org/v39/i06.
 Tom Fawcett (2006) “An introduction to ROC analysis”. Pattern Recognition Letters 27, 861–874. DOI: j.patrec.2005.10.010.
 James E. Hanley (1988) “The robustness of the “binormal” assumptions used in fitting ROC curves”. Medical Decision Making 8, 197–203. DOI: 10.1177/0272989X8800800308; PMID: 3398748.
 Donna Katzman McClish (1989) “Analyzing a Portion of the ROC Curve”. Medical Decision Making 9, 190–195. DOI: 10.1177/0272989X8900900307; PMID: 2668680.
 Nancy A. Obuchowski and Donna K. McClish (1997) “Sample size determination for diagnostic accurary studies involving binormal ROC curve indices”. Statistics in Medicine, 16, 1529–1542. DOI: 10.1002/(SICI)10970258(19970715)16:13<1529::AIDSIM565>3.0.CO;2H.
 Nancy A. Obuchowski, Micharl L. Lieber and Frank H. Wians Jr. (2004) “ROC Curves in Clinical Chemistry: Uses, Misuses, and Possible Solutions”. Clinical Chemistry, 50, 1118–1125. DOI: 10.1373/clinchem.2004.031823.
 Margaret Pepe, Gary Longton and Holly Janes (2009) “Estimation and Comparison of Receiver Operating Characteristic Curves”. The Stata journal 9, 1. PMID: 20161343.
 Xavier Robin, Natacha Turck et al. (2011) “pROC: an opensource package for R and S+ to analyze and compare ROC curves”. BMC Bioinformatics, 12, 77. DOI: 10.1186/147121051277.
 E. S. Venkatraman and Colin B. Begg (1996) “A distributionfree procedure for comparing receiver operating characteristic curves from a paired experiment”. Biometrika 83, 835–848. DOI: 10.1093/biomet/83.4.835
 E. S. Venkatraman (2000) “A Permutation Test to Compare Receiver Operating Characteristic Curves”. Biometrics 56, 1134–1138. DOI: 10.1111/j.0006341X.2000.01134.x.
 Kelly H. Zou, W. J. Hall and David E. Shapiro (1997) “Smooth nonparametric receiver operating characteristic (ROC) curves for continuous diagnostic tests”. Statistics in Medicine 18, 2143–2156. DOI: 10.1002/(SICI)10970258(19971015)16:19<2143::AIDSIM655>3.0.CO;23; PMID: 9330425 .