AUC unexpected value

Question

I have the following predictions after running a logistic regression model on a set of molecules we suppose that are predictive of tumors versus normals.

                  Predicted   class     
                      T        N          
                 T   29        5
  Actual class
                 N   993      912

I have a list of scores that range from predictions <0 (negative numbers) to predictions >0 (positive numbers). Then I have another column in my data.frame that indicated the labels (1== tumours and 0==normals) as predicted from the model. I tried to calculate the ROC using the library(ROC) in the following way:

 pred = prediction(prediction, labels)     
 roc = performance(pred, "tpr", "fpr")   
 plot(roc, lwd=2, colorize=TRUE)

Using:

       roc_full_data <- roc(labels, prediction)
       rounded_scores <- round(prediction, digits=1)
       roc_rounded <- roc(labels, prediction)

Call:

       roc.default(response = labels, predictor = prediction)
       Data: prediction in 917 controls (category 0) < 1022 cases (category1).
       Area under the curve: 1

The AUC is equal to 1. I'm not sure that I run all correctly or probably I'm doing something wrong in the interpretation of my results because it is quite rare that the AUC is equal to 1.

What exactly did you feed to these functions? A [reproducible example](http://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example) would be very helpful. — MrFlick, Mar 22 '17 at 20:08
Maybe you need gto do something like `auc(roc(pred, labels))`? — Fernando, Mar 22 '17 at 20:10

Jil Jung Juk · Accepted Answer · 2017-03-23T07:16:17.320

1

There is a typo in your x.measure which should have thrown an error. You have "for" and not "fpr". Try the following code.

performance(pred, measure = "tpr", x.measure = "fpr")
plot(perf)

# add a reference line to the graph
abline(a = 0, b = 1, lwd = 2, lty = 2)

# calculate AUC
perf.auc <- performance(pred, measure = "auc")
str(perf.auc)
as.numeric(perf.auc@y.values)

edited Mar 23 '17 at 07:16

answered Mar 23 '17 at 07:08

Jil Jung Juk

690
2
9
21

Sorry, I made a mistake during copy and paste from my code to here but the original was fpr. – Bfu38 Mar 23 '17 at 09:38

Fernando · Answer 2 · 2017-03-23T13:27:57.327

0

I use pROC to calculate AUC:

require(pROC)
set.seed(1)
pred = runif(100)
y = factor(sample(0:1, 100, TRUE))
auc = as.numeric(roc(response = y, predictor = pred)$auc)
print(auc) # 0.5430757

Or

require(AUC)
auc = AUC::auc(AUC::roc(pred, y))
print(auc) # 0.4569243

~~I can't explain why the results are different.~~

EDIT: The above aucs sum to 1.0, so one of the libs automatically 'inverted' the predictions.

edited Mar 23 '17 at 13:27

answered Mar 22 '17 at 20:06

Fernando

7,785
6
49
81

They are the same, but one inverted the prediction. Pure chance will give an AUC of 0.5, so everything less than that is better taken the opposite. – Jasper Mar 23 '17 at 09:49
@Jasper You're right, completely forgot about that! – Fernando Mar 23 '17 at 13:25

AUC unexpected value

2 Answers2