Easy way of counting precision, recall and F1-score in R

Question

I am using an rpart classifier in R. The question is - I would want to test the trained classifier on a test data. This is fine - I can use the predict.rpart function.

But I also want to calculate precision, recall and F1 score.

My question is - do I have to write functions for those myself, or is there any function in R or any of CRAN libraries for that?

score 30 · Answer 1 · edited Apr 01 '19 at 03:44

using the caret package:

library(caret)

y <- ... # factor of positive / negative cases
predictions <- ... # factor of predictions

precision <- posPredValue(predictions, y, positive="1")
recall <- sensitivity(predictions, y, positive="1")

F1 <- (2 * precision * recall) / (precision + recall)

A generic function that works for binary and multi-class classification without using any package is:

f1_score <- function(predicted, expected, positive.class="1") {
    predicted <- factor(as.character(predicted), levels=unique(as.character(expected)))
    expected  <- as.factor(expected)
    cm = as.matrix(table(expected, predicted))

    precision <- diag(cm) / colSums(cm)
    recall <- diag(cm) / rowSums(cm)
    f1 <-  ifelse(precision + recall == 0, 0, 2 * precision * recall / (precision + recall))

    #Assuming that F1 is zero when it's not possible compute it
    f1[is.na(f1)] <- 0

    #Binary F1 or Multi-class macro-averaged F1
    ifelse(nlevels(expected) == 2, f1[positive.class], mean(f1))
}

Some comments about the function:

It's assumed that an F1 = NA is zero
positive.class is used only in binary f1
for multi-class problems, the macro-averaged F1 is computed
If predicted and expected had different levels, predicted will receive the expected levels

hi thanks. I noticed that there is a warning when for instance one of the class never gets predicted. Do you think the calculation is still valid? — agenis, Dec 06 '17 at 14:55
Thank you, you are right. I made a little improvement to fix this problem. Now it's working right in such cases. — Adriano Rivolli, Dec 13 '17 at 11:51

score 25 · Accepted Answer · answered Dec 14 '11 at 09:22

25

The ROCR library calculates all these and more (see also http://rocr.bioinf.mpi-sb.mpg.de):

library (ROCR);
...

y <- ... # logical array of positive / negative cases
predictions <- ... # array of predictions

pred <- prediction(predictions, y);

# Recall-Precision curve             
RP.perf <- performance(pred, "prec", "rec");

plot (RP.perf);

# ROC curve
ROC.perf <- performance(pred, "tpr", "fpr");
plot (ROC.perf);

# ROC area under the curve
auc.tmp <- performance(pred,"auc");
auc <- as.numeric(auc.tmp@y.values)

...

answered Dec 14 '11 at 09:22

Itamar

2,111
13
16

7

... and for F1-score `performance(pred,"f")` gives a vector of F1-scores – smci Mar 04 '14 at 10:19
1

and I guess that also the predictions must contain scores of confidence or probability for each prediction? – peschü Mar 18 '15 at 06:42
1

Just to clarify: `Performance` uses the `prediction` object that is constructed from the scores (`predictions`) and labels (`y`) of each case. There is no additional number beyond that (such as confidence, etc.). – Itamar Mar 19 '15 at 07:57
1

@itamar Can you help me with computing area under precision recall curve. My question is [here](http://stackoverflow.com/questions/39508831/precision-recall-curve-computation-by-prroc-package-in-r) Thanks. – Newbie Sep 15 '16 at 12:00
1

ROCR doesn't work if in the problem there are more than 2 classes to predict – Matias Aug 01 '20 at 19:17
1

@peschü seems to be right. `ROCR` expects the predictions to be confidence values, so that it can tweak the cutoff threshold and therefore calculate a ROC curve. However if you don't have confidence values because of the nature of your classifier, the outputs are very confusing. It's probably possible to extract the raw F1 score but it's not clear how. – Migwell Jul 08 '21 at 11:16

score 15 · Answer 3 · edited Jul 10 '20 at 22:45

Just to update this as I came across this thread now, the confusionMatrix function in caretcomputes all of these things for you automatically.

cm <- confusionMatrix(prediction, reference = test_set$label)

# extract F1 score for all classes
cm[["byClass"]][ , "F1"] #for multiclass classification problems

You can substitute any of the following for "F1" to extract the relevant values as well:

"Sensitivity", "Specificity", "Pos Pred Value", "Neg Pred Value", "Precision", "Recall", "F1", "Prevalence", "Detection", "Rate", "Detection Prevalence", "Balanced Accuracy"

I think this behaves slightly differently when you're only doing a binary classifcation problem, but in both cases, all of these values are computed for you when you look inside the confusionMatrix object, under $byClass

score 7 · Answer 4 · answered Aug 24 '17 at 15:19

7

confusionMatrix() from caret package can be used along with a proper optional field "Positive" specifying which factor should be taken as positive factor.

confusionMatrix(predicted, Funded, mode = "prec_recall", positive="1")

This code will also give additional values such as F-statistic, Accuracy, etc.

answered Aug 24 '17 at 15:19

nin

89
1
4

or you can just (mode = "everything") to print all statistics. – Zawir Amin Apr 02 '20 at 00:28

score 4 · Answer 5 · answered Jun 17 '15 at 15:51

I noticed the comment about F1 score being needed for binary classes. I suspect that it usually is. But a while ago I wrote this in which I was doing classification into several groups denoted by number. This may be of use to you...

calcF1Scores=function(act,prd){
  #treats the vectors like classes
  #act and prd must be whole numbers
  df=data.frame(act=act,prd=prd);
  scores=list();
  for(i in seq(min(act),max(act))){
    tp=nrow(df[df$prd==i & df$act==i,]);        
    fp=nrow(df[df$prd==i & df$act!=i,]);
    fn=nrow(df[df$prd!=i & df$act==i,]);
    f1=(2*tp)/(2*tp+fp+fn)
    scores[[i]]=f1;
  }      
  print(scores)
  return(scores);
}

print(mean(unlist(calcF1Scores(c(1,1,3,4,5),c(1,2,3,4,5)))))
print(mean(unlist(calcF1Scores(c(1,2,3,4,5),c(1,2,3,4,5)))))

score 3 · Answer 6 · answered Aug 24 '17 at 15:41

3

We can simply get F1 value from caret's confusionMatrix function

result <- confusionMatrix(Prediction, Lable)

# View confusion matrix overall
result 

# F1 value
result$byClass[7]

answered Aug 24 '17 at 15:41

donghyeon kim

39
2

2

It is not F1 value – Siddhu Mar 12 '19 at 06:59

score 1 · Answer 7 · answered Jan 29 '17 at 17:45

You can also use the confusionMatrix() provided by caret package. The output includes,between others, Sensitivity (also known as recall) and Pos Pred Value(also known as precision). Then F1 can be easily computed, as stated above, as: F1 <- (2 * precision * recall) / (precision + recall)

Fasanya OluwaFunmibi Omotayo · Answer 8 · 2022-06-28T16:35:04.833

0

library(caret)

result <- confusionMatrix(Prediction, label)

#This shows all the measures you need including precision, recall and F1

result$byClass

edited Jun 28 '22 at 16:35

answered Jun 28 '22 at 16:33

Fasanya OluwaFunmibi Omotayo

1
1

Easy way of counting precision, recall and F1-score in R

8 Answers8

Linked