Confusion Matrix and caret package - rpart algorithm

Question

I'm developing a simple rpart model using caret package.

library(caret)
data <- fread(sprintf("%s/%s", dataDir, dataFile), header=T, stringsAsFactors = TRUE)
split <- createDataPartition(y = data$Audit, p = 0.7, list = FALSE)
trainSet <- data[split,]
test <- data[-split,]

fitControl <- trainControl( method = "repeatedcv",  number = 10,repeats = 10)


fitControl <- trainControl( method = "repeatedcv",  number = 10,repeats = 10)

fit <- train(Audit ~ Activity1 + Activity2 + Activity3 + Activity4 + Activity5 + Activity6 + Activity7 + Activity8, data = trainSet, method="rpart", trControl = fitControl)

Now I'm trying to create a confusion matrix:

preds<-predict(fit2, newdata = test)
cm<-confusionMatrix(preds,test$Audit)
print(cm)

When I see the output I see this:

     [,1] [,2]
[1,]    0    0
[2,]    4    0

This is really confusing to me and it doesn't make sense. I was expecting something like this to be my Confusion Matrix

preds A B U
    A 0 0 0
    B 0 4 0
    U 2 0 9

here is the snapshot of my data:

snapshot of the data

here is dput of my data:

dput(as.data.frame(data))
structure(list(Activity1 = structure(c(2L, 1L, 2L, 1L, 1L, 1L, 
1L, 2L, 1L, 1L, 2L, 1L, 1L, 2L, 1L, 1L, 2L, 1L, 1L, 2L, 2L, 2L, 
2L, 1L, 2L, 1L, 1L, 2L, 2L, 2L, 1L, 1L, 1L, 1L, 1L, 1L, 2L, 2L, 
1L, 1L, 1L, 1L, 2L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 1L, 2L, 2L, 1L, 
2L, 2L), .Label = c("a", "b"), class = "factor"), Activity2 = 
structure(c(2L, 
1L, 2L, 1L, 1L, 1L, 1L, 2L, 1L, 1L, 2L, 1L, 1L, 2L, 1L, 1L, 2L, 
1L, 1L, 2L, 2L, 2L, 2L, 1L, 2L, 1L, 1L, 2L, 2L, 2L, 1L, 1L, 1L, 
1L, 1L, 1L, 2L, 2L, 1L, 1L, 1L, 1L, 2L, 1L, 1L, 1L, 1L, 2L, 2L, 
2L, 1L, 2L, 2L, 1L, 2L, 2L), .Label = c("a", "b"), class = "factor"), 
Activity3 = structure(c(2L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L), .Label = c("a", "b"), class = "factor"), Activity4 = structure(c(2L, 
1L, 2L, 2L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 1L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 1L, 
2L, 2L, 1L, 2L, 2L, 2L, 2L, 2L, 1L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L), .Label = c("b", 
"e"), class = "factor"), Activity5 = structure(c(2L, 1L, 
2L, 2L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 1L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 1L, 2L, 
2L, 1L, 2L, 2L, 2L, 2L, 2L, 1L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L), .Label = c("b", "d"), class = "factor"), 
Activity6 = structure(c(2L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L), .Label = c("d", "e"), class = "factor"), Activity7 = structure(c(1L, 
1L, 1L, 2L, 1L, 1L, 1L, 1L, 1L, 2L, 1L, 2L, 2L, 1L, 2L, 1L, 
1L, 1L, 2L, 1L, 1L, 1L, 1L, 1L, 1L, 2L, 2L, 1L, 1L, 1L, 1L, 
2L, 2L, 1L, 1L, 2L, 1L, 1L, 2L, 1L, 1L, 1L, 1L, 2L, 2L, 1L, 
1L, 1L, 1L, 1L, 2L, 1L, 1L, 2L, 1L, 1L), .Label = c("a", 
"c"), class = "factor"), Activity8 = structure(c(2L, 2L, 
4L, 2L, 2L, 2L, 2L, 4L, 2L, 1L, 3L, 1L, 2L, 4L, 1L, 2L, 2L, 
2L, 2L, 2L, 4L, 2L, 3L, 2L, 4L, 2L, 1L, 2L, 3L, 4L, 2L, 2L, 
1L, 2L, 2L, 2L, 3L, 4L, 1L, 2L, 2L, 2L, 3L, 1L, 1L, 2L, 2L, 
3L, 2L, 2L, 2L, 3L, 2L, 2L, 4L, 3L), .Label = c("a", "c", 
"d", "e"), class = "factor"), Audit = structure(c(3L, 1L, 
3L, 2L, 1L, 1L, 1L, 3L, 3L, 2L, 3L, 2L, 2L, 3L, 2L, 1L, 3L, 
3L, 2L, 3L, 3L, 3L, 3L, 3L, 3L, 2L, 2L, 3L, 3L, 3L, 1L, 2L, 
2L, 1L, 3L, 2L, 3L, 3L, 2L, 1L, 3L, 3L, 3L, 2L, 2L, 3L, 3L, 
3L, 3L, 3L, 2L, 3L, 3L, 2L, 3L, 3L), .Label = c("A", "B", 
"U"), class = "factor")), .Names = c("Activity1", "Activity2", 
"Activity3", "Activity4", "Activity5", "Activity6", "Activity7", 
"Activity8", "Audit"), row.names = c(NA, -56L), class = "data.frame")

Here is the dput of my test data:

> dput(as.data.frame(test))
structure(list(Activity1 = structure(c(1L, 1L, 1L, 2L, 2L, 1L, 
1L, 2L, 1L, 1L, 2L, 2L, 1L, 2L, 2L), .Label = c("a", "b"), class = 
"factor"), 
Activity2 = structure(c(1L, 1L, 1L, 2L, 2L, 1L, 1L, 2L, 1L, 
1L, 2L, 2L, 1L, 2L, 2L), .Label = c("a", "b"), class = "factor"), 
Activity3 = structure(c(1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L), .Label = c("a", "b"), class = "factor"), 
Activity4 = structure(c(2L, 2L, 2L, 2L, 2L, 1L, 2L, 2L, 1L, 
2L, 2L, 2L, 2L, 2L, 2L), .Label = c("b", "e"), class = "factor"), 
Activity5 = structure(c(2L, 2L, 2L, 2L, 2L, 1L, 2L, 2L, 1L, 
2L, 2L, 2L, 2L, 2L, 2L), .Label = c("b", "d"), class = "factor"), 
Activity6 = structure(c(1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L), .Label = c("d", "e"), class = "factor"), 
Activity7 = structure(c(2L, 1L, 2L, 1L, 1L, 1L, 2L, 1L, 1L, 
1L, 1L, 1L, 2L, 1L, 1L), .Label = c("a", "c"), class = "factor"), 
Activity8 = structure(c(2L, 2L, 2L, 4L, 2L, 2L, 2L, 3L, 2L, 
2L, 3L, 2L, 2L, 4L, 3L), .Label = c("a", "c", "d", "e"), class = "factor"), 
Audit = structure(c(2L, 3L, 2L, 3L, 3L, 1L, 2L, 3L, 1L, 3L, 
3L, 3L, 2L, 3L, 3L), .Label = c("A", "B", "U"), class = "factor")), .Names = c("Activity1", 
"Activity2", "Activity3", "Activity4", "Activity5", "Activity6", 
"Activity7", "Activity8", "Audit"), class = "data.frame", row.names = c(NA, 
-15L))

Can anyone help me with this please? Thanks

In order to answer we would need to see the train and test data. Could you provide them with `dput`? — missuse, Mar 28 '18 at 16:23
Please [read this](https://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example). In order to solve your problem we must first recreate it. To recreate it we need the data and the code that produced the same result as you are showing. No one wants to type data from a picture. Let alone data that will not recreate the problem. — missuse, Mar 28 '18 at 16:39
Thanks @missuse for the info. I'm sorry I'm new to this website but I've updated the question with dput of my data. Hope that's ok? — vik2426, Mar 28 '18 at 16:51
No problem, we were all new at one point or another. could you also provide `dput` for the test data? One problem I can see so far is that the selected `cp` is 0. This is not good. That means the tree was not pruned at all. — missuse, Mar 28 '18 at 16:54
I can not recreate the problem with your data. I get a confusion matrix with 3 classes. Try it with the data in the post in a clean R session. On another note`cp` or`complexity parameter` in the rpat algorithm is the hyper parameter you are tuning with `caret` `train`. Due to very few cp's being tested (3 which is default) cp = 0 is selected, this means no pruning for the built model which is rarely a good idea. Set `tuneLength = 10` in train to select a better `cp`. — missuse, Mar 28 '18 at 17:05
Thanks again @missuse for your prompt reply. Does this mean that I'm not loading the data correctly? do you think confusion matrix isn't getting the data in correct format or something? I'll try again in a clean R session and update you. — vik2426, Mar 28 '18 at 17:18
I apologize I did not see you split the data. So the test set is in fact a subset of the data you provided. You can remove the test set from the post. The reason why I can not recreate the problem is perhaps because of the different data split in `createDataPartition`. Could you test the same code with an addition of `set.seed(1)` prior calling `createDataPartition`. Why are you calling `fit2` in predict when you fit a model called `fit`? — missuse, Mar 28 '18 at 17:22
Thanks so much. I just restarted my R and ran the same code again and managed to get the confusion matrix working absolutely fine. I've absolutely no idea why it wasn't working before. all I did was restarted my R on my windows machine. I have exactly same problem on another linux machine so I guess I'll just restart my r session there too. Thanks again for all your help @missuse — vik2426, Mar 28 '18 at 17:29

score 0 · Accepted Answer · answered Mar 28 '18 at 23:41

I have just found out the reason why my confusionMatrix was nbot working properly. This was because I was loading library(caret) as well as library(ModelMetrics) at the start of the code. as soon as I commented out library(ModelMetrics) and retarted my R, I was able to get the Confusion Matrix working perfectly fine.

Confusion Matrix and caret package - rpart algorithm

1 Answers1