R caret Error: Something is wrong; all the Accuracy metric values are missing:

Question

I'm trying to apply stacking on my dataset but here I am.

# Load library
library(DJL)
library(caret)
library(caretEnsemble)

# Load data and format the target attribute to avoid clutters
df <- dataset.engine.2015[, -c(1, 2)]
levels(df$Type) <- list(NA.D = "NA-D", NA.P = "NA-P", SC.P = "SC-P", TC.D = "TC-D", TC.P = "TC-P")

# Run
st.methods <- c("lda", "rpart", "glm", "knn", "svmRadial")
st.control <- trainControl(method = "repeatedcv", number = 5, repeats = 3, 
                           savePredictions = T, classProbs = T)
st.models  <- caretList(Type ~., data = df, trControl = st.control, methodList = st.methods)

Then I get this:

Something is wrong; all the Accuracy metric values are missing:
    Accuracy       Kappa    
 Min.   : NA   Min.   : NA  
 1st Qu.: NA   1st Qu.: NA  
 Median : NA   Median : NA  
 Mean   :NaN   Mean   :NaN  
 3rd Qu.: NA   3rd Qu.: NA  
 Max.   : NA   Max.   : NA  
 NA's   :1     NA's   :1    
Error: Stopping
In addition: There were 18 warnings (use warnings() to see them)

Can anyone help me to fix this error?

Could you try to make your example [reproducible](http://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example), else any advice would be mere speculation. — Silence Dogood, May 02 '17 at 11:45
Sorry, I forgot to load the package "caretEnsemble". Now, you can reproduce my error. Thanks for pointing that out! — user4143385, May 02 '17 at 12:15
Without a few observations from your input dataset `dataset.engine.2015`, your example is not reproducible. If your dataset is proprietary you can use options mentioned in first comment to anonymize it. Essentially, create a small test dataset on which you receive the error and post the output of `dput(DF)` where `DF` is the test dataset. — Silence Dogood, May 02 '17 at 12:56
I don't know what you mean by "not reproducible". The dataset (dataset.engine.2015) is accompanied with the package "DJL" so I believe you can simply implement my code to load it and reproduce my issue. Please advise if this is not what you asked. — user4143385, May 02 '17 at 13:45
I am sorry,please excuse my unfamiliarity with `DJL` package and my assumption that `dataset.engine.2015` is a user-defined dataset. — Silence Dogood, May 02 '17 at 15:28

score 3 · Answer 1 · answered May 02 '17 at 14:35

The glm model cannot be used for predicting categorical dependent variables with more than two categories. Try to delete glm from st.methods or substitute glm with, for example, multinom, gbm, randomForest.

Here are two useful experiment. In the first we consider only glm:

rm(list=ls())
library(DJL)
library(caret)
library(caretEnsemble)  
df <- dataset.engine.2015[, -c(1, 2)]
levels(df$Type) <- list(NA.D = "NA-D", NA.P = "NA-P", SC.P = "SC-P", TC.D = "TC-D", TC.P = "TC-P")

st.control <- trainControl(method = "repeatedcv", number = 5, repeats = 3, 
                           savePredictions = T, classProbs = T)

st.methods <- c("glm")
st.models  <- caretList(Type ~., data = df, trControl = st.control, methodList = st.methods)

Here is the error message:

Something is wrong; all the Accuracy metric values are missing:
    Accuracy       Kappa    
 Min.   : NA   Min.   : NA  
 1st Qu.: NA   1st Qu.: NA  
 Median : NA   Median : NA  
 Mean   :NaN   Mean   :NaN  
 3rd Qu.: NA   3rd Qu.: NA  
 Max.   : NA   Max.   : NA  
 NA's   :1     NA's   :1    
Error in train.default(x, y, weights = w, ...) : Stopping
Inoltre: There were 18 warnings (use warnings() to see them)

Now we substitute glm with multinom:

st.methods <- c("multinom")
st.models  <- caretList(Type ~., data = df, trControl = st.control, methodList = st.methods)
print(st.models)

The output is:

$multinom
Penalized Multinomial Regression 

1206 samples
   5 predictor
   5 classes: 'NA.D', 'NA.P', 'SC.P', 'TC.D', 'TC.P' 

No pre-processing
Resampling: Cross-Validated (5 fold, repeated 3 times) 
Summary of sample sizes: 964, 965, 965, 965, 965, 964, ... 
Resampling results across tuning parameters:

  decay  Accuracy   Kappa    
  0e+00  0.9306411  0.8518294
  1e-04  0.9300901  0.8506964
  1e-01  0.9328507  0.8564466

Accuracy was used to select the optimal model using  the largest value.
The final value used for the model was decay = 0.1.

Thanks for the solution! It works now but unfortunately caretEnsemble package doesn't provide stacking for multi-class (>2) problems: it returns "Not yet implemented for multiclass problems" when I try to create an ensemble solution using caretStack function. I wish I had known this earlier! I may have to find another package to complete this task. Please comment if anyone knows better options in R. — user4143385, May 03 '17 at 02:46

R caret Error: Something is wrong; all the Accuracy metric values are missing:

1 Answers1