0

I have a KNN model in caret and I'd like to optimize for median relative absolute error.

library(caret)
model <- train(
  close_price~ ., data = train.data, method = "knn",
  trControl = trainControl("cv", number = 10),
  preProcess = c("center", "scale"),
  metric = "MdRAE",
  tuneLength = 10
)

I tried to use MdRAE and MDRAE and they both returned this error.

Warning message:
In train.default(x, y, weights = w, ...) :
  The metric "MdRAE" was not in the result set. RMSE will be used instead

Is there a list of all the metrics available? I couldn't find it in the caret ebook.

StupidWolf
  • 45,075
  • 17
  • 40
  • 72
goollan
  • 765
  • 8
  • 19
  • Does "error optimizing " mean reducing the error? Where did you find these arguments `MdRAE` and `MDRAE`? – bbiasi May 27 '19 at 17:01
  • Yes. And they were just my guesses for how the metric might be listed. – goollan May 27 '19 at 17:09
  • So, as far as I know, it is not possible to optimize the error. What is usually done is the grid search in order to find the parameters and or hyperparameters that minimize the RMSE and the MAE. In your case, you want to find the best value of `K`, and this depends on the quality of your data. – bbiasi May 27 '19 at 17:14
  • [See here](https://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example) on making an R question that folks can help with. That includes a sample of data and a clear explanation of what you're trying to do. You haven't mentioned where you expect `MdRAE` to come from – camille May 28 '19 at 02:27

1 Answers1

3

This metric is not directly implemented in caret but you can easily do it yourself:

mdrae_summary <- function(data, lev=NULL, model=NULL) {
    c(MdRAE=median(abs(data$pred - data$obs)/data$obs))
}

model <- train(
  close_price~ ., data = train.data, method = "knn",
  trControl = trainControl("cv", number = 10, summaryFunction = mdrae_summary),
  preProcess = c("center", "scale"),
  metric = "MdRAE",
  tuneLength = 10
)
Pierre Gramme
  • 1,209
  • 7
  • 23
  • If I were to combine this with your answer from the other question (https://stackoverflow.com/questions/56229145/how-do-i-avoid-time-leakage-in-my-knn-model), would I just add `summaryFunction = mdrae_summary` to tr_ctrl? – goollan May 27 '19 at 17:24
  • Yes, as simple as that – Pierre Gramme May 28 '19 at 07:13