Undocumented error in dcast.data.table

Question

(This was posted previously at the data-table-help mailing list, but it's been a few weeks without comment, and I did a little more to try to debug it.)

I ran into a strange error that an internet search only turns up in the commit log of data.table:

# Error in dcast.data.table(test.table, as.formula(paste(class.col, "+",  : 
#   retFirst must be integer vector the same length as nrow(i)

This came up on running a previously tested working dcast.data.table expression, on a data.table I have subsetted by randomly resampling Trial with replacement. The offending section is this:

dcast.data.table(test.table, 
                 Class + Time + Trial ~ Channel,
                 value.var = "Voltage",
                 fun.aggregate=identity)

It seems to be choking on near-duplicate rows in the input table (i.e., the error is the same with or without the id column present in the table):

test.table <- structure(list(Trial = c(1169L, 1169L), Sample = c(155L, 155L
), Class = c(1L, 1L), Subject = structure(c(13L, 13L), .Label = c("s01", 
"s02", "s03", "s04", "s05", "s06", "s07", "s08", "s09", "s10", 
"s11", "s12", "s13"), class = "factor"), Channel = c(1L, 1L), 
    Voltage = structure(c(-0.992322316444497, -0.992322316444497
    ), "`scaled:center`" = -6.23438399446429e-16, "`scaled:scale`" = 1), 
    Time = c(201.149466192171, 201.149466192171), Baseline = c(0.688151312347969, 
    0.688151312347969), id = 1:2), .Names = c("Trial", "Sample", 
"Class", "Subject", "Channel", "Voltage", "Time", "Baseline", 
"id"), class = c("data.table", "data.frame"), row.names = c(NA, 
-2L), sorted = "id")    

test.table
#    Trial Sample Class Subject Channel    Voltage     Time  Baseline id
# 1:  1169    155     1     s13       1 -0.9923223 201.1495 0.6881513  1
# 2:  1169    155     1     s13       1 -0.9923223 201.1495 0.6881513  2
dcast.data.table(test.table, 
                  Class + Time + Trial ~ Channel,
                  value.var = "Voltage",
                  fun.aggregate=identity)
# Error in dcast.data.table(test.table, Class + Time + Trial ~ Channel,  : 
#   retFirst must be integer vector the same length as nrow(i)

Changing a single column in the dcast formula gets close to the output I am looking for:

test.table[2,Trial:=1170]
dcast.data.table(test.table, 
                  Class + Time + Trial ~ Channel,
                  value.var = "Voltage",
                  fun.aggregate=identity)
#    Class     Time Trial          1
# 1:     1 201.1495  1169 -0.9923223
# 2:     1 201.1495  1170 -0.9923223

What's bothering data.table? I tried changing keys and messing with the order of the formula terms just to see, because I don't understand the error, but that didn't work.

If I replace the function call with regular dcast from reshape2, I get a seemingly unrelated error:

# Error in vapply(indices, fun, .default) : values must be length 0, but FUN(X[[29]]) result is length 1

At this point in my code I don't care if the Trial values are correct, so I could work around this by replacing it in the formula with id, but I'm interested in a more general or robust solution.

It would be helpful if your example were [reproducible](http://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example). You have not included sample data such that we can run your code exactly and encounter the same error. — MrFlick, Jun 11 '14 at 01:26
I agree, and I mentioned in the question that I would like to know what the error means so that I have a clue on how to reproduce it in a concise way, instead of populating the question with a ton of (possibly irrelevant) code. — bright-star, Jun 11 '14 at 01:40
I wouldn't say the errors are un related, both appear to suggest there is a case where a return value is not the length it "should" be. A reproducible example will be required. Try and find a minimal subset of the data on which the error occurs, and then use gist.github or similar to host the data. — mnel, Jun 11 '14 at 02:29
I narrowed it down to `dcast` of identical rows and managed to get a working example. — bright-star, Jun 12 '14 at 00:22

score 2 · Accepted Answer · edited Jun 20 '20 at 09:12

Update: Fixed in commit 1253 of v1.9.3. From NEWS:

dcast.data.table provides better error message when fun.aggregate is specified but it returns length != 1. Closes git #693. Thanks to Trevor Alexander for reporting here on SO.

I agree that the error message should be more helpful in understanding the issue and it usually is in data.table. This is just a case I hadn't foreseen.

If you could please file the issue here as a bug, I'll fix it when I've some time.

Your problem, however, seems quite trivial RTFM to me. From ?dcast.data.table:

fun.aggregate - Should the data be aggregated before casting? If the formula doesn't identify single observation for each cell, then aggregation defaults to length with a message.

In the DETAILS section: "... fun.aggregate will have to be used. The aggregating function should take a vector as input and return a single value (or a list of length one) as output." ...

In your example, your formula's LHS results in two identical rows, which means fun.aggregate has to be used - which'll default to length if you dint use one (like reshape2:::dcast does). And you've used identity, which'll just return the values back. So it returns both the values for Voltage, which the function doesn't like.

The error message should be something like:

Error: fun.aggregate should return, for each unique group (from formula's LHS), a length 1 vector, but returns length=2 for a group.

Or something of that sort. Feel free to suggest better / clearer error messages.

PS: I don't understand what you mean by near-duplicate.

identical(test.table[1, list(Class, Time, Trial)], 
          test.table[2, list(Class, Time, Trial)])
# [1] TRUE

If you use id column on the LHS, then you should be able to get the desired result, as you can now uniquely identify the rows...

dcast.data.table(test.table, 
             Class + Time + Trial ~ Channel + id,
             value.var = "Voltage",
             fun.aggregate=identity)

#    Class     Time Trial        1_1        1_2
# 1:     1 201.1495  1169 -0.9923223 -0.9923223

The function only considers the columns given in the formula LHS to find out if there are/aren't unique rows, not if your actual input data has unique rows (if that was the confusion).

To answer OP's 2nd comment:

The only way currently to get a result (without error) is if your function returns a list:

dcast.data.table(test.table, 
             Class + Time + Trial ~ Channel,
             value.var = "Voltage",
             fun.aggregate=list)
#    Class     Time Trial                     1
# 1:     1 201.1495  1169 -0.9923223,-0.9923223

Then you can just check if the columns are all of length 1 and if so, unlist.

I'll edit the question to be more clear; sorry about that. I definitely don't have a problem with the *behavior* of `dcast.data.table`, and this error wasn't really irritating, just confusing. What I meant by near-duplicate was that the *input* table does not contain duplicates (when given unique ids as I did in the `dput`), but the formula I gave it causes non-unique rows to be generated. These subtleties were lost on me coming back to the insides of a function I wrote and tested two months ago and promptly forgot how it worked :) I think your proposed error message is perfect for this. — bright-star, Jun 12 '14 at 00:53
I filed the issue at github. I agree with your workaround too for the `id` column--it's what I went with today (see end of the question), but it's sort of unsatisfying because it has to be worked in based on the structure of the input; it would be nice if there was some way to 'coerce uniqueness' in a way that would not affect the structure of the output so much. — bright-star, Jun 12 '14 at 01:04

Undocumented error in dcast.data.table

1 Answers1

Update: Fixed in commit 1253 of v1.9.3. From NEWS: