0

I'm looking for the equivalent to the mapvalues function in R in sparkR, e.g.

x <- c("a", "b", "c")
mapvalues(x, c("a", "c"), c("A", "C"))

I have found equivalent functions for this in Scala, for example this. But I can't find it in the documentation for sparkR.

EDIT:

In the linked doumentation there is a function called map_values:

##D # Dataframe used throughout this doc
df <- createDataFrame(cbind(model = rownames(mtcars), mtcars))
tmp3 <- mutate(df, v3 = create_map(df$model, df$cyl))
head(select(tmp3, map_keys(tmp3$v3), map_values(tmp3$v3)))
head(select(tmp3, element_at(tmp3$v3, "Valiant")))

But it is not used in the same way, both map_values and map_keys are used in the same column in at where they were created: "v3" is not like I can use the variable v3 to map again values from model to cyl.

Shaido
  • 27,497
  • 23
  • 70
  • 73
Luis Ramon Ramirez Rodriguez
  • 9,591
  • 27
  • 102
  • 181
  • I do see it added in 2.3.0, check this page. https://spark.apache.org/docs/latest/api/R/index.html – Piyush Patel May 06 '20 at 21:49
  • `map_keys` and `map_values` are simply getting the keys/values from a map column (dictionary). As for the functionality of `mapvalues` in R, you need to implement it yourself. I'm unfortunately not familiar enough with R to help you (there should be some `map` function in SparkR to use I guess?). – Shaido May 13 '20 at 06:29

1 Answers1

1

The Spark function map_values unfortunately very different to the R function mapvalues. It returns the values from a map, rather than mapping values.

As others have mentioned there's no direct replacement for mapvalues in spark

Here is one way to do what you described

sparkR.session()

# Create initial dataframe
df <- createDataFrame(cbind(model = rownames(mtcars), mtcars))

# create a data frame with one column to find and one column to replace with
to_replace <- data.frame(find    = c('Valiant',      'Datsun 710',           'Ferrari Dino'),
                         replace = c('Valiant v2.0', 'Datsun 710 (Retired)', 'Ferrari Dino (Raptor Edition)'))

# good pratice would be to match only entries that exactly match the find column, but not those that contain using regular expressions
to_replace_rexp <- data.frame(find    = c('^Valiant$',      '^Datsun 710$',           '^Ferrari Dino$'),
                              replace = c('Valiant v2.0', 'Datsun 710 (Retired)', 'Ferrari Dino (Raptor Edition)'))
# Turn it into a spark dataframe
to_replace_spark <- createDataFrame(to_replace)
head(to_replace_spark)
#     find                       replace
#      Valiant                  Valiant v2.0
#   Datsun 710          Datsun 710 (Retired)
# Ferrari Dino Ferrari Dino (Raptor Edition)

# Left join the to replace table, adding two columns with matching results
joined_data <- SparkR::join(df, to_replace_spark, df$model == to_replace_spark$find, 'left_outer')
head(joined_data)
#         model  mpg cyl disp  hp drat    wt  qsec vs am gear carb       find              replace
#     Mazda RX4 Wag 21.0   6  160 110 3.90 2.875 17.02  0  1    4    4       <NA>                 <NA>
#         Mazda RX4 21.0   6  160 110 3.90 2.620 16.46  0  1    4    4       <NA>                 <NA>
# Hornet Sportabout 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2       <NA>                 <NA>
#        Datsun 710 22.8   4  108  93 3.85 2.320 18.61  1  1    4    1 Datsun 710 Datsun 710 (Retired)
#    Hornet 4 Drive 21.4   6  258 110 3.08 3.215 19.44  1  0    3    1       <NA>                 <NA>
#           Valiant 18.1   6  225 105 2.76 3.460 20.22  1  0    3    1    Valiant         Valiant v2.0

# Replace the model column with the replace column if it isn't empty
joined_data_coal <- SparkR::mutate(joined_data, model = SparkR::coalesce(joined_data$replace, joined_data$model))
head(joined_data_coal)
#                     model  mpg cyl  disp  hp drat    wt  qsec vs am gear carb         find
#                Ford Pantera L 15.8   8 351.0 264 4.22 3.170 14.50  0  1    5    4         <NA>
#                    Merc 450SL 17.3   8 275.8 180 3.07 3.730 17.60  0  0    3    3         <NA>
#                   Honda Civic 30.4   4  75.7  52 4.93 1.615 18.52  1  1    4    2         <NA>
# Ferrari Dino (Raptor Edition) 19.7   6 145.0 175 3.62 2.770 15.50  0  1    5    6 Ferrari Dino
#                 Mazda RX4 Wag 21.0   6 160.0 110 3.90 2.875 17.02  0  1    4    4         <NA>
#                     Mazda RX4 21.0   6 160.0 110 3.90 2.620 16.46  0  1    4    4         <NA>
                    replace
#                          <NA>
#                          <NA>
#                          <NA>
# Ferrari Dino (Raptor Edition)
#                          <NA>
#                          <NA>

# Drop the columns we joined
data_final <- drop(joined_data_coal, c("find", "replace"))
head(data_final)
#                     model  mpg cyl  disp  hp drat    wt  qsec vs am gear carb
#                Ford Pantera L 15.8   8 351.0 264 4.22 3.170 14.50  0  1    5    4
#                    Merc 450SL 17.3   8 275.8 180 3.07 3.730 17.60  0  0    3    3
#                   Honda Civic 30.4   4  75.7  52 4.93 1.615 18.52  1  1    4    2
# Ferrari Dino (Raptor Edition) 19.7   6 145.0 175 3.62 2.770 15.50  0  1    5    6
#                 Mazda RX4 Wag 21.0   6 160.0 110 3.90 2.875 17.02  0  1    4    4
#                     Mazda RX4 21.0   6 160.0 110 3.90 2.620 16.46  0  1    4    4

# Final results
head(filter(data_final, data_final$cyl == "6"))
                      model  mpg cyl  disp  hp drat    wt  qsec vs am gear carb
# Ferrari Dino (Raptor Edition) 19.7   6 145.0 175 3.62 2.770 15.50  0  1    5    6
#                 Mazda RX4 Wag 21.0   6 160.0 110 3.90 2.875 17.02  0  1    4    4
#                     Mazda RX4 21.0   6 160.0 110 3.90 2.620 16.46  0  1    4    4
#                     Merc 280C 17.8   6 167.6 123 3.92 3.440 18.90  1  0    4    4
#                Hornet 4 Drive 21.4   6 258.0 110 3.08 3.215 19.44  1  0    3    1
#                  Valiant v2.0 18.1   6 225.0 105 2.76 3.460 20.22  1  0    3    1
Charlie
  • 188
  • 3