0

I have a block of code that I have written to pull JSON data from an API and convert it to a data frame. The base code is:

hospitals_url <- "https://services1.arcgis.com/Hp6G80Pky0om7QvQ/arcgis/rest/services/Hospitals_1/FeatureServer/0/query?where=1%3D1&outFields=*&outSR=4326&f=json"
        hospitals_num <- c(0, 2000, 4000, 6000)
        hosget <- lapply(hospitals_num, function(num) {
          hospitals_url_loop <- paste(hospitals_url, "&resultOffset=", num)
          hospitals_json <- fromJSON(hospitals_url_loop)
          hospitals_df <- as.data.frame(hospitals_json$features$attributes)})
        
        hospitals_df <- do.call(rbind, hosget)}

I am trying to add this to a tryCatch() function so that I can be more engaged in error management and control. What I am trying to do is use the error and warning sections of thetryCatch() function to create variables containing errors (if there are any), and throw them in a list. I have the following:

tryCatch({
        hospitals_url <- "https://services1.arcgis.com/Hp6G80Pky0om7QvQ/arcgis/rest/services/Hospitals_1/FeatureServer/0/query?where=1%3D1&outFields=*&outSR=4326&f=json"
        hospitals_num <- c(0, 2000, 4000, 6000)
        hosget <- lapply(hospitals_num, function(num) {
          hospitals_url_loop <- paste(hospitals_url, "&resultOffset=", num)
          hospitals_json <- fromJSON(hospitals_url_loop)
          hospitals_df <- as.data.frame(hospitals_json$features$attributes)})
        
        hospitals_df <- do.call(rbind, hosget)},
      
      error = function(hosp_e){
              hosp_e <- simpleError("Error: Hospital data has not been collected due to an error. Please ensure that the number loop is correct.")
              print(hosp_e)},
      
      warning = function(hosp_w){
              hosp_w <- simpleWarning("Warning: Hospital data has not been collected. Please ensure that the number loop is correct.")
              print(hosp_w)},
        
      finally = {
              hospitals_log <- (paste("Hospitals Code Has Been Executed on ", Sys.Date()))
              hosp_error <- if (exists("hosp_e") == FALSE) {
                      ("No Error")}
              hosp_warning <- if (exists("hosp_w") == FALSE) {
                      ("No Warning")}
              print(paste(hospitals_log, ", ", hosp_error, ", ", hosp_warning))
              hospitals_output = list(hospitals_log, hosp_error, hosp_warning, paste("URL: ", hospitals_url))
              rm(hospitals_log, hosp_warning, hosp_error, hospitals_num, hosget, hospitals_url)
                })

When I create an error (e.g., adding an extra "l" to hospitals_url) my simpleerror prints (<simpleError: Error: Hospital data has not been collected due to an error. Please ensure that the number loop is correct.>), but is not stored as a variable and added to the list. How can I change my code to accomplish this?

I have looked at the following (here and here) but am now more confused... :(

wcbrown
  • 157
  • 7
  • 1
    Since your `tryCatch` encompasses the entire process, any error or warning in any step interrupts the process and does not re-enter it. I think you need `tryCatch` to encompass the code inside of your `lapply` anonfunc. – r2evans Aug 25 '21 at 19:00
  • @r2evans, could you provide an example of what you mean? It may be useful if I provide context on my anticipated outcome. I'm looking to get the dataframe, as well as a list of containing the designated error/no error and warning/no warning messages for logging purposes. – wcbrown Aug 25 '21 at 19:02
  • Another problem besides what @r2evans identified is that `hosp_e` and `hosp_w` are local variables in the error and warning functions. If you want something saved, you need to create a place to save it outside of those functions, and have those functions store something there. – user2554330 Aug 25 '21 at 19:05
  • @user2554330 Could you please explain that further with an example or a link to another SO question/answer? – wcbrown Aug 25 '21 at 19:07

1 Answers1

3

Here's a version that puts the tryCatch within the loop, and saves the errors and warnings that were generated:

  hospitals_url <- "https://services1.arcgis.com/Hp6G80Pky0om7QvQ/arcgis/rest/services/Hospitals_1/FeatureServer/0/query?where=1%3D1&outFields=*&outSR=4326&f=json"
  hospitals_num <- c(0, 2000, 4000, 6000)
  hosp_e <- list()
  hosp_w <- list()
  
  hosget <- lapply(hospitals_num, function(num) {
    tryCatch({
      hospitals_url_loop <- paste(hospitals_url, "&resultOffset=", num)
      hospitals_json <- fromJSON(hospitals_url_loop)
      hospitals_df <- as.data.frame(hospitals_json$features$attributes)},
      error = function(e){
        hosp_e <<- c(hosp_e, list(e))
        print(e)},
      
      warning = function(w){
        hosp_w <<- c(hosp_w, list(w))
        print(w)}     
      ) })
    
  hospitals_df <- do.call(rbind, hosget)

I didn't adapt your finally code to this rearrangement; I'll leave that to you. But at the end, hosp_e will be a list holding all the errors, and hosp_w will be a list holding all the warnings.

user2554330
  • 37,248
  • 4
  • 43
  • 90
  • That was exactly what I was looking for, thank you. I didn't realize that I had to initialize the error and warning list beforehand. I appreciate it! – wcbrown Aug 25 '21 at 19:21
  • 1
    An alternative is to `return(e)` (and `w`) within the `error=`/`warning=` blocks, then after-the-fact `hosp_e <- Filter(function(z) inherits(z, "error"), hosget)` (and then `inherits(z, "warning")`). That way, you don't have to `<<-` go outside scope. Get the "real" data then with `Filter(function(z) !inherits(z, c("error", "warning")), hosget)`. – r2evans Aug 25 '21 at 20:29
  • @r2evans: that's a nice idea. It might cause trouble with the warnings though: you would lose the real answer, wouldn't you? – user2554330 Aug 25 '21 at 20:42
  • 1
    I think you're already losing the answer by `tryCatch`ing the warning. You'd need to use `muffleWarning` (instead) to capture a warning and continue processing, I believe. – r2evans Aug 25 '21 at 20:59