Try this demonstration:
library(shiny)
.log <- function(...) message(format(Sys.time(), format = "[ %H:%M:%S ]"), " ", ...)
.read <- function(path) if (file.exists(path)) return(readRDS(path))
shinyApp(
  ui = fluidPage(
    textInput("txt", "Text: "),
    actionButton("btn", "Submit"),
    tableOutput("tbl")
  ),
  server = function(input, output, session) {
    .log("hello world")
    rv <- reactiveValues()
    rv$df <- data.frame(row = 0L, word = "a", stringsAsFactors = FALSE)[0,]
    observeEvent(req(input$btn), {
      .log("submit!")
      rv$df <- rbind(rv$df,
                     data.frame(row = input$btn, word = input$txt,
                                stringsAsFactors = FALSE))
      .log("saveRDS: ", nrow(rv$df))
      saveRDS(rv$df, "local.rds")
    })
    filedata <- reactiveFileReader(1000, session, "local.rds", .read)
    output$tbl <- renderTable(filedata())
  }
)

The engineering of this app:
- I use a 
reactiveValues like you did, in order to keep the in-memory data. (Note: iteratively adding rows to a frame is bad in the long-run. If this is low-volume adding, then you're probably fine, but it scales badly. Each time a row is added, it copies the entire frame, doubling memory consumption.) 
- I pre-fill the 
$df with a zero-row frame, just for formatting. Nothing fancy here. 
observe and observeEvent do not return something you are interested in, it should be operating completely by side-effect. It does return something, but it is really only meaningful to shiny internals. 
saveRDS as you do, nothing fancy, but it works. 
- I added a 
shiny::reactiveFileReader in order to demonstrate that the file was being saved. When the shiny table shows an update, it's because (1) the data was added to the underlying frame; (2) the frame was saved to the "local.rds" file; then (3) reactiveFileReader noticed that the underlying file exists and has changed, causing (4) it to call my .read function to read the contents and return it as reactive data into filedata. This block is completely unnecessary in general, just for demonstration here. 
- I create a function 
.read for this reactiveFileReader that is resilient to the file not existing first. If the file does not exist, it invisibly returns NULL. There may be better ways to do this.