What you need is a rolling mean, in the argument k (4 in my example) you provide an integer width of the rolling window. Check the documentation page for the rollmean function of the zoo package, ?rollmean.
zoo
library(zoo) 
library(dplyr)
df <- data.frame(number = 1:20)
  df %>% mutate(rolling_avg = rollmean(number, k = 4 , fill = NA, align = "right"))
RcppRoll
library(RcppRoll)
df %>% mutate(rolling_avg = roll_mean(number, n = 4, fill = NA, align = "right"))
Output
   number rolling_avg
1       1          NA
2       2          NA
3       3          NA
4       4         2.5
5       5         3.5
6       6         4.5
7       7         5.5
8       8         6.5
9       9         7.5
10     10         8.5
11     11         9.5
12     12        10.5
13     13        11.5
14     14        12.5
15     15        13.5
16     16        14.5
17     17        15.5
18     18        16.5
19     19        17.5
20     20        18.5
Using the other vector you provided in the comments:
  df <- data.frame(number = c(1,-3,5,4,3,2,-4,5,6,-4,3,2,3,-4,5,6,6,3,2))
  df %>% mutate(rolling_avg = rollmean(number, 4, fill = NA, align = "right"))
Output
   number rolling_avg
1       1          NA
2      -3          NA
3       5          NA
4       4        1.75
5       3        2.25
6       2        3.50
7      -4        1.25
8       5        1.50
9       6        2.25
10     -4        0.75
11      3        2.50
12      2        1.75
13      3        1.00
14     -4        1.00
15      5        1.50
16      6        2.50
17      6        3.25
18      3        5.00
19      2        4.25