{"id":956,"date":"2024-11-15T10:18:37","date_gmt":"2024-11-15T15:18:37","guid":{"rendered":"https:\/\/www.clayford.net\/statistics\/?p=956"},"modified":"2024-11-17T09:10:46","modified_gmt":"2024-11-17T14:10:46","slug":"the-statsfilter-function","status":"publish","type":"post","link":"https:\/\/www.clayford.net\/statistics\/the-statsfilter-function\/","title":{"rendered":"The stats::filter() function"},"content":{"rendered":"

The base R function filter()<\/code> can be used to calculate moving averages. This is one of the base R functions masked when the {dplyr} package is loaded.<\/p>\n

Before we see how it works, let\u2019s create some toy data.<\/p>\n

n <- 100\r\nt <- as.Date(seq(n), origin = as.Date("01\/01\/2022", "%m\/%d\/%Y"))\r\nset.seed(1)\r\ny <- cos(pi*seq(-2,2, length.out = n)) + rnorm(n, sd = 0.5)\r\nd <- data.frame(t, y)\r\n\r\nlibrary(ggplot2)\r\nggplot(d) +\r\n  aes(t, y) +\r\n  geom_line()<\/code><\/pre>\n
<\/p>\n
\n
3 day moving average<\/h2>\n
Let\u2019s say we want to calculate a 3 day moving average. Two ways to approach this:<\/p>\n
    \n
  1. \n
    on any given day, take average of day before, current day, and next day. In other words use data from both sides<\/em> of current day.<\/p>\n<\/li>\n
  2. \n
    on any given day, take average of prior two days and current day. In other words use data from only one side<\/em> of current day.<\/p>\n<\/li>\n<\/ol>\n
    The first approach is the default of the filter()<\/code> function. The first argument is the vector of data we want to calculate the moving average for. The second argument is the filter<\/em> we want to apply to the data. These are coefficients<\/em> we apply to the data before summing. For a 3 day moving average this is a vector of three 1\/3<\/code>, which we can create using rep(1\/3, 3)<\/code>. The sides=2<\/code> argument says use both sides of the data.<\/p>\n
    d$ma3 <- stats::filter(d$y, filter = rep(1\/3, 3), sides = 2)<\/code><\/pre>\n
    Let\u2019s look at the first three values. There\u2019s a NA for day 1 because we have no data for the prior day.<\/p>\n
    d$ma3[1:3]<\/code><\/pre>\n
    ## [1]        NA 0.7735613 1.1199731<\/code><\/pre>\n
    The first 3 day moving average is 0.7735613 calculated at day 2. This is the average of days 1, 2, and 3.<\/p>\n
    mean(y[1:3])<\/code><\/pre>\n
    ## [1] 0.7735613<\/code><\/pre>\n
    Notice we can get the same result by multiplying each data point by 1\/3<\/code> and summing. This is the \u201cfilter\u201d applied to the data by the filter()<\/code> function.<\/p>\n
    sum(1\/3 * y[1:3])<\/code><\/pre>\n
    ## [1] 0.7735613<\/code><\/pre>\n
    One reason to calculate a moving average is to smooth out day-to-day variation. Below we plot the original data with the 3 day moving average superimposed.<\/p>\n
    ggplot(d) +\r\n  aes(t, y) +\r\n  geom_line(alpha = 1\/4) +\r\n  geom_line(aes(y = ma3), color  = "red")<\/code><\/pre>\n
    ## Warning: Removed 2 rows containing missing values or values outside the scale range\r\n## (`geom_line()`).<\/code><\/pre>\n
    <\/p>\n
    The other approach is to use one side of the data. That means setting sides= 1<\/code>. Now we get 2 NAs at the beginning because days 1 and 2 did not have two prior days.<\/p>\n
    d$ma3 <- stats::filter(d$y, filter = rep(1\/3, 3), sides = 1)\r\nd$ma3[1:3]<\/code><\/pre>\n
    ## [1]        NA        NA 0.7735613<\/code><\/pre>\n<\/div>\n
    \n
    odd versus even numbers<\/h2>\n
    Odd numbers are good to use for sides = 2<\/code> because we\u2019ll get an equal number of days before and after the current day. If an even number is used, \u201cmore of the filter is forward in time than backward\u201d (?filter<\/code>).<\/p>\n
    For example, consider a 6 day moving average:<\/p>\n
    d$ma6 <- stats::filter(d$y, filter = rep(1\/6, 6), sides = 2)\r\nd$ma6[1:3]<\/code><\/pre>\n
    ## [1]        NA        NA 0.9133886<\/code><\/pre>\n
    The first 6 day moving average occurs at day 3 which is the average of the current day, the 2 previous days, and the 3 following days. Notice \u201cmore of the filter is forward in time<\/em> than backward\u201d:<\/p>\n
    1 2 3<\/strong> 4 5 6<\/p>\n
    sum(1\/6 * y[1:6])<\/code><\/pre>\n
    ## [1] 0.9133886<\/code><\/pre>\n<\/div>\n
    \n
    Other functions<\/h2>\n
    Three other functions for calculating moving averages are:<\/p>\n
      \n
    1. runmean()<\/code> from the {caTools} package<\/li>\n
    2. frollmean()<\/code> from the {data.table} package<\/li>\n
    3. roll_mean()<\/code> from the {RcppRoll} package<\/li>\n<\/ol>\n
      Below I use all three to replicate the stats::filter()<\/code> result.<\/p>\n
      For caTools::runmean()<\/code> we need to specify endrule = "NA"<\/code>, otherwise it uses an algorithm to calculate means in the extremes using smaller windows than 3.<\/p>\n
      For data.table::frollmean()<\/code> we need to specify align = "center"<\/code>, which is the same as sides = 2<\/code> for filter()<\/code>. Otherwise it defaults to align = "right"<\/code> which is equivalent to sides = 1<\/code> for filter()<\/code>.<\/p>\n
      For RcppRoll::roll_mean<\/code> we need to specify fill = NA<\/code> to pad the output with missing values in the extremes. Otherwise, in this case, it returns a vector of length 98 instead of 100.<\/p>\n
      d$ma3 <- stats::filter(d$y, filter = rep(1\/3, 3), sides = 2)\r\nd$ma3_runmean <- caTools::runmean(d$y, k = 3, endrule = "NA")\r\nd$ma3_frollmean <- data.table::frollmean(d$y, n = 3, align = "center")\r\nd$ma3_rcpproll <- RcppRoll::roll_mean(d$y, n = 3, align = "center",\r\n                                      fill = NA)\r\nhead(d[,c("ma3", "ma3_runmean", "ma3_frollmean", "ma3_rcpproll")])<\/code><\/pre>\n
      ##         ma3 ma3_runmean ma3_frollmean ma3_rcpproll\r\n## 1        NA          NA            NA           NA\r\n## 2 0.7735613   0.7735613     0.7735613    0.7735613\r\n## 3 1.1199731   1.1199731     1.1199731    1.1199731\r\n## 4 1.1049153   1.1049153     1.1049153    1.1049153\r\n## 5 1.0532159   1.0532159     1.0532159    1.0532159\r\n## 6 0.8003626   0.8003626     0.8003626    0.8003626<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"
      The base R function filter() can be used to calculate moving averages. This is one of the base R functions… Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13],"tags":[85,86],"class_list":["post-956","post","type-post","status-publish","format-standard","hentry","category-using-r","tag-filter","tag-moving-averages"],"_links":{"self":[{"href":"https:\/\/www.clayford.net\/statistics\/wp-json\/wp\/v2\/posts\/956","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.clayford.net\/statistics\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.clayford.net\/statistics\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.clayford.net\/statistics\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.clayford.net\/statistics\/wp-json\/wp\/v2\/comments?post=956"}],"version-history":[{"count":4,"href":"https:\/\/www.clayford.net\/statistics\/wp-json\/wp\/v2\/posts\/956\/revisions"}],"predecessor-version":[{"id":960,"href":"https:\/\/www.clayford.net\/statistics\/wp-json\/wp\/v2\/posts\/956\/revisions\/960"}],"wp:attachment":[{"href":"https:\/\/www.clayford.net\/statistics\/wp-json\/wp\/v2\/media?parent=956"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.clayford.net\/statistics\/wp-json\/wp\/v2\/categories?post=956"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.clayford.net\/statistics\/wp-json\/wp\/v2\/tags?post=956"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}