Matt Dowle and Arun Srinivasan’s1 data.table package is a data.frame twin with amazing performance. Often, it’s the fastest way to operate on in-memory tabular data, period (not just in R!).
I’ve used data.table for performance reasons in projects for many years now. There are a few things to watch out for outlined in these notes (I have stumbled across both in practice). Fortunately they are not that problematic to deal with.
R’s data.frame is simply an R list with extra attributes: class = “data.frame” and row.names = (integer or character vector of row names). Data.frame column names (if any) are taken from the list names.
That is, you can make a data.frame the usual way:
x <- data.frame(v = 1:2, a = c(pi, 5))or, like this if you prefer2:
y <- structure(
list(v = 1:2, a = c(pi, 5)),
class = "data.frame",
row.names = 1:2)
all.equal(x, y)## [1] TRUER defines the number of rows of a data.frame to be the length of its row.names vector attribute. The number of columns is simply the length of the list.
This simple scheme is very flexible! Each data.frame list entry (column) is effectively a memory pointer to a native R vector. And usually, R tries to defer copying data until it must. Consider the following example run on my laptop with 8GB RAM:
system.time({
x <- runif(4e8)
y <- runif(4e8, 0, 10)
})## user system elapsed
## 15.195 2.282 17.523system.time({d <- data.frame(y, x)})## user system elapsed
## 0.000 0.001 0.000Each vector x and y consume about 3GB of RAM for a total of about 6GB. Creation of the data.frame d above happens instantly without data copy because its columns simply contain pointers to y and x. (Of course, subsequently modifying the values would induce a copy.)
Re-arranging columns in a data.frame simply re-arranges pointers and need not always copy data:
system.time({print(tail(d[, order(names(d))]))})## x y
## 399999995 0.4109185 7.576306
## 399999996 0.3578239 7.045663
## 399999997 0.3312816 7.116747
## 399999998 0.6940979 3.164837
## 399999999 0.2465026 5.793221
## 400000000 0.8413648 7.473116## user system elapsed
## 0.003 0.000 0.006That operation (ordering columns and printing the tail) ran almost instantly on my laptop.
Data.tables are blobs of data behind a reference pointer that (mostly) emulate standard data.frame structure and behavior. They are almost always faster and more efficient than standard R data.frames (and I often encourage their use), but there exist situations where plain old data.frames can be more efficient.
First, let’s run rm and gc to free up the 6GB memory used by the previous example before proceeding:
rm(list = ls())
gc()## used (Mb) gc trigger (Mb) max used (Mb)
## Ncells 462458 24.7 973298 52.0 665888 35.6
## Vcells 869100 6.7 924660767 7054.7 800959386 6110.9Now consider this:
library(data.table)
system.time({
dt <- data.table(x = runif(4e8), y = runif(4e8, 0, 10))
})## user system elapsed
## 13.979 17.280 129.837Why did that take so long? Because data.table made copies of the data, which happen to be larger than the RAM in my laptop in this case, and thus operating system swap was involved.
Somewhat more problematically, unlike base-R data.frames, simply re-ordering data.table columns sometimes induces data copies. For example:
system.time({print(tail(dt[, order(names(dt)), with = FALSE]))})## x y
## 1: 0.21629333 9.35813882
## 2: 0.42472695 0.71434654
## 3: 0.28599030 8.08345845
## 4: 0.07070132 5.94907941
## 5: 0.23969760 0.01440646
## 6: 0.28483723 8.35151906## user system elapsed
## 1.669 13.798 88.547That’s a bit of a bummer–watch out for it. The work around here is usually to try to use “the data.table way” as much as possible to solve problems.
Another brass tack with data.table that you should be aware of is that data.table function arguments are call-by-reference, unlike the standard R function argument call-by-value convention. (See, for instance, https://cran.r-project.org/doc/manuals/r-release/R-lang.html#Argument-evaluation). I’ve actually personally wrestled with hard-to-find bugs resulting from this, so it is something to watch out for. To be fair, environment function arguments are also call-by-reference in R, so it may help to think of data.tables like that.
Be aware that data.table sorts using the ‘C’ locale (ASCII, basically), which is almost certainly not the way your R session defaults to sorting things.
Here is an example:
set.seed(1)
a <- sample(c(letters, LETTERS), 10)
x <- data.table(a = a)
setkeyv(x, "a") # sort the data.table by the contents of 'a'
a <- sort(a) # sort 'a' in plain old R and compare:
cbind(x, a)## a a
## 1: G a
## 2: H d
## 3: M G
## 4: Q H
## 5: a M
## 6: d n
## 7: n Q
## 8: r r
## 9: u u
## 10: w wForcing R to use the C local makes the sort order the same:
Sys.setlocale("LC_COLLATE", "C")## [1] "C"a <- sort(a)
cbind(x, a)## a a
## 1: G G
## 2: H H
## 3: M M
## 4: Q Q
## 5: a a
## 6: d d
## 7: n n
## 8: r r
## 9: u u
## 10: w w