rbindlist throws errors when combining tables with many columns

[rickhelmus]

Hello,

A bit of a corner case, but still relevant: when rbind()/rbindlist() is used to combine tables with many columns it fails:

tab1 <- as.data.table(matrix(0L, nrow = 10, ncol = 1E6))
tab2 <- as.data.table(matrix(0L, nrow = 10, ncol = 1E6))
m <- rbind(tab1, tab2)

Fails with: Error: protect(): protection stack overflow.

The tables are big, but not that big :-)

Confirmed on Windows with latest devel of data.table and on Linux with 1.18.2.1.

Thanks,
Rick

[MichaelChirico]

Interesting, even though it's an edge case, it hints that we're not beeing aggressive enough in UNPROTECT() in some loop which probably has performance implications for moderately wide tables.

For now, you can work around by batching. My understanding is there's no way to poll the protect stack size, but playing around I see it to be around 50,000 columns.

batch_max = 40000
JJ = ncol(tab1)
batches <- split(seq_len(JJ), rep(seq_len(JJ/batch_max), each=batch_max, length.out=JJ))
cbindlist(lapply(batches, \(batch) rbind(tab1[, batch, with=FALSE], tab2[, batch, with=FALSE])))

I find that still outperforms coercing to data.frame and using rbind.data.frame.

[aitap]

Interesting. The protect stack overflow usually happens when computing column classes:

data.table/src/rbindlist.c

Lines 329 to 331 in 0b57d38

	if (!R_compute_identical(PROTECT(getAttrib(thisCol, R_ClassSymbol)),
	PROTECT(getAttrib(firstCol, R_ClassSymbol)),
	0) && !ignoreattr) {

...but the actual runaway process is the way longestLevels is protected anew on every iteration of the per-column loop (#7585):

data.table/src/rbindlist.c

Lines 281 to 282 in 0b57d38

	SEXP longestLevels=R_NilValue; // just for ordered factor
	PROTECT_WITH_INDEX(longestLevels, &ILongestLevels); nprotect++;

nprotect keeps increasing, filling the protection stack with (in this case) R_NilValue.