Forum: help

Sparse random effect design matrix [ Reply ]
By: Christopher Stanton on 2013-01-03 02:21

I have an extremely large dataset with around 5 million observations on individual workers performing a variety of tasks. An observation is recorded at the worker-task level, and workers only perform 1 task per observation. There are around 23,000 workers and 2,000 tasks. I am able to estimate models with a random effect for tasks, but I am unable to estimate models with random effects for workers. When I attempt to estimate a model with worker random effects, I get a memory error.
Because the sparse storage requirement for tasks and workers should be exactly the same, I suspect the package is attempting to load a dense matrix in memory and then convert to a sparse matrix. Is it possible to pass a sparse matrix directly?

Any help would be greatly appreciated.

Chris