Determine the total buffer size before starting to read LH5 data from a list of files

[gipert]

Should we avoid resizing buffers every time a new file is read-appended and instead already allocate a buffer of the right total size? Would this approach be less memory hungry and faster?

[iguinn]

I think this would help with both memory and speed, but I'm not sure how we get the total size first without slowing things down (at least with core.read and store.read). I think this is a great idea for the data loader, as we could store sizes in the file DB.

Another option would be to allow the memory buffers for LGDO objects to be larger than the size. Then instead of increasing the size every time we add a new file, we could, say, double the size when the buffer is full. In other words, we could turn our arrays into C++ vectors.