Input formats

[igorbrigadir]

Input format should accept unflattened as well as flattened data, but also a list of IDs for basic counts (ids encode a timestamp so we can give back counts over time for a list of IDs only)

Ideally we should also support compressed files, and a directory of files. Eg: your dataset can be is 1 big xz compressed file with 1 tweet per line, or a directory of gz files with 1 file per hour or per day, etc.

It would also be nice to support v1.1 format tweets but this would be a lower priority than v2 formats.