passing interpretation info to uproot.iterate

[shallmann]

Hi, I am trying to find out how to use uproot.iterate on existing root files containing branches for which the correct interpretation is not discovered automatically.

On a single file, I can read the data doing something like:

my_interpretation = uproot.interpretation.jagged.AsJagged(uproot.interpretation.numerical.AsDtype('>i4'), header_bytes=6)
file = uproot.open("testfile.root")
branch = file['some_tree']['some_branch']
branch.array(interpretation=my_interpretation)

Is there a straightforward possibility to pass the interpretation information for the branches throwing errors (*) also to uproot.iterate?

Thanks a lot for suggestions!

(*) I am getting errors like: ValueError: basket 0 in tree/branch /some_tree:some_branch has the wrong number of bytes (266) for interpretation AsStridedObjects(Model_TTimeStamp_v1)

[jpivarski]

I checked to see if it might just be undocumented, but no, there currently isn't a way to pass custom interpretations to uproot.iterate. The TBranch.array method takes an interpretation as its first argument, but the interface to TTree.arrays, TTree.iterate, and uproot.iterate is complicated by the fact that the strings might not be branch names directly, but might be expressions to compute from branches (to enable aliases). At best, we could add a dict argument that maps branch name (not an expression) to custom interpretations. There are a few places in the code that would have to change to pass that down.

But still, it would be a complicated interface. If a branch name is misspelled in that dict, it just wouldn't take effect. Uproot 3 combined the list of branches to read with custom interpretations by accepting it as a dict, rather than a list, but those strings were real branch names, not expressions, in Uproot 3, so such an interface made more sense. Here, it would have to be a separate dict, which increases the possibility of error. Also, passing custom interpretations is uncommon. I see why you're doing it for this timestamp—you happen to know that timestamps are just 4-byte integers, but the streamer is being interpreted otherwise.

Actually, can you print out the

file.file.show_streamers("TTimeStamp")

to see how it's being interpreted? Looking at its header file,

https://github.com/root-project/root/blob/87a998d48803bc207288d90038e60ff148827664/core/base/inc/TTimeStamp.h#L81-L82

it ought to be a pair of 32-bit integers, not just one 32-bit integer.

266 bytes doesn't evenly divide into either 4 or 8. Its prime factorization is 2 × 7 × 19. I doubt the struct size is 2 or 7 (or anything larger), so maybe the basket this is obtaining is wrong somehow? The timestamp is in a branch by itself, right?

[shallmann]

Thanks for your reply. Then at least I did not miss anything obvious.

I hope indeed that custom interpretations are needed only in rare use cases in general - it is very cumbersome..., for me the timestamp branch was not the only one causing these kind of errors after all, I got something similar with the trigger mask... I think I can live with not using the uproot.iterate for the reader of these files.

For the 266 bytes interpretation, I just tried until I got out no errors and the correct time values. Anyway, the file.file.show_streamers("TTimeStamp") gives me

TTimeStamp (v1)
    fSec: int (TStreamerBasicType)
    fNanoSec: int (TStreamerBasicType)

and the times are in a separate branch file['EventHeader./EventHeader.fTime']

<TBranchElement 'EventHeader.fTime' at 0x7ff989ea0dc0>

If as you say the basket is wrong somehow, I have no means of fixing that when I just want to use externally provided files, right?

[jpivarski]

That TTimeStamp streamer agrees with the C++ header file; if you're interpreting the branch as ">i4", then you ought to be getting fSec and fNanoSec interleaved. Are you sure you don't want

uproot.interpretation.jagged.AsJagged(uproot.interpretation.numerical.AsDtype(
    [("fSec", ">i4"), ("fNanoSec", ">i4")]
), header_bytes=6)

?

[shallmann]

Oh that is indeed much more convenient to use, yes. Thanks a lot!

That leaves me with only one more problematic one to solve...

     TSnTriggerSetup version 3 as uproot.dynamic.Model_TSnTriggerSetup_v3 (184 bytes)
        (base): <TObject None None at 0x7ff98e86e850>
        TMap version 3 as uproot.dynamic.Model_TMap_v3 (139 bytes)
            (base): <TCollection (version 3) at 0x7ff98e86e9a0>
            fTable: None
            TSnTriggerSetup version 3 as uproot.dynamic.Model_TSnTriggerSetup_v3 (184 bytes)
                (base): <TObject None None at 0x7ff98e86edc0>
                TMap version 3 as uproot.dynamic.Model_TMap_v3 (139 bytes)
                    (base): <TCollection (version 3) at 0x7ff98e86efd0>
                    fTable: None
Base classes for TMap: (TCollection)
Members for TMap: (fTable)

expected 139 bytes but cursor moved by 12 bytes (through TMap)

[jpivarski]

There's something very peculiar about the fact that the TSnTriggerSetup and TMap deserializations are nested within themselves, and the sizes of the nested ones are not smaller. The number of bytes should include all constituents, which have non-zero size headers, so nested objects should have strictly fewer bytes than the outer ones. This might have accidentally gone into some recursive loop, which wasn't an infinite loop because it stopped at the first number of bytes check that failed. (We might be looking at the same TSnTriggerSetup and TMap objects—the reading position in the file might have jumped back.) This is likely a bug that I'll need to see a copy of the original file to diagnose (GitHub Issue, minimally reproducing example, and a small or one-event file that can be public).