Include isArray and arrayShape

[ccl-core]

Addresses: #649

Update: the final implementation uses:

{
          "@type": "cr:Field",
          "@id": "recordset/fieldname",
          "isArray": true,
          "arrayShape": "1,2,1"
       ...
}

======

Note that this is a bit different than the original proposal in #649 :

{
          "@type": "cr:Field",
          "@id": "recordset/fieldname",
          "isArray": true,
          "arrayShape": [
            1,
            2,
            1
          ]
       ...
}

This is a bit difficult for us to process using mlcroissant... When we create the graph from the jsonld data, we would loose the order information; moreover, as we create a triple for each element in the arrayShape, and triples need to be unique, we would loose dimensions which have the same size.

As an alternative I propose to use a @list, so we keep the order info more nicely and we don't loose data:

{
          "@type": "cr:Field",
          "@id": "recordset/fieldname",
          "isArray": true,
          "arrayShape": {
            "@list": [
              1,
              2,
              1
            ]
          },
       ...
}

WDYT?

In this PR:

• Introduces isArray and arrayShape
• Includes one example dataset (datasets/1.0/huggingface-baratilab-flow3d/metadata.json) which is a simplified version of https://huggingface.co/datasets/baratilab/Flow3D-V0-Ti-6Al-4V_fast_test (less fields, only one config) which includes the new fields.
• Updates all contexts

Still TODO: Remove repeated from all datasets and use isArray and arrayShape instead.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[marcenacp]

nit: references coming after source was more readable.

[ccl-core]

Yeah I agree... I don't know why it got changed, maybe it started to order those alphabetically? I could change it back manually, shall I do this?

[marcenacp]

An array of strings used to control the order. We shouldn't do it manually but programmatically. But this is not prio at all - so we can probably let it like this if we don't have time.

[marcenacp]

Resolved.

[benjelloun]

As we discussed offline, arrayShape can be represented more compactly as a string that contains space separated dimensions, e.g., "1 2 -1 3", where -1 corresponds to a dimension where the length is not specified. Even though this approach is encoding structured information in a string, it's unambiguous and easy to parse, so I think it's acceptable. We already do accept such input in other places, e.g., for bounding boxes.

[ccl-core]

@marcenacp WDYT if I add a "LATEST_VERSION" attr to CroissantVersion and is_latest_version method to Context in /usr/local/google/home/stanzelc/croissant/python/mlcroissant/mlcroissant/_src/core/context.py ? Then we could add the new elements to the context in rdf.py, include 1.1 to the versions we use for testing in /usr/local/google/home/stanzelc/croissant/python/mlcroissant/mlcroissant/_src/tests/versions.py

See example in cb40bea

[marcenacp]

LGTM, thanks! Just a few nits.