Updated docs with Croissant information.

[Reikyo]

No description provided.

[Reikyo]

Hi @amercader, just moving the conversation from #328 to here.

Many thanks for the changes to enable Croissant features as a plugin, and to provide a dedicated endpoint. I synced with these latest changes and tried them out, and all works as expected. I wouldn't have so easily figured out what's needed to enable these new features, so your work is a big help, and of course it then naturally fits in with your desired structure too.

I looked at the existing docs, and thought that maybe the Croissant info could better sit in the existing files rather than a new file. Firstly, the schema info naturally sits alongside other schema info in getting-started.md (FYI I notice that reference to dcat_us_recommended.yaml and dcat_us_full.yaml is currently missing here, in case it should be updated.) Secondly, the structured data info naturally sits alongside other structured data info in google-dataset-search.md, so I wrote a modified version of that file which discusses the structured_data plugin and the croissant plugin side-by-side, seeing as they are so similar in use and effect. I provided a couple of complete new examples there using the default CKAN data entry forms, and manually adjusted the output for the examples so that the data appears to sit at http://demo.ckan.org, as per the previous example.

Finally, when producing the new examples just mentioned, I noticed that the schemaorg profile currently outputs the same schema:contactPoint info twice. Looking at schemaorg.py, this is because of two calls to self._agent_graph, once for publisher and once for creator, even though it picks up the same contact info each time. Not sure if this is desired behaviour, so thought I'd flag here.

Please let me know what you think of the above when you have time.

[amercader]

Hi @Reikyo thanks for this, see below:

• I totally get that croissant and structured_data are really similar implementation-wise but I still think it's valuable to separate them in the docs as they target two different users and we want to have a single place to point people interested in CKAN - Croissant integration. You added all the parts that should be in the docs so don't worry I'll rearrange them myself. One think I want to add is a small example of actual usage of a Croissant enabled CKAN dataset, perhaps adapting the tensorflow_datasets example in the croissant package README. But alas when trying it out I got validation errors that we should address first.
• Valid output: Obviously we want the generated croissant output to be valid. Running the croissant validator in one of the CKAN dataset croissant endpoints gives the following:

mlcroissant validate --jsonld ~/Downloads/croissant.jsonld
W0117 16:00:54.340307 139754582292288 rdf.py:78] WARNING: The JSON-LD `@context` is not standard. Refer to the official @context (e.g., from the example datasets in https://github.com/mlcommons/croissant/tree/main/datasets/1.0). The different keys are: {'conformsTo', 'column', 'fileObject', 'citeAs', '@language', 'dataType', 'source', 'fileProperty', 'parentField', '@vocab', 'extract', 'includes', 'isLiveDataset', 'dct', 'jsonPath', 'md5', 'transform', 'rai', 'sc', 'regex', 'path', 'examples', 'key', 'separator', 'recordSet', 'replace', 'subField', 'data', 'references', 'repeated', 'format', 'fileSet', 'field'}

E0117 16:00:54.345736 139754582292288 validate.py:55] Found the following 1 error(s) during the validation:
  -  The current JSON-LD doesn't extend https://schema.org/Dataset

This looks pretty high level so maybe there's something obvious that needs to be tweaked in the profile?

This eventually needs to run in an automated test so we make sure the output remains valid in the future. I added the test in 7fd67fe so you can run it locally if you want but it's essentially the same as running the validator, only the test uses a dataset with all properties present.

I'm getting errors when providing the id_given fields, I need to look into that as well but if you have any ideas that would be great:

    def make_path(path: PathLike) -> abstract_path.Path:
      """Create a generic `pathlib.Path`-like abstraction.
    
      Depending on the input (e.g. `gs://`, `github://`, `ResourcePath`,...), the
      system (Windows, Linux,...), the function will create the right pathlib-like
      abstraction.
    
      Args:
        path: Pathlike object.
    
      Returns:
        path: The `pathlib.Path`-like abstraction.
      """
      is_windows = os.name == 'nt'
      if isinstance(path, str):
        uri_splits = path.split('://', maxsplit=1)
        if len(uri_splits) > 1:  # str is URI (e.g. `gs://`, `github://`,...)
          # On windows, `PosixGPath` is created for `gs://` paths
>         return _URI_PREFIXES_TO_CLS[uri_splits[0] + '://'](path)  # pytype: disable=bad-return-type
E         KeyError: '[\n  {\n    "@id": "my-custom-resource-id",\n    "@type": [\n      "http://'

/home/adria/.pyenv/versions/3.11.6/envs/ckan-ml/lib/python3.11/site-packages/etils/epath/register.py:100: KeyError

[amercader]

After much debugging I found out the issue causing the doesn't extend https://schema.org/Dataset validation error. The profile was using http://schema.org as namespace and the validator insists on https://schema.org. Anyway, that's fixed in 43840bc and now we are down to these failures, which seem fairly straightforward:

E               mlcroissant._src.core.issues.ValidationError: Found the following 3 error(s) during the validation:
E                 -  [Metadata(Test Croissant dataset) > FileObject()] At least one of these properties should be defined: ['md5', 'sha256'].
E                 -  [Metadata(Test Croissant dataset) > FileObject()] Property "https://schema.org/contentUrl" is mandatory, but does not exist.
E                 -  [Metadata(Test Croissant dataset) > FileSet(Resource 1)] Property "http://mlcommons.org/croissant/includes" is mandatory, but does not exist.

[Reikyo]

Hi @amercader - thanks for the continued efforts here, and apologies as I've been busy with another project which has diverted attention lately.

Firstly regarding documentation, yes please modify according to your preferred format and breakdown. I hope there's enough info in what I already supplied, just let me know if anything needs further clarification. I'll need to get some further input from others in the Croissant working group to be completely sure of the implementation details to be added to the docs.

Secondly I need to spend some time with those tests and validation errors to check through properly. I had a very quick look, and noted that @context wasn't treated as flexibly as was anticipated, and that the graph structure was demanded to be flat, essentially, rather than having sub-nodes embedded in a top-level @graph block. At least that was my initial interpretation. The current behaviour seems to be standard for rdflib, and so again I'll need to have a word with the rest of the team to see if it is in fact the Croissant validation procedure that needs to flex.

Please bear with me while I look into this, time is also further limited this week due to annual leave, but I'll try and get the needed conversations going at the very least.

[amercader]

No worries @Reikyo I can completely relate. There's no rush in getting this merged and I'd rather get it right. I'll also spend some time this week to work on docs and look at these errors, so I'm confident we can get it over the line soon.

[Reikyo]

@amercader Yes, agree on getting it right from launch, seeing as we've come this far already. I see what you mean with the http vs https issue for schema.org in @context. I'll have a word with the Croissant working group, as that looks like something on which the validator should perhaps be more flexible.

With that changed, I then also see other errors as I'm sure you are, which seem to be related to the nested structure that rdflib creates if there's more than one graph node. On a quick look, my first impression was that the validator is demanding a flat structure like {blah1, blah2, blah3} rather than a nested structure like [{blah1}, {blah2}, {blah3}], but the plugin is designed to give the latter (following the schemaorg.py plugin). However, seeing as the plugin and the validator both rely on rdflib, it seems to me that they should both be able to read and write the variants that this library allows for.

There could be more to it though, and I've noted that you flagged a possible issue with IDs too. I'll need a block of time to look at this all properly. As mentioned, I'm currently on leave but will aim to give this some focus next week. Thanks for your patience and continued support.

[amercader]

it seems to me that they should both be able to read and write the variants that this library allows for.

You are correct, CKAN transforms its metadata into an RDFlib graph and serializes it, ML Croissant takes this serialization and turns it into an RDFlib graph so that should work fine.

Turns out the failures were caused by the changes needed in the example and some fixes in the profile logic (36afa7d). I also introduced a change to the processors so you can pass a custom context (2fc9489 + a1390fa) so the validator stops complaining.

Anyway, the bottom line is that the validation tests are passing now 🚀

https://github.com/ckan/ckanext-dcat/actions/runs/13199110300

I think the metadata schema and the profile could be tweaked a bit more to align it better with the spec (e.g. hashes are mandatory in Croissant but not in CKAN) but what we have is an excellent first iteration that we can improve in the future.

If you can do some more testing on your end that would be great, I'd finish the docs next week.

[Reikyo]

@amercader I've had a look your recent changes individually and all makes sense. Thanks also for catching the resource_dict->subresource_dict typo in croissant.py too. In fact that prompted me to review a chunk of things in that area and the associated section of the croissant.yaml schema file. I'll make some tweaks tomorrow and push.

As you've seen, the subresources section I introduced is simply to allow people to describe the contents of container files e.g. ZIP files. A subresource can either be a cr:FileObject or a cr:FileSet. Each case has different needs, such as "includes" and "excludes" only being relevant for the latter. I've simply noted this in the help text for users, but it would be great if the presence of fields can actually be toggled based on the selection of another. So, for this example, if a subresource is flagged as cr:FileObject then "includes" and "excludes" fields wouldn't even be shown. Do you know if this kind of behaviour is possible?

Regarding certain required aspects of the Croissant spec, yes there are some differences with what comes out of vanilla CKAN when not using the bespoke Croissant schema. But we don't want to force users to change their schema because of this. As long as we have as many fields as possible mapped to a Croissant(-like) output via the profile, in as many cases as possible, then that will still be helpful and support adoption of the standard.

[amercader]

@amercader I've had a look your recent changes individually and all makes sense. Thanks also for catching the resource_dict->subresource_dict typo in croissant.py too. In fact that prompted me to review a chunk of things in that area and the associated section of the croissant.yaml schema file. I'll make some tweaks tomorrow and push.

That sounds great

As you've seen, the subresources section I introduced is simply to allow people to describe the contents of container files e.g. ZIP files. A subresource can either be a cr:FileObject or a cr:FileSet.

I was confused by this, as I was not finding any mention of sub resources in the spec. Why can't we use CKAN resources to model the Croissant Resources directly? We can apply the type property (cr:FileObject or a cr:FileSet) and the includes / excludes ones to the resource schema directly and we will avoid the complexity of the subresources. Unless I'm missing something.

[...] it would be great if the presence of fields can actually be toggled based on the selection of another. So, for this example, if a subresource is flagged as cr:FileObject then "includes" and "excludes" fields wouldn't even be shown. Do you know if this kind of behaviour is possible?

This is not well supported out the box in CKAN but I'm sure we can use a custom form snippet for the resource type field that uses some js to hide/show the includes/excludes fields when changing the value

Regarding certain required aspects of the Croissant spec, yes there are some differences with what comes out of vanilla CKAN when not using the bespoke Croissant schema. But we don't want to force users to change their schema because of this. As long as we have as many fields as possible mapped to a Croissant(-like) output via the profile, in as many cases as possible, then that will still be helpful and support adoption of the standard.

I very much agree with this approach.

[amercader]

BTW I'm trying to go one level deeper to make this more useful and expose RecordSet information. This will only be possible for data files that have been imported to the DataStore, for which the fields and data types are known (or at least inferred). I'll share my findings when I have something working.

[amercader]

@Reikyo my apologies, your comment appeared duplicated and when I deleted one of the copies stupid Github got rid of both. I'm really sorry!

[Reikyo]

@amercader Yes I originally tried to post when on the "Preview" tab but it didn't seem to work, so I went back to the "Write" tab and tried again ... but it was just being slow and it posted twice. I did actually delete the duplicate myself, but for some reason it still showed you both even though there was really only one there, so your deletion cleared everything. Basically GitHub was being a bit glitchy, but I saved a copy and here it is:

Original message (plus tweaks!):

As far as I was seeing, a resource is a distinct uploaded file. If a resource happens to be a container file (ZIP etc.), then it can have subresources which then don't require separate upload, hence aren't resources themselves, and for the purposes of this plugin we only require information about them to enrich the metadata of the associated resource.

For a particular resource, entering its subresource info on the resource's own edit page then makes it clear that subresources don't require separate upload, and also allows for automated parent-child identification in construction of the graph, which is what the "containedIn" line is all about in the croissant.py profile. Also, seeing as only the resource itself is uploaded, the hash is only needed there rather than for the individual subresources. I see you've introduced it for subresources too, but I'm not sure this is warranted. Apologies, my understanding does fall down a bit here, but this observation is supported by what's seen in the Croissant spec.

For example, in the following snippet from the spec, flores200_dataset.tar.gz is a top-level resource and is the only thing which is actually uploaded, and so the only thing with a hash to verify the download against. It contains a couple of cr:FileSet items and a couple of cr:FileObject items, none of which have hashes but all of which have the containedIn field instead, and all of which are subresources for the purposes of the CKAN plugin. The cr:FileObject items are simple files, and the cr:FileSet items are descriptions about groups of files:

{
  "@type": "cr:FileObject",
  "@id": "flores200_dataset.tar.gz",
  "description": "Flores 200 is hosted on a webserver.",
  "contentSize": "25585843 B",
  "contentUrl": "https://tinyurl.com/flores200dataset",
  "encodingFormat": "application/x-gzip",
  "sha256": "c764ffdeee4894b3002337c5b1e70ecf6f514c00"
},
{
  "@type": "cr:FileSet",
  "@id": "files-dev",
  "description": "dev files are inside the tar.",
  "containedIn": { "@id": "flores200_dataset.tar.gz" },
  "encodingFormat": "application/json",
  "includes": "flores200_dataset/dev/*.dev"
},
{
  "@type": "cr:FileSet",
  "@id": "files-devtest",
  "description": "devtest files are inside the tar.",
  "containedIn": { "@id": "flores200_dataset.tar.gz" },
  "encodingFormat": "application/json",
  "includes": "flores200_dataset/devtest/*.devtest"
},
{
  "@type": "cr:FileObject",
  "@id": "metadata-dev",
  "description": "Contains labels for the records in each line in the dev files.",
  "contentUrl": "flores200_dataset/metadata_dev.tsv",
  "containedIn": { "@id": "flores200_dataset.tar.gz" },
  "encodingFormat": "text/tsv"
},
{
  "@type": "cr:FileObject",
  "@id": "metadata-devtest",
  "description": "Contains labels for the records in each line in the devtest files.",
  "contentUrl": "flores200_dataset/metadata_devtest.tsv",
  "containedIn": { "@id": "flores200_dataset.tar.gz" },
  "encodingFormat": "text/tsv"
}

Basically, the unzipped file structure is as follows:

|- flores200_dataset/ <- The top-level unzipped root folder to which the above "includes" and "contentUrl"s are with respect to
    |- flores200_dataset/
        |- dev/
            |- file1.dev
            |- file2.dev
            |- etc.
        |- devtest/
            |- file1.devtest
            |- file2.devtest
            |- etc.
        |- metadata_dev.tsv
        |- metadata_devtest.tsv

This is, admittedly, if I've understood the spec correctly myself! One area the current approach falls short is that technically a container file can contain another container file, so a subresource can therefore be a container, and itself have subresources. Actually the issue is really just with the containedIn field (I'm thinking aloud here) ... if the user could specify the containedIn field, then they can override the default that would otherwise point to the top-level resource. This would require them to give IDs in order to specify the appropriate relationships, which could be error prone, but it's maybe better than nothing. This is another reason to have all subresource entries be done on the same edit page, rather than over separate edit pages, as then all information can be seen and edited at once for a top-level resource downwards.

So to give you a heads-up, the changes I'm looking to make are:

Resources

• Remove type field (cr:FileObject / cr:FileSet), as this will always be cr:FileObject

Subresources

• Remove hash field - please do correct my thinking above if you think I'm missing something
• Add contentUrl field (i.e. the filepath within the top-level resource)
• Add contentSize field (unlike for top-level resources which can be auto-generated, this will have to be manual)
• Add sameAs field
• Add containedIn field, and allow users to make their own ID chains as needed for the case of subresources belonging to other subresources. Some help notes will be given to support this.
• Ensure that only the needed fields are picked up for a subresource based on its cr:FileObject or cr:FileSet type, and output to the graph accordingly.

[amercader]

Thanks for the detailed answer @Reikyo , I'll have a proper read tomorrow and get back to you . IIRC I added the hash to subresources because the validator was moaning about it but I'll double check

[Reikyo]

@amercader Just a quick message to say I've made the changes mentioned yesterday and am just working through a couple of final issues. Will update again soon.

[Reikyo]

@amercader I've finalised the latest changes and pushed (this PR still includes the doc changes, so of course feel free to do whatever you need to do with those).

I found that the hash is needed for FileObjects which are top-levels resources (as expected, as these are the primary upload/downloads), but not for FileObjects which are subresources. The issue with the latter was solved by changing the output nodes from using cr:containedIn to containedIn. This could be achieved in two ways:

1. In croissant.py, insert "containedIn": "cr:containedIn" into the JSONLD_CONTEXT
   • This doesn't completely work, as it creates a new error
2. In croissant.py, change from CR.containedIn to SCHEMA.containedIn
   • This does work, but SCHEMA.containedIn actually looks deprecated, and is also intended for use with place info. The Croissant spec doesn't actually state if containedIn is from Croissant vocab or schema.org vocab. But as this method works, I've left it in. This works as it shows that the FileObject is indeed a subresource, not the primary upload/download top-level resource, and so isn't expected to have its own hash to verify a download against.

In a similar observation, FileSet output nodes used cr:excludes rather than simply excludes. This was solved by inserting "excludes": "cr:excludes" into the context, which matches the existing "includes": "cr:includes". This isn't covered by the Croissant spec, but I think this may be an oversight. Either way, this change works as intended, and doesn't introduce a new validation error.

I found that the size field must be present as a string in an output node if it's present at all, so I ensured that.

I found that FileObjects (both top-level resources and subresources) can only have one "Same as" entry, even though the Croissant spec states a cardinality of many, so I temporarily changed these multiple entry fields to single entry. Again, I think either the spec or the validator may need tweaking, they're not in sync. We'll then know which way to finalise this field.

I found that use of "@language": "en" in JSONLD_CONTEXT introduces a lot of additional nesting. Rather than giving output like:

"conformsTo": "http://mlcommons.org/croissant/1.0"

we instead have output like:

"conformsTo": {
    "@value": "http://mlcommons.org/croissant/1.0"
}

Although the latter is accepted by the validator, it does make everything more difficult to read. If you have an idea on how we can restrict this behaviour then that would be appreciated. I noticed that you've been looking into the graph structure with changes you made yesterday, so maybe there's something in that area you may already be familiar with. On that note, following your changes the graph creation now seems to be nesting subresources within the resource in the distribution block, whereas ideally all resources and subresources should sit side-by-side at the same level. Again, if you have some idea of how to get this behaviour that would be good.

Finally, I included a couple of example output files. ckan_full_dataset_croissant_2.json is the structure I was seeing when "@language": "en" was removed from JSONLD_CONTEXT, but I manually inserted it into the output to stop the validator complaining. ckan_full_dataset_croissant_3.json is the structure I was seeing when "@language": "en" was included in JSONLD_CONTEXT, and shows the extra nesting with lots of @value fields as mentioned above. Both of these outputs are from before your latest changes though. In both cases I also manually rearranged the nodes for clarity, the order is:

• dataset1
• catalog
• creator1
• creator2
• publisher1
• publisher2
• resource1
• subresource1
• subresource2

[amercader]

@Reikyo thanks

Forgot to mention that now subresources make sense, thanks for your explanation. It is true that is a bit cumbersome to deal with them with the standard CKAN support (which just takes resources into account), but it's the best we can do now.

See below for more comments:

I found that the hash is needed for FileObjects which are top-levels resources (as expected, as these are the primary upload/downloads), but not for FileObjects which are subresources. The issue with the latter was solved by changing the output nodes from using cr:containedIn to containedIn. [...]

In a similar observation, FileSet output nodes used cr:excludes rather than simply excludes. This was solved by inserting "excludes": "cr:excludes" into the context, which matches the existing "includes": "cr:includes". This isn't covered by the Croissant spec, but I think this may be an oversight. Either way, this change works as intended, and doesn't introduce a new validation error.

I'll defer to your judgement here as you know the spec better, we can always change the serialization in future versions

I found that the size field must be present as a string in an output node if it's present at all, so I ensured that.

I know, although the fix is easy I found that annoying and created an issue in the croissant repo for it a few days ago :) mlcommons/croissant#804

I found that use of "@language": "en" in JSONLD_CONTEXT introduces a lot of additional nesting. Rather than giving output like:

"conformsTo": "http://mlcommons.org/croissant/1.0"

we instead have output like:

"conformsTo": {
"@value": "http://mlcommons.org/croissant/1.0"
}

I agree that the compact form is much nicer but I don't think there's any workaround, that just seems to be how JSON-LD works if the context defines a default @language.

I'm happy to either leave it as is with the @value properties or remove the @language bit from the context that we are using and live with the warning in the validation test.

On that note, following your changes the graph creation now seems to be nesting subresources within the resource in the distribution block, whereas ideally all resources and subresources should sit side-by-side at the same level. Again, if you have some idea of how to get this behaviour that would be good.

I was trying to use framing to get a nicer output where instead of a list of entities under a @graph property we get a single dict for the sc:Dataset entity with the others embedded. But you are right, the main resource is nested in the containedIn property of the first subresource, which doesn't look nice:

• Current
• Framed

It's still a valid graph but it's not intuitive for humans so if I can't figure out how to keep resources and sub-resources side by side I'll revert it

Finally, I included a couple of example output files

That's great. The example Croissant outputs should go to the directory examples/croissant as each directory contains a particular format.
I think it would be nice that the included croissant example was the ones resulting from serializing the example CKAN dataset in examples/ckan/ckan_full_dataset_croissant.json, i.e. this one I got by exporting the serialization used in the validation test adding this:

diff --git a/ckanext/dcat/tests/profiles/croissant/test_validate.py b/ckanext/dcat/tests/profiles/croissant/test_validate.py
index e839f29..e1df0a2 100644
--- a/ckanext/dcat/tests/profiles/croissant/test_validate.py
+++ b/ckanext/dcat/tests/profiles/croissant/test_validate.py
@@ -30,6 +30,8 @@ def test_valid_output():
     croissant_dict = json.loads(
         s.g.serialize(format="json-ld", auto_compact=True, context=JSONLD_CONTEXT)
     )
+    with open("graph.jsonld", "w") as f:
+        f.write(json.dumps(croissant_dict))
 
     try:
         mlc.Dataset(croissant_dict)

Once we settle on all the serialization issues we can save the final example.

Thanks for your work on this, we are really close

[amercader]

@Reikyo I've created #339 to get everything ready to merge this feature and consolidate all changes in a single PR (it includes all your recent changes in this PR)

Have a look at the docs and let me know if you are missing something: https://docs.ckan.org/projects/ckanext-dcat/en/croissant/croissant/

[amercader]

Closing in favour of #339