diff --git a/README.md b/README.md
index 10514d880..f9d4f52d6 100644
--- a/README.md
+++ b/README.md
@@ -9,13 +9,33 @@ The mission of the W3C Data Privacy Vocabularies and Controls CG (DPVCG) is to d
License: All work produced by DPVCG and provided through this repo or elsewhere is provided by contributors under the [W3C Document License](https://www.w3.org/Consortium/Legal/2015/doc-license). A copy of the license is provided in the [LICENSE.md](./LICENSE.md) file.
-Outputs:
- * Primer: Introduction to the Data Privacy Vocabulary - [https://w3id.org/dpv/primer](https://w3id.org/dpv/primer)
- * Data Privacy Vocabulary (DPV) - [https://w3id.org/dpv](https://w3id.org/dpv)
- * GDPR terms for Data Privacy Vocabulary (DPV-GDPR) [https://w3id.org/dpv/dpv-gdpr](https://w3id.org/dpv/dpv-gdpr)
+## DPV Family of Documents
+
+**Core documents:**
+
+* [Primer for Data Privacy Vocabulary](https://www.w3id.org/dpv/primer): introductory document to DPV concepts
+
+> **Note:** Newcomers to the DPV are **strongly recommended to first read through the Primer** to familiarise themselves with the semantics and concepts of DPV.
+
+* [Data Privacy Vocabulary (DPV) Specification](https://www.w3id.org/dpv): (this document) formal and normative description of DPV and its concepts.
+
+**Serialisations of DPV:**
+
+- [Data Privacy Vocabulary serialised using SKOS+RDFS (DPV-SKOS)](https://www.w3id.org/dpv/dpv-skos): serialisation of DPV with SKOS and RDFS semantics
+- [Data Privacy Vocabulary serialised using OWL2 (DPV-OWL)](https://www.w3id.org/dpv/dpv-owl): serialisation of DPV using OWL2 semantics
+
+**Extensions to Concepts:**
+- [GDPR Extension for Data Privacy Vocabulary (DPV-GDPR)](https://www.w3id.org/dpv/dpv-gdpr): extends DPV concepts for GDPR
+- [Personal Data Categories Extension for Data Privacy Vocabulary (DPV-PD)](https://www.w3id.org/dpv/dpv-pd)
+- [Legal Extension providing Jurisdictions, Laws, and Authorities for Data Privacy Vocabulary (DPV-LEGAL)](https://www.w3id.org/dpv/dpv-legal)
+- [Guidelines for Adoption and Use of DPV](https://w3id.org/dpv/guides): [Using DPV in OWL2](https://w3id.org/dpv/guides/dpv-owl)
+
+**Other Resources:**
+- [NACE Taxonomy serialised in RDFS](https://www.w3id.org/dpv/dpv-nace)
-Publication:
- * Pandit H.J. et al. (2019) Creating a Vocabulary for Data Privacy. In: Panetto H., Debruyne C., Hepp M., Lewis D., Ardagna C., Meersman R. (eds) On the Move to Meaningful Internet Systems: OTM 2019 Conferences. OTM 2019. Lecture Notes in Computer Science, vol 11877. Springer, Cham. https://doi.org/10.1007/978-3-030-33246-4_44
+**Related Links**
+* For a general overview of the Data Protection Vocabularies and Controls Community Group \[[DPVCG](#bib-dpvcg "W3C Data Privacy Vocabularies and Controls Community Group (DPVCG)")\], its history, deliverables, and activities - refer to [DPVCG Website](https://www.w3.org/community/dpvcg/).
+* **Cite as:** The peer-reviewed article “[Creating A Vocabulary for Data Privacy](https://link.springer.com/chapter/10.1007%2F978-3-030-33246-4_44)” presents a historical overview of the DPVCG, and describes the methodology and structure of the DPV along with describing its creation. An open-access version can be accessed [here](http://hdl.handle.net/2262/91581), [here](http://doras.dcu.ie/23801/), and [here](https://aic.ai.wu.ac.at/~polleres/publications/pand-etal-2019ODBASE.pdf).
## Data Privacy Vocabulary (DPV)
@@ -34,22 +54,6 @@ The vocabulary provides terms to describe:
The namespace for DPV terms is `http://www.w3id.org/dpv#` with suggested prefix `dpv`. The IRI for DPV is currently redirected to serve the files hosted in this repository from GitHub pages i.e. `https://w3c.github.io/dpv/dpv/` (thanks to @bert-github for setting this up). Content-negotiation should therefore be supported for all files/serialisations of the DPV and its modules.
-## DPV Family of Documents
-
-* [DPV-Primer](https://www.w3id.org/dpv/dpv-primer): The Primer serves as an introductory document to DPV and provides an overview of its concepts.
-* [DPV](https://www.w3id.org/dpv/): The DPV Specification is the formal and normative description of DPV and its concepts. It provides a serialisation of the concepts as a taxonomy using SKOS.
-
-**Extensions to Concepts**
-
-* [DPV-GDPR](https://www.w3id.org/dpv/dpv-gdpr): Extension expands on the DPV vocabulary to provide the specific legal basis, rights, and concepts defined within GDPR. It expands or specialises the concepts in DPV for use with GDPR.
-* [DPV-PD](https://www.w3id.org/dpv/dpv-pd): Extension to the DPV providing a taxonomy of personal data categories.
-* [DPV-NACE](https://www.w3id.org/dpv-nace): [NACE](https://ec.europa.eu/eurostat/ramon/nomenclatures/index.cfm?TargetUrl=LST_NOM_DTL&StrNom=NACE_REV2) industry standard classification system used in the EU serialised in RDFS
-
-**Serialisations of DPV**
-
-* [DPV-SKOS](https://www.w3id.org/dpv/dpv-skos): A serialisation of the DPV using RDFS and SKOS to enable its use as a lightweight ontology for modelling or annotating information. This serialisation can be used in cases where the DPV is to be used as a 'data model' or 'schema' without formal logical assertions. It is suitable in cases where simple(r) inferences are required, or where the strict interpretation or restrictions of OWL are not needed, or the rules/constraints are expressed in another language (e.g. SWRL or SHACL).
-* [DPV-OWL](https://www.w3id.org/dpv/dpv-owl): a serialisation of the DPV specification using OWL language. It should be used where the additional semantic relationships offered by OWL (based on description logic) are needed for modelling knowledge and describing desired inferences. OWL offers more powerful (and complex) features compared to RDFS regarding expression of information and its use to produce desired inferences in a coherent manner.
-
### DPV and Modules
The term 'DPV' represents the entire vocabulary - with its concepts and terms as defined in the specification. Serialisations for this in `rdf+xml`, `json-ld`, and `turtle` are provided. The 'modules' in DPV are separate files for each of the hierarchies and concept taxonomies - for example 'purposes'. These are defined in the `rdf` folder with serialisations for each module. The `core` or `base` vocabulary or ontology is defined containing the top-level classes and data model (i.e. `PersonalDataHandling`).
diff --git a/documentation-generator/001_download_vocab_in_csv.py b/documentation-generator/001_download_vocab_in_csv.py
index 57af68fe3..2d5955579 100755
--- a/documentation-generator/001_download_vocab_in_csv.py
+++ b/documentation-generator/001_download_vocab_in_csv.py
@@ -27,6 +27,8 @@
'Purpose_properties',
'Context',
'Context_properties',
+ 'Risk',
+ 'Risk_properties',
'Processing',
'Processing_properties',
'ProcessingContext',
@@ -35,12 +37,15 @@
'TechnicalOrganisationalMeasure_properties',
'Entities',
'Entities_properties',
+ 'Entities_Authority',
+ 'Entities_Authority_properties',
'Entities_LegalRole',
'Entities_LegalRole_properties',
'Entities_Organisation',
- 'Entities_DataSubjects',
- 'Jurisdictions',
- 'Jurisdictions_properties',
+ 'Entities_DataSubject',
+ 'Entities_DataSubject_properties',
+ 'Jurisdiction',
+ 'Jurisdiction_properties',
'LegalBasis',
'LegalBasis_properties',
'Consent_properties',
@@ -48,6 +53,13 @@
'GDPR_LegalBasis',
'GDPR_LegalRights',
'GDPR_DataTransfers',
+ # DPV-Legal
+ 'legal_properties',
+ 'legal_Locations',
+ 'legal_Laws',
+ 'legal_Authorities',
+ 'legal_EU_EEA',
+ 'legal_EU_Adequacy',
)
from urllib import request
@@ -57,8 +69,11 @@ def download_csv(document_id, sheet_name, save_path='./vocab_csv'):
'''Download the sheet and save to given path'''
url = GOOGLE_EXPORT_LINK % (document_id, sheet_name)
print(f'Downloading {sheet_name}...', end='')
- request.urlretrieve(url, f'{save_path}/{sheet_name}.csv')
- print('DONE')
+ try:
+ request.urlretrieve(url, f'{save_path}/{sheet_name}.csv')
+ print('DONE')
+ except Exception as E:
+ print(f'ERROR :: {E}')
if __name__ == '__main__':
diff --git a/documentation-generator/002_parse_csv_to_rdf.py b/documentation-generator/002_parse_csv_to_rdf.py
index 97ce5e378..ac6312c16 100755
--- a/documentation-generator/002_parse_csv_to_rdf.py
+++ b/documentation-generator/002_parse_csv_to_rdf.py
@@ -16,6 +16,9 @@
# CSV FILES are in IMPORT_CSV_PATH
# RDF FILES are written to EXPORT_DPV_MODULE_PATH
+
+# CONTRIBUTION: If anyone is willing to turn these scripts into
+# an equivalent RML/R2RML or similar mappings, please let us know
########################################
IMPORT_CSV_PATH = './vocab_csv'
@@ -24,6 +27,8 @@
EXPORT_DPV_GDPR_PATH = '../dpv-gdpr'
EXPORT_DPV_GDPR_MODULE_PATH = '../dpv-gdpr/modules'
EXPORT_DPV_PD_PATH = '../dpv-pd'
+EXPORT_DPV_LEGAL_PATH = '../dpv-legal'
+EXPORT_DPV_LEGAL_MODULE_PATH = '../dpv-legal/modules'
# serializations in the form of extention: rdflib name
RDF_SERIALIZATIONS = {
@@ -76,6 +81,7 @@
DPV_NACE = Namespace('https://w3id.org/dpv/dpv-nace#')
DPV_GDPR = Namespace('https://w3id.org/dpv/dpv-gdpr#')
DPV_PD = Namespace('https://w3id.org/dpv/dpv-pd#')
+DPV_LEGAL = Namespace('https://w3id.org/dpv/dpv-legal#')
DPVS = Namespace('https://w3id.org/dpv/dpv-skos#')
DPVS_GDPR = Namespace('https://w3id.org/dpv/dpv-skos/dpv-gdpr#')
DPVS_PD = Namespace('https://w3id.org/dpv/dpv-skos/dpv-pd#')
@@ -111,6 +117,7 @@
'dpv-nace': DPV_NACE,
'dpv-gdpr': DPV_GDPR,
'dpv-pd': DPV_PD,
+ 'dpv-legal': DPV_LEGAL,
'dpvs': DPVS,
'dpvs-gdpr': DPVS_GDPR,
'dpvs-pd': DPVS_PD,
@@ -343,25 +350,30 @@ def serialize_graph(graph, filepath):
'classes': f'{IMPORT_CSV_PATH}/PersonalData.csv',
'properties': f'{IMPORT_CSV_PATH}/PersonalData_properties.csv',
'model': 'ontology',
- 'topconcept': DPV.PersonalData,
+ 'topconcept': BASE['PersonalData'],
},
'purposes': {
'classes': f'{IMPORT_CSV_PATH}/Purpose.csv',
'properties': f'{IMPORT_CSV_PATH}/Purpose_properties.csv',
'model': 'taxonomy',
- 'topconcept': DPV.Purpose,
+ 'topconcept': BASE['Purpose'],
},
'context': {
'classes': f'{IMPORT_CSV_PATH}/Context.csv',
'properties': f'{IMPORT_CSV_PATH}/Context_properties.csv',
'model': 'taxonomy',
- 'topconcept': DPV.Context,
+ 'topconcept': BASE['Context'],
+ },
+ 'risk': {
+ 'classes': f'{IMPORT_CSV_PATH}/Risk.csv',
+ 'properties': f'{IMPORT_CSV_PATH}/Risk_properties.csv',
+ 'model': 'ontology',
},
'processing': {
'classes': f'{IMPORT_CSV_PATH}/Processing.csv',
'properties': f'{IMPORT_CSV_PATH}/Processing_properties.csv',
'model': 'taxonomy',
- 'topconcept': DPV.Processing,
+ 'topconcept': BASE['Processing'],
},
'processing_context': {
'classes': f'{IMPORT_CSV_PATH}/ProcessingContext.csv',
@@ -372,30 +384,52 @@ def serialize_graph(graph, filepath):
'classes': f'{IMPORT_CSV_PATH}/TechnicalOrganisationalMeasure.csv',
'properties': f'{IMPORT_CSV_PATH}/TechnicalOrganisationalMeasure_properties.csv',
'model': 'taxonomy',
- 'topconcept': DPV.TechnicalOrganisationalMeasure,
+ 'topconcept': BASE['TechnicalOrganisationalMeasure'],
},
'entities': {
'classes': f'{IMPORT_CSV_PATH}/Entities.csv',
'properties': f'{IMPORT_CSV_PATH}/Entities_properties.csv',
'model': 'ontology',
- 'topconcept': DPV.Entity,
+ 'topconcept': BASE['Entity'],
+ },
+ 'entities_authority': {
+ 'classes': f'{IMPORT_CSV_PATH}/Entities_Authority.csv',
+ 'properties': f'{IMPORT_CSV_PATH}/Entities_Authority_properties.csv',
+ 'model': 'ontology',
+ 'topconcept': BASE['Authority'],
+ },
+ 'entities_legalrole': {
+ 'classes': f'{IMPORT_CSV_PATH}/Entities_LegalRole.csv',
+ 'properties': f'{IMPORT_CSV_PATH}/Entities_LegalRole_properties.csv',
+ 'model': 'ontology',
+ },
+ 'entities_organisation': {
+ 'classes': f'{IMPORT_CSV_PATH}/Entities_Organisation.csv',
+ 'model': 'ontology',
+ 'topconcept': BASE['Organisation'],
},
- 'jurisdictions': {
- 'classes': f'{IMPORT_CSV_PATH}/Jurisdictions.csv',
- 'properties': f'{IMPORT_CSV_PATH}/Jurisdictions_properties.csv',
+ 'entities_datasubject': {
+ 'classes': f'{IMPORT_CSV_PATH}/Entities_DataSubject.csv',
+ 'properties': f'{IMPORT_CSV_PATH}/Entities_DataSubject_properties.csv',
+ 'model': 'ontology',
+ 'topconcept': BASE['DataSubject'],
+ },
+ 'jurisdiction': {
+ 'classes': f'{IMPORT_CSV_PATH}/Jurisdiction.csv',
+ 'properties': f'{IMPORT_CSV_PATH}/Jurisdiction_properties.csv',
'model': 'ontology',
},
'legal_basis': {
'classes': f'{IMPORT_CSV_PATH}/LegalBasis.csv',
'properties': f'{IMPORT_CSV_PATH}/LegalBasis_properties.csv',
'model': 'taxonomy',
- 'topconcept': DPV.LegalBasis,
+ 'topconcept': BASE['LegalBasis'],
},
'consent': {
# 'classes': f'{IMPORT_CSV_PATH}/Consent.csv',
'properties': f'{IMPORT_CSV_PATH}/Consent_properties.csv',
- },
- }
+ },
+}
# this graph will get written to dpv.ttl
DPV_GRAPH = Graph()
@@ -567,6 +601,339 @@ def serialize_graph(graph, filepath):
# #############################################################################
+# DPV-LEGAL #
+# The structure of DPV-Legal spreadsheets is different than the rest of DPV
+# Therefore, it requires separate functions/code to handle
+
+DPV_LEGAL_CSV_FILES = {
+ f'{IMPORT_CSV_PATH}/legal_Authorities.csv',
+ f'{IMPORT_CSV_PATH}/legal_EU_Adequacy.csv',
+ f'{IMPORT_CSV_PATH}/legal_EU_EEA.csv',
+ f'{IMPORT_CSV_PATH}/legal_Laws.csv',
+ f'{IMPORT_CSV_PATH}/legal_Locations.csv',
+ f'{IMPORT_CSV_PATH}/legal_properties.csv',
+ }
+
+BASE = NAMESPACES['dpv-legal']
+DPV_LEGAL_GRAPH = Graph()
+for prefix, namespace in NAMESPACES.items():
+ DPV_LEGAL_GRAPH.namespace_manager.bind(prefix, namespace)
+graph = Graph()
+for prefix, namespace in NAMESPACES.items():
+ graph.namespace_manager.bind(prefix, namespace)
+proposed_terms = {}
+DEBUG('------')
+DEBUG(f'Processing DPV-LEGAL')
+for prefix, namespace in NAMESPACES.items():
+ DPV_LEGAL_GRAPH.namespace_manager.bind(prefix, namespace)
+
+DEBUG(f'Processing DPV-LEGAL classes and properties')
+# NOTE: There are currently no additional classes
+# >>> START
+# classes = extract_terms_from_csv(DPV_LEGAL_CSV_FILES, DPV_Class)
+# DEBUG(f'there are {len(classes)} classes in {name}')
+# returnval = add_triples_for_classes(classes, DPV_LEGAL_GRAPH)
+# if returnval:
+# proposed_terms.extend(returnval)
+# add collection representing concepts
+# DPV_LEGAL_GRAPH.add((BASE[f'LegalConcepts'], RDF.type, SKOS.Collection))
+# DPV_LEGAL_GRAPH.add((BASE[f'LegalConcepts'], DCT.title, Literal(f'Legal Concepts', datatype=XSD.string)))
+# for concept, _, _ in DPV_LEGAL_GRAPH.triples((None, RDF.type, SKOS.Concept)):
+# DPV_LEGAL_GRAPH.add((BASE[f'LegalConcepts'], SKOS.member, concept))
+properties = extract_terms_from_csv(
+ f'{IMPORT_CSV_PATH}/legal_properties.csv', DPV_Property)
+DEBUG(f'there are {len(properties)} properties in DPV-LEGAL')
+returnval = add_triples_for_properties(properties, graph)
+if returnval:
+ proposed_terms['ontology'] = returnval
+# serialize
+# DPV_LEGAL_GRAPH.load('ontology_metadata/dpv-legal.ttl', format='turtle')
+serialize_graph(graph, f'{EXPORT_DPV_LEGAL_MODULE_PATH}/ontology')
+
+DEBUG('Processing DPV-LEGAL Locations')
+graph = Graph()
+for prefix, namespace in NAMESPACES.items():
+ graph.namespace_manager.bind(prefix, namespace)
+proposed = []
+Location_schema = namedtuple('Legal_Location', (
+ 'Term', 'Label', 'ParentTerm', 'Alpha2', 'Alpha3', 'Numeric', 'M49',
+ 'broader', 'narrower', 'created', 'modified',
+ 'status', 'contributors', 'resolution'))
+concepts = extract_terms_from_csv(
+ f'{IMPORT_CSV_PATH}/legal_Locations.csv', Location_schema)
+for row in concepts:
+ if row.status not in VOCAB_TERM_ACCEPT:
+ proposed.append(row.Term)
+ continue
+ term = BASE[row.Term]
+ parent = row.ParentTerm.replace("dpv:", "")
+ graph.add((term, RDF.type, DPV[f'{parent}']))
+ graph.add((term, RDF.type, SKOS.Concept))
+ graph.add((term, DCT.title, Literal(row.Label, lang='en')))
+ graph.add((term, SKOS.prefLabel, Literal(row.Label, lang='en')))
+ if row.Alpha2:
+ graph.add((
+ term, BASE.iso_alpha2, Literal(row.Alpha2, datatype=XSD.string)))
+ graph.add((
+ term, BASE.iso_alpha3, Literal(row.Alpha3, datatype=XSD.string)))
+ graph.add((
+ term, BASE.iso_numeric, Literal(row.Numeric, datatype=XSD.string)))
+ if row.M49:
+ graph.add((
+ term, BASE.un_m49, Literal(row.M49, datatype=XSD.string)))
+ parents = [p.strip() for p in row.broader.split(',') if p]
+ for item in parents:
+ print(f'item: {item}')
+ prefix, parent = item.split(':')
+ parent = NAMESPACES[prefix][f'{parent}']
+ graph.add((term, SKOS.broaderTransitive, parent))
+ graph.add((parent, SKOS.narrowerTransitive, term))
+ # dct:created
+ graph.add((term, DCT.created, Literal(row.created, datatype=XSD.date)))
+ # dct:modified
+ if row.modified:
+ graph.add((term, DCT.modified, Literal(row.modified, datatype=XSD.date)))
+ # sw:term_status
+ graph.add((term, SW.term_status, Literal(row.status, lang='en')))
+ # dct:creator
+ if row.contributors:
+ authors = [a.strip() for a in row.contributors.split(',')]
+ for author in authors:
+ graph.add((term, DCT.creator, Literal(author, datatype=XSD.string)))
+ graph.add((BASE['LocationConcepts'], SKOS.member, term))
+graph.add((BASE['LocationConcepts'], RDF.type, SKOS.Collection))
+serialize_graph(graph, f'{EXPORT_DPV_LEGAL_MODULE_PATH}/locations')
+DPV_LEGAL_GRAPH += graph
+if proposed:
+ proposed_terms['location'] = proposed
+
+DEBUG('Processing DPV-LEGAL Laws')
+graph = Graph()
+for prefix, namespace in NAMESPACES.items():
+ graph.namespace_manager.bind(prefix, namespace)
+proposed = []
+Location_schema = namedtuple('Legal_Laws', (
+ 'term', 'label_en', 'label_de', 'time_start', 'time_end',
+ 'jurisdictions', 'webpage',
+ 'created', 'modified', 'status', 'contributors', 'resolution'))
+concepts = extract_terms_from_csv(
+ f'{IMPORT_CSV_PATH}/legal_Laws.csv', Location_schema)
+for row in concepts:
+ if row.status not in VOCAB_TERM_ACCEPT:
+ proposed.append(row.Term)
+ continue
+ term = BASE[row.term]
+ graph.add((term, RDF.type, DPV.Law))
+ graph.add((term, RDF.type, SKOS.Concept))
+ graph.add((term, DCT.title, Literal(row.label_en, lang='en')))
+ graph.add((term, SKOS.prefLabel, Literal(row.label_en, lang='en')))
+ if row.label_de:
+ graph.add((term, DCT.title, Literal(row.label_de, lang='de')))
+ graph.add((term, SKOS.prefLabel, Literal(row.label_de, lang='de')))
+ for loc in row.jurisdictions.split(','):
+ loc = loc.replace("dpv-legal:", "")
+ graph.add((term, DPV.hasJurisdiction, BASE[f'{loc}']))
+ graph.add((BASE[f'{loc}'], DPV.hasLaw, term))
+ graph.add((term, FOAF.homepage, Literal(row.webpage, datatype=XSD.anyURI)))
+ if row.time_start:
+ dct_temporal = BNode()
+ graph.add((term, DCT.temporal, dct_temporal))
+ graph.add((dct_temporal, RDF.type, TIME.ProperInterval))
+ dct_date = BNode()
+ graph.add((dct_temporal, TIME.hasBeginning, dct_date))
+ graph.add((dct_date, TIME.inXSDDate, Literal(row.time_start, datatype=XSD.date)))
+ if row.time_end:
+ dct_date = BNode()
+ graph.add((dct_temporal, TIME.hasEnd, dct_date))
+ graph.add((dct_date, TIME.inXSDDate, Literal(row.time_end, datatype=XSD.date)))
+ # dct:created
+ graph.add((term, DCT.created, Literal(row.created, datatype=XSD.date)))
+ # dct:modified
+ if row.modified:
+ graph.add((term, DCT.modified, Literal(row.modified, datatype=XSD.date)))
+ # sw:term_status
+ graph.add((term, SW.term_status, Literal(row.status, lang='en')))
+ # dct:creator
+ if row.contributors:
+ authors = [a.strip() for a in row.contributors.split(',')]
+ for author in authors:
+ graph.add((term, DCT.creator, Literal(author, datatype=XSD.string)))
+ graph.add((BASE['LawConcepts'], SKOS.member, term))
+graph.add((BASE['LawConcepts'], RDF.type, SKOS.Collection))
+serialize_graph(graph, f'{EXPORT_DPV_LEGAL_MODULE_PATH}/laws')
+DPV_LEGAL_GRAPH += graph
+if proposed:
+ proposed_terms['laws'] = proposed
+
+DEBUG('Processing DPV-LEGAL Authorities')
+graph = Graph()
+for prefix, namespace in NAMESPACES.items():
+ graph.namespace_manager.bind(prefix, namespace)
+proposed = []
+Location_schema = namedtuple('Legal_Laws', (
+ 'term', 'label_en', 'label_de', 'type', 'jurisdictions', 'laws', 'webpage',
+ 'created', 'modified', 'status', 'contributors', 'resolution'))
+concepts = extract_terms_from_csv(
+ f'{IMPORT_CSV_PATH}/legal_Authorities.csv', Location_schema)
+for row in concepts:
+ if row.status not in VOCAB_TERM_ACCEPT:
+ proposed.append(row.Term)
+ continue
+ term = BASE[row.term]
+ graph.add((term, RDF.type, DPV[f'{row.type.replace("dpv:","")}']))
+ graph.add((term, RDF.type, SKOS.Concept))
+ graph.add((term, DCT.title, Literal(row.label_en, lang='en')))
+ graph.add((term, SKOS.prefLabel, Literal(row.label_en, lang='en')))
+ if row.label_de:
+ graph.add((term, DCT.title, Literal(row.label_de, lang='de')))
+ graph.add((term, SKOS.prefLabel, Literal(row.label_de, lang='de')))
+ for loc in row.jurisdictions.split(','):
+ loc = loc.replace("dpv-legal:", "")
+ graph.add((term, DPV.hasJurisdiction, BASE[f'{loc}']))
+ graph.add((BASE[f'{loc}'], DPV.hasAuthority, term))
+ for law in row.laws.split(','):
+ law = law.replace("dpv-legal:", "")
+ graph.add((term, DPV.hasLaw, BASE[f'{law}']))
+ graph.add((BASE[f'{law}'], DPV.hasAuthority, term))
+ graph.add((term, FOAF.homepage, Literal(row.webpage, datatype=XSD.anyURI)))
+ # dct:created
+ graph.add((term, DCT.created, Literal(row.created, datatype=XSD.date)))
+ # dct:modified
+ if row.modified:
+ graph.add((term, DCT.modified, Literal(row.modified, datatype=XSD.date)))
+ # sw:term_status
+ graph.add((term, SW.term_status, Literal(row.status, lang='en')))
+ # dct:creator
+ if row.contributors:
+ authors = [a.strip() for a in row.contributors.split(',')]
+ for author in authors:
+ graph.add((term, DCT.creator, Literal(author, datatype=XSD.string)))
+ graph.add((BASE['AuthoritiesConcepts'], SKOS.member, term))
+graph.add((BASE['AuthoritiesConcepts'], RDF.type, SKOS.Collection))
+serialize_graph(graph, f'{EXPORT_DPV_LEGAL_MODULE_PATH}/authorities')
+DPV_LEGAL_GRAPH += graph
+if proposed:
+ proposed_terms['authorities'] = proposed
+
+DEBUG('Processing DPV-LEGAL EU-EEA Memberships')
+graph = Graph()
+for prefix, namespace in NAMESPACES.items():
+ graph.namespace_manager.bind(prefix, namespace)
+proposed = []
+Location_schema = namedtuple('Legal_EU_EEA', (
+ 'term', 'label', 'type', 'broader', 'time_start', 'time_end', 'members',
+ 'created', 'modified', 'status', 'contributors', 'resolution'))
+concepts = extract_terms_from_csv(
+ f'{IMPORT_CSV_PATH}/legal_EU_EEA.csv', Location_schema)
+for row in concepts:
+ if row.status not in VOCAB_TERM_ACCEPT:
+ proposed.append(row.Term)
+ continue
+ term = BASE[row.term]
+ graph.add((term, RDF.type, DPV[f'{row.type.replace("dpv:","")}']))
+ graph.add((term, RDF.type, SKOS.Concept))
+ graph.add((term, DCT.title, Literal(row.label, lang='en')))
+ if row.broader:
+ graph.add((term, SKOS.broaderTransitive, BASE[f'{row.broader.replace("dpv-legal:","")}']))
+ graph.add((BASE[f'{row.broader.replace("dpv-legal:","")}'], SKOS.narrowerTransitive, term))
+ for loc in row.members.split(','):
+ loc = loc.replace("dpv-legal:", "")
+ graph.add((term, DPV.hasCountry, BASE[f'{loc}']))
+ graph.add((term, SKOS.narrowerTransitive, BASE[f'{loc}']))
+ graph.add((BASE[f'{loc}'], SKOS.broaderTransitive, term))
+ if row.time_start:
+ dct_temporal = BNode()
+ graph.add((term, DCT.temporal, dct_temporal))
+ graph.add((dct_temporal, RDF.type, TIME.ProperInterval))
+ dct_date = BNode()
+ graph.add((dct_temporal, TIME.hasBeginning, dct_date))
+ graph.add((dct_date, TIME.inXSDDate, Literal(row.time_start, datatype=XSD.date)))
+ if row.time_end:
+ dct_date = BNode()
+ graph.add((dct_temporal, TIME.hasEnd, dct_date))
+ graph.add((dct_date, TIME.inXSDDate, Literal(row.time_end, datatype=XSD.date)))
+ # dct:created
+ graph.add((term, DCT.created, Literal(row.created, datatype=XSD.date)))
+ # dct:modified
+ if row.modified:
+ graph.add((term, DCT.modified, Literal(row.modified, datatype=XSD.date)))
+ # sw:term_status
+ graph.add((term, SW.term_status, Literal(row.status, lang='en')))
+ # dct:creator
+ if row.contributors:
+ authors = [a.strip() for a in row.contributors.split(',')]
+ for author in authors:
+ graph.add((term, DCT.creator, Literal(author, datatype=XSD.string)))
+ graph.add((BASE['EUEEAConcepts'], SKOS.member, term))
+graph.add((BASE['EUEEAConcepts'], RDF.type, SKOS.Collection))
+serialize_graph(graph, f'{EXPORT_DPV_LEGAL_MODULE_PATH}/eu_eea')
+DPV_LEGAL_GRAPH += graph
+if proposed:
+ proposed_terms['EU_EEA'] = proposed
+
+DEBUG('Processing DPV-LEGAL EU Adequacy Decisions')
+graph = Graph()
+for prefix, namespace in NAMESPACES.items():
+ graph.namespace_manager.bind(prefix, namespace)
+proposed = []
+Location_schema = namedtuple('Legal_EU_Adequacy', (
+ 'term', 'label', 'webpage', 'countryA', 'countryB',
+ 'time_start', 'time_end',
+ 'created', 'modified', 'status', 'contributors', 'resolution'))
+concepts = extract_terms_from_csv(
+ f'{IMPORT_CSV_PATH}/legal_EU_Adequacy.csv', Location_schema)
+for row in concepts:
+ if row.status not in VOCAB_TERM_ACCEPT:
+ proposed.append(row.Term)
+ continue
+ term = BASE[row.term]
+ graph.add((term, RDF.type, DPV.Law))
+ graph.add((term, RDF.type, DPV_GDPR['A45-3']))
+ graph.add((term, RDF.type, SKOS.Concept))
+ graph.add((term, DCT.title, Literal(row.label, lang='en')))
+ graph.add((term, FOAF.homepage, Literal(row.webpage, datatype=XSD.anyURI)))
+ graph.add((term, DPV.hasJurisdiction, BASE[f'{row.countryA.replace("dpv-legal:","")}']))
+ graph.add((term, DPV.hasJurisdiction, BASE[f'{row.countryB.replace("dpv-legal:","")}']))
+ if row.time_start:
+ dct_temporal = BNode()
+ graph.add((term, DCT.temporal, dct_temporal))
+ graph.add((dct_temporal, RDF.type, TIME.ProperInterval))
+ dct_date = BNode()
+ graph.add((dct_temporal, TIME.hasBeginning, dct_date))
+ graph.add((dct_date, TIME.inXSDDate, Literal(row.time_start, datatype=XSD.date)))
+ if row.time_end:
+ dct_date = BNode()
+ graph.add((dct_temporal, TIME.hasEnd, dct_date))
+ graph.add((dct_date, TIME.inXSDDate, Literal(row.time_end, datatype=XSD.date)))
+ # dct:created
+ graph.add((term, DCT.created, Literal(row.created, datatype=XSD.date)))
+ # dct:modified
+ if row.modified:
+ graph.add((term, DCT.modified, Literal(row.modified, datatype=XSD.date)))
+ # sw:term_status
+ graph.add((term, SW.term_status, Literal(row.status, lang='en')))
+ # dct:creator
+ if row.contributors:
+ authors = [a.strip() for a in row.contributors.split(',')]
+ for author in authors:
+ graph.add((term, DCT.creator, Literal(author, datatype=XSD.string)))
+ graph.add((BASE['AdequacyConcepts'], SKOS.member, term))
+graph.add((BASE['AdequacyConcepts'], RDF.type, SKOS.Collection))
+serialize_graph(graph, f'{EXPORT_DPV_LEGAL_MODULE_PATH}/eu_adequacy')
+DPV_LEGAL_GRAPH += graph
+if proposed:
+ proposed_terms['EU_Adequacy'] = proposed
+
+serialize_graph(DPV_LEGAL_GRAPH, f'{EXPORT_DPV_LEGAL_PATH}/dpv-legal')
+if proposed_terms:
+ with open(f'{EXPORT_DPV_LEGAL_PATH}/proposed.json', 'w') as fd:
+ json.dump(proposed_terms, fd)
+ DEBUG(f'exported proposed terms to {EXPORT_DPV_LEGAL_PATH}/proposed.json')
+else:
+ DEBUG('no proposed terms in DPV-LEGAL')
+
+# #############################################################################
+
# Save collected links as resource for generating HTML A HREF in JINJA2 templates
# file is in jinja2_resources/links_labels.json
diff --git a/documentation-generator/002_parse_csv_to_rdf_owl.py b/documentation-generator/002_parse_csv_to_rdf_owl.py
index 7e2e8e45e..8383c652a 100755
--- a/documentation-generator/002_parse_csv_to_rdf_owl.py
+++ b/documentation-generator/002_parse_csv_to_rdf_owl.py
@@ -392,25 +392,30 @@ def serialize_graph(graph, filepath):
'classes': f'{IMPORT_CSV_PATH}/PersonalData.csv',
'properties': f'{IMPORT_CSV_PATH}/PersonalData_properties.csv',
'model': 'ontology',
- 'topconcept': DPV.PersonalData,
+ 'topconcept': BASE['PersonalData'],
},
'purposes': {
'classes': f'{IMPORT_CSV_PATH}/Purpose.csv',
'properties': f'{IMPORT_CSV_PATH}/Purpose_properties.csv',
'model': 'taxonomy',
- 'topconcept': DPV.Purpose,
+ 'topconcept': BASE['Purpose'],
},
'context': {
'classes': f'{IMPORT_CSV_PATH}/Context.csv',
'properties': f'{IMPORT_CSV_PATH}/Context_properties.csv',
'model': 'taxonomy',
- 'topconcept': DPV.Context,
+ 'topconcept': BASE['Context'],
+ },
+ 'risk': {
+ 'classes': f'{IMPORT_CSV_PATH}/Risk.csv',
+ 'properties': f'{IMPORT_CSV_PATH}/Risk_properties.csv',
+ 'model': 'ontology',
},
'processing': {
'classes': f'{IMPORT_CSV_PATH}/Processing.csv',
'properties': f'{IMPORT_CSV_PATH}/Processing_properties.csv',
'model': 'taxonomy',
- 'topconcept': DPV.Processing,
+ 'topconcept': BASE['Processing'],
},
'processing_context': {
'classes': f'{IMPORT_CSV_PATH}/ProcessingContext.csv',
@@ -421,30 +426,52 @@ def serialize_graph(graph, filepath):
'classes': f'{IMPORT_CSV_PATH}/TechnicalOrganisationalMeasure.csv',
'properties': f'{IMPORT_CSV_PATH}/TechnicalOrganisationalMeasure_properties.csv',
'model': 'taxonomy',
- 'topconcept': DPV.TechnicalOrganisationalMeasure,
+ 'topconcept': BASE['TechnicalOrganisationalMeasure'],
},
'entities': {
'classes': f'{IMPORT_CSV_PATH}/Entities.csv',
'properties': f'{IMPORT_CSV_PATH}/Entities_properties.csv',
'model': 'ontology',
- 'topconcept': DPV.Entity,
+ 'topconcept': BASE['Entity'],
+ },
+ 'entities_authority': {
+ 'classes': f'{IMPORT_CSV_PATH}/Entities_Authority.csv',
+ 'properties': f'{IMPORT_CSV_PATH}/Entities_Authority_properties.csv',
+ 'model': 'ontology',
+ 'topconcept': BASE['Authority'],
+ },
+ 'entities_legalrole': {
+ 'classes': f'{IMPORT_CSV_PATH}/Entities_LegalRole.csv',
+ 'properties': f'{IMPORT_CSV_PATH}/Entities_LegalRole_properties.csv',
+ 'model': 'ontology',
+ },
+ 'entities_organisation': {
+ 'classes': f'{IMPORT_CSV_PATH}/Entities_Organisation.csv',
+ 'model': 'ontology',
+ 'topconcept': BASE['Organisation'],
+ },
+ 'entities_datasubject': {
+ 'classes': f'{IMPORT_CSV_PATH}/Entities_DataSubject.csv',
+ 'properties': f'{IMPORT_CSV_PATH}/Entities_DataSubject_properties.csv',
+ 'model': 'ontology',
+ 'topconcept': BASE['DataSubject'],
},
- 'jurisdictions': {
- 'classes': f'{IMPORT_CSV_PATH}/Jurisdictions.csv',
- 'properties': f'{IMPORT_CSV_PATH}/Jurisdictions_properties.csv',
+ 'jurisdiction': {
+ 'classes': f'{IMPORT_CSV_PATH}/Jurisdiction.csv',
+ 'properties': f'{IMPORT_CSV_PATH}/Jurisdiction_properties.csv',
'model': 'ontology',
},
'legal_basis': {
'classes': f'{IMPORT_CSV_PATH}/LegalBasis.csv',
'properties': f'{IMPORT_CSV_PATH}/LegalBasis_properties.csv',
'model': 'taxonomy',
- 'topconcept': DPV.LegalBasis,
+ 'topconcept': BASE['LegalBasis'],
},
'consent': {
# 'classes': f'{IMPORT_CSV_PATH}/Consent.csv',
'properties': f'{IMPORT_CSV_PATH}/Consent_properties.csv',
- },
- }
+ },
+}
# this graph will get written to dpv.ttl
DPV_GRAPH = Graph()
diff --git a/documentation-generator/002_parse_csv_to_rdf_skos.py b/documentation-generator/002_parse_csv_to_rdf_skos.py
index c9c61fa9e..cfe45b981 100755
--- a/documentation-generator/002_parse_csv_to_rdf_skos.py
+++ b/documentation-generator/002_parse_csv_to_rdf_skos.py
@@ -265,15 +265,17 @@ def add_triples_for_classes(classes, graph, model, topconcept):
if parent == "dpv:Concept":
continue
# assuming something like rdfs:Resource
- prefix, term = parent.split(':')
- prefix = prefix.replace("sc__", "")
+ prefix_str, term = parent.split(':')
+ prefix = prefix_str.replace("sc__", "")
# gets the namespace from registered ones and create URI
# will throw an error if namespace is not registered
# dpv internal terms are expected to have the prefix i.e. dpv:term
parent = NAMESPACES_DPV_SKOS[prefix][f'{term}']
graph.add((BASE[f'{cls.term}'], SKOS.broaderTransitive, parent))
- if model == 'ontology':
- graph.add((BASE[f'{cls.term}'], RDFS.subClassOf, parent))
+ if model == 'vocabulary' and "sc__" in prefix_str:
+ graph.add((BASE[f'{cls.term}'], RDFS.subClassOf, parent))
+ elif model == 'ontology':
+ graph.add((BASE[f'{cls.term}'], RDFS.subClassOf, parent))
else:
graph.add((BASE[f'{cls.term}'], SKOS.broaderTransitive, Literal(parent, datatype=XSD.string)))
@@ -370,13 +372,11 @@ def serialize_graph(graph, filepath):
# #############################################################################
# DPV #
-
DPV_CSV_FILES = {
'base': {
'classes': f'{IMPORT_CSV_PATH}/BaseOntology.csv',
'properties': f'{IMPORT_CSV_PATH}/BaseOntology_properties.csv',
'model': 'vocabulary',
- 'topconcept': '',
},
'personal_data': {
'classes': f'{IMPORT_CSV_PATH}/PersonalData.csv',
@@ -396,6 +396,11 @@ def serialize_graph(graph, filepath):
'model': 'taxonomy',
'topconcept': BASE['Context'],
},
+ 'risk': {
+ 'classes': f'{IMPORT_CSV_PATH}/Risk.csv',
+ 'properties': f'{IMPORT_CSV_PATH}/Risk_properties.csv',
+ 'model': 'ontology',
+ },
'processing': {
'classes': f'{IMPORT_CSV_PATH}/Processing.csv',
'properties': f'{IMPORT_CSV_PATH}/Processing_properties.csv',
@@ -405,8 +410,8 @@ def serialize_graph(graph, filepath):
'processing_context': {
'classes': f'{IMPORT_CSV_PATH}/ProcessingContext.csv',
'properties': f'{IMPORT_CSV_PATH}/ProcessingContext_properties.csv',
- 'model': 'taxonomy',
- 'topconcept': BASE['Context'],
+ 'model': 'vocabulary',
+ 'topconcept': BASE['ProcessingContext'],
},
'technical_organisational_measures': {
'classes': f'{IMPORT_CSV_PATH}/TechnicalOrganisationalMeasure.csv',
@@ -420,11 +425,32 @@ def serialize_graph(graph, filepath):
'model': 'ontology',
'topconcept': BASE['Entity'],
},
- 'jurisdictions': {
- 'classes': f'{IMPORT_CSV_PATH}/Jurisdictions.csv',
- 'properties': f'{IMPORT_CSV_PATH}/Jurisdictions_properties.csv',
+ 'entities_authority': {
+ 'classes': f'{IMPORT_CSV_PATH}/Entities_Authority.csv',
+ 'properties': f'{IMPORT_CSV_PATH}/Entities_Authority_properties.csv',
+ 'model': 'ontology',
+ 'topconcept': BASE['Authority'],
+ },
+ 'entities_legalrole': {
+ 'classes': f'{IMPORT_CSV_PATH}/Entities_LegalRole.csv',
+ 'properties': f'{IMPORT_CSV_PATH}/Entities_LegalRole_properties.csv',
+ 'model': 'ontology',
+ },
+ 'entities_organisation': {
+ 'classes': f'{IMPORT_CSV_PATH}/Entities_Organisation.csv',
+ 'model': 'ontology',
+ 'topconcept': BASE['Organisation'],
+ },
+ 'entities_datasubject': {
+ 'classes': f'{IMPORT_CSV_PATH}/Entities_DataSubject.csv',
+ 'properties': f'{IMPORT_CSV_PATH}/Entities_DataSubject_properties.csv',
+ 'model': 'ontology',
+ 'topconcept': BASE['DataSubject'],
+ },
+ 'jurisdiction': {
+ 'classes': f'{IMPORT_CSV_PATH}/Jurisdiction.csv',
+ 'properties': f'{IMPORT_CSV_PATH}/Jurisdiction_properties.csv',
'model': 'ontology',
- 'topconcept': '',
},
'legal_basis': {
'classes': f'{IMPORT_CSV_PATH}/LegalBasis.csv',
@@ -436,9 +462,8 @@ def serialize_graph(graph, filepath):
# 'classes': f'{IMPORT_CSV_PATH}/Consent.csv',
'properties': f'{IMPORT_CSV_PATH}/Consent_properties.csv',
'model': 'vocabulary',
- 'topconcept': '',
- },
- }
+ },
+}
# this graph will get written to dpv.ttl
DPV_GRAPH = Graph()
@@ -449,7 +474,7 @@ def serialize_graph(graph, filepath):
proposed = []
DEBUG('------')
model = module['model']
- topconcept = module['topconcept']
+ topconcept = module.get('topconcept')
DEBUG(f'Processing {name} {model}')
for prefix, namespace in NAMESPACES.items():
graph.namespace_manager.bind(prefix, namespace)
diff --git a/documentation-generator/003_generate_respec_html.py b/documentation-generator/003_generate_respec_html.py
index e877c5f9f..ca7f11311 100755
--- a/documentation-generator/003_generate_respec_html.py
+++ b/documentation-generator/003_generate_respec_html.py
@@ -13,6 +13,9 @@
EXPORT_DPV_GDPR_HTML_PATH = '../dpv-gdpr'
IMPORT_DPV_PD_PATH = '../dpv-pd/dpv-pd.ttl'
EXPORT_DPV_PD_HTML_PATH = '../dpv-pd'
+IMPORT_DPV_LEGAL_PATH = '../dpv-legal/dpv-legal.ttl'
+IMPORT_DPV_LEGAL_MODULES_PATH = '../dpv-legal/modules'
+EXPORT_DPV_LEGAL_HTML_PATH = '../dpv-legal'
import json
from rdflib import Graph, Namespace
@@ -106,9 +109,14 @@ def saved_label(item):
load_data('processing', f'{IMPORT_DPV_MODULES_PATH}/processing.ttl')
load_data('technical_organisational_measures', f'{IMPORT_DPV_MODULES_PATH}/technical_organisational_measures.ttl')
load_data('entities', f'{IMPORT_DPV_MODULES_PATH}/entities.ttl')
+load_data('entities_authority', f'{IMPORT_DPV_MODULES_PATH}/entities_authority.ttl')
+load_data('entities_legalrole', f'{IMPORT_DPV_MODULES_PATH}/entities_legalrole.ttl')
+load_data('entities_organisation', f'{IMPORT_DPV_MODULES_PATH}/entities_organisation.ttl')
+load_data('entities_datasubject', f'{IMPORT_DPV_MODULES_PATH}/entities_datasubject.ttl')
load_data('context', f'{IMPORT_DPV_MODULES_PATH}/context.ttl')
+load_data('risk', f'{IMPORT_DPV_MODULES_PATH}/risk.ttl')
load_data('processing_context', f'{IMPORT_DPV_MODULES_PATH}/processing_context.ttl')
-load_data('jurisdictions', f'{IMPORT_DPV_MODULES_PATH}/jurisdictions.ttl')
+load_data('jurisdiction', f'{IMPORT_DPV_MODULES_PATH}/jurisdiction.ttl')
load_data('legal_basis', f'{IMPORT_DPV_MODULES_PATH}/legal_basis.ttl')
load_data('consent', f'{IMPORT_DPV_MODULES_PATH}/consent.ttl')
g = Graph()
@@ -164,5 +172,38 @@ def saved_label(item):
fd.write(template.render(**TEMPLATE_DATA))
DEBUG(f'wrote DPV-PD spec at f{EXPORT_DPV_PD_HTML_PATH}/dpv-pd.html')
+# DPV-LEGAL: generate HTML
+
+with open(f'{EXPORT_DPV_LEGAL_HTML_PATH}/proposed.json') as fd:
+ TEMPLATE_DATA['proposed'] = json.load(fd)
+
+def load_legal_data(label, filepath):
+ DEBUG(f'loading data for {label}')
+ g = Graph()
+ g.load(filepath, format='turtle')
+ G = DataGraph()
+ G.load(g)
+ G.graph.ns = { k:v for k,v in G.graph.namespaces() }
+ # TODO: Take the instance variable so that template has contextual info
+ # e.g. Law can specify jurisdiction label, authorities, etc.,
+ TEMPLATE_DATA[f'{label}_terms'] = G.get_instances_of('skos_Concept')
+
+load_legal_data('ontology', f'{IMPORT_DPV_LEGAL_MODULES_PATH}/ontology.ttl')
+load_legal_data('locations', f'{IMPORT_DPV_LEGAL_MODULES_PATH}/locations.ttl')
+load_legal_data('laws', f'{IMPORT_DPV_LEGAL_MODULES_PATH}/laws.ttl')
+load_legal_data('authorities', f'{IMPORT_DPV_LEGAL_MODULES_PATH}/authorities.ttl')
+load_legal_data('EU_EEA', f'{IMPORT_DPV_LEGAL_MODULES_PATH}/eu_eea.ttl')
+load_legal_data('EU_Adequacy', f'{IMPORT_DPV_LEGAL_MODULES_PATH}/eu_adequacy.ttl')
+g = Graph()
+g.load(f'{IMPORT_DPV_LEGAL_PATH}', format='turtle')
+G.load(g)
+
+template = template_env.get_template('template_dpv_legal.jinja2')
+with open(f'{EXPORT_DPV_LEGAL_HTML_PATH}/index.html', 'w+') as fd:
+ fd.write(template.render(**TEMPLATE_DATA))
+DEBUG(f'wrote DPV-LEGAL spec at f{EXPORT_DPV_LEGAL_HTML_PATH}/index.html')
+with open(f'{EXPORT_DPV_LEGAL_HTML_PATH}/dpv-legal.html', 'w+') as fd:
+ fd.write(template.render(**TEMPLATE_DATA))
+DEBUG(f'wrote DPV-LEGAL spec at f{EXPORT_DPV_LEGAL_HTML_PATH}/dpv-legal.html')
DEBUG('--- END ---')
\ No newline at end of file
diff --git a/documentation-generator/003_generate_respec_html_owl.py b/documentation-generator/003_generate_respec_html_owl.py
index 3d62c7cac..70f89e722 100755
--- a/documentation-generator/003_generate_respec_html_owl.py
+++ b/documentation-generator/003_generate_respec_html_owl.py
@@ -105,7 +105,12 @@ def saved_label(item):
load_data('processing', f'{IMPORT_DPV_MODULES_PATH}/processing.ttl')
load_data('technical_organisational_measures', f'{IMPORT_DPV_MODULES_PATH}/technical_organisational_measures.ttl')
load_data('entities', f'{IMPORT_DPV_MODULES_PATH}/entities.ttl')
+load_data('entities_authority', f'{IMPORT_DPV_MODULES_PATH}/entities_authority.ttl')
+load_data('entities_legalrole', f'{IMPORT_DPV_MODULES_PATH}/entities_legalrole.ttl')
+load_data('entities_organisation', f'{IMPORT_DPV_MODULES_PATH}/entities_organisation.ttl')
+load_data('entities_datasubject', f'{IMPORT_DPV_MODULES_PATH}/entities_datasubject.ttl')
load_data('context', f'{IMPORT_DPV_MODULES_PATH}/context.ttl')
+load_data('risk', f'{IMPORT_DPV_MODULES_PATH}/risk.ttl')
load_data('processing_context', f'{IMPORT_DPV_MODULES_PATH}/processing_context.ttl')
load_data('jurisdictions', f'{IMPORT_DPV_MODULES_PATH}/jurisdictions.ttl')
load_data('legal_basis', f'{IMPORT_DPV_MODULES_PATH}/legal_basis.ttl')
diff --git a/documentation-generator/003_generate_respec_html_skos.py b/documentation-generator/003_generate_respec_html_skos.py
index f405d966d..b84174da1 100755
--- a/documentation-generator/003_generate_respec_html_skos.py
+++ b/documentation-generator/003_generate_respec_html_skos.py
@@ -105,9 +105,14 @@ def saved_label(item):
load_data('processing', f'{IMPORT_DPV_MODULES_PATH}/processing.ttl')
load_data('technical_organisational_measures', f'{IMPORT_DPV_MODULES_PATH}/technical_organisational_measures.ttl')
load_data('entities', f'{IMPORT_DPV_MODULES_PATH}/entities.ttl')
+load_data('entities_authority', f'{IMPORT_DPV_MODULES_PATH}/entities_authority.ttl')
+load_data('entities_legalrole', f'{IMPORT_DPV_MODULES_PATH}/entities_legalrole.ttl')
+load_data('entities_organisation', f'{IMPORT_DPV_MODULES_PATH}/entities_organisation.ttl')
+load_data('entities_datasubject', f'{IMPORT_DPV_MODULES_PATH}/entities_datasubject.ttl')
load_data('context', f'{IMPORT_DPV_MODULES_PATH}/context.ttl')
+load_data('risk', f'{IMPORT_DPV_MODULES_PATH}/risk.ttl')
load_data('processing_context', f'{IMPORT_DPV_MODULES_PATH}/processing_context.ttl')
-load_data('jurisdictions', f'{IMPORT_DPV_MODULES_PATH}/jurisdictions.ttl')
+load_data('jurisdiction', f'{IMPORT_DPV_MODULES_PATH}/jurisdiction.ttl')
load_data('legal_basis', f'{IMPORT_DPV_MODULES_PATH}/legal_basis.ttl')
load_data('consent', f'{IMPORT_DPV_MODULES_PATH}/consent.ttl')
g = Graph()
diff --git a/documentation-generator/004_generate_guides.py b/documentation-generator/004_generate_guides.py
new file mode 100755
index 000000000..d9d1f4562
--- /dev/null
+++ b/documentation-generator/004_generate_guides.py
@@ -0,0 +1,40 @@
+#!/usr/bin/env python3
+#author: Harshvardhan J. Pandit
+
+'''Generates ReSpec documentation for DPV using RDF and SPARQL'''
+
+# The vocabularies are modular
+
+EXPORT_HTML_PATH = '../guides'
+
+import json
+import logging
+# logging configuration for debugging to console
+logging.basicConfig(
+ level=logging.DEBUG, format='%(levelname)s - %(funcName)s :: %(lineno)d - %(message)s')
+DEBUG = logging.debug
+
+# JINJA2 for templating and generating HTML
+from jinja2 import FileSystemLoader, Environment
+JINJA2_FILTERS = {}
+TEMPLATE_DATA = {}
+
+template_loader = FileSystemLoader(searchpath='./jinja2_resources')
+template_env = Environment(
+ loader=template_loader,
+ autoescape=True, trim_blocks=True, lstrip_blocks=True)
+template_env.filters.update(JINJA2_FILTERS)
+
+# Generate HTML
+
+template = template_env.get_template('template_guides_index.jinja2')
+with open(f'{EXPORT_HTML_PATH}/index.html', 'w+') as fd:
+ fd.write(template.render(**TEMPLATE_DATA))
+DEBUG(f'wrote Guides index at f{EXPORT_HTML_PATH}/index.html')
+
+template = template_env.get_template('template_guides_owl2.jinja2')
+with open(f'{EXPORT_HTML_PATH}/dpv-owl.html', 'w+') as fd:
+ fd.write(template.render(**TEMPLATE_DATA))
+DEBUG(f'wrote Guide for DPV-OWL at f{EXPORT_HTML_PATH}/dpv-owl.html')
+
+DEBUG('--- END ---')
\ No newline at end of file
diff --git a/documentation-generator/005_generate_examples.py b/documentation-generator/005_generate_examples.py
new file mode 100755
index 000000000..e69de29bb
diff --git a/documentation-generator/006_generate_primer.py b/documentation-generator/006_generate_primer.py
new file mode 100755
index 000000000..d963af4ae
--- /dev/null
+++ b/documentation-generator/006_generate_primer.py
@@ -0,0 +1,35 @@
+#!/usr/bin/env python3
+#author: Harshvardhan J. Pandit
+
+'''Generates ReSpec documentation for DPV using RDF and SPARQL'''
+
+# The vocabularies are modular
+
+EXPORT_HTML_PATH = '../primer'
+
+import json
+import logging
+# logging configuration for debugging to console
+logging.basicConfig(
+ level=logging.DEBUG, format='%(levelname)s - %(funcName)s :: %(lineno)d - %(message)s')
+DEBUG = logging.debug
+
+# JINJA2 for templating and generating HTML
+from jinja2 import FileSystemLoader, Environment
+JINJA2_FILTERS = {}
+TEMPLATE_DATA = {}
+
+template_loader = FileSystemLoader(searchpath='./jinja2_resources')
+template_env = Environment(
+ loader=template_loader,
+ autoescape=True, trim_blocks=True, lstrip_blocks=True)
+template_env.filters.update(JINJA2_FILTERS)
+
+# Generate HTML
+
+template = template_env.get_template('template_primer.jinja2')
+with open(f'{EXPORT_HTML_PATH}/index.html', 'w+') as fd:
+ fd.write(template.render(**TEMPLATE_DATA))
+DEBUG(f'wrote Primer at {EXPORT_HTML_PATH}/index.html')
+
+DEBUG('--- END ---')
\ No newline at end of file
diff --git a/documentation-generator/README.md b/documentation-generator/README.md
index 99a872937..9dd017e06 100644
--- a/documentation-generator/README.md
+++ b/documentation-generator/README.md
@@ -18,7 +18,7 @@ In between steps 2 and 3, there can be a series of tests done to ensure the RDF
`./003_generate_respec_html.py` will generate HTML documentation for DPV and DPV-GDPR from RDF.
-
+The `9**` series offers convenience in running the other scripts in some combination. `902` executes all the RDF generation scripts, `903` executes all the HTML generation scripts, and `999` executes all scripts.
## How everything works
### Downloading CSV data
diff --git a/documentation-generator/changelog.py b/documentation-generator/changelog.py
index 2c5aca198..536389ae4 100755
--- a/documentation-generator/changelog.py
+++ b/documentation-generator/changelog.py
@@ -16,16 +16,21 @@
DPV_MODULES = (
'base',
- 'personal_data',
- 'purposes',
- 'processing',
- 'technical_organisational_measures',
+ 'consent',
'context',
- 'processing_context',
+ 'entities_authority',
+ 'entities_datasubject',
+ 'entities_legalrole',
+ 'entities_organisation',
'entities',
- 'jurisdictions',
+ 'jurisdiction',
'legal_basis',
- 'consent',
+ 'personal_data',
+ 'processing_context',
+ 'processing',
+ 'purposes',
+ 'risk',
+ 'technical_organisational_measures',
)
DPV_GDPR_MODULES = (
'legal_basis',
diff --git a/documentation-generator/devnotes.txt b/documentation-generator/devnotes.txt
new file mode 100644
index 000000000..aa6e8871b
--- /dev/null
+++ b/documentation-generator/devnotes.txt
@@ -0,0 +1,4 @@
+- 001_download script now uses a try...catch block to continue downloading other sheets if one causes an error
+- Scripts and templates updated to incorporate Entities and Risk modules
+- Updated DPV to v5
+- DPV-SKOS: fixed issue where certain concepts were not expressed as subclasses (e.g. ProcessingContext -> Context) using the "sc__" notation in spreadsheet. Updated HTML documentation which shows subclass if applies, else narrower than if applies - in that order.
\ No newline at end of file
diff --git a/documentation-generator/jinja2_resources/links_label.json b/documentation-generator/jinja2_resources/links_label.json
index 00d8a611b..c5678dde8 100644
--- a/documentation-generator/jinja2_resources/links_label.json
+++ b/documentation-generator/jinja2_resources/links_label.json
@@ -1 +1 @@
-{"https://eur-lex.europa.eu/eli/reg/2016/679/art_4/par_1/oj": "GDPR Art.4-1g", "https://www.specialprivacy.eu/": "SPECIAL Project", "https://eur-lex.europa.eu/eli/reg/2016/679/art_4/par_9/oj": "GDPR Art.4-9g", "https://eur-lex.europa.eu/eli/reg/2016/679/art_4/par_7/oj": "GDPR Art.4-7g", "https://www.w3.org/community/dpvcg/": "DPVCG", "https://eur-lex.europa.eu/eli/reg/2016/679/art_9/par_1/oj": "GDPR Art.9-1", "https://www.privacycommission.be/nl/model-voor-een-register-van-de-verwerkingsactiviteiten": "Belgian DPA ROPA Template", "https://eur-lex.europa.eu/eli/reg/2016/679/art_4/par_2/oj": "GDPR Art.4-2", "https://www.specialprivacy.eu/vocabs/processing": "SPECIAL Project", "https://eur-lex.europa.eu/eli/reg/2016/679/art_4/par_5/oj": "GDPR Art.4-5", "https://www.iso.org/iso-31000-risk-management.html": "ISO 31000", "https://eur-lex.europa.eu/eli/reg/2016/679/art_4/par_8/oj": "GDPR Art.4-8", "https://eur-lex.europa.eu/eli/reg/2016/679/art_4/par_10/oj": "GDPR Art.4-10", "https://eur-lex.europa.eu/eli/reg/2016/679/art_37/oj": "GDPR Art.37", "https://edpb.europa.eu/our-work-tools/our-documents/recommendations/recommendations-012020-measures-supplement-transfer_en": "EDPB Recommendations 01/2020 on Data Transfers", "https://eur-lex.europa.eu/eli/reg/2016/679/art_27/oj": "GDPR Art.27", "http://purl.org/adms": "ADMS controlled vocabulary", "https://eur-lex.europa.eu/eli/reg/2016/679/art_6/par_1/pnt_a/oj": "GDPR Art.6-1a", "https://eur-lex.europa.eu/eli/reg/2016/679/art_6/par_1/pnt_b/oj": "GDPR Art.6-1b", "https://eur-lex.europa.eu/eli/reg/2016/679/art_6/par_1/pnt_c/oj": "GDPR Art.6-1c", "https://eur-lex.europa.eu/eli/reg/2016/679/art_6/par_1/pnt_d/oj": "GDPR Art.6-1d", "https://eur-lex.europa.eu/eli/reg/2016/679/art_6/par_1/pnt_e/oj": "GDPR Art.6-1e", "https://eur-lex.europa.eu/eli/reg/2016/679/art_6/par_1/pnt_f/oj": "GDPR Art.6-1f", "https://eur-lex.europa.eu/eli/reg/2016/679/art_9/par_2/pnt_a/oj": "GDPR Art.9-2a", "https://eur-lex.europa.eu/eli/reg/2016/679/art_9/par_2/pnt_b/oj": "GDPR Art.9-2b", "https://eur-lex.europa.eu/eli/reg/2016/679/art_9/par_2/pnt_c/oj": "GDPR Art.9-2c", "https://eur-lex.europa.eu/eli/reg/2016/679/art_9/par_2/pnt_d/oj": "GDPR Art.9-2d", "https://eur-lex.europa.eu/eli/reg/2016/679/art_9/par_2/pnt_e/oj": "GDPR Art.9-2e", "https://eur-lex.europa.eu/eli/reg/2016/679/art_9/par_2/pnt_f/oj": "GDPR Art.9-2f", "https://eur-lex.europa.eu/eli/reg/2016/679/art_9/par_2/pnt_g/oj": "GDPR Art.9-2g", "https://eur-lex.europa.eu/eli/reg/2016/679/art_9/par_2/pnt_h/oj": "GDPR Art.9-2h", "https://eur-lex.europa.eu/eli/reg/2016/679/art_9/par_2/pnt_i/oj": "GDPR Art.9-2i", "https://eur-lex.europa.eu/eli/reg/2016/679/art_9/par_2/pnt_j/oj": "GDPR Art.9-2j", "https://eur-lex.europa.eu/eli/reg/2016/679/art_45/par_3/oj": "GDPR Art.45-3", "https://eur-lex.europa.eu/eli/reg/2016/679/art_46/par_2/pnt_a/oj": "GDPR Art.46-2a", "https://eur-lex.europa.eu/eli/reg/2016/679/art_46/par_2/pnt_b/oj": "GDPR Art.46-2b", "https://eur-lex.europa.eu/eli/reg/2016/679/art_46/par_2/pnt_c/oj": "GDPR Art.46-2c", "https://eur-lex.europa.eu/eli/reg/2016/679/art_46/par_2/pnt_d/oj": "GDPR Art.46-2d", "https://eur-lex.europa.eu/eli/reg/2016/679/art_46/par_2/pnt_e/oj": "GDPR Art.46-2e", "https://eur-lex.europa.eu/eli/reg/2016/679/art_46/par_2/pnt_f/oj": "GDPR Art.46-2f", "https://eur-lex.europa.eu/eli/reg/2016/679/art_46/par_3/pnt_a/oj": "GDPR Art.46-3a", "https://eur-lex.europa.eu/eli/reg/2016/679/art_46/par_3/pnt_b/oj": "GDPR Art.46-3b", "https://eur-lex.europa.eu/eli/reg/2016/679/art_49/par_1/pnt_a/oj": "GDPR Art.49-1a", "https://eur-lex.europa.eu/eli/reg/2016/679/art_49/par_1/pnt_b/oj": "GDPR Art.49-1b", "https://eur-lex.europa.eu/eli/reg/2016/679/art_49/par_1/pnt_c/oj": "GDPR Art.49-1c", "https://eur-lex.europa.eu/eli/reg/2016/679/art_49/par_1/pnt_d/oj": "GDPR Art.49-1d", "https://eur-lex.europa.eu/eli/reg/2016/679/art_49/par_1/pnt_e/oj": "GDPR Art.49-1e", "https://eur-lex.europa.eu/eli/reg/2016/679/art_49/par_1/pnt_f/oj": "GDPR Art.49-1f", "https://eur-lex.europa.eu/eli/reg/2016/679/art_49/par_1/pnt_g/oj": "GDPR Art.49-1g", "https://eur-lex.europa.eu/eli/reg/2016/679/art_49/par_2/oj": "GDPR Art.49-2", "https://eur-lex.europa.eu/eli/reg/2016/679/art_13/oj": "GDPR Art.13", "https://eur-lex.europa.eu/eli/reg/2016/679/art_14/oj": "GDPR Art.14", "https://eur-lex.europa.eu/eli/reg/2016/679/art_15/oj": "GDPR Art.15", "https://eur-lex.europa.eu/eli/reg/2016/679/art_16/oj": "GDPR Art.16", "https://eur-lex.europa.eu/eli/reg/2016/679/art_17/oj": "GDPR Art.17", "https://eur-lex.europa.eu/eli/reg/2016/679/art_18/oj": "GDPR Art.18", "https://eur-lex.europa.eu/eli/reg/2016/679/art_19/oj": "GDPR Art.19", "https://eur-lex.europa.eu/eli/reg/2016/679/art_20/oj": "GDPR Art.20", "https://eur-lex.europa.eu/eli/reg/2016/679/art_21/oj": "GDPR Art.21", "https://eur-lex.europa.eu/eli/reg/2016/679/art_22/oj": "GDPR Art.22", "https://eur-lex.europa.eu/eli/reg/2016/679/art_7/par_3/oj": "GDPR Art.7-3", "https://eur-lex.europa.eu/eli/reg/2016/679/art_77/oj": "GDPR Art.77", "https://edpb.europa.eu/system/files/2021-06/edpb_recommendations_202001vo.2.0_supplementarymeasurestransferstools_en.pdf": "EDPB Recommendations 01/2020 on Supplementary Measures and Transfer Tools", "https://eur-lex.europa.eu/eli/reg/2016/679/art_4/par_20/oj": "GDPR Art.4-20", "https://eur-lex.europa.eu/eli/reg/2016/679/art_46/pnt_c/oj": "GDPR Art.46", "https://edpb.europa.eu/sites/default/files/consultation/edpb_recommendations_202001_supplementarymeasurestransferstools_en.pdf": "EDPB Recommendations 01/2020 on Supplementary Measures and Transfer Tools", "https://eur-lex.europa.eu/eli/dec_impl/2021/914/oj": "Implementing Decision on SCC for Data Transfers", "https://enterprivacy.com/wp-content/uploads/2018/09/Categories-of-Personal-Information.pdf": "EnterPrivacy Categories of Personal Information"}
\ No newline at end of file
+{"https://eur-lex.europa.eu/eli/reg/2016/679/art_4/par_1/oj": "GDPR Art.4-1", "https://www.specialprivacy.eu/": "SPECIAL Project", "https://eur-lex.europa.eu/eli/reg/2016/679/art_4/par_9/oj": "GDPR Art.4-9g", "https://eur-lex.europa.eu/eli/reg/2016/679/art_4/par_7/oj": "GDPR Art.4-7g", "https://www.w3.org/community/dpvcg/": "DPVCG", "https://eur-lex.europa.eu/eli/reg/2016/679/art_9/par_1/oj": "GDPR Art.9-1", "https://www.privacycommission.be/nl/model-voor-een-register-van-de-verwerkingsactiviteiten": "Belgian DPA ROPA Template", "https://eur-lex.europa.eu/eli/reg/2016/679/art_4/par_2/oj": "GDPR Art.4-2", "https://www.specialprivacy.eu/vocabs/processing": "SPECIAL Project", "https://eur-lex.europa.eu/eli/reg/2016/679/art_4/par_5/oj": "GDPR Art.4-5", "https://www.iso.org/iso-31000-risk-management.html": "ISO 31000", "https://eur-lex.europa.eu/eli/reg/2016/679/art_27/oj": "GDPR Art.27", "http://purl.org/adms": "ADMS controlled vocabulary", "https://eur-lex.europa.eu/eli/reg/2016/679/art_4/par_8/oj": "GDPR Art.4-8", "https://eur-lex.europa.eu/eli/reg/2016/679/art_4/par_10/oj": "GDPR Art.4-10", "https://edpb.europa.eu/our-work-tools/our-documents/recommendations/recommendations-012020-measures-supplement-transfer_en": "EDPB Recommendations 01/2020 on Data Transfers", "https://eur-lex.europa.eu/eli/reg/2016/679/art_37/oj": "GDPR Art.37", "https://eur-lex.europa.eu/eli/reg/2016/679/art_4/par_26/oj": "GDPR Art.4-26", "https://eur-lex.europa.eu/eli/reg/2016/679/art_6/par_1/pnt_a/oj": "GDPR Art.6-1a", "https://eur-lex.europa.eu/eli/reg/2016/679/art_6/par_1/pnt_b/oj": "GDPR Art.6-1b", "https://eur-lex.europa.eu/eli/reg/2016/679/art_6/par_1/pnt_c/oj": "GDPR Art.6-1c", "https://eur-lex.europa.eu/eli/reg/2016/679/art_6/par_1/pnt_d/oj": "GDPR Art.6-1d", "https://eur-lex.europa.eu/eli/reg/2016/679/art_6/par_1/pnt_e/oj": "GDPR Art.6-1e", "https://eur-lex.europa.eu/eli/reg/2016/679/art_6/par_1/pnt_f/oj": "GDPR Art.6-1f", "https://eur-lex.europa.eu/eli/reg/2016/679/art_9/par_2/pnt_a/oj": "GDPR Art.9-2a", "https://eur-lex.europa.eu/eli/reg/2016/679/art_9/par_2/pnt_b/oj": "GDPR Art.9-2b", "https://eur-lex.europa.eu/eli/reg/2016/679/art_9/par_2/pnt_c/oj": "GDPR Art.9-2c", "https://eur-lex.europa.eu/eli/reg/2016/679/art_9/par_2/pnt_d/oj": "GDPR Art.9-2d", "https://eur-lex.europa.eu/eli/reg/2016/679/art_9/par_2/pnt_e/oj": "GDPR Art.9-2e", "https://eur-lex.europa.eu/eli/reg/2016/679/art_9/par_2/pnt_f/oj": "GDPR Art.9-2f", "https://eur-lex.europa.eu/eli/reg/2016/679/art_9/par_2/pnt_g/oj": "GDPR Art.9-2g", "https://eur-lex.europa.eu/eli/reg/2016/679/art_9/par_2/pnt_h/oj": "GDPR Art.9-2h", "https://eur-lex.europa.eu/eli/reg/2016/679/art_9/par_2/pnt_i/oj": "GDPR Art.9-2i", "https://eur-lex.europa.eu/eli/reg/2016/679/art_9/par_2/pnt_j/oj": "GDPR Art.9-2j", "https://eur-lex.europa.eu/eli/reg/2016/679/art_45/par_3/oj": "GDPR Art.45-3", "https://eur-lex.europa.eu/eli/reg/2016/679/art_46/par_2/pnt_a/oj": "GDPR Art.46-2a", "https://eur-lex.europa.eu/eli/reg/2016/679/art_46/par_2/pnt_b/oj": "GDPR Art.46-2b", "https://eur-lex.europa.eu/eli/reg/2016/679/art_46/par_2/pnt_c/oj": "GDPR Art.46-2c", "https://eur-lex.europa.eu/eli/reg/2016/679/art_46/par_2/pnt_d/oj": "GDPR Art.46-2d", "https://eur-lex.europa.eu/eli/reg/2016/679/art_46/par_2/pnt_e/oj": "GDPR Art.46-2e", "https://eur-lex.europa.eu/eli/reg/2016/679/art_46/par_2/pnt_f/oj": "GDPR Art.46-2f", "https://eur-lex.europa.eu/eli/reg/2016/679/art_46/par_3/pnt_a/oj": "GDPR Art.46-3a", "https://eur-lex.europa.eu/eli/reg/2016/679/art_46/par_3/pnt_b/oj": "GDPR Art.46-3b", "https://eur-lex.europa.eu/eli/reg/2016/679/art_49/par_1/pnt_a/oj": "GDPR Art.49-1a", "https://eur-lex.europa.eu/eli/reg/2016/679/art_49/par_1/pnt_b/oj": "GDPR Art.49-1b", "https://eur-lex.europa.eu/eli/reg/2016/679/art_49/par_1/pnt_c/oj": "GDPR Art.49-1c", "https://eur-lex.europa.eu/eli/reg/2016/679/art_49/par_1/pnt_d/oj": "GDPR Art.49-1d", "https://eur-lex.europa.eu/eli/reg/2016/679/art_49/par_1/pnt_e/oj": "GDPR Art.49-1e", "https://eur-lex.europa.eu/eli/reg/2016/679/art_49/par_1/pnt_f/oj": "GDPR Art.49-1f", "https://eur-lex.europa.eu/eli/reg/2016/679/art_49/par_1/pnt_g/oj": "GDPR Art.49-1g", "https://eur-lex.europa.eu/eli/reg/2016/679/art_49/par_2/oj": "GDPR Art.49-2", "https://eur-lex.europa.eu/eli/reg/2016/679/art_13/oj": "GDPR Art.13", "https://eur-lex.europa.eu/eli/reg/2016/679/art_14/oj": "GDPR Art.14", "https://eur-lex.europa.eu/eli/reg/2016/679/art_15/oj": "GDPR Art.15", "https://eur-lex.europa.eu/eli/reg/2016/679/art_16/oj": "GDPR Art.16", "https://eur-lex.europa.eu/eli/reg/2016/679/art_17/oj": "GDPR Art.17", "https://eur-lex.europa.eu/eli/reg/2016/679/art_18/oj": "GDPR Art.18", "https://eur-lex.europa.eu/eli/reg/2016/679/art_19/oj": "GDPR Art.19", "https://eur-lex.europa.eu/eli/reg/2016/679/art_20/oj": "GDPR Art.20", "https://eur-lex.europa.eu/eli/reg/2016/679/art_21/oj": "GDPR Art.21", "https://eur-lex.europa.eu/eli/reg/2016/679/art_22/oj": "GDPR Art.22", "https://eur-lex.europa.eu/eli/reg/2016/679/art_7/par_3/oj": "GDPR Art.7-3", "https://eur-lex.europa.eu/eli/reg/2016/679/art_77/oj": "GDPR Art.77", "https://edpb.europa.eu/system/files/2021-06/edpb_recommendations_202001vo.2.0_supplementarymeasurestransferstools_en.pdf": "EDPB Recommendations 01/2020 on Supplementary Measures and Transfer Tools", "https://eur-lex.europa.eu/eli/reg/2016/679/art_4/par_20/oj": "GDPR Art.4-20", "https://eur-lex.europa.eu/eli/reg/2016/679/art_46/pnt_c/oj": "GDPR Art.46", "https://edpb.europa.eu/sites/default/files/consultation/edpb_recommendations_202001_supplementarymeasurestransferstools_en.pdf": "EDPB Recommendations 01/2020 on Supplementary Measures and Transfer Tools", "https://eur-lex.europa.eu/eli/dec_impl/2021/914/oj": "Implementing Decision on SCC for Data Transfers", "https://enterprivacy.com/wp-content/uploads/2018/09/Categories-of-Personal-Information.pdf": "EnterPrivacy Categories of Personal Information"}
\ No newline at end of file
diff --git a/documentation-generator/jinja2_resources/macro_dpv_document_family.jinja2 b/documentation-generator/jinja2_resources/macro_dpv_document_family.jinja2
new file mode 100644
index 000000000..40e03f010
--- /dev/null
+++ b/documentation-generator/jinja2_resources/macro_dpv_document_family.jinja2
@@ -0,0 +1,43 @@
+{% macro dpv_document_family(document=None) %}
+{# references.json contains the title/href for documents which is automatically expanded by ReSpec ; this macro contains only the document listing and text #}
+ Newcomers to the DPV are strongly recommended to first read through the Primer to familiarise themselves with the semantics and concepts of DPV. Related LinksDPV Family of Documents
+
+
+
+
+
+
+
The peer-reviewed article “Creating A Vocabulary for Data Privacy” presents a historical overview of the DPVCG, and describes the methodology and structure of the DPV along with describing its creation. An open-access version can be accessed here, here, and here.
+This document is published by the Data Privacy Vocabularies and Controls Community Group (DPVCG) as a deliverable and report of its work in creating and maintaining the Data Privacy Vocabulary (DPV).
+Contributing to the DPV and its extensions The DPVCG welcomes participation regarding the DPV, including expansion or refinement of its terms, addressing open issues, and welcomes suggestions on their resolution or mitigation.
+For contributions to the DPV, please see the section on GitHub. The current list of open issues and their discussions to date can be found at GitHub issues.
+DPV-LEGAL extends the [[[DPV]]] to provide concepts related to ...
+The namespace for terms in DPV-LEGAL is https://www.w3id.org/dpv/dpv-legal#
+ The suggested prefix for the namespace is dpv-legal
+ The DPV-LEGAL vocabulary and its documentation is available on GitHub.
The Data Privacy Vocabulary (DPV) provides terms to annotate and categorise instances of legally compliant personal data handling. In particular, the vocabulary provides Location
, Law
, and Authority
as top-level concepts. Since these concepts are specifically defined within the scope of jurisdictional laws, their implementation is provided as a separate vocabulary that extends the DPV, thereby permitting continued usage of DPV as a jurisdiction-agnostic and generic vocabulary.
This vocabulary, termed as DPV-LEGAL, provides several collections of concepts based on extending those from the DPV regarding:
+We welcome further contributions to refine or extend this data
+{% if not prefix %}{{term.iri|fragment_this}}{% else %}{{term.iri|fragment_this}}{% endif %}
Label: | +{{term.skos_prefLabel__en}} | +
Label (DE): | +{{term.skos_prefLabel__de}} | +
Website: | +link | +
Temporal start: | +{{ term.dct_temporal.time_hasBeginning.time_inXSDDate }} + |
Temporal end: | +{{ term.dct_temporal.time_hasEnd.time_inXSDDate }} | +
Jurisdictions: | +{% if term.dpv_hasJurisdiction is sequence %}{% for loc in term.dpv_hasJurisdiction|sort(attribute='iri') %}{{ loc.iri|prefix_this }}{{ ", " if not loop.last }}{% endfor %}{% else %}{{ term.dpv_hasJurisdiction.iri|prefix_this }}{% endif %} | +
Authority | +{% if term.dpv_hasAuthority is sequence %}{% for loc in term.dpv_hasAuthority %}{{ loc.iri|prefix_this }}{{ ", " if not loop.last }}{% endfor %}{% else %}{{ term.dpv_hasAuthority.iri|prefix_this }}{% endif %} | +
Concept Created: | ++ |
Concept Modified: | ++ |
Contributor(s): | ++ {% if term.dct_creator is sequence and not term.dct_creator is string %}{% for person in term.dct_creator|sort %} + {{person}}{{', ' if not loop.last }} + {% endfor %}{% else %} + {{term.dct_creator}} + {% endif %} + | +
See Also: | ++ {% if term.skos_related is sequence and not term.skos_related is string %}{% for link in term.skos_related %} + {{link|prefix_this}}{{", " if not loop.last}} + {% endfor %}{% else %} + {{term.skos_related|prefix_this}} + {% endif %} + | +
{% if not prefix %}{{term.iri|fragment_this}}{% else %}{{term.iri|fragment_this}}{% endif %}
Label: | +{{term.dct_title}} | +
Type: | +{% for t in term.rdf_type %}{% if t.iri.startswith('https://w3id.org/dpv#') %}{{ t|prefix_this }}{% endif %}{% endfor %} | +
has broader: | +{{ term.skos_broaderTransitive|prefix_this }} | +
Temporal start: | +{{ term.dct_temporal.time_hasBeginning.time_inXSDDate }} + |
Temporal end: | +{{ term.dct_temporal.time_hasEnd.time_inXSDDate }} | +
Members: | +{% if term.dpv_hasCountry is sequence %}{% for loc in term.dpv_hasCountry|sort(attribute='iri') %}{{ loc.iri|prefix_this }}{{ ", " if not loop.last }}{% endfor %}{% else %}{{ term.dpv_hasCountry.iri|prefix_this }}{% endif %} | +
Concept Created: | ++ |
Concept Modified: | ++ |
Contributor(s): | ++ {% if term.dct_creator is sequence and not term.dct_creator is string %}{% for person in term.dct_creator|sort %} + {{person}}{{', ' if not loop.last }} + {% endfor %}{% else %} + {{term.dct_creator}} + {% endif %} + | +
See Also: | ++ {% if term.skos_related is sequence and not term.skos_related is string %}{% for link in term.skos_related %} + {{link|prefix_this}}{{", " if not loop.last}} + {% endfor %}{% else %} + {{term.skos_related|prefix_this}} + {% endif %} + | +
{% if not prefix %}{{term.iri|fragment_this}}{% else %}{{term.iri|fragment_this}}{% endif %}
Label: | +{{term.dct_title}} | +
Jurisdictions: | +{% if term.dpv_hasJurisdiction is sequence %}{% for loc in term.dpv_hasJurisdiction|sort(attribute='iri') %}{{ loc.iri|prefix_this }}{{ ", " if not loop.last }}{% endfor %}{% else %}{{ term.dpv_hasJurisdiction.iri|prefix_this }}{% endif %} | +
Temporal start: | +{{ term.dct_temporal.time_hasBeginning.time_inXSDDate }} + |
Temporal end: | +{{ term.dct_temporal.time_hasEnd.time_inXSDDate }} | +
Website: | +link | +
Concept Created: | ++ |
Concept Modified: | ++ |
Contributor(s): | ++ {% if term.dct_creator is sequence and not term.dct_creator is string %}{% for person in term.dct_creator|sort %} + {{person}}{{', ' if not loop.last }} + {% endfor %}{% else %} + {{term.dct_creator}} + {% endif %} + | +
See Also: | ++ {% if term.skos_related is sequence and not term.skos_related is string %}{% for link in term.skos_related %} + {{link|prefix_this}}{{", " if not loop.last}} + {% endfor %}{% else %} + {{term.skos_related|prefix_this}} + {% endif %} + | +
{% if not prefix %}{{term.iri|fragment_this}}{% else %}{{term.iri|fragment_this}}{% endif %}
Label: | +{{term.dct_title}} | +
Type: | +{% for t in term.rdf_type %}{% if t.iri.startswith('https://w3id.org/dpv#') %}{{ t|prefix_this }}{% endif %}{% endfor %} | +
has broader: | +{% if term.skos_broaderTransitive is sequence %}{% for loc in term.skos_broaderTransitive|sort(attribute="iri") %}{{ loc|prefix_this }}{{ ", " if not loop.last }}{% endfor %}{% else %}{{ term.skos_broaderTransitive|prefix_this }}{% endif %} | +
Concept Created: | ++ |
Concept Modified: | ++ |
Contributor(s): | ++ {% if term.dct_creator is sequence and not term.dct_creator is string %}{% for person in term.dct_creator|sort %} + {{person}}{{', ' if not loop.last }} + {% endfor %}{% else %} + {{term.dct_creator}} + {% endif %} + | +
See Also: | ++ {% if term.skos_related is sequence and not term.skos_related is string %}{% for link in term.skos_related %} + {{link|prefix_this}}{{", " if not loop.last}} + {% endfor %}{% else %} + {{term.skos_related|prefix_this}} + {% endif %} + | +
The following terms have been proposed for inclusion, and are under discussion. They are provided here for illustrative purposes and should not be considered as part of DPV.
+ {% for name, terms in proposed.items() %} + {{name}} +This document lists the various guides created by the DPVCG and the community providing guidance for the adoption and use of DPV in terms of its concepts and serialisations, or regarding the application of DPV for specific applications or domains.
+The DPVCG invites contributions regarding additional guides as well as updates to existing guides. + {{ dpv_document_family(document='dpv-guides') }} + {{ sotd() }} +
The [[[DPV-GUIDE-OWL2]]] provides guidance for the use of DPV as an OWL2 ontology, and explains how DPV can be easily encoded in a low-complexity profile of OWL2 called OWL2-PL.
+This document acts as a guide presenting how [[[DPV]]], through its OWL2 encoding (i.e. [[DPV-OWL]]) can be used as an OWL2 vocabulary by easily encoding it in a low-complexity profile of OWL2 called OWL2-PL.
+ {{ dpv_document_family(document='dpv-guide-owl2') }} + {{ sotd() }} +DPV’s concept are treated as OWL2 classes, and DPV’s properties as OWL2 object properties. For example, the term Location represent the class of all points on the earth’s surface; the term Collect denotes the class of all possible data collection operations. The term Marketing represents the class of all possible varieties of purposes related to marketing, including all kinds of DirectMarketing, Advertising, etc.
+The reason for encoding DPV concepts as classes is simple: they are meant to be extended and refined for specific domains and use cases, as described in section 4 of the primer. For example, the purpose AcademicResearch may be specialized by introducing two subclasses, one for medical research and one for AI research. The processing category Collect, that denotes all personal data collection operations, can be specialized by two subclasses, one for the direct collection from the data subject, and one for the import of personal data from a third party, such as Open-ID or a Solid repository, for example. The new subclasses, in turn can be further specialized to support extreme extensibility and granularity. For example, the class Location can be refined by a class EULocation representing all the coordinates that fall within the borders of the European Union; in turn, EULocation can be refined by further terms, one for each country belonging to the Union; such terms can be further refined by classes that represent increasingly specific areas, such as regions, cities, streets, buildings, and even rooms in a building.
+OWL2 provides strong interoperability guarantees: its direct semantics constitutes a crisp semantic specification, that guarantees that all compliant software agents understand and process DPV and its extensions in the same way.
+The formal semantics of OWL2 provides also other guarantees, for example it can be formally proved that the compliance checking algorithms never return any false positive or negative. This robustness property and flexibility, extensibility and interoperability, make OWL2 the recommended machine-understandable encoding for automated compliance checking (of privacy policies – or records of processing – with respect to data protection regulations and the data subjects’ consent to personal data processing).
+OWL2 does not mean poor performance: for example, using the specialized Java compliance checker developed in SPECIAL a compliance check takes only a few hundred μ-seconds.
+It is not necessary to master OWL2 in its full complexity, to encode DPV in OWL2; a few simple features suffice, namely:
+subclass assertions (to state – say – that DataSubjectRight is a subclass of Right)
property range assertions (e.g. to say that the values of the hasPurpose property are instances of Purpose)
No complex classes are needed, only class names. Optionally, DPV’s semantics can be refined by asserting formally that two classes are disjoint:
+disjoint classes assertions (e.g. Purpose and Recipient have no instance in common)
Such disjointness assertions may help in identifying wrong uses of DPV (like the assignment of a recipient to property hasPurpose).
+To give a concrete example, let us show how the term MedicalHealth is encoded in OWL2. In the following examples we use the Manchester syntax of OWL2.
+ + + + +In a few cases, it may be appropriate to introduce new terms that are instances of a class, in particular when the new term represents:
+a single, specific recipient or controller,
a single location, such as those specified by GPS coordinates,
other single data values, such as specific email addresses (like “jane.doe@provider.org”), specific social security numbers, and the like.
The rule of thumb to decide whether a term should be a class or an instance is based on the question: can the new term possibly be further refined? If the answer is “yes”, then make it a class, otherwise it can be made an instance. In case of doubt, making the new term a class is the safest option.
+Instances can be added to the ontology with:
+instance assertions (e.g. stating that the ACME company is an instance of Recipient).
The OWL2 keyword for declaring instances is “Individual”. Its use is illustrated in the following example:
+ +The low-complexity profile OWL2-PL allows to define properties and classes with little more than the OWL2 keywords illustrated in the previous section. The complete list of keywords that can be used for this purpose is the following:
+ +No other OWL2 keywords are accepted in the OWL2-PL ontologies that formalize terminologies such as the DPV and its extensions.
+In the above list, only the last feature has not yet been explained. It is applied in the use cases related to policies and compliance in order to define functional properties of the data processing operations (that is, properties that may have only one value). This task is under the responsibility of the policy language designer. The users who extend the DPV, as illustrated in the previous section, do not need to use this feature.
+The OWL2 encoding of DPV can be used to encode data usage policies. A data usage policy can be:
+the privacy policy of a company or organization,
the record of processing maintained by a company or organization,
the consent statement of a data subject,
the objective part of a personal data protection regulation such as the GDPR.
Data usage policies are modeled as classes. More precisely, a policy is nothing but the class of permitted data processing operations. A policy P1 (like the privacy policy of a company) complies with a policy P2 (like the consent policy of a data subject, or the objective part of the GDPR) if all the operations permitted by P1 are also permitted by P2, that is,
+P1 complies with P2 if, and only if, P1 is a subclass of P2.
+A policy can be specified in a compact way by describing the properties of its permitted operations.
+In a personal data protection use case, the relevant properties of such operations include (not exclusively) the category of data being processed, the purpose of the processing, the kind of processing (e.g. data collection, data analysis, etc.), the recipients to which the data is transferred, and other properties related to how data is stored and protected, and the legal basis of the processing.
+For example, let Policy1 be a privacy policy that permits the collection of email addresses for advertising purposes, based on the consent of the data subjects. This policy – seen as a class of operations – contains all the possible operations with the following DPV features:
+their hasProcessing attribute is some kind of data collection (i.e. an instance of Collection)
their hasPersonalData attribute is some email address
their hasPurpose attribute is some kind of advertising (i.e. an instance of Advertising)
their hasLegalBasis attribute is some kind of consent (either explicit or implicit)
This class of operations is encoded in OWL2-PL as follows:
+ +where A6-1-a-consent is a superclass of A6-1-a-explicit-consent and A6-1-a-non-explicit-consent. In a similar way, one can encode a consent statement that permits all the operations that collect contact information for marketing purposes:
+ +Now, since EmailAddress is a subclass of Contact and Advertising is a subclass of Marketing, all the operations that belong to Policy1 satisfy also the requirements for being in Policy2 as well. Then Policy1 is a subclass of Policy2, that is, the privacy policy complies with the data subject’s consent.
+It is also easy to say that some properties of a policy must have a single value (an instance). For example, the policy Policy3 stating that the email address “user@provider.com” (and only that mail address) can be transferred to the ACME company (and only that company) for any marketing purposes is encoded in OWL2-PL as follows:
+ +The expressions {user@provider.com} and {myext:ACME} denote the classes that contain only user@provider.com and ACME, respectively. Therefore, all the operations permitted by Policy3 process only the address “user@provider.com” and share it only with ACME, as required. Any other operation is forbidden.
+Interestingly, every OWL2 reasoner can be used to check compliance, because they are all able to check whether a class is a subclass of another class. Of course, the reasoners specialized for OWL2-PL are in general more efficient than generic OWL2 reasoners. The part of OWL2-PL aimed at encoding policies supports the above syntax, that is, policies can be defined using the keywords “EquivalentTo”, “and”, and “some” as shown above. Each policy must have a unique definition.
+This kind of automated compliance checking has several interesting applications, including the following:
+Consent re-use: when a new application or business line is deployed, its data usage policy can be automatically checked for compliance with the consent statements that are already available; in this way, the users whose consent already covers the new processing need not be asked again for consent.
Data re-purposing: an organization wants to use its data for a purpose different from that declared when the data was collected. Similarly to consent re-use, it can be checked automatically which data can be used for the new purpose based on the available consent.
Data sharing: The personal information contained in a container like a Solid pod or an Open-ID account can be labeled with a consent policy that determines whether a third party can access it, based on the data usage policy of the requester. This method improves the existing access control language by taking into account additional relevant information such as purposes and transfers to third parties.
Dynamic query control: when an employee queries personal data, the purpose of the query and the data categories involved in the query can be checked on-the-fly for compliance with the available consent.
Compliance with the regulations: the personal data usage policies of a company can be checked for compliance with the objective part of the data protection regulations. Every single compliance check verifies a number of restrictions, including (not exclusively): (i) verifying whether international data transfers comply with GDPR’s restrictions; (ii) verifying whether special data categories are protected with the technical and organizational measures prescribed by the GDPR.
As it should be expected, modeling the GDPR is more complex, but its encoding into OWL2 and OWL2-PL is the responsibility of knowledge engineers, and it does not concern end users. The interested reader may find more details about the OWL2 encoding of the GDPR in several papers and documents, such as [1] and [2].
+The complete grammar of the policy language of OWL2-PL can be found in [3], together with a corresponding JSON version designed for the developers that are not familiar with OWL2.
+OWL2-PL and the semantic policy framework have been developed by the H2020 projects SPECIAL and TRAPEZE, funded by the European Union under grants n. 731601 and 883464, respectively.
+[1] Piero A. Bonatti, Sabrina Kirrane, Iliana M. Petrova, Luigi Sauro: Machine Understandable Policies and GDPR Compliance Checking. Künstliche Intell. 34(3): 303-315 (2020)
+[2] P.A. Bonatti, L. Ioffredo, G. Luongo, S. Mosi, I.M. Petrova, L. Sauro: Formalization of the GDPR and of the Pilots’ Policies. SPECIAL Report D5. Available at: https://specialprivacy.ercim.eu/images/documents/report-D5.pdf
+[3] P.A. Bonatti, J. Langens, L. Sauro: Policy Language – v1. TRAPEZE Report D2.1. Available at: https://trapeze-project.eu/resources/
+