Harvard Tech for Social Good: Data Warehouse Integration for Importing SDMX Format Related Tables #354

CChenalds17 · 2025-01-16T05:48:11Z

Harvard Tech for Social Good: Added Data Warehouse integration for importing SDMX Format Related Tables

DsdComponents

Stores the general frontend components and the functions that call Data Warehouse

DimensionDropdown.jsx

Parameters:

dimensionId - ID of the dimension
options - array of options to be displayed
selectedValues - array of selected values
onChange - callback function to handle changes in the selected options
customStyles - optional custom styles to be applied to the dropdown

Returns:

A frontend multi-select dropdown component with specified parameters

DropdownSection.jsx

Parameters:

title - title of the section
options - options for the dropdown select
selectedOption - currently selected option
onChange - function to call when the selected option changes
loading - boolean indicator for whether the data is currently loading
error - error message to display if an error occurs
placeholder - placeholder text for the dropdown select
customStyles - custom styles for the dropdown select

Returns:

A frontend section that contains the selection dropdowns for a given value

DsdForm.jsx

Parameters:

setRequest - state variable setter function that mutates request, the request/response payload state variable
urlChanged - function to read the API url created based on user inputs

Returns:

A frontend component containing all the selection / input fields

dsdFunctions.jsx

propagateAgencyOptions

Outputs:
A list of agencies, each specifying

id (string): The ID of the agency.
name (string): The name of the agency in English, or the ID if no English name is available.

restrictDataflowOptions

Inputs:

agencyParam (string): The ID of the agency to filter dataflows.

Outputs:
A list of dataflow objects, where each object contains:

name (string): The name of the dataflow.
id (string): The ID of the dataflow.
dsdId (string): The ID of the associated DSD (Data Structure Definition).
dataflowAgency (string): The ID of the agency that owns the dataflow.
If agencyParam is not provided, an empty list is returned.

updateDsd

Inputs:

dataflow (object): The dataflow object containing:
- dataflowAgency (string): The ID of the agency owning the dataflow.
- id (string): The ID of the dataflow.
dimensions (object): A mapping of dimension IDs to an array of their selected values.
dataflowVersion (string, optional): The version of the dataflow (default is "1.0").

Outputs:
An object containing:

updatedDimensions (object): A mapping of dimension IDs to their available values.
apiUrl (string): The constructed API URL used for the request.
apiResponse (object): The full API response from the server.
sdmxImplementation (array): A placeholder array for additional implementation details.
If an error occurs, an object with an error property is returned.

propagateDataflowVersions

Inputs:

dataflow (object): An object containing information about the selected dataflow:
- dataflowAgency (string): The ID of the agency owning the dataflow.
- value (string): The dataflow ID.

Outputs:

Returns an array of strings, where each string represents a version of the dataflow

updateDimensions

Inputs:

dataflow (object): The dataflow object containing:
- dataflowAgency (string): The ID of the agency owning the dataflow.
- dsdId (string): The ID of the associated DSD.
dataflowVersion (string, optional): The version of the dataflow (default is "1.0").

Outputs:
An object containing:

dimensionSelections (object): A mapping of dimension IDs to an empty array, initialized for each dimension.
updatedDimensions (object): A mapping of dimension IDs to their available values, updated from the API response.
If an error occurs, an object with an error property is returned.

fetchFunctions.jsx

fetchAgencies

Inputs:

setLoading (function): A function to update the loading state for agency fetching. Accepts a callback.
setAgencyOptions (function): A function to update the list of agency options. Expects an array of objects with the following keys:
- value (string): The agency ID.
- label (string): The agency name.
setError (function): A function to set error messages related to agency fetching. Accepts a callback.

fetchDataflows

Inputs:

setLoading (function): A function to update the loading state for dataflow fetching. Accepts a callback.
setDataflowOptions (function): A function to update the list of dataflow options. Expects an array of objects with the following keys:
- value (string): The dataflow ID.
- label (string): The dataflow name.
- dataflowAgency (string): The ID of the agency owning the dataflow.
- dsdId (string): The ID of the associated DSD.
setError (function): A function to set error messages related to dataflow fetching. Accepts a callback.
selectedAgency (object): The selected agency object with the following keys:
- value (string): The agency ID.
- label (string): The agency name.

fetchDataflowVersions

Inputs:

setLoading (function): A function to update the loading state for fetching dataflow versions. Accepts a callback.
setDataflowVersionOptions (function): A function to update the list of dataflow version options. Expects an array of objects with the following keys:
- value (string): The version number of the dataflow.
- label (string): A descriptive label for the version (e.g., Version 1.0).
setError (function): A function to set error messages related to fetching dataflow versions. Accepts a callback.
selectedDataflow (object): An object containing information about the selected dataflow:
- value (string): The dataflow ID.
- dataflowAgency (string): The ID of the agency owning the dataflow.

fetchDimensions

Inputs:

setLoading (function): A function to update the loading state for dimension fetching. Accepts a callback.
setError (function): A function to set error messages related to dimension fetching. Accepts a callback.
setDimensionOptions (function): A function to update the available dimension options. Expects an object where each key is a dimension ID, and the value is an array of objects with:
- value (string): The dimension value ID.
- label (string): The dimension value name.
setDimensionSelections (function): A function to update the selected dimension values. Expects an object where each key is a dimension ID, and the value is an empty array.
selectedDataflow (object): The selected dataflow object with the following keys:
- value (string): The dataflow ID.
- dataflowAgency (string): The ID of the agency owning the dataflow.
- dsdId (string): The ID of the associated DSD.

fetchDsd

Inputs:

selectedDataflow (object): The selected dataflow object with the following keys:
- value (string): The dataflow ID.
- dataflowAgency (string): The ID of the agency owning the dataflow.
- dsdId (string): The ID of the associated DSD.
dimensionSelections (object): A mapping of dimension IDs to arrays of their selected values.
setDsdResult (function): A function to update the DSD result. Expects an object with:
- updatedDimensions (object): A mapping of dimension IDs to their available values.
- apiUrl (string): The constructed API URL.
- apiResponse (object): The API response.
- sdmxImplementation (array): Placeholder for implementation details.
urlChanged (function): A function to update the URL. Expects a string (formatted URL).
setError (function): A function to set error messages related to DSD fetching. Accepts a callback.
setLoading (function): A function to update the loading state for DSD fetching. Accepts a callback.

testing making PR from charliechen branch to dev-t4sg

…ce done implementing. look at database-test-app/src/update_dsd for continuing work and TestUpdateDSD for testing.

- Currently testing app is broken with the new input - New input has not been debugged (some results may be invalid) - Documentation is still in progress - Resulting output is a dictionary with each key having an array of objects, and each object having an id, description and name, description is sometimes blank depending on the label

- Able to receive agency, dataflow, dataflow_version, and an object with dimensions - Will output updated_dimensions, api_response, api_url, sdmx_implementation

…lder

- Abstracted away components - Added helper functions - Organized styles

Charlie/midpoint updates

…ible labels

…es, cleaned up code

Charlie/mvp0 final

meomancer · 2025-01-17T01:28:24Z

django_project/geosight/data/migrations/0112_indicatorvaluedataset.py

@@ -0,0 +1,33 @@
+# Generated by Django 3.2.16 on 2024-11-02 15:14


Please revert this as latest code already handle this (please pull main branch)

meomancer · 2025-01-17T01:28:33Z

django_project/frontend/src/pages/Dashboard/index.jsx

@@ -26,6 +26,7 @@ import { EmbedConfig } from "../../utils/embed";
 import { LEFT, RIGHT } from "../../components/ToggleButton";
 import { ProjectOverview } from "./Toolbars";

+


Please revert this

meomancer · 2025-01-17T01:28:36Z

django_project/frontend/src/pages/Dashboard/index.jsx

@@ -35,6 +36,7 @@ export default function Dashboard({ children }) {
    user_permission
  } = useSelector(state => state.dashboard.data);

+


Please revert this

meomancer · 2025-01-17T01:29:05Z

.gitignore

@@ -25,6 +25,7 @@ django_project/frontend/package-lock.json
 django_project/frontend/webpack-stats.dev.json
 django_project/frontend/webpack-stats.prod.json
 django_project/geosight/georepo/api/mock/create_responses.py
+django_project/frontend/package.json


Please revert this or the package json is not found

meomancer · 2025-01-17T01:29:34Z

django_project/package.json

@@ -0,0 +1,14 @@
+{


Not sure what is this file

meomancer · 2025-01-17T01:29:37Z

django_project/package-lock.json

@@ -0,0 +1,13 @@
+{


Not sure what is this file

meomancer · 2025-01-17T01:30:38Z

django_project/frontend/src/pages/Admin/Importer/Form/Extensions/SDMX/IndicatorValue.jsx

@@ -49,8 +49,8 @@ import { updateDataWithSetState } from "../../utils";
 */
 export const SDMXIndicatorValue = forwardRef(
  ({
-     data, setData, files, setFiles, indicatorList


Please revert this (as no actual changes here)

meomancer · 2025-01-17T01:32:21Z

django_project/frontend/src/pages/Admin/Importer/Form/Extensions/SDMX/DsdComponents/config.js

@@ -0,0 +1,10 @@
+// config.js
+const API_URLS = {


Ee need to make the domain an input
So user can use other sdmx api to handle this

Hi! I believe our current implementation only handles one sdmx implementation (if that's what you mean). So, we don't really account for multiple sdmx implementations (which probably have different API behavior). We can include a "dummy" variable for the sdmx implementation though, if that's what you want. Alternatively, once we include multiple api implementations, we can easily add that to the function

meomancer · 2025-01-17T01:33:20Z

deployment/docker/Dockerfile

 # For Mac M1/M2 use the below code
-# FROM --platform=amd64 ubuntu:20.04 AS prod
+FROM --platform=amd64 ubuntu:20.04 AS prod


Please revert this (As it is for local)

Addressing Irwan's Changes

…taset.py per review feedback

CChenalds17 and others added 30 commits October 3, 2024 15:06

test commit

de967bd

change for pushing to my own branch

c2debfd

Merge pull request #1 from MichaelCai0301/charliechen/test-branch

91f023c

testing making PR from charliechen branch to dev-t4sg

Created a new app to build update_dsd and test it. Delete that app on…

2c78b6b

…ce done implementing. look at database-test-app/src/update_dsd for continuing work and TestUpdateDSD for testing.

Basics of update_dsd completed

000e5c1

- Able to receive agency, dataflow, dataflow_version, and an object with dimensions - Will output updated_dimensions, api_response, api_url, sdmx_implementation

Database Testing Frontend App Portion (Not Complete)

684fa6e

finish functions, work on frontend (NOT DONE)

cb07ca9

Add small visual changes + axios call for update_dsd

eed5952

edits to everything

fc87c73

updates

b5a987b

move files into django_project->frontend->src->pages->databasePlaceHo…

a753834

…lder

First commit of unorganized completed merging of database and frontend

75fdfea

Completed majority of my deliverables for Midpoint

9e56b3a

- Abstracted away components - Added helper functions - Organized styles

updated dropdown labels to include tags

697f0c7

removed url input field

99689d1

resets table upon agency change

03c93db

Merge pull request #12 from MichaelCai0301/charlie/midpoint-updates

2d8a1b7

Charlie/midpoint updates

Migrated DropdownSection to native GeoSight dropdowns

1e651e8

fully transferred all components to native ui. Also added aria/access…

c170c45

…ible labels

Implementing dataflow version dropdown and integrating everything else

c018dc9

Added handling for dataflow version

6a42333

removed style edits in Select.jsx

5115fa3

Final MVP0: added 'Load Data' button, changed dropdown menu appearanc…

edee039

…es, cleaned up code

MVP0: resolved import bug

de6eaf6

Merge pull request #19 from MichaelCai0301/charlie/mvp0-final

a2c9fc7

Charlie/mvp0 final

added doc

3e99208

added doc

e65c102

Merge pull request #20 from MichaelCai0301/charlie/mvp0-final

f1b6ea3

Charlie/mvp0 final

Merge branch 'unicef-drp:main' into dev-t4sg

758a766

deleted unncessary file

272cbd2

meomancer reviewed Jan 17, 2025

View reviewed changes

django_project/package.json Outdated

@@ -0,0 +1,14 @@

{

Copy link

Collaborator

meomancer Jan 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure what is this file

meomancer reviewed Jan 17, 2025

View reviewed changes

django_project/package-lock.json Outdated

@@ -0,0 +1,13 @@

{

Copy link

Collaborator

meomancer Jan 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure what is this file

meomancer reviewed Jan 17, 2025

View reviewed changes

CChenalds17 and others added 9 commits February 2, 2025 17:29

Merge remote-tracking branch 'upstream/main' into temp-branch

f68a040

reverted gitignore per review feedback

c8acefe

reverted Dockerfile per review feedback

efb9113

reverted whitespace changes per review feedback

b6dca12

reverted whitespace changes per review feedback

dd75ed6

deleted package.json and package-lock.json per review feedback

73cb58f

Merge pull request #21 from MichaelCai0301/temp-branch

8aeaf0f

Addressing Irwan's Changes

removed whitespace

dcf3b46

deleted django_project/geosight/data/migrations/0112_indicatorvalueda…

cfa2a8f

…taset.py per review feedback

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Harvard Tech for Social Good: Data Warehouse Integration for Importing SDMX Format Related Tables #354

Harvard Tech for Social Good: Data Warehouse Integration for Importing SDMX Format Related Tables #354

CChenalds17 commented Jan 16, 2025

meomancer Jan 17, 2025 •

edited

Loading

meomancer Jan 17, 2025

meomancer Jan 17, 2025

meomancer Jan 17, 2025

meomancer Jan 17, 2025

meomancer Jan 17, 2025

meomancer Jan 17, 2025

meomancer Jan 17, 2025 •

edited

Loading

MichaelCai0301 Jan 30, 2025

meomancer Jan 17, 2025

		@@ -0,0 +1,33 @@
		# Generated by Django 3.2.16 on 2024-11-02 15:14

		@@ -26,6 +26,7 @@ import { EmbedConfig } from "../../utils/embed";
		import { LEFT, RIGHT } from "../../components/ToggleButton";
		import { ProjectOverview } from "./Toolbars";

		@@ -35,6 +36,7 @@ export default function Dashboard({ children }) {
		user_permission
		} = useSelector(state => state.dashboard.data);

Harvard Tech for Social Good: Data Warehouse Integration for Importing SDMX Format Related Tables #354

Are you sure you want to change the base?

Harvard Tech for Social Good: Data Warehouse Integration for Importing SDMX Format Related Tables #354

Conversation

CChenalds17 commented Jan 16, 2025

DsdComponents

DimensionDropdown.jsx

DropdownSection.jsx

DsdForm.jsx

dsdFunctions.jsx

propagateAgencyOptions

restrictDataflowOptions

updateDsd

propagateDataflowVersions

updateDimensions

fetchFunctions.jsx

fetchAgencies

fetchDataflows

fetchDataflowVersions

fetchDimensions

fetchDsd

meomancer Jan 17, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

meomancer Jan 17, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

meomancer Jan 17, 2025 •

edited

Loading

meomancer Jan 17, 2025 •

edited

Loading