Home

Data Bridge

version 0.1.0 BETA

Data Bridge is a Python3 application designed to transfer data from various data sources. Most of the transfer is done by creating a profile for your transfer, and have the Data Bridge read it to parse data.

A command usually looks like this:

python3 data-bridge.py --profile profilefile.json

The application will perform a sync based on the configurations defined in the profile json. The JSON file comprises of two main objects: source and dest. The app will first perform a query based on the source configuration. It'll then iterate through the results one row at a time, and perform the query defined in the dest config. To read more about how this works, please see profile.

Requirements

• MongoDB v4.0
• Python3
• unixodbc-dev - Needed in the Linux environment.
• pyodbc - Needed for SQL import.
• pymongo - Needed for MongoDB.
• python3-dateutil - Recording dates.
• pytz - Date formatter.

Installation

• Download the latest version (v0.1.0) from the tags, or here.
• Install all the required dependencies.
• Run it like this:

python3 data-bridge.py --profile {profilefile.file}

Options

Most of the magic is set in each profile json, which will is explained more here. There're a few optional args that can be used to compliment the sync.

• --help - Use this to open help and see a list of available options.
• --upsert - This will perform an "insert" if the item to update does not exist in the destination dataset.