feat(py): Add support for storing models in S3

[syntaxsdev]

Within the Python client, users will be able to directly store models to an S3 compatible object storage.
AWS S3 connections will automatically be consumed.

Description

The bulk of the changes were done in clients/python/src/_client.py
With E2E tests that were added to test functionality by creating a Minio instance

This implementation is within the ModelRegistry class

Example usage:

mr = ModelRegistry(...)
mr.save_to_s3(
  path="models",
  bucket_name="default",
  endpoint_url="xxx",
  access_key_id="xxx",
  secret_access_key="xxx"
)

You can path both files and paths to the path parameter.
If a path is nested (ex. data/models) then an s3_prefix variable will be required to allow for proper folder naming.
Environment variables are picked up, but variables supplied via direct params are preferred over environment variables that are mounted.

How Has This Been Tested?

A local and remote version of Minio S3

1. Changes to the Makefile target were made to add a local Minio instance for the same Kind cluster than the Makefile target deploy-latest-mr uses.
2. An .env file is generated so that vars can be used for the added tests
3. Three new tests were added that single for storing a singular model to S3 and for storing a path of models, recursively

Tests are passing when completely rebuilding and testing the new e2e workflow

To test, run make test-e2e as per usual.

Merge criteria:

• All the commits have been signed-off (To pass the DCO check)

• The commits have meaningful messages; the author will squash them after approval or in case of manual merges will ask to merge with squash.
• Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
• The developer has manually tested the changes and verified that the changes work.
• Code changes follow the kubeflow contribution guidelines.

If you have UI changes

• The developer has added tests or explained why testing cannot be added.
• Included any necessary screenshots or gifs if it was a UI change.
• Verify that UI/UX changes conform the UX guidelines for Kubeflow.

[tarilabs]

thank you @syntaxsdev for this!

some initial comments below.
which type of test can we consider to make sure the functionality is covered?

I'm thinking we could have some dedicated e2e test by extending the current opt-in pytest mechanism and deploy minio in that "scenario" of e2e testing. Do you have some additional ideas?

[tarilabs]

I would like to make sure the end results works as expected for KServe (not necessarily as a test in the PR itself, making the check manually can also do)

otherwise
/lgtm
already thanks @syntaxsdev

[tarilabs]

( I also believe this PR need rebasing to account for more recent openapi and generated changes )

[syntaxsdev]

Confirmed functionality works similarly by going through other kubeflow saving model demos

[rareddy]

@tarilabs @syntaxsdev one question I have is that I was assuming we are using pytest libraries for any cluster management too, is that a wrong assumption?

[tarilabs]

no we are not "orchestrating cluster setup from pytest" if that was the question, as the request was to stick to makefile and scripting; the downside is netting on a basic common ground, but I see the advantage as reuse from Go or else in the future if needed.

[rareddy]

Correct for reuse and manageability, this was the topic in shift left. @lugi0 might have some comments on this. I am not asking to change this PR but maybe we can capture a follow-up if that is the overall direction we need to be writing the tests with, if not we can ignore.

[tarilabs]

I'm favourable of revising the setup, as you know I prsonally prefer to reduce Makefile and scripts to the minimum as possible, but we need indeed to account for multiple factors.

[rareddy]

@lucferbux can you look into the issues with the UI tests here?

[lucferbux]

Agh, this might happen due to latest config PR cc @christianvogt I'm gonna take a look

[christianvogt]

A fix for the tests has been posted here #832

[dhirajsb]

API looks good to me, we should verify Makefile changes on all platforms.

[tarilabs]

thank you @syntaxsdev , and all for this PR, this is a good foundation step to make life easier to the DS + MLOps Eng to orchestrate the Store-then-Register flow!

/lgtm
/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tarilabs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [tarilabs]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment