[Idea] Support OpenDAL

[alberttwong]

Is there an existing issue for this?

• I have searched the existing issues

Description of the bug

OpenDAL offers a unified data access layer, empowering users to seamlessly and efficiently retrieve data from diverse storage services. Our goal is to deliver a comprehensive solution for any languages, methods, integrations, and services.

ParadeDB uses Apache OpenDAL to integrate with object stores. If we support OpenDAL, all databases that use OpenDAL will be able to use Hudi

Steps To Reproduce

n/a

Expected behavior

n/a

Screenshots / Logs

No response

Software information

n/a

Additional context

No response

[alberttwong]

apache/opendal#5065

[Xuanwo]

Thank you for bringing this up. I'm happy to help.

[kazdy]

there's a crate that brings opendal integrations to object_store hudi-rs uses: https://crates.io/crates/object_store_opendal
It may be worth considering this instead of replacing it with opendal completely?

[xushiyan]

@Xuanwo and I had a chat - we don't have to bridge the APIs as the current object_store footprint is still small within the codebase - the API migration won't be too much of changes. And we should make use of opendal's capabilities to support more storage systems natively.

[Xuanwo]

Hi, @xushiyan and @alberttwong, I'm working on this now. One blocker here is opendal doesn't support construct from url yet, I will add this support in opendal first.

Tracked at apache/opendal#5407