v4.0.0

v4.0.0 (2024-05-09)

⛔️ BREAKING CHANGES

• exporter: Add S3Exporter that implements cached S3 exporting of files from the local disk. (d17b2d2)
• exporter: Add BaseExporter and LocalExporter implementations that support exporting data to disk, along with corresponding Pipeline steps. (6ce13cf)

✨ Features

• exporter: Add LocalManifest support for LocalExporter which simplifies caching logic and enables S3 manifest translations. (2199ff0)
• exporter: Add support for multiple data export using LocalExporter. (ff988b6)
• data source: Add support for reading manifest files from S3 buckets in S3Ingester. (9c68a9b)
• pipeline: Add disable_cache parameter to Pipeline execution. (da1e31a)

🐛 Bug Fixes

• data cleaning: Fix newline characters breaking CSV reading using Arrow. (3a7e594)
• tuning: Delete logging of storage URI to minimize risk of accidentally logging credentials. (054692d)

🛠️ Code Refactoring

• data source: Extract shared S3 logic to utils which can be then used by S3Exporter. (97a7974)