diff --git a/CHANGELOG.md b/CHANGELOG.md index c8c80eb..f6c1de6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,3 +3,27 @@ This document outlines major changes between releases. ## [Unreleased] + +## [0.1.2] - 2024-09-25 + +### Added +- More debug logs (#20) + +## [0.1.1] - 2024-09-17 + +### Added +- Allow to configure several sources (#16) + +### Fixed +- Don't create redundant delete markers (#17) +- Don't user btoken for separate lifecycle container (#19) + +## [0.1.0] - 2024-07-26 + +### Added +- Basic lifecycle logic (#1, #2, #3, #4) + +[0.1.0]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-lifecycler/compare/27189a38bb...v0.1.0 +[0.1.1]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-lifecycler/compare/v0.1.0...v0.1.1 +[0.1.2]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-lifecycler/compare/v0.1.1...v0.1.2 +[Unreleased]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-lifecycler/compare/v0.1.2...master diff --git a/README.md b/README.md index e7b42d3..ad8a32d 100644 --- a/README.md +++ b/README.md @@ -1 +1,133 @@ # FrostFS S3 Lifecycler + +The purpose of this service is to provide lifecycle management of objects +(https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html). + +This service works with objects and lifecycle configurations uploaded to FrostFS mainly by using +[s3-gw](https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw). + + +## Build and run + +To build service use the following command: + +```shell +$ make +``` + +To run service use the following command: + +```shell +$ ./bin/frostfs-s3-lifecycler --config config.yaml +``` + +Minimal config example to run with [dev-env](https://git.frostfs.info/TrueCloudLab/frostfs-dev-env) can be: + +```yaml +wallet: + path: ./frostfs-dev-env/wallets/wallet.json + address: NbUgTSFvPmsRxmGeWpuuGeJUoRoi6PErcM + passphrase: "" + +morph: + rpc_endpoint: + - address: ws://morph-chain.frostfs.devenv:30333/ws + +credential: + use: wallets + source: + wallets: + - path: ./frostfs-dev-env/wallets/wallet.json + address: NbUgTSFvPmsRxmGeWpuuGeJUoRoi6PErcM + passphrase: "" + +frostfs: + peers: + 0: + priority: 1 + weight: 1 + address: s01.frostfs.devenv:8080 +``` + +More detailed description of each parameter can be found [here](docs/configuration.md). + +## Description + +Once running service listens new epoch notification from FrostFS network. +On each epoch service lists all users from `frostfsid` +[contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/frostfsid) and check each +user bucket lifecycle configuration to validate if objects or multipart uploads should be deleted according to this +configuration. + +> **NOTE** +> +> Lifecycler handles only expiration actions (transition isn't supported). + +If object should be deleted or multipart aborted lifecycler service perform delete action. It must have user credential +for this operation, so service must be provided with private key of each user. It can be done by specifying wallets +in config: + +```yaml +credential: + use: wallets + source: + wallets: + - path: ./frostfs-dev-env/wallets/wallet.json + address: NbUgTSFvPmsRxmGeWpuuGeJUoRoi6PErcM + passphrase: "" +``` + +Currently only wallet source is supported. + +### S3 + +To apply lifecycle configuration to bucket we must invoke +[PutBucketLifecycleConfiguration](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycleConfiguration.html) +method on [s3-gw](https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw). It can be done using +[AWS CLI](https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-lifecycle-configuration.html). + +Lifecycle configuration example: + +```json +{ + "Rules": [ + { + "Status": "Enabled", + "ID": "Abort 1 day old multiparts", + "AbortIncompleteMultipartUpload": { + "DaysAfterInitiation": 1 + } + }, + { + "Status": "Enabled", + "ID": "Expire objects with prefix after specific date", + "Expiration": { + "Date": "2024-07-31T06:32:00Z" + }, + "Filter": { + "Prefix": "prefix" + } + }, + { + "Status": "Enabled", + "ID": "Expire objects by tags after specific date", + "Expiration": { + "Date": "2024-07-31T06:32:00Z" + }, + "Filter": { + "And": { + "Prefix": "tags", + "Tags": [{ + "Key":"tag-key", + "Value":"tag-val" + }] + } + } + } + ] +} +``` + +Details can be found in AWS documentaion +* https://docs.aws.amazon.com/AmazonS3/latest/userguide/intro-lifecycle-rules.html +* https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-configuration-examples.html diff --git a/VERSION b/VERSION index b82608c..5366600 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -v0.1.0 +v0.1.2