Release v0.1.2
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
This commit is contained in:
parent
8b6d93c94d
commit
1cf24ae500
3 changed files with 157 additions and 1 deletions
24
CHANGELOG.md
24
CHANGELOG.md
|
@ -3,3 +3,27 @@
|
||||||
This document outlines major changes between releases.
|
This document outlines major changes between releases.
|
||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
## [0.1.2] - 2024-09-25
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- More debug logs (#20)
|
||||||
|
|
||||||
|
## [0.1.1] - 2024-09-17
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- Allow to configure several sources (#16)
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
- Don't create redundant delete markers (#17)
|
||||||
|
- Don't user btoken for separate lifecycle container (#19)
|
||||||
|
|
||||||
|
## [0.1.0] - 2024-07-26
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- Basic lifecycle logic (#1, #2, #3, #4)
|
||||||
|
|
||||||
|
[0.1.0]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-lifecycler/compare/27189a38bb...v0.1.0
|
||||||
|
[0.1.1]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-lifecycler/compare/v0.1.0...v0.1.1
|
||||||
|
[0.1.2]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-lifecycler/compare/v0.1.1...v0.1.2
|
||||||
|
[Unreleased]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-lifecycler/compare/v0.1.2...master
|
||||||
|
|
132
README.md
132
README.md
|
@ -1 +1,133 @@
|
||||||
# FrostFS S3 Lifecycler
|
# FrostFS S3 Lifecycler
|
||||||
|
|
||||||
|
The purpose of this service is to provide lifecycle management of objects
|
||||||
|
(https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html).
|
||||||
|
|
||||||
|
This service works with objects and lifecycle configurations uploaded to FrostFS mainly by using
|
||||||
|
[s3-gw](https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw).
|
||||||
|
|
||||||
|
|
||||||
|
## Build and run
|
||||||
|
|
||||||
|
To build service use the following command:
|
||||||
|
|
||||||
|
```shell
|
||||||
|
$ make
|
||||||
|
```
|
||||||
|
|
||||||
|
To run service use the following command:
|
||||||
|
|
||||||
|
```shell
|
||||||
|
$ ./bin/frostfs-s3-lifecycler --config config.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
Minimal config example to run with [dev-evn](https://git.frostfs.info/TrueCloudLab/frostfs-dev-env) can be:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
wallet:
|
||||||
|
path: ./frostfs-dev-env/wallets/wallet.json
|
||||||
|
address: NbUgTSFvPmsRxmGeWpuuGeJUoRoi6PErcM
|
||||||
|
passphrase: ""
|
||||||
|
|
||||||
|
morph:
|
||||||
|
rpc_endpoint:
|
||||||
|
- address: ws://morph-chain.frostfs.devenv:30333/ws
|
||||||
|
|
||||||
|
credential:
|
||||||
|
use: wallets
|
||||||
|
source:
|
||||||
|
wallets:
|
||||||
|
- path: ./frostfs-dev-env/wallets/wallet.json
|
||||||
|
address: NbUgTSFvPmsRxmGeWpuuGeJUoRoi6PErcM
|
||||||
|
passphrase: ""
|
||||||
|
|
||||||
|
frostfs:
|
||||||
|
peers:
|
||||||
|
0:
|
||||||
|
priority: 1
|
||||||
|
weight: 1
|
||||||
|
address: s01.frostfs.devenv:8080
|
||||||
|
```
|
||||||
|
|
||||||
|
More detailed description of each parameter can be found [here](docs/configuration.md).
|
||||||
|
|
||||||
|
## Description
|
||||||
|
|
||||||
|
Once running service listens new epoch notification from FrostFS network.
|
||||||
|
On each epoch service lists all users from `frostfsid`
|
||||||
|
[contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/frostfsid) and check each
|
||||||
|
user bucket lifecycle configuration to validate if objects or multipart uploads should be deleted according to this
|
||||||
|
configuration.
|
||||||
|
|
||||||
|
> **NOTE**
|
||||||
|
>
|
||||||
|
> Lifecycler handles only expiration actions (transition isn't supported).
|
||||||
|
|
||||||
|
If object should be deleted or multipart aborted lifecycler service perform delete action. It must have user credential
|
||||||
|
for this operation, so service must be provided with private key of each user. It can be done by specifying wallets
|
||||||
|
in config:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
credential:
|
||||||
|
use: wallets
|
||||||
|
source:
|
||||||
|
wallets:
|
||||||
|
- path: ./frostfs-dev-env/wallets/wallet.json
|
||||||
|
address: NbUgTSFvPmsRxmGeWpuuGeJUoRoi6PErcM
|
||||||
|
passphrase: ""
|
||||||
|
```
|
||||||
|
|
||||||
|
Currently only wallet source is supported.
|
||||||
|
|
||||||
|
### S3
|
||||||
|
|
||||||
|
To apply lifecycle configuration to bucket we must invoke
|
||||||
|
[PutBucketLifecycleConfiguration](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycleConfiguration.html)
|
||||||
|
method on [s3-gw](https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw). It can be done using
|
||||||
|
[AWS CLI](https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-lifecycle-configuration.html).
|
||||||
|
|
||||||
|
Lifecycle configuration example:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"Rules": [
|
||||||
|
{
|
||||||
|
"Status": "Enabled",
|
||||||
|
"ID": "Abort 1 day old multiparts",
|
||||||
|
"AbortIncompleteMultipartUpload": {
|
||||||
|
"DaysAfterInitiation": 1
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Status": "Enabled",
|
||||||
|
"ID": "Expire objects with prefix after specific date",
|
||||||
|
"Expiration": {
|
||||||
|
"Date": "2024-07-31T06:32:00Z"
|
||||||
|
},
|
||||||
|
"Filter": {
|
||||||
|
"Prefix": "prefix"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Status": "Enabled",
|
||||||
|
"ID": "Expire objects by tags after specific date",
|
||||||
|
"Expiration": {
|
||||||
|
"Date": "2024-07-31T06:32:00Z"
|
||||||
|
},
|
||||||
|
"Filter": {
|
||||||
|
"And": {
|
||||||
|
"Prefix": "tags",
|
||||||
|
"Tags": [{
|
||||||
|
"Key":"tag-key",
|
||||||
|
"Value":"tag-val"
|
||||||
|
}]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Details can be found in AWS documentaion
|
||||||
|
* https://docs.aws.amazon.com/AmazonS3/latest/userguide/intro-lifecycle-rules.html
|
||||||
|
* https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-configuration-examples.html
|
||||||
|
|
2
VERSION
2
VERSION
|
@ -1 +1 @@
|
||||||
v0.1.0
|
v0.1.2
|
||||||
|
|
Loading…
Reference in a new issue