Release v0.1.2 #24
3 changed files with 157 additions and 1 deletions
24
CHANGELOG.md
24
CHANGELOG.md
|
@ -3,3 +3,27 @@
|
|||
This document outlines major changes between releases.
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [0.1.2] - 2024-09-25
|
||||
|
||||
### Added
|
||||
- More debug logs (#20)
|
||||
|
||||
## [0.1.1] - 2024-09-17
|
||||
|
||||
### Added
|
||||
- Allow to configure several sources (#16)
|
||||
|
||||
### Fixed
|
||||
- Don't create redundant delete markers (#17)
|
||||
- Don't user btoken for separate lifecycle container (#19)
|
||||
|
||||
## [0.1.0] - 2024-07-26
|
||||
|
||||
### Added
|
||||
- Basic lifecycle logic (#1, #2, #3, #4)
|
||||
|
||||
[0.1.0]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-lifecycler/compare/27189a38bb...v0.1.0
|
||||
[0.1.1]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-lifecycler/compare/v0.1.0...v0.1.1
|
||||
[0.1.2]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-lifecycler/compare/v0.1.1...v0.1.2
|
||||
[Unreleased]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-lifecycler/compare/v0.1.2...master
|
||||
|
|
132
README.md
132
README.md
|
@ -1 +1,133 @@
|
|||
# FrostFS S3 Lifecycler
|
||||
|
||||
The purpose of this service is to provide lifecycle management of objects
|
||||
(https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html).
|
||||
|
||||
This service works with objects and lifecycle configurations uploaded to FrostFS mainly by using
|
||||
[s3-gw](https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw).
|
||||
|
||||
|
||||
## Build and run
|
||||
|
||||
To build service use the following command:
|
||||
|
||||
```shell
|
||||
$ make
|
||||
```
|
||||
|
||||
To run service use the following command:
|
||||
|
||||
```shell
|
||||
$ ./bin/frostfs-s3-lifecycler --config config.yaml
|
||||
```
|
||||
|
||||
Minimal config example to run with [dev-env](https://git.frostfs.info/TrueCloudLab/frostfs-dev-env) can be:
|
||||
|
||||
```yaml
|
||||
wallet:
|
||||
path: ./frostfs-dev-env/wallets/wallet.json
|
||||
address: NbUgTSFvPmsRxmGeWpuuGeJUoRoi6PErcM
|
||||
passphrase: ""
|
||||
|
||||
morph:
|
||||
rpc_endpoint:
|
||||
- address: ws://morph-chain.frostfs.devenv:30333/ws
|
||||
|
||||
credential:
|
||||
use: wallets
|
||||
source:
|
||||
wallets:
|
||||
- path: ./frostfs-dev-env/wallets/wallet.json
|
||||
address: NbUgTSFvPmsRxmGeWpuuGeJUoRoi6PErcM
|
||||
passphrase: ""
|
||||
|
||||
frostfs:
|
||||
peers:
|
||||
0:
|
||||
priority: 1
|
||||
weight: 1
|
||||
address: s01.frostfs.devenv:8080
|
||||
```
|
||||
|
||||
More detailed description of each parameter can be found [here](docs/configuration.md).
|
||||
|
||||
## Description
|
||||
|
||||
Once running service listens new epoch notification from FrostFS network.
|
||||
On each epoch service lists all users from `frostfsid`
|
||||
[contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/frostfsid) and check each
|
||||
user bucket lifecycle configuration to validate if objects or multipart uploads should be deleted according to this
|
||||
configuration.
|
||||
|
||||
> **NOTE**
|
||||
>
|
||||
> Lifecycler handles only expiration actions (transition isn't supported).
|
||||
|
||||
If object should be deleted or multipart aborted lifecycler service perform delete action. It must have user credential
|
||||
for this operation, so service must be provided with private key of each user. It can be done by specifying wallets
|
||||
in config:
|
||||
|
||||
```yaml
|
||||
credential:
|
||||
use: wallets
|
||||
source:
|
||||
wallets:
|
||||
- path: ./frostfs-dev-env/wallets/wallet.json
|
||||
address: NbUgTSFvPmsRxmGeWpuuGeJUoRoi6PErcM
|
||||
passphrase: ""
|
||||
```
|
||||
|
||||
Currently only wallet source is supported.
|
||||
|
||||
### S3
|
||||
|
||||
To apply lifecycle configuration to bucket we must invoke
|
||||
[PutBucketLifecycleConfiguration](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycleConfiguration.html)
|
||||
method on [s3-gw](https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw). It can be done using
|
||||
[AWS CLI](https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-lifecycle-configuration.html).
|
||||
|
||||
Lifecycle configuration example:
|
||||
|
||||
```json
|
||||
{
|
||||
"Rules": [
|
||||
{
|
||||
"Status": "Enabled",
|
||||
"ID": "Abort 1 day old multiparts",
|
||||
"AbortIncompleteMultipartUpload": {
|
||||
"DaysAfterInitiation": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"Status": "Enabled",
|
||||
"ID": "Expire objects with prefix after specific date",
|
||||
"Expiration": {
|
||||
"Date": "2024-07-31T06:32:00Z"
|
||||
},
|
||||
"Filter": {
|
||||
"Prefix": "prefix"
|
||||
}
|
||||
},
|
||||
{
|
||||
"Status": "Enabled",
|
||||
"ID": "Expire objects by tags after specific date",
|
||||
"Expiration": {
|
||||
"Date": "2024-07-31T06:32:00Z"
|
||||
},
|
||||
"Filter": {
|
||||
"And": {
|
||||
"Prefix": "tags",
|
||||
"Tags": [{
|
||||
"Key":"tag-key",
|
||||
"Value":"tag-val"
|
||||
}]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Details can be found in AWS documentaion
|
||||
* https://docs.aws.amazon.com/AmazonS3/latest/userguide/intro-lifecycle-rules.html
|
||||
* https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-configuration-examples.html
|
||||
|
|
2
VERSION
2
VERSION
|
@ -1 +1 @@
|
|||
v0.1.0
|
||||
v0.1.2
|
||||
|
|
Loading…
Reference in a new issue