133 lines
3.7 KiB
Markdown
133 lines
3.7 KiB
Markdown
# FrostFS S3 Lifecycler
|
|
|
|
The purpose of this service is to provide lifecycle management of objects
|
|
(https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html).
|
|
|
|
This service works with objects and lifecycle configurations uploaded to FrostFS mainly by using
|
|
[s3-gw](https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw).
|
|
|
|
|
|
## Build and run
|
|
|
|
To build service use the following command:
|
|
|
|
```shell
|
|
$ make
|
|
```
|
|
|
|
To run service use the following command:
|
|
|
|
```shell
|
|
$ ./bin/frostfs-s3-lifecycler --config config.yaml
|
|
```
|
|
|
|
Minimal config example to run with [dev-env](https://git.frostfs.info/TrueCloudLab/frostfs-dev-env) can be:
|
|
|
|
```yaml
|
|
wallet:
|
|
path: ./frostfs-dev-env/wallets/wallet.json
|
|
address: NbUgTSFvPmsRxmGeWpuuGeJUoRoi6PErcM
|
|
passphrase: ""
|
|
|
|
morph:
|
|
rpc_endpoint:
|
|
- address: ws://morph-chain.frostfs.devenv:30333/ws
|
|
|
|
credential:
|
|
use: wallets
|
|
source:
|
|
wallets:
|
|
- path: ./frostfs-dev-env/wallets/wallet.json
|
|
address: NbUgTSFvPmsRxmGeWpuuGeJUoRoi6PErcM
|
|
passphrase: ""
|
|
|
|
frostfs:
|
|
peers:
|
|
0:
|
|
priority: 1
|
|
weight: 1
|
|
address: s01.frostfs.devenv:8080
|
|
```
|
|
|
|
More detailed description of each parameter can be found [here](docs/configuration.md).
|
|
|
|
## Description
|
|
|
|
Once running service listens new epoch notification from FrostFS network.
|
|
On each epoch service lists all users from `frostfsid`
|
|
[contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/frostfsid) and check each
|
|
user bucket lifecycle configuration to validate if objects or multipart uploads should be deleted according to this
|
|
configuration.
|
|
|
|
> **NOTE**
|
|
>
|
|
> Lifecycler handles only expiration actions (transition isn't supported).
|
|
|
|
If object should be deleted or multipart aborted lifecycler service perform delete action. It must have user credential
|
|
for this operation, so service must be provided with private key of each user. It can be done by specifying wallets
|
|
in config:
|
|
|
|
```yaml
|
|
credential:
|
|
use: wallets
|
|
source:
|
|
wallets:
|
|
- path: ./frostfs-dev-env/wallets/wallet.json
|
|
address: NbUgTSFvPmsRxmGeWpuuGeJUoRoi6PErcM
|
|
passphrase: ""
|
|
```
|
|
|
|
Currently only wallet source is supported.
|
|
|
|
### S3
|
|
|
|
To apply lifecycle configuration to bucket we must invoke
|
|
[PutBucketLifecycleConfiguration](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycleConfiguration.html)
|
|
method on [s3-gw](https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw). It can be done using
|
|
[AWS CLI](https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-lifecycle-configuration.html).
|
|
|
|
Lifecycle configuration example:
|
|
|
|
```json
|
|
{
|
|
"Rules": [
|
|
{
|
|
"Status": "Enabled",
|
|
"ID": "Abort 1 day old multiparts",
|
|
"AbortIncompleteMultipartUpload": {
|
|
"DaysAfterInitiation": 1
|
|
}
|
|
},
|
|
{
|
|
"Status": "Enabled",
|
|
"ID": "Expire objects with prefix after specific date",
|
|
"Expiration": {
|
|
"Date": "2024-07-31T06:32:00Z"
|
|
},
|
|
"Filter": {
|
|
"Prefix": "prefix"
|
|
}
|
|
},
|
|
{
|
|
"Status": "Enabled",
|
|
"ID": "Expire objects by tags after specific date",
|
|
"Expiration": {
|
|
"Date": "2024-07-31T06:32:00Z"
|
|
},
|
|
"Filter": {
|
|
"And": {
|
|
"Prefix": "tags",
|
|
"Tags": [{
|
|
"Key":"tag-key",
|
|
"Value":"tag-val"
|
|
}]
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
Details can be found in AWS documentaion
|
|
* https://docs.aws.amazon.com/AmazonS3/latest/userguide/intro-lifecycle-rules.html
|
|
* https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-configuration-examples.html
|