No description
Find a file
Denis Kirillov eb51a33881
All checks were successful
/ DCO (pull_request) Successful in 54s
/ Builds (pull_request) Successful in 1m20s
/ Vulncheck (pull_request) Successful in 1m22s
/ Lint (pull_request) Successful in 1m45s
/ Tests (pull_request) Successful in 1m22s
Release v0.1.2
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-09-25 14:09:05 +03:00
.docker [#18] Update Go version 2024-08-29 08:36:11 +03:00
.forgejo/workflows [#18] Update Go version 2024-08-29 08:36:11 +03:00
cmd/s3-lifecycler [#20] Add metric to see number of dropped logs 2024-09-24 17:47:03 +03:00
config [#20] Add metric to see number of dropped logs 2024-09-24 17:47:03 +03:00
docs [#20] Add metric to see number of dropped logs 2024-09-24 17:47:03 +03:00
internal [#20] Add metric to see number of dropped logs 2024-09-24 17:47:03 +03:00
.dockerignore [#1] Add basic repository structure 2024-07-02 12:14:51 +03:00
.gitignore [#1] Add basic repository structure 2024-07-02 12:14:51 +03:00
.gitlint [#1] Add basic repository structure 2024-07-02 12:14:51 +03:00
.golangci.yml [#18] Update Go version 2024-08-29 08:36:11 +03:00
.pre-commit-config.yaml [#1] Add basic repository structure 2024-07-02 12:14:51 +03:00
CHANGELOG.md Release v0.1.2 2024-09-25 14:09:05 +03:00
CONTRIBUTING.md [#1] Add basic repository structure 2024-07-02 12:14:51 +03:00
CREDITS.md [#1] Add basic repository structure 2024-07-02 12:14:51 +03:00
go.mod [#18] Update Go version 2024-08-29 08:36:11 +03:00
go.sum [#20] Fix go.sum 2024-09-23 18:53:32 +03:00
help.mk [#1] Add basic repository structure 2024-07-02 12:14:51 +03:00
Makefile [#18] Update Go version 2024-08-29 08:36:11 +03:00
README.md Release v0.1.2 2024-09-25 14:09:05 +03:00
VERSION Release v0.1.2 2024-09-25 14:09:05 +03:00

FrostFS S3 Lifecycler

The purpose of this service is to provide lifecycle management of objects (https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html).

This service works with objects and lifecycle configurations uploaded to FrostFS mainly by using s3-gw.

Build and run

To build service use the following command:

$ make

To run service use the following command:

$ ./bin/frostfs-s3-lifecycler --config config.yaml

Minimal config example to run with dev-env can be:

wallet:
  path: ./frostfs-dev-env/wallets/wallet.json
  address: NbUgTSFvPmsRxmGeWpuuGeJUoRoi6PErcM
  passphrase: ""

morph:
  rpc_endpoint:
    - address: ws://morph-chain.frostfs.devenv:30333/ws

credential:
  use: wallets
  source:
    wallets:
      - path: ./frostfs-dev-env/wallets/wallet.json
        address: NbUgTSFvPmsRxmGeWpuuGeJUoRoi6PErcM
        passphrase: ""

frostfs:
  peers:
    0:
      priority: 1
      weight: 1
      address: s01.frostfs.devenv:8080

More detailed description of each parameter can be found here.

Description

Once running service listens new epoch notification from FrostFS network. On each epoch service lists all users from frostfsid contract and check each user bucket lifecycle configuration to validate if objects or multipart uploads should be deleted according to this configuration.

NOTE

Lifecycler handles only expiration actions (transition isn't supported).

If object should be deleted or multipart aborted lifecycler service perform delete action. It must have user credential for this operation, so service must be provided with private key of each user. It can be done by specifying wallets in config:

credential:
  use: wallets
  source:
    wallets:
      - path: ./frostfs-dev-env/wallets/wallet.json
        address: NbUgTSFvPmsRxmGeWpuuGeJUoRoi6PErcM
        passphrase: ""

Currently only wallet source is supported.

S3

To apply lifecycle configuration to bucket we must invoke PutBucketLifecycleConfiguration method on s3-gw. It can be done using AWS CLI.

Lifecycle configuration example:

{
    "Rules": [
        {
            "Status": "Enabled",
            "ID": "Abort 1 day old multiparts",
            "AbortIncompleteMultipartUpload": {
                "DaysAfterInitiation": 1
            }
        },
        {
            "Status": "Enabled",
            "ID": "Expire objects with prefix after specific date",
            "Expiration": {
                "Date": "2024-07-31T06:32:00Z"
            },
            "Filter": {
                "Prefix": "prefix"
            }
        },
        {
            "Status": "Enabled",
            "ID": "Expire objects by tags after specific date",
            "Expiration": {
                "Date": "2024-07-31T06:32:00Z"
            },
            "Filter": {
                "And": {
                    "Prefix": "tags",
                    "Tags": [{
                        "Key":"tag-key",
                        "Value":"tag-val"
                    }]
                }
            }
        }
    ]
}

Details can be found in AWS documentaion