FrostFS S3 Lifecycler

The purpose of this service is to provide lifecycle management of objects (https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html).

This service works with objects and lifecycle configurations uploaded to FrostFS mainly by using s3-gw.

Build and run

To build service use the following command:

$ make

To run service use the following command:

$ ./bin/frostfs-s3-lifecycler --config config.yaml

Minimal config example to run with dev-env can be:

wallet:
  path: ./frostfs-dev-env/wallets/wallet.json
  address: NbUgTSFvPmsRxmGeWpuuGeJUoRoi6PErcM
  passphrase: ""

morph:
  rpc_endpoint:
    - address: ws://morph-chain.frostfs.devenv:30333/ws

credential:
  use: wallets
  source:
    wallets:
      - path: ./frostfs-dev-env/wallets/wallet.json
        address: NbUgTSFvPmsRxmGeWpuuGeJUoRoi6PErcM
        passphrase: ""

frostfs:
  peers:
    0:
      priority: 1
      weight: 1
      address: s01.frostfs.devenv:8080

More detailed description of each parameter can be found here.

Description

Once running service listens new epoch notification from FrostFS network. On each epoch service lists all users from frostfsid contract and check each user bucket lifecycle configuration to validate if objects or multipart uploads should be deleted according to this configuration.

NOTE

Lifecycler handles only expiration actions (transition isn't supported).

If object should be deleted or multipart aborted lifecycler service perform delete action. It must have user credential for this operation, so service must be provided with private key of each user. It can be done by specifying wallets in config:

credential:
  use: wallets
  source:
    wallets:
      - path: ./frostfs-dev-env/wallets/wallet.json
        address: NbUgTSFvPmsRxmGeWpuuGeJUoRoi6PErcM
        passphrase: ""

Currently only wallet source is supported.

S3

To apply lifecycle configuration to bucket we must invoke PutBucketLifecycleConfiguration method on s3-gw. It can be done using AWS CLI.

Lifecycle configuration example:

{
    "Rules": [
        {
            "Status": "Enabled",
            "ID": "Abort 1 day old multiparts",
            "AbortIncompleteMultipartUpload": {
                "DaysAfterInitiation": 1
            }
        },
        {
            "Status": "Enabled",
            "ID": "Expire objects with prefix after specific date",
            "Expiration": {
                "Date": "2024-07-31T06:32:00Z"
            },
            "Filter": {
                "Prefix": "prefix"
            }
        },
        {
            "Status": "Enabled",
            "ID": "Expire objects by tags after specific date",
            "Expiration": {
                "Date": "2024-07-31T06:32:00Z"
            },
            "Filter": {
                "And": {
                    "Prefix": "tags",
                    "Tags": [{
                        "Key":"tag-key",
                        "Value":"tag-val"
                    }]
                }
            }
        }
    ]
}

Details can be found in AWS documentaion

• https://docs.aws.amazon.com/AmazonS3/latest/userguide/intro-lifecycle-rules.html
• https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-configuration-examples.html