TrueCloudLab/frostfs-node

Fork 27

Incorrect APE processing with object attribute condition for `DeleteObject` #1572

New issue

Open

opened 2024-12-19 07:59:58 +00:00 by dkirillov · 3 comments

dkirillov commented

2024-12-19 07:59:58 +00:00

Member

Expected Behavior

APE processing rules works the same way on any node in system

Current Behavior

APE processing rules works different depends on if node is container one or not

Possible Solution

Don't use local head here

headerObjSDK, err := c.headerProvider.GetHeader(ctx, prm.Container, *prm.Object, true)

Steps to Reproduce (for bugs)

Check netmap:

frostfs-dev-env$ frostfs-cli netmap snapshot -r s01.frostfs.devenv:8080 -w wallets/wallet.json  
Enter password > 
Epoch: 1081
Node 1: 022bb4041c50d607ff871dec7e4cd7778388e0ea6849d84ccbd9aa8f32e16a8131 ONLINE /dns4/s01.frostfs.devenv/tcp/8080 
	Continent: Europe
	Country: Russia
	CountryCode: RU
	Location: Moskva
	Price: 22
	SubDiv: Moskva
	SubDivCode: MOW
	UN-LOCODE: RU MOW
	User-Agent: FrostFS/0.44.0-rc.11-52-gc6a09195
Node 2: 02ac920cd7df0b61b289072e6b946e2da4e1a31b9ab1c621bb475e30fa4ab102c3 ONLINE /dns4/s03.frostfs.devenv/tcp/8080 
	Continent: Europe
	Country: Sweden
	CountryCode: SE
	Location: Stockholm
	Price: 11
	SubDiv: Stockholms l�n
	SubDivCode: AB
	UN-LOCODE: SE STO
	User-Agent: FrostFS/0.44.0-rc.11-52-gc6a09195
Node 3: 038c862959e56b43e20f79187c4fe9e0bc7c8c66c1603e6cf0ec7f87ab6b08dc35 ONLINE /dns4/s04.frostfs.devenv/tcp/8082/tls /dns4/s04.frostfs.devenv/tcp/8080 
	Continent: Europe
	Country: Finland
	CountryCode: FI
	Location: Helsinki (Helsingfors)
	Price: 44
	SubDiv: Uusimaa
	SubDivCode: 18
	UN-LOCODE: FI HEL
	User-Agent: FrostFS/0.44.0-rc.11-52-gc6a09195
Node 4: 03ff65b6ae79134a4dce9d0d39d3851e9bab4ee97abf86e81e1c5bbc50cd2826ae ONLINE /dns4/s02.frostfs.devenv/tcp/8080 
	Continent: Europe
	Country: Russia
	CountryCode: RU
	Location: Saint Petersburg (ex Leningrad)
	Price: 33
	SubDiv: Sankt-Peterburg
	SubDivCode: SPE
	UN-LOCODE: RU LED
	User-Agent: FrostFS/0.44.0-rc.11-52-gc6a09195

Create container:

frostfs-dev-env$ frostfs-cli container create -r s01.frostfs.devenv:8080 -w wallets/wallet.json  --policy 'REP 1 CBF 1 SELECT 1 FROM *' --await

J8FTRwivt2o7gF17jwdFbANdKTfC3Ddm7MBBFkqNMPKX

Check container nodes:

frostfs-dev-env$ frostfs-cli container nodes -r s01.frostfs.devenv:8080 -w wallets/wallet.json  --cid J8FTRwivt2o7gF17jwdFbANdKTfC3Ddm7MBBFkqNMPKX
Descriptor #1, REP 1:
	Node 1: 02ac920cd7df0b61b289072e6b946e2da4e1a31b9ab1c621bb475e30fa4ab102c3 ONLINE /dns4/s03.frostfs.devenv/tcp/8080 
		Continent: Europe
		Country: Sweden
		CountryCode: SE
		Location: Stockholm
		Price: 11
		SubDiv: Stockholms l�n
		SubDivCode: AB
		UN-LOCODE: SE STO
		User-Agent: FrostFS/0.44.0-rc.11-52-gc6a09195

Add chain for that container:

frostfs-dev-env$ cat chain.json
{
  "ID": "Zm9yLWJ1Zw==",
  "Rules": [
    {
      "Status": "AccessDenied",
      "Actions": {
        "Inverted": false,
        "Names": [
          "DeleteObject"
        ]
      },
      "Resources": {
        "Inverted": false,
        "Names": [
          "native:object//J8FTRwivt2o7gF17jwdFbANdKTfC3Ddm7MBBFkqNMPKX/*",
          "native:container//J8FTRwivt2o7gF17jwdFbANdKTfC3Ddm7MBBFkqNMPKX"
        ]
      },
      "Any": false,
      "Condition": [
        {
          "Op": "StringEquals",
          "Kind": "Request",
          "Key": "$Actor:publicKey",
          "Value": "031a6c6fbbdf02ca351745fa86b9ba5a9452d785ac4f7fc2b7548ca2a46c4fcf4a"
        },
        {
          "Op": "StringLike",
          "Kind": "Resource",
          "Key": "FilePath",
          "Value": "home/*"
        }
      ]
    }
  ],
  "MatchType": "DenyPriority"
}

frostfs-dev-env$ frostfs-adm -c frostfs-adm.yml morph ape add-rule-chain --path chain.json --chain-id for-bug --target-name J8FTRwivt2o7gF17jwdFbANdKTfC3Ddm7MBBFkqNMPKX --target-type container

Create object:

frostfs-dev-env$ frostfs-cli object put -r s01.frostfs.devenv:8080 -w wallets/wallet.json --cid J8FTRwivt2o7gF17jwdFbANdKTfC3Ddm7MBBFkqNMPKX --attributes FilePath=home/obj --file Makefile

6gvUBhnpk7xiB681MxQTXyCeb9Aeuz7Ng16Bj2E87hPr

Check object nodes:

frostfs-dev-env$ frostfs-cli object nodes -r s01.frostfs.devenv:8080 -w wallets/wallet.json  --cid J8FTRwivt2o7gF17jwdFbANdKTfC3Ddm7MBBFkqNMPKX --oid 6gvUBhnpk7xiB681MxQTXyCeb9Aeuz7Ng16Bj2E87hPr

Object 6gvUBhnpk7xiB681MxQTXyCeb9Aeuz7Ng16Bj2E87hPr stores payload in 1 data objects:
- 6gvUBhnpk7xiB681MxQTXyCeb9Aeuz7Ng16Bj2E87hPr
	Required nodes:
		- 02ac920cd7df0b61b289072e6b946e2da4e1a31b9ab1c621bb475e30fa4ab102c3
	Confirmed nodes:
		- 02ac920cd7df0b61b289072e6b946e2da4e1a31b9ab1c621bb475e30fa4ab102c3

Delete object using container node:

frostfs-dev-env$ frostfs-cli object delete -r s03.frostfs.devenv:8080 -w wallets/wallet.json  --cid J8FTRwivt2o7gF17jwdFbANdKTfC3Ddm7MBBFkqNMPKX --oid 6gvUBhnpk7xiB681MxQTXyCeb9Aeuz7Ng16Bj2E87hPr
Enter password > 
rpc error: remove object via client: status: code = 2048 message = access to object operation denied: ape denied request: status: code = 2048 message = access to object operation denied

Delete object using non-container node:

frostfs-dev-env$ frostfs-cli object delete -r s01.frostfs.devenv:8080 -w wallets/wallet.json  --cid J8FTRwivt2o7gF17jwdFbANdKTfC3Ddm7MBBFkqNMPKX --oid 6gvUBhnpk7xiB681MxQTXyCeb9Aeuz7Ng16Bj2E87hPr
Enter password > 
Object removed successfully.

Context

We cannot use bucket policies like:

{
  "Statement":[
    {
      "Action":[
        "s3:*"
      ],
      "Effect":"Allow",
      "Principal":{
        "AWS":[
          "arn:aws:iam:::user/epaas-test-user-2"
        ]
      },
      "Resource":[
        "arn:aws:s3:::epaas-test-2/home",
        "arn:aws:s3:::epaas-test-2/home/*"
      ]
    }
  ],
  "Version":"2012-10-17"
}

is s3-gw

Regression

Probably not. The same problem also for v0.42.15

Your Environment

Version used: 0.44.0-rc.11-52-gc6a09195
Server setup and configuration: devenv

## Expected Behavior APE processing rules works the same way on any node in system ## Current Behavior APE processing rules works different depends on if node is container one or not ## Possible Solution Don't use `local` head here https://git.frostfs.info/TrueCloudLab/frostfs-node/src/commit/df05057ed46632e7746fcaa26731987a9070b2e5/pkg/services/object/ape/request.go#L124 ## Steps to Reproduce (for bugs)  1. Check netmap: ``` frostfs-dev-env$ frostfs-cli netmap snapshot -r s01.frostfs.devenv:8080 -w wallets/wallet.json Enter password > Epoch: 1081 Node 1: 022bb4041c50d607ff871dec7e4cd7778388e0ea6849d84ccbd9aa8f32e16a8131 ONLINE /dns4/s01.frostfs.devenv/tcp/8080 Continent: Europe Country: Russia CountryCode: RU Location: Moskva Price: 22 SubDiv: Moskva SubDivCode: MOW UN-LOCODE: RU MOW User-Agent: FrostFS/0.44.0-rc.11-52-gc6a09195 Node 2: 02ac920cd7df0b61b289072e6b946e2da4e1a31b9ab1c621bb475e30fa4ab102c3 ONLINE /dns4/s03.frostfs.devenv/tcp/8080 Continent: Europe Country: Sweden CountryCode: SE Location: Stockholm Price: 11 SubDiv: Stockholms l�n SubDivCode: AB UN-LOCODE: SE STO User-Agent: FrostFS/0.44.0-rc.11-52-gc6a09195 Node 3: 038c862959e56b43e20f79187c4fe9e0bc7c8c66c1603e6cf0ec7f87ab6b08dc35 ONLINE /dns4/s04.frostfs.devenv/tcp/8082/tls /dns4/s04.frostfs.devenv/tcp/8080 Continent: Europe Country: Finland CountryCode: FI Location: Helsinki (Helsingfors) Price: 44 SubDiv: Uusimaa SubDivCode: 18 UN-LOCODE: FI HEL User-Agent: FrostFS/0.44.0-rc.11-52-gc6a09195 Node 4: 03ff65b6ae79134a4dce9d0d39d3851e9bab4ee97abf86e81e1c5bbc50cd2826ae ONLINE /dns4/s02.frostfs.devenv/tcp/8080 Continent: Europe Country: Russia CountryCode: RU Location: Saint Petersburg (ex Leningrad) Price: 33 SubDiv: Sankt-Peterburg SubDivCode: SPE UN-LOCODE: RU LED User-Agent: FrostFS/0.44.0-rc.11-52-gc6a09195 ``` 2. Create container: ``` frostfs-dev-env$ frostfs-cli container create -r s01.frostfs.devenv:8080 -w wallets/wallet.json --policy 'REP 1 CBF 1 SELECT 1 FROM *' --await J8FTRwivt2o7gF17jwdFbANdKTfC3Ddm7MBBFkqNMPKX ``` 3. Check container nodes: ``` frostfs-dev-env$ frostfs-cli container nodes -r s01.frostfs.devenv:8080 -w wallets/wallet.json --cid J8FTRwivt2o7gF17jwdFbANdKTfC3Ddm7MBBFkqNMPKX Descriptor #1, REP 1: Node 1: 02ac920cd7df0b61b289072e6b946e2da4e1a31b9ab1c621bb475e30fa4ab102c3 ONLINE /dns4/s03.frostfs.devenv/tcp/8080 Continent: Europe Country: Sweden CountryCode: SE Location: Stockholm Price: 11 SubDiv: Stockholms l�n SubDivCode: AB UN-LOCODE: SE STO User-Agent: FrostFS/0.44.0-rc.11-52-gc6a09195 ``` 4. Add chain for that container: ``` frostfs-dev-env$ cat chain.json { "ID": "Zm9yLWJ1Zw==", "Rules": [ { "Status": "AccessDenied", "Actions": { "Inverted": false, "Names": [ "DeleteObject" ] }, "Resources": { "Inverted": false, "Names": [ "native:object//J8FTRwivt2o7gF17jwdFbANdKTfC3Ddm7MBBFkqNMPKX/*", "native:container//J8FTRwivt2o7gF17jwdFbANdKTfC3Ddm7MBBFkqNMPKX" ] }, "Any": false, "Condition": [ { "Op": "StringEquals", "Kind": "Request", "Key": "$Actor:publicKey", "Value": "031a6c6fbbdf02ca351745fa86b9ba5a9452d785ac4f7fc2b7548ca2a46c4fcf4a" }, { "Op": "StringLike", "Kind": "Resource", "Key": "FilePath", "Value": "home/*" } ] } ], "MatchType": "DenyPriority" } frostfs-dev-env$ frostfs-adm -c frostfs-adm.yml morph ape add-rule-chain --path chain.json --chain-id for-bug --target-name J8FTRwivt2o7gF17jwdFbANdKTfC3Ddm7MBBFkqNMPKX --target-type container ``` 5. Create object: ``` frostfs-dev-env$ frostfs-cli object put -r s01.frostfs.devenv:8080 -w wallets/wallet.json --cid J8FTRwivt2o7gF17jwdFbANdKTfC3Ddm7MBBFkqNMPKX --attributes FilePath=home/obj --file Makefile 6gvUBhnpk7xiB681MxQTXyCeb9Aeuz7Ng16Bj2E87hPr ``` 6. Check object nodes: ``` frostfs-dev-env$ frostfs-cli object nodes -r s01.frostfs.devenv:8080 -w wallets/wallet.json --cid J8FTRwivt2o7gF17jwdFbANdKTfC3Ddm7MBBFkqNMPKX --oid 6gvUBhnpk7xiB681MxQTXyCeb9Aeuz7Ng16Bj2E87hPr Object 6gvUBhnpk7xiB681MxQTXyCeb9Aeuz7Ng16Bj2E87hPr stores payload in 1 data objects: - 6gvUBhnpk7xiB681MxQTXyCeb9Aeuz7Ng16Bj2E87hPr Required nodes: - 02ac920cd7df0b61b289072e6b946e2da4e1a31b9ab1c621bb475e30fa4ab102c3 Confirmed nodes: - 02ac920cd7df0b61b289072e6b946e2da4e1a31b9ab1c621bb475e30fa4ab102c3 ``` 7. Delete object using container node: ``` frostfs-dev-env$ frostfs-cli object delete -r s03.frostfs.devenv:8080 -w wallets/wallet.json --cid J8FTRwivt2o7gF17jwdFbANdKTfC3Ddm7MBBFkqNMPKX --oid 6gvUBhnpk7xiB681MxQTXyCeb9Aeuz7Ng16Bj2E87hPr Enter password > rpc error: remove object via client: status: code = 2048 message = access to object operation denied: ape denied request: status: code = 2048 message = access to object operation denied ``` 8. Delete object using non-container node: ``` frostfs-dev-env$ frostfs-cli object delete -r s01.frostfs.devenv:8080 -w wallets/wallet.json --cid J8FTRwivt2o7gF17jwdFbANdKTfC3Ddm7MBBFkqNMPKX --oid 6gvUBhnpk7xiB681MxQTXyCeb9Aeuz7Ng16Bj2E87hPr Enter password > Object removed successfully. ``` ## Context We cannot use bucket policies like: ```json { "Statement":[ { "Action":[ "s3:*" ], "Effect":"Allow", "Principal":{ "AWS":[ "arn:aws:iam:::user/epaas-test-user-2" ] }, "Resource":[ "arn:aws:s3:::epaas-test-2/home", "arn:aws:s3:::epaas-test-2/home/*" ] } ], "Version":"2012-10-17" } ``` is s3-gw ## Regression Probably not. The same problem also for `v0.42.15` ## Your Environment  * Version used: `0.44.0-rc.11-52-gc6a09195` * Server setup and configuration: devenv

dkirillov added the

bug

triage

labels 2024-12-19 07:59:58 +00:00

alexvanin commented

2024-12-19 10:20:51 +00:00

Owner

If this is going to be fixed for object.Delete operation, we should adopt this change in S3 gateway. Gateway now creates tombstones manually for multipart objects (TrueCloudLab/frostfs-s3-gw#559). So gateway should adopt these changes as well.

If this is going to be fixed for `object.Delete` operation, we should adopt this change in S3 gateway. Gateway now creates tombstones manually for multipart objects (https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/issues/559). So gateway should adopt these changes as well.

aarifullin commented

2024-12-19 11:15:06 +00:00

Member

@dkirillov correctly highlighted about local head-ing. It seems this is the only way is to force CheckAPE to fetch a full header. So, the third parameter (that's currently always true) should be able to be set by incoming value (to false)

 		headerObjSDK, err := c.headerProvider.GetHeader(ctx, prm.Container, *prm.Object, true)
 		if err == nil {
 			header = headerObjSDK.ToV2().GetHeader()
 		}

The problem is that this may cause performance degradation in a scenario when a lot of delete requests are coming to non-container nodes. So, we need to compare a load results with true/false

@dkirillov correctly highlighted about local _head_-ing. It seems this is the only way is to force `CheckAPE` to fetch a full header. So, the third parameter (that's currently always `true`) should be able to be set by incoming value (to `false`) https://git.frostfs.info/TrueCloudLab/frostfs-node/src/commit/f0c43c8d80e79b12111e8e8d1574e847f5162ac1/pkg/services/object/ape/request.go#L124-L127 The problem is that this may cause performance degradation in a scenario when a lot of delete requests are coming to non-container nodes. So, we need to compare a load results with `true`/`false`

dkirillov referenced this issue

2024-12-20 09:57:11 +00:00

Improve logs for APE checking #1574

fyrchik commented

2024-12-25 10:45:24 +00:00

Owner

The problem is not that the second parameter is true.
Setting it to false will merely reduce the probability of triggering this problem.
The problem is that we may ignore an error and violate APE processing rules.

The problem is not that the second parameter is true. Setting it to `false` will merely reduce the probability of triggering this problem. The problem is that we may ignore an error and _violate_ APE processing rules.