TrueCloudLab/frostfs-s3-gw
18
3
Fork 20
Code Issues 29 Pull requests 7 Projects Releases 23 Packages 1 Activity Actions

Directory sync commands don't set the right ACL to objects #49

New issue
Closed
opened 2023-03-08 23:38:11 +00:00 by snegurochka · 1 comment
snegurochka commented 2023-03-08 23:38:11 +00:00
Member
Copy link

Original issue: https://github.com/nspcc-dev/neofs-s3-gw/issues/685

I have tried to use the following commands with parameter "--acl public-read-write":

PS C:\temp> aws --no-verify-ssl s3 cp c:\temp\testdir\d2  s3://b-test-800 --endpoint-url http://172.26.163.38:8084 --acl public-read-write --recursive

PS C:\temp> aws --no-verify-ssl s3 sync c:\temp\testdir\d2  s3://b-test-700 --endpoint-url http://172.26.163.38:8084 --acl public-read-write

with the same result - objects ACL for AllUsers set to public-read instead of public-read-write

PS C:\Users\a.anikeev> aws --no-verify-ssl s3api get-object-acl --bucket b-test-800 --key d1f2.txt --endpoint-url http://172.26.163.38:8084
{
    "Owner": {
        "DisplayName": "NS9iuCpxq8VzpKWLSwceguV5pz2YAYtG6M",
        "ID": "NS9iuCpxq8VzpKWLSwceguV5pz2YAYtG6M"
    },
    "Grants": [
        {
            "Grantee": {
                "ID": "02efc49d370eb40238b85d8469439b5dd70dac7b1567aaf251eef311a312098b13",
                "Type": "CanonicalUser"
            },
            "Permission": "READ"
        },
        {
            "Grantee": {
                "Type": "Group",
                "URI": "http://acs.amazonaws.com/groups/global/AllUsers"
            },
            "Permission": "READ"
        }
    ]
}

Log:

Aug 26 13:16:59 az neofs-s3-gw[5874]: 2022-08-26T13:16:59.959Z        info        api/router.go:167        call method        {"status": 200, "request_id": "509a910b-007b-41ea-b906-7e81f4b82725", "method": "PutObject", "description": "OK"}
Aug 26 13:17:28 az neofs-s3-gw[5874]: 2022-08-26T13:17:28.620Z        error        handler/util.go:25        could not put bucket acl        {"request_id": "f9a5b410-62e3-488c-8d34-dd815bf54d69", "method": "PutObject", "bucket_name": "b-test-800", "object_name": "d2f1.log", "error": "save eACL via connection pool: wait eacl presence on client: context canceled"}
Aug 26 13:17:28 az neofs-s3-gw[5874]: 2022-08-26T13:17:28.620Z        error        api/router.go:158        something went wrong        {"status": 500, "request_id": "f9a5b410-62e3-488c-8d34-dd815bf54d69", "method": "PutObject", "description": "Internal Server Error"}
Aug 26 13:17:28 az neofs-s3-gw[5874]: 2022-08-26T13:17:28.635Z        error        handler/util.go:25        could not put bucket acl        {"request_id": "68f0b520-3279-4e66-a94d-811c46d8b436", "method": "PutObject", "bucket_name": "b-test-800", "object_name": "d2f3.txt", "error": "save eACL via connection pool: wait eacl presence on client: context canceled"}
Aug 26 13:17:28 az neofs-s3-gw[5874]: 2022-08-26T13:17:28.635Z        error        handler/util.go:25        could not put bucket acl        {"request_id": "db63ae59-649a-449f-9cb8-a19089f73eff", "method": "PutObject", "bucket_name": "b-test-800", "object_name": "d2f4.txt", "error": "save eACL via connection pool: wait eacl presence on client: context canceled"}
Aug 26 13:17:28 az neofs-s3-gw[5874]: 2022-08-26T13:17:28.635Z        error        api/router.go:158        something went wrong        {"status": 500, "request_id": "db63ae59-649a-449f-9cb8-a19089f73eff", "method": "PutObject", "description": "Internal Server Error"}
Aug 26 13:17:28 az neofs-s3-gw[5874]: 2022-08-26T13:17:28.635Z        error        api/router.go:158        something went wrong        {"status": 500, "request_id": "68f0b520-3279-4e66-a94d-811c46d8b436", "method": "PutObject", "description": "Internal Server Error"}
Aug 26 13:17:28 az neofs-s3-gw[5874]: 2022-08-26T13:17:28.732Z        error        handler/util.go:25        could not put bucket acl        {"request_id": "2511a93a-79bb-4b62-b676-e9038c38864f", "method": "PutObject", "bucket_name": "b-test-800", "object_name": "test.txt", "error": "save eACL via connection pool: wait eacl presence on client: context canceled"}
Aug 26 13:17:28 az neofs-s3-gw[5874]: 2022-08-26T13:17:28.732Z        error        api/router.go:158        something went wrong        {"status": 500, "request_id": "2511a93a-79bb-4b62-b676-e9038c38864f", "method": "PutObject", "description": "Internal Server Error"}
Aug 26 13:17:30 az neofs-s3-gw[5874]: 2022-08-26T13:17:30.090Z        info        api/router.go:167        call method        {"status": 200, "request_id": "162fefe1-f8ff-4cba-8315-527cd86b7bf0", "method": "PutObject", "description": "OK"}
Aug 26 13:18:00 az neofs-s3-gw[5874]: 2022-08-26T13:18:00.929Z        info        api/router.go:167        call method        {"status": 200, "request_id": "d2d52c15-cab0-4f93-907e-0d3f5a0cfc74", "method": "PutObject", "description": "OK"}
Aug 26 13:18:29 az neofs-s3-gw[5874]: 2022-08-26T13:18:29.399Z        error        handler/util.go:25        could not put bucket acl        {"request_id": "acc557f3-25a2-4a35-a2e6-a379e0ee0b64", "method": "PutObject", "bucket_name": "b-test-800", "object_name": "d2f4.txt", "error": "save eACL via connection pool: wait eacl presence on client: context canceled"}
Aug 26 13:18:29 az neofs-s3-gw[5874]: 2022-08-26T13:18:29.399Z        error        api/router.go:158        something went wrong        {"status": 500, "request_id": "acc557f3-25a2-4a35-a2e6-a379e0ee0b64", "method": "PutObject", "description": "Internal Server Error"}
Aug 26 13:18:29 az neofs-s3-gw[5874]: 2022-08-26T13:18:29.414Z        error        handler/util.go:25        could not put bucket acl        {"request_id": "66f42737-3525-496b-a422-3d36ecb7d6a4", "method": "PutObject", "bucket_name": "b-test-800", "object_name": "d2f3.txt", "error": "save eACL via connection pool: wait eacl presence on client: context canceled"}
Aug 26 13:18:29 az neofs-s3-gw[5874]: 2022-08-26T13:18:29.414Z        error        api/router.go:158        something went wrong        {"status": 500, "request_id": "66f42737-3525-496b-a422-3d36ecb7d6a4", "method": "PutObject", "description": "Internal Server Error"}
Aug 26 13:19:02 az neofs-s3-gw[5874]: 2022-08-26T13:19:02.702Z        info        api/router.go:167        call method        {"status": 200, "request_id": "429a71d2-8c14-4330-b604-660626fe0b7a", "method": "PutObject", "description": "OK"}
Aug 26 13:19:31 az neofs-s3-gw[5874]: 2022-08-26T13:19:31.034Z        error        handler/util.go:25        could not put bucket acl        {"request_id": "5cdb2ea3-f6c3-4728-805b-62743930a175", "method": "PutObject", "bucket_name": "b-test-800", "object_name": "d2f3.txt", "error": "save eACL via connection pool: wait eacl presence on client: context canceled"}
Aug 26 13:19:31 az neofs-s3-gw[5874]: 2022-08-26T13:19:31.034Z        error        api/router.go:158        something went wrong        {"status": 500, "request_id": "5cdb2ea3-f6c3-4728-805b-62743930a175", "method": "PutObject", "description": "Internal Server Error"}

Product versions:

s3 gateway
Version: v0.23.0-36-g3b343d1-dirty
GoVersion: go1.18.4

NeoFS Storage node
Version: v0.31.0
GoVersion: go1.18.4

NeoGo
Version: 0.99.1
GoVersion: go1.18.4

PS C:\Users\a.anikeev> aws --version
aws-cli/2.7.21 Python/3.9.11 Windows/10 exe/AMD64 prompt/off

s3 gateway config:

default_policy: REP 1 IN X CBF 1 SELECT 1 FROM * AS X
listen_address: 0.0.0.0:8084
logger:
  level: debug
max_clients_count: 600
max_clients_deadline: 60s
peers:
  '0':
    address: node1.neofs:8080
    priority: '1'
    weight: '1'
  '1':
    address: node2.neofs:8080
    priority: '2'
    weight: '0.25'
  '2':
    address: node3.neofs:8080
    priority: '2'
    weight: '0.25'
  '3':
    address: node4.neofs:8080
    priority: '2'
    weight: '0.25'
pool_error_threshold: 100
pprof:
  address: localhost:8085
  enabled: true
prometheus:
  address: localhost:8086
  enabled: true
resolve_order:
- nns
rpc_endpoint: http://node1.neofs:40332
tree:
  service: 172.26.163.38:8080
wallet:
  address: ''
  passphrase: ''
  path: /etc/neofs/s3/wallet.json
Original issue: https://github.com/nspcc-dev/neofs-s3-gw/issues/685 I have tried to use the following commands with parameter "--acl public-read-write": ``` PS C:\temp> aws --no-verify-ssl s3 cp c:\temp\testdir\d2 s3://b-test-800 --endpoint-url http://172.26.163.38:8084 --acl public-read-write --recursive ``` ``` PS C:\temp> aws --no-verify-ssl s3 sync c:\temp\testdir\d2 s3://b-test-700 --endpoint-url http://172.26.163.38:8084 --acl public-read-write ``` with the same result - objects ACL for AllUsers set to public-read instead of public-read-write ``` PS C:\Users\a.anikeev> aws --no-verify-ssl s3api get-object-acl --bucket b-test-800 --key d1f2.txt --endpoint-url http://172.26.163.38:8084 { "Owner": { "DisplayName": "NS9iuCpxq8VzpKWLSwceguV5pz2YAYtG6M", "ID": "NS9iuCpxq8VzpKWLSwceguV5pz2YAYtG6M" }, "Grants": [ { "Grantee": { "ID": "02efc49d370eb40238b85d8469439b5dd70dac7b1567aaf251eef311a312098b13", "Type": "CanonicalUser" }, "Permission": "READ" }, { "Grantee": { "Type": "Group", "URI": "http://acs.amazonaws.com/groups/global/AllUsers" }, "Permission": "READ" } ] } ``` **Log:** ``` Aug 26 13:16:59 az neofs-s3-gw[5874]: 2022-08-26T13:16:59.959Z info api/router.go:167 call method {"status": 200, "request_id": "509a910b-007b-41ea-b906-7e81f4b82725", "method": "PutObject", "description": "OK"} Aug 26 13:17:28 az neofs-s3-gw[5874]: 2022-08-26T13:17:28.620Z error handler/util.go:25 could not put bucket acl {"request_id": "f9a5b410-62e3-488c-8d34-dd815bf54d69", "method": "PutObject", "bucket_name": "b-test-800", "object_name": "d2f1.log", "error": "save eACL via connection pool: wait eacl presence on client: context canceled"} Aug 26 13:17:28 az neofs-s3-gw[5874]: 2022-08-26T13:17:28.620Z error api/router.go:158 something went wrong {"status": 500, "request_id": "f9a5b410-62e3-488c-8d34-dd815bf54d69", "method": "PutObject", "description": "Internal Server Error"} Aug 26 13:17:28 az neofs-s3-gw[5874]: 2022-08-26T13:17:28.635Z error handler/util.go:25 could not put bucket acl {"request_id": "68f0b520-3279-4e66-a94d-811c46d8b436", "method": "PutObject", "bucket_name": "b-test-800", "object_name": "d2f3.txt", "error": "save eACL via connection pool: wait eacl presence on client: context canceled"} Aug 26 13:17:28 az neofs-s3-gw[5874]: 2022-08-26T13:17:28.635Z error handler/util.go:25 could not put bucket acl {"request_id": "db63ae59-649a-449f-9cb8-a19089f73eff", "method": "PutObject", "bucket_name": "b-test-800", "object_name": "d2f4.txt", "error": "save eACL via connection pool: wait eacl presence on client: context canceled"} Aug 26 13:17:28 az neofs-s3-gw[5874]: 2022-08-26T13:17:28.635Z error api/router.go:158 something went wrong {"status": 500, "request_id": "db63ae59-649a-449f-9cb8-a19089f73eff", "method": "PutObject", "description": "Internal Server Error"} Aug 26 13:17:28 az neofs-s3-gw[5874]: 2022-08-26T13:17:28.635Z error api/router.go:158 something went wrong {"status": 500, "request_id": "68f0b520-3279-4e66-a94d-811c46d8b436", "method": "PutObject", "description": "Internal Server Error"} Aug 26 13:17:28 az neofs-s3-gw[5874]: 2022-08-26T13:17:28.732Z error handler/util.go:25 could not put bucket acl {"request_id": "2511a93a-79bb-4b62-b676-e9038c38864f", "method": "PutObject", "bucket_name": "b-test-800", "object_name": "test.txt", "error": "save eACL via connection pool: wait eacl presence on client: context canceled"} Aug 26 13:17:28 az neofs-s3-gw[5874]: 2022-08-26T13:17:28.732Z error api/router.go:158 something went wrong {"status": 500, "request_id": "2511a93a-79bb-4b62-b676-e9038c38864f", "method": "PutObject", "description": "Internal Server Error"} Aug 26 13:17:30 az neofs-s3-gw[5874]: 2022-08-26T13:17:30.090Z info api/router.go:167 call method {"status": 200, "request_id": "162fefe1-f8ff-4cba-8315-527cd86b7bf0", "method": "PutObject", "description": "OK"} Aug 26 13:18:00 az neofs-s3-gw[5874]: 2022-08-26T13:18:00.929Z info api/router.go:167 call method {"status": 200, "request_id": "d2d52c15-cab0-4f93-907e-0d3f5a0cfc74", "method": "PutObject", "description": "OK"} Aug 26 13:18:29 az neofs-s3-gw[5874]: 2022-08-26T13:18:29.399Z error handler/util.go:25 could not put bucket acl {"request_id": "acc557f3-25a2-4a35-a2e6-a379e0ee0b64", "method": "PutObject", "bucket_name": "b-test-800", "object_name": "d2f4.txt", "error": "save eACL via connection pool: wait eacl presence on client: context canceled"} Aug 26 13:18:29 az neofs-s3-gw[5874]: 2022-08-26T13:18:29.399Z error api/router.go:158 something went wrong {"status": 500, "request_id": "acc557f3-25a2-4a35-a2e6-a379e0ee0b64", "method": "PutObject", "description": "Internal Server Error"} Aug 26 13:18:29 az neofs-s3-gw[5874]: 2022-08-26T13:18:29.414Z error handler/util.go:25 could not put bucket acl {"request_id": "66f42737-3525-496b-a422-3d36ecb7d6a4", "method": "PutObject", "bucket_name": "b-test-800", "object_name": "d2f3.txt", "error": "save eACL via connection pool: wait eacl presence on client: context canceled"} Aug 26 13:18:29 az neofs-s3-gw[5874]: 2022-08-26T13:18:29.414Z error api/router.go:158 something went wrong {"status": 500, "request_id": "66f42737-3525-496b-a422-3d36ecb7d6a4", "method": "PutObject", "description": "Internal Server Error"} Aug 26 13:19:02 az neofs-s3-gw[5874]: 2022-08-26T13:19:02.702Z info api/router.go:167 call method {"status": 200, "request_id": "429a71d2-8c14-4330-b604-660626fe0b7a", "method": "PutObject", "description": "OK"} Aug 26 13:19:31 az neofs-s3-gw[5874]: 2022-08-26T13:19:31.034Z error handler/util.go:25 could not put bucket acl {"request_id": "5cdb2ea3-f6c3-4728-805b-62743930a175", "method": "PutObject", "bucket_name": "b-test-800", "object_name": "d2f3.txt", "error": "save eACL via connection pool: wait eacl presence on client: context canceled"} Aug 26 13:19:31 az neofs-s3-gw[5874]: 2022-08-26T13:19:31.034Z error api/router.go:158 something went wrong {"status": 500, "request_id": "5cdb2ea3-f6c3-4728-805b-62743930a175", "method": "PutObject", "description": "Internal Server Error"} ``` **Product versions:** ``` s3 gateway Version: v0.23.0-36-g3b343d1-dirty GoVersion: go1.18.4 NeoFS Storage node Version: v0.31.0 GoVersion: go1.18.4 NeoGo Version: 0.99.1 GoVersion: go1.18.4 PS C:\Users\a.anikeev> aws --version aws-cli/2.7.21 Python/3.9.11 Windows/10 exe/AMD64 prompt/off ``` **s3 gateway config:** ``` default_policy: REP 1 IN X CBF 1 SELECT 1 FROM * AS X listen_address: 0.0.0.0:8084 logger: level: debug max_clients_count: 600 max_clients_deadline: 60s peers: '0': address: node1.neofs:8080 priority: '1' weight: '1' '1': address: node2.neofs:8080 priority: '2' weight: '0.25' '2': address: node3.neofs:8080 priority: '2' weight: '0.25' '3': address: node4.neofs:8080 priority: '2' weight: '0.25' pool_error_threshold: 100 pprof: address: localhost:8085 enabled: true prometheus: address: localhost:8086 enabled: true resolve_order: - nns rpc_endpoint: http://node1.neofs:40332 tree: service: 172.26.163.38:8080 wallet: address: '' passphrase: '' path: /etc/neofs/s3/wallet.json ```
dkirillov commented 2024-05-24 06:59:41 +00:00
Member
Copy link

We will not support object acl at all after #372, so this sync command will return AccessControlListNotSupported (the same as AWS does in case of ACL disabled). If we use --acl private everything works fine.

We will not support object acl at all after #372, so this `sync` command will return `AccessControlListNotSupported` (the same as AWS does in case of ACL disabled). If we use `--acl private` everything works fine.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
support/v0.31
support/v0.30
support/v0.29
support/v0.28
support/v0.27
v0.33.0-pre
v0.32.0
v0.31.3
v0.31.2
v0.30.9
v0.31.1
v0.31.0
v0.31.0-rc.6
v0.31.0-rc.5
v0.31.0-rc.4
v0.30.8
v0.31.0-rc.3
v0.30.7
v0.31.0-rc.2
v0.31.0-rc.1
v0.30.6
v0.30.5
v0.30.4
v0.30.3
v0.30.2
v0.30.1
v0.30.0
v0.29.3
v0.30.0-rc.5
v0.29.2
v0.30.0-rc.4
v0.30.0-rc.3
v0.30.0-rc.2
v0.29.1
v0.30.0-rc.1
v0.29.0
v0.28.2
v0.29.0-rc.11
v0.29.0-rc.10
v0.29.0-rc.9
v0.29.0-rc.8
v0.29.0-rc.7
v0.29.0-rc.6
v0.29.0-rc.5
v0.29.0-rc.4
v0.29.0-rc.3
v0.29.0-rc.2
v0.29.0-rc.1
v0.28.1
v0.28.0
v0.28.0-rc.1
v0.27.0
v0.27.0-rc.2
v0.27.0-rc.1
Labels
Clear labels   
P0

Highest

  
P1

High

  
P2

Medium

  
P3

Low

  
good first issue

Good for newcomers

  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff

  
blocked

The issue or PR is blocked by something

  
bug

Something is not working

  
config

Configuration format update or breaking change

  
discussion

Talking about something in order to reach a decision or to exchange ideas

  
documentation

Documentation related

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
go

Language features adoption

  
help wanted

Extra attention is needed

  
internal

   
invalid

Something is wrong

  
kludge

This is a kludge and we know it

  
observability

Related to metrics, tracing and logs

  
perfomance

Perfomance related

  
question

More information is needed

  
refactoring

Refactoring

  
wontfix

This won't be fixed

No labels
P0
P1
P2
P3
good first issue
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-s3-gw#49
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?