TrueCloudLab/frostfs-s3-gw
18
3
Fork 20
Code Issues 29 Pull requests 9 Projects Releases 23 Packages Activity Actions 3

Flaky create bucket operation (complex location constraint) #161

New issue
Closed
opened 2023-07-05 14:51:13 +00:00 by anikeev-yadro · 3 comments
anikeev-yadro commented 2023-07-05 14:51:13 +00:00
Member
Copy link

Expected Behavior

All buckets should be created.

Current Behavior

Some buckets was not created.

Steps to Reproduce (for bugs)

1.Create following location constraint sync-rep:

root@aanikeev-node1:~# cat /etc/frostfs/s3/placement-policy.json
{
    "load-1-1": "REP 1 CBF 1 SELECT 1 FROM *",
    "load-1-4": "REP 1 CBF 1 SELECT 4 FROM *",
    "load-3-3": "REP 3 CBF 1 SELECT 3 FROM *",
    "sync-rep": "REP 2 IN X1 REP 1 IN X2 CBF 1 SELECT 2 FROM MSK AS X1 SELECT 2 FROM SPB AS X2 FILTER SubDivCode EQ MOW AS MSK FILTER SubDivCode EQ SPE AS SPB"

2.Set logging level to debug:

root@aanikeev-node1:~# cat /etc/frostfs/s3/conf.d/90_logger.yml
logger:
  level: debug

3.Try to create 10 buckets with this location constraint:

Bucket: 1
+ aws --no-verify-ssl s3api create-bucket --bucket sync-rep-bucket-s1 --endpoint-url https://172.26.161.37 --object-lock-enabled-for-bucket --acl public-read-write --create-bucket-configuration LocationConstraint=sync-rep
urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.26.161.37'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings

An error occurred (InvalidLocationConstraint) when calling the CreateBucket operation (reached max retries: 0): The specified location (Region) constraint is not valid.

Bucket: 2
+ aws --no-verify-ssl s3api create-bucket --bucket sync-rep-bucket-s2 --endpoint-url https://172.26.161.37 --object-lock-enabled-for-bucket --acl public-read-write --create-bucket-configuration LocationConstraint=sync-rep
urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.26.161.37'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings

Bucket: 3
+ aws --no-verify-ssl s3api create-bucket --bucket sync-rep-bucket-s3 --endpoint-url https://172.26.161.37 --object-lock-enabled-for-bucket --acl public-read-write --create-bucket-configuration LocationConstraint=sync-rep
urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.26.161.37'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings

An error occurred (InvalidLocationConstraint) when calling the CreateBucket operation (reached max retries: 0): The specified location (Region) constraint is not valid.

Bucket: 4
+ aws --no-verify-ssl s3api create-bucket --bucket sync-rep-bucket-s4 --endpoint-url https://172.26.161.37 --object-lock-enabled-for-bucket --acl public-read-write --create-bucket-configuration LocationConstraint=sync-rep
urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.26.161.37'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings

An error occurred (InvalidLocationConstraint) when calling the CreateBucket operation (reached max retries: 0): The specified location (Region) constraint is not valid.

Bucket: 5
+ aws --no-verify-ssl s3api create-bucket --bucket sync-rep-bucket-s5 --endpoint-url https://172.26.161.37 --object-lock-enabled-for-bucket --acl public-read-write --create-bucket-configuration LocationConstraint=sync-rep
urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.26.161.37'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings

Bucket: 6
+ aws --no-verify-ssl s3api create-bucket --bucket sync-rep-bucket-s6 --endpoint-url https://172.26.161.37 --object-lock-enabled-for-bucket --acl public-read-write --create-bucket-configuration LocationConstraint=sync-rep
urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.26.161.37'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings

An error occurred (InvalidLocationConstraint) when calling the CreateBucket operation (reached max retries: 0): The specified location (Region) constraint is not valid.

Bucket: 7
+ aws --no-verify-ssl s3api create-bucket --bucket sync-rep-bucket-s7 --endpoint-url https://172.26.161.37 --object-lock-enabled-for-bucket --acl public-read-write --create-bucket-configuration LocationConstraint=sync-rep
urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.26.161.37'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings

An error occurred (InvalidLocationConstraint) when calling the CreateBucket operation (reached max retries: 0): The specified location (Region) constraint is not valid.
 
Bucket: 8
+ aws --no-verify-ssl s3api create-bucket --bucket sync-rep-bucket-s8 --endpoint-url https://172.26.161.37 --object-lock-enabled-for-bucket --acl public-read-write --create-bucket-configuration LocationConstraint=sync-rep
urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.26.161.37'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings

Bucket: 9
+ aws --no-verify-ssl s3api create-bucket --bucket sync-rep-bucket-s9 --endpoint-url https://172.26.161.37 --object-lock-enabled-for-bucket --acl public-read-write --create-bucket-configuration LocationConstraint=sync-rep
urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.26.161.37'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings

An error occurred (InvalidLocationConstraint) when calling the CreateBucket operation (reached max retries: 0): The specified location (Region) constraint is not valid.

Bucket: 10
+ aws --no-verify-ssl s3api create-bucket --bucket sync-rep-bucket-s10 --endpoint-url https://172.26.161.37 --object-lock-enabled-for-bucket --acl public-read-write --create-bucket-configuration LocationConstraint=sync-rep
urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.26.161.37'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings

An error occurred (InvalidLocationConstraint) when calling the CreateBucket operation (reached max retries: 0): The specified location (Region) constraint is not valid.
  1. As we can see some buckets was not created because `The specified location (Region) constraint is not valid

6.In the s3-gw logs I didn't see any errors

Jul 05 13:16:28 aanikeev-node1 frostfs-s3-gw[185123]: 2023-07-05T13:16:28.341Z        info        api/router.go:158  , "method": "CreateBucket", "bucket": "sync-rep-bucket-s2", "host": "172.26.161.37", "remote_host": "127.0.0.1"}
Jul 05 13:16:40 aanikeev-node1 frostfs-s3-gw[185123]: 2023-07-05T13:16:40.377Z        info        handler/put.go:750 8a01", "method": "CreateBucket", "bucket": "sync-rep-bucket-s2", "container_id": "CkoMmWazCVu4UtKsyAsxydGZo6ScyMGBLWt
Jul 05 13:16:40 aanikeev-node1 frostfs-s3-gw[185123]: 2023-07-05T13:16:40.395Z        info        api/router.go:237  "method": "CreateBucket", "bucket": "sync-rep-bucket-s2", "status": 200, "description": "OK"}
Jul 05 13:16:49 aanikeev-node1 frostfs-s3-gw[185123]: 2023-07-05T13:16:49.104Z        info        api/router.go:158  , "method": "CreateBucket", "bucket": "sync-rep-bucket-s5", "host": "172.26.161.37", "remote_host": "127.0.0.1"}
Jul 05 13:17:05 aanikeev-node1 frostfs-s3-gw[185123]: 2023-07-05T13:17:05.142Z        info        handler/put.go:750 8388", "method": "CreateBucket", "bucket": "sync-rep-bucket-s5", "container_id": "7h4Ptwr7wbU3xbHJ9SzUn8kWhWNfYfRfEBB
Jul 05 13:17:05 aanikeev-node1 frostfs-s3-gw[185123]: 2023-07-05T13:17:05.162Z        info        api/router.go:237  "method": "CreateBucket", "bucket": "sync-rep-bucket-s5", "status": 200, "description": "OK"}
Jul 05 13:17:13 aanikeev-node1 frostfs-s3-gw[185123]: 2023-07-05T13:17:13.799Z        info        api/router.go:158  , "method": "CreateBucket", "bucket": "sync-rep-bucket-s8", "host": "172.26.161.37", "remote_host": "127.0.0.1"}
Jul 05 13:17:28 aanikeev-node1 frostfs-s3-gw[185123]: 2023-07-05T13:17:28.837Z        info        handler/put.go:750 7755", "method": "CreateBucket", "bucket": "sync-rep-bucket-s8", "container_id": "HiG7GBr6VLMTJD5uF6A48zuAgxdDkef9TQw
Jul 05 13:17:28 aanikeev-node1 frostfs-s3-gw[185123]: 2023-07-05T13:17:28.855Z        info        api/router.go:237  "method": "CreateBucket", "bucket": "sync-rep-bucket-s8", "status": 200, "description": "OK"}
  1. Network map:
service@aanikeev-node2:~/aanikeev/configuration$ frostfs-cli netmap snapshot -r localhost:8080 -g                    Epoch: 501
Node 1: 0216b3a6058062ae6e0619171a92d8da3d9ba66df1ab986ea3f0d35bd5a35132a9 ONLINE /ip4/172.26.162.229/tcp/8080
        Continent: Europe
        Country: Russia
        CountryCode: RU
        ExternalAddr: /ip4/172.26.161.103/tcp/8080
        Location: Saint Petersburg (ex Leningrad)
        Node: 172.26.160.155
        Price: 10
        SubDiv: Sankt-Peterburg
        SubDivCode: SPE
        UN-LOCODE: RU LED
        role: alphabet
Node 2: 022d99879ed3e8484a2655e7bb06f433a5bac34d76da9ca6499d28401d3388a3b3 ONLINE /ip4/172.26.162.225/tcp/8080
        Continent: Europe
        Country: Russia
        CountryCode: RU
        ExternalAddr: /ip4/172.26.161.138/tcp/8080
        Location: Saint Petersburg (ex Leningrad)
        Node: 172.26.160.125
        Price: 10
        SubDiv: Sankt-Peterburg
        SubDivCode: SPE
        UN-LOCODE: RU LED
        role: alphabet
Node 3: 02735f91ae5609f6464c491337e0946155382bd7ec1fc9581de422960666297a68 ONLINE /ip4/172.26.162.137/tcp/8080
        Continent: Europe
        Country: Russia
        CountryCode: RU
        ExternalAddr: /ip4/172.26.161.37/tcp/8080
        Location: Moskva
        Node: 172.26.160.126
        Price: 10
        SubDiv: Moskva
        SubDivCode: MOW
        UN-LOCODE: RU MOW
        role: alphabet
Node 4: 0374065fe4e3eaaf0e4e77931735de2f6f68d850b06d86315a10b3c39792e5ce4a ONLINE /ip4/172.26.162.10/tcp/8080
        Continent: Europe
        Country: Russia
        CountryCode: RU
        ExternalAddr: /ip4/172.26.161.163/tcp/8080
        Location: Moskva
        Node: 172.26.160.176
        Price: 10
        SubDiv: Moskva
        SubDivCode: MOW
        UN-LOCODE: RU MOW
        role: alphabet

8.Test policy in playground:

root@aanikeev-node4:~# frostfs-cli container policy-playground --wallet /etc/frostfs/storage/wallet.json --rpc-endpoint 127.0.0.1:8080
Enter password >
> ls
         1: id=02735f91ae5609f6464c491337e0946155382bd7ec1fc9581de422960666297a68 attrs={Continent:Europe Country:Russia CountryCode:RU ExternalAddr:/ip4/172.26.161.37/tcp/8080 Location:Moskva Node:172.26.160.126 Price:10 SubDiv:Moskva SubDivCode:MOW UN-LOCODE:RU MOW role:alphabet}
         2: id=0374065fe4e3eaaf0e4e77931735de2f6f68d850b06d86315a10b3c39792e5ce4a attrs={Continent:Europe Country:Russia CountryCode:RU ExternalAddr:/ip4/172.26.161.163/tcp/8080 Location:Moskva Node:172.26.160.176 Price:10 SubDiv:Moskva SubDivCode:MOW UN-LOCODE:RU MOW role:alphabet}
         3: id=0216b3a6058062ae6e0619171a92d8da3d9ba66df1ab986ea3f0d35bd5a35132a9 attrs={Continent:Europe Country:Russia CountryCode:RU ExternalAddr:/ip4/172.26.161.103/tcp/8080 Location:Saint Petersburg (ex Leningrad) Node:172.26.160.155 Price:10 SubDiv:Sankt-Peterburg SubDivCode:SPE UN-LOCODE:RU LED role:alphabet}
         4: id=022d99879ed3e8484a2655e7bb06f433a5bac34d76da9ca6499d28401d3388a3b3 attrs={Continent:Europe Country:Russia CountryCode:RU ExternalAddr:/ip4/172.26.161.138/tcp/8080 Location:Saint Petersburg (ex Leningrad) Node:172.26.160.125 Price:10 SubDiv:Sankt-Peterburg SubDivCode:SPE UN-LOCODE:RU LED role:alphabet}

> eval REP 2 IN X1 REP 1 IN X2 CBF 1 SELECT 2 FROM MSK AS X1 SELECT 2 FROM SPB AS X2 FILTER SubDivCode EQ MOW AS MSK FILTER SubDivCode EQ SPE AS SPB
         1: [02735f91ae5609f6464c491337e0946155382bd7ec1fc9581de422960666297a68 0374065fe4e3eaaf0e4e77931735de2f6f68d850b06d86315a10b3c39792e5ce4a]
         2: [0216b3a6058062ae6e0619171a92d8da3d9ba66df1ab986ea3f0d35bd5a35132a9 022d99879ed3e8484a2655e7bb06f433a5bac34d76da9ca6499d28401d3388a3b3]

Regression

No

Version

FrostFS S3 Gateway
Version: v0.27.0-rc.1-70-g499f4c64
GoVersion: go1.19.7

Your Environment

Virtual
4 nodes

## Expected Behavior All buckets should be created. ## Current Behavior Some buckets was not created. ## Steps to Reproduce (for bugs) 1.Create following location constraint `sync-rep`: ``` root@aanikeev-node1:~# cat /etc/frostfs/s3/placement-policy.json { "load-1-1": "REP 1 CBF 1 SELECT 1 FROM *", "load-1-4": "REP 1 CBF 1 SELECT 4 FROM *", "load-3-3": "REP 3 CBF 1 SELECT 3 FROM *", "sync-rep": "REP 2 IN X1 REP 1 IN X2 CBF 1 SELECT 2 FROM MSK AS X1 SELECT 2 FROM SPB AS X2 FILTER SubDivCode EQ MOW AS MSK FILTER SubDivCode EQ SPE AS SPB" ``` 2.Set logging level to `debug`: ``` root@aanikeev-node1:~# cat /etc/frostfs/s3/conf.d/90_logger.yml logger: level: debug ``` 3.Try to create 10 buckets with this location constraint: ``` Bucket: 1 + aws --no-verify-ssl s3api create-bucket --bucket sync-rep-bucket-s1 --endpoint-url https://172.26.161.37 --object-lock-enabled-for-bucket --acl public-read-write --create-bucket-configuration LocationConstraint=sync-rep urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.26.161.37'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings An error occurred (InvalidLocationConstraint) when calling the CreateBucket operation (reached max retries: 0): The specified location (Region) constraint is not valid. Bucket: 2 + aws --no-verify-ssl s3api create-bucket --bucket sync-rep-bucket-s2 --endpoint-url https://172.26.161.37 --object-lock-enabled-for-bucket --acl public-read-write --create-bucket-configuration LocationConstraint=sync-rep urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.26.161.37'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings Bucket: 3 + aws --no-verify-ssl s3api create-bucket --bucket sync-rep-bucket-s3 --endpoint-url https://172.26.161.37 --object-lock-enabled-for-bucket --acl public-read-write --create-bucket-configuration LocationConstraint=sync-rep urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.26.161.37'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings An error occurred (InvalidLocationConstraint) when calling the CreateBucket operation (reached max retries: 0): The specified location (Region) constraint is not valid. Bucket: 4 + aws --no-verify-ssl s3api create-bucket --bucket sync-rep-bucket-s4 --endpoint-url https://172.26.161.37 --object-lock-enabled-for-bucket --acl public-read-write --create-bucket-configuration LocationConstraint=sync-rep urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.26.161.37'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings An error occurred (InvalidLocationConstraint) when calling the CreateBucket operation (reached max retries: 0): The specified location (Region) constraint is not valid. Bucket: 5 + aws --no-verify-ssl s3api create-bucket --bucket sync-rep-bucket-s5 --endpoint-url https://172.26.161.37 --object-lock-enabled-for-bucket --acl public-read-write --create-bucket-configuration LocationConstraint=sync-rep urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.26.161.37'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings Bucket: 6 + aws --no-verify-ssl s3api create-bucket --bucket sync-rep-bucket-s6 --endpoint-url https://172.26.161.37 --object-lock-enabled-for-bucket --acl public-read-write --create-bucket-configuration LocationConstraint=sync-rep urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.26.161.37'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings An error occurred (InvalidLocationConstraint) when calling the CreateBucket operation (reached max retries: 0): The specified location (Region) constraint is not valid. Bucket: 7 + aws --no-verify-ssl s3api create-bucket --bucket sync-rep-bucket-s7 --endpoint-url https://172.26.161.37 --object-lock-enabled-for-bucket --acl public-read-write --create-bucket-configuration LocationConstraint=sync-rep urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.26.161.37'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings An error occurred (InvalidLocationConstraint) when calling the CreateBucket operation (reached max retries: 0): The specified location (Region) constraint is not valid. Bucket: 8 + aws --no-verify-ssl s3api create-bucket --bucket sync-rep-bucket-s8 --endpoint-url https://172.26.161.37 --object-lock-enabled-for-bucket --acl public-read-write --create-bucket-configuration LocationConstraint=sync-rep urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.26.161.37'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings Bucket: 9 + aws --no-verify-ssl s3api create-bucket --bucket sync-rep-bucket-s9 --endpoint-url https://172.26.161.37 --object-lock-enabled-for-bucket --acl public-read-write --create-bucket-configuration LocationConstraint=sync-rep urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.26.161.37'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings An error occurred (InvalidLocationConstraint) when calling the CreateBucket operation (reached max retries: 0): The specified location (Region) constraint is not valid. Bucket: 10 + aws --no-verify-ssl s3api create-bucket --bucket sync-rep-bucket-s10 --endpoint-url https://172.26.161.37 --object-lock-enabled-for-bucket --acl public-read-write --create-bucket-configuration LocationConstraint=sync-rep urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.26.161.37'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings An error occurred (InvalidLocationConstraint) when calling the CreateBucket operation (reached max retries: 0): The specified location (Region) constraint is not valid. ``` 5. As we can see some buckets was not created because `The specified location (Region) constraint is not valid 6.In the s3-gw logs I didn't see any errors ``` Jul 05 13:16:28 aanikeev-node1 frostfs-s3-gw[185123]: 2023-07-05T13:16:28.341Z info api/router.go:158 , "method": "CreateBucket", "bucket": "sync-rep-bucket-s2", "host": "172.26.161.37", "remote_host": "127.0.0.1"} Jul 05 13:16:40 aanikeev-node1 frostfs-s3-gw[185123]: 2023-07-05T13:16:40.377Z info handler/put.go:750 8a01", "method": "CreateBucket", "bucket": "sync-rep-bucket-s2", "container_id": "CkoMmWazCVu4UtKsyAsxydGZo6ScyMGBLWt Jul 05 13:16:40 aanikeev-node1 frostfs-s3-gw[185123]: 2023-07-05T13:16:40.395Z info api/router.go:237 "method": "CreateBucket", "bucket": "sync-rep-bucket-s2", "status": 200, "description": "OK"} Jul 05 13:16:49 aanikeev-node1 frostfs-s3-gw[185123]: 2023-07-05T13:16:49.104Z info api/router.go:158 , "method": "CreateBucket", "bucket": "sync-rep-bucket-s5", "host": "172.26.161.37", "remote_host": "127.0.0.1"} Jul 05 13:17:05 aanikeev-node1 frostfs-s3-gw[185123]: 2023-07-05T13:17:05.142Z info handler/put.go:750 8388", "method": "CreateBucket", "bucket": "sync-rep-bucket-s5", "container_id": "7h4Ptwr7wbU3xbHJ9SzUn8kWhWNfYfRfEBB Jul 05 13:17:05 aanikeev-node1 frostfs-s3-gw[185123]: 2023-07-05T13:17:05.162Z info api/router.go:237 "method": "CreateBucket", "bucket": "sync-rep-bucket-s5", "status": 200, "description": "OK"} Jul 05 13:17:13 aanikeev-node1 frostfs-s3-gw[185123]: 2023-07-05T13:17:13.799Z info api/router.go:158 , "method": "CreateBucket", "bucket": "sync-rep-bucket-s8", "host": "172.26.161.37", "remote_host": "127.0.0.1"} Jul 05 13:17:28 aanikeev-node1 frostfs-s3-gw[185123]: 2023-07-05T13:17:28.837Z info handler/put.go:750 7755", "method": "CreateBucket", "bucket": "sync-rep-bucket-s8", "container_id": "HiG7GBr6VLMTJD5uF6A48zuAgxdDkef9TQw Jul 05 13:17:28 aanikeev-node1 frostfs-s3-gw[185123]: 2023-07-05T13:17:28.855Z info api/router.go:237 "method": "CreateBucket", "bucket": "sync-rep-bucket-s8", "status": 200, "description": "OK"} ``` 7. Network map: ``` service@aanikeev-node2:~/aanikeev/configuration$ frostfs-cli netmap snapshot -r localhost:8080 -g Epoch: 501 Node 1: 0216b3a6058062ae6e0619171a92d8da3d9ba66df1ab986ea3f0d35bd5a35132a9 ONLINE /ip4/172.26.162.229/tcp/8080 Continent: Europe Country: Russia CountryCode: RU ExternalAddr: /ip4/172.26.161.103/tcp/8080 Location: Saint Petersburg (ex Leningrad) Node: 172.26.160.155 Price: 10 SubDiv: Sankt-Peterburg SubDivCode: SPE UN-LOCODE: RU LED role: alphabet Node 2: 022d99879ed3e8484a2655e7bb06f433a5bac34d76da9ca6499d28401d3388a3b3 ONLINE /ip4/172.26.162.225/tcp/8080 Continent: Europe Country: Russia CountryCode: RU ExternalAddr: /ip4/172.26.161.138/tcp/8080 Location: Saint Petersburg (ex Leningrad) Node: 172.26.160.125 Price: 10 SubDiv: Sankt-Peterburg SubDivCode: SPE UN-LOCODE: RU LED role: alphabet Node 3: 02735f91ae5609f6464c491337e0946155382bd7ec1fc9581de422960666297a68 ONLINE /ip4/172.26.162.137/tcp/8080 Continent: Europe Country: Russia CountryCode: RU ExternalAddr: /ip4/172.26.161.37/tcp/8080 Location: Moskva Node: 172.26.160.126 Price: 10 SubDiv: Moskva SubDivCode: MOW UN-LOCODE: RU MOW role: alphabet Node 4: 0374065fe4e3eaaf0e4e77931735de2f6f68d850b06d86315a10b3c39792e5ce4a ONLINE /ip4/172.26.162.10/tcp/8080 Continent: Europe Country: Russia CountryCode: RU ExternalAddr: /ip4/172.26.161.163/tcp/8080 Location: Moskva Node: 172.26.160.176 Price: 10 SubDiv: Moskva SubDivCode: MOW UN-LOCODE: RU MOW role: alphabet ``` 8.Test policy in playground: ``` root@aanikeev-node4:~# frostfs-cli container policy-playground --wallet /etc/frostfs/storage/wallet.json --rpc-endpoint 127.0.0.1:8080 Enter password > > ls 1: id=02735f91ae5609f6464c491337e0946155382bd7ec1fc9581de422960666297a68 attrs={Continent:Europe Country:Russia CountryCode:RU ExternalAddr:/ip4/172.26.161.37/tcp/8080 Location:Moskva Node:172.26.160.126 Price:10 SubDiv:Moskva SubDivCode:MOW UN-LOCODE:RU MOW role:alphabet} 2: id=0374065fe4e3eaaf0e4e77931735de2f6f68d850b06d86315a10b3c39792e5ce4a attrs={Continent:Europe Country:Russia CountryCode:RU ExternalAddr:/ip4/172.26.161.163/tcp/8080 Location:Moskva Node:172.26.160.176 Price:10 SubDiv:Moskva SubDivCode:MOW UN-LOCODE:RU MOW role:alphabet} 3: id=0216b3a6058062ae6e0619171a92d8da3d9ba66df1ab986ea3f0d35bd5a35132a9 attrs={Continent:Europe Country:Russia CountryCode:RU ExternalAddr:/ip4/172.26.161.103/tcp/8080 Location:Saint Petersburg (ex Leningrad) Node:172.26.160.155 Price:10 SubDiv:Sankt-Peterburg SubDivCode:SPE UN-LOCODE:RU LED role:alphabet} 4: id=022d99879ed3e8484a2655e7bb06f433a5bac34d76da9ca6499d28401d3388a3b3 attrs={Continent:Europe Country:Russia CountryCode:RU ExternalAddr:/ip4/172.26.161.138/tcp/8080 Location:Saint Petersburg (ex Leningrad) Node:172.26.160.125 Price:10 SubDiv:Sankt-Peterburg SubDivCode:SPE UN-LOCODE:RU LED role:alphabet} > eval REP 2 IN X1 REP 1 IN X2 CBF 1 SELECT 2 FROM MSK AS X1 SELECT 2 FROM SPB AS X2 FILTER SubDivCode EQ MOW AS MSK FILTER SubDivCode EQ SPE AS SPB 1: [02735f91ae5609f6464c491337e0946155382bd7ec1fc9581de422960666297a68 0374065fe4e3eaaf0e4e77931735de2f6f68d850b06d86315a10b3c39792e5ce4a] 2: [0216b3a6058062ae6e0619171a92d8da3d9ba66df1ab986ea3f0d35bd5a35132a9 022d99879ed3e8484a2655e7bb06f433a5bac34d76da9ca6499d28401d3388a3b3] ``` ## Regression No ## Version ``` FrostFS S3 Gateway Version: v0.27.0-rc.1-70-g499f4c64 GoVersion: go1.19.7 ``` ## Your Environment Virtual 4 nodes
anikeev-yadro added the
bug
 label 2023-07-05 14:51:13 +00:00
anikeev-yadro changed title from Flaky create bucket operation () to Flaky create bucket operation (complex location constraint) 2023-07-05 14:51:38 +00:00
anikeev-yadro commented 2023-07-05 15:32:52 +00:00
Author
Member
Copy link
No description provided.
logs.tar.gz
3.1 MiB
dkirillov commented 2023-07-06 07:55:50 +00:00
Member
Copy link

It isn't s3 gateway problem. Nginx routes requests to different gates and several of them don't have sync-rep policy in their location contstraint map

It isn't s3 gateway problem. Nginx routes requests to different gates and several of them don't have `sync-rep` policy in their location contstraint map
anikeev-yadro commented 2023-07-06 10:02:21 +00:00
Author
Member
Copy link

I suggest to close.

I suggest to close.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
support/v0.31
support/v0.30
support/v0.29
support/v0.28
support/v0.27
v0.32.0
v0.31.3
v0.31.2
v0.30.9
v0.31.1
v0.31.0
v0.31.0-rc.6
v0.31.0-rc.5
v0.31.0-rc.4
v0.30.8
v0.31.0-rc.3
v0.30.7
v0.31.0-rc.2
v0.31.0-rc.1
v0.30.6
v0.30.5
v0.30.4
v0.30.3
v0.30.2
v0.30.1
v0.30.0
v0.29.3
v0.30.0-rc.5
v0.29.2
v0.30.0-rc.4
v0.30.0-rc.3
v0.30.0-rc.2
v0.29.1
v0.30.0-rc.1
v0.29.0
v0.28.2
v0.29.0-rc.11
v0.29.0-rc.10
v0.29.0-rc.9
v0.29.0-rc.8
v0.29.0-rc.7
v0.29.0-rc.6
v0.29.0-rc.5
v0.29.0-rc.4
v0.29.0-rc.3
v0.29.0-rc.2
v0.29.0-rc.1
v0.28.1
v0.28.0
v0.28.0-rc.1
v0.27.0
v0.27.0-rc.2
v0.27.0-rc.1
Labels
Clear labels   
P0

Highest

  
P1

High

  
P2

Medium

  
P3

Low

  
good first issue

Good for newcomers

  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff

  
blocked

The issue or PR is blocked by something

  
bug

Something is not working

  
config

Configuration format update or breaking change

  
discussion

Talking about something in order to reach a decision or to exchange ideas

  
documentation

Documentation related

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
go

Language features adoption

  
help wanted

Extra attention is needed

  
internal

   
invalid

Something is wrong

  
kludge

This is a kludge and we know it

  
observability

Related to metrics, tracing and logs

  
perfomance

Perfomance related

  
question

More information is needed

  
refactoring

Refactoring

  
wontfix

This won't be fixed

No labels
P0
P1
P2
P3
good first issue
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-s3-gw#161
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?