REP3 in policy "REP 1 IN SPB REP 1 IN MSK REP3" doesn't work

anikeev-yadro commented

2023-07-04 09:07:37 +00:00

Expected Behavior

We must be able to create third replica of object somewhere using following policy:

REP 1 IN SPB  
REP 1 IN MSK 
REP 3

Current Behavior

We cannot be able to create third replica of object somewhere.

Steps to Reproduce (for bugs)

1.Network map

Node 1: 0216b3a6058062ae6e0619171a92d8da3d9ba66df1ab986ea3f0d35bd5a35132a9 ONLINE /ip4/172.26.162.229/tcp/8080
        Continent: Europe
        Country: Russia
        CountryCode: RU
        ExternalAddr: /ip4/172.26.161.103/tcp/8080
        Location: Saint Petersburg (ex Leningrad)
        Node: 172.26.160.155
        Price: 10
        SubDiv: Sankt-Peterburg
        SubDivCode: SPE
        UN-LOCODE: RU LED
        role: alphabet
Node 2: 022d99879ed3e8484a2655e7bb06f433a5bac34d76da9ca6499d28401d3388a3b3 ONLINE /ip4/172.26.162.225/tcp/8080
        Continent: Europe
        Country: Russia
        CountryCode: RU
        ExternalAddr: /ip4/172.26.161.138/tcp/8080
        Location: Saint Petersburg (ex Leningrad)
        Node: 172.26.160.125
        Price: 10
        SubDiv: Sankt-Peterburg
        SubDivCode: SPE
        UN-LOCODE: RU LED
        role: alphabet
Node 3: 02735f91ae5609f6464c491337e0946155382bd7ec1fc9581de422960666297a68 ONLINE /ip4/172.26.162.137/tcp/8080
        Continent: Europe
        Country: Russia
        CountryCode: RU
        ExternalAddr: /ip4/172.26.161.37/tcp/8080
        Location: Moskva
        Node: 172.26.160.126
        Price: 10
        SubDiv: Moskva
        SubDivCode: MOW
        UN-LOCODE: RU MOW
        role: alphabet
Node 4: 0374065fe4e3eaaf0e4e77931735de2f6f68d850b06d86315a10b3c39792e5ce4a ONLINE /ip4/172.26.162.10/tcp/8080
        Continent: Europe
        Country: Russia
        CountryCode: RU
        ExternalAddr: /ip4/172.26.161.163/tcp/8080
        Location: Moskva
        Node: 172.26.160.176
        Price: 10
        SubDiv: Moskva
        SubDivCode: MOW
        UN-LOCODE: RU MOW
        role: alphabet

2.Using policy playgroung to test the following policy:

REP 1 IN SPB  
REP 1 IN MSK

> eval REP 1 IN X1 REP 1 IN X2 CBF 1 SELECT 2 FROM MSK AS X1 SELECT 2 FROM SPB AS X2 FILTER Location EQ Moskva AS MSK FILTER Location EQ 'Saint Petersburg (ex Leningrad)' AS SPB
         1: [02735f91ae5609f6464c491337e0946155382bd7ec1fc9581de422960666297a68 0374065fe4e3eaaf0e4e77931735de2f6f68d850b06d86315a10b3c39792e5ce4a]
         2: [0216b3a6058062ae6e0619171a92d8da3d9ba66df1ab986ea3f0d35bd5a35132a9 022d99879ed3e8484a2655e7bb06f433a5bac34d76da9ca6499d28401d3388a3b3]

3.Then add REP3 to the policy:

> eval REP 1 IN X1 REP 1 IN X2 REP 3 CBF 1 SELECT 2 FROM MSK AS X1 SELECT 2 FROM SPB AS X2 FILTER Location EQ Moskva AS MSK FILTER Location EQ 'Saint Petersburg (ex Leningrad)' AS SPB
         1: [02735f91ae5609f6464c491337e0946155382bd7ec1fc9581de422960666297a68 0374065fe4e3eaaf0e4e77931735de2f6f68d850b06d86315a10b3c39792e5ce4a 02735f91ae5609f6464c491337e0946155382bd7ec1fc9581de422960666297a68 0374065fe4e3eaaf0e4e77931735de2f6f68d850b06d86315a10b3c39792e5ce4a]
         2: [0216b3a6058062ae6e0619171a92d8da3d9ba66df1ab986ea3f0d35bd5a35132a9 022d99879ed3e8484a2655e7bb06f433a5bac34d76da9ca6499d28401d3388a3b3 0216b3a6058062ae6e0619171a92d8da3d9ba66df1ab986ea3f0d35bd5a35132a9 022d99879ed3e8484a2655e7bb06f433a5bac34d76da9ca6499d28401d3388a3b3]
         3: []

4.We cannot be able to create container with this policy:

anikeev@NB-1670:~/neofs/xk6-frostfs$ ./frostfs-cli --rpc-endpoint 172.26.161.37:8080 --wallet scenarios/files/wallet.json container create --name f_test_1 --policy "REP 1 IN X1 REP 1 IN X2 REP 3 CBF 1 SELECT 2 FROM MSK AS X1 SELECT 2 FROM SPB AS X2 FILTER Location EQ Moskva AS MSK FILTER Location EQ 'Saint Petersburg (ex Leningrad)' AS SPB" --await
Enter password >
the number of nodes '0' in selector is not enough for the number of replicas '3', use --force option to skip this check

5.Even we create container with this policy (with --force flag) we cannot be able to PUT object to REP3:

anikeev@NB-1670:~/neofs/xk6-frostfs$ ./frostfs-cli --rpc-endpoint 172.26.161.37:8080 -w scenarios/files/wallet.json object put --file /tmp/summary.json --cid GLK61rVo7rVd55M4uukEFqz2jcghvj7cPWauAzT6wWg --copies-number 0,1,1
Enter password >
 1079 / 1079 [===================================================================================================================================================================================================================] 100.00% 0s
rpc error: client failure: status: code = 1024 message = incomplete object PUT by placement

Regression

No

Your Environment

Virtual
4 nodes

## Expected Behavior We must be able to create third replica of object somewhere using following policy: ``` REP 1 IN SPB REP 1 IN MSK REP 3 ``` ## Current Behavior We cannot be able to create third replica of object somewhere. ## Steps to Reproduce (for bugs) 1.Network map ``` Node 1: 0216b3a6058062ae6e0619171a92d8da3d9ba66df1ab986ea3f0d35bd5a35132a9 ONLINE /ip4/172.26.162.229/tcp/8080 Continent: Europe Country: Russia CountryCode: RU ExternalAddr: /ip4/172.26.161.103/tcp/8080 Location: Saint Petersburg (ex Leningrad) Node: 172.26.160.155 Price: 10 SubDiv: Sankt-Peterburg SubDivCode: SPE UN-LOCODE: RU LED role: alphabet Node 2: 022d99879ed3e8484a2655e7bb06f433a5bac34d76da9ca6499d28401d3388a3b3 ONLINE /ip4/172.26.162.225/tcp/8080 Continent: Europe Country: Russia CountryCode: RU ExternalAddr: /ip4/172.26.161.138/tcp/8080 Location: Saint Petersburg (ex Leningrad) Node: 172.26.160.125 Price: 10 SubDiv: Sankt-Peterburg SubDivCode: SPE UN-LOCODE: RU LED role: alphabet Node 3: 02735f91ae5609f6464c491337e0946155382bd7ec1fc9581de422960666297a68 ONLINE /ip4/172.26.162.137/tcp/8080 Continent: Europe Country: Russia CountryCode: RU ExternalAddr: /ip4/172.26.161.37/tcp/8080 Location: Moskva Node: 172.26.160.126 Price: 10 SubDiv: Moskva SubDivCode: MOW UN-LOCODE: RU MOW role: alphabet Node 4: 0374065fe4e3eaaf0e4e77931735de2f6f68d850b06d86315a10b3c39792e5ce4a ONLINE /ip4/172.26.162.10/tcp/8080 Continent: Europe Country: Russia CountryCode: RU ExternalAddr: /ip4/172.26.161.163/tcp/8080 Location: Moskva Node: 172.26.160.176 Price: 10 SubDiv: Moskva SubDivCode: MOW UN-LOCODE: RU MOW role: alphabet ``` 2.Using policy playgroung to test the following policy: ``` REP 1 IN SPB REP 1 IN MSK ``` ``` > eval REP 1 IN X1 REP 1 IN X2 CBF 1 SELECT 2 FROM MSK AS X1 SELECT 2 FROM SPB AS X2 FILTER Location EQ Moskva AS MSK FILTER Location EQ 'Saint Petersburg (ex Leningrad)' AS SPB 1: [02735f91ae5609f6464c491337e0946155382bd7ec1fc9581de422960666297a68 0374065fe4e3eaaf0e4e77931735de2f6f68d850b06d86315a10b3c39792e5ce4a] 2: [0216b3a6058062ae6e0619171a92d8da3d9ba66df1ab986ea3f0d35bd5a35132a9 022d99879ed3e8484a2655e7bb06f433a5bac34d76da9ca6499d28401d3388a3b3] ``` 3.Then add REP3 to the policy: ``` > eval REP 1 IN X1 REP 1 IN X2 REP 3 CBF 1 SELECT 2 FROM MSK AS X1 SELECT 2 FROM SPB AS X2 FILTER Location EQ Moskva AS MSK FILTER Location EQ 'Saint Petersburg (ex Leningrad)' AS SPB 1: [02735f91ae5609f6464c491337e0946155382bd7ec1fc9581de422960666297a68 0374065fe4e3eaaf0e4e77931735de2f6f68d850b06d86315a10b3c39792e5ce4a 02735f91ae5609f6464c491337e0946155382bd7ec1fc9581de422960666297a68 0374065fe4e3eaaf0e4e77931735de2f6f68d850b06d86315a10b3c39792e5ce4a] 2: [0216b3a6058062ae6e0619171a92d8da3d9ba66df1ab986ea3f0d35bd5a35132a9 022d99879ed3e8484a2655e7bb06f433a5bac34d76da9ca6499d28401d3388a3b3 0216b3a6058062ae6e0619171a92d8da3d9ba66df1ab986ea3f0d35bd5a35132a9 022d99879ed3e8484a2655e7bb06f433a5bac34d76da9ca6499d28401d3388a3b3] 3: [] ``` 4.We cannot be able to create container with this policy: ``` anikeev@NB-1670:~/neofs/xk6-frostfs$ ./frostfs-cli --rpc-endpoint 172.26.161.37:8080 --wallet scenarios/files/wallet.json container create --name f_test_1 --policy "REP 1 IN X1 REP 1 IN X2 REP 3 CBF 1 SELECT 2 FROM MSK AS X1 SELECT 2 FROM SPB AS X2 FILTER Location EQ Moskva AS MSK FILTER Location EQ 'Saint Petersburg (ex Leningrad)' AS SPB" --await Enter password > the number of nodes '0' in selector is not enough for the number of replicas '3', use --force option to skip this check ``` 5.Even we create container with this policy (with `--force` flag) we cannot be able to PUT object to REP3: ``` anikeev@NB-1670:~/neofs/xk6-frostfs$ ./frostfs-cli --rpc-endpoint 172.26.161.37:8080 -w scenarios/files/wallet.json object put --file /tmp/summary.json --cid GLK61rVo7rVd55M4uukEFqz2jcghvj7cPWauAzT6wWg --copies-number 0,1,1 Enter password > 1079 / 1079 [===================================================================================================================================================================================================================] 100.00% 0s rpc error: client failure: status: code = 1024 message = incomplete object PUT by placement ``` ## Regression No ## Your Environment Virtual 4 nodes

anikeev-yadro added the

bug

label 2023-07-04 09:07:37 +00:00

fyrchik self-assigned this 2023-07-04 10:17:25 +00:00

fyrchik commented

2023-07-04 10:28:11 +00:00

Owner

At one point we had the intention to allow REP N SELECT 3 policies (note the ommitted IN .. clause for REP). This case is explicitly processed in the code.
I see 3 options:

Save compatibility. In this case we must add a special selector for some rep: REP 1 will work, but for multiple reps SELECT 1 IN * AS ALL may be required.
Partially lose compatibility: behave the old way unless we have at lest 2 REP statements with AND without selector.
Lose compatibility: REP always selects from ALL nodes.

At one point we had the intention to allow `REP N SELECT 3` policies (note the ommitted `IN ..` clause for `REP`). This case is explicitly processed in the code. I see 3 options: 1. Save compatibility. In this case we must add a special selector for some rep: `REP 1` will work, but for multiple reps `SELECT 1 IN * AS ALL` may be required. 2. Partially lose compatibility: behave the old way unless we have at lest 2 `REP` statements with AND without selector. 3. Lose compatibility: `REP` always selects from ALL nodes.

fyrchik commented

2023-07-06 10:23:43 +00:00

Owner

Stayed on the last option.