After install on all nodes epoch froze on 3st epoch #195

New issue

Closed

opened 2023-03-31 14:54:38 +00:00 by anikeev-yadro · 1 comment

anikeev-yadro commented

2023-03-31 14:54:38 +00:00

Member

After install on all nodes epoch froze on 3st epoch.

Version

NeoFS Storage node
Version: v0.35.0-55-g5cafdacb
GoVersion: go1.18.4

Your Environment

Virtual

After install on all nodes epoch froze on 3st epoch. ## Version ``` NeoFS Storage node Version: v0.35.0-55-g5cafdacb GoVersion: go1.18.4 ``` ## Your Environment Virtual

journal_buky.tar.gz

1 MiB

journal_az.tar.gz

1.1 MiB

journal_glagoli.tar.gz

1.1 MiB

journal_vedi.tar.gz

1.1 MiB

anikeev-yadro added the

triage

label 2023-03-31 14:54:38 +00:00

anikeev-yadro commented

2023-04-03 05:56:46 +00:00

Author

Member

Reproduce steps:

Create 50 containers witout --await flag in loop

root@az:~# ./create.sh
Fri Mar 31 12:38:48 UTC 2023
number is 1
container ID: 5T1MAAiJ4o4QrGhXeJyojwJE9HddnBdcqZDLucsbbTUJ
number is 2
container ID: CcMLyHozXTgyKXeQXfDBBkmJqXS6HkMAXju4afncTahT
number is 3
container ID: 5vv97qpBYpJ5xpgPH5BH93aQaY8XUuX7XYc3cxMsQjfs
number is 4
container ID: Bnk6RWyZ2fQxgS7ZQKFrbU8ZRoB1LS4KzcoStvFHbwoR
number is 5
container ID: 7yYEwiVmcA5MJPxkijEbCZ4HFHK4mPKT2i9Yf8BxaxiC
number is 6
container ID: C64Bu3AH34iJWEmSqUBmqytwL5kBoEL4PmRF3bYn4FfB
number is 7
container ID: E3FknmyXcX73T7Aqjt6BnvRQCqmne9tKzsAACyM2nHV2
number is 8
container ID: 3zSE977N81RwaNeaM9pEuJxisBgo5acw84eBFpfMQYMJ
number is 9
container ID: 4uiD9HAbg5NXJRr8BRYtRtvvuSLR8yJnM3fRArS8UtBL
number is 10
container ID: CfoWUxxUGV5idyZWEx3AL6xqWedhiotswwFBKaYb16xi
number is 11
container ID: GuXpQG9u32ZhHZB3pxgogHCe3oJAvMGS2399pHJUW4Fg
number is 12
container ID: 8VVM98uF3gye4j8itUf16p5hbueUNbsD1CkyTzMk26Gh
number is 13
container ID: 9muDsk5w2BWKM7gemJknR6FsMpSxgaQgf612H17jUVgD
number is 14
container ID: 6c2V3Sc5KJNRuRwHtwPZkhYMDot4YVvvqmFYVQXx5XEB
number is 15
container ID: 2oALuGJTjkSTZyvmzZWAK2NqCrG1xf2ifK1UBw1YBFne
number is 16
container ID: CmLec476M6gtMmnPK6SR4NzbeQcaSocN433MF71RN6Yd
number is 17
container ID: 7CVXJzUm3as2MfCwajHvRX2xVrSjEk732fuK9M3f7Qja
number is 18
container ID: 79VuBwb6HdWo3HDegS79yNXNzdVwAFZZDCsYmpuuMDES
number is 19
container ID: A8tA42C8dSEpXzSE7LL37S7iARGbjst9xpGYPiZNYmJ6
number is 20
container ID: 2jPGYF9ga3JcnxDATbBbUVUcFdGhScWGBduzFnnTYgqW
number is 21
container ID: 2vR5ofjPxfJqsnrydxh9eeZ6ZcexAcnjrWKMbntQahxC
number is 22
container ID: AncY6FxSaL8A12m4PTk7hvQpATJZykMbDmAcT6XkvhHT
number is 23
container ID: Adg8K6BP5QEbiGtxMkv7ku9JCyghhxqRP1VmaELBq6Qw
number is 24
container ID: p293LwE4f1bS9wgoq1VKYTTS5aGrBiGdh1rJcfu4iiu
number is 25
container ID: 4J9Q8BNFFnMGvDXwmrnSCTRhAbZf9qaFB8Uj3FJirFPD
number is 26
container ID: CUs6EBnbMthocMy89zn3jydwBg8YKmSPHdSgdFUaD5P
number is 27
container ID: 3M2g8EgY313zRJEj29B1CaXy3hkAsqxBrRYe8DScD8jw
number is 28
container ID: AWseL3GvoBHGqpVTVdTx6tgf8yzkJXcyZt8DsLtB6Gri
number is 29
container ID: DbgAH9pKdYn4h5GV18X8oeyGMzHtEbjDxmPEMcwX7Xwo
number is 30
container ID: BTxAibwJ45CjD7bymijFZLNSLm1UbSDoPD4jpTSH2xZB
number is 31
container ID: F41P5WFmUHhQBPKQxxSMp7dNa3HCEteb64RhLYoK93bA
number is 32
container ID: 6oRRiownZxnN4nzBCtD7YWPRqfXWTkXXmuJ4kvXk4sJs
number is 33
container ID: 5MABvyZM1KftJmg2na22SFphRWB25gXjiG8QEKVvkvN
number is 34
container ID: GdWXsPohCQtqpijEEDdR8LJo7ycVyJyhQGue3ubMHEJH
number is 35
container ID: CVwZorYfpY43zwXfKi8fwJnsrdtKzgvbeJXmWcM66JiN
number is 36
container ID: 9VQREH5cPiijV4MFSyF8ZAJUc9Yvyke7SpvyQtyr2v4K
number is 37
container ID: BqG2muizznnn39Tvyyk5p9WZ8ErKmVn8VZuYqyUmhW9S
number is 38
container ID: 6TcabPDnLnYmeYhDAbgMPeYJtESaA1uJ5gCtM7jbobx
number is 39
container ID: CMnoYYDmeL81n84GowGf3GLjQrctK5sQapz9j2y5aHh5
number is 40
container ID: 59XfYGENuTYkFxeHRLuV6xZr6bTMHuNopEzxMp588yWB
number is 41
container ID: GyYGF5pfbMDXWjKnCtje322dahfEnedw9TwxNREgDA9A
number is 42
container ID: 9DisPo2eXPme2afXXVm9ZRhwukL9cjfifuGggisDbSCj
number is 43
container ID: 3jBLG5ybjQ9gZWRKE7pys4DJqSDWNNoonqxAKe6oRXid
number is 44
container ID: 7yKoEMmTmLHnE1wHDFpCPuvKRTL6u6fmaeF4BpjCMoS9
number is 45
container ID: 9uqnd8vRjcNCoCHNyVcznswgrcjT2W5a1K59pU26RPvy
number is 46
container ID: CPRzkXdn5KKQSax82VyQUXE8qsZKRLfRbMdMCodRfcta
number is 47
container ID: Rs9CPL8E7p8c8HkiZHPRrkqrsVzujAB9ULNcbfdULB2
number is 48
container ID: 7QHUVDZRJpG3y4QKfVeKju6xSrRJwCVMj2p8HL4y7A4A
number is 49
container ID: Ax3yzz3RR7HwGcxkmJ4y3A67QYmJFqNovJ7bsD7oLe4p
number is 50
container ID: FpLadVJtF6rEyZLZMEe6N88JTNMyuxZGFQ9Uu6uJbwVH

and tick epochs in parallel:

service@az:~/aanikeev/configuration$ neofs-adm morph force-new-epoch -c config.yaml
Current epoch: 10, increase to 11.
Waiting for transactions to persist...
service@az:~/aanikeev/configuration$ neofs-adm morph force-new-epoch -c config.yaml
Current epoch: 14, increase to 15.
Waiting for transactions to persist...
service@az:~/aanikeev/configuration$ neofs-adm morph force-new-epoch -c config.yaml
Current epoch: 15, increase to 16.
Waiting for transactions to persist...
service@az:~/aanikeev/configuration$ neofs-adm morph force-new-epoch -c config.yaml
Current epoch: 16, increase to 17.
Waiting for transactions to persist...
service@az:~/aanikeev/configuration$ neofs-adm morph force-new-epoch -c config.yaml
Current epoch: 17, increase to 18.
Waiting for transactions to persist...
service@az:~/aanikeev/configuration$ neofs-adm morph force-new-epoch -c config.yaml
Current epoch: 18, increase to 19.
Waiting for transactions to persist...
service@az:~/aanikeev/configuration$ neofs-adm morph force-new-epoch -c config.yaml
Current epoch: 19, increase to 20.
Waiting for transactions to persist...

See errors in log:

Mar 31 12:48:01 glagoli neofs-ir[19231]: 2023-03-31T12:48:01.139Z        info        reputation/process_put.go:30        ignore reputation value        {"reason": "invalid epoch number", "trust_epoch": 19, "local_epoch": 18}

Check epoch in map:

root@buky:~# neofs-cli netmap snapshot -r localhost:8080 -g
Epoch: 20
Node 1: 027e0e94094ce9c15c2487e3a363e87b74d5af52ff3ad25577d6f599087b5ba9e4 ONLINE /ip4/172.26.162.187/tcp/8080
        Continent: Europe
        Country: Russia
        CountryCode: RU
        Deployed: YACZROKH
        ExternalAddr: /ip4/172.26.161.102/tcp/8080
        Location: Saint Petersburg (ex Leningrad)
        Node: 172.26.160.121
        Price: 10
        SubDiv: Sankt-Peterburg
        SubDivCode: SPE
        UN-LOCODE: RU LED
Node 2: 02c066da761aab395d6a1c048e8c67021fb6e7afaa9a5a419e3b3a0464137f0b09 ONLINE /ip4/172.26.162.39/tcp/8080
        Continent: Europe
        Country: Finland
        CountryCode: FI
        Deployed: YACZROKH
        ExternalAddr: /ip4/172.26.161.123/tcp/8080
        Location: Helsinki (Helsingfors)
        Node: 172.26.160.224
        Price: 10
        SubDiv: Uusimaa
        SubDivCode: 18
        UN-LOCODE: FI HEL
Node 3: 034a41d617a22d7052fb00914dcda4079d2d0c3b945ac537d0c7b843975cc3ba47 ONLINE /ip4/172.26.162.168/tcp/8080
        Continent: Europe
        Country: Sweden
        CountryCode: SE
        Deployed: YACZROKH
        ExternalAddr: /ip4/172.26.161.170/tcp/8080
        Location: Stockholm
        Node: 172.26.160.29
        Price: 10
        SubDiv: Stockholms l�n
        SubDivCode: AB
        UN-LOCODE: SE STO
Node 4: 03c9878e7e809f59bab68e8d9162b418b814f33a736478c0143677a0cf9e23de28 ONLINE /ip4/172.26.162.88/tcp/8080
        Continent: Europe
        Country: Russia
        CountryCode: RU
        Deployed: YACZROKH
        ExternalAddr: /ip4/172.26.161.250/tcp/8080
        Location: Moskva
        Node: 172.26.160.12
        Price: 10
        SubDiv: Moskva
        SubDivCode: MOW
        UN-LOCODE: RU MOW

Reproduce steps: 1. Create 50 containers witout `--await` flag in loop ``` root@az:~# ./create.sh Fri Mar 31 12:38:48 UTC 2023 number is 1 container ID: 5T1MAAiJ4o4QrGhXeJyojwJE9HddnBdcqZDLucsbbTUJ number is 2 container ID: CcMLyHozXTgyKXeQXfDBBkmJqXS6HkMAXju4afncTahT number is 3 container ID: 5vv97qpBYpJ5xpgPH5BH93aQaY8XUuX7XYc3cxMsQjfs number is 4 container ID: Bnk6RWyZ2fQxgS7ZQKFrbU8ZRoB1LS4KzcoStvFHbwoR number is 5 container ID: 7yYEwiVmcA5MJPxkijEbCZ4HFHK4mPKT2i9Yf8BxaxiC number is 6 container ID: C64Bu3AH34iJWEmSqUBmqytwL5kBoEL4PmRF3bYn4FfB number is 7 container ID: E3FknmyXcX73T7Aqjt6BnvRQCqmne9tKzsAACyM2nHV2 number is 8 container ID: 3zSE977N81RwaNeaM9pEuJxisBgo5acw84eBFpfMQYMJ number is 9 container ID: 4uiD9HAbg5NXJRr8BRYtRtvvuSLR8yJnM3fRArS8UtBL number is 10 container ID: CfoWUxxUGV5idyZWEx3AL6xqWedhiotswwFBKaYb16xi number is 11 container ID: GuXpQG9u32ZhHZB3pxgogHCe3oJAvMGS2399pHJUW4Fg number is 12 container ID: 8VVM98uF3gye4j8itUf16p5hbueUNbsD1CkyTzMk26Gh number is 13 container ID: 9muDsk5w2BWKM7gemJknR6FsMpSxgaQgf612H17jUVgD number is 14 container ID: 6c2V3Sc5KJNRuRwHtwPZkhYMDot4YVvvqmFYVQXx5XEB number is 15 container ID: 2oALuGJTjkSTZyvmzZWAK2NqCrG1xf2ifK1UBw1YBFne number is 16 container ID: CmLec476M6gtMmnPK6SR4NzbeQcaSocN433MF71RN6Yd number is 17 container ID: 7CVXJzUm3as2MfCwajHvRX2xVrSjEk732fuK9M3f7Qja number is 18 container ID: 79VuBwb6HdWo3HDegS79yNXNzdVwAFZZDCsYmpuuMDES number is 19 container ID: A8tA42C8dSEpXzSE7LL37S7iARGbjst9xpGYPiZNYmJ6 number is 20 container ID: 2jPGYF9ga3JcnxDATbBbUVUcFdGhScWGBduzFnnTYgqW number is 21 container ID: 2vR5ofjPxfJqsnrydxh9eeZ6ZcexAcnjrWKMbntQahxC number is 22 container ID: AncY6FxSaL8A12m4PTk7hvQpATJZykMbDmAcT6XkvhHT number is 23 container ID: Adg8K6BP5QEbiGtxMkv7ku9JCyghhxqRP1VmaELBq6Qw number is 24 container ID: p293LwE4f1bS9wgoq1VKYTTS5aGrBiGdh1rJcfu4iiu number is 25 container ID: 4J9Q8BNFFnMGvDXwmrnSCTRhAbZf9qaFB8Uj3FJirFPD number is 26 container ID: CUs6EBnbMthocMy89zn3jydwBg8YKmSPHdSgdFUaD5P number is 27 container ID: 3M2g8EgY313zRJEj29B1CaXy3hkAsqxBrRYe8DScD8jw number is 28 container ID: AWseL3GvoBHGqpVTVdTx6tgf8yzkJXcyZt8DsLtB6Gri number is 29 container ID: DbgAH9pKdYn4h5GV18X8oeyGMzHtEbjDxmPEMcwX7Xwo number is 30 container ID: BTxAibwJ45CjD7bymijFZLNSLm1UbSDoPD4jpTSH2xZB number is 31 container ID: F41P5WFmUHhQBPKQxxSMp7dNa3HCEteb64RhLYoK93bA number is 32 container ID: 6oRRiownZxnN4nzBCtD7YWPRqfXWTkXXmuJ4kvXk4sJs number is 33 container ID: 5MABvyZM1KftJmg2na22SFphRWB25gXjiG8QEKVvkvN number is 34 container ID: GdWXsPohCQtqpijEEDdR8LJo7ycVyJyhQGue3ubMHEJH number is 35 container ID: CVwZorYfpY43zwXfKi8fwJnsrdtKzgvbeJXmWcM66JiN number is 36 container ID: 9VQREH5cPiijV4MFSyF8ZAJUc9Yvyke7SpvyQtyr2v4K number is 37 container ID: BqG2muizznnn39Tvyyk5p9WZ8ErKmVn8VZuYqyUmhW9S number is 38 container ID: 6TcabPDnLnYmeYhDAbgMPeYJtESaA1uJ5gCtM7jbobx number is 39 container ID: CMnoYYDmeL81n84GowGf3GLjQrctK5sQapz9j2y5aHh5 number is 40 container ID: 59XfYGENuTYkFxeHRLuV6xZr6bTMHuNopEzxMp588yWB number is 41 container ID: GyYGF5pfbMDXWjKnCtje322dahfEnedw9TwxNREgDA9A number is 42 container ID: 9DisPo2eXPme2afXXVm9ZRhwukL9cjfifuGggisDbSCj number is 43 container ID: 3jBLG5ybjQ9gZWRKE7pys4DJqSDWNNoonqxAKe6oRXid number is 44 container ID: 7yKoEMmTmLHnE1wHDFpCPuvKRTL6u6fmaeF4BpjCMoS9 number is 45 container ID: 9uqnd8vRjcNCoCHNyVcznswgrcjT2W5a1K59pU26RPvy number is 46 container ID: CPRzkXdn5KKQSax82VyQUXE8qsZKRLfRbMdMCodRfcta number is 47 container ID: Rs9CPL8E7p8c8HkiZHPRrkqrsVzujAB9ULNcbfdULB2 number is 48 container ID: 7QHUVDZRJpG3y4QKfVeKju6xSrRJwCVMj2p8HL4y7A4A number is 49 container ID: Ax3yzz3RR7HwGcxkmJ4y3A67QYmJFqNovJ7bsD7oLe4p number is 50 container ID: FpLadVJtF6rEyZLZMEe6N88JTNMyuxZGFQ9Uu6uJbwVH ``` and tick epochs in parallel: ``` service@az:~/aanikeev/configuration$ neofs-adm morph force-new-epoch -c config.yaml Current epoch: 10, increase to 11. Waiting for transactions to persist... service@az:~/aanikeev/configuration$ neofs-adm morph force-new-epoch -c config.yaml Current epoch: 14, increase to 15. Waiting for transactions to persist... service@az:~/aanikeev/configuration$ neofs-adm morph force-new-epoch -c config.yaml Current epoch: 15, increase to 16. Waiting for transactions to persist... service@az:~/aanikeev/configuration$ neofs-adm morph force-new-epoch -c config.yaml Current epoch: 16, increase to 17. Waiting for transactions to persist... service@az:~/aanikeev/configuration$ neofs-adm morph force-new-epoch -c config.yaml Current epoch: 17, increase to 18. Waiting for transactions to persist... service@az:~/aanikeev/configuration$ neofs-adm morph force-new-epoch -c config.yaml Current epoch: 18, increase to 19. Waiting for transactions to persist... service@az:~/aanikeev/configuration$ neofs-adm morph force-new-epoch -c config.yaml Current epoch: 19, increase to 20. Waiting for transactions to persist... ``` 2. See errors in log: ``` Mar 31 12:48:01 glagoli neofs-ir[19231]: 2023-03-31T12:48:01.139Z info reputation/process_put.go:30 ignore reputation value {"reason": "invalid epoch number", "trust_epoch": 19, "local_epoch": 18} ``` 3. Check epoch in map: ``` root@buky:~# neofs-cli netmap snapshot -r localhost:8080 -g Epoch: 20 Node 1: 027e0e94094ce9c15c2487e3a363e87b74d5af52ff3ad25577d6f599087b5ba9e4 ONLINE /ip4/172.26.162.187/tcp/8080 Continent: Europe Country: Russia CountryCode: RU Deployed: YACZROKH ExternalAddr: /ip4/172.26.161.102/tcp/8080 Location: Saint Petersburg (ex Leningrad) Node: 172.26.160.121 Price: 10 SubDiv: Sankt-Peterburg SubDivCode: SPE UN-LOCODE: RU LED Node 2: 02c066da761aab395d6a1c048e8c67021fb6e7afaa9a5a419e3b3a0464137f0b09 ONLINE /ip4/172.26.162.39/tcp/8080 Continent: Europe Country: Finland CountryCode: FI Deployed: YACZROKH ExternalAddr: /ip4/172.26.161.123/tcp/8080 Location: Helsinki (Helsingfors) Node: 172.26.160.224 Price: 10 SubDiv: Uusimaa SubDivCode: 18 UN-LOCODE: FI HEL Node 3: 034a41d617a22d7052fb00914dcda4079d2d0c3b945ac537d0c7b843975cc3ba47 ONLINE /ip4/172.26.162.168/tcp/8080 Continent: Europe Country: Sweden CountryCode: SE Deployed: YACZROKH ExternalAddr: /ip4/172.26.161.170/tcp/8080 Location: Stockholm Node: 172.26.160.29 Price: 10 SubDiv: Stockholms l�n SubDivCode: AB UN-LOCODE: SE STO Node 4: 03c9878e7e809f59bab68e8d9162b418b814f33a736478c0143677a0cf9e23de28 ONLINE /ip4/172.26.162.88/tcp/8080 Continent: Europe Country: Russia CountryCode: RU Deployed: YACZROKH ExternalAddr: /ip4/172.26.161.250/tcp/8080 Location: Moskva Node: 172.26.160.12 Price: 10 SubDiv: Moskva SubDivCode: MOW UN-LOCODE: RU MOW ```