Node have put object in one shard then try to remove from another shard #93

New issue

Open

opened 2023-03-08 23:24:54 +00:00 by snegurochka · 1 comment

snegurochka commented

2023-03-08 23:24:54 +00:00

Member

Original issue: https://github.com/nspcc-dev/neofs-node/issues/2266

I saw that object was PUT to shard 8HPVZ4ETzwTP9iBT9y959A

2023-02-21T02:58:34.099Z	info	log/log.go:13	local object storage operation	{"shard_id": "8HPVZ4ETzwTP9iBT9y959A", "address": "B13hwjP8iEEhWbHrAnRmasb9KDcPXhvxNbA2umBesSMz/EMUB5U2traWn8nA2c8u3Yuwuwkm2SLqCsCsPyjKtr58x", "op": "metabase PUT"}

then node tried to remove this object from another shard FBYYAY8qX4kkEMVJqUPhWU

2023-02-21T03:04:14.258Z	info	log/log.go:13	local object storage operation	{"shard_id": "FBYYAY8qX4kkEMVJqUPhWU", "address": "B13hwjP8iEEhWbHrAnRmasb9KDcPXhvxNbA2umBesSMz/EMUB5U2traWn8nA2c8u3Yuwuwkm2SLqCsCsPyjKtr58x", "op": "metabase DELETE"}
2023-02-21T03:04:14.422Z	debug	shard/delete.go:92	can't remove object from blobStor	{"shard_id": "FBYYAY8qX4kkEMVJqUPhWU", "object_address": "B13hwjP8iEEhWbHrAnRmasb9KDcPXhvxNbA2umBesSMz/EMUB5U2traWn8nA2c8u3Yuwuwkm2SLqCsCsPyjKtr58x", "error": "status: code = 2049 message = object not found"}

And one hour later node failed to replicate this object

2023-02-21T04:02:34.653Z	debug	policer/check.go:231	shortage of object copies detected	{"component": "Object Policer", "object": "B13hwjP8iEEhWbHrAnRmasb9KDcPXhvxNbA2umBesSMz/EMUB5U2traWn8nA2c8u3Yuwuwkm2SLqCsCsPyjKtr58x", "shortage": 1}
2023-02-21T04:02:34.662Z	error	replicator/process.go:62	could not replicate object	{"component": "Object Replicator", "node": "02ac920cd7df0b61b289072e6b946e2da4e1a31b9ab1c621bb475e30fa4ab102c3", "object": "B13hwjP8iEEhWbHrAnRmasb9KDcPXhvxNbA2umBesSMz/EMUB5U2traWn8nA2c8u3Yuwuwkm2SLqCsCsPyjKtr58x", "error": "(*putsvc.RemoteSender) could not send object: (*putsvc.remoteTarget) could not put object to [/dns4/s03.neofs.devenv/tcp/8080]: write object via client: status: code = 1024 message = object has expired"}

Logs

obj-nn-test-node2.spb.yadro.com-s01-log.txt.zip
obj-nn-test-node2.spb.yadro.com-s02-log.txt.zip
obj-nn-test-node2.spb.yadro.com-s03-log.txt.zip
obj-nn-test-node2.spb.yadro.com-s04-log.txt.zip
obj-nn-test-node2.spb.yadro.com-s3_gate-log.txt.zip

Version

0.35.0-47-gdea97133

Your Environment

DEVENV

Original issue: https://github.com/nspcc-dev/neofs-node/issues/2266 I saw that object was PUT to shard `8HPVZ4ETzwTP9iBT9y959A` ``` 2023-02-21T02:58:34.099Z info log/log.go:13 local object storage operation {"shard_id": "8HPVZ4ETzwTP9iBT9y959A", "address": "B13hwjP8iEEhWbHrAnRmasb9KDcPXhvxNbA2umBesSMz/EMUB5U2traWn8nA2c8u3Yuwuwkm2SLqCsCsPyjKtr58x", "op": "metabase PUT"} ``` then node tried to remove this object from another shard `FBYYAY8qX4kkEMVJqUPhWU` ``` 2023-02-21T03:04:14.258Z info log/log.go:13 local object storage operation {"shard_id": "FBYYAY8qX4kkEMVJqUPhWU", "address": "B13hwjP8iEEhWbHrAnRmasb9KDcPXhvxNbA2umBesSMz/EMUB5U2traWn8nA2c8u3Yuwuwkm2SLqCsCsPyjKtr58x", "op": "metabase DELETE"} 2023-02-21T03:04:14.422Z debug shard/delete.go:92 can't remove object from blobStor {"shard_id": "FBYYAY8qX4kkEMVJqUPhWU", "object_address": "B13hwjP8iEEhWbHrAnRmasb9KDcPXhvxNbA2umBesSMz/EMUB5U2traWn8nA2c8u3Yuwuwkm2SLqCsCsPyjKtr58x", "error": "status: code = 2049 message = object not found"} ``` And one hour later node failed to replicate this object ``` 2023-02-21T04:02:34.653Z debug policer/check.go:231 shortage of object copies detected {"component": "Object Policer", "object": "B13hwjP8iEEhWbHrAnRmasb9KDcPXhvxNbA2umBesSMz/EMUB5U2traWn8nA2c8u3Yuwuwkm2SLqCsCsPyjKtr58x", "shortage": 1} 2023-02-21T04:02:34.662Z error replicator/process.go:62 could not replicate object {"component": "Object Replicator", "node": "02ac920cd7df0b61b289072e6b946e2da4e1a31b9ab1c621bb475e30fa4ab102c3", "object": "B13hwjP8iEEhWbHrAnRmasb9KDcPXhvxNbA2umBesSMz/EMUB5U2traWn8nA2c8u3Yuwuwkm2SLqCsCsPyjKtr58x", "error": "(*putsvc.RemoteSender) could not send object: (*putsvc.remoteTarget) could not put object to [/dns4/s03.neofs.devenv/tcp/8080]: write object via client: status: code = 1024 message = object has expired"} ``` ## Logs [obj-nn-test-node2.spb.yadro.com-s01-log.txt.zip](https://github.com/nspcc-dev/neofs-node/files/10794809/obj-nn-test-node2.spb.yadro.com-s01-log.txt.zip) [obj-nn-test-node2.spb.yadro.com-s02-log.txt.zip](https://github.com/nspcc-dev/neofs-node/files/10794812/obj-nn-test-node2.spb.yadro.com-s02-log.txt.zip) [obj-nn-test-node2.spb.yadro.com-s03-log.txt.zip](https://github.com/nspcc-dev/neofs-node/files/10794815/obj-nn-test-node2.spb.yadro.com-s03-log.txt.zip) [obj-nn-test-node2.spb.yadro.com-s04-log.txt.zip](https://github.com/nspcc-dev/neofs-node/files/10794816/obj-nn-test-node2.spb.yadro.com-s04-log.txt.zip) [obj-nn-test-node2.spb.yadro.com-s3_gate-log.txt.zip](https://github.com/nspcc-dev/neofs-node/files/10794817/obj-nn-test-node2.spb.yadro.com-s3_gate-log.txt.zip) ## Version ``` 0.35.0-47-gdea97133 ``` ## Your Environment DEVENV

fyrchik added this to the v0.38.0 milestone 2023-05-18 08:47:42 +00:00

fyrchik added the

bug

frostfs-node

labels 2023-05-18 08:47:50 +00:00

acid-ant self-assigned this 2024-03-11 18:29:06 +00:00

acid-ant commented

2024-03-13 07:28:20 +00:00

Member

Unfortunately, logs no more available. Problems with shard ID and shard mode fixed in scope of #918 and #959

Unfortunately, logs no more available. Problems with shard ID and shard mode fixed in scope of https://git.frostfs.info/TrueCloudLab/frostfs-node/pulls/918 and https://git.frostfs.info/TrueCloudLab/frostfs-node/issues/959

acid-ant closed this issue

2024-03-13 07:28:20 +00:00

acid-ant reopened this issue

2024-06-03 06:03:13 +00:00

fyrchik modified the milestone from v0.38.0 to v0.41.0

2024-06-03 06:58:55 +00:00

fyrchik modified the milestone from v0.41.0 to v0.42.0

2024-06-14 07:09:11 +00:00

fyrchik modified the milestone from v0.42.0 to v0.43.0

2024-07-23 06:34:48 +00:00

fyrchik modified the milestone from v0.43.0 to v0.44.0

2024-09-30 11:51:39 +00:00