Commit graph

327 commits

Author SHA1 Message Date
882c068410 [#1334] metabase: Store upgrade flag
This allows to check if metabase upgrade was not completed.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-30 09:06:20 +03:00
6c2146bbc1 [#1334] metabase: Add upgrade from v2 to v3
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-30 09:06:20 +03:00
76f67ea34e [#1323] metabase: Bump version
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-22 08:21:40 +00:00
7d0d781db1 [#1323] metabase: Drop user attribute index
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-22 08:21:40 +00:00
0f08a2efba [#1323] metabase: Resolve funlen linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-22 08:21:40 +00:00
7bf20c9f1f [#1323] metabase: Add expiration epoch buckets
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-22 08:21:40 +00:00
2542d4f5df [#1323] metabase: Drop payload checksum index
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-22 08:21:40 +00:00
15dae8685e [#1323] metabase: Drop ownerID index
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-22 08:21:40 +00:00
7bca428db0 [#1322] Use new protobuf marshaler
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-22 07:17:41 +00:00
cfda9003a7 [#1318] meta: Add test TestInhumeECObject
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-08-20 16:21:27 +03:00
6ff0b0996b [#1318] metrics: Fix container_size_bytes for EC
When node put chunk into EC container, `policer` may remove it as redundant.
This chunk marked as removed. When parent object removed and `gc` start iterating over chunk,
node count removing chunk twice.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-08-20 15:33:28 +03:00
8e51d7849a [#1295] getSvc: Assemble complex EC object headers without linking object
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-06 16:48:12 +03:00
3119f2fbc3 [#1257] metabase: Delete EC gc marks and split info
EC parent and split gc marks should be deleted after last EC chunk delete.
Also should delete split info for EC parent and split parent.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-19 17:14:14 +03:00
27caa8a72f [#1256] metabase: Put split parent object ID for EC chunks
It is required to save split parent ID too, not only split ID.
Otherwise inhume operation works incorrect: shard with last part may be skipped
and parent object will be available.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-17 18:08:07 +03:00
56eeb630b6 [#1217] Fix grammar mistakes and misspelling
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-07-01 19:14:25 +03:00
76cf7a051b [#1178] shard: Check metabase existence before read shard ID
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-06-17 09:59:15 +03:00
b9fcaad21f [#1168] shard: Set Disabled as default mode for components
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-06-11 15:13:38 +00:00
3f1961157e [#1163] metabase: Handle multiple splitInfos for EC
For REP updating split info is handled explicitly by a high-level PUT logic.
For EC it is trickier, because the address of an object we put is only
distantly related to a split info.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-06-06 16:26:29 +03:00
2e074d3846 [#1163] metabase: Properly save EC parent split ID
Search by SplitID should return all parts of a complex object.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-06-05 12:40:16 +03:00
806236da78 [#1121] node: Change mode of shard components
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-06-05 05:55:24 +00:00
cc2449beaf [#1158] metabase: Fix EC storage schema
Do not store EC info twice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-06-04 17:24:40 +03:00
5aacb8fc86 [#1144] metabase: Save parent attributes for ec-chunks
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-31 19:55:32 +03:00
f8e33f8e3a [#1144] metabase: Proprely choose root OID for EC-splitted objects
* If EC-parent is a part of Split itself, then save to root bucket
  its parent;
* If EC-parent is not a part of Split itself, then save to root bucket
  OID of this EC-parent.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-31 19:53:32 +03:00
f0edebea18 [#1144] metabase: Support ec parent filter for Search
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-31 19:53:32 +03:00
6130650bb6 [#1147] node: Implement Lock\Delete requests for EC object
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-05-30 08:13:04 +00:00
40b04c00ef [#1141] metabase: Fix IsUserObject method
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-20 13:24:15 +03:00
89a80e9a0f [#1141] metabase: Fix putUniqueIndexItem
* `GetECHeader` is not correct way to determine if an object's got
  EC-header: `ECHeader` must be used for that.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-20 13:24:08 +03:00
d45d086acd [#1129] policer: Add EC chunk replication
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-05-16 16:28:48 +03:00
5c582e96fd [#1136] metabase: Fix creation of ECInfoError
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-05-15 11:04:27 +00:00
112a7c690f [#1103] node: Implement Get\Head requests for EC object
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-04-24 18:15:53 +03:00
2c4b50a71e Reapply "[#972] Use require.ElementsMatch() where possible"
This reverts commit 7627d08914.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-10 12:09:34 +00:00
76398c06b0 [#1080] metabase: Add StorageID metric
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-10 10:00:08 +03:00
7b1adfed3e [#1080] metabase: Open bucket for container counter once
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-10 10:00:08 +03:00
d5194ab2a6 [#949] metabase: fix shard.UpdateID()
metabase.Open() now reports metabase mode metric. shard.UpdateID()
needs to read shard ID from metabase => needs to open metabase.
It caused reporting 'shard undefined' metrics. To avoid reporting
wrong metrics metabase.GetShardID() was added which also opens
metabase and does not report metrics.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-04-01 17:27:34 +03:00
81a0346a96 [#949] metabase: fix metabase mode metric
It used to always show CLOSED regardless of actual mode.
Now metric represents actual metabase mode of operations.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-04-01 17:27:34 +03:00
942d83611b [#874] engine: Revert Check object existance concurrently
This reverts commit f526f49995.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-01 08:42:34 +00:00
c09c701613 [#1048] metabase: Fix drop buckets during resync
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-19 14:28:31 +03:00
d433b49265 [#973] node: Resolve perfsprint linter
`fmt.Errorf can be replaced with errors.New` and `fmt.Sprintf can be replaced with string addition`

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-11 17:55:50 +03:00
66a26b7775 [#973] node: Resolve revive: unused-parameter linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-11 17:11:49 +03:00
d6534fd755 [#1016] frostfs-node: Fix gopls issues
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-01 12:13:43 +03:00
918613546f [#1008] metabase: Do not update storageID on put
There may be a race condition between put an object and
flushing the writecache:
1. Put object to the writecache
2. Writecache flushes object to the blobstore and sets blobstore's
storageID
3. Put object to the metabase, set writecache's storageID

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-28 11:01:50 +03:00
7470c383dd [#997] metabase: Drop toMoveIt bucket
It is not used.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-21 10:06:05 +03:00
adf7ebab5b [#996] metabase: Speed up bucket creation
Most of the time it exits, e.g. when it is per-container and use on each
object PUT. Bbolt implementation first tries to create bucket and then
returns it if it exists. Create operation uses cursor and thus is not
very lightweight, we can avoid it.

```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
                 │     old     │                new                 │
                 │   sec/op    │   sec/op     vs base               │
Put/parallel-8     174.4µ ± 3%   163.3µ ± 3%  -6.39% (p=0.000 n=10)
Put/sequential-8   263.3µ ± 2%   259.0µ ± 1%  -1.64% (p=0.000 n=10)
geomean            214.3µ        205.6µ       -4.05%

                 │     old      │                 new                 │
                 │     B/op     │     B/op      vs base               │
Put/parallel-8     275.3Ki ± 3%   281.1Ki ± 4%       ~ (p=0.063 n=10)
Put/sequential-8   413.0Ki ± 2%   426.6Ki ± 2%  +3.29% (p=0.003 n=10)
geomean            337.2Ki        346.3Ki       +2.70%

                 │     old     │                 new                 │
                 │  allocs/op  │  allocs/op   vs base                │
Put/parallel-8      678.0 ± 1%    524.5 ± 2%  -22.64% (p=0.000 n=10)
Put/sequential-8   1.329k ± 0%   1.183k ± 0%  -10.91% (p=0.000 n=10)
geomean             949.1         787.9       -16.98%
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-20 15:42:58 +00:00
7627d08914 Revert "[#972] Use require.ElementsMatch() where possible"
This reverts commit 6d9707ff1f.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-19 15:36:01 +00:00
6d9707ff1f [#972] Use require.ElementsMatch() where possible
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-19 13:13:09 +00:00
d19ade23c8 [#959] node: Set mode to shard's components when open it
Avoid opening database for `metabase` and `cache` in `Degraded` mode.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-02-09 14:04:01 +00:00
cfc5ce7853 [#964] metabase: Drop GC marks if object not found
GC inhumes expired locks and tombstones on all the shards.
So it could be GC mark without object.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-08 07:54:39 +00:00
5ed330e436 [#927] metabase: Delete GC marks
`key` is changed inside `db.get`, so encode address again after get.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-24 18:51:16 +03:00
f526f49995 [#874] engine: Check object existance concurrently
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-23 09:28:29 +03:00
57171907e3 [#904] metabase: Return if object was actuall inserted
This requires to count metrics properly.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-17 19:04:39 +03:00