All error counting and hangling logic is present on the engine level.
Currently, we pass engine metrics with shard ID metric to shard, then
export 3 methods to manipulate these metrics.
In this commits all methods are removed and error counter is tracked on
the engine level exlusively.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
- `reportShardErrorBackground()` no longer differs from
`reportShardError()`, reflect this in its name;
- reuse common pieces of code to make it simpler.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
Target container ID is taken from tombstone: cmd/frostfs-node/object.go:507
Also object of type `TOMBSTONE` contains objectID, so tombstone and
tombstoned object must have the same containerID.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
* Remove `relay` field from put streamer as it's no longer used;
* Fix initialization of `Relay` object writer parameter.
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
If local EC chunk found, but remote node is off, then `HEAD --raw` request
returns object not found.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
* `Patch` can't be applied for non-regular type object (tombstones,
locks etc.)
* Complex object parts can't be patched. So, if an object has EC/Split
header, it won't be patched.
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
Created grpc connection should be established, so perform Healthcheck request
to check connection is ok.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
For EC chunks need to return EC parent object ID as
EC chunks don't have own attributes but inherit parent's.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
FSTree file counter used by writecache. As writecache has now only one
storage, so it is required to use real object size to get writecache
size more accurate than `count * max_object_size`.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
Add tests for `CreateViper` and `ReloadViper` to ensure that no extra
files, except *.yaml, *.yml, *.json, are loaded from config directory.
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
* Tree and object service have the same log for checking APE. So,
this check should be moved to common package.
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
Do not fail on same latest version to run compact on upgraded metabase.
Use NoSync on compact.
Log every batch on bucket delete stage.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
Since `frostfs-cli control shards rebuild` command was added,
there is no need for background rebuild now.
For failover tests used used value 1 to rebuild only schema change.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
If slice is altered in `for` loop, we cannot use range over its
length: it may cause panic if slice gets shorter.
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
* Split the logic of write target initialization to different packages;
* Refactor patch and put services: since both service initialize the target
themselves.
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
Since Go 1.22 a "for" statement with a "range" clause is able
to iterate through integer values from zero to an upper limit.
gopatch script:
@@
var i, e expression
@@
-for i := 0; i <= e - 1; i++ {
+for i := range e {
...
}
@@
var i, e expression
@@
-for i := 0; i <= e; i++ {
+for i := range e + 1 {
...
}
@@
var i, e expression
@@
-for i := 0; i < e; i++ {
+for i := range e {
...
}
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
The `TestGCDropsObjectInhumedFromWritecache` test was flaky because a
running asynchronous rebuild operation prevented GC from deleting the
object. A test-only shard option `WithDisabledRebuild` has been added
to fix this.
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
exportloopref is deprecated.
gopatch:
```
@@
var index, value identifier
var slice expression
@@
for index, value := range slice {
...
-value := value
...
}
@@
var index, value identifier
var slice expression
@@
for index, value := range slice {
...
-index := index
...
}
@@
var value identifier
var channel expression
@@
for value := range channel {
...
-value := value
...
}
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
Benchmark results:
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/pilorama
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
│ old │ new │
│ sec/op │ sec/op vs base │
ForestSortedIteration/bbolt,root-8 207.2µ ± 6% 173.6µ ± 6% -16.23% (p=0.000 n=10)
ForestSortedIteration/bbolt,leaf-8 3.910µ ± 5% 3.928µ ± 7% ~ (p=0.529 n=10)
geomean 28.46µ 26.11µ -8.27%
```
They are not representative, as the worst case is when we have multiple
items of different lengths. However, `FileName` is usually less than 100
in practice, so the asymptotics is the same.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
Renamed parameters `min/max` to avoid conflicts with
predeclared identifiers.
Replaced background context with parent context without
cancellation in closer functions in frostfs-node.
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
When node put chunk into EC container, `policer` may remove it as redundant.
This chunk marked as removed. When parent object removed and `gc` start iterating over chunk,
node count removing chunk twice.
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
* Also, resolve dependencies and conflicts for object service
by creating stub for `Patch` method.
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
The synchronized service reload protocol added in systemd version 253
requires that the service provides a MONOTONIC_USEC field alongside the
RELOADING=1 notification message for synchronization purposes. The value
carried in this field must be the system CLOCK_MONOTONIC timestamp at
the time the notification message was generated as systemd compares it
to other CLOCK_MONOTONIC timestamps taken by pid1.
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
This reverts commit 327d364f34.
Reverted due to the problem with reload signal sent by systemd.
`frostfs-ir` service reconfigures correctly and service's
statuses are being reported to systemd. However, since we
replaced `go:linkname` & `nanotime()` with `time.Since()`,
systemd refuses to accept reload signal response from
`frostfs-ir`. To maintain correct behaviour it was decided to
revevrt systemd-related changes until a better solution is
found.
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
EC parent and split gc marks should be deleted after last EC chunk delete.
Also should delete split info for EC parent and split parent.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
* Methods `Head`, `Get`, `GetRangeHash` should no longer use APE pre-checks
as that leads only to incorrect rule chain processing for requests:
1. Immediate return with `NoRuleFound` may be unexpected as some `Allow`
rule is actually defined but can't be matched yet as it gets no object
attributes;
2. Immdediate return with `Allow` may be incorrect as some `Deny` rule
is actually defined but can't bet matched yet as it gets no object
attirbutes;
3. Pre-check breaks compatibility for converted EACL-tables.
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
It is required to save split parent ID too, not only split ID.
Otherwise inhume operation works incorrect: shard with last part may be skipped
and parent object will be available.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
getSvc may change the values of some fields, so Head will affect Delete
or Put. In this case, the change is necessary so that the session token
is stored in the tombstone object (EC assemble calls `ForgetTokens`).
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
Now EC objects assembling is performed concurrently.
Also fixed issue with an error in case of getting
EC object via non-container node.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
Previously we used pointer, this could have worked,
because most of the time, the netmap is cached.
This didn't work, however, because `lastNm` field was always nil.
Rework the mechanism completely:
1. Use epoch to track netmap versions, as it it simpler and
is unrelated to the TTL of an underlying cache.
2. Fix a bug where the epoch could change while mutex was unlocked.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
* We used several utility functions to parse frostfsid client
subject and extended subject. However, following the changes
in TrueCloudLab/frostfs-contract#97, these utility functions
have become public. So there is no more need to have them here.
* There was a mismatch of slice parameter required length between
frostfs-node's and frostfs-contract's utility functions,
`checkStackItem()` solves this problem.
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
When AddByPath() is called concurrently on 2 different nodes,
internal path components may be created twice. This violates some
of our assumptions in GetByPath() and, indirectly, in S3 handling of
GetSubTree() results.
Add a test for the correct behaviour, fixes will follow.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
* Make session token expired at `current_epoch + 1` but
not at `current_epoch` when it's still valid.
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
* Refactor object and tree service - they should instantiate
chain router cheking the bearer token. If there are no bearer
token rules, then defaul chain router is used.
* Fix unit-tests.
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
* Unlike default chain router, `BearerChainFedRouter` performs checks for
overrides defined in the bearer token;
* Add unit-test for the introduced router.
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
