Commit graph

3236 commits

Author SHA1 Message Date
Evgenii Stratonikov
01a226b3ec [#2037] services/object: Fix concurrent map writes in traverser
```
fatal error: concurrent map writes

goroutine 4337 [running]:
github.com/nspcc-dev/neofs-node/pkg/services/object/put.(*traversal).submitProcessed(...)
        github.com/nspcc-dev/neofs-node/pkg/services/object/put/distributed.go:78
github.com/nspcc-dev/neofs-node/pkg/services/object/put.(*distributedTarget).iteratePlacement.func1()
        github.com/nspcc-dev/neofs-node/pkg/services/object/put/distributed.go:198 +0x265
github.com/panjf2000/ants/v2.(*goWorker).run.func1()
        github.com/panjf2000/ants/v2@v2.4.0/worker.go:68 +0x97
created by github.com/panjf2000/ants/v2.(*goWorker).run
        github.com/panjf2000/ants/v2@v2.4.0/worker.go:48 +0x65
```

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-10 10:54:50 +03:00
Pavel Karpy
110f6e7864 [#2000] cli: Provide a bearer token to spawned HEAD by DELETE
If a `neofs-cli object delete` operation is performing using a bearer token,
add it to the new `HEAD` requests that collects children OIDs.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-09 18:11:47 +03:00
Evgenii Stratonikov
c38ad2d339 [#1906] writecache: Do not require read-only mode in Flush
It was needed before we started to flush during transition to
`degraded` mode. Now it is confusing.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-08 16:51:58 +03:00
Evgenii Stratonikov
51a9306e41 [#2024] services/object: Unify status errors
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-08 15:56:36 +03:00
Evgenii Stratonikov
871be9d63d [#2024] services/object: Cover corner cases for children OutOfRange
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-08 15:56:36 +03:00
Pavel Karpy
3eb2ac985d [#1972] node: Fix object format unit tests
Includes:
1. Unused func removal;
2. Err check of the `Sign` method.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-08 13:18:15 +03:00
Pavel Karpy
76b87c6d94 [#1972] node: Do not save objects if node not in a container
Do not use node's local storage if it is clear that an object will be
removed anyway as a redundant. It requires moving the changing local storage
logic from the validation step to the local target implementation.
It allows performing any relations checks (e.g. object locking) only if a
node is considered as a valid container member and is expected to store
(stored previously) all the helper objects (e.g. `LOCK`, `TOMBSTONE`, etc).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-08 13:18:15 +03:00
Pavel Karpy
59eebc5eeb [#1972] cli: Fix lifetime flag in the lock command
That part of the code was refactored incorrectly.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-08 13:18:15 +03:00
Pavel Karpy
f79386e538 [#1972] node: Fix errors comments in the Put service
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-08 13:18:15 +03:00
Evgenii Stratonikov
d2cce62934 [#1818] writecache: Increase error counter on background errors
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-07 16:07:20 +03:00
Evgenii Stratonikov
a4a6d547a8 [#1818] writecache: Update storage ID during flush
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-07 16:07:20 +03:00
Evgenii Stratonikov
681400eed8 [#1818] metabase: Add UpdateStorageID operation
By default writecache puts the whole object to update storage ID.
This logic comes from the times when we needed to put objects
in the metabase by the writecache itself. Now this is done by the
blobstor at unmarshaling objects during flush only to update storage ID
is an overkill.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-07 16:07:20 +03:00
Evgenii Stratonikov
b580846630 [#1818] writecache: Reuse FSTree flushing code between flushes
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-07 16:07:20 +03:00
Evgenii Stratonikov
6e2f7e291d [#1818] writecache: Remove unused variable
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-07 16:07:20 +03:00
Pavel Karpy
7a75b3aaaf [#1699] meta: Do not return SplitInfoError on Delete
It is not an error: removing virtual object is expected and should be just
skipped. Getting a virtual object with `raw` flag is considered as an
impossible action, all the virtual objects removals will be handled via
their children's removals implicitly.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-07 16:04:09 +03:00
Evgenii Stratonikov
9cd8441dd5 [#1732] pilorama: Fill parent mark correctly
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-07 13:02:45 +03:00
Evgenii Stratonikov
163d8d778d [#1732] pilorama: Fix backwards log insertion
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-07 13:02:45 +03:00
Evgenii Stratonikov
52d85ca463 [#1732] pilorama: Improve logical error handling
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-07 13:02:45 +03:00
Evgenii Stratonikov
ba3db7fed5 [#1996] engine: Always select proper shard for a tree
Currently there is a possibility for modifying operations to fail
because of I/O errors and a new tree to be created on another shard.
This commit adds existence check for modifying operations.
Read operations remain as they are, not to slow things.
`TreeDrop` is an exception, because this is a tree removal and trying
multiple shards is not an unwanted behaviour.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-07 13:02:45 +03:00
Pavel Karpy
2e89176892 [#1971] cli: Unify CID and OID flags provision
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-03 15:23:16 +03:00
Evgenii Stratonikov
855de87b62 [#2007] services/object: Allocate memory on-demand in GET_RANGE
For big objects we want to get OutOfRange error before all the memory is
allocated.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-03 15:02:32 +03:00
Evgenii Stratonikov
289a7827c4 [#2007] services/object: Fix comment
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-03 15:02:32 +03:00
Pavel Karpy
abf4a63585 [#1991] cli: Refine container placement description
Not to confuse a user by mixing a replication vector number with its copy
number.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-02 15:14:19 +03:00
Pavel Karpy
3ee4260647 [#2009] github: Run CI on support branches
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-02 14:27:36 +03:00
Evgenii Stratonikov
245da1e50e [#1992] writecache: Allow to open in NOSYNC mode
Applicable only to FSTree as we cannot handle corrupted databases
properly yet.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-01 15:29:01 +03:00
Evgenii Stratonikov
a386448ab9 [#1992] neofs-node: Allow to open fstree in NOSYNC mode
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-01 15:29:01 +03:00
Evgenii Stratonikov
3c1f788642 [#1994] docs: Update storage node configuration
Reflect the reality after a not so recent refactoring.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-01 15:29:01 +03:00
Evgenii Stratonikov
0de9efa685 [#1992] fstree: Allow working in SYNC mode
Make O_SYNC the default and allow to opt-out explicitly.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-01 15:29:01 +03:00
Pavel Karpy
694d888219 Release v0.34.0 - Marado (마라도, 馬羅島)
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-31 15:07:53 +03:00
Pavel Karpy
d739e06289 [#1980] make: Update linter version in docker command
To be in sync with GitHub actions.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-31 15:07:53 +03:00
Pavel Karpy
cd2faf29a9 [#1980] go.mod: Update neofs-sdk to rc7
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-31 15:07:53 +03:00
Evgenii Stratonikov
34501685b7 [#1969] local_object_storage: Move ErrObjectIsExpired to another package
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-31 11:41:24 +03:00
Evgenii Stratonikov
56de2f1363 [#1969] local_object_storage: Simplify logic error construction
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-31 11:41:24 +03:00
Evgenii Stratonikov
fcdbf5e509 [#1969] local_object_storage: Add a type for logical errors
All logic errors are wrapped in `logicerr.Logical` type and do not
affect shard error counter.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-31 11:41:24 +03:00
Leonard Lyubich
98034005f1 [#1900] node: Fix loosing the "maintenance" status
In previous implementation node lost maintenance status after successful
switching to it. For example, after some period of time node sent
bootstrap requests with the "online" state instead of "maintenance".

Make `startMaintenance` method to set maintenance status in the
`networkState`.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-28 18:30:59 +03:00
Leonard Lyubich
db92e96e40 [#1900] node: Refactor bootstrap methods
In previous implementation bootstrapping state was chosen according to
bool flag which was not convenient.

Create separate method to boostrap with "online" and the current state.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-28 18:30:59 +03:00
Leonard Lyubich
e8c5f03c30 [#1905] shard: Don't log read-only errors of write-cache
There is no need to log `writecache.ErrReadOnly` errors in `Delete`
method of the `Shard`.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-28 18:30:45 +03:00
Leonard Lyubich
b1fa084756 [#1905] shard: Decrease severity level of write-cache failure logs
In previous implementation `Shard.Delete` logged writecache's removal
failures in `error` level. There is a need to decrease severity of these
log records since they aren't critical and don't require individual
review.

Change level of the message to `info`.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-28 18:30:45 +03:00
Leonard Lyubich
98ac525272 [#1978] cli/object: Gather all related object in delete session
Object removal session should reflect all objects related to the
removing one.

Make `OpenSessionViaClient` to gather the split members of the original
object in order to spread the session to them.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-28 13:49:41 +03:00
Stanislav Bogatyrev
7653a1f626 [#409] Fix package building doc typos
Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2022-10-28 12:58:32 +03:00
Stanislav Bogatyrev
43fe156f7c [#409] Fix more lintian warnings
Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2022-10-28 12:58:32 +03:00
c640374d98 [#409] Return OS release suffix to package version
Signed-off-by: Dmitriy Zabolotskiy <d.zabolotskiy@yadro.com>
2022-10-28 12:58:32 +03:00
1e0b1919ed [#409] Fix lintian warnings
Signed-off-by: Dmitriy Zabolotskiy <d.zabolotskiy@yadro.com>
2022-10-28 12:58:32 +03:00
Stanislav Bogatyrev
bbfc1d9263 [#409] Fix neofs-locode-db dependency
And update CHANGELOG.md =)

Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2022-10-28 12:58:32 +03:00
Stanislav Bogatyrev
e1a085ffd5 [#409] Remove docker target
It would not work this way for non-root users. Let's postpone.

Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2022-10-28 12:58:32 +03:00
Stanislav Bogatyrev
ff67e903ca [#409] Add default package changelog
Make it work with simple `dpkg-buildpackage`

Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2022-10-28 12:58:32 +03:00
92e9782c44 [#409] Debian packaging
Debian package building, including:

 - users creation if not exist;
 - dirs creation and ownership/ACLs set;
 - man/doc/examples installation;
 - shell completion generation;
 - locode DB binary download

Using:
	make debpackage

Signed-off-by: Dmitriy Zabolotskiy <d.zabolotskiy@yadro.com>
2022-10-28 12:58:32 +03:00
Evgenii Stratonikov
846ff515e6 [#1812] policer: Do not remove copies if there are maintenance nodes
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-26 19:13:17 +03:00
Evgenii Stratonikov
280e56f4bb [#1893] neofs-node: Do not fail unless all gRPC endpoints are unavailable
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-26 12:48:38 +03:00
Evgenii Stratonikov
1e6588e761 [#1974] shard: Do not panic in degraded mode
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-26 12:41:12 +03:00