Commit graph

1326 commits

Author SHA1 Message Date
Alex Vanin
2271944064 [#945] metabase: Use cleanup instead of defer in tests
Cleanup is more idiomatic way to clean test data

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 16:10:01 +03:00
Alex Vanin
e41aba610d [#945] metabase: Fix containers listing
Container listing should not ignore tombstone and
storage group objects which are not stored in
primary buckets.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 16:10:01 +03:00
Pavel Karpy
8956f015fc [#941] morph/notary: Delete useless "low till" check
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-25 19:16:05 +03:00
Pavel Karpy
78aab096fb [#941] morph/notary: Add dynamic notary deposit till calculation
Do not set `till` as some constant:
use maximum of two values instead:
1. currentDepositTill;
2. currentHeight+epochDuration+constant.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-25 19:16:05 +03:00
Evgenii Stratonikov
0057eeb0f7 [#901] *: release worker pools where possible
Some of the pools are initialized during config initialization,
so it isn't possible currently to release them in one place.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-25 18:53:36 +03:00
Evgenii Stratonikov
10afd26354 [#901] engine: release shard pools in Close
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-25 18:53:36 +03:00
Evgenii Stratonikov
0beaed2ef4 [#901] util: implement Release for PseudWorkerPool
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-25 18:53:36 +03:00
Pavel Karpy
616013cb8a [#933] write-cache: Use real number of objects in counters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-25 18:49:10 +03:00
Pavel Karpy
29955a3036 [#933] FSTree: Add NumberOfObjects method
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-25 18:49:10 +03:00
Alex Vanin
eaecc438f8 [#932] neofs-cli: Support ContainerAliasFee global config parameter
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-19 19:15:56 +03:00
Pavel Karpy
b148b85af4 [#919] pkg/object: Do not accept objects with empty owner
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-18 12:15:06 +03:00
Evgenii Stratonikov
7cb3d0cb4a [#885] policer: remove objects for removed container
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-18 12:14:14 +03:00
Evgenii Stratonikov
1a1435be3d [#791] neofs-lens: inspect objects by address
```
> neofs-lens list --path ./blob/blobovnicza/1/1/0
6ay4GfhR9RgN28d5ufg63toPetkYHGcpcW7G3b7QWSek/9ibXu6v4uTwLEcME5vyHev6Zi8LpxqiWTe1dahKpAbb6
H3VBttoLQoknzMDgnVNyLZ8EpkDnQjnaxDr9fnAWeEHA/Hw1titdGh7BrTe2yLotiYbVh9FQaRRNhoNzXTyetpFgt

> neofs-lens inspect --path ./blob/blobovnicza/1/1/0 \
    --address 6ay4GfhR9RgN28d5ufg63toPetkYHGcpcW7G3b7QWSek/9ibXu6v4uTwLEcME5vyHev6Zi8LpxqiWTe1dahKpAbb6 \
    --header --out payload
Version: v2.1
Type: REGULAR
CID: 6ay4GfhR9RgN28d5ufg63toPetkYHGcpcW7G3b7QWSek
ID: 9ibXu6v4uTwLEcME5vyHev6Zi8LpxqiWTe1dahKpAbb6
Owner: 2dokPzmmcLnnR21jQB3qPppTQRgwMNMKEWD
CreatedAt: 0
PayloadSize: 32
Attributes:
  foo: bar

> hexdump -C payload
00000000  ff 6c d4 71 c4 83 f1 5f  b9 0b ad b3 7c 58 21 b6  |.l.q..._....|X!.|
00000010  d9 55 26 a4 1a 95 04 68  0b 4e 7c 8b 76 3a 1b 1d  |.U&....h.N|.v:..|
00000020
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-18 11:55:02 +03:00
Leonard Lyubich
e0f0188466 [#907] container/put: Work with named containers
Add name and zone arguments to `Put` method of wrapper over the Container
contract client. Pass result of `container.GetNativeNameWithZone` function
to the method in `Put` helper function. Due to this, the storage node will
call the method depending on the presence of the container name in the
attributes.

Make IR to listen `putNamed` notification event. The event is processed like
`put` event, but with sanity check of the container attributes.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-18 11:54:34 +03:00
Leonard Lyubich
7db47c88bf [#907] morph/container: Add native name and zone to PutArgs
Add `PutArgs.SetNativeNameWithZone` method which sets native name and zone
for container. Call `putNamed` method of Container contract if name is set,
otherwise call `put` method.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-18 11:54:34 +03:00
Leonard Lyubich
36c5e4c527 [#907] morph/event/container: Parse PutNamed event
Implement `ParsePutNamedNotary` function which parses `PutNamed` structure
from `event.NotaryEvent`. Share common code with `ParsePutNotary` function.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-18 11:54:34 +03:00
Leonard Lyubich
a9494412da [#907] morph/event/container: Define PutNamed event
Define `PutNamed` structure of notary notification from `putNamed` method of
Container contract. Embed `Put` type in order to inherit methods and
parsing of common parts with `put` method.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-18 11:54:34 +03:00
Leonard Lyubich
3173bf345e [#907] morph/event: Implement Op to string converter
Implement `StringFromOpcode` function that tries to retrieve `string` to
`Op`. Add a comment about neo-go source code that is used for implementation
of converters.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-18 11:54:34 +03:00
Evgenii Stratonikov
123328a2f4 blobstor: close all opened blobovniczas
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-18 10:44:50 +03:00
Leonard Lyubich
1b3637aad7 [#884] shard: Mark expired tombstones as garbage
Mark expired tombstones as garbage in `Shard.HandleExpiredTombstones`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-15 16:09:44 +03:00
Leonard Lyubich
8efeba8010 [#833] morph/subscriber: Change value cast for NotificationEventID
`WSClient` of Neo Go v0.97.3 sets value of notification with
`NotificationEventID` to `subscriptions.NotificationEvent` type which wraps
previously used `state.NotificationEvent`.

Change type cast and pull `state.NotificationEvent` structure from new type.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-15 13:56:19 +03:00
Leonard Lyubich
283ccc04b4 [#833] services/netmap: Support new fields of netmap.NetworkInfo
Make the implementation of network info source (Netmap V2 service
dependency) to read MillisecondsPerBlock sidechain parameter and NeoFS
network parameters depending on the client version.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-15 13:56:19 +03:00
Leonard Lyubich
7582689de4 [#833] morph/netmap: Support ListConfig contract method
Implement `Client.ListConfig` method which calls `listConfig` method of
Netmap contract. Implement `Wrapper.IterateConfigParameters` method which
uses previous one. Implement `wrapper.WriteConfig` helper function which
allows to interpret parameters by names.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-15 13:56:19 +03:00
Leonard Lyubich
cb22e2bf29 [#833] morph/client: Implement MsPerBlock method
Add `Client.MsPerBlock` method which reads MillisecondsPerBlock network
parameter.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-15 13:56:19 +03:00
Leonard Lyubich
3929f6b396 [#833] Upgrade Neo Go and NeoFS API Go libraries
Upgrade Neo Go lib to release v0.97.3.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-15 13:56:19 +03:00
Pavel Karpy
96a65df32b [#873] innerring: Reuse atomic IR's epochDuration instead of RPC call
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-15 12:22:25 +03:00
Pavel Karpy
505d92bb06 [#873] morph/notary: Not warn at "low till value" notary deposit error
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-15 12:22:25 +03:00
Pavel Karpy
6638f7d9e6 [#873] innerring: Use multihanders in the New epoch timer
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-15 12:22:25 +03:00
Pavel Karpy
4afccab015 [#873] innerring: Delete duration from notary configs
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-15 12:22:25 +03:00
Pavel Karpy
e5748bfc96 [#873] innerring/notary: Change notary deposit logic
IR tries to keep 1:3 proportion of GAS and
notary balances respectively. If that proportion
has been messed(means that notary balance is
lower than required) it sends half of its
GAS balance to the notary service.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-15 12:22:25 +03:00
Pavel Karpy
a25bd2112d [#873] innerring/timers: Add multihandlers functionality for new epoch timer
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-15 12:22:25 +03:00
Pavel Karpy
d2096b392c [#873] morph/client: Add function that calculates notary deposit
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-15 12:22:25 +03:00
Evgenii Stratonikov
d55456f3ac [#905] morph/client: handle latest NNS version
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-14 15:21:12 +03:00
Leonard Lyubich
3e4f0c1eb9 [#674] object/put: Handle job submission errors
Close wait channel on shard pool error in order to prevent deadlock.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-14 10:20:39 +03:00
Leonard Lyubich
40a4a7faa2 [#674] object/put: Use pseudo worker pool for local operations
After storage engine started to limit number of PUT operations there is no
need to limited worker pool in Object Put service.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-14 10:20:39 +03:00
Leonard Lyubich
07130855aa [#674] util: Rename SyncWorkerPool with PseudoWorkerPool
`SyncWorkerPool` name is more appropriate for worker pool of size 1.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-14 10:20:39 +03:00
Leonard Lyubich
43ccab3294 [#674] storage engine: Fix unit test failed on shard pool NPE
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-14 10:20:39 +03:00
Leonard Lyubich
5b1975d52a [#674] storage engine: Use per-shard worker pools for PUT operation
Make `StorageEngine` to use non-blocking worker pools with the same
(configurable) size for PUT operation. This allows you to switch to using
more free shards when overloading others, thereby more evenly distributing
the write load.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-14 10:20:39 +03:00
Pavel Karpy
501c78f327 [#902] util/locode: Fix parsing minutes
Convert minutes of received coordinates
to decimal parts of degree, and do not
use decimal part of float as storage for
minutes: "5915N 01806E" is
59.25N 18.10E, not 59°15'N 18°06'E.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-12 22:25:01 +03:00
Evgenii Stratonikov
2f149f95d4 [#883] network: parse URI properly
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-12 15:50:51 +03:00
Alex Vanin
d8f7fed10a [#881] acl: Use session token from request at object.Put
Session token can be present in both object header and
request meta header. They are the same during initial object
placement.

At the object replication, storage node puts object without
any session tokens attached to the request. If container's eACL
denies object.Put for USER role (use bearer to upload), then
replication might fail on objects with session tokens of the
signed by container owner. It is incorrect, so use session
token directly from request meta header.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-08 17:18:32 +03:00
Evgenii Stratonikov
dd678cd976 [#868] blobstor: allow to decompress objects on-the-fly
We should be able to read whatever we have written earlier.
Compression setting applies only to the new objects.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-08 16:01:45 +03:00
Evgenii Stratonikov
cc377b34d2 [#868] blobstor: initialize (de-)compressors in Init
Do not log in options constructors. Also failure to
initialize compression module (possibly due to invalid options) is
certainly an error deserving proper treatment.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-08 16:01:45 +03:00
Evgenii Stratonikov
b8ba677c85 [#882] policer: add CID to the error message
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-08 08:21:01 +03:00
Alex Vanin
e41e74b5fc [#872] services/container: Ignore passed routes in load router
Higher level solution for original issue with lost announcements
from single node container.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-05 19:29:27 +03:00
Leonard Lyubich
8a1593fdcc [#791] writecache: Export OpenDB function
The function will be useful for the tool which works with write-cache parts.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-05 10:51:10 +03:00
Leonard Lyubich
ca4c9d4673 [#791] writecache: Implement iterator over small objects
There is a need to list addresses of the small objects stored in WriteCache
database.

Implement `IterateDB` function which accepts BoltDB instance and iterate
over all saved objects and passes their addresses to the hander.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-05 10:51:10 +03:00
Leonard Lyubich
f73c5c2259 [#791] blobovnicza: Support read-only mode
There is a need to open Blobovnicza instances in read-only mode in some
cases.

Add `ReadOnly` option. Do not create dir path in RO. Open underlying BoltDB
instance with ReadOnly flag. Document thal all writing operations should not
be called in ro (otherwise BoltDB txs fail).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-05 10:51:10 +03:00
Leonard Lyubich
cc5d3288a1 [#791] blobovnicza: Don't iterate over range buckets in Iterate
`Blobovnicza` can be initialized with any number of range buckets, and
reconstructed with different size limit. In previous implementation
`Iterate` could miss some stored objects if we construct `Blobovnicza` with
smaller number of ranges.

Make `Iterate` to traverse all buckets regardless of current instance
bounds.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-05 10:51:10 +03:00
Leonard Lyubich
5e2ca0d04b [#791] blobovnicza: Implement method to iterate over object addresses
In previous implementation `Blobovnicza.Iterate` op decoded object data only
and passed it to the handler. There is a need to iterate over all addresses
of the stored objects.

Add `DecodeAddresses` and `WithoutData` methods of `IteratePrm` type. Add
`Address` method to `IterationElement` type. Make `Iterate` to decode object
addresses if `DecodeAddress` was called and not read the data if
`WithoutData` was called. Implement `IterateAddresses` helper function to
simplify the code.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-05 10:51:10 +03:00