Compare commits

...

1938 commits

Author SHA1 Message Date
79ba34714a [#79] cli: Fix panic when setting domain for container
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-03-01 11:11:11 +03:00
6d4f48f37a [TrueCloudLab#78] .github: Fix CODEOWNERS
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-28 16:44:50 +03:00
e9f3c24229 [#65] Use strings.Cut instead of strings.Split* where possible
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-28 13:39:14 +03:00
88e3868f47 [#37] cli: Add nns-name and nns-zone for container create
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-28 13:37:23 +03:00
6925fb4c59 [TrueCloudLab/hrw#2] node: Use typed HRW methods
Update HRW lib and use typed HRW methods to sort shards and nodes

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-02-28 13:36:25 +03:00
c3a7039801 [TrueCloudLab/hrw#2] node: Optimize shard hash
Compute shard hash only once

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-02-28 13:36:25 +03:00
a1ab25b33e [#72] .github: Fix CODEOWNERS
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-22 16:29:58 +03:00
73bb590cb1 [#64] node: Use pool_size_local and separate pool for local puts
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-02-22 13:43:19 +03:00
7eaf159a8b [#63] adm: Fix contract wallet creation
Create contract wallet only by init and update-config command.

Close #63

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-02-22 10:08:37 +03:00
cb5468abb8 [#66] node: Replace interface{} with any
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-02-21 16:47:07 +03:00
3d873237d5 [#44] Update Changelog
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-02-21 10:00:28 +03:00
633c5a35de [#44] adm: Support multiple configs
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-02-21 10:00:28 +03:00
5f06232d34 [#44] cli: Support multiple configs
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-02-21 10:00:28 +03:00
bed5a36235 [#44] ir: Support multiple configs
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-02-21 10:00:28 +03:00
87e69b9349 [#44] node: Support multiple configs
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-02-21 10:00:28 +03:00
Pavel Karpy
337049b2ce [#56] node: Allow reading expired locked object
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-21 09:56:57 +03:00
Pavel Karpy
3beef10f89 [#61] node: Do not fetch missing objects
If an object is missing in a `meta`, shard should not look for it in
a `blobstor`.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 14:47:38 +03:00
5303736acd [#62] Fix CHANGELOG.md
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-02-20 14:25:21 +03:00
22f3c7d080 [#1868] Reload config for pprof and metrics on SIGHUP
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-20 13:53:27 +03:00
2b755ddb12 [#2260] node: Use a separate client cache for PUT service
Currently, under a mixed load one failed PUT can lead to closing
connection for all concurrent GETs. For PUT it does no harm: we have
many other nodes to choose from. For GET we are limited by `REP N`
factor, so in case of failover we can close the connection with the only
node posessing an object, which leads to failing the whole operation.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
0b61a3c961 [#2260] network/cache: Ignore clients only on Dial errors
The problem is that accidental timeout errors can make us to ignore
other nodes for some time. The primary purpose of the whole ignore
mechanism is not to degrade in case of failover. For this case,
closing connection and limiting the amount of dials is enough.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
bf1e59bb83 [#2260] network/cache: Ignore context cancelled errors
Timeouts on client side should node affect inter-node communication.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
2567f8020e [#2260] services/object: Do not assemble object with TTL=1
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
d1d123d180 [#2234] writecache: Fix possible panic in initFlushMarks
In case we have many small objects in the write-cache, `indices` should
not be reused between iterations.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
315141dc2c [#2252] fstree: Allow concurrent writes
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
Pavel Karpy
b422ac9f94 [#2164] node: Fix multi-client error reporting
Missing `ReportError` method did not allow casing multi-client interface to
`errorReporter` interface and dropping broken connections.
`replicationClient` embeds that interface, and it is widely used across
node's code. Embedded interface does not allow casting its parent structure
to `errorReporter` and breaks multi client error reporting logic.
Multi-client scheme is extremely hard to maintain, it makes unpredictable
casts and does not allow tracking code flow, so it will be refactored in the
future anyway.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 13:53:27 +03:00
Pavel Karpy
95ee905861 [#2244] node: Fix subscriptions lock
Subscribing without async listening could lead to a dead-lock in the
`neo-go` client.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 13:53:27 +03:00
Pavel Karpy
07ec51ea60 [#2244] node: Add object address to WC's operations
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 13:53:27 +03:00
Pavel Karpy
dbbbef9ddb [#2244] node: Update expired storage ID by WC
Previously, node could get an "infinite" small object: it could be expired
and thus could not be flushed (update its storage ID) to metabase => could
not be marked as flushed => node never removes such object and repeat all
the cycle one more time. If object exists and is not marked with GC (meta
returns `ErrObjectIsExpired`, not `ObjectNotFound` and not
`ObjectAlreadyRemoved`), its ID is safe to update _in the same_ bbolt
transaction.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 13:53:27 +03:00
351fdd9fa2 [#2246] node: Allow to configure tombsone lifetime
Currently, DELETE service sets tombstone expiration epoch to
`current epoch + 5`. This works less than ideal in private networks
where an epoch can be e.g. 10 minutes. In this case, after a node is
unavailable for more than 1 hour, already deleted objects have a chance
to reappear.

After this commit tombstone lifetime can be configured.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
6fd88a036f [#2241] metrics: Fix request count metrics names
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
2272c55c4d [#2238] engine: Add test for component initialization failures
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
5cb2c5ae62 [#2238] engine: Add test for component initialization failures
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
59748b7ae8 [#2238] neofs-node: Gracefully handle shard initialization errors
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
427fe276f2 [#2238] shard: Try closing all components
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
c53903ccd0 [#2238] engine: Make Open and Init similar
1. Both could initialize shards in parallel.
2. Both should close shards after an error.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
e0309e398c [#2239] writecache: Fix possible deadlock
LRU `Peek`/`Contains` take LRU mutex _inside_ of a `View` transaction.
`View` transaction itself takes `mmapLock` [1], which is lifted after tx
finishes (in `tx.Commit()` -> `tx.close()` -> `tx.db.removeTx`)

When we evict items from LRU cache mutex order is different:
first we take LRU mutex and then execute `Batch` which _does_ take
`mmapLock` in case we need to remap. Thus the deadlock.

[1] 8f4a7e1f92/db.go (L708)

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
58367e4df6 [#2232] pilorama: Merge in-queue batches
To achieve high performance we must choose proper values for both
batch size and delay. For user operations we want to set low delay.
However it would prevent tree synchronization operations to form big
enough batches. For these operations, batching gives the most benefit
not only in terms of on-CPU execution cost, but also by speeding up
transaction persist (`fsync`).
In this commit we try merging batches that are already
_triggered_, but not yet _started to execute_. This way we can still
query batches for execution after the provided delay while also allowing
multiple formed batches to execute faster.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
236c4af615 [#2224] adm: Use native neo-go sessions in dump-hashes
If we had lots of domains in one zone, `dump-hashes` for all others
can miss some domains, because we need to restrict ourselves with _some_
number.
In this commit we use neo-go sessions by default, with a proper
failback to in-script iterator unwrapping.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
Pavel Karpy
40822adb51 [#2213] node: Do not return object expired object
"Object is expired" means that object is presented in `meta` but it is not
`ObjectNotFound` error. Previous implementation made `shard` search for an
object without `meta` which was an error.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 13:53:27 +03:00
Roman Khimov
ad93d4db7c CHANGELOG: add more fancy glyphs
How could you forget adding it?

Signed-off-by: Roman Khimov <roman@nspcc.ru>
2023-02-20 13:53:27 +03:00
Roman Khimov
a6f071d66f CHANGELOG: fix whitespacing errors
Signed-off-by: Roman Khimov <roman@nspcc.ru>
2023-02-20 13:53:27 +03:00
9afe86ba3e [#2212] morph: Fix subscription restoration
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
c43b2dbac9 [#1465] Add log entry for morph components shutdown action
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-17 12:13:00 +03:00
85cf1f47ac [#1465] node: Prevent process from killing by systemd when shutting down
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-17 12:13:00 +03:00
362f24953a [#47] shard: Switch container size metric from physical to logical capacity
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-02-17 12:03:42 +03:00
Pavel Karpy
901d62567d [#57] node: Broadcast link objects
It boosts object assembling by an _average_ container node.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-17 11:58:27 +03:00
Aleksey Pastukhov
269a4e9b50 [#53] Fix dirty in version
Signed-off-by: Aleksey Pastukhov <a.pastukhov@yadro.com>
2023-02-16 11:10:43 +03:00
3e5bc394b5 [#48] adm: Add initialize test for 1 node
Single node is used in dev-env, worth testing.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-16 09:07:18 +03:00
1d3669232e [#48] adm: Allow using nonzero magic with local client
neo-go actor API uses `getVersion` call which returned incorrect magic.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-16 09:07:18 +03:00
204cd3a11c [#31] fstree: Optimize treePath
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-10 12:49:31 +03:00
dee4498c1e [#31] fstree: Do not check for a file existence twice
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-10 12:49:31 +03:00
abbecf49d6 [#31] fstree: Speedup string-to-address conversion
```
name                  old time/op    new time/op    delta
_addressFromString-8    1.25µs ±30%    1.02µs ± 6%  -18.49%  (p=0.000 n=9+9)

name                  old alloc/op   new alloc/op   delta
_addressFromString-8      352B ± 0%      256B ± 0%  -27.27%  (p=0.000 n=9+10)

name                  old allocs/op  new allocs/op  delta
_addressFromString-8      6.00 ± 0%      4.00 ± 0%  -33.33%  (p=0.000
n=10+10)
```

Also, assure compiler that `s` doesn't escape:
Before this commit:
```
./fstree.go:74:24: leaking param: s
./fstree.go:90:6: moved to heap: addr
```

After this commit:
```
./fstree.go:74:24: s does not escape
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-10 12:49:31 +03:00
ab21d90cfb [#1794] shard: Add increasing case for the payload size metric
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-02-09 13:30:23 +03:00
5ffa826897 [#1] Update Changelog
Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-02-06 17:41:14 +03:00
cb016d53a6 [#1] Fix comments and error messages
Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-02-06 17:41:14 +03:00
c761a95eef [#1] Fix project name in control service
Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-02-06 17:41:14 +03:00
f825cfac78 [#1] Fix project name in comments here and there
Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-02-06 17:41:14 +03:00
c6645ef775 [#1] Documentation rebranding
Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-02-06 17:41:14 +03:00
488eece25f [#1] Fix Debian package references to old upstream
Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-02-06 17:41:14 +03:00
1cedd446bb [#1] Fix sample configs
- Update sample configs to match unit tests
- Remove Docker container for N3 testnet
  Will return with updates when FrostFS enters N3

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-02-06 17:41:14 +03:00
1858f11146 [#1] Fix viper env prefix in cli tools
Changing env prefix and corresponding example config files.

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-02-06 17:41:14 +03:00
Pavel Karpy
73bc1b0b68 [#38] node: Fix linter warnings
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-06 17:27:54 +03:00
515c60bdf4 [#1889] adm: Add command morph netmap-candidates
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-06 17:26:34 +03:00
ee24815748 [#1889] Move flag --config in cmd/frostfs-adm/internal/commonflags/flags.go
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-06 17:26:34 +03:00
2b09564355 [#1889] Move netmap.go and exit.go from cli to cmd/internal/common
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-06 17:26:34 +03:00
5a9d6a09d8 [#8] cli: Set flag mode required for control shards set-mode
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-06 16:45:03 +03:00
2540598779 [#36] Packaging: allow any version
Also updated changelog

Signed-off-by: Dmitriy Zabolotskiy <d.zabolotskiy@yadro.com>
2023-01-31 16:33:32 +03:00
5d64a354cb [#1] Fix naming in FrostFS Adm help output
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-01-31 11:24:50 +03:00
d7e9e2ef9e [#1] Fix naming in FrostFS Lens help output
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-01-31 11:24:50 +03:00
d31d8c5335 [#1] Fix naming in FrostFS CLI help output
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-01-31 11:24:50 +03:00
406ff1360f [#1] Fix version output for all compiled binaries
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-01-31 11:24:50 +03:00
Pavel Karpy
89a0266f5e [#1794] metrics: Track physical object capacity per shard
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-01-26 20:06:28 +03:00
Evgenii Stratonikov
9513f163aa [#2116] metrics: Track physical object capacity in the container
Currently we track based on `PayloadSize`, because it is already stored
in the metabase and it is easier to calculate without slowing down the
whole system.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-01-26 20:06:28 +03:00
d65a95a2c6 [#28] pilorama: Remove LogMove struct
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
46c62be7e8 [#28] Fix linter issues
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
072a7d61ab [#2176] neofs-node: Do not cache full container list
We rarely need to list all containers: as one example
we need it for tree service synchronization once per epoch.
Given that cache TTL has the order of block time it makes no sense
to cache the list of all containers.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
8078af3424 [#2176] neofs-node: Do not invalidate old cache items
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
e5a40b90b6 [#2176] neofs-node: Peek during cache invalidation
`Get` needs write mutex and makes our item move to top.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
93eb72ef44 [#2176] neofs-node: Remove unused field from morphContainerWriter
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
8ee590794f [#1962] cli: common.PrintVerbose prints via cobra.Command.Printf
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
cff4184cd3 [#2128] Add doc for extended headers
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
c72576e72f [#2208] engine: Log time-consuming shard operations
Currently the only way to tell whether `evacuate/set-mode` is finished
is to set a very big timeout and _hope_ that the operation will finish.
In this commit we add INFO logs for such operations which should
simplify the life of an administrator.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
87f0e3ea25 [#2208] fstree: Rename file after write
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
792319a044 [#2208] fstree: Remove file if there was an error during write
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
67c97c6804 [#2210] services/tree: Drop messages not in queue
Currently, under high load clients are blocked on channel send
and the number of goroutines can increase indefinitely.
In this commit we drop replication messages if send/recv queue is full
and rely on a background synchronization.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
25d5995cef [#2210] pilorama: Allocate bucket name outside of batches
1. Reduce allocations inside transactions.
2. Do not encode container ID to string: it allocates a lot and takes more
space.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
165a600624 [#2210] pilorama: Reduce the amount of keys per node
Under high load we are limited by the _amount_ of keys we need to update
in a single transaction. In this commit we try storing all state
with a single key.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Pavel Karpy
64a5294b27 [#2200] shard: Do not fetch big objects from blobovniczas
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Pavel Karpy
91757329ae [#2200] shard: Fix blobstor obj fetching
In the previous implementation any non-nil error that preceded object
fetching from blobstor led to iterating over every storage (in other words,
no storage ID information was taken into account). Now storage ID is
skipped only if metabase (storage ID source) returns any error.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Pavel Karpy
cf1a91a758 [#2206] blobovnicza: Use Latin letters in the code
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Pavel Karpy
c33ad3c474 [#2164] node: Use reconnect_interval from config
Not always the default one.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
0d8366f475 [#2207] object/acl: Return status error for expired session token
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
6451f019d2 [#2203] shard: Do not panic in Close after unsuccessful Init
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Evgenii Stratonikov
6efa93be0a [#1621] services/tree: Return Apply result asyncronously
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Evgenii Stratonikov
ac81c70c09 [#1621] pilorama: Batch related operations
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
9009612a82 [#2198] blobovniczatree: Properly handle concurrent active blobovnicza update
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
cedbd380f2 [#2197] pilorama: Close database in degraded mode
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Pavel Karpy
1d21b1e3e8 [#1978] node: Do not drop clients on split errors
After the reconnection interval feature there was an bug related to the big
objects collecting: split error is returned from a client directly, not
via API status and was considered as a connection error.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
554b85411f [#2190] services/object: Log service error with INFO level
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
b0ad1b9ed2 [#2193] pilorama: Use do in TreeMove
It should be similar to a `TreeAddByPath`. `applyOperation` is used for
`Apply` when the operation can be inserted in the middle of a log.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
ba393e3e91 [#2188] engine: Fix panic during setting shard mode
Under load changing shard mode can lead to it being removed from the
list during some other PUT.
```
Dec 28 07:01:26 az neofs-node[364505]: panic: runtime error: invalid memory address or nil pointer dereference
Dec 28 07:01:26 az neofs-node[364505]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xc9fbb1]
Dec 28 07:01:26 az neofs-node[364505]: goroutine 11791912 [running]:
Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).putToShard(0xc000435490, {0xc0003f7a28?, 0xc0001192c0?}, 0x2, {0x0, 0x>
Dec 28 07:01:26 az neofs-node[364505]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:91 +0x1b1
Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).put.func1(0xc000435490?, {0xc0003f7a28?, 0xc0001192c0?})
Dec 28 07:01:26 az neofs-node[364505]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:71 +0x19c
Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).iterateOverSortedShards(0x1?, {{0x62, 0x23, 0xfe, 0x60, 0x67, 0xd5, 0x>
Dec 28 07:01:26 az neofs-node[364505]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/shards.go:225 +0xc8
Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).put(0xc000435490, {0x1?})
Dec 28 07:01:26 az neofs-node[364505]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:66 +0x2a9
Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).Put.func1()
Dec 28 07:01:26 az neofs-node[364505]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:43 +0x2a
Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).execIfNotBlocked(0x8?, 0x38?)
Dec 28 07:01:26 az neofs-node[364505]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/control.go:147 +0xcf
Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).Put(0xc4df775a80?, {0x0?})
Dec 28 07:01:26 az neofs-node[364505]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:42 +0x65
Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.Put(0xc06d928b80?, 0xc06b1b8dc8?)
Dec 28 07:01:26 az neofs-node[364505]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:158 +0x19
Dec 28 07:01:26 az neofs-node[364505]: main.engineWithoutNotifications.Put({0x20301b?}, 0x20301b?)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
118783c4cf [#26] .github: Fix CODEOWNERS
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:23 +03:00
3d57f4c961 [#2179] test: Fix test TestEvacuateNetwork/multiple_shards,_evacuate_many
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-01-24 13:37:49 +03:00
d4d2a8c865 [#11] Rename NeoFS to FrostFS in docs
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-12 08:55:47 +03:00
3d1d2ee7b1 [#11] Regenerate proto files
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-12 08:55:47 +03:00
054bc4a727 [#11] Rename contract-related NeoFS occurences
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-12 08:55:47 +03:00
9cb4b4cc17 [#11] Rename neofsid contract to frostfsid
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-12 08:55:47 +03:00
023396e6a4 [#11] adm: Remove NeoFS mentions
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-12 08:55:47 +03:00
19d180b510 [#11] Rename neofs contract to frostfs
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-12 08:55:47 +03:00
e355442532 [#10] .docker: Update go version to 1.18
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-01-10 12:27:25 +03:00
cc71e6902a [#7] Rebranding leftovers
- logo update
- docker image build fixes
- README and other texts updates

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-01-09 11:41:51 +03:00
fdb0affc31 [#5] frost-node: Used generic cache
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
9936b112b8 [#5] blobstor: Use generic LRU cache
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
f0be0befc5 [#5] services/object_manager: Use generic LRU cache
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
1b3374ac7f [#5] services/tree: User generic LRU cache
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
d0a0432a51 [#5] innerring: Use generic LRU cache
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
4155c1bdff [#5] writecache: Use generic LRU cache
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
8f61cc1dcc [#5] policer: Use generic LRU client
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
55b403e0ee [#5] morph/client: Use generic LRU cache
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
d8fb9c85eb [#5] go.mod: Bump supported go version to 1.18
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
6f8dd816fb [#2189] go.mod: Update nats-io
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
26560b6b8d [#2184] go.mod: Update dependencies
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
0272218eb9 [#2184] compression: Properly calculate upper bound
If the data is not compressible allocating `len(data)` will lead to a
slice reallocation. For a compressible data the results for small size
are flaky and we allocate a bit more. However, it feels right to use a
provided function if we need to pick any size at all.

```
name                                                           old time/op    new time/op    delta
Compression/size=128/zeroed_slice-8                              2.23µs ±12%    2.06µs ± 6%   -7.35%  (p=0.009 n=10+10)
Compression/size=128/not_so_random_slice_(block_=_123)-8         19.0µs ±10%    15.8µs ±16%  -17.09%  (p=0.000 n=9+10)
Compression/size=128/random_slice-8                              17.6µs ±15%    16.1µs ±16%     ~     (p=0.075 n=10+10)
Compression/size=1024/zeroed_slice-8                             3.05µs ±11%    2.84µs ±10%     ~     (p=0.089 n=10+10)
Compression/size=1024/not_so_random_slice_(block_=_123)-8        18.1µs ± 6%    18.2µs ±12%     ~     (p=0.971 n=10+10)
Compression/size=1024/random_slice-8                             48.6µs ± 6%    45.6µs ± 5%   -6.07%  (p=0.006 n=10+9)
Compression/size=32768/zeroed_slice-8                            26.8µs ± 3%    28.7µs ± 8%   +7.23%  (p=0.001 n=10+10)
Compression/size=32768/not_so_random_slice_(block_=_123)-8       44.3µs ± 8%    43.7µs ±13%     ~     (p=0.762 n=8+10)
Compression/size=32768/random_slice-8                            97.3µs ±32%    68.9µs ±15%  -29.13%  (p=0.000 n=10+10)
Compression/size=33554432/zeroed_slice-8                         29.8ms ± 9%    30.3ms ±17%     ~     (p=1.000 n=9+9)
Compression/size=33554432/not_so_random_slice_(block_=_123)-8    33.1ms ±14%    30.3ms ±11%   -8.61%  (p=0.043 n=10+10)
Compression/size=33554432/random_slice-8                         41.7ms ± 3%    30.1ms ± 8%  -27.72%  (p=0.000 n=9+10)

name                                                           old alloc/op   new alloc/op   delta
Compression/size=128/zeroed_slice-8                                128B ± 0%      144B ± 0%  +12.50%  (p=0.000 n=10+10)
Compression/size=128/not_so_random_slice_(block_=_123)-8           384B ± 0%      144B ± 0%  -62.50%  (p=0.000 n=10+10)
Compression/size=128/random_slice-8                                384B ± 0%      144B ± 0%  -62.50%  (p=0.000 n=10+10)
Compression/size=1024/zeroed_slice-8                             1.02kB ± 0%    1.15kB ± 0%  +12.50%  (p=0.000 n=10+10)
Compression/size=1024/not_so_random_slice_(block_=_123)-8        1.02kB ± 0%    1.15kB ± 0%  +12.50%  (p=0.000 n=10+10)
Compression/size=1024/random_slice-8                             2.56kB ± 0%    1.15kB ± 0%  -55.00%  (p=0.000 n=10+10)
Compression/size=32768/zeroed_slice-8                            32.8kB ± 0%    41.0kB ± 0%  +25.00%  (p=0.000 n=10+10)
Compression/size=32768/not_so_random_slice_(block_=_123)-8       32.8kB ± 0%    41.0kB ± 0%  +25.00%  (p=0.000 n=10+10)
Compression/size=32768/random_slice-8                            81.9kB ± 0%    41.0kB ± 0%  -50.00%  (p=0.000 n=10+10)
Compression/size=33554432/zeroed_slice-8                         33.6MB ± 0%    33.6MB ± 0%   +0.02%  (p=0.000 n=9+9)
Compression/size=33554432/not_so_random_slice_(block_=_123)-8    33.6MB ± 0%    33.6MB ± 0%   +0.02%  (p=0.000 n=8+10)
Compression/size=33554432/random_slice-8                         75.5MB ± 0%    33.6MB ± 0%  -55.55%  (p=0.000 n=10+10)

name                                                           old allocs/op  new allocs/op  delta
Compression/size=128/zeroed_slice-8                                1.00 ± 0%      1.00 ± 0%     ~     (all equal)
Compression/size=128/not_so_random_slice_(block_=_123)-8           2.00 ± 0%      1.00 ± 0%  -50.00%  (p=0.000 n=10+10)
Compression/size=128/random_slice-8                                2.00 ± 0%      1.00 ± 0%  -50.00%  (p=0.000 n=10+10)
Compression/size=1024/zeroed_slice-8                               1.00 ± 0%      1.00 ± 0%     ~     (all equal)
Compression/size=1024/not_so_random_slice_(block_=_123)-8          1.00 ± 0%      1.00 ± 0%     ~     (all equal)
Compression/size=1024/random_slice-8                               2.00 ± 0%      1.00 ± 0%  -50.00%  (p=0.000 n=10+10)
Compression/size=32768/zeroed_slice-8                              1.00 ± 0%      1.00 ± 0%     ~     (all equal)
Compression/size=32768/not_so_random_slice_(block_=_123)-8         1.00 ± 0%      1.00 ± 0%     ~     (all equal)
Compression/size=32768/random_slice-8                              2.00 ± 0%      1.00 ± 0%  -50.00%  (p=0.000 n=10+10)
Compression/size=33554432/zeroed_slice-8                           1.00 ± 0%      1.00 ± 0%     ~     (all equal)
Compression/size=33554432/not_so_random_slice_(block_=_123)-8      1.00 ± 0%      1.00 ± 0%     ~     (all equal)
Compression/size=33554432/random_slice-8                           2.00 ± 0%      1.00 ± 0%  -50.00%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
874be99076 [#2184] go.mod: Update neo-go to v0.100.1
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
71c64ae253 Release v0.35.0
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-12-30 11:07:35 +03:00
Mikhail Petrov
cfdb53a788 [#614] Add logo for dark theme
Signed-off-by: Mikhail Petrov <mike@nspcc.ru>
2022-12-30 11:07:35 +03:00
a2fe912d1a [#2164] neofs-node: Use a separate client cache for replicator
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
741482c26f [#2164] neofs-node: Allow to set reconnect interval
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
6f5edac730 [#2164] network/cache: Do not reconnect to failed clients immediately
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
f3caf6acfe [#2164] network/cache: Separate mutex for addr
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
0ace28e43d [#2175] blobovniczatree: Close all non-active blobovniczas
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
c1cf418956 [#2175] blobovniczatree: Make function parameters more descriptive
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
b4e90cdf51 [#2165] pilorama: Optimize TreeApply when used for synchronization
Because synchronization _most likely_ will have apply already existing
operations, it is much faster to check their presence in a read
transaction. However, always doing this will degrade the perfomance
for normal `Apply`. And, let's be honest, it is already not good.
Thus we add a separate parameter which specifies whether this logic is
enabled.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
f9fcd85363 [#2165] services/tree: Remember starting height for the synchronization
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
06137dbf8e [#2165] services/tree: Do not export synchronizeAllTrees
It is used only privately.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
c299b98afe [#2165] services/tree: Parallelize synchronization
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
de9957e076 [#2165] services/tree: Always synchronize all containers
In case of split-brain we must synchronize everything.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
d3054e577a [#2165] services/tree: Allow to set custom synchronization interval
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
6a4e5e6f0a [#2144] node: Try node's private key if dynamic token fetching failed
`GETRANGEHASH` request spawns `GETRANGE` requests if an object could not be
found locally. If the original request contains session, it can be static
and, therefore, fetching session key can not be performed successfully.
As the best effort a node could request object's range with its own key.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
86a4fba571 [#2144] node: Clarify KeyStorage.GetKey method
Actualize the doc, fix API status error return.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
b7a99c757a [#2097] cli: Clarify help for --expire-at parameter for commands object lock/put and bearer create
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-12-30 11:07:35 +03:00
e406036629 [#2166] go.mod: Update dependencies
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
21717262ec [#2016] shard: Check meta first on Get
`meta` should prevent returning removed objects (`GCMark` and `TS` relations
are `meta` abstractions).

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
74ec71446f [#2167] shard: Do not use write-cache by default in Head
Both `meta` and `write-cache` are expected to have a fast underlying disk,
so it does not seem like an optimisation. Moreover, `write-cache`'s `Head`
is a `Get` with payload cutting, it _must_ use more memory for no reason
(`meta` was created for such requests). Also, `write-cache` does not allow
performing any "meta" relations checks (such as locking, tombstoning).

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
1608fd1c07 [#2167] write-cache: Add "write-cache" to its logs
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
eea2892109 [#1956] node: Lock shard's mode on its methods switch
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
33d279a3f2 [#2152] cli: Do not search for LOCK objects when delete container when session provided
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
04b5ec759b [#2139] object/put: Use sync.Pool for temporary payloads
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
9e0decd12d [#2162] services/tree: Close connection after the syncronization
There was a goroutine leak here.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
e1c3bdbfa6 [#1621] pilorama: Remove Timestamp field from nodeInfo
It is already present in `Meta`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
1044adbe94 [#1621] pilorama: Improve memory allocation
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
2539d466a6 [#1621] pilorama: Seek after cursor invalidation
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
e9ba8931f8 [#1621] pilorama: Simplify bucket creation
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
fe7ddfdc6a [#1621] pilorama: Compare memory forests properly
Node children are not sorted and could occur in any order.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-30 11:07:35 +03:00
Pavel Karpy
306609030a [#2159] node: Add tree replication timeout configuration
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
edb1428248 [#2022] Add metric readonly to get shards mode
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-12-30 11:07:35 +03:00
e5c304536b [#2161] pilorama: Do not apply already existing operations
Speeds up synchronization a bit.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
3bb5a320d7 [#2154] services/tree: Do not log an error when synchronizing container of 1 node
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
387d1e2977 [#2127] services/tree: Randomize node order for synchronization
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
21c58c92a9 [#2145] meta: Do allow force inhuming a locked object
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
a30310b2ca [#2145] cli: Do not print "rpc error:" twice
That error part is already included in GRPC's library.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
b207dc424f [#2158] policer: Reduce default cache size
We use cache to avoid policing the same object multiple times in a short
time span (< 30 seconds). If we have 200_000 objects in a blobstor, it is a bit useless
-- if it takes 1 second to process an object and we have `replicator.pool_size: 20`
in config, the next iteration will happen in 10_000 second which is much
larger than 30 second. However we still consume a lot of memory, so it
makes sense to use saner default.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
0244f2e5ce [#2156] core/netmap: Change node addresses iteration order
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Leonard Lyubich
a68ff67ed8 [#2106] cli: Verify container owner in container delete command
In NeoFS containers can be removed on behalf of its owner only. To
improve user experience, there is a need to add ownership check to the
removal command of the NeoFS CLI.

Check container ownership in `container delete` command `Run` function.
The check can be skipped by `--force` option.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
73ef5b18c7 [#2092] go.mod: Update SDK
Includes a fix that adds error messages' text, it affects incomplete object
put error message.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
b413094704 [#2095] node: Fix collecting child objects
Stop child objects collection if the last returned object (the most "left"
object in the collected chain) starts exactly from the `GETRANGE`'s `from`
value.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-12-30 11:07:35 +03:00
Pavel Karpy
350eecfa13 [#2095] node: Do not allow GETRANGE requests with zero length
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-12-30 11:07:35 +03:00
Pavel Karpy
d6196c3971 [#2095] cli: Do not panic on object range
Also, includes range parsing error messages enhancement.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-12-30 11:07:35 +03:00
0b78af467e [#2140] engine: Fix error handling in TreeMove
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
923f84722a Move to frostfs-node
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-28 15:04:29 +03:00
Evgenii Stratonikov
42554a9298 [#2068] writecache: Remove deleted objects from the writecache
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-02 11:52:05 +03:00
Evgenii Stratonikov
4a49ea0855 [#2068] writecache: Allow to open FSTree in read-only mode
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-02 11:52:05 +03:00
Evgenii Stratonikov
857d2dc3f5 [#2068] writecache: Optimize initial flush existence checking
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-02 11:52:05 +03:00
Evgenii Stratonikov
63f604e948 [#2068] blobstor: Allow to provide storage ID in Exists
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-02 11:52:05 +03:00
Evgenii Stratonikov
6ad2b5d5b8 [#2068] blobovnicza: Add Exists method
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-02 11:52:05 +03:00
9eccf3bbf5 [#1486] Update changelog
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-12-02 11:43:33 +03:00
b0eeb0dfcf [#1486] node: Use endless notary deposit for side chain
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-12-02 11:43:33 +03:00
ba74244d1b [#1486] ir: Use endless notary deposit for side chain
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-12-02 11:43:33 +03:00
b6a40241f8 [#1486] morph: Add method for endless notary deposit
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-12-02 11:43:33 +03:00
Pavel Karpy
d54022eacc [#2047] node: Do not send chunk twice on request forwarding
That could happen if a node forwards request to a node that closed the
connection during the original object stream.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-12-02 11:27:48 +03:00
Evgenii Stratonikov
bd25db5d4a [#1984] metrics: Use separate metrics for success/failed requests
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-01 14:18:10 +03:00
Evgenii Stratonikov
e21c472dc7 [#1984] services/object: Increase put_req_count after the request is processed
As it is specified in metrics description.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-01 14:18:10 +03:00
afabd6be91 [#2038] neofs-cli: Check the sufficiency of the number of nodes in the selector for replicas
Perform this check on container creation.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-12-01 11:32:27 +03:00
Pavel Karpy
50d28b72c3 [#2075] node, ir: Log notary state with info
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 19:26:33 +03:00
Pavel Karpy
0f41c09207 [#2075] node: Do not make notary requests on shutdown
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 19:26:33 +03:00
Pavel Karpy
b8c30b88f6 [#2075] morph: Do not do notary deposit with zero balance
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 19:26:33 +03:00
Pavel Karpy
c01d4ecb50 [#2080] morph: Close morph clients
Could be related to "websocket users limit reached" on the `neo-go` server
side when an SN/IR is rebooting repeatedly.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 19:26:33 +03:00
Pavel Karpy
3d0768a1d3 [#2061] node: Unify meta.Get benchmarks
Make them get exactly one (different) object per a bench iteration.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 18:29:14 +03:00
Pavel Karpy
bc905f169d [#2061] meta: Add parallel bench for Get
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 18:29:14 +03:00
Pavel Karpy
761e82fecd [#2079] cli: Do not panic in object hash
Sign RPC requests with the provided key.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 16:58:52 +03:00
Pavel Karpy
960a2d0629 [#2081] ir: Set default key in IR's SDK clients
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 16:58:52 +03:00
8d15c14be6 [#2078] adm: Pack parameters for setPrice invocation
Contract arguments have to be packed.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-11-30 16:58:52 +03:00
Pavel Karpy
51963abce7 [#1972] node: Fix errors comments in the Put service
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 16:58:52 +03:00
Evgenii Stratonikov
7335a52f29 [#1732] pilorama: Improve logical error handling
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:58:52 +03:00
Evgenii Stratonikov
ae7b473768 [#2064] blobovniczatree: Remove index too big log
There is no need to log about a situation which is expected.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:53:18 +03:00
8a77b4638a [#2012] Add commands neofs-cli acl basic/extended print to show ACL table in human readable format
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-11-30 16:45:37 +03:00
Evgenii Stratonikov
59db66cdb6 [#2091] neofs-adm: Sign blocks properly in tests
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:45:13 +03:00
Evgenii Stratonikov
6882887bdd [#2091] neofs-adm: Do not query hashes via network
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:45:13 +03:00
Evgenii Stratonikov
c1ea6fd854 [#2091] morph/client: Use notary.FakeContractAccount wrapper
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:45:13 +03:00
Evgenii Stratonikov
b93be8869b [#2091] morph/client: Simplify code
1. Replace `mn` function with a `sigCount`.
2. Use `notary.FakeMultisigAccount` for account creation.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:45:13 +03:00
Evgenii Stratonikov
cfefebd5b3 [#2091] neofs-adm: Do not use deprecated methods
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:45:13 +03:00
Evgenii Stratonikov
6af52c46d8 [#2091] morph/client: Do not use deprecated methods
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:45:13 +03:00
51e886dd67 [#2090] neofs-cli: Remove --header from object get
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-11-30 11:46:02 +03:00
79130f781e [#2089] neofs-cli: Remove -g option from neofs-cli control ... and neofs-cli container create commands
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-11-30 09:14:50 +03:00
Evgenii Stratonikov
dd76ceadf1 [#2060] go.mod: Retract v1.* versions
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-29 09:29:42 +03:00
Pavel Karpy
ca3596dd05 [#2104] cli: OID signature in output
OID signature should always be present in an object; it does not relate
to the object split.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-29 08:52:09 +03:00
Evgenii Stratonikov
816c74d185 [#2075] morph/client: Ignore error if a transaction already exists
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
71853348b9 [#2063] morph/client: Support new hash format in morph nns client
Signed-off-by: Vladimir Domnich <v.domnich@yadro.com>
2022-11-19 11:01:04 +03:00
Pavel Karpy
ed4351aab0 [#2074] write-cache: Do not flush same object twice
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
dd225906a0 [#2074] write-cache: Remove unused variables
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
9f7ac6bf9f [#2069] innerring: Do not panic in Head
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
98e6dc5ce8 [#409] debian: Refactor storage service paths
Separate User data and Service data:
 - /var/lib/neofs/storage for service persistence
 - /srv/neofs for user data

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
660c38d07e [#2062] services/policer: Use a proper key for object cache
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
b1025bdb42 [#2057] meta: Fail write operations in R/O mode
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
fdeea1dfac [#2057] meta: Fix concurrent mode changes
Includes:
1. mode change read lock operation in every exported method that r/w the
underlying database;
2. returning `ErrDegradedMode` logical error if any exported method is
called in degraded (without a metabase) mode.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
3d6defd3e8 [#2057] meta: Do not lock the whole meta on GET
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
fa231b8c56 [#2057] blobstor: Block operations on a mode change
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
1779664644 [#2058] services/policer: Fix panic in shardPolicyWorker
```
2022/11/15 08:40:56 worker exits from a panic: runtime error: index out of range [0] with length 0
2022/11/15 08:40:56 worker exits from panic: goroutine 1188 [running]:
github.com/panjf2000/ants/v2.(*goWorker).run.func1.1()
	github.com/panjf2000/ants/v2@v2.4.0/worker.go:58 +0x10c
panic({0x1042b60, 0xc0015ae018})
	runtime/panic.go:1038 +0x215
github.com/nspcc-dev/neofs-node/pkg/services/policer.(*Policer).shardPolicyWorker.func1()
	github.com/nspcc-dev/neofs-node/pkg/services/policer/process.go:65 +0x366
github.com/panjf2000/ants/v2.(*goWorker).run.func1()
	github.com/panjf2000/ants/v2@v2.4.0/worker.go:68 +0x97
created by github.com/panjf2000/ants/v2.(*goWorker).run
	github.com/panjf2000/ants/v2@v2.4.0/worker.go:48 +0x68
```

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
b673d9e472 [#2053] engine: Do not switch mode because of logical errors
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
9a20498f34 [#1940] Removing all trees by container ID if tree ID is empty in pilorama.Forest.TreeDrop
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
426fe97990 [#2026] neofs-adm: Make contract update idempotent
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
634792077e [#1502] node: Store lock object on every container node
Includes extending listing methods in the Storage Engine with object types.
It allows tuning replication/policer algorithms: container nodes do
not remove `LOCK` objects as redundant and try to fulfill `LOCK` placement
on the ohter container nodes.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
fe09cd9c70 [#1502] core: Add AddressWithType
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
3b61cb4f49 [#1502] engine: Check all shards for LOCK'ing before inhuming
It allows keeping all the locked objects safe after metabase
resynchronization. Currently, all `LOCK` objects are broadcast to all nodes
in a container, it guarantees `LOCK` object presence in a regular situation.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
34e8d2ba56 [#1502] shard: Add IsLocked method
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
9a039ba582 [#1502] meta: Add IsLocked method
It gets an object and returns its locking status.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
7ef0303e13 [#2003] neofs-node: Allow to configure replicator pool size
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
659011c143 [#2048] network/cache: Optimize ClientCache
1. Remove a layer of indirection for mutex, `ClientCache` is already
   used by pointer.
2. Fix duplication of a `AllowExternal` field.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
597ed18269 [#2048] neofs-node: Use a separate client cache for client operations
Background workers can prevent user operations to complete because of
locking in cache.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
d5a14041e0 [#2040] node: Do not attach tokens in the assembly process
A container node is expected to have full "get" access to assemble the
object.
A non-container node is expected to forward any request to a container node.
Any token is expected to be issued for an original request sender not for a
node so any new request is invalid by design with that token.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
fd61bdadcb [#2040] node: Attach original meta to the spawned requests
Do not lose meta information of the original requests: cache session and
bearer tokens of the original request b/w a new generated ones. Middle
request wrappers should not contain any meta information, since it is
useless (e.g. ACL service checks only the original tokens).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
d65604ad30 [#1985] blobstor: Allow to report multiple errors to caller
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
f2d7e65e39 [#2035] engine: Allow moving to degraded from background workers
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
c85bea15ef [#1978] cli: Add children to the static session on DELETE
If an external session is provided and is not opened by CLI itself, add
children objects to it too. It fixes "not found" errors when removing a big
object with a predefined session (from a file).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
f48b1de54b [#2029] cli: Fix panic caused by flag redefinition
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
09a59fef56 [#2029] cli: Allow attaching static session to object hash
All the other object commands already have it.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
481b48b942 [#2028] node: Check session token's NBF and IAT
ACL service did not check "Not Valid Before" and "Issued At" claims.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
aadd2ad050 [#2028] node: Do not wrap malformed request errors
After presenting request statuses on the API level, all the errors are
unwrapped before sending to the caller side. It led to a losing invalid
request's context.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
2522d924b9 [#2037] services/object: Fix concurrent map writes in traverser
```
fatal error: concurrent map writes

goroutine 4337 [running]:
github.com/nspcc-dev/neofs-node/pkg/services/object/put.(*traversal).submitProcessed(...)
        github.com/nspcc-dev/neofs-node/pkg/services/object/put/distributed.go:78
github.com/nspcc-dev/neofs-node/pkg/services/object/put.(*distributedTarget).iteratePlacement.func1()
        github.com/nspcc-dev/neofs-node/pkg/services/object/put/distributed.go:198 +0x265
github.com/panjf2000/ants/v2.(*goWorker).run.func1()
        github.com/panjf2000/ants/v2@v2.4.0/worker.go:68 +0x97
created by github.com/panjf2000/ants/v2.(*goWorker).run
        github.com/panjf2000/ants/v2@v2.4.0/worker.go:48 +0x65
```

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
37f813604f [#2000] cli: Provide a bearer token to spawned HEAD by DELETE
If a `neofs-cli object delete` operation is performing using a bearer token,
add it to the new `HEAD` requests that collects children OIDs.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
b0e94b6a6b [#1906] writecache: Do not require read-only mode in Flush
It was needed before we started to flush during transition to
`degraded` mode. Now it is confusing.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
aa478f1def [#2024] services/object: Unify status errors
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
3875fef542 [#2024] services/object: Cover corner cases for children OutOfRange
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
b9fb0d6050 [#1972] node: Fix object format unit tests
Includes:
1. Unused func removal;
2. Err check of the `Sign` method.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
aab398f4f5 [#1972] node: Do not save objects if node not in a container
Do not use node's local storage if it is clear that an object will be
removed anyway as a redundant. It requires moving the changing local storage
logic from the validation step to the local target implementation.
It allows performing any relations checks (e.g. object locking) only if a
node is considered as a valid container member and is expected to store
(stored previously) all the helper objects (e.g. `LOCK`, `TOMBSTONE`, etc).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
a77392e9ce [#1972] cli: Fix lifetime flag in the lock command
That part of the code was refactored incorrectly.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
2849e465f9 [#1699] meta: Do not return SplitInfoError on Delete
It is not an error: removing virtual object is expected and should be just
skipped. Getting a virtual object with `raw` flag is considered as an
impossible action, all the virtual objects removals will be handled via
their children's removals implicitly.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
a3e7365cbd [#1732] pilorama: Fill parent mark correctly
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
134f2ba02e [#1732] pilorama: Fix backwards log insertion
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
2ef38cfbc4 [#1996] engine: Ignore pilorama.ErrTreeNotFound for write operations
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
a455ec18c3 [#2007] services/object: Allocate memory on-demand in GET_RANGE
For big objects we want to get OutOfRange error before all the memory is
allocated.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
ff5526038d [#2007] services/object: Fix comment
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
d8d3588e1b [#1996] engine: Always select proper shard for a tree
Currently there is a possibility for modifying operations to fail
because of I/O errors and a new tree to be created on another shard.
This commit adds existence check for modifying operations.
Read operations remain as they are, not to slow things.
`TreeDrop` is an exception, because this is a tree removal and trying
multiple shards is not an unwanted behaviour.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-03 15:29:23 +03:00
Pavel Karpy
08efe6eb11 [#2010] github: Run CI on support branches
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-02 14:29:17 +03:00
Evgenii Stratonikov
777fd32d4f [#1818] writecache: Increase error counter on background errors
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-02 14:24:02 +03:00
Evgenii Stratonikov
bffb0f894c [#1818] writecache: Update storage ID during flush
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-02 14:24:02 +03:00
Evgenii Stratonikov
5cf75404dc [#1818] metabase: Add UpdateStorageID operation
By default writecache puts the whole object to update storage ID.
This logic comes from the times when we needed to put objects
in the metabase by the writecache itself. Now this is done by the
blobstor at unmarshaling objects during flush only to update storage ID
is an overkill.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-02 14:24:02 +03:00
Evgenii Stratonikov
b64b14eb54 [#1818] writecache: Reuse FSTree flushing code between flushes
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-02 14:24:02 +03:00
Evgenii Stratonikov
a56927e3d4 [#1818] writecache: Remove unused variable
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-02 14:24:02 +03:00
bb52857b2b [#1338] neofs-cli: Add support to store/restore/delete binary objects
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-11-01 15:30:00 +03:00
Pavel Karpy
1f82c583e3 [#1971] cli: Unify CID and OID flags provision
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-01 15:07:19 +03:00
Pavel Karpy
7daa57d4d2 [#1991] cli: Refine container placement description
Not to confuse a user by mixing a replication vector number with its copy
number.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-01 15:00:00 +03:00
51e3810285 [#1689] Add new command morph list-containers in neofs-adm
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-11-01 09:43:02 +03:00
Evgenii Stratonikov
98a152256b [#1992] writecache: Allow to open in NOSYNC mode
Applicable only to FSTree as we cannot handle corrupted databases
properly yet.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-01 09:42:26 +03:00
Evgenii Stratonikov
b6930f2219 [#1992] neofs-node: Allow to open fstree in NOSYNC mode
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-01 09:42:26 +03:00
Evgenii Stratonikov
148da5fdbb [#1994] docs: Update storage node configuration
Reflect the reality after a not so recent refactoring.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-01 09:42:26 +03:00
Evgenii Stratonikov
f564430b90 [#1992] fstree: Allow working in SYNC mode
Make O_SYNC the default and allow to opt-out explicitly.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-01 09:42:26 +03:00
Pavel Karpy
694d888219 Release v0.34.0 - Marado (마라도, 馬羅島)
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-31 15:07:53 +03:00
Pavel Karpy
d739e06289 [#1980] make: Update linter version in docker command
To be in sync with GitHub actions.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-31 15:07:53 +03:00
Pavel Karpy
cd2faf29a9 [#1980] go.mod: Update neofs-sdk to rc7
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-31 15:07:53 +03:00
Evgenii Stratonikov
34501685b7 [#1969] local_object_storage: Move ErrObjectIsExpired to another package
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-31 11:41:24 +03:00
Evgenii Stratonikov
56de2f1363 [#1969] local_object_storage: Simplify logic error construction
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-31 11:41:24 +03:00
Evgenii Stratonikov
fcdbf5e509 [#1969] local_object_storage: Add a type for logical errors
All logic errors are wrapped in `logicerr.Logical` type and do not
affect shard error counter.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-31 11:41:24 +03:00
Leonard Lyubich
98034005f1 [#1900] node: Fix loosing the "maintenance" status
In previous implementation node lost maintenance status after successful
switching to it. For example, after some period of time node sent
bootstrap requests with the "online" state instead of "maintenance".

Make `startMaintenance` method to set maintenance status in the
`networkState`.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-28 18:30:59 +03:00
Leonard Lyubich
db92e96e40 [#1900] node: Refactor bootstrap methods
In previous implementation bootstrapping state was chosen according to
bool flag which was not convenient.

Create separate method to boostrap with "online" and the current state.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-28 18:30:59 +03:00
Leonard Lyubich
e8c5f03c30 [#1905] shard: Don't log read-only errors of write-cache
There is no need to log `writecache.ErrReadOnly` errors in `Delete`
method of the `Shard`.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-28 18:30:45 +03:00
Leonard Lyubich
b1fa084756 [#1905] shard: Decrease severity level of write-cache failure logs
In previous implementation `Shard.Delete` logged writecache's removal
failures in `error` level. There is a need to decrease severity of these
log records since they aren't critical and don't require individual
review.

Change level of the message to `info`.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-28 18:30:45 +03:00
Leonard Lyubich
98ac525272 [#1978] cli/object: Gather all related object in delete session
Object removal session should reflect all objects related to the
removing one.

Make `OpenSessionViaClient` to gather the split members of the original
object in order to spread the session to them.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-28 13:49:41 +03:00
Stanislav Bogatyrev
7653a1f626 [#409] Fix package building doc typos
Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2022-10-28 12:58:32 +03:00
Stanislav Bogatyrev
43fe156f7c [#409] Fix more lintian warnings
Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2022-10-28 12:58:32 +03:00
c640374d98 [#409] Return OS release suffix to package version
Signed-off-by: Dmitriy Zabolotskiy <d.zabolotskiy@yadro.com>
2022-10-28 12:58:32 +03:00
1e0b1919ed [#409] Fix lintian warnings
Signed-off-by: Dmitriy Zabolotskiy <d.zabolotskiy@yadro.com>
2022-10-28 12:58:32 +03:00
Stanislav Bogatyrev
bbfc1d9263 [#409] Fix neofs-locode-db dependency
And update CHANGELOG.md =)

Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2022-10-28 12:58:32 +03:00
Stanislav Bogatyrev
e1a085ffd5 [#409] Remove docker target
It would not work this way for non-root users. Let's postpone.

Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2022-10-28 12:58:32 +03:00
Stanislav Bogatyrev
ff67e903ca [#409] Add default package changelog
Make it work with simple `dpkg-buildpackage`

Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2022-10-28 12:58:32 +03:00
92e9782c44 [#409] Debian packaging
Debian package building, including:

 - users creation if not exist;
 - dirs creation and ownership/ACLs set;
 - man/doc/examples installation;
 - shell completion generation;
 - locode DB binary download

Using:
	make debpackage

Signed-off-by: Dmitriy Zabolotskiy <d.zabolotskiy@yadro.com>
2022-10-28 12:58:32 +03:00
Evgenii Stratonikov
846ff515e6 [#1812] policer: Do not remove copies if there are maintenance nodes
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-26 19:13:17 +03:00
Evgenii Stratonikov
280e56f4bb [#1893] neofs-node: Do not fail unless all gRPC endpoints are unavailable
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-26 12:48:38 +03:00
Evgenii Stratonikov
1e6588e761 [#1974] shard: Do not panic in degraded mode
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-26 12:41:12 +03:00
Evgenii Stratonikov
7395ab8ef7 [#1944] metabase: Assume static buckets are created on Init
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-26 12:41:12 +03:00
Evgenii Stratonikov
e4bc3d0e9d [#1944] metabase: Recreate static buckets instead of resetting
From the `Bucket.ForEach` doc:
```
The provided function must not modify the bucket; this will result in undefined behavior.
```

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-26 12:41:12 +03:00
Leonard Lyubich
3c6daa2995 [#1883] cli/shards: Exclude degraded-read-write mode from help
The mode is not expected to be used by the user. It is supported only
for developers.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-26 08:24:40 +03:00
Leonard Lyubich
eb206d6e59 [#1883] config/example: Actualize desc of shard modes
Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-26 08:24:40 +03:00
Leonard Lyubich
c4a2a283ae [#1883] cli/shards: Make supporting new modes easier
Add new shard modes as a map entry to automatically parse them in
`set-mode` command. The change also automatically adds new modes to help
message.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-26 08:24:40 +03:00
Evgenii Stratonikov
3b939d190c [#1957] engine: Move shard to read-only if cannot move to degraded
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-26 08:20:53 +03:00
Leonard Lyubich
7f54462e60 [#1920] cli: Print used RPC timeout in verbose exec
Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-26 08:20:36 +03:00
Pavel Karpy
9a0824766c [#1713] node: Add Neo RPC switch interval to the config
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-25 17:38:48 +03:00
Pavel Karpy
0631b38da1 [#1713] ir: Add Neo RPC switch interval to the config
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-25 17:38:48 +03:00
Pavel Karpy
d240e2dc87 [#1713] morph: Allow switch interval configuration
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-25 17:38:48 +03:00
Pavel Karpy
d09ec2e3fe [#1615] node/config: Remove unused default constants
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-25 17:38:48 +03:00
Pavel Karpy
17f7d0a2ee [#1615] morph: Switch to a more prioritized RPC node
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-25 17:38:48 +03:00
Pavel Karpy
7c0aa69d11 [#1615] morph: Fix the switch comment message
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-25 17:38:48 +03:00
Evgenii Stratonikov
713fdab177 [#1907] shard: Return from Close after GC has stopped
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-25 11:54:45 +03:00
Evgenii Stratonikov
4fe1aad30f [#1966] neofs-adm: Fix signature creation for multiactor scenario
`SignTx` from neo-go appended verification script only if `Scripts`
array was not completely filled.

```
can't persist transactions: transaction b9cd02fb81721c7316c908687301eeaf8ef384e381476632c24a77bdf9ffacd5 failed to verify: witness #1: unknown verification contract
```

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-25 11:22:00 +03:00
Evgenii Stratonikov
2dcf8d51be [#1966] neofs-adm: Set VerifyBlocks: true and fix NetworkFee calculation
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-25 11:22:00 +03:00
Pavel Karpy
41f93dcc1d [#1953] blobstor: Log any object removal operation
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-24 21:52:18 +03:00
Pavel Karpy
942cc38146 [#1953] engine: Add refactored storage helpers
After the refactor there are new storage characteristics: a type and
a general storage id (that could be stringified).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-24 21:52:18 +03:00
Pavel Karpy
0371d15b2f [#1953] shard: Fix debug log messages
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-24 21:52:18 +03:00
Pavel Karpy
f8180447a1 [#1938] meta: Make version error messages more descriptive
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-24 21:46:18 +03:00
Evgenii Stratonikov
f396ffcc0f [#1959] CHANGELOG.md: Update neofs-adm remarks
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
6c9423cfb0 [#1959] neofs-adm: Add tests for more commands
Also, fix `restore-containers`, but it is hard to test it now.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
7c3ac7150d [#1959] neofs-adm: Remove newTempDir function
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
2404a267b9 [#1959] neofs-adm: Move testdata generation to a separate function
Later we could make it a separate tests an run neofs-adm tests on CI.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
6c21e2cc28 [#1959] neofs-adm: Fix set-config for >4 nodes
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
c87c3d3721 [#1959] neofs-adm: Refactor tests
Make it easier to test both common deployment scenarios.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
02bea52f40 [#1959] neofs-adm: Allow to test all commands with a local dump
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
8648b102db [#1959] neofs-adm: Dump all blocks in file
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
23622ed283 [#1959] neofs-adm: Check transaction validity in tests
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
942155be6e [#1959] neofs-adm: Fix double multisig
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
2484c1d35e [#1959] neofs-adm: Verify blocks and transactions in tests
Also update to the actual contracts version.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
926830bb9c [#1961] network/cache: Do not return nil client
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 17:56:34 +03:00
Leonard Lyubich
a4573d5026 [#1933] cli: Document sessions
Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-24 17:45:22 +03:00
Leonard Lyubich
7b418c36b4 [#1930] services/session: Log calling Create RPC
There is a need to check if session is opened during system
testing/debug.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-24 17:45:22 +03:00
Leonard Lyubich
8bba490c30 [#1933] cli: Support binary sessions
There is a need to support NeoFS-binary sessions along with JSON ones in
NeoFS CLI.

Provide generic `common.ReadBinaryOrJSON` functions which tries to
decode NeoFS-binary structure and falls back to JSON format. Use this
function in all places with token reading.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-24 17:45:22 +03:00
Evgenii Stratonikov
04b67f3ba5 [#1949] neofs-adm: Fix epoch tick for >4 node committee
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 13:36:08 +03:00
Evgenii Stratonikov
8796807040 [#1936] network/cache: Handle dial errors
After an SDK update `Dial` can return an error.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 13:28:53 +03:00
Evgenii Stratonikov
a189eca5d5 [#1908] docs: Describe SIGHUP behaviour
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 13:28:26 +03:00
Evgenii Stratonikov
1beafea0b5 [#1869] shard: Add logs for SetMode operations on reload
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 13:28:26 +03:00
Evgenii Stratonikov
87be4f1629 [#1869] shard: Restore shard mode on failed reloads
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 13:28:26 +03:00
Evgenii Stratonikov
c785e11b20 [#1869] shard: Allow to reload metabase on SIGHUP
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 13:28:26 +03:00
Evgenii Stratonikov
f769fc83fc [#1869] shard: Embed gcCfg as raw struct
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 13:28:26 +03:00
Leonard Lyubich
60e9de8d63 [#1916] control: Check maintenance allowance on Control server
In previous implementation turning to maintenance mode using NeoFS CLI
required NeoFS API endpoint. This was not convenient from the user
perspective. It's worth to move networks settings' check to the server
side.

Add `force_maintenance` field to `SetNetmapStatusRequest.Body` message
of Control API. Add `force` flag to `neofs-cli control set-status`
command which sets corresponding field in the requests body if status is
`maintenance`. Force flag is ignored for any other status.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-24 09:20:24 +04:00
Leonard Lyubich
c50603494b [#1916] cli/control: Pre-check maintenance mode in set-status
Node's maintenance state can be set only if the network allows it.

Make `neofs-cli control set-status` to read current network settings and
check is maintenance state is allowed. Fail the execution if the mode is
not allowed.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-24 09:20:24 +04:00
Pavel Karpy
e1be0180f6 [#1329] tree: Sync trees when a node first time appears in a container
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-21 18:47:56 +03:00
Leonard Lyubich
217b030d20 [#1946] cli/internal: Fix test by wallet non-closing
Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-21 15:21:59 +03:00
Leonard Lyubich
810087d8b9 [#1946] Bump neo-go module to v0.99.4
Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-21 15:21:59 +03:00
Evgenii Stratonikov
de055f27c7 [#1942] neofs-adm: Fix multisig checks in tests
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-21 10:16:51 +03:00
Pavel Karpy
7d456cb4d0 [#1902] Update CHANGELOG
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
49c38d642d [#1902] engine: Search for the tree IDs in every shard
Iterate over every shard and search for the container's trees. Final result
is a concatenation of shards' results. It is considered that one fixed tree
is placed on one fixed shard but the different trees of a fixed container
could be placed on different shards.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
aa37078570 [#1902] node: Sync all the trees on bootstrap
Do not limit synchronization by "system" and "version" tree IDs.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
1766ca2039 [#1902] tree: Allow synchronize all the container trees
Add `SynchronizeAllTrees` method of the Tree service. It allows fetching
tree IDs and sync all of them. Share common logic b/w the new method and
the `SynchronizeTree`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
1805c6606d [#1332] cli: Add tree list
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
6d4beea187 [#1902] tree: Extend grpc service with ListTrees method
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
24e9e3f3bf [#1902] engine, shard: Implement TreeList method
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
2e199c7ab1 [#1902] shard: Fix pilorama disabled err message
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
19850ef157 [#1902] pilorama: Add TreeList method
To both `bolt` and `memory` forests; extend `Forest` interface.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
3aa9938b8f [#1902] Update protoc to v3.21.7
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Evgenii Stratonikov
9ec01bb9c1 [#1931] control: Allow to clear errors in SetShardMode RPC
It hasn't been working since the initial implementation 7fb15fa1d0.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-20 15:51:31 +03:00
Evgenii Stratonikov
a95fad833e [#1909] docs: Add shard modes description
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-20 08:53:58 +03:00
Evgenii Stratonikov
0d14ef69f0 [#1922] neofs-node: Allow to go online after maintenance
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-19 10:14:56 +03:00
Evgenii Stratonikov
1980ed968a [#1926] neofs-node: Use writecache.max_object_size from the config
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-18 16:44:28 +03:00
Evgenii Stratonikov
13e818e2ca [#1910] .github: Update golangci-lint version
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-18 15:08:26 +03:00
Evgenii Stratonikov
0d65888005 [#1910] .golangci.yml: Add predeclared linker
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-18 15:08:26 +03:00
Evgenii Stratonikov
1cb892c579 [#1910] .golangci.yml: Add misspell linker
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-18 15:08:26 +03:00
Evgenii Stratonikov
9e46799cd4 [#1910] .golangci.yml: Add more useful linkers
Steal them from the neo-go repo.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-18 15:08:26 +03:00
Evgenii Stratonikov
d772e35aba [#1910] .golangci.yml: Add godot linker
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-18 15:08:26 +03:00
Evgenii Stratonikov
347912ea0b [#1910] .golangci.yml: Move unused to a list of default linters
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-18 15:08:26 +03:00
Evgenii Stratonikov
0a1530afa0 [#1917] neofs-cli: Support timeout in control commands
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-18 10:58:10 +03:00
Pavel Karpy
4c63be6629 [#1329] Update CHANGELOG
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-17 20:53:34 +03:00
Pavel Karpy
bc67f77d86 [#1329] tree: Fix sync applying
Add the node position in a container and the container size to the CID
descriptor that is passed to the `TreeApply`. Previously, `checkValid` does
not allow any log operarations.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-17 20:53:34 +03:00
Pavel Karpy
c16a31ef3e [#1329] node: Set undefined netmap status by default
It allows general usage of `controlNetmapStatus` getter: does not matter of
handler (`NewEpoch` handler of initial setter).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-17 20:53:34 +03:00
Pavel Karpy
f4a3fa2977 [#1329] tree: Sync tree on startup
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-17 20:53:34 +03:00
Pavel Karpy
f5f560d903 [#1329] tree: Do not sync trees that are from external containers
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-17 20:53:34 +03:00
Pavel Karpy
ea9a5690cb [#1918] Fix v0.33.0 CHANGELOG record
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-17 20:42:32 +03:00
Evgenii Stratonikov
e2970bf892 Release v0.33.0
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-17 19:09:42 +03:00
Evgenii Stratonikov
b40dd10b7a [#1913] go.mod: Update neofs-api-go to v2.14.0
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-17 19:09:42 +03:00
Evgenii Stratonikov
f524c812c1 [#1913] go.mod: Update neo-contract to v0.16.0
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-17 19:09:42 +03:00
Leonard Lyubich
d96139b201 [#1900] node: Do not override maintenance state on bootstrap
In previous implementation node always sent bootstrap requests with
"online" network state. This provoked switching to online of the nodes
under maintenance.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-17 18:59:22 +03:00
Pavel Karpy
f76516a883 [#1860] meta: Fix 0,1 -> 2+ version migration
In the 2nd version, there was a database format change: buckets have changed
their keys, so it becomes impossible to check the version in the 1 -> 2+
migrations because of different buckets that store info about the version.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-17 13:42:05 +03:00
Leonard Lyubich
6eb82c4cbf [#1890] docs: Describe maintenance mode's behavior
Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-17 13:40:56 +03:00
Evgenii Stratonikov
8145e79d7f [#1903] neofs-adm: Expand update parameters in MakeCall
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-17 13:37:48 +03:00
anastasia prasolova
bcf3df354b Add CODEOWNERS file
Signed-off-by: anastasia prasolova <anastasia@nspcc.ru>
2022-10-16 21:09:07 +03:00
Leonard Lyubich
1406d096a2 [#1680] service/object: Fail all operations in maintenance mode
Storage node should not provide NeoFS Object API service when it is
under maintenance.

Declare `Common` service that unifies behavior of all object operations.
The implementation pre-checks if node is under maintenance and returns
`apistatus.NodeUnderMaintenance` if so. Use `Common` service as a first
logical processor in object service pipeline.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-14 13:54:32 +04:00
Leonard Lyubich
05420173cc [#1894] services/object: Ignore unrelated sessions in client
In some scenarios original session can be unrelated to the objects which
are read internally by the node. For example, node requests child
objects when removing the parent one.

Tune internal NeoFS API client used by node's Object API server to
ignore unrelated sessions in `GetObject` / `HeadObject` / `PayloadRange`
ops.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-14 12:37:31 +03:00
Evgenii Stratonikov
2050a623ac [#1896] neofs-node: Set MAINTENANCE status on update
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-14 11:46:19 +03:00
Pavel Karpy
ade19077ee [#1332] Update Changelog
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-13 20:01:48 +03:00
Pavel Karpy
beb1998ed1 [#1332] cli: Print missing nodes message verbosely
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-13 20:01:48 +03:00
Pavel Karpy
6e4f7180fe [#1332] cli: Implement tree add-by-path command
It is `TreeService.AddNodeByPath` method implementation.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-13 20:01:48 +03:00
Pavel Karpy
3d8349d7f8 [#1332] cli: Implement tree get-by-path command
It is `TreeService.GetNodeByPath` method implementation.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-13 20:01:48 +03:00
Pavel Karpy
2c02e66939 [#1332] cli: Implement tree add command
It is `TreeService.Add` method implementation.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-13 20:01:48 +03:00
Pavel Karpy
13c4a9f4b8 [#1332] tree: Make SignMessage public
It will allow reusing signing routine in other components
(e.g. `neofs-cli`).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-13 20:01:48 +03:00
Evgenii Stratonikov
80d3c7f9d6 [#1892] neofs-adm: Support MaintenanceModeAllowed network setting
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-13 17:45:47 +03:00
Evgenii Stratonikov
f2793060c5 [#1892] neofs-cli: Display maintenance status in netmap nodeinfo
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-13 17:45:47 +03:00
Pavel Karpy
31c623636d [#1863] node: Fix shard id in the object counter metrics
If shard ID is stored in metabase (it is not the first time boot), read it,
set it, use it (not a generated one) in the metrics writer.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-13 13:06:41 +03:00
8714fc42b5 [#1765] Use hex format to print storage node ID
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-10-13 12:55:21 +03:00
Pavel Karpy
5a2daadd37 [#1770] Update Changelog
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-12 18:11:05 +03:00
Pavel Karpy
b6806ea6b9 [#1770] node: Support logger config rereading
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-12 18:11:05 +03:00
Pavel Karpy
8c75cb1dad [#1770] node: Validate logger config section
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-12 18:11:05 +03:00
Pavel Karpy
d7c7022bbd [#1770] logger: Support runtime level reloading
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-12 18:11:05 +03:00
Pavel Karpy
f037022a7a [#1770] logger: Refactor Logger component
Make it store its internal `zap.Logger`'s level. Also, make all the
components to accept internal `logger.Logger` instead of `zap.Logger`; it
will simplify future refactor.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-12 18:11:05 +03:00
Evgenii Stratonikov
4baf00aa21 [#1884] services/object: Fallback to GET in GET_RANGE
Current spec allows denying GET_RANGE requests from other storage nodes.
However, GET should always be allowed and it is enough to perform
GET_RANGE locally

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-12 17:05:51 +03:00
Leonard Lyubich
dde4d4df2a [#1878] services/object: Fix child check in GET
In previous implementation `ObjectService.Get` RPC handler failed with
`parent address in child object differs` while assembling the "big"
object. This was caused by the child check which required parent
reference to be set in all child objects. The check was impracticable
because not all elements of the split-chain have a link to the parent.

Make `execCtx.isChild` to return `true` if parameterized object has no
parent header in its own header.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-12 16:56:37 +03:00
Leonard Lyubich
9a006ac14f [#1865] cli/object: Do not open remote sessions in reading ops
It does not make sense to open remote sessions with the storage node in
`get`, `head`, `search`, `range` and `hash` sub-commands of `neofs-cli
object` command.

Do not use NeoFS API `SessionService` in mentioned commands. Decode
object session from JSON file specified `--session` flag. Perform some
sanity checks instantly on CLI side.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-12 13:22:19 +03:00
Evgenii Stratonikov
49eab6318c [#1867] control: Fix degraded-read-only mode parsing
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-12 11:20:48 +03:00
Evgenii Stratonikov
c0199dee93 [#1867] services/control: Interpret empty list of IDs as all shards
In neofs-cli the flag is still required, but `all` can be used to
process all shards.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-12 11:20:48 +03:00
Evgenii Stratonikov
b632260995 [#1867] neofs-cli: Support multiple shard IDs in control subcommands
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-12 11:20:48 +03:00
Evgenii Stratonikov
19c0a74e94 [#1867] services/control: Allow to provide multiple shard IDs to some commands
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-12 11:20:48 +03:00
74d2f2c8d3 [#1854] cli: Unify help messages
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-10-12 09:53:11 +03:00
Pavel Karpy
9e2edfedc6 [#1833] node: Update local node info on nil updates
If the contract returns a netmap that does not contain the node, update
local `NodeInfo`. It fixes `neofs-cli netmap nodeinfo` command that printed
"state: online" previously.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-12 09:52:51 +03:00
Evgenii Stratonikov
c5fdb7bedf [#1876] .github: Add an action monitoring config updates
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-12 09:33:28 +03:00
Leonard Lyubich
050ad2762c [#1680] replicator: Consider NODE_UNDER_MAINTENANCE as OK
Node response with `NODE_UNDER_MAINTENANCE` status signals that the node
was switched to maintenance mode. There is a delay between the actual
switch and the reflection in the network map of up to one epoch. To
speed up the reaction to the maintenance, it is required to recognize
such node responses in the Policer.

Make `Policer.processNodes` to exclude elements with shortage decreasing
on `NODE_UNDER_MAINTENANCE` status response.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-11 12:54:27 +03:00
Leonard Lyubich
e99e25b52f [#1680] replicator: Consider nodes under maintenance as OK
Nodes under maintenance SHOULD not respond to object requests. Based on
this, storage node's Policer SHOULD consider such nodes as problem ones.
However, to prevent spam with the new replicas, on the contrary, Policer
should consider them normal.

Make `Policer.processNodes` to exclude elements if `IsMaintenance()`
with shortage decreasing.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-11 12:54:27 +03:00
Leonard Lyubich
df5d7bf729 [#1680] replicator: Work with netmap.NodeInfo in TaskResult
Make `replicator.TaskResult` to accept `netmap.NodeInfo` type instead of
uint64 in order to clarify the meaning and prevent passing the random
numbers.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-11 12:54:27 +03:00
Leonard Lyubich
e6f8904040 [#1680] policer: Refactor tracking the processed nodes
Add clear methods with docs. Use the methods instead of direct map
and bool instructions.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-11 12:54:27 +03:00
feaa9eace7 [#1768] Add healthcheck method to the Tree service
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-10-11 12:53:45 +03:00
Leonard Lyubich
807c0a1321 [#1859] services/object: Do not session check relation in PUT
It doesn't make sense to check object relation in session check of
`ObjectService.Put` RPC which has been spawned by `ObjectService.Delete`
with session. Session issuer can't predict identifier of the tombstone
object to be created.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-10 20:09:47 +03:00
Evgenii Stratonikov
2d43892fc9 [#1840] neofs-node: Use blobstor paths to identify shard
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-10 11:14:55 +03:00
Evgenii Stratonikov
4b005d3178 [#1840] blobstor: Return info about all components
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-10 11:14:55 +03:00
Evgenii Stratonikov
6d7ffefec5 [#1840] blobstor/test: Add tests for auxiliary functions
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-10 11:14:55 +03:00
Evgenii Stratonikov
328691c94f [#1840] blobstor: Return Path from components
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-10 11:14:55 +03:00
Evgenii Stratonikov
9b241e4a17 [#1840] neofs-node: Allow to use mode: disabled in config
Currently, when removing shard special care must be taken with respect
to shard numbering. `mode: disabled` allows to leave shard configuration
in place while also ignoring it during initialization. This makes
disk replacement much more convenient.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-10 11:14:55 +03:00
Pavel Karpy
1360273fec [#1856] node: Fix config rereading
Config was reread from the file only once in two SIGHUPs.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-10 10:35:52 +03:00
Evgenii Stratonikov
b2aa9947c2 [#1829] engine: Delete split objects properly
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-07 16:35:46 +03:00
5e493b7f1c [#1704] Add command container nodes to output list of nodes for container, grouped by replica (#1704)
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-10-07 14:58:01 +03:00
Evgenii Stratonikov
6557f5d249 [#1839] engine: Handle Inhume errors properly
If shard is in read-only or degraded mode, there is no need to increase
error counter.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-07 14:25:52 +03:00
Evgenii Stratonikov
7356ee91ff [#1837] services/object: Optimize uniqueIDWriter
Avoid encoding object ID to string.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-07 13:29:15 +03:00
Evgenii Stratonikov
2ac42b70ce [#1826] services/object: Parallelize object search
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-07 13:29:15 +03:00
Evgenii Stratonikov
90bfe0bad9 [#1826] neofs-cli: Add --timeout flag
Allow to specify it everywhere `--rpc-endpoint` flag is present.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-07 13:29:15 +03:00
Evgenii Stratonikov
0cb8e7f6f1 [#1837] services/object: Fix log messages
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-07 13:29:15 +03:00
Evgenii Stratonikov
ca8dc872b2 [#1846] neofs-node: Make morph.cache_ttl equal to block time by default
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-07 10:34:55 +03:00
Leonard Lyubich
e54b52ec03 [#1420] object/acl: Fix correlation of object session to request
In previous implementation of `neofs-node` app object session was not
checked for substitution of the object related to it. Also, for access
checks, the session object was substituted instead of the one from the
request. This, on the one hand, made it possible to inherit the session
from the parent object for authorization for certain actions. On the
other hand, it covered the mentioned object substitution, which is a
critical vulnerability.

Next changes are applied to processing of all Object service requests:
 - check if object session relates to the requested object
 - use requested object in access checks.

Disclosed problem of object context inheritance will be solved within

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-07 10:34:38 +03:00
Evgenii Stratonikov
5834f9807e [#1847] services/policer: Provide container ID in logs
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-07 09:58:16 +03:00
Evgenii Stratonikov
c20c2e3b39 [#1845] morph/client: Use 0 as OfflineState in update peer
As the documentation prescribes.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-06 12:26:25 +03:00
Pavel Karpy
01ddb3f8e6 [#1770] node: Refactor application configuration
Split all the fields in `cfg` structure on:
1. `applicationConfiguration`;
2. `internals`; // shared entities for an application work, such as
`context.Context`
3. `shared`; // holder for the shared entities b/w;
4. `cfgXXX`; // configuration for internal services.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-05 14:25:33 +04:00
Leonard Lyubich
713aea06fa [#1681] node: Block only Object service ops under maintenance
In previous implementation node blocked any operation of local object
storage in maintenance mode. There is a need to perform some storage
operations like data evacuation or restoration.

Do not call block storage engine in maintenance mode. Make all Object
service operations to return `apistatus.NodeUnderMaintenance` error from
each local op.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
082602b668 [#1680] Update changelog
Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
a7668618c9 [#1680] cli/netmap: Support MAINTENANCE mode
Make `netmap snapshot` command to print `MAINTENANCE` state of the nodes
with `IsMaintenance()` flag set.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
eb1fba5182 [#1680] morph/netmap: Adopt to recent contract changes
After recent Netmap contract changes all read methods which return
network map (either candidates or snapshots) encode node descriptors
into same structure.

Decode `netmap.Node` contract-side structure from the call results.
Replace node state with the value from the `netmap.Node.State` field.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
d6c01199c8 [#1680] node: Return MAINTENANCE status in corresponding mode
Make storage node to return `NODE_UNDER_MAINTENANCE` status
error on each local object operation if the node is in `MAINTENANCE`
mode.

Pass `apistatus.NodeUnderMaintenance` to `StorageEngine.BlockExecution`
during `ControlService.SetNetmapStatus` RPC processing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
9fcc80ea38 [#1680] node/control: Send request to switch to MAINTENANCE state
After recent changes `MAINTENANCE` state is reflected in the Sidechain.
Storage node should switch its state to "maintenance" during serving the
`ControlService.SetNetmapStatus` RPC with correspoding status in the
request.

Call `UpdatePeerState` operation of Netmap contract's client in
`control.NodeState` provider on Storage node app side. The op is
executed if `BlockExecution` on local object storage is succeeded.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
6a2ec21641 [#1681] ir/netmap: Check MAINTENANCE state in UpdateState
Storage node can be requested to be switched into `MAINTENANCE` state.
Inner Ring should accept such requests only if network configuration
allows it.

Make `Processor` of Netmap contract's notifications to depend on
`state.NetworkSettings`. Make `Processor.processUpdatePeer` to call
`MaintenanceModeAllowed` if notification event relates to `MAINTENANCE`
mode`. Share singe `state.NetworkSettings` provider in Inner Ring
application.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
bdb8243a5a [#1681] morph/netmap: Support MAINTENANCE state notification
After recent changes Netmap contract can send `UpdateState` notification
event with `MAINTENANCE` node's state. There is a need to provide
functionality to work with the status.

Provide `UpdatePeer.Maintenance` method. Support new state in
`ParseUpdatePeer` and `ParseUpdatePeerNotary` functions.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
0903b2af93 [#1680] morph/netmap: Allow to set MAINTENANCE state
After recent changes in NeoFS protocol storage nodes can be in
`MAINTENANCE` state. There is a need to support this state in
`UpdateState` operation.

Add `UpdatePeerPrm.SetMaintenance` method which makes node to be
switched into `MAINTENANCE` mode after the `UpdatePeerState` operation.

New functionality is going to be used in Storage node application for
Control API serving.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
6b99f2a0df [#1680] ir/netmap: Correlate node's state with network settings
Inner Ring should allow registering of storage nodes with `MAINTENANCE`
state in the NeoFS network only if its configuration allows this status.

Make `networkSettings.MaintenanceModeAllowed` to call
`MaintenanceModeAllowed` method of underlying Netmap contract's client
in order to assert state allowance.

From now nodes will be accepted to the network with `MAINTENANCE` state
only with the appropriate network configuration.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
876e132d22 [#1680] cli/netmap: Print MaintenanceModeAllowed config
After recent changes network configuration provided by NeoFS storage
nodes contains `MaintenanceModeAllowed` flag. There is
a need to support this value in NeoFS CLI application.

Print `MaintenanceModeAllowed` flag in `netmap netinfo` command.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
7a3a827d35 [#1680] node/netmap: Support MaintenanceModeAllowed config in RPC
After recent changes network configuration stored in the Netmap contract
of the NeoFS Sidechain contains `MaintenanceModeAllowed` flag. There is
a need to support this value in Storage node application.

Make `NetmapService.NetworkInfo` RPC server of the storage node to set
`MaintenanceModeAllowed` flag according to corresponding value in the
Sidechain.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
0826310b2a [#1681] morph/netmap: Support MaintenanceModeAllowed config
After recent changes in the NeoFS API protocol network configuration
contains `MaintenanceModeAllowed` boolean flag. There is a need to
support the config value in all NeoFS applications.

Provide `Client.MaintenanceModeAllowed` method which read the config
from the Sidechain. Extend `NetworkConfiguration` structure with
`MaintenanceModeAllowed` field.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
d2d4191868 [#1680] morph/netmap: Support HomomorphicHashingDisabled config
`NetworkConfiguration` represents NeoFS network configuration stored in
the Sidechain. In previous implementation the configuration missed flag
of disabled homomorphic hashing.

Add `NetworkConfiguration.HomomorphicHashingDisabled` boolean field.
Decode the field in `Client.ReadNetworkConfiguration` method. Print this
value in `netmap netinfo` command of NeoFS CLI.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
e0194dbde5 [#1681] morph/netmap: Refactor reading the boolean configurations
`readBoolConfig` method is going to be reused for reading other
configuration values. All boolean settings are `false` by default, so it
makes sense to return default value on missing key directly from
`readBoolConfig`.

Handle `ErrConfigNotFound` case in `readBoolConfig` method. Change
`HomomorphicHashDisabled` method to call `readBoolConfig` directly.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
f64ae55806 [#1681] ir/netmap: Require MAINTENANCE mode to be allowed by network
There is a need to prevent limitless abuse of MAINTENANCE status of the
storage nodes. To do this, configuration of the NeoFS network is going
to be extended with the flag which allows the state. Until this is done,
it makes sense to prepare a site for this in the code.

Define `state.NetworkSettings` interface as an abstraction of global
network configuration within the `state` package. Make
`NetMapCandidateValidator` to depend on `NetworkSettings` and provide
corresponding field setter. Change `VerifyAndUpdate` method's behavior
to return an error for candidates with MAINTENANCE state if this state
is disallowed by the network configuration. Provide `NetworkSettings`
from the wrapper over Netmap contract's client on Inner Ring application
side. The provider is implemented to statically disallow MAINTENANCE
mode in order to save previous behavior.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
42fb40e841 [#1680] morph/netmap: Pre-refactor processing of node states
New network status of storage nodes is going to be introduced. To
simplify the addition, it would be useful to prepare the code for this.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
4c94faac67 [#1797] node/control: Verify states in SetNetmapStatus RPC server
In previous implementation storage node interpreted all status values
sent in `SetNetmapStatus` RPC as `OFFLINE` except `ONLINE` and
`MAINTENANCE`. This could lead to incorrect processing of new values,
and also didn't allow detection of problems with sending garbage values.

Make implementation of `NodeState` interface used by Control API server
to deny requests with statuses other than protocol-declared enum.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
8858840751 [#1796] ir/netmap: Allow to call AddPeer with ONLINE state only
In previous implementation Inner Ring allowed storage nodes with any
state to register in the network. According to the current design, only
nodes with ONLINE state are allowed to enter the network map.

Create new `state` sub-package of `nodevalidation` package of Inner Ring
application. Define `state.NetMapCandidateValidator` type and provide
`NodeValidator` interface required by the Inner Ring's processor of
`Netmap` contract's notification events. Embed new validator into the
one used by the Inner Ring application.

From now all `AddPeer` notifications with node state other than `ONLINE`
will be denied.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
8bf82d738b [#1704] cli: Add force option to the command container create
Validate policy before container creation

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-10-05 09:14:13 +03:00
Pavel Karpy
4eb0ed11f8 [#1809] node: Do not boot up if metabase is outdated
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 12:32:10 +03:00
Evgenii Stratonikov
8b3b16fe62 [#1825] writecache: Flush cache when moving to the DEGRADED mode
Degraded mode allows us to operate without an SSD,
thus writecache should be unavailable in this mode.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-04 12:13:09 +03:00
Evgenii Stratonikov
236414df49 [#1817] network: Allow to use network addresses from the iterator
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-04 12:12:18 +03:00
Evgenii Stratonikov
76893bdc50 [#1817] go.mod: Update neofs-sdk-go
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-04 12:12:18 +03:00
Evgenii Stratonikov
2e3ef817f4 [#1819] engine: Increase error counter for PUT errors
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-04 10:11:52 +03:00
Evgenii Stratonikov
b89e71fa78 [#1819] common: Add ErrNoSpace
Add a common error for this case because it is not an error
which should increase error counter. Single error simplifies checks on
the call-site.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-04 10:11:52 +03:00
Evgenii Stratonikov
af56574849 [#1819] engine: Fix error counter in Inhume
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-04 10:11:52 +03:00
Pavel Karpy
1e35c12cc1 [#1770] node: Validate config before apply it
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
8ebe95747e [#1770] node: Do not lock on shard's Close call
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
ab3ef7110e [#1770] node: Reread config files on SIGHUP
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
6fc3268ebf [#1770] node: Support configuration reread on SIGHUP
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
887afeaddb [#1770] engine: Do not lock on shard init
Init can take a lot of time. Because the mutex is taken, all new operations
are blocked.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
fbd5bc8c38 [#1770] engine: Support configuration reload
Currently, it only supports changing the compound of the shards.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
91b56ad3e8 [#1770] node: Read storage config in a separate struct
It will allow rereading config values and will simplify distinguishing them
from the custom values in the `cfg` structure.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
5c69e19016 [#1770] fstree: Depth parameter int -> uint64
Negative values have no sense. On the other hand it differs from the
blobovnicza's configuration and prevents unification.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
4aa4694152 [#1770] node: Do not init local storage in reading config routine
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
2d7166f8d0 [#1770] shard: Move NewEpoch event routing on SE level
It will allow dynamic shard management. Closing a shard does not allow
removing event handlers.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
9374823950 [#1770] node: Do not create meta dir on config read
It is created in `Open` anyway.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
76cfcc242c [#1820] neofs-adm: Add wallet-address flag in refill command
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-10-03 10:11:38 +03:00
Evgenii Stratonikov
1edc048870 [#1697] services/object: Return proper error if session token is missing
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-30 16:13:01 +03:00
5284ac53f9 [#1821] neofs-cli: Fix description of cli commands
1. nodeinfo - updated description of node.
2. eacl - fixed misprint.

Signed-off-by: Vladimir Domnich <v.domnich@yadro.com>
2022-09-28 16:09:24 +03:00
Evgenii Stratonikov
0a411908ee [#1806] writecache: Allow to ignore read errors during flush
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
f2045c10d7 [#1806] shard: Check each component mode when setting mode
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
1c62f1b2c4 [#1806] neofs-cli: Add control flush-cache command
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
51b8f26a31 [#1806] services/control: Allow to flush write-cache
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
3d882e9f47 [#1806] engine: Allow to flush write-cache
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
0b4c867ef1 [#1806] writecache: Allow to start flush manually
Allow user to initiate flushing objects from a writecache.
We need this in 2 cases:
1. During writecache storage schema update, it should be flushed with
   the old version of node and started clean with a new one.
2. During SSD replacement, to avoid data loss.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
55148404ee [#1806] writecache: Use interfaces for blobstor and metabase
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
36ab1a2472 [#1731] neofs-cli: Make --endpoint flag optional
Allow to provide it using config file.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
Leonard Lyubich
93742d37b7 [#1793] node/netmap: Change interface of the latest network map reader
Replace `ProcessCurrentNetMap` method of `NodeState` interface with
`ReadCurrentNetMap` one with two changes:
 * Replace network map type from NeoFS SDK package with the
   protocol-generated message. This replaces all the business logic to
   the application layer.
 * Support error return. This allows to cover problem node states.

Return an error from `NodeState.ReadCurrentNetMap` method implemeted
through `atomic.Value` if `Store` method has not been called yet.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-27 09:03:17 +03:00
Leonard Lyubich
485a5418d2 [#1793] node: Serve NetmapService.NetmapSnapshot RPC
There is no more need to serve the same request on Control API.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-27 09:03:17 +03:00
Evgenii Stratonikov
59de20fbba [#1764] neofs-node: Allow to check configuration
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-27 08:57:20 +03:00
Evgenii Stratonikov
0fb5c51ac9 [#1764] neofs-node: Validate config before usage
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-27 08:57:20 +03:00
Evgenii Stratonikov
6f45cc81fc [#1764] neofs-node: Use constants for storage types
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-27 08:57:20 +03:00
Evgenii Stratonikov
9113793688 [#1764] neofs-node: Allow to return error from IterateShards
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-27 08:57:20 +03:00
Evgenii Stratonikov
898689ec14 [#1731] services/replicator: Unify Task interface with other parameters
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-24 13:47:48 +03:00
Evgenii Stratonikov
4e043a801c [#1731] services/control: Replicate object over network in EvacuateShard RPC
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-24 13:47:48 +03:00
Evgenii Stratonikov
a49137349b [#1731] engine: Allow to use user handler for evacuated objects
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-24 13:47:48 +03:00
Pavel Karpy
93ae3f0b19 [#1808] .github: Add changelog workflow
It checks whether the CHANGELOG.md was updated in a just-created PR.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-23 10:20:25 +04:00
Evgenii Stratonikov
7cc68cf4d4 [#1810] blobovniczatree: Add a test for blobovnicva usage
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-23 10:19:26 +04:00
c7f85994e5 [nspcc-dev#1692] cli: Remove --generate-key option in neofs-cli container delete
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-09-20 07:19:46 +04:00
bb02913c39 [nspcc-dev#1128] cli: Remove WIF and NEP2 support in --wallet argument
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-09-19 14:21:37 +03:00
Evgenii Stratonikov
3df98ce7ba [#1731] engine: Return the amount of actually moved objects in Evacuate
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-19 11:33:52 +03:00
Evgenii Stratonikov
8fc88487db [#1731] neofs-cli: Add control shards evacuate command
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-19 11:33:52 +03:00
Evgenii Stratonikov
091d7d30f6 [#1731] services/control: Allow to evacuate data from shard
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-19 11:33:52 +03:00
Evgenii Stratonikov
a51b76056e [#1731] engine: Add Evacuate command
Make it possible to move all data from 1 shard to other shards.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-19 11:33:52 +03:00
Evgenii Stratonikov
7377979e12 [#1731] engine: Move single shard PUT to a separate function
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-19 11:33:52 +03:00
Evgenii Stratonikov
4208f7c0cf [#1741] neofs-adm: Use ValidUntilBlock to wait for tx persist
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-15 14:27:16 +03:00
Evgenii Stratonikov
5321f8ef9c [#1786] engine: Unify parameter setters
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-15 10:28:48 +03:00
Evgenii Stratonikov
b064fb24d8 [#1616] engine: Do not use batches in delete
Use a simple loop at the callsite. This way we remove as much as we can.
Also, `Delete` metrics is more meaningful now.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-15 10:28:48 +03:00
Evgenii Stratonikov
cda8f9df2e [#1786] services/control: Remove WithDeletedObjectHandler option
Use storage engine directly instead. It is already provided in the
options.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-15 10:28:48 +03:00
Evgenii Stratonikov
ae1dab29bc [#1482] metabase: Encode database keys in binary
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-15 10:28:26 +03:00
Leonard Lyubich
d6fef68a62 Release v0.32.0
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-14 17:52:49 +04:00
Leonard Lyubich
d895a272ad docs: Improve release instructions
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-14 17:52:49 +04:00
Leonard Lyubich
ae655b74f0 *: Exec make fmts
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-14 17:52:49 +04:00
Evgenii Stratonikov
4b8b575ca4 [#1782] go.mod: Update neofs-sdk-go
Add fixes for key parameter in `object.Get` operation.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-14 13:18:19 +04:00
Pavel Karpy
532c41ca05 [#1658] meta: Do not check object expiration in counters
Do not perform operations that produce unused results in `syncCounter`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy
431e331373 [#1658] shard: Update metric counters
Use meta's operation results to change the metrics. Support typed object
counters.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy
ad47e2a985 [#1658] meta: Add logic counter
- Meta now supports (and requires) inc/dec labeled counters
- The new logic counter is incremented on `Put` operations and is
decremented on `Inhume` and `Delete` operations that are performed on
_stored_ objects only
- Allow force counters sync. "Force" mode should be used on metabase resync
and should not be used on a regular meta start

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy
d872862710 [#1658] meta: Force counters resync process
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy
beb7f2a048 [#1634] node: Change default epoch in tests
Do not treat objects with expiration as expired by default.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy
edef26a4fd [#1658] engine: Update metrics interfaces
It now supports typed object counter metrics.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy
175e2b9fa0 [#1658] metrics: Add object type label to the counters
It will allow to use labeled object counters, e.g.: "phy" for all
the physically stored objects, "logic" for only available objects. Also, it
will allow to add typed (regular, TS, SG, LOCK) counters if needed.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Leonard Lyubich
a789159e5b [#1189] config/example: Mention persistent_sessions.path default
In current implementation storage node app uses in-memory session
storage if `persistent_sessions.path` value is empty/missing in the
config.

Clarify default behavior in `config/example/node.yaml`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-13 21:29:04 +04:00
Leonard Lyubich
d25b7e177b [#1189] node/config: Fix PersistentSessionsConfig.Path method's docs
Method never returns `PersistentStatePathDefault` value, and this is
correct.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-13 21:29:04 +04:00
Pavel Karpy
4083a62679 [#1628] tree: Pre-allocate errors in ACL checks
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 10:33:50 +03:00
Pavel Karpy
5adf089c1d [#1628] tree: Log unacceptable bearer attachment
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 10:33:50 +03:00
Pavel Karpy
2ffcd02ac3 [#1628] tree: Document ACL checks in tree service
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 10:33:50 +03:00
Pavel Karpy
8d0906c6ab [#1628] tree: Skip eACL filters for tree requests
Do not call `CalculateAction` for the eACL checks since it requires object
headers that are meaningless in the tree context.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 10:33:50 +03:00
Pavel Karpy
876e014b5d [#1628] tree: Make ACL checks the same way as for object requests
1. Do not require a request to be signed by the container owner if a
bearer token is missing
2. Do not check the system role since public requests are not expected to
be signed by IR or a container node (unlike the object requests)

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 10:33:50 +03:00
Pavel Karpy
4f18893d9b [#1628] node: Move common EACLSource interface to core pkg
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 10:33:50 +03:00
Pavel Karpy
4afb928ab6 [#1674] *: Expire entities after the expiration epoch
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-12 19:56:59 +04:00
Evgenii Stratonikov
f1572a674b [#1775] go.mod: Update neofs-sdk-go
Make use of fixes for the placement policy parser.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-12 19:10:49 +04:00
Evgenii Stratonikov
d25ec52459 [#1778] neofs-node: Panic if config type is invalid
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-12 17:06:00 +03:00
Evgenii Stratonikov
60b2930417 [#1778] neofs-node: Fix blobovnicza config type
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-12 17:06:00 +03:00
Evgenii Stratonikov
9da5d784cb [#1630] neofs-node: Remove trees on container removal event
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-12 09:54:15 +03:00
Evgenii Stratonikov
87bd49563e [#1761] pilorama: Use batch size of 1 for tests
Improve tests speed by a lot and use more iterations in `ApplyRandom` test.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-12 09:54:15 +03:00
Evgenii Stratonikov
a2bb3a2a96 [#1630] pilorama: Support dropping trees
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-12 09:54:15 +03:00
Leonard Lyubich
699b534416 [#1763] node/netmap: Write log message about parsed NewEpoch event
There is a need to have the ability to track NeoFS timeline on storage
nodes. Epochs tick on notifications receipt, so the most obvious way to
know about received epochs is logging the events.

Wrap `morphEvent.ParseNewEpoch` event parser into function which writes
log message about new epoch number.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-12 09:53:05 +04:00
Leonard Lyubich
3a206b5d7a [#1763] morph/event: Write log message about new notification event
Sometimes it is useful to track sidechain events processed by the node.
For example, we need some easy way to look up for log messages about new
epoch notifications.

Make `subscriber.Subscriber` to log names of all events received from
the sidechain notification channel.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-12 09:53:05 +04:00
Leonard Lyubich
8e8265d5ea [#1707] blobovnicza/get: Iterate over all buckets regardless of config
Re-configuration of Blobovnicza's object size limit must not affect
already stored objects. In previous implementation `Blobovnicza` didn't
see objects stored in buckets which became too big after size limit
re-configuration. For example, lets consider 1st configuration with 64KB
size limit for stored objects. Lets assume that we stored object of 64KB
size, and re-configured `Blobovnicza` with 32KB limit. After reboot
object should be still available, but actually it isn't. This is caused
by `Get` operation algorithm which iterates over configured size ranges
only, and doesn't process any other existing size bucket. By the way,
increasing of the object size limit didn't lead to the problem even in
previous implementation.

Make `Blobovnicza.Get` method to iterate over all size buckets
regardless of current configuration. This covers the described scenario.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-08 12:24:28 +04:00
Pavel Karpy
8e6e89aca3 [#1711] amd: Cache committee actor in the init context
Also simplify the code using `invoker.Invoker`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-08 10:09:44 +03:00
Pavel Karpy
60aa53651b [#1711] adm: Append witnesses only if they are missing
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-08 10:09:44 +03:00
Pavel Karpy
1e57565f6c [#1711] adm: Implement local client's methods
Implement methods that `actor.Actor` requires:
1. `InvokeFunction` -- wrapper over `InvokeScript`
2. `GetVersion` returns default struct
3. `CalculateNetworkFee` copied and simplified from Neo-go server side.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-08 10:09:44 +03:00
Pavel Karpy
4bd8608b37 [#1711] *: Simplify code using neo-go actors
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-08 10:09:44 +03:00
Evgenii Stratonikov
4354359aed [#1746] network: Set timeout for streaming operations
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 16:00:51 +04:00
Evgenii Stratonikov
0140ac354b [#1759] services/tree: Make logs more descriptive
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 12:57:33 +04:00
Evgenii Stratonikov
74c861342e [#1753] services/tree: Add parent ID to the GetNodeByPath response
Currently, you need to use `GetSubTree` to get parent, which seems an
overkill.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 10:52:37 +03:00
Evgenii Stratonikov
8a47f9ba12 [#1753] services/tree: Remove MaxGetSubTreeCount constant
It was used only in `GetOpLog` and ignored on the server side.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 10:52:37 +03:00
Evgenii Stratonikov
9840936a4f [#1753] services/tree: Do not restrict depth in GetSubTree
Previously, the depth was restricted because with BFS the amount of
nodes we have in memory blows up exponentially. With DFS is is linear,
so we can process trees of arbitrary depth.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 10:52:37 +03:00
c4c2840b52 [#1632] neofs-adm: Update autogenerated storage node config
Replace `diable_cache` with `cache_ttl` value.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-09-07 10:34:46 +03:00
944665dd5d [#1632] neofs-node: Update docs
Replace `diable_cache` with `cache_ttl` value

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-09-07 10:34:46 +03:00
Evgenii Stratonikov
e4dcc4d6a9 [#1749] neofs-adm: Register contract hashes in NNS in 2 formats
NNS proposal describes string N3 address format, however we must also
have hex-string for backwards compatibility.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 09:58:05 +03:00
Evgenii Stratonikov
fe0c6db67d [#1749] neofs-adm: Delete NNS records before updating
In case we already have the record and it is invalid, we should
overwrite it instead of having several conflicting records.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 09:58:05 +03:00
Evgenii Stratonikov
0818d8d43a [#1751] .github: Add label checklist to templates
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 09:57:50 +03:00
Pavel Karpy
fa18100489 [#1714] lens: Add open*COMPONENT* funcs
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-06 18:09:18 +04:00
Pavel Karpy
01d7c007aa [#1714] Update CHANGELOG
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-06 18:09:18 +04:00
Pavel Karpy
2132b78aba [#1714] lens: Add meta subcommand
Includes:
1. `inspect`
2. `list-garbage`
3. `list-graveyard`

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-06 18:09:18 +04:00
Pavel Karpy
adff08ad02 [#1714] lens: Separate commands
Separate `inspect` and `list` on `write-cache` and `blobovnicza`
subcommands.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-06 18:09:18 +04:00
Pavel Karpy
387953c1e0 [#1756] make: Fix commit abbrev length in version
Make it uniform across all our repos.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-06 10:53:38 +03:00
Evgenii Stratonikov
fef4f6d155 [#1748] neofs-adm: Allow to dump hashes from a custom zone
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 21:31:32 +03:00
Evgenii Stratonikov
d4ac5bdb97 [#1748] neofs-adm: Allow to parse hashes in multiple formats
NEO NNS proposal uses addresses. We should eventually use the same,
but must stay compatible now.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 21:31:32 +03:00
Evgenii Stratonikov
4ccacb89e8 [#1748] neofs-adm: Set custom contract hashes as N3 address
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 21:31:32 +03:00
Evgenii Stratonikov
11e6f03aa2 [#1748] neofs-adm: Refactor NNS price handling
Make `nnsRegisterDomainScript` simpler.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 21:31:32 +03:00
Evgenii Stratonikov
9290a7e1fe [#1748] neofs-adm: Remove addRecord from nnsRegisterDomainScript
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 21:31:32 +03:00
Evgenii Stratonikov
eb5e1121c5 [#1748] neofs-adm: Move dump-hashes command to a separate file
I have just lost 3 minutes trying to find it.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 21:31:32 +03:00
Leonard Lyubich
e26e70ffcf [#1652] cli/container: Mention new flag in the CHANGELOG
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 16:22:16 +04:00
Leonard Lyubich
33143b18a9 [#1652] cli/container: Invert pre-check flag of set-eacl command
Flag `--pre-check` of `set-eacl` command found to be in demand in most
cases. based on this, it makes sense to add its action to the default
behavior.

Pre-check container extensibility by default. Rename flag to
`--no-precheck` and invert its action.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 16:22:16 +04:00
Leonard Lyubich
46b815c863 [#1652] cli/container: Move eaclPathFrom flag var into the struct
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 16:22:16 +04:00
Leonard Lyubich
491a908af1 [#1652] cli/container: Pre-check ACL extensibility in set-eacl
Container ACL in NeoFS can be extended only for container in which the
corresponding option is enabled. In previous implementation command
`set-eacl` could hang up on modifying eACL of the non-existent or
non-extendable container. To improve UX, there is a need to pre-check
the availability of `SETEACL` operation.

Add boolean `precheck` flag to `set-eacl` cmd which reads the container
before the actual transaction formation. If flag is set, command fails
on non-extendable container ACL.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 16:22:16 +04:00
Pavel Karpy
17059b34ea [#1743] morph: Fix non-notary calls
Some methods add "IR" suffix to its names in notary enabled envs
because of contract logic. It was broken due to incorrect notary state
reading (tryNotary != notary is enabled).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-02 12:11:57 +03:00
Evgenii Stratonikov
e6cf0b3d42 [#1745] writecache: Set flush parameters based on max size
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
c04126f35e [#1745] writecache: Remove IsErrNotFound
We specify the error in the doc-comment, and it is the same for all our
components.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
9e41e85295 [#1745] writecache: Actualize docs
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
e9c6ee2623 [#1745] writecache: Rename constants for default values
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
177e8e01b1 [#1745] neofs-node: Remove memcache_capacity from the configuration
It is unused since ddaed283e9 .

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
bda084f331 [#1745] writecache: Simplify object counters
Remove unused option and additional pointers to db/fstree.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
20abdaeed4 [#1745] writecache: Set flush mark only on success
Set flush mark in the inside the flush worker because writing to the blobstor
can fail. Because each evicted object must be deleted, it is reasonable
to do this in the evict callback.

The evict callback is protected by LRU mutex and thus potentially interferes
with `Get` and `Iterate` methods. This problem will be addressed in the
future.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 11:59:53 +03:00
Pavel Karpy
82839cb1c9 [#1711] morph: Add nolint for AddNetworkFee
Waits for `v0.99.3` neo-go unreleased yet.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-02 10:41:18 +03:00
Pavel Karpy
62b293b7ab [#1711] adm: Add error context
To make all the calls in the command to be in the same form.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-02 10:41:18 +03:00
Pavel Karpy
a99474c40e [#1711] adm: Drop deprecated neo-go calls
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-02 10:41:18 +03:00
Pavel Karpy
6d277a57aa [#1711] morph: Simplify code using neo-go actors
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-02 10:41:18 +03:00
Pavel Karpy
4582b8f0d4 [#1711] morph: Drop deprecated neo-go calls
Does not include notary actor and does not drop the deprecated
`AddNetworkFee`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-02 10:41:18 +03:00
Leonard Lyubich
ae6ebccd9c [#1632] node/container: Don't add new container to missed cache record
If container listing cache on node's side is missing (for particular
owner), then updating it as a reaction to successful container creation
leads to potentially invalid cache value for a period of time equivalent
to cache TTL.

Immediately return from `ttlContainerLister.update` method if owner's
container list isn't cached.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 11:29:55 +04:00
Leonard Lyubich
fa3124fc33 [#1632] node: Configure TTL of Sidechain caches
From now cache TTL can be parameterized in the `neofs-node` app using
`cache_ttl` config key. `disable_cache` value is no longer supported.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 11:29:55 +04:00
Leonard Lyubich
5ce8315cd8 [#1632] node/container: Update LIST cache on successful writes
In previous implementation storage node responded with the outdated
container list after successful creation/removal up until cache
invalidation due to TTL. In order to decrease the probability of
outdated responses node should update its cache on event receipts.

Implement `ttlContainerLister.update` method which actualizes cached
list of the owner's containers. Make node to call `update` method
on `PutSuccess`/`DeleteSuccess` notifications from the `Container`
contract.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 11:29:55 +04:00
Leonard Lyubich
d15a7d8d3d [#1632] node/container: Update GET cache on successful removal
In previous implementation storage node responded with the removed
container up until cache invalidation due to TTL. In order to avoid
false-positive responses node should update its cache on `DeleteSuccess`
events.

Make node to call `handleRemoval` method of the container cache which
leads to subsequent `apistatus.ErrContainerNotFound` errors.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 11:29:55 +04:00
Leonard Lyubich
a529149e6f [#1632] node: Cache Sidechain errors along with the values
In previous implementation failed requests to the Sidechain weren't
cached. It makes sense to cache errors along with the values in order to
decrease potential load spikes onto Sidechain nodes.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 11:29:55 +04:00
Leonard Lyubich
7c1babb7d6 [#1632] node: Subscribe on the successful container creations/removals
There is a need to sync container-related caching mechanism with the
actual Sidechain changes. To do this, node should be able to listen
incoming notifications about container ops.

Define `PutSuccess` / `DeleteSuccess` notification event's parsers.
Subscribe to these events in node app. As initial implementation node
will log event receipts. Later handling is going to be practically
complicated.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 11:29:55 +04:00
Evgenii Stratonikov
d8a00c365a [#1691] blobovniczatree: Fix active blobovnicza caching
Maintain an invariant that any blobovnicza is present either
in `opened` or in `active` map. Otherwise, the logic becomes too
complicate because it is not obvious when we should close the blobovnicza.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-31 19:11:48 +03:00
Evgenii Stratonikov
dfac4e1c0b [#1691] blobovniczatree: Do not fill cache during initialization
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-31 19:11:48 +03:00
Evgenii Stratonikov
df88afeef3 [#1738] neofs-adm: Clean tx hash list after awaiting
Do not query application log for the same transaction more than once.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 16:53:00 +03:00
Evgenii Stratonikov
ae52d53609 [#1698] pilorama: Add a test for the empty FileName
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 16:52:51 +03:00
Evgenii Stratonikov
180e5e938f [#1735] go.mod: Update API and SDK
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 12:27:46 +03:00
Evgenii Stratonikov
78bf17c0b0 [#1686] blobstor/*: Remove unneeded tests
The blobovniczatree test also checks the amount of objects we can put,
so leave it here.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
b9a2055e1c [#1686] blobstor/*: Return ErrReadOnly for modifying operations
This check should occur on the shard level, but because
blobstor components expose `Open(readOnly bool)` interface,
it is reasonable to expect an error here.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
6c2d3b020f [#1686] fstree: Return proper error for concurrent Delete
If the file doesn't exist, return `apistatus.ObjectNotFound`.
First check is still there as a shortcut.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
0b95a21701 [#1686] blobstor: Add generic tests
This tests check that each blobstor component behaves similarly when
same methods are being used. It is intended to serve as a specification
for all future components.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
b2d4cc556e [#1686] blobovnicza: Remove IterateObjects
It is used only once, makes sense to inline.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
5f95498364 [#1686] blobstor: Add generic tests
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
9792e6a4ef [#1686] metabase: Add generic tests
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
482a7e7f2f [#1686] pilorama: Add generic tests
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
b8b9d25f9d [#1686] local_object_storage: Add generic tests
Use them for writecache as a simple example.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
bc20851cd1 [#1686] shard: Open all components in case metabase has failed
All components must be opened first, before any other operation is
performed.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Pavel Karpy
c7c1c257e1 [#1658] node: Init object counter on meta's Init
Includes:
1. Renaming counter key to distinguish logical and physical objects
2. Version update dropping since changes could be done in a compatible way

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-25 19:20:33 +03:00
Pavel Karpy
cd6f8e051a [#1658] Update CHANGELOG
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-25 19:20:33 +03:00
Pavel Karpy
9c7b3ce799 [#1658] node: Read metrics from meta on startup
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-25 19:20:33 +03:00
Pavel Karpy
745f72fff0 [#1658] node: Support object counter metrics
Increment shard's object counter on successful `Put` calls and decrement on
`Delete`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-25 19:20:33 +03:00
Pavel Karpy
067ab44e0b [#1658] metrics: Add object counter metric
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-25 19:20:33 +03:00
Pavel Karpy
626766db08 [#1658] meta: Add object counter
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-25 19:20:33 +03:00
Evgenii Stratonikov
61f0d85834 [#1727] neofs-adm: Allow to register custom TLD for contracts
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-25 12:31:00 +03:00
Evgenii Stratonikov
ddbe129cb2 [#1729] neofs-adm: Change default domain expiration time
It is a noop for current contract versions, but will matter in the next
release.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-25 12:31:00 +03:00
Evgenii Stratonikov
5827a926c9 [#1729] go.mod: Update neofs-contract to v0.15.5
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-25 12:31:00 +03:00
Evgenii Stratonikov
19585c63ed [#1730] go.mod: Update neofs-sdk-go
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-24 17:17:42 +03:00
e68c80df0a [#1728] Don't describe default value twice
Cobra will mention default value automatically.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-08-24 16:44:44 +03:00
31b6af5667 [#1728] Update neofs-adm morph deploy command description
Add more details about expected file names in contract dir.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-08-24 16:44:44 +03:00
Pavel Karpy
b54f34d7df [#1683] adm: Do not deploy custom contract with a group
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-24 13:52:55 +03:00
Pavel Karpy
bbd6b9780f [#1683] adm: Minimize registration price
Register NNS domain in one TX:
1. Set minimal (`1`) registration price;
2. Register domain;
3. Return registration price back.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-24 13:52:55 +03:00
Evgenii Stratonikov
d57c57010f [#1726] neofs-cli: Truncate file before writing the object
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-23 18:07:12 +03:00
Evgenii Stratonikov
40f1cf36e1 [#1719] neofs-cli: Save object with proper permissions
`curl` uses 0644, for example.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-23 18:07:12 +03:00
Evgenii Stratonikov
871fae8321 [#1705] go.mod: Update neo-go to v0.99.2
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-23 11:59:23 +03:00
Evgenii Stratonikov
e4370c4129 [#1702] neofs-adm: Set HighPriority attribute for committee transactions
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-23 11:59:23 +03:00
Pavel Karpy
0720d96c9d [#1687] go.mod: Update neofs-sdk-go
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-22 18:59:57 +03:00
Pavel Karpy
d40ed00438 [#1687] go.mod: Update neofs-api-go
It contains the signature fix for go `v1.19`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-22 18:59:57 +03:00
Pavel Karpy
56792d5eaf [#1687] .github: Update actions to the latest version
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-22 18:59:57 +03:00
Pavel Karpy
99903e1aba [#1687] Support v1.19 Golang
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-22 18:59:57 +03:00
Pavel Karpy
30341f2192 [#1687] *: Perform go fmt using go v1.19
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-22 18:59:57 +03:00
Pavel Karpy
37ab26bfa9 [#1296] cli: "ID" -> "OID"
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-22 14:29:50 +04:00
Evgenii Stratonikov
e8c6dce466 [#1523] blobstor: Rename compression.CConfig to Config
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
0042c1d4a6 [#1584] blobovniczatree: move public methods to different files
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
26b4a258e0 [#1523] neofs-node: Refactor configuration
1. Move compression parameters to the `shard` section.
2. Allow to use multiple sub-storage components in the blobstor.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
13cdbde2e2 [#1523] blobstor: Remove unused types
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
0c9e4e6a35 [#1523] blobstor: Unify request dispatch logic
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
266458fe5c [#1523] blobstor: Unify fstree and blobovnicza interfaces
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
5aa3defc67 [#1523] writecache: Remove cloneBytes
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
ddaed283e9 [#1523] writecache: Simplify logic
1. Remove in-memory cache. It doesn't persist objects and if we want
   more speed, `NoSync` option can be used for the bolt DB.
2. Put to the metabase in a synchronous fashion. This considerably
   simplifies overall logic and plays nicely with the metabase bolt DB
   batch settings.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
4176b2a1bc [#1523] local_object_storage: Unify parameters for the Iterate operation
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
73f8bb3e5f [#1523] shard: Store generic storage ID in metabase
Allow to extend blobstor with more storage sub-systems. Currently
objects stored in the FSTree have empty byte slice descriptor and object
from blobovnicza tree have the same id as earlier. Each such change in
the identifier formation should be accompanied with metabase version
increase.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
9eb018672c [#1523] blobstor: Unify parameters for Put operation
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
7d6df543d7 [#1523] blobstor: Export single Get, GetRange and Delete methods
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
ca15083a50 [#1523] local_object_storage: Unify parameters for the Exists operation
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
d75d030a90 [#1523] local_object_storage: Unify parameters for the Delete operation
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
36c88f0dc8 [#1523] local_object_storage: Unify parameters for the GetRange operation
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
d9a2f280c9 [#1523] local_object_storage: Unify parameters for the Get operation
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
b621f5983a [#1523] local_object_storage: Move blobovnicza tree to a separate package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Pavel Karpy
5139dc9864 [#1706] cli: Do not duplicate payload on SG put
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-19 09:49:47 +04:00
Evgenii Stratonikov
4e361e2ab4 [#1647] writecache: Open DB in sync mode
Close #1647.

Initially, `Sync: false` was provided because we can already lose
objects cached in memory. However, future changes in writecache will
remove inmemory cache and speed up it via other means.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-16 15:27:18 +04:00
Evgenii Stratonikov
89cba2cc7b [#1684] .github: Update golangci-lint version
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-15 10:57:31 +03:00
Evgenii Stratonikov
f8b106ac85 [#1684] *: Fix linter warnings
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-15 10:57:31 +03:00
Leonard Lyubich
67af4c89a8 [#1649] cli: Refactor --owner flag definition in container list cmd
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-08-15 10:18:51 +03:00
Leonard Lyubich
7a26b2a57a [#1649] cli: Add option to print attributes in list
Define `--with-attr` flag of `container list` which makes the command to
request and print user attributes for each found element.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-08-15 10:18:51 +03:00
Leonard Lyubich
2740bf7ee4 [#1649] cli: Add option to print attributes in list-objects
Define `--with-attr` flag of `container list-objects` which makes the
command to request and print user attributes for each object from the
container.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-08-15 10:18:51 +03:00
e07921fc46 Use only v-prefixed tags for version calculation
This allows us to use any custom tags along with tags that denote releases

Signed-off-by: Vladimir Domnich <v.domnich@yadro.com>
2022-08-10 10:34:07 +03:00
Pavel Karpy
da2975a2f9 [#1664] write-cache: Fix panic on Delete operation
If an object is found in the Write-cache and is placed at the end of
the in-memory cache, the memory counter update operation tries to
dereference the index that is out of the sliced array. Moreover, even if
panic does not appear, the counter is updated with the wrong value.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-09 20:18:26 +03:00
a.y.volkov
81684b6f04 [#1665] neofs-lens: Use stdout as default output
Signed-off-by: a.y.volkov <a.y.volkov@yadro.com>
2022-08-09 13:23:32 +03:00
Pavel Karpy
cd71de69a0 Release v0.31.0
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 17:04:33 +03:00
Pavel Karpy
5ff1df285b Release v0.30.2
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 17:04:33 +03:00
Evgenii Stratonikov
058538768b [#1640] Release v0.30.1
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 17:04:33 +03:00
Pavel Karpy
a5da665e69 [#1656] meta: Fix MatchCommonPrefix operation
Return all the objects on the empty common prefix search without search
optimizations that breaks boltDB logic.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 16:32:00 +03:00
Pavel Karpy
156ba85326 [#1634] node: Do not return expired objects
If an object has not been marked for removal by the GC in the current epoch
yet but has already expired, respond with `ErrObjectNotFound` api status.
Also, optimize shard iteration: a node must stop any iteration if the object
 is found but gonna be removed soon.
All the checks are performed by the Metabase.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 16:31:49 +03:00
Pavel Karpy
9aba0ba512 [#1634] meta: Add epoch state
It allows performing expiration checks on the stored objects.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 16:31:49 +03:00
Pavel Karpy
a97dee008c [#1648] morph: Change endpoint priority order
The lowest value means the highest priority.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 16:11:24 +03:00
Pavel Karpy
2467be0117 [#1648] morph: Init endpoints before first client creation
Sort the endpoint by their priority before the first WS client creation to
start the morph client with the highest priority endpoint.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 16:11:24 +03:00
Pavel Karpy
1c1b5043fc [#1653] morph: Handle chain notifications via pool
Handling notification in a synchronous manner may lead to a blocking state
if a handler uses neo-go client.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 17:06:53 +04:00
Pavel Karpy
8f44335925 [#1651] cli: Fix CID setting in eACL creation
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 17:04:42 +04:00
Evgenii Stratonikov
9a5f9d6f0e [#1643] innerring: Exit if we cannot bind to the control endpoint
Return listen errors in a synchronous fashion.
Another solution would be to use buffered channel, but this is not
scalable: for each new similar runner we would need to extend the
buffer.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-04 15:41:41 +03:00
Pavel Karpy
0a60524a9c [#1654] ir, node: Drop deprecated profiler and metrics config sections
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 15:38:35 +03:00
Pavel Karpy
713cfa5610 [#1655] cli: Do not force specifying session lifetime
We have the default value which is also printed in the help messages but any
call that does not specify that flag leads to an error.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 15:37:50 +03:00
Pavel Karpy
c2918fce3a [#1645] node: Support EACL_NOT_FOUND status
Remove internal `ErrEACLNotFound` error.
Also, update `neofs-api-go` and `neofs-sdk-go` libraries.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-01 20:45:36 +03:00
Pavel Karpy
8c8ae56960 [#1645] go.mod: Update dependencies
- `neofs-api-go` to `v2.13.1`;
- `neofs-sdk-go` to `7a99cc916c8e`.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-01 20:45:36 +03:00
Evgenii Stratonikov
8ffc2fdf5e [#1642] engine: Do not increase error counter if the pilorama is disabled
After a4adb79db new logical error could be returned. Do not increase
error counter in this case.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-30 18:39:22 +03:00
Evgenii Stratonikov
4558f30575 [#1644] services/tree: Do not use deprecated gRPC options
Get rid of `grpc.WithInsecure`.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-30 18:39:08 +03:00
Evgenii Stratonikov
8d0884e74f [#1644] storagelog: Fix doc comment
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-30 18:39:08 +03:00
Evgenii Stratonikov
7ecf54c6df [#1631] neofs-adm: Set group for updated alphabet contracts
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 12:09:48 +03:00
Evgenii Stratonikov
d3eac7626d [#1631] neofs-adm: Set group for a proper contract
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 12:09:48 +03:00
Evgenii Stratonikov
5efcd34680 [#1639] go.mod: Update neofs-sdk-go to v1.0.0-rc.6
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 12:07:53 +03:00
Evgenii Stratonikov
5d02613c33 [#1633] neofs-adm: Allow to set custom NNS zone for deployed contracts
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 12:02:40 +03:00
Evgenii Stratonikov
6146df4998 [#1633] neofs-adm: Allow to provide arguments to deployed contracts
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 12:02:40 +03:00
Evgenii Stratonikov
dd30703a6b [#1633] neofs-adm: Allow to update arbitrary contracts
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 12:02:40 +03:00
Evgenii Stratonikov
fc06b6e89a [#1633] neofs-adm: Allow to deploy arbitrary contracts
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 12:02:40 +03:00
Evgenii Stratonikov
38558a3238 [#1633] neofs-adm: Start chain only on happy path
If the error is returned, it will not be closed properly.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 12:02:40 +03:00
Evgenii Stratonikov
642d7328ab [#1633] neofs-adm: Fix dump generation in init
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 12:02:40 +03:00
Evgenii Stratonikov
4abe5a7245 shard: add more checks for GetRange parameters
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 11:22:04 +03:00
Evgenii Stratonikov
72586f17d4 shard: fix GetRange for objects stored in the write-cache
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 11:22:04 +03:00
Evgenii Stratonikov
07465849a4 [#1637] go.mod: Update neo-go to v0.99.1
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-28 20:11:45 +03:00
Alex Vanin
71fd86f220 [#1636] ir: Listen balance contract in balance processor
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-07-28 18:47:20 +03:00
Pavel Karpy
8162b27264 [#1627] node: Add tree service to the config framework
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-27 12:15:56 +03:00
Pavel Karpy
a0fae0443f [#1627] node: Add missing tree service to the config files
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-27 12:15:56 +03:00
Evgenii Stratonikov
54fe7667fb [#1625] go.mod: Update dependencies
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-25 19:00:07 +03:00
Evgenii Stratonikov
2a1c5557f2 [#1624] Release v0.30.0
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-25 16:56:02 +03:00
Evgenii Stratonikov
10c855efef [#1624] go.mod: Update dependencies
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-25 16:56:02 +03:00
Evgenii Stratonikov
91361c4fe2 [#1624] CHANGELOG.md: Fix v0.29.0 release name
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-25 16:56:02 +03:00
Evgenii Stratonikov
25a0f539ea [#1624] config: Fix tree section example
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-25 16:56:02 +03:00
Pavel Karpy
589a54805d [#1618] node: Use OID/CID from the request in eACL checks
Also, try to fetch object header info from the local storage to find as much
object info as possible for the requests which do not assume returning
object header as a response.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-25 09:41:11 +03:00
Evgenii Stratonikov
6049022f7e [#1612] neofs-cli: Remove RunE functions from commands
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-22 16:18:09 +03:00
Evgenii Stratonikov
a52e7c2c99 [#1612] neofs-cli: Unify expiration flags
Use `expire-at` everywhere expiration epoch is expected.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-22 16:18:09 +03:00
Evgenii Stratonikov
175e9da3a7 [#1612] go.mod: Update cobra dependency
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-22 16:18:09 +03:00
Evgenii Stratonikov
7e717e80ef [#1623] services/tree: Fix error message
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-22 15:13:03 +03:00
Pavel Karpy
2f4e465cf4 [#1620] node: Remove debug attribute printing
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-22 13:50:36 +03:00
Pavel Karpy
ee9312dcfb [#1619] example: Fix IR RPC
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-22 12:37:25 +03:00
Pavel Karpy
fb1fac02e9 [#1619] ir: Fix RPC endpoint iteration
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-22 12:37:25 +03:00
Evgenii Stratonikov
1691364653 [#1559] local_object_storage: Fix tests and some data races
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
50e28f22f9 [#1559] shard: Change Degraded mode string representation
It is a flag, but is a `degraded-read-write` mode for a user.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Leonard Lyubich
fabe717d32 [#1549] shard: Turn to read-only mode on metabase failure
If metabase can't be opened in the default mode, try opening shard
first in `ReadOnly` mode and then in `DegradedReadOnly`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
4944490ffb [#1559] local_object_storage: Move shard to the DegradedReadOnly mode
`Degraded` mode can be set by the administrator if needed.
Modifying operations in this mode can lead node into an inconsistent state
because metabase checks such as lock checking are not performed.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
3df62769c0 [#1559] local_object_storage: Allow to set mode for all components
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
1e786233bf [#1559] local_object_storage: Provide readOnly flag to Open
We should be able to reopen storage in readonly in runtime.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Leonard Lyubich
e38b0aa4ba [#1549] engine: Disable shard on blobovnicza init failure
There is a need to support working w/o shard if it has problems with
blobovnicza tree.

Make `BlobStor.Init` to return new `ErrInitBlobovniczas` error. Remove
shard from storage engine's shard set if it returned this error from
`Init` call. So if some of the shards (but not all) return this error,
the node will be able to continue working without them.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
40a56c6b42 [#1559] engine: Do not count logical errors as storage ones
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
c8911d72d0 [#1559] shard: Do not consult metabase in a degraded mode
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
339864b720 [#1559] local_object_storage: Move shard.Mode to a separate package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
d8ba954aff [#1559] shard: Use Set prefix for parameter setting
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
7b882b26d8 [#1559] shard: Remove public functions
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
34d319fed2 [#1559] metabase: Use Set prefix for parameter setting
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
f58234aa2f [#1559] metabase: Remove public functions
Reduce public interface of this package. Later each result will contain
an additional status, so it makes more sense to use the same functions
and result processing everywhere.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
30c7925b3c [#1609] morph/client: Retry connecting to failed endpoint during the switch
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 16:08:42 +03:00
Evgenii Stratonikov
7410827db8 [#1609] config: Allow to prioritize N3 endpoints
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 16:08:42 +03:00
Evgenii Stratonikov
aed83d1660 [#1609] config: Remove mainchain section
It is deprecated and unused, there is no need to have it in the
documentation.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 16:08:42 +03:00
Pavel Karpy
a4ed91f9b5 [#1614] node: Update health metric on its change
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-21 16:07:47 +03:00
Pavel Karpy
581a9901c9 [#1614] metrics: Add health metrics
Also, rename metrics structure since it collects not only storage metrics
now.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-21 16:07:47 +03:00
Evgenii Stratonikov
2455b72844 [#1607] CHANGELOG.md: add tree service info
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
b549cc314c [#1607] services/tree: allow to customize some parameters
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
2334e94fdb [#1607] neofs-node: Initialize storage before other services
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
a4adb79db7 [#1607] pilorama: Enable tree service explicitly
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
d62723f038 [#1505] pilorama: Provide timeout to bbolt.Open
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
26041f18bf [#1505] pilorama: Allow to customize database parameters
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
f80e52fbea [#1334] services/tree: Document *.proto files
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
3904b0e017 [#1333] neofs-cli: Add control synchronize-tree command
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
bfdd68dcb3 [#1333] services/control: Allow to synchronize local trees
Do not check that a node indeed belongs to the container, because the
synchronization will fail in this case anyway.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
0dc1a4e336 [#1333] neofs-node: Initialize tree service before the control one
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
c7b10598f9 [#1333] engine: Increase error counter for pilorama errors
1. Modifying operations are not expected to fail, unless the shard is
   read-only.
2. `Get*` operations should increase error counter too, unless the
   error is `ErrTreeNotFound`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
5e843a73f9 [#1333] services/control: Return pilorama info in ListShards RPC
Do not return backend type from the service for now, because memory
backend is expected to vanish.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
8f4ee1aded [#1333] local_object_storage: Support ReadOnly mode in pilorama
The tricky part here is the engine itself: we stop iteration on
`ErrReadOnly` because it is better to synchronize the shard later than
to have partial trees stored in 2 shards.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
735931c842 [#1481] pilorama: Fix TreeApply
Current implementation prevents invalid operations to become valid at
some later point (consider adding a child to the non-existent parent and
then adding the parent). This seems to diverge from the paper algorithm
and complicates implementation. Make it simpler.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
5ffbeb76e6 [#1456] services/tree: Wait some time before reconnecting after failure
In case node is down or failing for some reason, we can expect `Dial` to
fail. In case we actively try to replicate and `Dial` always takes 2
seconds, replication-related channels quickly become full. That affects
latency of all other write operations.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
39f47f61c6 [#1445] services/tree: Cache the list of container nodes
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
a2edfec0c3 [#1446] services/tree: Use grpc.WithInsecure only for nodes without TLS
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
ad3038d16d [#1444] pilorama: Fix TreeMove in bbolt backend
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
33d8fb187a [#1427] services/tree: Parallelize replicator
Before this commit the replication channel was quickly filled under
heavy load. This lead to the continuously increasing latency for all
write operations. Now it looks better.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
8027b7bb6b [#1444] pilorama: Optimize internal encoding/decoding
```
name                      old time/op    new time/op    delta
ApplySequential/bbolt-8     55.5µs ± 4%    55.5µs ± 3%     ~     (p=1.000 n=10+7)
ApplyReorderLast/bbolt-8     108µs ± 6%     112µs ± 8%     ~     (p=0.077 n=9+9)

name                      old alloc/op   new alloc/op   delta
ApplySequential/bbolt-8     28.8kB ± 3%    27.7kB ± 6%   -3.79%  (p=0.005 n=10+10)
ApplyReorderLast/bbolt-8    41.4kB ± 5%    38.9kB ± 5%   -6.19%  (p=0.001 n=10+9)

name                      old allocs/op  new allocs/op  delta
ApplySequential/bbolt-8        262 ± 2%       235 ±10%  -10.41%  (p=0.000 n=10+10)
ApplyReorderLast/bbolt-8       684 ± 6%       616 ± 7%  -10.04%  (p=0.000 n=10+9)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
b04f712773 [#1446] services/tree: Cache connections to the container nodes
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
4437cd7113 [#1442] pilorama: Generate timestamp based on node position in the container
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
3caa982283 [#1442] services/tree: Fix log message for failed Apply
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
01d2e06a9b [#1401] services/tree: Marshal public key once
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
1f5a650b05 [#1401] services/tree: Retransmit queries to container nodes
Also fix a bug with replicator using the multiaddress instead of
<host>:<port> format expected by gRPC library.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
3312924b82 [#1431] pilorama: Use Batch for write transactions
Helps a lot in case of concurrent request flow.

```
name                      old time/op    new time/op    delta
ApplySequential/bbolt-8     78.0µs ± 9%    59.8µs ± 4%  -23.39%  (p=0.000 n=10+9)
ApplyReorderLast/bbolt-8     143µs ± 5%     113µs ±15%  -21.06%  (p=0.000 n=10+10)

name                      old alloc/op   new alloc/op   delta
ApplySequential/bbolt-8     56.9kB ± 8%    28.9kB ± 3%  -49.22%  (p=0.000 n=10+10)
ApplyReorderLast/bbolt-8    87.3kB ± 3%    40.9kB ±10%  -53.16%  (p=0.000 n=10+10)

name                      old allocs/op  new allocs/op  delta
ApplySequential/bbolt-8        224 ±11%       262 ± 5%  +16.93%  (p=0.000 n=9+10)
ApplyReorderLast/bbolt-8       518 ± 4%       674 ±11%  +30.09%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
f0a67f948d [#1431] pilorama: Cache attributes in the index
Currently to find a node by path we iterate over all the children on
each level. This is far from optimal and scales badly with the number of
nodes on a single level. Thus we introduce "indexed attributes" for
which an additional information is stored and which can be use in
`*ByPath` operations. Currently this set only includes `FileName`
attribute but this may change in future.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
f5d35571d0 [#1431] engine: Add benchmark for Select vs TreeGetByPath
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
f4bc1c601a [#1329] services/tree: Synchronize from the last stored height
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
835170e452 [#1329] pilorama: Allow to benchmark all tree backends
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
536857ea5a [#1329] services/tree: Implement GetOpLog RPC
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
886baf3136 [#1426] services/tree: Do not replicate to a local node
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
7703dd5d7f [#1419] pilorama: Create new nodes in path if needed
Consider a node `{FileName: "dir", Attribute: "xxx"}`. In case we add
a new node by path `["dir", "file.txt"]`, create a new intermediate node
with a single attribute.

`GetByPath` now also considers only nodes with a single attribute while building a path.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
ad48918a97 [#1406] pilorama: Return parent from TreeGetMeta
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Denis Kirillov
b30f14978d [#1404] services/tree: Fix nodeId in GetSubTree
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
45321b3f6d [#1328] services/tree: Fix grpc import path
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
16e3421825 [#1328] services/tree: Implement access control
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
94b9e13431 [#1328] services/tree: Fix proto field numbers
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
36e24f7f78 [#1342] neofs-node: Use the default endpoint for tree service
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
8844d9b2db [#1344] pilorama: Document errors for Get* methods
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
d8ad68d613 [#1344] engine: Log errors in Tree* operations
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
910db42748 [#1344] pilorama: Use require.ErrorIs in tests
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
aea855e8f3 [#1326] services/tree: Implement GetSubTree RPC
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
62154da17c [#1324] services/tree: Implement Object Tree Service
Object Tree Service allows changing trees assotiated with
the container in runtime.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
46f4ce2773 [#1324] engine: Implement Forest interface for storage engine
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
8cf71b7f1c [#1324] local_object_storage: Implement tree service backend
In this commit we implement algorithm for CRDT trees from
https://martin.klepmann.com/papers/move-op.pdf

Each tree is identified by the ID of a container it belongs to
and the tree name itself. Essentially, it is a sequence of operations
which should be applied in chronological order to get a usual tree
representation.

There are 2 backends for now: bbolt database and in-memory.
In-memory backend is here for debugging and will eventually act
as a memory-cache for the on-disk database.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
198beae703 [#1613] neofs-cli: Print lock object ID
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-19 17:15:15 +03:00
Evgenii Stratonikov
5c8d725447 [#1610] neofs-adm: Fix generated storage config
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-19 16:19:41 +03:00
Evgenii Stratonikov
11c9df6f00 [#1611] shard: Print shard ID in component logs
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-19 15:57:53 +03:00
Pavel Karpy
9f7a22e2aa [#1561] object: Return OUT_OF_RANGE status
Replace `ErrRangeOutOfBounds` error from `pkg/core/object` package with
`ObjectOutOfRange` from `apistatus` package. That error is returned by
storage node's server as NeoFS API statuses.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-19 13:29:45 +03:00
Evgenii Stratonikov
ccba07fe19 [#1608] neofs-adm: Register candidates in a single transaction
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-19 13:28:17 +03:00
Evgenii Stratonikov
59d08b8dae [#1608] neofs-adm: Print info about already performed steps
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-19 13:28:17 +03:00
Evgenii Stratonikov
5e1636e50c [#1604] docs: Describe storage node configuration
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-19 11:12:52 +03:00
Evgenii Stratonikov
eeef83a74c [#1604] config: Change yaml example a bit
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-19 11:12:52 +03:00
Evgenii Stratonikov
e5c0338f8c [#1602] neofs-ir: Fix defaults for pprof and prometheus
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-18 19:41:22 +03:00
Evgenii Stratonikov
3a0b06d5da [#1602] config: Rename metrics and profiler sections
Depracate old names and remove them in the next release.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-18 19:41:22 +03:00
Evgenii Stratonikov
ac46d1a11f [#1602] config: Enable metrics and profiler services with a flag
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-18 19:41:22 +03:00
Evgenii Stratonikov
12dc5c3395 [#1563] config: Replace 127.0.0.1 with localhost
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-18 19:41:22 +03:00
Evgenii Stratonikov
4a8dfd0ce4 [#1606] Remove deprecated functions
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-18 12:41:06 +03:00
Evgenii Stratonikov
6cc180391e [#1606] go.mod: Update neo-go
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-18 12:41:06 +03:00
Pavel Karpy
cb0bb7207c [#1461] shard: Add a separate ErrLockObjectRemoval
Do not return `meta.ErrLockObjectRemoval` from shard's methods, add shard's
own error for that.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
558cc1193a [#1461] engine: Clarify force removal
Document force removal behaviour in all the Storage engine parts.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
b1e54dec9f [#1461] cli: Add context to GetCurrentEpoch helper
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
51afcc1182 [#1461] engine, policer: Force remove objects w/o container
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
18ec5d7c8e [#1461] meta: Return error on lock object removal
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
b5c56d459a [#1461] engine: Add force lock removal tests
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
63c00e785d [#1461] shard: Fix option naming
`WitDeletedLockCallback` => `WithDeletedLockCallback`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
5122be34e7 [#1461] shard: Add lock tests
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
eaf96bccf7 [#1461] meta: Add lock tests
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
c6cf8e5c0b [#1461] engine: Fix LOCK test
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
9c5ef3bab8 [#1461] node: Allow force LOCK removal
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
25826bd96d [#1461] node: Do not accept lock object w/o expiration
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
de31bb27b6 [#1461] cli: Require LOCK object to be expired
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
595fc60d72 [#1461] cli: Reorganize common functions
Make `ParseEpoch` and `GetCurrentEpoch` funcs public and move the first one
to the `common` package.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
fed9e6679d [#1461] node: Unlock locked object on its lock removal
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Evgenii Stratonikov
72c044e2eb [#1599] engine: Parallelize shard initialization
Shard is intended to be used as a separate failure domain,
which usually resides on a separate disk. Thus, sequential
initialization is bound by IO and this change speeds up thing a bit.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-15 13:44:54 +03:00
Evgenii Stratonikov
6eb5260562 [#1500] neofs-cli: Check if container contains LOCK objects before removal
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-15 13:25:56 +03:00
Pavel Karpy
a421344727 [#1605] node: Stream forwarder requests
Do not hold objects in memory in the forwarded Get/GetRange requests.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-15 11:16:35 +03:00
Pavel Karpy
89ebc278b8 [#1596] *: Remove .gitkeep files
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-14 15:10:57 +03:00
Evgenii Stratonikov
c4b86cf1f1 [#1592] Remove debug builds
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-14 10:39:51 +03:00
Evgenii Stratonikov
b8508585a5 [#1592] cmd: Ignore error when fetching a defined flag
There is an error only if the flag is not defined, such errors should be
caught during debugging.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-14 10:39:51 +03:00
Evgenii Stratonikov
f7d5045876 [#1592] neofs-lens: Add --version flag
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-14 10:39:51 +03:00
Evgenii Stratonikov
5569ff82ef [#1587] Do not print build time in version
This makes our build more reproducible.
Also print `Component` and `GoVersion`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-14 10:39:51 +03:00
Pavel Karpy
b3272e7cf1 [#1367] adm: Support homomorphic hashing config
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
4a316ceae9 [#1365] morph: Do not return errors if config key is missing
Return default values instead of casting errors in `HomomorphicHashDisabled`
method.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
9bd1951ca4 [#1365] adm: Add homomorphic hash disabling option
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
89118e9da0 [#1365] node: Calculate object homomorphic hash flexibly
Do not calculate and do not write homomorphic hash for containers that were
configured to store objects without hash.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
ae4740f99f [#1365] ir: Check homomorphic hash flexibly in audit
Do not perform that check if it was turned off for the container being
checked.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
8a61d33c65 [#1365] ir: Check homomorphic hash setting on ContainerPut
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
fa9c910648 [#1365] morph: Add HomomorphicHashDisabled config getter
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
ae86d2907c [#1365] cli: Calculate homomorphic hash flexibly
Do not use homomorphic hash in storage group for containers that have
`homomorphic_hashing_disabled` set to `true`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
68e583f143 [#1365] cli: Sync container with network config
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
49ba1ef1b6 [#1365] cli: Add SyncContainerSettings func to internal client
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Evgenii Stratonikov
1119014e53 [#1579] config: Move storage.default to shard.default
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-13 12:31:42 +03:00
Evgenii Stratonikov
0ab168a00a [#1564] .github: Remove assign-to-project action
It doesn't work now and needs to be done differently.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-13 11:35:48 +03:00
Evgenii Stratonikov
f4a3fc5f11 [#1594] neofs-adm: Support better completion for set-policy
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-13 11:00:35 +03:00
Pavel Karpy
466b9b4273 [#1593] adm: Support changing network config
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-12 21:03:43 +03:00
Pavel Karpy
1e034c8d48 [#1532] node: Support signature status codes
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-12 18:22:59 +03:00
Pavel Karpy
dac4a838fa [#1490] Update the CHANGELOG
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-12 17:35:46 +03:00
Pavel Karpy
96a24b5721 [#1490] cli: Require SG members to be unique
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-12 17:35:46 +03:00
Pavel Karpy
5d5d21e3d0 [#1490] node: Require SG members to be unique
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-12 17:35:46 +03:00
Evgenii Stratonikov
0504c3e0c6 [#1266] object/acl: Check bearer token container ID
If the container ID is not nil and not equal to the container ID in the
request, consider bearer token invalid.

See also nspcc-dev/neofs-api#207.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-12 12:25:02 +03:00
Leonard Lyubich
3a2c025843 Release v0.29.0
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-07-08 12:32:36 +03:00
Pavel Karpy
1658242e00 [#1590] node: Smart memory allocation in GetRange
Allocate memory only if a node chosen as the forwarded request receiver
has responded with a successful status.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-08 12:04:03 +03:00
Pavel Karpy
fed5e76e7f [#1590] node: Check status of forwarded requests
After fixing version fields in forwarded requests, a node does not check
statuses since errors are not covered by direct call error checks.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-08 12:04:03 +03:00
Leonard Lyubich
fdf62e8562 [#1586] Upgrade NeoFS SDK Go to rc#5
Error checkers now support wrapped errors so there is no need to
explicitly unwrap errors in `Policer`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-07-07 14:23:41 +03:00
Leonard Lyubich
9a11a75b77 [#1570] Upgrade NeoFS SDK Go with changed reputation API
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-07-06 18:21:24 +03:00
Pavel Karpy
9a6da336db [#1581] node: Do not lose API version on forwarding
Forwarded requests contained zero version in their meta header. It did not
allow responding with API statuses (`v0.0` version considered to be older
than `v2.11`) to the forwarding node and, therefore, did not allow analyzing
responses.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 17:06:24 +03:00
Pavel Karpy
c8506b247e [#1582] *: Fix linter warnings
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 11:56:01 +03:00
Stanislav Bogatyrev
df0d352305 [#1578] Reorder .gitignore
Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2022-07-06 10:00:19 +03:00
Stanislav Bogatyrev
930b51b57d [#1578] Minor Makefile fixes
- Add VERSION file to make it work without .git folder
- Remove obsolete GO111MODULE flag
- Move linter version to variable
- clean .cache on `make clean`
- Move help target to a separate help.mk file

Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2022-07-06 10:00:19 +03:00
Pavel Karpy
a153e040cb [#1402] ir: Do not use map for combining SG ID and SG object
Also, add corresponding structure for that.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 09:56:26 +03:00
Pavel Karpy
ec07fda97b [#1402] pkg: Move SGSource to the core directory
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 09:56:26 +03:00
Pavel Karpy
27304455bf [#1402] ir: Filter expired SGs in the audit process
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 09:56:26 +03:00
Pavel Karpy
7864959d0c [#1402] ir: Move GetSG interface
`auditor` does not need to request SG: processor will fetch that info before
audit context initialization.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 09:56:26 +03:00
Pavel Karpy
6370c2e160 [#1402] ir: Make ClientCache not depend on audit.Task
That allows using `ClientCache` for storage group searching before task
context is initialized. Also, that makes it more general purpose.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 09:56:26 +03:00
Pavel Karpy
5f658c499c [#1402] ir: Collect SG from the object
Do not use `Marshal()` with object's payload. Use `ReadFromObject` func from
SDK instead. That allows checking both attributes and SG body's expiration
epoch.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 09:56:26 +03:00
Pavel Karpy
12d41918e6 [#1402] cli: Support storage group expiration
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 09:56:26 +03:00
Pavel Karpy
cfca88fc5c [#1402] cli: object <-> SG conversion via SDK
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 09:56:26 +03:00
Pavel Karpy
bc5882fc89 [#1402] cli: Add cid flag constant
Also, move common flag constants in `root` file of the `storagegroup`
package.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 09:56:26 +03:00
Evgenii Stratonikov
6f2363cf31 [#1438] README.md: add build instructions
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-05 14:47:12 +03:00
Leonard Lyubich
c165d1a9b5 [#1556] Upgrade NeoFS SDK Go with changed container API
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-07-05 11:26:06 +03:00
Evgenii Stratonikov
f699e82ea7 [#1560] morph/client: Remove customFees type
It is private, is a simple map and there is no complex logic to be wrapped in methods.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-04 09:29:01 +03:00
Evgenii Stratonikov
0ccea802e9 [#1560] morph/client: Perform RPC switch and restore in one step
Otherwise we could switch infinitely if subscription restore has failed.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-04 09:29:01 +03:00
Evgenii Stratonikov
6358f4d746 [#1053] config: Remove notification endpoints
Missed this in #1170.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-30 10:51:57 +03:00
Evgenii Stratonikov
ee27c66952 [#1555] util/gendoc: Remove trailing dot from the help message
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-28 13:46:02 +03:00
Evgenii Stratonikov
ad81b6c90a [#1555] neofs-adm: Remove trailing dots from help messages
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-28 13:46:02 +03:00
Evgenii Stratonikov
b41658db04 [#1555] neofs-adm: Allow to remove nodes from the netmap
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-28 13:46:02 +03:00
Evgenii Stratonikov
32fd3692bf [#1555] neofs-adm: Fix cached group signer
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-28 13:46:02 +03:00
Evgenii Stratonikov
5c2ef79080 [#1547] .github: Assign issues to project when labeled
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-27 10:11:43 +03:00
Leonard Lyubich
305dd7598f [#1533] acl: Upgrade NeoFS SDK Go with refactored basic ACL
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-25 13:57:21 +03:00
Evgenii Stratonikov
b13dca8052 [#1545] .github: Auto assign new issues to the project
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-24 20:27:45 +03:00
Evgenii Stratonikov
451343c751 [#1539] neofs-adm: Retrieve storage wallet passwords from config
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-24 13:40:21 +03:00
Pavel Karpy
fc7940a5ea [#1544] cli: Init bearer flags in list-objects cmd
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-24 13:40:07 +03:00
Pavel Karpy
bb52372108 [#1544] go.mod: Update SDK
The updated version supports reading network configuration uint64 values
that consist of less than 8 bytes.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-24 13:40:07 +03:00
Evgenii Stratonikov
9b3e1bd27b [#1307] neofs-adm: Add tests for morph init command
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-24 10:33:34 +03:00
Evgenii Stratonikov
189507dc89 [#1541] neofs-cli: allow to print shards list output in JSON
Eventually more parameters will be supported (#1390) and after blobstor
configuration refactoring the output will certainly change. Implement
the simplest approach now.

Marshaling the result directly results in too ugly names and they cannot
be easily customized. Marshaling the results via `jsonpb` is better but
is not that flexible in terms of what we want to output.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-24 10:33:22 +03:00
Evgenii Stratonikov
9816d59ec0 [#1323] neofs-cli: Reuse JSON flag for multiple commands
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-24 10:33:22 +03:00
Pavel Karpy
e812d78672 [#1537] node: Remove "trust" from non-reputation logs
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-23 10:06:36 +03:00
Leonard Lyubich
b67974a8d3 [#xxx] Upgrade NeoFS SDK Go with changed container sessions
After recent changes in NeoFS SDK Go library session tokens aren't
embedded into `container.Container` and `eacl.Table` structures.

Group value, session token and signature in a structure for container
and eACL.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-22 16:38:57 +03:00
Evgenii Stratonikov
633b4e7d2d [#1483] metabase: Add VERSION.md
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-21 17:48:28 +03:00
Evgenii Stratonikov
6f243a2a76 [#1483] metabase: Store version
The main problem is to distinguish the case of initial initialization
and update from version 0. We can't do this at `Open`, because of
`resync_metabase` flag. Thus, the following approach was taken:
1. During `Open` check whether the metabase was initialized.
2. Check for the version in `Init` or write the new one if the metabase
   is new.
3. Update version in `Reset`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-21 17:48:28 +03:00
Evgenii Stratonikov
7df50297cd [#1520] shard: Ignore errors on metabase refill
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-21 17:25:28 +03:00
Evgenii Stratonikov
78ea450c25 [#1502] shard: Process locks on metabase refill
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-21 11:07:26 +03:00
Evgenii Stratonikov
972ca83e23 [#1524] writecache: Add some bolt parameters to the configuration
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-20 17:04:35 +03:00
Evgenii Stratonikov
07e06249d5 [#1524] metabase: Add some bolt parameters to the configuration
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-20 17:04:35 +03:00
Evgenii Stratonikov
d6043abc24 [#1524] neofs-node: Reuse boltdb config for different components
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-20 17:04:35 +03:00
Evgenii Stratonikov
0408acc40e services/control: Autogenerate stable marshalers
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-20 16:52:58 +03:00
Leonard Lyubich
d2d1694113 [#1513] cli/storagegroup: Fix linter remarks
`staticcheck` linter scolds on potential nil-dereference since it
doesn't know `common.ExitOnErr` definitely exists. Explicit `return`
fixes linter's remark and prevents potential bugs.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-17 15:53:18 +03:00
Leonard Lyubich
13e74fce8a [#1513] ir/settlement: Do not allocate intermediate slice of nodes
After recent changes `buildContainer` method returns two-dimensional
slice of `NodeInfo` so there is no need to flatten it to build slice of
`common.NodeInfo`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-17 15:53:18 +03:00
Leonard Lyubich
c408a6a0db [#1513] morph/netmap: Use constant states in unit tests
Use values of the node state enumeration from Netmap contract instead of
numeric literals.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-17 15:53:18 +03:00
Leonard Lyubich
eb7b8bf7c4 [#1513] node: Encode node's public key into binary format once
Avoid per-iteration key encoding due to the immutability of the key.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-17 15:53:18 +03:00
Leonard Lyubich
4574ba646d [#1513] cli/storagegroup: Fix parsing container and SG flags
Read `id` flag with the storage group ID. Prevent NPE-panic if flag is
missing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-17 15:53:18 +03:00
Leonard Lyubich
820acebb7d [#1513] morph/netmap: Return result by value from net config reader
Make `ReadNetworkConfiguration` method to return `NetworkConfiguration`
by value in order to follow storage engine improvements and prevent heap
escaping.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-17 15:53:18 +03:00
Leonard Lyubich
808ba87e82 [#1513] morph/netmap: Use node state constants from Netmap contract
`Netmap` contract exports enumeration of the node states.

Replace using literals and constants from NeoFS API Go V2 with the
values provided by contract.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-17 15:53:18 +03:00
Leonard Lyubich
21d2f8f861 [#1513] Upgrade NeoFS SDK Go with changed netmap package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-17 15:53:18 +03:00
Evgenii Stratonikov
24b4c1ecf4 [#1484] neofs-cli: Print error on a separate line from a progress bar
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-16 16:38:33 +03:00
Evgenii Stratonikov
e42b7f6a65 [#1484] neofs-cli: Print progress bar after the header has been sent
Make it similar to `object get`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-16 16:38:33 +03:00
Evgenii Stratonikov
b68778ea90 [#1380] neofs-cli: move container command to a separate package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-16 09:27:07 +03:00
Evgenii Stratonikov
fd48b96082 [#1384] neofs-cli: move storagegroup command to a separate package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-16 09:27:07 +03:00
Evgenii Stratonikov
a219e3a667 [#1383] neofs-cli: move object command to a separate package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-16 09:27:07 +03:00
Evgenii Stratonikov
12bc5607f7 [#1382] neofs-cli: move netmap command to a separate package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-16 09:27:07 +03:00
Evgenii Stratonikov
81f925d5a0 [#1516] metabase: Optimize ListWithCursor for long listings
Cache buckets outside of the main loop and allocate memory for the
resulting offset only once.

```
name                        old time/op    new time/op    delta
ListWithCursor/1_item-8       6.45µs ±14%    5.79µs ±11%  -10.24%  (p=0.002 n=10+10)
ListWithCursor/10_items-8     20.9µs ±17%    17.3µs ± 9%  -17.27%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     153µs ±12%     131µs ± 9%  -14.63%  (p=0.000 n=10+10)

name                        old alloc/op   new alloc/op   delta
ListWithCursor/1_item-8       2.31kB ± 0%    1.91kB ± 0%  -17.46%  (p=0.000 n=10+10)
ListWithCursor/10_items-8     6.94kB ± 0%    5.50kB ± 0%  -20.78%  (p=0.000 n=8+8)
ListWithCursor/100_items-8    53.3kB ± 0%    41.5kB ± 0%  -22.18%  (p=0.000 n=10+10)

name                        old allocs/op  new allocs/op  delta
ListWithCursor/1_item-8         40.0 ± 0%      34.0 ± 0%  -15.00%  (p=0.000 n=10+10)
ListWithCursor/10_items-8        121 ± 0%       100 ± 0%  -17.36%  (p=0.000 n=10+10)
ListWithCursor/100_items-8       930 ± 0%       758 ± 0%  -18.49%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 20:49:41 +03:00
Evgenii Stratonikov
a93373fe71 [#1516] metabase: Cache graveyard buckets in ListWithCursor
```
name                        old time/op    new time/op    delta
ListWithCursor/1_item-8       6.40µs ±13%    6.45µs ±14%     ~     (p=0.739 n=10+10)
ListWithCursor/10_items-8     30.9µs ±21%    20.9µs ±17%  -32.49%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     274µs ±27%     153µs ±12%  -44.09%  (p=0.000 n=10+10)

name                        old alloc/op   new alloc/op   delta
ListWithCursor/1_item-8       2.26kB ± 0%    2.31kB ± 0%   +2.46%  (p=0.000 n=10+10)
ListWithCursor/10_items-8     10.8kB ± 0%     6.9kB ± 0%  -36.07%  (p=0.000 n=8+8)
ListWithCursor/100_items-8    96.8kB ± 0%    53.3kB ± 0%  -44.98%  (p=0.000 n=10+10)

name                        old allocs/op  new allocs/op  delta
ListWithCursor/1_item-8         39.0 ± 0%      40.0 ± 0%   +2.56%  (p=0.000 n=10+10)
ListWithCursor/10_items-8        192 ± 0%       121 ± 0%  -36.98%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     1.72k ± 0%     0.93k ± 0%  -45.93%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>

name                        old time/op    new time/op    delta
ListWithCursor/1_item-8       5.23µs ±19%    5.26µs ±15%     ~     (p=0.853 n=10+10)
ListWithCursor/10_items-8     27.2µs ±15%    18.0µs ±19%  -33.80%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     250µs ±13%     139µs ±15%  -44.27%  (p=0.000 n=10+10)

name                        old alloc/op   new alloc/op   delta
ListWithCursor/1_item-8       1.99kB ± 0%    2.04kB ± 0%   +2.82%  (p=0.000 n=8+8)
ListWithCursor/10_items-8     10.3kB ± 0%     6.4kB ± 0%  -37.83%  (p=0.000 n=8+10)
ListWithCursor/100_items-8    93.9kB ± 0%    50.4kB ± 0%  -46.37%  (p=0.000 n=10+10)

name                        old allocs/op  new allocs/op  delta
ListWithCursor/1_item-8         35.0 ± 0%      36.0 ± 0%   +2.86%  (p=0.000 n=10+10)
ListWithCursor/10_items-8        184 ± 0%       113 ± 0%  -38.59%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     1.67k ± 0%     0.88k ± 0%  -47.29%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 20:49:41 +03:00
Evgenii Stratonikov
af4db8a73b [#1516] metabase: Cache address key and do not decode address twice
```
name                        old time/op    new time/op    delta
ListWithCursor/1_item-8       10.6µs ± 1%     6.4µs ±13%  -39.62%  (p=0.000 n=7+10)
ListWithCursor/10_items-8     75.3µs ± 2%    30.9µs ±21%  -58.97%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     726µs ± 2%     274µs ±27%  -62.28%  (p=0.000 n=10+10)

name                        old alloc/op   new alloc/op   delta
ListWithCursor/1_item-8       3.19kB ± 0%    2.26kB ± 0%  -29.21%  (p=0.000 n=10+10)
ListWithCursor/10_items-8     20.7kB ± 0%    10.8kB ± 0%  -47.68%  (p=0.000 n=10+8)
ListWithCursor/100_items-8     196kB ± 0%      97kB ± 0%  -50.65%  (p=0.000 n=7+10)

name                        old allocs/op  new allocs/op  delta
ListWithCursor/1_item-8         55.0 ± 0%      39.0 ± 0%  -29.09%  (p=0.000 n=10+10)
ListWithCursor/10_items-8        346 ± 0%       192 ± 0%  -44.51%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     3.25k ± 0%     1.72k ± 0%  -47.13%  (p=0.000 n=9+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 20:49:41 +03:00
Evgenii Stratonikov
504f45e9ee [#1516] metabase: Add benchmark for ListWithCursor
```
name                        time/op
ListWithCursor/1_item-8     10.6µs ± 1%
ListWithCursor/10_items-8   75.3µs ± 2%
ListWithCursor/100_items-8   726µs ± 2%

name                        alloc/op
ListWithCursor/1_item-8     3.19kB ± 0%
ListWithCursor/10_items-8   20.7kB ± 0%
ListWithCursor/100_items-8   196kB ± 0%

name                        allocs/op
ListWithCursor/1_item-8       55.0 ± 0%
ListWithCursor/10_items-8      346 ± 0%
ListWithCursor/100_items-8   3.25k ± 0%
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 20:49:41 +03:00
Pavel Karpy
36f4929e52 [#1507] node: Do not handle object concurrently by the policer
Cache object that are being processed. That prevents concurrent
object handling when there is a few number of objects and object handling
takes more time that the policer needs for starting that object handling one
more time.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-15 20:43:32 +03:00
Pavel Karpy
256165045b [#1508] node: Do not replicate object twice
If placement contains two vectors with intersecting nodes it was possible to
send the object to the nodes twice.
Also optimizes requests: do not ask about storing the object twice from the
same node.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-15 20:33:04 +03:00
Pavel Karpy
df8a3807fe [#1508] node: Remove unused replicator code
The node does not support asynchronous object replication anymore, so it
does not need to have replicator worker, channel and `AddTask` function.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-15 20:33:04 +03:00
Leonard Lyubich
2e4a1cb6df [#1518] Upgrade NeoFS SDK Go with changed subnet package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-15 17:02:49 +03:00
Evgenii Stratonikov
f602d05b0a [#1494] *: Fix linter warnings
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 12:26:10 +03:00
Evgenii Stratonikov
bbf8b8e74d [#1494] services/object: Do not ignore bearer token decode errors
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 12:26:10 +03:00
Evgenii Stratonikov
795d1e0789 [#1494] go.mod: Update neofs-sdk-go
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 12:26:10 +03:00
Evgenii Stratonikov
aab6094a7c [#1515] neofs-node: Cache max object size
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 11:14:37 +03:00
Evgenii Stratonikov
9269ed344d [#1510] placement: Cache container nodes
In case we have lots of objects in a single container,
`GetContainerNodes` invoked indirectly by a policer can be seen in
pprof.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-14 20:17:43 +03:00
Evgenii Stratonikov
0dad0114f0 [#1511] CHANGELOG.md: Fill unreleased section
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-10 16:21:07 +03:00
Pavel Karpy
8c77387982 [#1509] morph: Log WS client closing error
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-10 12:53:44 +03:00
Pavel Karpy
1708cc0cc7 Fix changelog
It contained duplicated `Fixed`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-09 16:17:30 +03:00
Evgenii Stratonikov
3db4526437 [#1498] neofs-adm: Save wallets as pretty JSON
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-09 16:16:46 +03:00
Evgenii Stratonikov
9e430abf83 [#1498] go.mod: Update neo-go to master
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-09 16:16:46 +03:00
Evgenii Stratonikov
38772e1a2e writecache: provide timeout to bbolt.Open
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-09 14:19:47 +03:00
Pavel Karpy
68a0112193 Release v0.28.3
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-08 14:21:58 +03:00
Pavel Karpy
9f2dd441c3 [#1466] Update changelog
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-08 11:06:19 +03:00
Pavel Karpy
4ed3063212 [#1466] adm: Remove notary flag in for subnet command
It is possible to deduce that information from the network.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-08 11:06:19 +03:00
Pavel Karpy
815e02f185 [#1466] adm: Do not use morph client in adm
Using morph client leads to requiring WebSocket endpoints.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-08 11:06:19 +03:00
Evgenii Stratonikov
60789dd475 [#1489] neofs-adm: Include alphabet contracts to the group
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-08 11:01:16 +03:00
Evgenii Stratonikov
91ed0d20ff [#1417] go.mod: Update neo-go to v0.99.0
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-08 10:27:49 +03:00
Leonard Lyubich
fa009db140 [#1464] ir/container: Fix verifying the operations within sessions
In previous implementation `verifySignature` method of container
processor worked incorrectly for operations without a key and with
session: processor tried to verify signature with one of the bound owner
keys instead of session one.

Use `VerifySessionDataSignature` method to check the signature if
session is used. Refactor `verifySignature` a bit with session check
highlighting for readability.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-07 23:26:30 +03:00
Leonard Lyubich
f6d121a4e4 [#1485] Upgrade NeoFS SDK Go module
New version includes the fix of the session signing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-07 16:34:15 +03:00
Leonard Lyubich
0937513c14 [#1485] ir/container: Accept eACL only if extension is allowed
In order to extend container ACL `F` bit must be set in basic ACL.

Make `Container` contract processor to deny eACL tables bound to
non-extendable containers.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-07 16:34:15 +03:00
Evgenii Stratonikov
af7d15cc1a [#1487] innerring: Log errors in processors
Log errors for network operations. The only places where we are not
interested in errors are `Submit` in pool and unmarshaling.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-07 14:11:58 +03:00
Pavel Karpy
79d72a6713 [#1460] Update the changelog
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
010253a97a [#1460] blobovnicza: Do not use pointers as the results
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
0e4a1beecf [#1460] blobstor: Do not use pointers as the results
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
08bf8a68f1 [#1460] engine: Do not use pointers as the results
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
7b6363f4c6 [#1460] shard: Do not use pointers as the results
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
9b2932609b [#1460] meta: Do not use pointers as the results
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
a580429996 [#1460] meta: Add a benchmark on Get operation
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
b0c7b7851a [#1418] blobstor: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
14366bbd89 [#1418] engine: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
5f57db6bf8 [#1418] shard: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
6d4d920bad [#1418] meta: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Evgenii Stratonikov
a46f585fb3 [#1074] neofs-cli: Inline some of the functions
They are quite small, however will produce problems during subsequent refactoring.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-06 13:54:29 +03:00
Evgenii Stratonikov
9efec21d34 [#1074] neofs-cli: Move session preparation to modules/session package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-06 13:54:29 +03:00
Evgenii Stratonikov
a2bcb3e0ce [#1074] neofs-cli: Move common API flags to a separate package
TTL and XHeader flags are reused between multiple commands.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-06 13:54:29 +03:00
Evgenii Stratonikov
736e09a70d [#1074] neofs-cli: Move PrettyPrintUnixTime to the common package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-06 13:54:29 +03:00
Evgenii Stratonikov
b432558aaa Release v0.28.2
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 17:38:55 +03:00
Leonard Lyubich
72708296cc Upgrade NeoFS SDK Go to v1.0.0-rc.4 and NeoFS API Go to v2.12.2
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-03 17:17:50 +03:00
Evgenii Stratonikov
c41d9c3fbe [#1474] neofs-node/config: Remove "shard_num" from the "grpc" section
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:44:06 +03:00
Evgenii Stratonikov
73a88c2965 [#1474] neofs-node/config: Remove num from the "grpc" section
It isn't needed and can be replaced with additional checks during the
iteration.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:44:06 +03:00
Leonard Lyubich
0e28902b0f [#1471] eacl: Fix incorrect request denial with incomplete object header
Node shouldn't perform eACL verification during GET/HEAD request
processing until full object header is received. Otherwise, for some
eACL tables request may be falsely rejected.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-03 14:42:54 +03:00
Leonard Lyubich
c89035d544 [#1471] eacl: Add testcase with incomplete list of object headers
Scenario:
 * HEAD request of some object
 * 1st eACL record allows op for objects with specific user attribute
 * 2nd eACL record forbids op by object ID
 * node doesn't store the requested object locally

With this scenario node shouldn't deny request.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-03 14:42:54 +03:00
Leonard Lyubich
f9504c1cba [#1471] eacl: Don't process object headers twice in response processing
It is redundant to process object headers in responses w/o object field
since result will be the same.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-03 14:42:54 +03:00
Leonard Lyubich
8b94cb22d3 [#1471] eacl: Prevent aborting on local storage failure
Request processing should not be interrupted in case of local storage
failure since error case in normal for relay nodes.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-03 14:42:54 +03:00
Leonard Lyubich
a97ea2aa21 [#1471] eacl: Add testcase with local storage failure
ACL service should not deny request on local storage failure since in
this case relay nodes won't be able to continue the operation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-03 14:42:54 +03:00
Pavel Karpy
fdd54b0a03 [#1441] ir: Do not validate subnet removal
It is useless process since subnet owner is able to delete subnet without an
Alphabet approval. The Alphabet should only validate netmap state after
removal:
1. Update nodes' attributes if they were included in the deleted subnet;
2. Remove nodes without any subnet entrance.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 14:36:58 +03:00
Evgenii Stratonikov
feef9a98f7 [#1462] writecache: remove useless logs
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
2ae7c94cd6 [#1462] *: Remove log.With invocations
`log.With` is suitable during initialization, but in other places it induces
some overhead, even when branches with logging are not taken.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
43e776dfb1 [#1462] writecache: Persist objects in batches
`Update` becomes a botleneck under a heavy load.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
6ad87e7959 [#1462] writecache: Fill flush marks during startup
Some of the objects are already flushed, don't do it twice.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
f2a7503964 [#1462] fstree: Allow to fetch file content lazily
If we should process address based on some condition, there is no need
to read file content in memory.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
54d4503701 [#1462] writecache: Rename flushWorkersCount
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
0148209168 [#1462] writecache: Use Timer instead of Ticker in flush
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
5073a37930 [#1433] metabase: Optimize Select
For some filters we can scan only a subset of keys instead of checking
each key.

```
name                       old time/op  new time/op  delta
Select/string_equal-8      49.3µs ± 4%  11.0µs ± 4%  -77.68%  (p=0.000 n=10+10)
Select/string_not_equal-8  7.01ms ± 5%  7.06ms ±10%     ~     (p=0.971 n=10+10)
Select/common_prefix-8      118µs ± 6%    79µs ± 5%  -33.04%  (p=0.000 n=10+9)
Select/unknown-8           21.3µs ± 4%   3.2µs ± 4%  -84.88%  (p=0.000 n=10+9)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:25:07 +03:00
Evgenii Stratonikov
5c48588c64 [#1433] metabase: Add more tests for Select
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:25:07 +03:00
Evgenii Stratonikov
ea76c989ca [#1433] metabase: Add a benchmark for Select
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:25:07 +03:00
Evgenii Stratonikov
b602fff01f [#1463] blobovnicza: Remove pointers from struct members
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 13:42:54 +03:00
Evgenii Stratonikov
d9f0ac8909 [#1463] blobovnicza: Determine db size via os.Stat
Currently we use `(*bbolt.Bucket).Stats().KeyN` for estimating database
size. However, it iterates over all pages in bucket and thus heavily
depends on the bucket size. This commit replaces initial size estimation
with a single `os.Stat` call.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 13:42:54 +03:00
Pavel Karpy
021aa97965 [#1418] blobovnicza: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Pavel Karpy
281befec67 [#1418] blobstor: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Pavel Karpy
babd382ba5 [#1418] engine: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Pavel Karpy
6e752f36dc [#1418] shard: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Pavel Karpy
e265ce2d52 [#1418] meta: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Pavel Karpy
1c100fb4b0 [#1418] writecache: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Pavel Karpy
da3ae202f0 [#1436] node: Log service initialization/boot up
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-02 16:16:39 +03:00
Evgenii Stratonikov
d84d52924a [#1467] pkg/util: Allow to load default account from a wallet
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-02 16:13:20 +03:00
Leonard Lyubich
2da8396a9f [#1454] cli/util: Support container sessions in sign session-token
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-01 17:41:45 +03:00
Leonard Lyubich
ff576d125d [#1454] container: Fix reading session token from request origin
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-01 17:41:45 +03:00
Leonard Lyubich
1c30414a6c [#1454] Upgrade NeoFS SDK Go module with new IDs
Core changes:
 * avoid package-colliding variable naming
 * avoid using pointers to IDs where unnecessary
 * avoid using `idSDK` import alias pattern
 * use `EncodeToString` for protocol string calculation and `String` for
  printing

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-01 17:41:45 +03:00
Leonard Lyubich
cc6209e8a0 [#1449] object/delete: Fix NPE in collectChain
`head.previous` method returns (nil, nil) if object has no previous
neighbor.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-30 17:59:16 +03:00
Leonard Lyubich
96cdc04705 [#1449] policer: Unwrap status HEAD response
Helper function `client.IsErrObjectNotFound` doesn't support error
unwrapping, so we need to do it on caller side.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-30 17:59:16 +03:00
Leonard Lyubich
e96ea4635c [#1449] policer: Fix selection of new storage candidates
`Policer` should pass list of selected candidates into `WithNodes`
method of `replicator.Task`. In previous implementation `processNodes`
method passed an opposite list: failed nodes and/or the local one.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-30 17:59:16 +03:00
Evgenii Stratonikov
855cbf5a3a [#1159] services/container: Remove ContainerWithContext struct
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-30 15:40:22 +03:00
Leonard Lyubich
2bcc0051ab [#1423] session: Get session issuer from token structure
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-27 15:45:33 +03:00
Leonard Lyubich
2dec3a0a93 [#1423] ir/container: Fix error message related to container removal
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-27 15:45:33 +03:00
Leonard Lyubich
883f2f1ac6 [#1423] local storage: Activate skipped unit tests after fix
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-27 15:45:33 +03:00
Leonard Lyubich
4c8ec20e32 [#1423] session: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-27 15:45:33 +03:00
Evgenii Stratonikov
dda56f1319 [#1437] neofs-cli: Move util command to a separate package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-26 12:03:39 +03:00
Pavel Karpy
786c920fb2 [#1430] go.mod: Update neo-go
v0.98.2 => v0.98.3

That patch includes fixing panic when using WS client with a closed
connection.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-25 13:54:58 +03:00
Pavel Karpy
57c5fccc8c [#1428] node/acl: Make OID optional
Not all the NeoFS requests must contain OID in their bodies (or must NOT
contain them at all). Do not pass object address in helper functions, pass
CID and OID separately instead.
Also, fixed NPE in the ACL service: updated SDK library brought errors
when working with `Put` and `Search` requests without OID fields.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-25 12:11:03 +03:00
Pavel Karpy
d69eb2aaf3 [#1428] node/acl: Make container ID as required param
Change pointer to value in request information since requests could not
exist without container ID.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-25 12:11:03 +03:00
Evgenii Stratonikov
6cb9c13c5e [#1424] neofs-cli: Fail immediately if a client can't be created
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-25 09:46:41 +03:00
Evgenii Stratonikov
295ec3700a [#1424] neofs-cli: Fail immediately if a key can't be fetched
If the key can't be fetched, an error is always returned, so it makes
sense to fail the whole command inside of a `key.Get*()`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-25 09:46:41 +03:00
Evgenii Stratonikov
d9c5ca5e77 [#1381] neofs-cli: Move control command to a separate module
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-25 09:46:41 +03:00
Evgenii Stratonikov
71d823f192 [#1381] neofs-cli: Move JSON and verbose printing to a common package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-25 09:46:41 +03:00
Evgenii Stratonikov
c57114def3 [#1422] morph/client: Fetch group key properly
NNS contract `resolve` method returns an array.
`neofs-adm` is already up to date.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-23 19:24:46 +03:00
Leonard Lyubich
86552cf3ae engine: make corruptSubDir to ignore blobovnicza dir
There is a dir for Blobovnicza tree in BlobStor, so we need to ignore
it.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-23 15:33:14 +03:00
Leonard Lyubich
bb25ecbd15 [#1400] owner: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-23 15:33:14 +03:00
Leonard Lyubich
f8ac4632f8 [#1335] policer: Prevent potential object loss
In previous implementation `Policer` considered local object copy as
redundant on processing single placement vector.

Make `Policer` to call redundant copy callback after full placement
processing. Also fix 404 error parsing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-23 15:24:23 +03:00
Pavel Karpy
9f62d25b50 [#1224] ir: Delete nodes that do not have subnets
Also, update their subnet entrance on subnet removal events.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-23 08:16:24 +03:00
Pavel Karpy
b3e1e90c01 [#1224] ir: Listen to subnet removal events
IR must listen to subnet removal notifications always (regardless of notary
mode).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-23 08:16:24 +03:00
Pavel Karpy
71f18ba9ec [#1224] morph: Call Alphabet methods flexibly
Call `UpdateStateIR` and `AddPeerIR` method instead of `UpdateState` and
`AddPeer` if calling client is configured as Alphabet in notary enabled
environment.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-23 08:16:24 +03:00
Pavel Karpy
a129fc98da [#1224] morph: Add configuration getters
Includes `WithNotary` and `IsAlpha` methods to the `StaticClient`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-23 08:16:24 +03:00
Pavel Karpy
f0c9a68ad3 [#1224] morph: Delete wrong comment
Current implementation does not use MurmurHash3.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-23 08:16:24 +03:00
Evgenii Stratonikov
df48ddcb3d Revert "[#1141] morph/client: Allow to use more integer types as arguments"
This reverts commit 9349f422fd9a29ec529ef0648bbfabd1f9ff7651.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 16:45:15 +03:00
Evgenii Stratonikov
4761857fb3 [#1396] cmd: Allow to autogenerate documentation for all commands
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 13:35:22 +03:00
Evgenii Stratonikov
c6325fdc91 [#1379] neofs-cli: Move accounting command to a separate package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 13:20:34 +03:00
Evgenii Stratonikov
56f33436dd [#1379] neofs-cli: Move exitOnErr to internal package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 13:20:34 +03:00
Evgenii Stratonikov
094534e31a [#1379] neofs-cli: Add key.GetOrGenerate helper
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 13:20:34 +03:00
Evgenii Stratonikov
cd46a7478e [#1379] neofs-cli: Bind key-related arguments to viper
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 13:20:34 +03:00
Evgenii Stratonikov
a198a4858c [#1379] neofs-cli: Move client creation to internal/client
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 13:20:34 +03:00
Evgenii Stratonikov
cbc4ca800d [#1379] neofs-cli: Move common flags to a separate package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 13:20:34 +03:00
Pavel Karpy
4be5dce848 [#1405] ir: Fix settlement log message
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-20 12:40:12 +03:00
Pavel Karpy
b11f35a5c7 [#1403] CHANGELOG: Update changelog
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-20 12:40:12 +03:00
Evgenii Stratonikov
3f2889a4e9 [#1392] eacl/v2: Use EncodeToString where possible
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 12:04:10 +03:00
Evgenii Stratonikov
63cdc16088 [#1386] eacl/v2: Do not allocate intermediate slices for XHeaders
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 12:04:10 +03:00
Evgenii Stratonikov
915f87959e [#1386] eacl/v2: Allocate []eaclSDK.Header slice immediately
There is no point in having intermediate `[]sessionSDK.XHeader`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 12:04:10 +03:00
Evgenii Stratonikov
b24589b62d [#1386] eacl/v2: Use raw structs where possible
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 12:04:10 +03:00
Evgenii Stratonikov
3f4475f97b [#1386] services/object: Fail eACL check if the request is invalid
Parse all headers beforehand and reject invalid requests.
Another approach would be to remember the error and check
it after `CalculateAction`, which is a bit faster.
The rule of thumb here is "first validate, then use".

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 12:04:10 +03:00
Pavel Karpy
f99a0498da [#1403] util: Add details to errors
Make error messages more descriptive when parsing LOCODE from string errors
appear.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-20 10:53:27 +03:00
Leonard Lyubich
aeb9884218 [#1389] crypto: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-19 10:12:14 +03:00
Leonard Lyubich
5e50ddd7f5 [#1389] storagegroup: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-19 10:12:14 +03:00
Evgenii Stratonikov
0f376a5d83 [#1388] services/control: Extend tests for ShardInfo marshaling
It is nice to have different paths for different components and also
check that the information returned is different for different shards.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-17 13:48:07 +03:00
Leonard Lyubich
f15e6e888f [#1377] oid, cid: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-16 15:33:22 +03:00
Evgenii Stratonikov
f65898a354 [#1141] morph/client: Use boolean type directly
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-16 15:09:35 +03:00
Evgenii Stratonikov
230a5cd037 [#1141] morph/client: Remove unneeded argument conversions
Client supports `util.Uint160` directly, no need to convert it to bytes.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-16 15:09:35 +03:00
Evgenii Stratonikov
1e8391d216 [#1141] morph/client: Allow to use more integer types as arguments
Also, use `*big.Int` as integer value, see nspcc-dev/neo-go#2413.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-16 15:09:35 +03:00
Evgenii Stratonikov
1c7195666c [#722] neofs-adm: Unify pasword retrieval functions
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-16 15:07:31 +03:00
Evgenii Stratonikov
b685af487f [#722] neofs-adm: Replace path with filepath package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-16 15:07:31 +03:00
Evgenii Stratonikov
5bd6624eb6 [#722] neofs-adm: Provide contract state to getContractDeployParameters
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-16 15:07:31 +03:00
Evgenii Stratonikov
efb6545bfe [#722] neofs-adm: Allow to initialize local dump
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-16 15:07:31 +03:00
Evgenii Stratonikov
c55950bd70 [#722] neofs-adm: Check returned tx hash
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-16 15:07:31 +03:00
Evgenii Stratonikov
37f9d083fb [#722] neofs-adm: Hide N3 client behind an interface
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-16 15:07:31 +03:00
Evgenii Stratonikov
57200e18cd [#1375] neofs-adm: Do not update NNS group if the key is the same
If the group key is already set, do not send any transactions.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-13 12:31:46 +03:00
Leonard Lyubich
3a188bb2e5 [#1371] bearer: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-12 15:16:17 +03:00
Leonard Lyubich
ae92074272 [#1369] checksum: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-12 09:48:31 +03:00
Leonard Lyubich
088df0e2a9 [#1369] version: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-12 09:48:31 +03:00
Leonard Lyubich
7a57d7b076 [#1369] audit: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-12 09:48:31 +03:00
Evgenii Stratonikov
875f0e79a2 [#1346] neofs-adm: do not ask password twice
When a contract wallet is created, there is no need to decrypt
newly-created accounts.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-11 12:01:11 +03:00
tasselsd
379c70cc90 [#1363] Use network package to parse and verify the announced address.
Signed-off-by: tasselsd <tasselsd@outlook.com>
2022-05-11 11:22:34 +03:00
Alex Vanin
8f1a0fb6e7 [#1358] Update CHANGELOG.md
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-05-11 11:15:54 +03:00
Alex Vanin
75b6f854e1 [#1358] example: Update README.md
Mention Inner ring and CLI applications.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-05-11 11:15:54 +03:00
Alex Vanin
5988653fc6 [#1358] example: Drop comments in ir.env file
Field descriptions are available in ir.yaml file.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-05-11 11:15:54 +03:00
Alex Vanin
d6be5d4087 [#1358] example: Add configs for inner ring application
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-05-11 11:15:54 +03:00
Elizaveta Chichindaeva
cc7a723d77 [#1320] English Check
Signed-off-by: Elizaveta Chichindaeva <elizaveta@nspcc.ru>
2022-05-11 10:40:02 +03:00
Leonard Lyubich
d99800ee93 Release v0.28.1
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-05 11:55:35 +03:00
Leonard Lyubich
cd545f0160 [#1351] cli: Fix connection scheme loss during endpoint parsing
In previous implementation NeoFS CLI app used `network.Address.HostAddr`
as a server URI, which caused scheme loss since host address doesn't
contain it.

Rename `HostAddr` to `URIAddr` and make it to return URI address with
`grpcs` scheme if TLS is enabled. Make `TLSEnabled` unexported since it
was used to provide default `tls.Config` only (it is used by default in
SDK).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-05 10:43:27 +03:00
Leonard Lyubich
3a44010180 Release v0.28.0 - Heuksando (흑산도, 黑山島)
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-04-29 19:05:02 +03:00
Evgenii Stratonikov
a4769d8624 *: use require.ErrorIs where possible
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-29 16:47:57 +03:00
Evgenii Stratonikov
3c39e6df11 [#1319] blobstor: Compress big objects in a streaming fashion
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-29 16:46:40 +03:00
Evgenii Stratonikov
1219ff89d4 [#1319] services/object_manager: Fix error message
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-29 16:46:40 +03:00
Evgenii Stratonikov
057d53459b [#1319] services/object: Remove Write method from local and remote targets
They work with prepared objects only.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-29 16:46:40 +03:00
Evgenii Stratonikov
4ea03c01b5 [#1319] services/object: Store payload directly in distributedTarget
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-29 16:46:40 +03:00
Pavel Karpy
9da7df4d42 [#1318] metabase: Init metabase in test
`Init()` creates buckets. It prevents NPE in tests.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
486ec8a2e5 [#1318] metabase: Reuse comparing func from require
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
1caeb0b648 [#1318] metabase: Add missing error check in tests
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
6f2723b47e [#1318] node: Fix linter
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
2aec5736e0 [#1318] node/gc: Fix tombstones handling
Handle only expired tombstones.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
a275a71a87 [#1318] node: Use new tombstone handling
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
24ca8ca5c7 [#1318] object: Change default ttl
This is need to reuse object service without passing any API v2 structs.
Also, fix comment.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
7799f8e4cf [#1318] engine: Change tombstone clear process
- Delete objects physically on tombstone's arrival;
- Store information about tombstones in the Graveyard;
- Clear Graveyard every epoch based on the information about TS in the
network.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
e4cfeec449 [#1318] service: Add tombstone checker service
The service fetches tombstones from the network via object service, every
request is handled in the following order:
1. checks local LRU cache;
2. checks local storage engine;
3. tries to find object in the placement nodes.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
2583f608e8 [#1318] metabase: Add DropGraves method
Method clean up Graveyard and should be called by the routine that handles
tombstone expiration.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
daab30c391 [#1318] metabase: Add iteration offset
Add offset element to the iterations over deleted objects (both the
Graveyard and the Garbage buckets).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
fe8076e60a [#1318] metabase: Add immediate object deletion
Mark objects with GC immediately after a Tombstone is received.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
8107c8d1a9 [#1318] metabase: Separate buckets with TS and GC marks
It allows storing information about object in both ways at the same time:
1. Metabase should know if an object is covered by a tombstone (that is
not expired yet);
2. It should be possible to physically delete objects covered by a
tombstone immediately (mark with GC) but keep tombstone knowledge.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Evgenii Stratonikov
b74fb2b932 [#1316] neofs-cli: Print object ID in base-58
It was missed from dd9bd05ba.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-25 10:54:18 +03:00
Evgenii Stratonikov
5736b834c3 [#1286] services/object: Validate object when created with sesssion token
All fields set by a user should be verified.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-25 10:53:22 +03:00
Evgenii Stratonikov
c6a9c5cd8c [#1283] services/object: Disallow creating objects without a session token
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-25 10:53:22 +03:00
Evgenii Stratonikov
a254fd6bc8 [#1216] neofs-cli: Allow to use session token from file
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-25 10:47:43 +03:00
Evgenii Stratonikov
a95bdb1811 [#1216] neofs-cli: Allow to create and save session token
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-25 10:47:43 +03:00
Evgenii Stratonikov
8d79168129 [#1216] neofs-cli: Reuse key retrieving code between modules
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-25 10:47:43 +03:00
Evgenii Stratonikov
006d6e8b48 [#1216] neofs-cli: Reuse GetSDKClient between CLI modules
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-25 10:47:43 +03:00
Evgenii Stratonikov
fd8c00400d [#1216] neofs-cli: Rename token subcommand to bearer
It works only with the bearer token.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-25 10:47:43 +03:00
Pavel Karpy
bd27837364 [#1321] node: Register GC event channel before shard init
Morph "NewEpoch" event handling was registered in a closure over
`addNewEpochNotificationHandler` func. That may lead to the data race:
if a shard was initialized before the event registration, everything works
as planned, but if registration was made earlier, it was not able to
include GC handlers since a shard has not called `eventChanInit` yet and,
therefore, it has not registered handler yet.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-25 10:39:37 +03:00
Leonard Lyubich
d4569946c5 [#1313] container: Do not invalidate cache on Container.Delete
In previous implementation `Container.Delete` operation caused local
node's cache invalidation (container itself, eACL and listings). Any
subsequent `Container.Get` operation reversed invalidation. Given the
low latency sensitivity of deleting a container, there is no need to
touch the cache. With this approach, all pending deletion operations on
the node via the NeoFS API protocol will be delayed by the cache TTL.

Do not call cache invalidation ops in `morphContainerWriter.Delete`.
Remove no longer needed `InvalidateContainerListByCID` and
`InvalidateContainer` methods.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-04-25 10:34:12 +03:00
Evgenii Stratonikov
606dfa3414 [#1308] neofs-adm: Add command to dump GAS balances
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-12 17:37:40 +03:00
Evgenii Stratonikov
f1e91313db [#1311] neofs-adm: Set stdout as default output
Cobra `Command` prints to stderr by default.
`neofs-cli` already sets the output properly.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-12 17:37:26 +03:00
Evgenii Stratonikov
6c08ec9ca2 [#1312] morph/netmap: Fix panic during parameter parsing
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-12 17:37:12 +03:00
Alex Vanin
8054311eee [#1170] neofs-adm: Update storage config template
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-04-12 17:36:37 +03:00
Alex Vanin
6da8c6635f Release candidate v0.28.0-rc.3
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-04-08 16:35:22 +03:00
Alex Vanin
2fd2f9b0bc Update go mod
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-04-08 16:35:22 +03:00
Pavel Karpy
9ac2f4b5a4 [#1170] CI: Update golang linter
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy
b6e5aac604 [#1170] pkg/morph: Delete unused function
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy
01e69f2f7a [#1170] pkg/morph: Add worker pool
Add worker pool to the listener to prevent blocking. It is used only for
notary notifications and new block events handling since it uses RPC
calls. That may lead to the deadlock state: neo-go cannot send RPC until
notification channel is read but notification channel cannot be read since
neo-go client cannot send RPC.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy
4ed5b1ceef [#1170] pkg/morph: Invalidate cache on RPC switch
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy
6508136204 [#1170] ir, node: Stop the app on WS connection lost
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy
2427593da6 [#1170] pkg/morph: Add inactive mode callback
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy
424cc6d495 [#1170] go.mod: Update neo-go
github.com/nspcc-dev/neo-go v0.98.0 => v0.98.2

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy
fdf1338d65 [#1170] config: Delete notification endpoints
Also, change `http` to `ws` in examples.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy
3e45b4a085 [#1170] pkg/morph/subscriber: Adopt new WS client
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy
402f488bec [#1170] pkg/morph: Change HTTP for WS client
Updated client now supports subscription to chain notifications and RPC
switch between provided RPC endpoints.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy
71c75dc7e8 [#1170] cmd/node: Remove useless function variable
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Evgenii Stratonikov
622ea4818f [#1250] *: Remove io/ioutil imports
It is deprecated starting from go1.16.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-07 17:23:28 +03:00
Evgenii Stratonikov
3bdab77c42 [#1250] go.mod: Bump supported go version to 1.17
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-07 17:23:28 +03:00
Evgenii Stratonikov
e8bf18c0b4 [#1208] innerring: Fix race condition between handling new epoch and block
Before resetting the timer, ensure the block tick is processed.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-07 11:52:24 +03:00
Evgenii Stratonikov
4770cb8bf6 [#1208] innerring: Disallow to tick timer twice on the same height
Provide current heights as an argument to ticker.
Zero height disables any checks, thus corresponding to the old
behaviour. If non-zero height is used, ignore the tick if the height
is less than the timer tick state.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-07 11:52:24 +03:00
Evgenii Stratonikov
c0e65dadaf [#1292] morph/event: Make listener methods use pointer receiver
After `Listen` has raw receiver and calls `listen` which has pointer
receiver, leading to ineffectual assign to `started` field.
Always use `listener` by pointer, to avoid similar bugs.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-07 11:48:57 +03:00
Evgenii Stratonikov
bc0c47738b [#1295] neofs-adm: Add contract group wallet to config template
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-07 11:46:20 +03:00
Evgenii Stratonikov
82fda42316 [#1294] neofs-adm: Use Global scope where needed
Provide explicit argument to `sendCommitteeTx` signifying whether a tx
should try to use group signer.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-05 18:23:14 +03:00
Evgenii Stratonikov
7d670129c9 [#1294] neofs-adm: Emit ASSERT after the transfers
NEP-17 transfer method returns boolean value signifying if the operation
was successful. Fail the transaction if it was not.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-05 18:23:14 +03:00
Evgenii Stratonikov
55362f0607 [#1294] neofs-adm: Check persisted tx status
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-05 18:23:14 +03:00
Leonard Lyubich
629a4f79cd [#1293] metabase: Fix out-of-range panic in freePotentialLocks
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-04-05 15:38:51 +03:00
Alex Vanin
e62e02815b [#1291] listener: Stop only once
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-04-05 14:46:40 +03:00
Alex Vanin
77f4a5844b [#1291] innerring: Don't stop the same listener twice
When IR node configured without main chain, both
`morphListener` and `mainnetListener` are pointing
into single listener component. We should not call
`Stop()` twice, because it may trigger channel
closing in neo-go or other components and it
can throw panic.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-04-05 14:46:40 +03:00
Evgenii Stratonikov
d311585de6 [#1289] neofs-adm: Allow to change parameters of the Policy contract
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-05 13:28:42 +03:00
Evgenii Stratonikov
a5cf38dcbf [#1288] neofs-adm: Allow to store garbage in alphabet wallet dir
Check the full file name instead of just prefix. Also, fix a bug where
a single missing wallet could lead to an incorrect size calculation.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-05 13:11:50 +03:00
Evgenii Stratonikov
923db59149 [#1213] morph/client: Remember last succesfully used client
Prevent attemts to connect to the failing endpoint multiple times.
This speeds up all invocations a bit and removes excessive error logs.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-01 17:07:12 +03:00
Evgenii Stratonikov
9bbb136e4a [#1213] morph/client: Protect cached values with a mutex
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-01 17:07:12 +03:00
Evgenii Stratonikov
a8d2001b35 [#1213] morph/client: Use a separate cache for every client
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-01 17:07:12 +03:00
Evgenii Stratonikov
61f60b8461 [#1213] morph/client: Unlock mutex to perform network requests
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-01 17:07:12 +03:00
Evgenii Stratonikov
48eb87d32c [#1149] writecache: Prevent corruption in Head
Also, remove optimization comments:
1. Having to maintain an execute the same logic for headers as for
   objects is quite inefficient, as it increases memory footprint.
2. Unmarshaling object is a cheap operation if data slice is in memory.
3. For unmarshaling header-only, I think we need SDK support.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:40:25 +03:00
Evgenii Stratonikov
ff1912aa2a services/acl: check session token expiration epoch
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:37:29 +03:00
Evgenii Stratonikov
68903c9fd9 [#1143] shard: Support degraded mode in Get and GetRange
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
Evgenii Stratonikov
f058cead8f [#1143] shard: Handle some errors in degraded mode
If metabase is corrupted for some reason, failback to
checking blobstor directly.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
Evgenii Stratonikov
08e7914729 [#1143] blobstor: Implement existsSmall check
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
Evgenii Stratonikov
aa0cc1f824 [#1143] blobovnicza: Copy object data in Get
Data returned from `*bbolt.Bucket.Get()` is only valid for the lifetime
of the transaction.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
Evgenii Stratonikov
6472a170eb [#1143] shard: Introduce explicit Degraded mode
`Degraded` mode is set automatically after error counter is over the
threshold. `ReadOnly` mode can still be set by an administrator.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
Alex Vanin
9eb70c18c3 Backport release v0.27.7 changelog
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-30 15:40:13 +03:00
Alex Vanin
e4a8ed589b [#1278] neofs-node: Cache IRFetcher
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-30 14:22:12 +03:00
Alex Vanin
be6ae3c066 [#1278] neofs-node: Use global cached netmap source in services
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-30 14:22:12 +03:00
Alex Vanin
7ed84d1755 [#1278] acl: Return netmap.Source interface
Application can provide cached netmap source in this case.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-30 14:22:12 +03:00
Evgenii Stratonikov
cf119e4ca9 [#1163] services/audit: Randomize the order of PDP checks
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-30 10:56:33 +03:00
Evgenii Stratonikov
dd9bd05bac [#1239] neofs-cli: Use pointer-less slices for object ID
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-29 16:02:05 +03:00
Evgenii Stratonikov
882236a03b [#1239] morph/client: Remove intermediate conversion in morph client
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-29 16:02:05 +03:00
Evgenii Stratonikov
6936195afa [#1239] util/attributes: Remove excessive slice copy during parsing
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-29 16:02:05 +03:00
Evgenii Stratonikov
2ad8016d75 [#1239] innerring: Use pointer-less slices for object IDs
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-29 16:02:05 +03:00
Pavel Karpy
6ec104d686 [#1255] node/session: Rename constant
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-29 09:35:10 +03:00
Pavel Karpy
90a8c52bdb [#1255] object: Add persistent storage usage
Use persistent storage usage in the node if it was configured so.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-29 09:35:10 +03:00
Pavel Karpy
9cda3121ab [#1255] node/config: Add persistent storage
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-29 09:35:10 +03:00
Pavel Karpy
016eaa25f3 [#1255] node/session: Add encryption tests
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-29 09:35:10 +03:00
Pavel Karpy
01ed366e99 [#1255] node/session: Add encryption
Add `WithEncryption` option that passes ECDSA key to the persistent session
storage. It uses 32 bytes from marshalled ECDSA key in ASN.1 DER from in
AES-256 algorithm encryption in Galois/Counter Mode.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-29 09:35:10 +03:00
Pavel Karpy
a884ad56d9 [#1255] node/session: Add persistent tests
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-29 09:35:10 +03:00
Pavel Karpy
455b9fb325 [#1255] node/session: Add persistent session storage
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-29 09:35:10 +03:00
Pavel Karpy
929c9851a6 [#1255] node/session: Create separate dir for in-memory storage
Move in-memory session storage to the separate directory of `storage`. It is
done for future support of different kind of session storages.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-29 09:35:10 +03:00
Evgenii Stratonikov
2a69aaf976 [#1157] network/cache: Optimize client fetch from multiClient
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 18:11:20 +03:00
Evgenii Stratonikov
a4261243fc [#1157] network/cache: Cache multiclients based on public key only
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 18:11:20 +03:00
Evgenii Stratonikov
de5a2f6574 [#1262] metabase: Remove list index in place Delete
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
Evgenii Stratonikov
43867a3093 [#1262] metabase: Do not allocate intermediate slices for indices
```
name              old alloc/op   new alloc/op   delta
Put/parallel-8       123kB ± 4%     119kB ± 3%  -2.72%  (p=0.006 n=10+9)
Put/sequential-8     170kB ± 1%     168kB ± 1%  -1.42%  (p=0.000 n=10+10)

name              old allocs/op  new allocs/op  delta
Put/parallel-8         473 ± 1%       469 ± 0%  -0.87%  (p=0.000 n=10+10)
Put/sequential-8       792 ± 0%       787 ± 0%  -0.58%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
Evgenii Stratonikov
0e9b6be3fd [#1262] metabase: Remove intermediate allocations in decodeList
```
name              old time/op    new time/op    delta
Put/parallel-8      1.57ms ±11%    1.51ms ± 3%   -4.06%  (p=0.043 n=9+10)
Put/sequential-8    5.16ms ± 2%    5.16ms ± 3%     ~     (p=1.000 n=9+10)

name              old alloc/op   new alloc/op   delta
Put/parallel-8       126kB ± 4%     123kB ± 4%   -2.54%  (p=0.016 n=8+10)
Put/sequential-8     171kB ± 1%     170kB ± 1%     ~     (p=0.182 n=9+10)

name              old allocs/op  new allocs/op  delta
Put/parallel-8         565 ± 2%       473 ± 1%  -16.18%  (p=0.000 n=9+10)
Put/sequential-8       819 ± 1%       792 ± 0%   -3.34%  (p=0.000 n=9+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
Evgenii Stratonikov
d45df614fb [#1262] metabase: Optimize decodeList
Prevent additional allocation during `append` in `Put`.

```
name              old alloc/op   new alloc/op   delta
Put/parallel-8       131kB ± 1%     126kB ± 4%  -3.87%  (p=0.005 n=8+8)
Put/sequential-8     172kB ± 1%     171kB ± 1%  -0.73%  (p=0.028 n=10+9)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
Evgenii Stratonikov
456e1584d6 [#1262] metabase: Add benchmarks for Put
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
Evgenii Stratonikov
f1223b46df [#1262] blobovnicza: Make helper in Put function idempotent
`Batch` can execute the function multiple times leading to multiple
increases of a size approximation.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
Alex Vanin
35ad6f188e Backport release v0.27.6 changelog
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-28 15:16:54 +03:00
Evgenii Stratonikov
571ae843ad [#1198] neofs-cli: Use io.Copy instead of io.CopyBuffer
The buffer size value is somewhat arbitrary and making in configurable
doesn't make much sense, given that we can't really restrict total resource
consumption in other places. `Copy` uses 32 KiB buffer by default, which
is not big. This approach is also more flexible as we can get rid of
buffer completely by implementing `ReaderFrom`, `WriterTo` interfaces.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-25 18:21:48 +03:00
Evgenii Stratonikov
cbe07120da [#1261] neofs-cli: Allow to use relative epoch for bearer token
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-25 17:58:53 +03:00
Evgenii Stratonikov
f2c1bc4bfb [#1261] neofs-cli: Allow to create eACL with empty container ID
Empty CID can be used in bearer token eACL.
See https://github.com/nspcc-dev/neofs-api/issues/207 .

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-25 17:58:53 +03:00
Evgenii Stratonikov
9b2523a408 [#1261] neofs-cli: Allow to create bearer tokens
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-25 17:58:53 +03:00
Evgenii Stratonikov
ae8e38cace [#1261] neofs-cli: Fix help message for acl extended create
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-25 17:58:53 +03:00
Alex Vanin
683439970a [#1270] neofs-node: Add timeout for grpc GracefulStop()
GracefulStop() may be blocked until all server-side streams
are finished. There is no control over such streams yet, so
application may be frozen in shutdown stage.

Naive solution is to add timeout for GracefulStop(). At this
point healthy connection will be finished and unhealthy
connections will be terminated by Stop().

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-25 17:39:24 +03:00
Evgenii Stratonikov
ad92493b86 [#1268] blobstor: Cleanup zstd encoders/decoders
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-25 14:05:06 +03:00
Evgenii Stratonikov
4253931699 Release candidate v0.28.0-rc.2
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-24 17:51:59 +03:00
Alex Vanin
7e06d0aa69 [#1253] ir: Call UpdateStateIR method to remove dead storage node
Alphabet nodes in notary enabled environment cannot call `UpdateState`
method to remove unwanted storage nodes from the network map,
because this method checks witness of the storage node.

To force storage node state update, alphabet nodes should invoke
new method `UpdateStateIR` which is similar to `AddPeerIR`.

State update initiated by the storage node itself is processed
the same way as before -- alphabet nods resign such transaction.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-24 11:52:19 +03:00
Alex Vanin
e81081e0e0 [#1253] ir: Update AddPeer method name for notary enabled env
`Register` was renamed to `AddPeerIR` for consistency with
`UpdateState` changes in
https://github.com/nspcc-dev/neofs-contract/pull/227

This is protocol breaking change for notary enabled environment.
Luckily, there is no notary enabled environments anywhere except
of neofs-dev-env, so we can do such thing. We should avoid such
changes in the future, though.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-24 11:52:19 +03:00
Evgenii Stratonikov
414ba6e0a2 [#1244] nats: Split client creation into 2 stages
Create and connect to an endpoint using separate functions.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-24 11:51:49 +03:00
Evgenii Stratonikov
2b0460c532 [#1233] neofs-cli: Fix split info marshaling
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-23 14:52:49 +03:00
Evgenii Stratonikov
e2062013cf network: remove unused constants
Fix linter complaints. These constants are unused after
nspcc-dev/neofs-node#1232.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-22 11:55:19 +03:00
Alex Vanin
c8b585b991 [#1259] neofs-cli: Use more cmd.PrintErr*()
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-21 19:20:19 +03:00
Alex Vanin
44138adacf [#1259] neofs-cli: Return non-zero exit code in acl extended create command failures
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-21 19:20:19 +03:00
Evgenii Stratonikov
32badab11a [#1252] neofs-cli: Print details for AccessDenied errors
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-21 19:20:01 +03:00
Evgenii Stratonikov
2848001dfb [#1246] object/acl: Return more concise description for eACL errors
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-21 19:20:01 +03:00
Evgenii Stratonikov
800d01e28c [#1233] neofs-cli: Do not print info if output format is strict
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-21 14:52:05 +03:00
Evgenii Stratonikov
5eef0f46c5 [#1233] neofs-cli: Respect format flags for SplitInfo output
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-21 14:52:05 +03:00
Alex Vanin
5c5279688b Release candidate v0.28.0-rc.1
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-18 10:17:08 +03:00
Alex Vanin
7c5cdd2144 Update neofs-sdk-go to latest master
Fixes missing message text in internal errors

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-18 10:17:08 +03:00
Leonard Lyubich
318639e5bf [#1247] *: Clarify docs about returned errors from apistatus package
`apistatus` package provides types which implement build-in `error`
interface. Add `error of type` pattern when documenting these errors in
order to clarify how these errors should be handled (e.g. `errors.Is` is
not good).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-17 16:34:00 +03:00
Leonard Lyubich
459bdcf04b [#1247] object/acl: Return ObjectAccessDenied status error
Return `apistatus.ObjectAccessDenied` error on access violation from ACL
service. Write reason in format of the errors from the previous
implementation. These errors are returned by storage node's server as
NeoFS API statuses.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-17 16:34:00 +03:00
Leonard Lyubich
967650f2ed [#1247] container: Return ContainerNotFound status error
Replace `core/container.ErrNotFound` error returned by `Source.Get`
interface method with `apistatus.ContainerNotFound` status error. This
error is returned by storage node's server as NeoFS API statuses.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-17 16:34:00 +03:00
Leonard Lyubich
70ffdf3478 [#1247] object: Return NOT_FOUND and ALREADY_REMOVED statuses
Replace `ErrNotFound`/`ErrAlreadyRemoved` error from
`pkg/core/object` package with `ObjectNotFound`/`ObjectAlreadyRemoved`
one from `apistatus` package. These errors are returned by storage
node's server as NeoFS API statuses.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-17 16:34:00 +03:00
Leonard Lyubich
f32c9670ad [#1247] object: Return status errors from util.KeyStorage
Return `SessionTokenExpired`/`SessionTokenNotFound` error from
`apistatus` package if private session token is expired/missing. These
errors are returned by storage node's server as NeoFS API statuses.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-17 16:34:00 +03:00
Leonard Lyubich
a4ee59977e [#1247] session: Remove unused storage.ErrNotFound error
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-17 16:34:00 +03:00
Evgenii Stratonikov
737d40f2d9 [#1152] network: Reuse URI parsing function from api-go
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-17 16:33:08 +03:00
Pavel Karpy
52e742bac1 [#1243] node: Add "hot" notifications
Wrap engine with notifications writer (if configured so) to allow sending
notifications right after the object is saved in the local storage.
"Hot" notifications are sent for objects with the following tick epoch
values:
1. 0;
2. *current epoch*.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-17 14:27:52 +03:00
Pavel Karpy
5dba64fcc5 [#1243] node/cfg: Add notification config
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-17 14:27:52 +03:00
Pavel Karpy
16543a1904 [#1243] node/object: Hide engine behind interface
Do not use engine as explicit dependency in the put service. This allows
wrapping storage engine.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-17 14:27:52 +03:00
Pavel Karpy
a4a29f3442 [#1243] node: Rename notifications init func
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-17 14:27:52 +03:00
Pavel Karpy
d02720c910 [#1243] nats: Fix comments
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-17 14:27:52 +03:00
Pavel Karpy
7ebeb4c89b [#1243] node/object: Fix comment
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-17 14:27:52 +03:00
Pavel Karpy
9ce0bbe90f [#1245] cli: Fix linter
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-17 12:09:00 +03:00
Pavel Karpy
5506b7af29 [#1245] cli: Add object notification to CLI
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-17 12:09:00 +03:00
Evgenii Stratonikov
ad4583fe85 [#1185] neofs-cli: Add progress bar to object put/get
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-16 17:28:35 +03:00
Pavel Karpy
c3db12d71b [#1210] reputation: Resolve race condition
Make all epoch independent in reputation process. Do not reset any timers
related to reputation. Make it possible to finish iteration after the
unexpected `NewEpoch` event.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-16 17:18:30 +03:00
Pavel Karpy
77d847dbea [#1210] timers: Add IterationsTicker
It allows specifying duration in blocks and a desired number of handler
calls in that period.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-16 17:18:30 +03:00
Pavel Karpy
13af4e6046 [#1210] reputation: Improve debug logs
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-16 17:18:30 +03:00
Alex Vanin
0adb29c035 [#1214] Fix metabase put logging
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-16 17:17:18 +03:00
Alex Vanin
6ff30a9543 [#1238] Use tagged version of neofs-sdk-go
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-16 15:52:34 +03:00
Alex Vanin
362cda53d2 [#1238] Call String() explicitly instead of using stringer pointer
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-16 15:29:31 +03:00
Alex Vanin
8f476f3c4d [#1238] Adopt neofs-node for non pointer slices in SDK
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-16 15:29:31 +03:00
Alex Vanin
9fad29dfe0 [#1238] Adopt neofs-ir for non pointer slices in SDK
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-16 15:29:31 +03:00
Alex Vanin
7f3195b197 [#1238] Adopt neofs-cli for non pointer slices in SDK
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-16 15:29:31 +03:00
Alex Vanin
8ae869b95e [#1238] Update API libraries
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-16 15:29:31 +03:00
Evgenii Stratonikov
f0ec35478a [#1236] neofs-node: Neofs-node: Remove mutex from ttlNetCache
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-15 17:26:26 +03:00
Evgenii Stratonikov
22b1208a20 [#1236] neofs-node: Remove mutex from lruNetCache
We already use thread-safe LRU and mutex shouldn't be taken while making
network requests.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-15 17:26:26 +03:00
Leonard Lyubich
98c84670e3 [#1175] core/object: Check emptiness of LOCK members
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
9feb5f9405 [#1175] engine: Fix AddShard implementation
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
9dff07200c [#1181] local storage: Process expired locks similar to tombstones
There is a need to process expired `LOCK` objects similar to `TOMBSTONE`
ones: we collect them on `Shard`, notify all other shards about
expiration so they could unlock the objects, and only after that mark
lockers as garbage.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
ebd84f6dd4 [#1175] object/put: Perform additional container broadcast of LOCKs
`LOCK` objects should be broadcast across container like `TOMBSTONE`
ones.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
323dea95c6 [#1181] Do not use deprecated elements
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
e3f4e48d87 [#1175] Fix typo in error return docs
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
d92b1d1bf8 [#1175] CLI: Implement command to lock the objects
Implement `lock` command and add it to `object` section. The command
accepts container argument (string `cid.ID`) and list of locked objects
(string `oid.ID` list). From the provided input `LOCK` object is
constructed and stored using NeoFS API protocol.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
8a2a096680 [#1175] services/sign: Convert error to status return
In previous implementation `SignService` converted all `error` values to
`INTERNAL` server failure status. That was done for simplification only.
There is a need to transmit status errors as corresponding status
messages.

Make `SignService` to unwrap errors and convert them to status message
during writing to the response. Non-status errors are converted to
`INTERNAL` server failures. Status errors can also be wrapped in the
depths of the executable code, so `SignService` tries to unwrap them.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
ced854bc2e [#1175] object/fmt: Handle LOCK objects
Make `FormatValidator.ValidateContent` to verify payload of `LOCK`
objects. Pass locked objects to `Locker` interface. Require from
`Locker.Lock` to return `apistatus.IrregularObjectLock` error on a
corresponding condition.

Also add error return to `DeleteHandler.DeleteObjects` method. Require
from method to return `apistatus.ObjectLocked` error on a corresponding
condition. Adopt implementations.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
d163008b63 [#1175] storage engine: Fix Lock implementation
`Lock` must return `apistatus.IrregularObjectLock` error instead of
`apistatus.ObjectLocked` one.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
07107c8bfd [#1175] metabase: Update locker list during Inhume
`Inhume` operation can potentially mark lockers as garbage. There is a
need to update locker list in locked bucket.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
26b8e33ed7 [#1175] metabase: Fix encoding of updated locker list
Even single locker must be encoded using `encodeList`, or subsequent
`decodeList` call will fail.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
88cefa7c5f [#1175] storage engine: Return status error on Delete of locked object
Make `StorageEngine.Delete` to forward first encountered
`apistatus.ObjectLocked` error during shard processing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
97b6ec8150 [#1175] storage engine: Return status error on Inhume of locked object
Make `StorageEngine.Inhume` to forward first encountered
`apistatus.ObjectLocked` error during shard processing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
545861d1bc [#1175] storage engine: Implement Lock operation
Implement `StorageEngine.Lock` method which works similar to `Inhume`
but calls `Lock` on the processing shards.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
6ed85ff1e1 [#1175] shard: Implement Lock operation
Implement `Shard.Lock` method which required rw mode and calls `Lock` on
underlying metabase.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
e21eedee06 [#1175] shard: Return status error on Inhume of locked object
Make `Shard.Inhume` to forward `apistatus.ObjectLocked` returned by
underlying metabase.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
b585791d6e [#1175] metabase: Return status error on Lock of irregular object
Make `DB.Lock` to return `apistatus.IrregularObjectLock` if at least one
of the locked objects is irregular (not of type REGULAR).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
23fcacd3f2 [#1175] metabase: Return status error on Inhume of locked object
Make `DB.Inhume` to return `apistatus.ObjectLocked` if at least one of
the inhumed objects is locked.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
303eb2e078 [#1176] metabase: Ignore locked objs in IterateCoveredByTombstones
Make `DB.IterateCoveredByTombstones` to not pass locked objects to the
handler. The method is used by GC, therefore it will not consider locked
objects as candidates for deletion even if their tombstone is expired.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
dba3e58dc5 [#1176] metabase: Ignore locked objects in IterateExpired
Make `DB.IterateExpired` to not pass locked objects to the handler. The
method is used by GC, therefore it will not consider them as candidates
for deletion.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
9508633a7e [#1175] metabase: Work with LOCK objects
After introduction of LOCK objects (of type `TypeLock`) complicated
extended its behavior:
  * create `lockers` container bucket (LCB) during PUT;
  * remove object from LCB during DELETE;
  * look up object in LCB during EXISTS;
  * get object from LCB during GET;
  * list objects from LCB during LIST with cursor;
  * select objects from LCB during SELECT with '*'.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
9f13674a10 [#1175] metabase: Implement LOCK operation
Implement `DB.Lock` method which marks list of the objects as locked by
another object. Only regular objects can be locked.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
14d27455f3 [#1175] metabase: Add container bucket for LOCKs
Create class of container buckets with `LOCKED` suffix. Put identifiers
of the objects of type `LOCK` to these buckets during `DB.Put`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
e90ac9c6f9 [#1181] metabase: Remove unused CleanUp method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
637077a883 [#1181] CLI: Simplify object type output
There is no need to switch between object types since `Type` implements
`fmt.Stringer`.

Replace swich-case in `printHeader` with `%s` format notation function.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Evgenii Stratonikov
2181e033b8 go.mod: update dependencies
Fix Dependabot alert.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-15 11:37:55 +03:00
Alex Vanin
b6720d5f97 [#1231] Update new SDK Client interface
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-14 13:18:48 +03:00
Pavel Karpy
697c12a5e9 [#1223] amd: Fix subnet node command
Added `-w` flag.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-10 19:23:39 +03:00
Alex Vanin
e38d7dda6b [#1230] replicator: Improve error log
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-10 19:19:18 +03:00
Leonard Lyubich
14ea7b2c47 [#1219] object/delete: Change approach of the tombstone broadcast
After `putsvc.Service` started to support additional container broadcast
of the saved objects there is no more need to perform broadcast of
tombstone object in `deletesvc.Service`.

Make `putsvc.Service` to perform additional broadcast of `TOMBSTONE`
objects. Remove `broadcastTombstone` stage from `deletesvc.execCtx`,
from now it is encapsulated in `saveTombstone` stage. Remove no longer
needed `putsvc.PutInitPrm.WithTraverseOption` method.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-05 16:23:21 +03:00
Leonard Lyubich
a55af18ad1 [#1219] object/put: Support additional container broadcast
There are several cases when we need to spread the object around the
container after its primary placement (e.g. objects of type TOMBSTONE).
It'd be convenient to support this feature in `putsvc.Service`.

Add additional stage of container broadcast after the object is stored.
This stage is carried out no more than once and does not affect the
outcome of the main PUT operation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-05 16:23:21 +03:00
Evgenii Stratonikov
0bf59522f7 [#1191] object/acl: check session token verb
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-05 16:03:16 +03:00
Pavel Karpy
5e90d85020 [#1183] node: Use NATS client as notification writer
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-05 15:57:24 +03:00
Pavel Karpy
1e96f62294 [#1183] node/config: Add NATS configuration
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-05 15:57:24 +03:00
Pavel Karpy
79c6f52a27 [#1183] node: Add NATS client
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-05 15:57:24 +03:00
Evgenii Stratonikov
51f12ea22b *: Tidy go.mod
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-05 13:43:54 +03:00
Pavel Karpy
03b601b594 [#1183] node: Init notificator on startup
It uses logger as a notification writer. NATS messaging will be implemented
later.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-05 11:41:18 +03:00
Pavel Karpy
46cf15f03c [#1183] node/config: Add notification configuration
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-05 11:41:18 +03:00
Pavel Karpy
90ff066940 [#1183] node/notificator: Implement notificator
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-05 11:41:18 +03:00
Pavel Karpy
7d0e3648a2 [#1183] go.mod: Update neofs-sdk-go
v0.0.0-20220201141054-6a7ba33b59ef => v1.0.0-rc.2

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-05 11:41:18 +03:00
Pavel Karpy
0576db747b [#1183] go.mod: Drop replace section
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-05 11:41:18 +03:00
Pavel Karpy
cf17686321 [#1111] control: Rename aliaces
`objectSDKAddress` -> `addressSDK`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-05 10:06:14 +03:00
Pavel Karpy
99b31e3235 [#1111] object/acl: Refactor service
Make all operations that related to `neofs-api-go` library be placed in `v2`
packages. They parse all v2-versioned structs info `neofs-sdk-go`
abstractions and pass them to the corresponding `acl`/`eacl` packages. `v2`
packages are the only packages that do import `neofs-api-go` library. `eacl`
and `acl` provide public functions that only accepts `sdk` structures.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-05 10:06:14 +03:00
Leonard Lyubich
7ccd1625af [#1214] *: Use single Object type in whole project
Remove `Object` and `RawObject` types from `pkg/core/object` package.
Use `Object` type from NeoFS SDK Go library everywhere. Avoid using the
deprecated elements.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 17:45:00 +03:00
Evgenii Stratonikov
19ad349b27 [#1204] shard: Save ID in the metabase
`AddShard` must return shard id, so we temporarily open metabase
there.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 17:13:13 +03:00
Evgenii Stratonikov
1fe9cd4d36 [#1186] engine: Allow to skip metabase in GetRange
Similarly to `Get`. Also fix a bug where `ErrNotFound` is returned
instead of `ErrRangeOutOfBounds`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 16:07:25 +03:00
Evgenii Stratonikov
2b5550ccf6 [#1186] blobstor: Fix comment
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 16:07:25 +03:00
Evgenii Stratonikov
b33fb0f739 [#1186] blobstor: Unify errors for Get and GetRangeBig
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 16:07:25 +03:00
Evgenii Stratonikov
69e1e6ca20 [#1186] engine: Read object directly from blobstor in case of conflicts
Metabase is expected to contain actual information about objects stored
in shard. If the object is present in metabase but is missing from
blobstor, peform an additional attempt to fetch it directly without
consulting metabase. Such a situation is unexpected, so error counter
is increased for the shard which has the object in the metabase. We
don't increase error counter for the shard which has the object in
blobstor, because some garbage can be expected there. In this
implementation there is no overhead for objects which are really
missing, i.e. are not present in any metabase.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 16:07:25 +03:00
Evgenii Stratonikov
6e6f3648d2 [#1188] metabase: log PUT / DELETE operations
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 15:21:35 +03:00
Evgenii Stratonikov
e292321ccf [#1114] neofs-node: Make some parameters optional in relay mode
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 14:30:55 +03:00
Evgenii Stratonikov
ce76325840 [#1146] engine: remove TODO and add a comment
Parallel check seems to be slower, so remove TODO.
```
name \ time/op      old          new           pool
Exists/2_shards-40  52.3µs ±23%   91.6µs ± 6%   84.2µs ± 1%
Exists/4_shards-40  72.7µs ±11%  121.8µs ± 8%  116.9µs ± 5%

name \ alloc/op     old          new           pool
Exists/2_shards-40  5.00kB ± 0%   5.03kB ± 0%   5.16kB ± 0%
Exists/4_shards-40  9.89kB ± 0%   9.93kB ± 1%  10.19kB ± 0%

name \ allocs/op    old          new           pool
Exists/2_shards-40     112 ± 1%      115 ± 1%      117 ± 1%
Exists/4_shards-40     207 ± 1%      211 ± 1%      216 ± 1%
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 12:31:41 +03:00
Evgenii Stratonikov
dab6bcabe4 [#1146] engine: Add benchmarks for exists
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 12:31:41 +03:00
Pavel Karpy
0e0a675f35 [#1212] morph: Fix NPE in multi client
Share multi cache between single clients.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-04 12:16:11 +03:00
Alex Vanin
cbc816d57d [#1211] node: Fix object get init error check
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-04 11:47:24 +03:00
Alex Vanin
bc45a79d1f [#1211] innerring: Fix object get init error check
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-04 11:47:24 +03:00
Alex Vanin
99749ea042 [#1211] neofs-cli: Specify address in session token
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-04 11:47:24 +03:00
Alex Vanin
1b5cb2dfe2 [#1211] neofs-cli: Specify verb in object session token
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-04 11:47:24 +03:00
Alex Vanin
ca17eb0eb4 [#1211] innerring: Fix objectIDs list reader
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-04 11:47:24 +03:00
Alex Vanin
6d3e626f20 [#1211] neofs-cli: Fix objectIDs list reader
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-04 11:47:24 +03:00
Alex Vanin
4935016bf8 [#1211] neofs-cli: Set bearer and session token in all object service requests
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-04 11:47:24 +03:00
Stanislav Bogatyrev
08f0a0b557 [#1167] Fix typos in README
Based on changes suggested by @roschler.

Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2022-03-04 10:50:16 +03:00
Leonard Lyubich
b5bdcfc076 [#1194] client: Support request X-headers
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-03 15:38:38 +03:00
Leonard Lyubich
dd6d7d2d10 [#1209] Upgrade NeoFS SDK Go
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-03 15:38:38 +03:00
Alex Vanin
09db5e387d [#1200] cli: Mention filter key prefixes in eACL creation command
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-01 10:36:41 +03:00
Evgenii Stratonikov
e10b8f53d6 [#1151] morph/client: Cache notary transaction heights
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-01 10:35:54 +03:00
Leonard Lyubich
e0dce1043a [#1195] Adopt recent changes in NeoFS SDK
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-02-28 10:43:03 +03:00
Evgenii Stratonikov
a8d10704d5 [#1182] neofs-cli: use BasicACL.String() for user output
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-22 11:26:34 +03:00
ZhangTao1596
dd0e10d306 [#1180] ir/internal: Fix audit range hash type
Signed-off-by: ZhangTao1596 <zhangtao@ngd.neo.org>
2022-02-14 12:36:02 +03:00
Alex Vanin
838a3fdb85 [#1103] example: Add neofs-cli config example
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-02-11 15:36:26 +03:00
Evgenii Stratonikov
47cd10a4d7 [#767] services/object: replace duplicate link in TODO
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 13:59:27 +03:00
Evgenii Stratonikov
e288a0188d [#749] neofs-adm: refactor password reading into a separate function
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 13:26:27 +03:00
Evgenii Stratonikov
e21d054fe2 [#749] morph/client: set group signer scope
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 13:26:27 +03:00
Evgenii Stratonikov
fa947b85a6 [#749] morph/client: reuse TestInvokePrm in InvokePrm
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 13:26:27 +03:00
Evgenii Stratonikov
9e2f7ac371 [#749] morph/client: allow to fetch contract group key
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 13:26:27 +03:00
Evgenii Stratonikov
4838d3bb80 [#749] morph/client: cache NNS contract hash
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 13:26:27 +03:00
Evgenii Stratonikov
050a4bb2b0 [#1115] *: link TODOs to corresponding issues
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 12:58:59 +03:00
Evgenii Stratonikov
692790a899 [#1115] services/object: reuse errors from core service
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 12:58:59 +03:00
Evgenii Stratonikov
4caa330ddc [#1115] morph/client: reuse errors from container contract
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 12:58:59 +03:00
Evgenii Stratonikov
f9d3111825 [#1115] blobovnicza: remove GetRange method
It is unused and incorrect as described in FIXME.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 12:58:59 +03:00
Evgenii Stratonikov
18f6fba6ea [#1115] neofs-ir: determine config type based on extension
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 12:58:59 +03:00
Evgenii Stratonikov
c1530dec5e [#1115] shard/gc: provide parsed addresses in HandleExpiredTombstones
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 12:58:59 +03:00
Evgenii Stratonikov
7ca06aeae2 [#1115] neofs-adm: generate wallets with 0644 permission
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 12:58:59 +03:00
Evgenii Stratonikov
a91b59fff9 [#1115] neofs-adm: use request timeout in N3 client
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 12:58:59 +03:00
Evgenii Stratonikov
0695ec4125 [#1133] services/session: remove expired tokens every epoch
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-08 14:58:32 +03:00
Evgenii Stratonikov
b3b3b8b20f [#1090] neofs-adm: Add history of commands to a storage configurator
It also persists between sessions.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-08 14:53:20 +03:00
Evgenii Stratonikov
8263582dde [#1090] neofs-adm: add interactive configurator for storage node
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-08 14:53:20 +03:00
Pavel Karpy
1667ec9e6d [#1131] *: Adopt SDK changes
`object.Address` has been moved to `object/address`
`object.ID` has been moved to `object/id`

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-02-08 09:45:38 +03:00
Evgenii Stratonikov
1c821d6c36 [#625] morph/client: remove unused interface
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-08 09:43:54 +03:00
Evgenii Stratonikov
c34cfa1f35 [#625] client/container: remove intermediate wrapper
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-08 09:43:54 +03:00
Evgenii Stratonikov
819d80a7a9 [#625] client/audit: remove intermediate wrapper
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-08 09:43:54 +03:00
Evgenii Stratonikov
767ee5c0cd [#625] client/reputation: remove intermediate wrapper
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-08 09:43:54 +03:00
Evgenii Stratonikov
97d18bc515 [#625] client/netmap: remove intermediate wrapper
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-08 09:43:54 +03:00
Evgenii Stratonikov
8c5c3ac9e8 [#625] client/neofsid: remove intermediate wrapper
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-08 09:43:54 +03:00
Evgenii Stratonikov
8474abb911 [#625] client/neofs: remove intermediate wrapper
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-08 09:43:54 +03:00
Evgenii Stratonikov
6f50fefbea [#625] client/balance: remove intermediate wrapper
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-08 09:43:54 +03:00
Pavel Karpy
c7a8c762e0 [#1136] cli: Allow usage of empty wallet passwords
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-02-07 14:42:46 +03:00
Alex Vanin
b86b06c0ca [#1101] neofs-cli: Set expiration in session token based on network info
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-02-07 13:29:25 +03:00
Evgenii Stratonikov
1deea4e8c6 [#749] neofs-adm: Set signer for committee transactions
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-04 11:49:55 +03:00
Evgenii Stratonikov
8fa2b364a1 [#749] neofs-adm: add group scope to force-new-epoch command
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-04 11:49:55 +03:00
Evgenii Stratonikov
e4bc9c7fad [#749] neofs-adm: save group public key in NNS
Query `group.neofs` instead of retrieving individual manifest.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-04 11:49:55 +03:00
Evgenii Stratonikov
def1bbc84c [#749] neofs-adm: create wallet for signing contract group
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-04 11:49:55 +03:00
Evgenii Stratonikov
b416285eb0 [#749] neofs-adm: include neofs contracts in a group
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-04 11:49:55 +03:00
Evgenii Stratonikov
e03cb91b64 [#1118] neofs-node: add shard_ro_error_threshold config setting
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-03 15:14:27 +03:00
Evgenii Stratonikov
ed50cf6207 [#1118] services/control: return error counter in ListShards
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-03 15:14:27 +03:00
Evgenii Stratonikov
7fb15fa1d0 [#1118] services/control: allow to reset error counter in SetShardMode
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-03 15:14:27 +03:00
Evgenii Stratonikov
6ad2624552 [#1118] engine: allow to set error threshold
There are certain errors which are not expected during usual node
operation and which tell us that something is wrong with the shard.
To prevent possible data corruption, move shard in read-only mode after
amount of errors exceeded some threshold. By default no actions are performed.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-03 15:14:27 +03:00
Leonard Lyubich
4f3323f084 [#1101] *: Adopt interface changes of API client from SDK library
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-02-03 14:00:14 +03:00
Evgenii Stratonikov
674f520da7 [#1132] *: Use path/filepath package when working with files
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-03 10:12:07 +03:00
Evgenii Stratonikov
0decb95591 [#625] morph/client: make all constructors return no error
In some places we panic, in some we return error, in some (audit) just return a client.
However in all of the places static client is created immediately before
the sugared-client creation.

This commit makes all constructors to just return a client for the sake
of code simplification and unification.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-01 15:47:54 +03:00
Evgenii Stratonikov
3c5b62d839 [#625] morph/client: make method names constant
We don't use custom names and the only place where custom method option
is used it provides the default name and can be omitted.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-01 15:47:54 +03:00
Alex Vanin
35dec2f494 Release v0.27.5
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-01-31 14:32:44 +03:00
Evgenii Stratonikov
a6cb608dc6 [#1023] .golangci.yml: remove outdated govet settings
Placement policy has been moved to SDK recently.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 13:01:58 +03:00
Evgenii Stratonikov
476528361e [#1023] *: fix linter errors
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 13:01:58 +03:00
Evgenii Stratonikov
eb99b65134 [#1023] morph/event: remove notaryRequest from subnet.Delete
Subnet removal is done by owner.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 13:01:58 +03:00
Evgenii Stratonikov
9d3609d4c1 [#1023] neofs-node/reputation: remove epoch from common parameters
It is unused and is present in both of `Announce*TrustPrm`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 13:01:58 +03:00
Evgenii Stratonikov
f96bb8d3b8 [#1023] innerring/subnet: fix subnet removal event processing
Caught by `unused` linter.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 13:01:58 +03:00
Evgenii Stratonikov
a82ac0edcd [#1023] services/object: remove TTL from Get parameters
It was unused an should be provided as a `CallOption` instead.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 13:01:58 +03:00
Evgenii Stratonikov
312071b9bd [#1023] .golangci.yml: add unused linter
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 13:01:58 +03:00
Evgenii Stratonikov
5a03a14940 [#1094] metabase: move Select filters sanity check outside of transaction
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:40 +03:00
Evgenii Stratonikov
651adf46c6 [#1094] blobovnicza: calculate size in Init properly
If pre-existing blobovnicza is initialized, it's size should be updated
even if all buckets are in place.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:40 +03:00
Evgenii Stratonikov
418b553920 [#1094] shard: remove TODO for shard mode description
It was added back in 2fb379b7 when we had many shard modes. Now we have
only two and comments for constants are rather descriptive.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:40 +03:00
Evgenii Stratonikov
ed7c732676 [#1094] shard: unify collection of expired objects
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:40 +03:00
Evgenii Stratonikov
fbb95cff14 [#1094] services/object: remove some FIXME comments
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:40 +03:00
Evgenii Stratonikov
1bdbd6ed22 [#1094] eacl/v2: preallocate slice in headersFromObject
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:40 +03:00
Evgenii Stratonikov
3407fef799 [#1086] neofs-cli: add command for shard restore
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:14 +03:00
Evgenii Stratonikov
18cfd8b042 [#1086] services/control: implement RestoreShard RPC
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:14 +03:00
Evgenii Stratonikov
2ec4a3c897 [#1086] neofs-cli: add command for shard dump
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:14 +03:00
Evgenii Stratonikov
0e60b1d6c9 [#1086] services/control: implement DumpShard RPC
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:14 +03:00
Evgenii Stratonikov
aa53418119 [#1086] services/control: fix comments
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:14 +03:00
Pavel Karpy
679c922e0c [#1120] cli: Fix printing config file in verbose mode
It was broken since `initConfig` was made before every execution of the
command and flags have not been read by cobra yet, so it was impossible to
print config file path if `verbose` flag was set in command line not in
config file.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-01-31 11:24:16 +03:00
Alex Vanin
08e83a2bc7 [#1092] neofs-cli: Add 'acl extended create' command
Follows neofs-cli refactor scheme from #1074

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-01-28 10:39:18 +03:00
Evgenii Stratonikov
e976a55358 config: replace use_write_cache with writecache.enabled
This is the way things are done with `grpc.tls` and in neo-go.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-27 16:44:56 +03:00
Leonard Lyubich
20b3ff84b3 [#1110] ir/container: Fix check of SetEACL operation
In previous implementation IR incorrectly verified `SetEACL` event of
`Container` contract. The incorrect behavior could be reproduced in two
ways:
  1. Create container using session, and perform `SetEACL` operation
  with a key that is different from the session one.
  2. Create container using session, and perform `SetEACL` w/o a
  session, but sign it using session key from the `Put` operation.

The problem was in the `checkSetEACL` validation method of IR container
processor. It always used session token used for container creation
during session ownership check.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-01-27 16:37:24 +03:00
Pavel Karpy
6f338c660c [#1038] adm: Do not cut data args in NNS
This reverts commit 185a2827cc.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-01-27 12:08:36 +03:00
Evgenii Stratonikov
adb80bebb2 [#1085] shard: rename Evacuate to Dump
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
d06425c852 [#1085] shard: allow to ignore errors in Evacuate
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
36eebb5932 [#1085] writecache: allow to ignore errors during iteration
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
cd75638ce3 [#1085] blobstor: allow to ignore errors during iteration
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
c954f0e71b [#1085] fstree: allow to ignore errors during iteration
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
e53ad2f468 [#1085] blobovnicza: allow to ignore errors during iteration
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
9c60ab893c [#1085] shard: allow to ignore errors during restore
We could also ignore errors during evacuate, but this requires
unmarshaling objects first which slowers the process considerably.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
0ef3d5ab03 [#1085] writecache: persist in-memory objects before shutdown
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
3215210d0c [#1085] shard: Close components in reverse order
Write-cache should be closed before blobstor and metabase.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
b432ec3a03 [#1085] shard/test: add object integrity check for Evacuate
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
c0f6c988f0 [#1085] shard: dump data from write-cache
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
ad01aaf8bf [#1085] writecache: add read-only mode
In read-only mode modifying operations are immediately returned with
error and all background operations are suspended.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
9f963e001b [#1085] writecache: set metaOnly flag for receiving from priority channel
This can lead to objects being flushed twice.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
7eedb23eb7 [#1085] shard: allow to dump/restore objects
Dump contains magic and a list of objects prefixed by object size in bytes.
We can't use proto-marshaled list because this requires having all dump
in memory. Using TAR induces 512 byte overhead for each object which can
be a problem in some cases.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Leonard Lyubich
257069a132 [#1084] cli/accounting: Convert balance response to Fixed8
After recent changes balance response contains contract precision.

Convert balance response to Fixed8. Use `fixedn.ToString` function to
print the converted value.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-01-26 14:10:02 +03:00
Leonard Lyubich
87a45983a1 [#1084] accounting: Keep balance precision from contract
Do not convert balance response of Balance contract to Fixed8.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-01-26 14:10:02 +03:00
Evgenii Stratonikov
dbb04d78b9 [#1103] neofs-cli: mention IR in healthcheck help message
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 14:09:42 +03:00
Evgenii Stratonikov
5c2b2e137d [#1103] neofs-cli: disallow generating key for control commands
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 14:09:42 +03:00
Evgenii Stratonikov
007158ebe1 [#1103] neofs-cli: allow to set password in config
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 14:09:42 +03:00
Evgenii Stratonikov
0c46827a08 go.mod: update tzhash version
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-24 14:13:36 +03:00
Evgenii Stratonikov
5bed27a62f [#1102] Makefile: use go install instead of go get
Remove warning and prevent `go.mod` updates when building `protoc` target.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-24 12:59:43 +03:00
Pavel Karpy
fee2f5a330 [#1100] *: Adopt new SDK's owner.ID API
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-01-21 17:58:30 +03:00
Leonard Lyubich
4b5404047c [#1098] object/search: Fix error loss in SignService.Search
Error loss led to incorrect response formation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-01-21 14:54:22 +03:00
Pavel Karpy
291fd2f183 [#1096] go.mod: Add SDK replace
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-01-21 10:47:33 +03:00
Pavel Karpy
202dbfb95d [#1096] eacl: Rename eacl import aliases
Since `Validator` has been moved to SDK it is more readable to see `eaclSDK`
than `acl` import.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-01-21 10:47:33 +03:00
Pavel Karpy
ed156cd738 [#1096] eacl: Use validator from SDK
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-01-21 10:47:33 +03:00
Evgenii Stratonikov
759421ebbf [#1083] neofs-cli: use single flag for key and wallet
Currently have static priority of what key is used irregardless of
whether a flag was provided via CLI or in config. This makes it
impossible to override some of the config settings. While we could try
to check if the key is provided by CLI by binding CLI flag under to
viper under a different name the same problem would occur for config/environment
variables. Fixing all of this with current set of keys is too complicate.
In this commit we revert changes from #610 and use a single flag for all types of keys.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-21 10:46:53 +03:00
ZhangTao1596
ac82899e85 [#1088] writecache: Fix small flushed when persist
Signed-off-by: ZhangTao1596 <zhangtao@ngd.neo.org>
2022-01-19 12:55:33 +03:00
Pavel Karpy
7c02a2e251 [#1087] *: Adopt SDK changes
- Update `neofs-sdk-go`:
v0.0.0-20211230072947-1fe37df88f80 => v0.0.0-20220113123743-7f3162110659

- Add client interface that duplicates SDK's client behaviour and new
`MultiAddressClient` interface that has method that iterates over wrapped
clients.

- Also start using simple client mode that does not require parsing statuses
outside the SDK library.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-01-14 17:29:03 +03:00
Alex Vanin
58d90eec7d Release v0.27.4
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-01-13 15:44:24 +03:00
Evgenii Stratonikov
486d5c2e86 [#1060] writecache: compress big object if needed
Small objects use `blobstor.Put`, so no changes are required.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-13 14:20:36 +03:00
Evgenii Stratonikov
0d969d7a06 [#1060] blobstor: allow to disable compression based on content-type
For some data compression makes little sense, as it is already compressed.
This commit allows to leave such data unchanged based on `Content-Type`
attribute. Currently exact, prefix and suffix matching are supported.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-13 14:20:36 +03:00
Alex Vanin
0f1eb743af [#1077] service/object: Don't override status of payload overtake
Reverse payload overtake triggers direct payload overtake that
sets status and error. We should not override that.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-01-13 10:30:18 +03:00
Evgenii Stratonikov
5828f43e52 [#851] util/rand: use single random source
It is much more convenient to skip source creation.
Also fix some bugs:
1. `cryptoSource.Int63()` now returns number in [0, 1<<63) as required
   by `rand.Source` interface.
2. Replace `cryptoSource.Uint63()` with `cryptoSource.Uint64` to allow
   generate uint64 numbers directly (see rand.Source64 docs).

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-12 19:29:27 +03:00
Evgenii Stratonikov
c35cdb3684 [#1041] autocomplete: fetch output stream from provided command
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-12 18:33:48 +03:00
Evgenii Stratonikov
c01acba3ce [#1041] autocomplete: beautify help message
Indent sections corresponding to different shells for faster navigation
and remove `#` at the line beginning.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-12 18:33:48 +03:00
Evgenii Stratonikov
efabffbfd0 [#1041] cmd/*: factor out autocomplet command generation
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-12 18:33:48 +03:00
Leonard Lyubich
fe29080ebb [#1025] ir: Read IR list from role management contract w/o mainchain
If mainchain is disabled in IR config then the node should read inner
ring list via role management contract.

Use `NeoFSAlphabetList` method of morph client as IR lister if
`withoutMainNet` flag is set in IR app.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-01-12 18:30:36 +03:00
Leonard Lyubich
0b0c61f4a5 [#1063] node/object: Do not skip empty search responses in splitter
In previous implementation `Search` method of transport splitter skipped
responses with empty ID list.

Replace while-loop with do-while one in `TransportSplitter.Search`
method implementation in order to send responses with empty result too.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-01-11 12:02:54 +03:00
ZhangTao1596
6830d54a3c [#1069] morph/balance: Fix transfer to
Signed-off-by: ZhangTao1596 <zhangtao@ngd.neo.org>
2022-01-11 12:02:11 +03:00
Leonard Lyubich
e74ff7c15a [#1062] node/object: Change the sequence of servers
In previous implementation ACL server was the 1st (except metric server
in some cases) server in pipeline of Object service servers. This led to
the fact that errors of this handler could not be reduced to status
responses.

Nest object ACL server into signature and response servers to support
common response format.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-01-11 11:08:55 +03:00
Alex Vanin
287bfd28e0 Release v0.27.3
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-12-30 14:29:05 +03:00
Pavel Karpy
97fd330fae [#1059] control: Fix ListShards wrapper name
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
77808c7b41 [#1059] control: Provide shard's real mode
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
375394dc99 [#1059] shard: Add shard mode to shard Info
Provide shard mode information via `DumpInfo()`. Delete atomic field from
Shard structure since it duplicates new field.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
86b90eb944 [#1059] cli: Add SetShardMode operation
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
88bb8e2df9 [#1059] control: Add SetShardMode implementation to ctrl svc
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
fa78a37787 [#1059] control: Add unit test for SetShardMode
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
29c69f37eb [#1059] control: Add SetShardMode RPC wrappers
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
25665cb0fd [#1059] control: Add marshalling and setters to SetShardMode
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
4533784e11 [#1059] control: Recompile proto files
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
41dd385b7b [#1059] control: Add SetShardMode rpc to proto files
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Alex Vanin
90f05d4448 [#1066] cli: Support basic ACL constants without final flag
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-12-30 14:07:18 +03:00
Evgenii Stratonikov
56b8793520 [#1056] neofs-adm: deploy proxy contract without arguments
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-30 13:54:07 +03:00
Evgenii Stratonikov
185a2827cc Revert "[#1038] adm: Do not cut data args in NNS"
This reverts commit acb4a9e5b4.
2021-12-30 10:20:54 +03:00
Evgenii Stratonikov
549546dea1 [#1035] neofs-adm: reuse NEF and manifest during alphabet contract update
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-30 10:20:54 +03:00
Evgenii Stratonikov
4673c81451 [#1035] neofs-adm: update contracts in a single transaction
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-30 10:20:54 +03:00
Pavel Karpy
a8ba573ec8 [#1057] node/config: Use mode from config in shard/s constructor
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-29 18:06:15 +03:00
Pavel Karpy
8b8a815fb3 [#1057] node/config: Add shard mode config param
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-29 18:06:15 +03:00
Pavel Karpy
93bd6be743 [#1057] shard: Do not allow memory change operations in "read-only"
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-29 18:06:15 +03:00
Pavel Karpy
4f756bf121 [#1057] shard: Make shard to have only two mode
Shard's mode was not used in the Node, so added only two modes whose roles
are clear. More modes will be added in the future.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-29 18:06:15 +03:00
Alex Vanin
2109211c55 Release v0.27.2
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-12-28 17:06:27 +03:00
Alex Vanin
abca196990 Update neofs-api-go and neofs-sdk-go repositories
1. Adds mandatory timeout for all NeoFS API message
   transmissions to avoid connection freeze and
   goroutine leak.

2. Fixes inconsistency in object placement function.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-12-28 17:06:27 +03:00
Evgenii Stratonikov
e96eb3e00b [#1054] neofs-node: add epoch metric
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-28 15:34:20 +03:00
Evgenii Stratonikov
e1137aa09f [#1054] innerring: add epoch metric
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-28 15:34:20 +03:00
Pavel Karpy
876b0c53de [#1048] control: Rename Blobstore to Blobstor
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
f5a9735e1c [#1048] cli: Add shard list operation
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
284188f8f9 [#1048] node: Fill shard's info with its components' infos
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
0e5410603e [#1048] control: Add ListShards implementation to ctrl svc
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
4e989e7133 [#1048] control: Add unit test for ListShards
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
f88a12eaa7 [#1048] control: Add ListShards RPC wrappers
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
a6b3e16975 [#1048] control: Add marshalling and setters to ListShards
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
a42b3d37f6 [#1048] control: Recompile proto files
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
20f11c88ca [#1048] control: Add ListShards rpc to proto files
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Alex Vanin
5d46035ae8 [#1052] Tidy INFO logs
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-12-27 14:28:01 +03:00
Alex Vanin
96efe0f294 Release v0.27.1
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-12-20 16:10:05 +03:00
Alex Vanin
e610d1ea5f [#1045] Provide sanity check of session token in container service
Without sanity check, container service provides successful response,
even though such request will never be approved by Alphabet nodes.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-12-20 14:51:07 +03:00
Pavel Karpy
acb4a9e5b4 [#1038] adm: Do not cut data args in NNS
The updated version of the `NNS` contract now supports the third `data`
argument that may provide additional information.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-20 14:43:29 +03:00
Alex Vanin
bca7cf9470 [#1047] policer: Check context before job selection
When application is being terminated, replicator routine
might be on the object picking phase. Storage is terminated
asynchronously, thus `Select()` may return corresponding
error. If we don't process `context.Done()` in this case,
then application freezes on shutdown.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-12-20 14:42:57 +03:00
Pavel Karpy
11466ea3cc [#1031] adm: Use 10 as maxConnPerHost in neo-go client
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-16 18:55:53 +03:00
Pavel Karpy
c49e53ba9d [#1031] morph: Add maxConnPerHost option
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-16 18:55:53 +03:00
Pavel Karpy
63e035bd8a [#1031] node: Add maxConnPerHost to config
It allows configuring number of neo-go client opened connections per one
host.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-16 18:55:53 +03:00
Pavel Karpy
90dbf3d944 [#1039] cli: Make return codes depend on errors
Cast client errors to well-known public `sdk` errors and return appropriate
exit code.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-16 16:03:41 +03:00
Pavel Karpy
df0d76dc19 [#1032] node/container/router: Do not check nil route points
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-16 16:02:29 +03:00
Pavel Karpy
fd9514f90e [#1032] node/container/router: Fix interface description
After #872 `NextStage` method description becomes outdated.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-16 16:02:29 +03:00
ZhangTao1596
1be0c42726 [#1022] writecache: Fix db counter decrement and flushed big object
Signed-off-by: ZhangTao1596 <zhangtao@ngd.neo.org>
2021-12-15 14:23:49 +03:00
Alex Vanin
2a031c5542 [#1013] Support autocomplete in neofs-adm
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-12-14 17:34:10 +03:00
Leonard Lyubich
fc42844fbf [#1028] writecache: Evict flushed big objects
Make `flushBigObjects` routine to call `evictObjects` which was
successfully written to `BlobStor`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-14 13:49:47 +03:00
Leonard Lyubich
d246e5193b [#1028] writecache: Mark big objects as flushed after write
Make `flushBigObjects` routine to mark objects which are written to
`BlobStor`. This prevents already flushed objects from being written on
the next iterator tick.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-14 13:49:47 +03:00
Stanislav Bogatyrev
bd4d17ef9e [#1034] Add shorthand for generate-key flag
Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2021-12-10 19:54:41 +03:00
Stanislav Bogatyrev
b705f9460b [#1033] README typo fixes
Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2021-12-10 14:31:43 +03:00
Alex Vanin
8c58c458e5 [#1027] Fix CHANGELOG formatting
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-12-09 16:58:02 +03:00
Alex Vanin
4e65abc766 [#1027] Add NeoFS contract address in N3 Mainnet
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-12-09 16:58:02 +03:00
Alex Vanin
2359001085 [#1027] Update contract version in README file
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-12-09 16:58:02 +03:00
Alex Vanin
fd321b6feb Release v0.27.0 - Sinjido (신지도, 薪智島)
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-12-09 10:37:06 +03:00
Alex Vanin
904759d5bf Update neo-go to the latest v0.98.0 release
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-12-09 10:37:06 +03:00
Pavel Karpy
cb246dfab7 [#1015] docs: Add subnetwork managing HOWTO
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-09 09:47:50 +03:00
Pavel Karpy
14f49df658 [#1015] docs: Add subnetwork creation HOWTO
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-09 09:47:50 +03:00
Alex Vanin
d89ffbfbdf [#1020] neofs-adm: Make key flag non global in subnet
Key flag should not be global because `subnet get` command
does not require any credentials.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-12-08 18:49:41 +03:00
Alex Vanin
3bb67075f4 [#1020] neofs-adm: Do not ask key in morph subnet get command
There is no need to provide credential to fetch public data from
side chain.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-12-08 18:49:41 +03:00
Pavel Karpy
15ff98c229 [#1024] adm: Change private key to wallet in subnet flags
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-08 15:57:22 +03:00
Leonard Lyubich
90fd883e32 [#915] blobovnicza: Use mean bucket size multiplier instead of min
For fullness estimation of `Blobovnicza` we use number of object stored
in each size bucket. In previous implementation we multiplied the number
by the difference in bucket boundaries. This expression rather
estimated the minimum volume (and for the smallest bucket, the maximum)
of objects in the bucket.

Multiply number of objects by mean bucket size.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-08 14:24:49 +03:00
Leonard Lyubich
2920c5203b [#915] blobovnicza: Re-use Bolt transaction in syncFullnessCounter
Make `syncFullnessCounter` to accept `bbolt.Tx` argument of Bolt
transaction within which counter should be synchronized. Pass
corresponding transaction during `Init`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-08 14:24:49 +03:00
Leonard Lyubich
feb0a65efb [#1008] ir/container: Customize fee for named container registration
In notary disabled environment, approval of container creation with nice
name attribute takes much more additional GAS than other operations
(due to NNS invocation).

Morph library changes:
  * add the ability to specify per-op fees using `StaticClient` options;
  * add the ability to customize fee for `Put` operation with named
    container in container morph client.

Inner Ring changes:
  * add `fee.named_container_register` config value which specifies
    additional GAS fee for the approvals of the named container
    registrations;
  * pass the config value to `WithCustomFeeForNamedPut` option of
    container morph client.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-08 13:57:01 +03:00
Pavel Karpy
b4c36a109d [#1019] adm: Fetch subnet SH from NNS
Also, delete `-a` (`--contract`) flag.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-08 13:33:36 +03:00
Leonard Lyubich
f9bcb6f5e3 [#1007] metabase: Copy BoltDB value returned by IsSmall
According to BoltDB documentation bucket `value is only valid for the
life of the transaction`.

Make `DB.IsSmall` copy value slice in order to prevent potential memory
corruptions (e.g. `runtime.stringtobyteslice` cast).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-07 18:59:42 +03:00
Alex Vanin
c383fc6929 [#1014] neofs-adm: Print NNS contract version
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-12-07 18:55:44 +03:00
Leonard Lyubich
3d5eb2a396 [#1008] ir/netmap: Call Register method on peer approval
After https://github.com/nspcc-dev/neofs-contract/issues/154 alphabet
nodes should call `Register` method for approval of the notary
notifications spawned by `AddPeer` method.

Call `register` method for peer approval in Netmap processor.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-07 17:15:44 +03:00
Alex Vanin
89eca449b6 Release candidate v0.27.0-rc.1
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-12-03 14:33:57 +03:00
Evgenii Stratonikov
a5e952cc72 [#1006] neofs-adm: skip missing entries in dump-hashes
We have `subnet` contract in list, but it isn't currently deployed.
This commit skips missing NNS entries to handle such situation. In
future we may optimize this to be done in 1 round-trip.

As a nice side-effect, dump-hashes for notary-enabled environment works
even if notary is disabled -- it just prints zero hash for proxy
contract.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-03 13:24:17 +03:00
Sergio Nemirowski
4ad2961c51 [#1005] neofs-node: fix usage message
Signed-off-by: Sergio Nemirowski <sergio@nspcc.ru>
2021-12-03 10:37:24 +03:00
Evgenii Stratonikov
ca894fee23 [#749] neofs-adm: make alphabet size check more robust
We can now store wallet for signing manifest groups in the same dir.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-02 17:24:16 +03:00
Evgenii Stratonikov
40b51b3586 [#995] neofs-adm: implement deposit-notary command
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-02 17:24:16 +03:00
Evgenii Stratonikov
fba8890224 [#995] neofs-adm: split out tx context
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-02 17:24:16 +03:00
Evgenii Stratonikov
02be6c83a6 morph/client: update morph container wrapper
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-02 14:16:23 +03:00
Evgenii Stratonikov
d1be5b5f9e [#878] neofs-node: default to secure TLS settings
Support TLS >=1.2 only and strong cipher suites.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-02 12:05:41 +03:00
Leonard Lyubich
973e50ad72 [#979] ir/subnet: Register parser and handler of RemoveNode event
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 19:27:16 +03:00
Leonard Lyubich
1b698867a7 [#979] adm/subnet: Fix typos
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Leonard Lyubich
dd2998d724 [#979] adm/subnet: Rename id flag to node in node cmd
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Leonard Lyubich
5f915dfb43 [#979] adm/subnet: Rename id flag to admin in admin cmd
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Leonard Lyubich
cc8a0037e0 [#979] adm/subnet: Fix print out of subnet info in get cmd
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Leonard Lyubich
6f41cd86ed [#979] Upgrade SDK NeoFS Go dependency
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Leonard Lyubich
3fcd7bc948 [#979] adm/subnet: Always set group ID parameter in client cmds
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Leonard Lyubich
3550ed9fe4 [#979] ir/subnet: Fix notification event names
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Leonard Lyubich
4d79195ede [#979] adm/subnet: Fix notary issues
Only `subnet create` command can generate notary requests.

Remove global `non-notary` flag. Add `notary` flag to `create` cmd.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Leonard Lyubich
98e0792b08 [#979] adm/subnet: Add commands to add/remove nodes
Implement `ManageNodes` operation on morph subnet client.
 Add `node add` and `node remove` commands to `subnet` section.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Leonard Lyubich
579c3717a5 [#979] adm/subnet: Inherit pre-run functions in child commands
Cobra commands don't call `PreRun` functions on parent by default. We
need to do it in `subnet` command of admin utility in order to inherit
viper bindings.

Add `inheritPreRun` function which makes sub-commands to call `PreRun`
functions before its own `PreRun`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Leonard Lyubich
6facc26cb9 [#979] adm/subnet: Fix handling of non-notary flag in initClient
`initSubnetClient` must call `EnableNotarySupport` only if non-notary
flag is not set.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Leonard Lyubich
ef57ba3fec [#979] ir/subnet: Fix notary event name of subnet creation
Notary event name equals to the name of the method which throws the
event.

Define name const of notary subnet creation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Leonard Lyubich
ca2c6ad77f [#979] morph/client: Return zero values from CalculateNonceAndVUB
Make `CalculateNonceAndVUB` to return zero values if notary isn't
enabled.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Leonard Lyubich
4b8ca75274 [#979] adm: Fix problems after testing
Use persistent flags on parent command in order to inherit flags in
sub-commands. Turn on notary mode of morph client in `subnet` command of
admin utility for notary environments.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Leonard Lyubich
069a174129 [#979] morph/client: Get proxy contract address from NNS
Make `Client.EnableNotarySupport` method to call `NNSContractAddress`
for proxy contract if it is not specified in corresponding option.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Leonard Lyubich
9622c9c858 [#979] adm: Register NNS domain of the deployed contracts
Make `deployContracts` method to call `nnsRegisterDomain` in order to
register NNS domain for deployed contracts.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Leonard Lyubich
2f2ca2e0bd [#979] adm: Deploy missing contracts during updating
Make `nnsResolveHash` function to return declared error on `token not
found` fault exception. Catch this error in `deployContracts` method,
and switch to deployment if updating contract is missing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Leonard Lyubich
e8f8e58e90 [#979] adm: Add subnet-related commands
Add `subnet` command which contains all subnet-related commands. Add
sub-commands:
  * `create` for creation;
  * `remove` for removal;
  * `get` for reading;
  * `admin` for admin management;
  * `client` for client management.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Leonard Lyubich
70f17dc778 [#979] adm: Deploy and update Subnet contract
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Pavel Karpy
1cee1b8f93 [#1002] ir: Add subnet check to the container Put process
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-01 16:26:15 +03:00
Pavel Karpy
2245bf85d8 [#1002] morph: Add UserAllowed method to subnetwork client
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-01 16:26:15 +03:00
Pavel Karpy
5b8f195563 [#1000] ir/nodeValidators: Ignore zero subnet validation
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-01 12:27:35 +03:00
Evgenii Stratonikov
98f288c946 [#984] neofs-adm: fix contract hash check in NNS
Don't update if the hash is already there.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-29 11:19:36 +03:00
Evgenii Stratonikov
7765ab3f02 [#984] neofs-adm: replace addRoot with register for NNS
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-29 11:19:36 +03:00
Evgenii Stratonikov
63a8b530f6 [#984] neofs-adm: compare NEF.Checksum to check if contract was updated
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-29 11:19:36 +03:00
Pavel Karpy
6207a2f261 [#988] cli: Add subnet flag to container create
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-29 11:03:56 +03:00
Pavel Karpy
acf7331034 [#990] object: Fix linter
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-28 17:18:08 +03:00
Pavel Karpy
8988ac2729 [#990] ir/nodeValidators: Add subnet entrance node Validation on AddPeer call
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-28 17:18:08 +03:00
Pavel Karpy
a079a8f727 [#990] nodeValidators: Add subnet entrance validator
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-28 17:18:08 +03:00
Pavel Karpy
b6db699b6e [#990] morph/client: Add NodeAllowed subnet method
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-28 17:18:08 +03:00
Leonard Lyubich
0fb838b169 [#983] node: Configure subnets
Add `subnet` sub-section to `node` section of storage node config. Add
`entries` value which allows to enumerate subnets for entrance. Add
`exit_zero` value which allows to not enter zero subnet.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-28 16:42:46 +03:00
Leonard Lyubich
b27c72c02a [#973] morph/subnet: Fix docs and comments of notification events
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-28 16:15:18 +03:00
Leonard Lyubich
52fc5bde6e [#973] ir/subnet: Remove alphabet re-signature of Delete request
Subnet contract doesn't work with alphabet signatures.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-28 16:15:18 +03:00
Leonard Lyubich
41eaa1e246 [#973] ir: Listen and process Put/Delete events of Subnet contract
Define notification events, implement parsers. Add morph client of
Subnet contract. Listen, verify and approve events in Inner Ring app.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-28 16:15:18 +03:00
Pavel Karpy
214c2bd0cb [#987] ir: Add RemoveNode parser and handler
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-28 14:38:44 +03:00
Pavel Karpy
81dc17718e [#987] ir: Add parsing subnet script hash from NNS contract
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-28 14:38:44 +03:00
Pavel Karpy
0c6cdd0afd [#987] morph/event: Add RemoveNode event and its parser
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-28 14:38:44 +03:00
Alex Vanin
011d0f605b [#965] replicator: Make HandleTask function public
Continues replication executed in separate pool of goroutines,
so there is no need in worker to handle replication tasks
asynchronously.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-26 15:39:38 +03:00
Alex Vanin
a74a402a7d [#965] policer: Implement continuous replication
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-26 15:39:38 +03:00
Alex Vanin
20f0b29a6e [#975] morph/event: Update notary preparator tests
Now includes testcases for notary
requests with 3 and 4 witnesses.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-26 15:37:17 +03:00
Alex Vanin
a0ff5b1bf8 [#975] morph/client: Use 4 witnesses in notary request
Add invoker witness as the third witness. Required for
netmap methods checks.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-26 15:37:17 +03:00
Alex Vanin
6f23dbfefe [#971] innerring: Use VUB pointer for initial committee vote
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-25 18:09:41 +03:00
Leonard Lyubich
7f5fb130c0 [#961] *: Support NeoFS API status returns
Upgrade NeoFS API Go library to version with status returns. Make all API
clients to pull out and return errors from failed statuses. Make signature
service to respond with status if client version supports it.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-24 09:26:40 +03:00
Leonard Lyubich
44d0d453ef [#961] client: Clarify error returns in docs
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-24 09:26:40 +03:00
Alex Vanin
378474b02c [#950] neofs-cli: Attach x-headers in object range
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 17:10:01 +03:00
Alex Vanin
1f911830a5 [#948] engine: Define EndOfListing error in component packages
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
c80c83b0b8 [#948] metabase: Simplify cursor structure
Use bbolt bucket keys to seek offset in root bucket
and container buckets.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
468caa83d9 [#948] Adopt sdk-go client
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
08bdd0d561 [#948] engine: Fix comments of object listing with cursor
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
aa9ce8a853 [#948] engine: Define cursor for object listing as a type
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
164cd10af8 [#948] engine: Hide shard IDs in ListWithCursor
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
0f6d8f6eea [#948] metabase: Use seek for faster cursor listing
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
8d471c7e36 [#948] core/object: Move listing objects error to core
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
c02c7bee5b [#948] engine: Add ListWithCursor method
Interface for shard.ListWithCursor invocations.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
5b6be7bc1c [#948] engine: Fix test file path
Before: `<t.Name()>.<id>.blobstor`
After:  `<t.Name()>/<id>.blobstor`
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
1950724a04 [#948] shard: Add ListWithCursor method
Interface for metabase.ListWithCursor method invocation.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
1a829a521f [#948] metabase: Add ListWithCursor method
ListWithCursor allows listing physically stored objects
from metabase with small chunks. Cursor tracks last
processed object, therefore new chunks are returned
on each request.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Pavel Karpy
60e3ea978f [#971] *: Group all 3rd parties imports
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
7ac3145980 [#971] *: Add optional parameters to audit morph client calls
Adapt all audit wrapper calls to new
structures.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
8a2f5c980b [#971] morph/audit: Add optional parameters
Add optional parameters to the client call
signature.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
ed4810a020 [#971] *: Add optional parameters to balance morph client calls
Adapt all balance wrapper calls to new
structures.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
644baf4985 [#971] morph/balance: Add optional parameters
Add optional parameters to the client call
signature.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
dbf3a2f2fb [#971] *: Add optional parameters to container morph client calls
Adapt all container wrapper calls to new
structures.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
af33dd65b2 [#971] morph/container: Add optional parameters
Add optional parameters to the client call
signature.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
822d73fb02 [#971] *: Add notification TX hash to neofsid morph client calls
Add hash of the TX that generated notification
to neofsid event structures. Adapt all
neofsid wrapper calls to new structures.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
404c62c5c4 [#971] morph/neofsid: Add optional parameters
Add optional parameters to the client call
signature.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
bad739258e [#971] *: Add notification TX hash to neofs/netmap morph client calls
Add hash of the TX that generated notification
to neofs/netmap event structures. Adapt all
neofs/netmap wrapper calls to new structures.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
c25f5a86ae [#971] morph/netmap: Add optional parameters
Add optional parameters to the client call
signature. Group parameters of a client call
into struct to improve future codebase
support.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
3114be39d0 [#971] morph/neofs: Add optional parameters
Add optional parameters to the client call
signature. Group parameters of a client call
into struct to improve future codebase
support.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
1db6d316c2 [#971] morph/client: Adapt signature changes in wrappers
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
3849d13e0b [#971] morph/client: Group Invoke and TestInvoke params
Also:
- add `validUntilBlock` and nonce calculation;
- pass notification hashes to notary calls.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
8e1f187822 [#971] ir: Drop CLI validators voting functionality
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
c167ae26f9 [#971] morph/event: Change notification parser's signature
Parsers should have original notification
structure to be able to construct internal
event structure that contains necessary
for unique nonce calculation information.
So notification parsers take raw notification
structure instead of slice of stack items.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
3666ae7ad2 [#971] morph/subscriber: Adapt extended notification structure
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Alex Vanin
c9f2804885 [#976] innerring: Ignore non-HALT netmap.AddPeer notary invocations
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-18 09:57:16 +03:00
Alex Vanin
93dce149ba [#976] morph/client: Add script verifier
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-18 09:57:16 +03:00
Alex Vanin
cb36f8b50e [#868] blobstor: Speed up compression test
Compression test does not require wide blobovnicza tree.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-16 13:35:34 +03:00
Evgenii Stratonikov
aba09bb853 [#903] config: support floating-point sizes
Rounding is done using the suffix boundary (i.e. floating point value
with `gb` suffix will return size which is an integer number of
megabytes).

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-16 12:43:55 +03:00
Evgenii Stratonikov
e7fd980951 [#859] config: rename writecache.size_limit to writecache.capacity
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-15 13:14:24 +03:00
Evgenii Stratonikov
c015b04ed5 [#859] config: rename shallow_* to width and depth
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-15 13:14:24 +03:00
Evgenii Stratonikov
684e3e0ae0 [#859] config: rename mem_size to memcache_capacity
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-15 13:14:24 +03:00
Evgenii Stratonikov
a180f09523 [#859] config: rename max_size to max_object_size
`_size` suffix by itself makes me think about restrictions on database
size.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-15 13:14:24 +03:00
Evgenii Stratonikov
da51c119d5 [#859] config: rename opened_cache_size to opened_cache_capacity
Capacity is a common term to use with cache, see e.g. leveldb
configuration.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-15 13:14:24 +03:00
Evgenii Stratonikov
0759d8d0e5 [#859] config: rename small_size_limit to small_object_size
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-15 13:14:24 +03:00
Evgenii Stratonikov
cbf8ca12b0 [#859] config: rename refill_metabase to resync_metabase
This flag enables metabase content synchronization with blobstor on
start.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-15 13:14:24 +03:00
Alex Vanin
e4087893a1 [#966] engine: Fix test after neofs-api-go replacement
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-15 13:06:10 +03:00
Alex Vanin
9a4d4c03b7 [#918] config/testnet: Add control service endpoint
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-15 13:06:10 +03:00
Leonard Lyubich
68a454fc56 [#969] config/example: Add note about shard config in relay node
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-15 06:47:25 +03:00
Leonard Lyubich
fb89d29574 [#969] node: Do not require shard config in relay mode
Relay storage node doesn't exec local object operations, so it doesn't need
shard configuration.

Add `required` bool parameter to `engineconfig.IterateShards`. Make it to
panic if it is `true`, and immediately return otherwise. Pass `false` if
node is configured as relay in app (it also prevents panic).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-15 06:47:25 +03:00
Evgenii Stratonikov
95893927aa *: replace neofs-api-go with neofs-sdk-go
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-12 17:29:09 +03:00
Leonard Lyubich
6c0b29e3e3 [#922] storage engine: Prevent any operations after first Close call
Make `BlockExecution` / `ResumeExecution` to not release per-shard worker
pools. Make `StorageEngine.Close` to block these methods and any
data-related operations. It is still releases the pools.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
d6457ee485 [#922] shard: Allow multiple gc.stop() calls
Use `sync.Once` to prevent locks of stopping GC. It will also allow to
safely call `Shard.Close` multiple  times.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
10f0bd91d6 [#922] engine: Change interface of container operations
Add `error` to return. Improve docs.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
a537334f33 [#922] engine: Fix typos and improve naming related to exec blocks
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
6b1ce99c35 [#922] node: Block data ops of storage engine via Control API
There is a need to block execution of local object storage operations if
node is put into maintenance mode (resume if the node is taken out of
maintenance mode).

Call `BlockExecution` method if `ControlService.SetNetmapStatus` was called
with `MAINTENANCE` status. Call `ResumeExecution` if it was called with
another status.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
29644e9bc2 [#922] cli: Support MAINTENANCE netmap status
Accept `maintenance` netmap status in `control set-status` command and
resolve it to `control.NetmapStatus_MAINTENANCE` value.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
8f8b638b95 [#922] control: Add MAINTENANCE netmap status
Add `MAINTENANCE` value to `NetmapStatus` enum in Control API. The status is
going to be used to toggle maintenance mode of the storage node.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
ec04e787aa [#922] storage engine: Support operation blocking
There is a need to disable execution of local data operation on storage
engine in runtime. If storage engine ops are blocked, node will act like
always but all local object operations will be denied.

Implement `BlockExecution` / `ResumeExecution` methods on `StorageEngine`
which blocks / resumes the execution of data ops. Wait for the completion of
all operations executed at the time of the call. Return error passed to
`BlockExecution` from all data-related methods until `ResumeExecution` call.
Make `Close` to block operations as well.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
cea1de3a27 [#922] storage engine: Remove unused Exists method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
68565d9617 [#938] ir/netmap: Call AddPeer method if existing candidate was updated
In previous implementation IR handler of `AddPeer` notification didn't send
registration to contract if existing peer changed has changed its
information. as a consequence, the network map members could not update the
information without going into offline.

Change `processAddPeer` handler to check if
  * candidate in the network map is a brand new
  * or information about the network map member was changed
and call `AddPeer` method if so.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-10 18:05:30 +03:00
Leonard Lyubich
0ec8f529ab [#842] object/delete: Set tombstone local node as tombstone owner
All objects in NeoFS must have owner ID. In previous implementation Object
Delete service handler set owner ID from request session token. If removal
was executed w/o a session, object with tombstone was prepared incorrectly.
In order to fix this node should set its own ID and become an owner of the
tombstone object.

Extend `NetworkInfo` interface required by Object.Delete handler with
`LocalNodeID` method which returns `owner.ID` of the local node. Implement
the method on `networkState` component of storage node application which is
updated on each node state change in NeoFS network map. Set owner returned
by `LocalNodeID` call as tombstone object's owner in Delete handler.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-10 16:18:02 +03:00
Leonard Lyubich
d421022547 [#842] object/acl: Check owner and key only if sticky bit is set
In previous implementation node returns "access denied" on Object.Put with
object with unset owner. Although object owner must be set, its absence
should not be considered as access error. The same applies to sender key.

Check owner ID and public key emptiness only if sticky bit is set.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-10 16:18:02 +03:00
Pavel Karpy
5e5211305c [#936] Update CHANGELOG
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-10 16:11:57 +03:00
Pavel Karpy
6cab1635d4 [#936] ir: Do not require number of Alphabet contracts to be set
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-10 16:11:57 +03:00
Pavel Karpy
b492201a84 [#936] morph/nns: Add record existence check
Also, add ErrNNSRecordNotFound error that
indicates that required hash is not presented
in `NNS` contract.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-10 16:11:57 +03:00
Leonard Lyubich
e500efb9b3 [#957] services/reputation: Refactor usage of NeoFS API client
The client needs of the Reputation service are limited and change not often.
Interface changes of the client library should not affect the operation of
various service packages, if they do not change their requirements for
the provided functionality. To localize the use of the base client and
facilitate further support, an auxiliary package is implemented that will
only be used by the Reputation service.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-03 18:30:09 +03:00
Leonard Lyubich
b3708fc530 [#957] services/object: Refactor usage of NeoFS API client
The client needs of the Object service are limited and change not often.
Interface changes of the client library should not affect the operation of
various service packages, if they do not change their requirements for
the provided functionality. To localize the use of the base client and
facilitate further support, an auxiliary package is implemented that will
only be used by the Object service.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-03 18:30:09 +03:00
Leonard Lyubich
bbc2b873ab [#950] cli: Refactor usage of NeoFS API client
The client needs of the CLI application are limited and change not often.
Interface changes of the client library should not affect the operation of
various application packages, if they do not change their requirements for
the provided functionality. To localize the use of the base client and
facilitate further support, an auxiliary package is implemented that will
only be used by the CLI application.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-03 18:26:36 +03:00
Leonard Lyubich
a3414b36dd [#958] cli/locode: Open RO database in info command
CLI `util locode info` command does not write operations. Opening a
`locodebolt.DB` instance in RW mode is redundant.

Provide `locodebolt.ReadOnly()` option to `locodebolt.DB` constructor in
order to create READ flock over BoltDB file.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-03 10:19:45 +03:00
Alex Vanin
cee4f3142f Release v0.26.1
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-02 17:08:37 +03:00
Alex Vanin
a437ffc3ed [#910] innerring: Make notary deposit on notification instead of timer
Timer is not suitable for notary deposits because it can never fire
in case of desynchronization or external epoch changes. Notary deposits
must be handled on new epoch event.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-02 11:28:02 +03:00
Alex Vanin
45f244eb77 [#955] locode: Use the nearest polygon when continent can't be found
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-02 10:30:23 +03:00
Evgenii Stratonikov
1462824ab8 [#947] writecache: refactor object persisting
a1696a8 introduced some logic which in some situations prevented big objects
to be persisted in FSTree. In this commit a refactoring is done with the
goal of simplifying the code and also checking #866 issue.

1. Split a monstrous function into multiple simple ones: memory objects
   can only be small and for writing through the cache we can do a dispatch
   in `Put` itself.
2. Determine objects to be put in database before the actual update
   as setting up a transaction has non-zero overhead.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-01 15:52:56 +03:00
Evgenii Stratonikov
088c894f44 [#733] neofs-adm: read contracts on start
Fail early and reduce disk operations when reading from archive.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-01 15:17:18 +03:00
Evgenii Stratonikov
0b6350d463 [#733] neofs-adm: fetch contracts release from Github
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-01 15:17:18 +03:00
Evgenii Stratonikov
c1e1b65ad9 [#733] neofs-adm: allow to read contract from tar archive
Assume contracts path is tar.gz archive if it is not a directory.
In theory we could support raw tar but it has little value as our
distribution format is tar.gz and this is the primary use-case for this
feature (download and use).

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-01 15:17:18 +03:00
Leonard Lyubich
88e37ea372 [#946] ir: Refactor usage of NeoFS API client
The client needs of the IR application are very limited and rarely change.
Interface changes of the client library should not affect the operation of
various application packages, if they do not change their requirements for
the provided functionality. To localize the use of the base client and
facilitate further support, an auxiliary package is implemented that will
only be used by the IR application.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-01 11:41:57 +03:00
Evgenii Stratonikov
49c9dbfba8 [#877] config/engine: use default section for shards
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-28 15:04:14 +03:00
Evgenii Stratonikov
cb9bf00ceb [#877] neofs-node/config: allow to provide default values
In case we have multiple sections with similar structure (e.g. shards)
having defaults in a single place is easier to work with.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-28 15:04:14 +03:00
Evgenii Stratonikov
118c3b3cfe [#940] neofs-adm: print contract version in dump-hashes
```
nns         (unknown):  7061fbd31562664b58f422c3dee4acfd70dba8af
alphabet 0  (v0.12.1):  2392438eb31100857c0f161c66791872b249aa13
alphabet 1  (v0.12.1):  83ef4226d5d6519ca9c99a5de13b1b5ca223a6ad
alphabet 2  (v0.12.1):  6250927beaa9aa5a00171379dcb7187b0c91d17d
alphabet 3  (v0.12.1):  1d6a2519ba41a139b2ced1bfd5013938271a7578
alphabet 4  (v0.12.1):  b65fc7a3c31cf57a90d7eb1c0e9909e4ca69133c
alphabet 5  (v0.12.1):  f95b6ff8cd3b027c9911c18115518ad8c5d2f591
alphabet 6  (v0.12.1):  5b17c579bf56884fd68af152432b3b5aee7aee76
audit       (v0.12.1):  85fe181f4aa3cbdc94023d97c69001ece0730398
balance     (v0.12.1):  dc1ec98d9d0c5f9dfade16144defe08cffc5ca55
container   (v0.12.1):  1b6e68d299b570e1cb7e86eadfdc06aa2e8e0cc5
neofsid     (v0.12.1):  0a64ce753653cc97c0467e1334d9d3678ca8c682
netmap      (v0.12.1):  7c5bdb23e36cc7cce95bf42f3ab9e452c2501df1
reputation  (v0.12.1):  7ad824fd1eeb1565be2cee3889214b9aa605d2fc
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-28 09:35:52 +03:00
Alex Vanin
fbed86da2c [#943] service/object: Remove KeyStore from CommonPrm
There is no point to pass key storage in parameters because
it can be defined on the service level of application.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 20:25:25 +03:00
Alex Vanin
c30aa20b04 [#943] service/object: Refactor private key fetching during execution
`CommonPrm` structure has private key for remote operations.
It obtained in the beginning of request processing. However,
not every operation triggers remote calls. Therefore, key
might not be used. It is important to avoid early key fetching
because `TokenStore` now returns error if session token does not
exist. This is valid case when container nodes receive request with
session token (for ACL pass) and they should process request locally.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 20:25:25 +03:00
Alex Vanin
2fbdcbdee1 [#943] service/object: Check session token expiration
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 20:25:25 +03:00
Alex Vanin
508a28fdc0 [#943] service/object: Do not sign put service message with session token
This is invalid operation for storage nodes that receive part of split
object. While object is signed by session token, the message itself
should be signed by the node key.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 20:25:25 +03:00
Alex Vanin
46fa07b7cc [#943] service/object: Return error if requested session token is not available
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 20:25:25 +03:00
Alex Vanin
0f598289f6 [#942] neofs-cli: Return endpoint flag to control netmap-snapshot command
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 19:37:19 +03:00
Alex Vanin
2271944064 [#945] metabase: Use cleanup instead of defer in tests
Cleanup is more idiomatic way to clean test data

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 16:10:01 +03:00
Alex Vanin
e41aba610d [#945] metabase: Fix containers listing
Container listing should not ignore tombstone and
storage group objects which are not stored in
primary buckets.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 16:10:01 +03:00
Pavel Karpy
8956f015fc [#941] morph/notary: Delete useless "low till" check
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-25 19:16:05 +03:00
Pavel Karpy
78aab096fb [#941] morph/notary: Add dynamic notary deposit till calculation
Do not set `till` as some constant:
use maximum of two values instead:
1. currentDepositTill;
2. currentHeight+epochDuration+constant.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-25 19:16:05 +03:00
Evgenii Stratonikov
0057eeb0f7 [#901] *: release worker pools where possible
Some of the pools are initialized during config initialization,
so it isn't possible currently to release them in one place.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-25 18:53:36 +03:00
Evgenii Stratonikov
10afd26354 [#901] engine: release shard pools in Close
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-25 18:53:36 +03:00
Evgenii Stratonikov
0beaed2ef4 [#901] util: implement Release for PseudWorkerPool
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-25 18:53:36 +03:00
Pavel Karpy
616013cb8a [#933] write-cache: Use real number of objects in counters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-25 18:49:10 +03:00
Pavel Karpy
29955a3036 [#933] FSTree: Add NumberOfObjects method
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-25 18:49:10 +03:00
Evgenii Stratonikov
41a9261069 [#935] config/example: quote spaces in node.env
```
> source config/example/node.env
config/example/node.env: строка 14: /dns4/s02.neofs.devenv/tcp/8081: Нет такого файла или каталога
config/example/node.env: строка 16: MSK: команда не найдена
config/example/node.env: строка 35: 028f42cfcb74499d7b15b35d9bff260a1c8d27de4f446a627406a382d8961486d6: команда не найдена
config/example/node.env: строка 48: https://rpc2.morph.fs.neo.org:40341: Нет такого файла или каталога
config/example/node.env: строка 49: wss://rpc2.morph.fs.neo.org:40341/ws: Нет такого файла или каталога
config/example/node.env: строка 53: https://rpc2.n3.nspcc.ru:30341: Нет такого файла или каталога
```

New implementation is still basic, but we don't need to parse any env
file, just have a correct example.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-25 11:10:01 +03:00
Pavel Karpy
208d150500 [#937] node: Init node state before gRPC server start
Fix panic when node processes RPC requests
before state initialization is finished.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-22 15:05:00 +03:00
Alex Vanin
99f9f8dd08 Release v0.26.0 - Udo (우도, 牛島)
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-19 20:00:00 +03:00
Alex Vanin
eaecc438f8 [#932] neofs-cli: Support ContainerAliasFee global config parameter
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-19 19:15:56 +03:00
Evgenii Stratonikov
751147793f [#927] neofs-adm: provide better error message if endpoint is missing
Note that we cannot mark `--rpc-endpoint` flag as required because
it can be taken from config.

Before:
```
Error: can't create N3 client: failed to get network magic: Post "": unsupported protocol scheme ""
```

Now:
```
Error: can't create N3 client: missing endpoint
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-19 17:47:29 +03:00
Evgenii Stratonikov
f1202a5738 [#927] neofs-adm: fix dump-config command
```
Error: can't to initialize context: can't read alphabet wallets dir: open : no such file or directory
```
It needs neither contracts nor wallets, just N3 client.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-19 17:47:29 +03:00
Evgenii Stratonikov
c93c5e726c [#927] neofs-adm: add fee flags to morph init
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-19 17:47:29 +03:00
Evgenii Stratonikov
26e11a732d [#927] neofs-adm: provide container alias fee on deploy
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-19 17:47:29 +03:00
Pavel Karpy
0866c1fb90 [#920] cli: Rename RPC flag for control service
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-18 12:19:02 +03:00
Pavel Karpy
f0252e00c1 [#920] cli: Do not add rpc flags to util
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-18 12:19:02 +03:00
Pavel Karpy
b148b85af4 [#919] pkg/object: Do not accept objects with empty owner
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-18 12:15:06 +03:00
Evgenii Stratonikov
7cb3d0cb4a [#885] policer: remove objects for removed container
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-18 12:14:14 +03:00
Evgenii Stratonikov
1a1435be3d [#791] neofs-lens: inspect objects by address
```
> neofs-lens list --path ./blob/blobovnicza/1/1/0
6ay4GfhR9RgN28d5ufg63toPetkYHGcpcW7G3b7QWSek/9ibXu6v4uTwLEcME5vyHev6Zi8LpxqiWTe1dahKpAbb6
H3VBttoLQoknzMDgnVNyLZ8EpkDnQjnaxDr9fnAWeEHA/Hw1titdGh7BrTe2yLotiYbVh9FQaRRNhoNzXTyetpFgt

> neofs-lens inspect --path ./blob/blobovnicza/1/1/0 \
    --address 6ay4GfhR9RgN28d5ufg63toPetkYHGcpcW7G3b7QWSek/9ibXu6v4uTwLEcME5vyHev6Zi8LpxqiWTe1dahKpAbb6 \
    --header --out payload
Version: v2.1
Type: REGULAR
CID: 6ay4GfhR9RgN28d5ufg63toPetkYHGcpcW7G3b7QWSek
ID: 9ibXu6v4uTwLEcME5vyHev6Zi8LpxqiWTe1dahKpAbb6
Owner: 2dokPzmmcLnnR21jQB3qPppTQRgwMNMKEWD
CreatedAt: 0
PayloadSize: 32
Attributes:
  foo: bar

> hexdump -C payload
00000000  ff 6c d4 71 c4 83 f1 5f  b9 0b ad b3 7c 58 21 b6  |.l.q..._....|X!.|
00000010  d9 55 26 a4 1a 95 04 68  0b 4e 7c 8b 76 3a 1b 1d  |.U&....h.N|.v:..|
00000020
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-18 11:55:02 +03:00
Leonard Lyubich
e0f0188466 [#907] container/put: Work with named containers
Add name and zone arguments to `Put` method of wrapper over the Container
contract client. Pass result of `container.GetNativeNameWithZone` function
to the method in `Put` helper function. Due to this, the storage node will
call the method depending on the presence of the container name in the
attributes.

Make IR to listen `putNamed` notification event. The event is processed like
`put` event, but with sanity check of the container attributes.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-18 11:54:34 +03:00
Leonard Lyubich
7db47c88bf [#907] morph/container: Add native name and zone to PutArgs
Add `PutArgs.SetNativeNameWithZone` method which sets native name and zone
for container. Call `putNamed` method of Container contract if name is set,
otherwise call `put` method.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-18 11:54:34 +03:00
Leonard Lyubich
36c5e4c527 [#907] morph/event/container: Parse PutNamed event
Implement `ParsePutNamedNotary` function which parses `PutNamed` structure
from `event.NotaryEvent`. Share common code with `ParsePutNotary` function.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-18 11:54:34 +03:00
Leonard Lyubich
a9494412da [#907] morph/event/container: Define PutNamed event
Define `PutNamed` structure of notary notification from `putNamed` method of
Container contract. Embed `Put` type in order to inherit methods and
parsing of common parts with `put` method.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-18 11:54:34 +03:00
Leonard Lyubich
3173bf345e [#907] morph/event: Implement Op to string converter
Implement `StringFromOpcode` function that tries to retrieve `string` to
`Op`. Add a comment about neo-go source code that is used for implementation
of converters.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-18 11:54:34 +03:00
Evgenii Stratonikov
123328a2f4 blobstor: close all opened blobovniczas
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-18 10:44:50 +03:00
Evgenii Stratonikov
1ebe188cdc [#926] Makefile: declare images target as phony
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-18 10:43:36 +03:00
Leonard Lyubich
1b3637aad7 [#884] shard: Mark expired tombstones as garbage
Mark expired tombstones as garbage in `Shard.HandleExpiredTombstones`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-15 16:09:44 +03:00
Leonard Lyubich
8efeba8010 [#833] morph/subscriber: Change value cast for NotificationEventID
`WSClient` of Neo Go v0.97.3 sets value of notification with
`NotificationEventID` to `subscriptions.NotificationEvent` type which wraps
previously used `state.NotificationEvent`.

Change type cast and pull `state.NotificationEvent` structure from new type.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-15 13:56:19 +03:00
Leonard Lyubich
e5e5395830 [#833] cli/netmap: Beautify output of MillisecondsPerBlock parameter
Convert milliseconds to `time.Duration` value and use its stringer in
`netinfo` command.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-15 13:56:19 +03:00
Leonard Lyubich
8f9b1fe090 [#833] cli: Write MillisecondsPerBlock and NeoFS network config
Print MillisecondsPerBlock and NeoFS network parameters in `netinfo` command
of `netmap` section.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-15 13:56:19 +03:00
Leonard Lyubich
283ccc04b4 [#833] services/netmap: Support new fields of netmap.NetworkInfo
Make the implementation of network info source (Netmap V2 service
dependency) to read MillisecondsPerBlock sidechain parameter and NeoFS
network parameters depending on the client version.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-15 13:56:19 +03:00
Leonard Lyubich
7582689de4 [#833] morph/netmap: Support ListConfig contract method
Implement `Client.ListConfig` method which calls `listConfig` method of
Netmap contract. Implement `Wrapper.IterateConfigParameters` method which
uses previous one. Implement `wrapper.WriteConfig` helper function which
allows to interpret parameters by names.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-15 13:56:19 +03:00
Leonard Lyubich
cb22e2bf29 [#833] morph/client: Implement MsPerBlock method
Add `Client.MsPerBlock` method which reads MillisecondsPerBlock network
parameter.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-15 13:56:19 +03:00
Leonard Lyubich
3929f6b396 [#833] Upgrade Neo Go and NeoFS API Go libraries
Upgrade Neo Go lib to release v0.97.3.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-15 13:56:19 +03:00
Pavel Karpy
96a65df32b [#873] innerring: Reuse atomic IR's epochDuration instead of RPC call
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-15 12:22:25 +03:00
Pavel Karpy
505d92bb06 [#873] morph/notary: Not warn at "low till value" notary deposit error
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-15 12:22:25 +03:00
Pavel Karpy
14f01fa953 [#873] Update CHANGELOG
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-15 12:22:25 +03:00
Pavel Karpy
fae2c21165 [#873] innerring/config: Delete useless defaults
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-15 12:22:25 +03:00
Pavel Karpy
6638f7d9e6 [#873] innerring: Use multihanders in the New epoch timer
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-15 12:22:25 +03:00
Pavel Karpy
4afccab015 [#873] innerring: Delete duration from notary configs
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-15 12:22:25 +03:00
Pavel Karpy
e5748bfc96 [#873] innerring/notary: Change notary deposit logic
IR tries to keep 1:3 proportion of GAS and
notary balances respectively. If that proportion
has been messed(means that notary balance is
lower than required) it sends half of its
GAS balance to the notary service.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-15 12:22:25 +03:00
Pavel Karpy
a25bd2112d [#873] innerring/timers: Add multihandlers functionality for new epoch timer
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-15 12:22:25 +03:00
Pavel Karpy
39b04ff749 [#873] morph/notary: Not warn at "low till value" notary deposit error
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-15 12:22:25 +03:00
Pavel Karpy
2f2258733b [#873] node: Delete useless notary deposit Amount, Duration envs
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-15 12:22:25 +03:00
Pavel Karpy
8374c3d2f5 [#873] node: Delete useless notary timer
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-15 12:22:25 +03:00
Pavel Karpy
2a3a6cc0ba [#873] node: Use dynamic notary deposit duration
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-15 12:22:25 +03:00
Pavel Karpy
5e816dc01a [#873] node: Make deposits depend on epoch and balances
SN tries to keep 1:3 proportion of GAS and
notary balances respectively. If that proportion
has been messed(means that notary balance is
lower than required) it sends half of its
GAS balance to the notary service.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-15 12:22:25 +03:00
Pavel Karpy
d2096b392c [#873] morph/client: Add function that calculates notary deposit
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-15 12:22:25 +03:00
Evgenii Stratonikov
d55456f3ac [#905] morph/client: handle latest NNS version
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-14 15:21:12 +03:00
anastasia prasolova
029c577a33 [#916] makefile: Removed -i key from docker run
Interactive terminal doesn't work in Jenkins jobs.

Signed-off-by: anastasia prasolova <anastasia@nspcc.ru>
2021-10-14 15:19:33 +03:00
Leonard Lyubich
3e4f0c1eb9 [#674] object/put: Handle job submission errors
Close wait channel on shard pool error in order to prevent deadlock.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-14 10:20:39 +03:00
Leonard Lyubich
40a4a7faa2 [#674] object/put: Use pseudo worker pool for local operations
After storage engine started to limit number of PUT operations there is no
need to limited worker pool in Object Put service.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-14 10:20:39 +03:00
Leonard Lyubich
07130855aa [#674] util: Rename SyncWorkerPool with PseudoWorkerPool
`SyncWorkerPool` name is more appropriate for worker pool of size 1.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-14 10:20:39 +03:00
Leonard Lyubich
43ccab3294 [#674] storage engine: Fix unit test failed on shard pool NPE
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-14 10:20:39 +03:00
Leonard Lyubich
2126235f0e [#674] node: Configure size of per-shard worker pools
Add `shard_pool_size` config to `storage` section. Set app default to 20.
Pass the value to `WithShardPoolSize` option.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-14 10:20:39 +03:00
Leonard Lyubich
5b1975d52a [#674] storage engine: Use per-shard worker pools for PUT operation
Make `StorageEngine` to use non-blocking worker pools with the same
(configurable) size for PUT operation. This allows you to switch to using
more free shards when overloading others, thereby more evenly distributing
the write load.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-14 10:20:39 +03:00
Alex Vanin
b08c6b3f30 [#913] config/testnet: Fix configuration after testnet redeploy
Contracts section can be omitted in neofs-node v0.25.x. But
upcoming changes in NNS contract are going to be incompatible
with v0.25.x, so it is better to stick with smart contract
addresses in config file.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-13 14:26:49 +03:00
Pavel Karpy
501c78f327 [#902] util/locode: Fix parsing minutes
Convert minutes of received coordinates
to decimal parts of degree, and do not
use decimal part of float as storage for
minutes: "5915N 01806E" is
59.25N 18.10E, not 59°15'N 18°06'E.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-12 22:25:01 +03:00
Evgenii Stratonikov
2f149f95d4 [#883] network: parse URI properly
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-12 15:50:51 +03:00
Evgenii Stratonikov
f836e7c1dc [#888] neofs-adm: use constant for update methods
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-12 15:50:22 +03:00
Evgenii Stratonikov
31b3c71457 [#888] neofs-adm: allow to work with multiple NNS versions
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-12 15:50:22 +03:00
Evgenii Stratonikov
6571f9214f [#888] neofs-adm: update container contract deploy parameters
It supports NNS domain and zone now.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-12 15:50:22 +03:00
Evgenii Stratonikov
8ddd0aab55 [#888] neofs-adm: set contract hashes on read
Non-alphabet contract are always deployed from committee
so it makes sense to calculate hashes in one place.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-12 15:50:22 +03:00
Evgenii Stratonikov
6bcd4811e2 [#888] neofs-adm: update NNS wrappers
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-12 15:50:22 +03:00
Evgenii Stratonikov
bca41f87af [#888] neofs-adm: deploy NNS contract first
Container contract uses actual NNS interface. This also aleviates some
pain related to update as neofs-adm code itself uses current NNS
version.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-12 15:50:22 +03:00
Evgenii Stratonikov
1177f0ca78 [#888] neofs-adm: deploy contracts with Global scope
Container contract now calls NNS contract to register TLD for nice
names. This must be witnessed by a committee.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-12 15:50:22 +03:00
Evgenii Stratonikov
ec27a96b8c [#888] neofs-adm: dump NNS contract hash
This can be useful for debugging.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-12 15:50:22 +03:00
Sergio Nemirowski
57007f4a52 [#858] config/example: Set log levels in verbosity order
Signed-off-by: Sergio Nemirowski <sergio@nspcc.ru>
2021-10-12 12:16:53 +03:00
Sergio Nemirowski
62093c1524 change vmagent port to 443
Update vmagent to an actual collectors configuration, that use default https port now.

Signed-off-by: Sergio Nemirowski <sergio@nspcc.ru>
2021-10-12 12:07:41 +03:00
Alex Vanin
e500bd13a9 [#906] neofs-adm/docs: Fix notes from Fyrchik
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-11 18:33:45 +03:00
Elizaveta Chichindaeva
6c155f62c4 [#906] neofs-adm/docs: Fix language
Signed-off-by: Elizaveta Chichindaeva <elizaveta@nspcc.ru>
2021-10-11 18:33:45 +03:00
Alex Vanin
d50e8f2e72 [#906] neofs-adm/docs: Add step-by-step deploy guide
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-11 18:33:45 +03:00
Alex Vanin
3c47cff717 [#906] neofs-adm: Add README file
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-11 18:33:45 +03:00
Alex Vanin
d8f7fed10a [#881] acl: Use session token from request at object.Put
Session token can be present in both object header and
request meta header. They are the same during initial object
placement.

At the object replication, storage node puts object without
any session tokens attached to the request. If container's eACL
denies object.Put for USER role (use bearer to upload), then
replication might fail on objects with session tokens of the
signed by container owner. It is incorrect, so use session
token directly from request meta header.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-08 17:18:32 +03:00
Alex Vanin
0126f18531 [#886] cli: Fix RPC flags for control service
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-08 17:18:18 +03:00
Evgenii Stratonikov
dd678cd976 [#868] blobstor: allow to decompress objects on-the-fly
We should be able to read whatever we have written earlier.
Compression setting applies only to the new objects.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-08 16:01:45 +03:00
Evgenii Stratonikov
cc377b34d2 [#868] blobstor: initialize (de-)compressors in Init
Do not log in options constructors. Also failure to
initialize compression module (possibly due to invalid options) is
certainly an error deserving proper treatment.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-08 16:01:45 +03:00
Pavel Karpy
fa1dc31320 [#894] config/example: Fix comments
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-08 11:30:10 +03:00
Evgenii Stratonikov
00f14d4dcd [#857] config: use size suffixes where possible
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-08 11:26:24 +03:00
Evgenii Stratonikov
a0abf10c8e [#857] config: allow to parse sizes with suffix
The code is taken from viper as `cast` package that we use
doesn't have needed converter.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-08 11:26:24 +03:00
Evgenii Stratonikov
b8ba677c85 [#882] policer: add CID to the error message
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-08 08:21:01 +03:00
Pavel Karpy
4912cdaaca [#886] Update CHANGELOG.md
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-07 18:54:34 +03:00
Pavel Karpy
4615ff1392 [#886] cli: Fix API flags
Bind API cobra flags(`ttl' and xhdr')
to the Viper.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-07 18:54:34 +03:00
Pavel Karpy
6b1e6456ab [#887] make: Update linter
`v1.30` linter version does not support
`revive` linter, so update it to `v1.42.1`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-07 16:30:02 +03:00
Pavel Karpy
40b2a03b49 [#874] node/config: Add information about comments to README.md
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-07 14:48:17 +03:00
Pavel Karpy
17dcc1299f [#874] node/config: Add comments for config values
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-07 14:48:17 +03:00
Angira Kekteeva
95620dcfbd [#875] ci: Update linter
Update golangci version, replace golint by revive

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-10-07 11:01:07 +03:00
Alex Vanin
e41e74b5fc [#872] services/container: Ignore passed routes in load router
Higher level solution for original issue with lost announcements
from single node container.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-05 19:29:27 +03:00
Alex Vanin
f08636c518 [#872] cmd/neofs-node: Don't lose local container size estimations
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-05 19:29:27 +03:00
Leonard Lyubich
4d7915a366 [#791] neofs-lens: Implement object listing
Implement `list` command for object listing. Support listing of the objects
from write-cache and blobovnicza.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-05 10:51:10 +03:00
Leonard Lyubich
522cbab47c [#791] cmd: Implement application observing storage engine
Implement skeleton of `neofs-lens` app which is going to be used for working
with storage engine's data.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-05 10:51:10 +03:00
Leonard Lyubich
8a1593fdcc [#791] writecache: Export OpenDB function
The function will be useful for the tool which works with write-cache parts.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-05 10:51:10 +03:00
Leonard Lyubich
ca4c9d4673 [#791] writecache: Implement iterator over small objects
There is a need to list addresses of the small objects stored in WriteCache
database.

Implement `IterateDB` function which accepts BoltDB instance and iterate
over all saved objects and passes their addresses to the hander.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-05 10:51:10 +03:00
Leonard Lyubich
f73c5c2259 [#791] blobovnicza: Support read-only mode
There is a need to open Blobovnicza instances in read-only mode in some
cases.

Add `ReadOnly` option. Do not create dir path in RO. Open underlying BoltDB
instance with ReadOnly flag. Document thal all writing operations should not
be called in ro (otherwise BoltDB txs fail).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-05 10:51:10 +03:00
Leonard Lyubich
cc5d3288a1 [#791] blobovnicza: Don't iterate over range buckets in Iterate
`Blobovnicza` can be initialized with any number of range buckets, and
reconstructed with different size limit. In previous implementation
`Iterate` could miss some stored objects if we construct `Blobovnicza` with
smaller number of ranges.

Make `Iterate` to traverse all buckets regardless of current instance
bounds.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-05 10:51:10 +03:00
Leonard Lyubich
5e2ca0d04b [#791] blobovnicza: Implement method to iterate over object addresses
In previous implementation `Blobovnicza.Iterate` op decoded object data only
and passed it to the handler. There is a need to iterate over all addresses
of the stored objects.

Add `DecodeAddresses` and `WithoutData` methods of `IteratePrm` type. Add
`Address` method to `IterationElement` type. Make `Iterate` to decode object
addresses if `DecodeAddress` was called and not read the data if
`WithoutData` was called. Implement `IterateAddresses` helper function to
simplify the code.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-05 10:51:10 +03:00
Pavel Karpy
8c59ade4ed [#854] cli: Do not use global flags
Also delete `ttl` and `xhdr` flags from
`accounting balance` command and refactor
command initialization.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-04 12:12:21 +03:00
Evgenii Stratonikov
4ccb3d05d8 [#835] neofs-adm: rename contract update method
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-09-30 20:57:24 +03:00
Alex Vanin
f2a61451b7 [#867] governance: Add option to disable governance sync
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-30 20:57:13 +03:00
Leonard Lyubich
4661f65975 [#645] client/cache: Check response public key in all client operations
There is a need to check if public key in the RPC response matches the
public key of the related storage node declared in network map.

Define `ErrWrongPublicKey` error. Implement RPC response handler's
constructor `AssertKeyResponseCallback` which checks public key. Construct
handler and pass it to client's option `WithResponseInfoHandler`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-30 20:57:00 +03:00
Leonard Lyubich
78e4a87dca [#645] client/cache: Use public key for cache key calculation
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-30 20:57:00 +03:00
Leonard Lyubich
312e356a75 [#645] object: Compare public keys in reputation client constructor
In previous implementation of reputation client constructor compared network
addresses to check the membership of the network map.

Replace network addresses comparison with public key comparison.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-30 20:57:00 +03:00
Leonard Lyubich
81e0396c13 [#645] ir/rpc: Revert using client cache as interface
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-30 20:57:00 +03:00
Leonard Lyubich
3a497a3c53 [#645] object/put: Pass NodeInfo instead of AddressGroup to relay func
Replace `network.AddressGroup` parameter type of relay function with
`client.NodeInfo`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-30 20:57:00 +03:00
Leonard Lyubich
e473f3ac91 [#645] *: Use helper functions to build client.NodeInfo structures
Helper functions from core/client package allow to set public keys of
storage nodes.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-30 20:57:00 +03:00
Leonard Lyubich
2d441a4cc6 [#645] placement/traverser: Rename Key method of Node type to PublicKey
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-30 20:57:00 +03:00
Leonard Lyubich
3d3d30560a [#645] core/client: Implement helper functions to fill NodeInfo
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-30 20:57:00 +03:00
Leonard Lyubich
4ec7e24b85 [#645] core/client: Add public key to NodeInfo structure
There is a need to process announced public keys of storage nodes for
intra-container communication.

Implement `PublicKey` / `SetPublicKey` methods of `client.NodeInfo` type.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-30 20:57:00 +03:00
Leonard Lyubich
7b228b7603 [#645] *: Construct clients from client.NodeInfo in API client cache
There is a need to have the ability to expand the data needed for client
construction.

Replace `network.AddressGroup` parameter of client cache interfaces with
`client.NodeInfo`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-30 20:57:00 +03:00
Leonard Lyubich
91cc33bdb9 [#645] core/client: Define node information structure
Define `client.NodeInfo` structure which groups data required for API client
construction.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-30 20:57:00 +03:00
Alex Vanin
8aa290c8c3 [#627] processors/netmap: Read address of contract from client wrapper
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-30 18:02:58 +03:00
Leonard Lyubich
8fc1505351 [#627] ir: Read addresses of contracts from the client wrappers
After recent changes morph client wrappers provide contract address getter.
It can be used to compose notification parsers and handlers.

Use `ContractAddress` method in constructors of notification parsers and
handlers. Remove no longer used script hash parameters of event processors.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-30 14:47:04 +03:00
Leonard Lyubich
d6c0307431 [#627] morph: Inherit internal.StaticClient interface in all wrappers
There is a need to provide contract address getter from all contract client
wrappers.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-30 14:47:04 +03:00
Leonard Lyubich
1860f5040c [#627] morph: Define partial interface of StaticClient type
There is a need to inherit some methods of `StaticClient` type. In order to
not inherit all method via type embedding we can group sub-set of methods
and inherit it.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-30 14:47:04 +03:00
Leonard Lyubich
6ce30c959c [#627] morph: Add contract address getter to StaticClient
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-30 14:47:04 +03:00
Alex Vanin
e558cdd9dd [#867] governance: Add more logs of the alphabet and inner ring lists
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-30 13:56:34 +03:00
Leonard Lyubich
be05bed0b3 [#860] Re-compile protobuf files after make protoc target update
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-29 17:51:33 +03:00
Leonard Lyubich
ea6bc8e892 [#860] Makefile: use protoc-gen-go-grpc in protoc target
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-29 17:51:33 +03:00
Leonard Lyubich
6f35405dca [#860] Update google.golang.org/grpc module to v1.41.0
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-29 17:51:33 +03:00
Sergio Nemirowski
78c8e91a9d fix #856, update issue templates
This comments out user notes that should not go to the reports.

Signed-off-by: Sergio Nemirowski <sergio@nspcc.ru>
2021-09-29 15:32:13 +03:00
Leonard Lyubich
66d54826e9 config/testnet: Update image version to v0.25.1
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-29 13:45:13 +03:00
Leonard Lyubich
ec06f02181 Update changelog for release v0.25.1
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-29 13:45:13 +03:00
Leonard Lyubich
287683c590 [#863] node: Do not open main chain client connection
In current implementation storage node doesn't use main chain client.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-29 12:55:48 +03:00
Leonard Lyubich
50c4c5116d [#863] node: Prevent panic about missing Neo RPC endpoints
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-29 12:55:48 +03:00
Leonard Lyubich
8efea6e066 [#792] ir: Support multiple notification endpoints
Accept notification endpoints as string slice from config. Work with the
first successfully initialized WSClient.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-28 18:47:06 +03:00
Alex Vanin
c8ba562fe9 [#853] config: Update config records for 0.25 release
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-27 18:00:06 +03:00
Alex Vanin
989ed5353c Release v0.25.0 - Mungapdo (문갑도, 文甲島)
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-27 16:37:19 +03:00
Alex Vanin
409b72cb5b [#786] neofs-node: Ignore proxy contract in notary disabled environment
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-27 16:20:31 +03:00
Leonard Lyubich
0e3e8db5c0 [#845] node: Configure Object PUT remote and local pools separately
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-27 16:20:19 +03:00
Leonard Lyubich
ee20200c2e [#845] object/put: Separate pools for local and remote operations
In previous implementation Object PUT used single pool of workers for local
and remote ops, but these ops are heterogeneous.

Use remote/local pool for remote/local operations in PUT service. At first
the pools are configured with the same size.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-27 16:20:19 +03:00
Leonard Lyubich
3b2b6007c6 [#845] object/put: Make distributed target to define is node is local
In previous implementation `distributedTarget` didn't check if next node is
local. This check was performed by the handlers (target initializer and
relay func).

Make `distributedTarget` to calculate node's locality. Pass locality flag to
the handlers.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-27 16:20:19 +03:00
Leonard Lyubich
c96c455125 [#840] shard: Fix metabase Inhume parameters for tombstones
Replace `WithAddresses` with `WithTombstoneAddress` on `InhumePrm`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-27 11:27:41 +03:00
Leonard Lyubich
73fb1a886c [#849] policer: Write message about redundant local object copy
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-27 11:27:41 +03:00
Leonard Lyubich
d613a856ce [#849] policer: Log object address in processNodes method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-27 11:27:41 +03:00
Leonard Lyubich
14329ab565 [#840] metabase: Distinguish objects with tombstones and GC marks
Each object from graveyard has tombstone or GC mark. If object has
tombstone, metabase should return `ErrAlreadyRemoved` on object requests.
This is the case when user clearly removed the object from container. GC
marks are used for physical removal which can appear even if object is still
presented in container (Control service, Policer job, etc.). In this case
metabase should return 404 error on object requests.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-27 11:27:41 +03:00
Leonard Lyubich
02e6df683a [#840] shard: Return only physically stored objects from List
`List` method of `Shard` must return only physically stored objects.

Use `AddPhyFilter` to select only phy objects.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-27 11:27:41 +03:00
Alex Vanin
b97cdbea9e [#839] service/object: Add high level functions to control request forwarding
Applicable for object/get execution context.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-27 11:10:50 +03:00
Alex Vanin
b3464e8140 [#839] service/object: Do not use request forwarding when assembling object
Forwarding mechanism resends original request. During split object chain traversal,
storage node performs multiple `object.Head` requests on each child. If request
forwarding happens, then `object.Head` returns object ID of the original request.
This produces infinite assembly loop.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-27 11:10:50 +03:00
Pavel Karpy
057d3ac06b [#846] innerring: Reuse original TX nonce from SN's NR
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-27 11:10:15 +03:00
Pavel Karpy
5702349cb2 [#846] innerring: Use epoch as nonce in cleanup netmap process
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-27 11:10:15 +03:00
Pavel Karpy
e3c0288e50 [#846] morph/notary: Add nonce parameter to notary invocation method
This prevents notary requests collisions
for TXs that contains equals hashable fields.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-27 11:10:15 +03:00
Pavel Karpy
3a8f0edac1 [#846] morph/event: Add notary request to addPeer event
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-27 11:10:15 +03:00
Pavel Karpy
5e74830c38 [#837] morph: Add WithSingleClient client constructor option
`WithSingleClient` allows Morph client
creation with existing raw neo-go client.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-24 19:14:28 +03:00
Leonard Lyubich
264ab489bb [#848] ir/container: Prevent potential NPE in key conversion
Set `Curve` field in `ecdsa.PublicKey` instance from `keys.PublicKey` one in
`checkKeyOwnership` method of container processor.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-24 18:57:00 +03:00
Alex Vanin
3e5c7e0ade [#836] neofs-adm: Fix proxy contract deploy arguments
Proxy contract does not include notary flag because
contract is useless without notary subsystem.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-23 09:55:46 +03:00
Alex Vanin
4c30757439 [#836] neofs-adm: Do not define contract owner
Contract owners are removed in neofs-contract v0.11.0.
Now side chain committee has rights to update contracts.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-23 09:55:46 +03:00
Alex Vanin
ce8a906bb5 [#834] neofs-adm: Update NNS contract during contract update
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-22 18:23:59 +03:00
Alex Vanin
d996004d80 [#834] neofs-adm: Use committee signature when updating alphabet contract
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-22 18:23:59 +03:00
Alex Vanin
6b3a9e6fcc [#834] neofs-adm: Update contract update if it exists
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-22 18:23:59 +03:00
Alex Vanin
60636d4c1d [#834] neofs-adm: Throw error if deploy produce non-HALT state
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-22 18:23:59 +03:00
Leonard Lyubich
504abdda06 [#789] shard: Exclude object.ErrAlreadyRemoved errors in refillMetabase
Tombstone and "alive" objects can be both stored in BlobStor. They can
appear during iterating in different order. Metabase returns
`ErrAlreadyRemoved` error if object is inhumed.

Ignore `object.ErrAlreadyRemoved` errors of `metabase.Put`in Shard's
`refillMetabase` operation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich
d938c7267b [#789] node/config: Add Shard's refill_metabase config to examples
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich
b5fff810f4 [#789] cmd/node: Add refill_metabase config to shard section
Implement `RefillMetabase` method to `shardconfig.Config` type which reads
`refill_metabase` config value. Pass the result to `WithRefillMetabase`
option.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich
6bf7a00cfe [#789] shard: Add option to refill metabase on initialization
Add `WithRefillMetabase` option constructor which allows to set flag to
refill metabase.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich
b9c22e21b1 [#789] shard: Add method to refill the metabase
There is a need to refill Metabase data with the objects from BlobStor.

Implement `refillMetabase` method which iterates over all objects from
BlobStor and saves them in Metabase.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich
8d016d2529 [#789] blobstor: Implement iterator
There is a need to be able to process all objects saved in `BlobStor`.

Implement `BlobStor.Iterate` method which iterates over all objects.
Implement `IterateBinaryObjects` and `IterateObjects` helper functions to
simplify the code.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich
b618a44d69 [#789] blobovnicza: Implement iterator
There is a need to be able to process all stored objects saved in
`Blobovnicza`.

Implement `Blobovnicza.Iterate` method which iterates over all objects.
Implement `IterateObjects` helper function to simplify the code.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich
55c94a0152 [#789] metabase: Implement Reset method
In the previous implementation of the metabase, there was no possibility of
reinitializing the metabase: clearing information about existing objects and
bringing it back to its initial state. This operation can be useful in
cases when the stored metadata about objects has lost (or possibly lost)
relevance, and you need to generate data from scratch. Also at the
initialization stage, static resources of the base were not created -
container-independent buckets.

Make `Metabase.Init` method to allocate graveyard, container-size and
to-move-it buckets in underlying BoltDB instance. Implement `Metabase.Reset`
method: it works like `Init` but clean up all static buckets and removes
other ones. Due to the logical similarity, the methods share a single piece
of code.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Pavel Karpy
7a15e649ba [#825] event/notaryPreparator: Add unit test for incorrect Main TX script
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 18:07:50 +03:00
Pavel Karpy
1b4f8f7e9f [#825] event/notaryPreparator: Add unit test for incorrect NR
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 18:07:50 +03:00
Pavel Karpy
5d791a4366 [#825] morph/event: Fix commentary
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 18:07:50 +03:00
Pavel Karpy
f0beb4abc9 [#825] event/notaryPreparator: Add args checking unit test
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 18:07:50 +03:00
Leonard Lyubich
368e280413 [#776] cmd/node: Configure write-cache size limit
Add `size_limit` config value to write-cache section. Add `SizeLimit` method
of `writecache.Config` type. Pass its value to `WithMaxCacheSize` option on
app construction.

Reflect config update in examples. Cover new value in unit test.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:07:36 +03:00
Leonard Lyubich
a1696a81b6 [#776] writecache: Limit size of used disk space
There is a need to limit disk space used by write-cache. It is almost
impossible to calculate the value exactly. It is proposed to estimate the
size of the cache by the number of objects stored in it.

Track amounts of objects saved in DB and FSTree separately. To do this,
`ObjectCounters` interface is defined. It is generalized to a store of
numbers that can be made persistent (new option `WithObjectCounters`). By
default DB number is calculated as key number in default bucket, and FS
number is set same to DB since it is currently hard to read the actual value
from `FSTree` instance. Each PUT/DELETE operation to DB or FS
increases/decreases corresponding counter. Before each PUT op an overflow
check is performed with the following formula for evaluating the occupied
space: `NumDB * MaxDBSize + NumFS * MaxFSSize`. If next PUT can cause
write-cache overflow, object is written to the main storage.

By default maximum write-cache size is set to 1GB.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:07:36 +03:00
Stanislav Bogatyrev
0a130177d6 [#725] Simplify Makefile a little
Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2021-09-15 17:58:58 +03:00
Stanislav Bogatyrev
03b5aad2e4 [#785] Remove unused layer from builder images
Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2021-09-15 17:58:58 +03:00
Pavel Karpy
e950dfe44f [#785] docker: Do not use Alpine image for building bins
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 17:58:58 +03:00
Pavel Karpy
71df291cda [#785] docs: Add updating Go instructions
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 17:58:58 +03:00
Pavel Karpy
80df695a25 [#785] make: Add image-storage-testnet to images target deps
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 17:58:58 +03:00
Pavel Karpy
654a3a6e60 [#785] docs: Actualize release instructions
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 17:58:58 +03:00
Pavel Karpy
94f3937c0e [#785] make: Add preparing release binaries
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 17:58:58 +03:00
Pavel Karpy
9fa4a254b2 [#785] make: Add docker/* target wrapper
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 17:58:58 +03:00
Leonard Lyubich
3258d9c616 [#790] writecache: Log each writing operation
Call `storagelog.Write` in all places after the successful writing op.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-13 15:02:13 +03:00
Leonard Lyubich
4f73c00776 [#790] blobstor: Log each writing operation
Call `storagelog.Write` in all places after the successful writing op.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-13 15:02:13 +03:00
Leonard Lyubich
3cfb58aabd [#790] storage engine: Add common template of log messages
There is a need to keep track of each local storage change. Log messages are
the most convenient way to do it.

Implement function which writes log message about the completed writing
operation in storage engine.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-13 15:02:13 +03:00
Alex Vanin
d840627816 [#786] cmd/neofs-node: Use NNS to find contract script hashes missing in config
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:35 +03:00
Alex Vanin
b6dfa6c118 [#786] pkg/innerring: Use NNS to find contract script hashes missing in config
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:35 +03:00
Alex Vanin
f43f389399 [#786] morph/client: Add NNS contract address getter
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:35 +03:00
Alex Vanin
affae68061 [#812] cmd/neofs-node: Support notary notifications in reputation service
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:07 +03:00
Alex Vanin
dc26a09ec3 [#812] pkg/innerring: Support notary notifications in reputation processor
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:07 +03:00
Alex Vanin
a7f6a3df78 [#812] morph/client: Add underline client getter in reputation wrapper
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:07 +03:00
Alex Vanin
e2c2e27c60 [#812] morph/event: Expand reputation put event with notary notification
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:07 +03:00
Alex Vanin
90823a7d05 [#812] cmd/neofs-node: Fix reputation wrapper import typo
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:07 +03:00
Pavel Karpy
288a8784d0 [#815] node: Make notary deposit before bootstrap
Bootstrap process involves `addPeer` contract
call => it is necessary to make deposit before
booting up.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 14:50:52 +03:00
Pavel Karpy
b303e49408 [#815] morph/event/netmap: Add updateState notary notification support
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 14:50:52 +03:00
Pavel Karpy
74afcbe409 [#815] pkg/morph/netmap: Add Morph() method
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 14:50:52 +03:00
Pavel Karpy
00caed8d3d [#815] morph/event/netmap: Add addPeer notary notification support
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 14:50:52 +03:00
Alex Vanin
01df4ffa61 [#811] service/container: Hide cache invalidation logic in Writer interface
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:50:18 +03:00
Alex Vanin
49b6b5b49d [#811] cmd/node: Use cache invalidator in container service
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:50:18 +03:00
Alex Vanin
4a1f0de8f4 [#811] service/container: Invalidate cache records on update requests
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:50:18 +03:00
Pavel Karpy
a8d41f596d [#822] event/notaryPreparator: Add unit test for correct NR
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 10:32:43 +03:00
Pavel Karpy
3b9ef4f63c [#822] event/notaryPreparator: Do not pass PACK opcode
Do not pass high level `PACK` opcode to
notary parsers. Add opcode amount check.
Delete `PACK` cases in notary parsers.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 10:32:43 +03:00
Pavel Karpy
4f3de1a9af [#822] moprh/event: Add parsers from Op struct
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 10:32:43 +03:00
Leonard Lyubich
61b4baf736 [#778] services/object: Wrap last client's error into errIncompletePut
Make `errIncompletePut` to be a structure which wraps single client error.
Wrap error of the last client into `errIncompletePut` during placement
execution.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-10 16:49:37 +03:00
Leonard Lyubich
02f2a98bcc [#778] services/object: Return the composite error of PUT operation
In previous implementation Object service's handler returned const error in
case of failure (full or partial) of PUT operation. This did not even allow
us to roughly guess what the reason is. Not as a complete solution, but to
alleviate some cases where all nodes in a container return the same error,
it is suggested to return the error of the last server that responded.

Return latest server error from placement loop of `iteratePlacement` method
of `distributedTarget` type.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-10 16:49:37 +03:00
Evgenii Stratonikov
5f86d54721 [#816] object/acl: fix eACL target processing
Ignore role if public keys are present.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-09-10 15:15:36 +03:00
Leonard Lyubich
7a13053fab [#818] object/acl: Fit sticky bit specification
In previous implementation sticky bit could disrupt access of container
nodes to replication. According to NeoFS specification sticky bit should not
affect the requests sent by nodes from SYSTEM group.

Add role check to `stickyBitCheck`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-10 15:14:59 +03:00
Pavel Karpy
ba77bb44e4 [#807] ir: Merge ContractProcessor and NotaryContractProcessor interfaces
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-09 18:37:45 +03:00
Pavel Karpy
539da27ccb [#807] morph/event/container: Add setEACL notary support
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-09 18:37:45 +03:00
Pavel Karpy
e03b44ffc1 [#807] morph/event/container: Add delete notary support
Includes:
- `delete` container notary notification;
- refactor in processor bindings;
- fixed conflicts in `put` notifications.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-09 18:37:45 +03:00
Pavel Karpy
19e97e4d7c [#807] node: Fix notary container invocation
Create non-notary container wrapper and
use it in load announcing since it shouldn't
be invoked as notary request.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-09 18:37:45 +03:00
Pavel Karpy
45aac7bb7f [#807] event/listener: Add event information to error logs
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-09 18:37:45 +03:00
Pavel Karpy
ffde45d164 [#807] morph/container: Add opcode check
Parsers return error if unexpected opcode
returns.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-09 18:37:45 +03:00
Pavel Karpy
6f41a979ca [#814] morph/event/container: Fix param amount check
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-09 14:43:29 +03:00
Pavel Karpy
b4059f652e [#814] morph/event: Fix NotaryPreparator parsing
Add checking for call flag in TX's script
and do not pass it to parsers.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-09 14:43:29 +03:00
Pavel Karpy
1ed5323850 [#801] morph/wrappers: Sync commentary and struct names
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-09 14:07:57 +03:00
Pavel Karpy
d0160b23e5 [#801] linter: Fix errcheck warning
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-09 14:07:57 +03:00
Pavel Karpy
425c02b0ec [#801] morph/netmap: Add wrapper options
Add constructor options on wrapper level
of `netmap` contract.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-09 14:07:57 +03:00
Alex Vanin
74051556de Release v0.24.1
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-07 17:56:32 +03:00
Alex Vanin
1edf40f4d6 [#798] morph/subscriber: Remove RPC Init timeout
More convenient way is to fail straight away and
expect external restart.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-07 17:22:25 +03:00
Alex Vanin
4874b4ae92 [#798] cmd/neofs-node: Save latest processed block number
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-07 17:22:25 +03:00
Alex Vanin
cdb3b71070 [#798] neofs-node/config: Add persistent_state section
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-07 17:22:25 +03:00
Alex Vanin
005f54e61e [#798] pkg/innerring: Save latest processed block number
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-07 17:22:25 +03:00
Alex Vanin
2bcf22ad79 [#798] pkg/innerring: Store withoutMainNet flag in server
withoutMainNet flag will be reused in runtime server start
to enable main chain block number state saver.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-07 17:22:25 +03:00
Alex Vanin
67471a8914 [#798] morph/subscriber: Await for RPC initialization
Remote RPC node might be in the transition state and produce
events from the past. We should avoid listening such nodes.
To do that subscriber component can await minimal height of
remote node. If minimal height is not reached, then throw
error.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-07 17:22:25 +03:00
Alex Vanin
7f074a775e [#798] util/state: Implement persistent storage for application state
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-07 17:22:25 +03:00
Pavel Karpy
b4378f7c11 [#770] Update changelog
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
4a81781c0c [#770] logger: Delete spaces in log message keys
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
2f343a15e5 [#770] node/cmd: Add TryNotary to SN container wrapper
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
1fedd8f860 [#770] morph/event/container: Sync comment and method name
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
722b844aa2 [#770] node: Add notary deposit timer
Storage Node needs to have notary deposit
for successful notary request sending.
Add notary deposit on startup(and wait for
its acceptance). Add notary deposit timer,
its config in `morph` section and env vars
for its tuning.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
e29bcd98e2 [#770] node/config: Add notary deposit config
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
2ef5e86aaf [#770] node/config: Add uint32 casting
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
d77b2d1b76 [#770] node/config: Add proxy contract to config
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
13f1273e82 [#770] ir/container: Support notary Put
Implement `NotaryContractProcessor` by IR
container processor. Add support for notary
`put` container operation. Do not parse `put`
non-notary notifications in notary enabled
environment.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
9921358f09 [#770] pkg/morph: Add NotarySignAndInvokeTX
Add `NotarySignAndInvokeTX` method to morph
client. This function allows invoking notary
request with passed main TX(not creating a
new one). It signs passed main TX with
client's key.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
f4a521cfd8 [#770] pkg/innerring: Make all morph wrappers Alphabet
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
79b350b628 [#770] morph: Support non-alpha notary request by wrappers
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
94d431e56e [#770] morph/client: Support non-alpha notary request
Add `NotaryInvokeNotAlpha` to low-level
client. It creates and sends notary request
that must be signed by Alphabet nodes, but
does not sign it by current node's private
key.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
b46adf188c [#770] pkg/morph/container: Add Morph() method
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
46df288d33 [#770] pkg/morph: Add Morph() method
Add access to low-level morph client in
wrappers

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
4e5618aecb [#770] pkg/innerring: Add NotaryContractProcessor interface
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
d40f898ee5 [#770] morph/event/container: Add notary Put parser
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
c042f6a429 [#770] pkg/morph/event: Add notary notifications support
Add handlers and parsers functionality for listener.
Separate notification and notary events by files.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
8f2924d6cf [#770] pkg/morph/event: Add notary request preparator
Add preparator for notary requests. Is parses
raw notary requests, checks if it should be
handled by Alphabet node. If handling is required,
returns `NotaryEvent` that contains information
about contract scripthash, method name and
arguments of the call.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
8d1f2e66e6 [#770] pkg/innerring: Fix commentary to bind functions
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
d252aa4a3e [#770] pkg/morph: Rename all parsers and handlers structs/interfaces
Prepare all listening structures for notary events:
rename(add prefix/suffix 'notification') all
notification specific handlers/parsers.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
c87bc70536 [#770] pkg/morph/event: Fix typo in AddPeer parser
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
f0cbf2e99d [#770] pkg/morph/subscriber: Add subscription for notary
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
e24489d33c [#770] pkg/morph: Fix commentaries for Listener interface
Refactor comments so they start with the names of
methods.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
c0e25d1706 [#770] morph/client: Fix comment for fallbackTime
`fallbackTime` is delta b/w `ValidUntilBlock` of
the main transaction and block when `fallback`
transaction is sent.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
3400a071f9 [#770] ir: Reuse read notary config
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Leonard Lyubich
846dd8c89e [#645] network: Remove no longer used IsLocalAddress function
In recent changes, the locality criterion for a node has been changed to
compare public keys.

Remove no longer used `IsLocalAddress` function and `LocalAddressSource`
interface.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-07 09:53:18 +03:00
Leonard Lyubich
358e3ed8c4 [#645] *: Change the locality condition of the node from the placement
Some software components regulate the way of working with placement arrays
when a local node enters it. In the previous implementation, the locality
criterion was the correspondence between the announced network address
(group) and the address with which the node was configured. However, by
design, network addresses are not unique identifiers of storage nodes in the
system.

Change comparisons by network addresses to comparisons by keys in all
packages with the logic described above. Implement `netmap.AnnouncedKeys`
interface on `cfg` type in the storage node application.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-07 09:53:18 +03:00
Leonard Lyubich
f14987c06d [#645] core: Define interface of the tool for working with netmap keys
Announced keys of storage nodes are required for many application components
to function.

Define a unified interface for the utility for working with public keys of
nodes. Add a method to check if the key has been advertised by the local
node in the application context.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-07 09:53:18 +03:00
Leonard Lyubich
22926e8f28 [#645] placement/traverser: Extend Node type with public key
There is a need to process public keys of the placement result.

Implement `Node.PublicKey` method which returns storage node's key announced
in netmap.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-07 09:53:18 +03:00
Leonard Lyubich
fe90456dcc [#645] placement: Overload result of Traverser.Next method
In previous implementation `placement.Traverser.Next` method returned slice
of `network.AddressGroup` elements. There is a need to process keys of
storage nodes besides network addresses for intra-container communication.

Wrap `network.AddressGroup` in a new type `placement.Node` that summarizes
the storage node information required for communication. Return slice of
`Node` instances from `Traverser.Next` method. Fix compilation breaks in
dependent packages.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-07 09:53:18 +03:00
Pavel Karpy
3c848b2cad [#788] cli: Wrap/sync errors
Add context to error messages. Sync error
messages for errors with the same context.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 09:52:53 +03:00
Leonard Lyubich
ba234699cd [#794] ir/neofs: Print tx IDs in LE on Withdraw and Deposit events
IDs come from NeoFS contract in big endian, but it is customary to write in
the node logs in little endian.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-06 18:11:38 +03:00
Denis Kirillov
60b3bf3ce6 [#796] Remove aio image
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-09-03 14:58:26 +03:00
Leonard Lyubich
3ff2e31472 [#762] Reflect changes in changelog
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-02 11:33:17 +03:00
Leonard Lyubich
3f293cb55e [#762] storage/shard: Try to read headers from write-cache
Shard should try to read object headers from write-cache if it is enabled.

Extend `writecache.Cache` interface with `Head` method. Call the method in
`Shard.Head` if `Shard.hasWriteCache` returns true.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-02 11:33:17 +03:00
Leonard Lyubich
aa2151fbeb [#762] write-cache: Implement Head operation
Write cache should be able to execute HEAD operations according to spec.

Add simple implementation of `Head` method through the `Get` one. Leave
notes for future optimization.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-02 11:33:17 +03:00
Leonard Lyubich
81722c373b [#746] morph/client: Don't cache GAS contract address on NeoFS-side
Caching is performed inside `GetNativeContractHash` method of neo-go client,
so the additional cache level is redundant.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-02 11:04:38 +03:00
Leonard Lyubich
0502abc3e0 [#746] Reflect changes in changelog
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-02 11:04:38 +03:00
Leonard Lyubich
18c49c2cb9 [#746] morph/client: Break loop in iterateClients on NeoFS error
There is no need to continue iterating over Neo RPC endpoints in case of
some address-independent error (e.g. NeoFS logic error).

Unwrap and immediately return `neofsError` errors from loop in
`iterateClients`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-02 11:04:38 +03:00
Leonard Lyubich
a2b81f13b0 [#746] morph/client: Simplify loop control in iterateClients
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-02 11:04:38 +03:00
Leonard Lyubich
3b7e884e74 [#746] morph/client: Distinguish between neo-go errors and NeoFS logic
Implement `error` interface on new `neofsError` type which is a wrapper over
NeoFS-specific error. Wrap all `Client` errors except neo-go client API ones
into `neofsError`. Wrapped errors are going to be used for multi-endpoint
loop control.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-02 11:04:38 +03:00
Leonard Lyubich
85bd2a1cdf [#746] morph: Add error return of MagicNumber method
Since morph `Client` works in multi-client mode, there is an error case when
we can not get network magic when all endpoints are unavailable.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-02 11:04:38 +03:00
Leonard Lyubich
ad7ad12a0c [#746] morph: Implement and use multi-client
There is a need to work with a set of Neo RPC nodes in order not to depend
on the failure of some nodes while others are active.

Support "multi-client" mode of morph `Client` entity. If instance is not
"multi-client", it works as before. Constructor `New` creates multi-client,
and each method performs iterating over the fixed set of endpoints until
success. Opened client connections are cached (without eviction for now).

Storage (as earlier) and IR (from now) nodes can be configured with multiple
Neo endpoints. As above, `New` creates multi-client instance, so we don't
need initialization changes on app-side.

`Wait` and `GetDesignateHash` methods of `Client` return an error from now
to detect connection errors. `NotaryEnabled` method is removed as unused.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-02 11:04:38 +03:00
Alex Vanin
f9218bf84f [#787] util/attributes: Restore escape characters after caching the result
As soon as resulting list and cache operate with the attribute pointer,
we can add attribute structure immediately and set restored key and
value of the attribute later.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-01 14:29:26 +03:00
Alex Vanin
f09b7e48af [#787] util/attributes: Fix order of require.Equal arguments
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-01 14:29:26 +03:00
Alex Vanin
2d7e9f0c40 [#787] util/attributes: Return consistent list of attributes
Attributes are linked to each other through parents, so they can
be returned in any order. However, it will be better to return
the list in consistent order to reduce entropy.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-01 14:29:26 +03:00
Alex Vanin
d8a04726ad [#787] util/attributes: Reuse attribute struct to update parents list
Parser should reuse existing attributes from the cache to update list
of the parent links in it. Parent links should be unique.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-01 14:29:26 +03:00
Alex Vanin
c1326efaf5 [#782] config/testnet: Fix examples in README
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-08-31 14:43:55 +03:00
Pavel Karpy
4806cf33c3 Release v0.24.0 - Anmyeondo (안면도, 安眠島)
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-30 18:53:45 +03:00
Leonard Lyubich
e738699fcc [#676] services/container: Cache the results of read operations
In previous implementation Container service handlers didn't cache the
results of `Get` / `GetEACL` / `List` operations. As a consequence of this,
high load on the service caused neo-go client's connection errors. To avoid
this there is a need to use cache. Object service already uses `Get` and
`GetEACL` caches.

Implement cache of `List` results. Share already implemented cache of Object
service with the Container one. Provide new instance of read-only container
storage (defined as an interface)to morph executor's constructor on which
container service is based. Write operations remained unchanged.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-08-30 18:01:26 +03:00
Pavel Karpy
c54f524df9 [#773] writecache: Delete unused dbSize param
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-27 15:05:53 +03:00
Pavel Karpy
9b32b5523d [#760] cli: Support COMMON_PREFIX matchtype
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-27 15:03:15 +03:00
Pavel Karpy
4ff98a7e2b [#760] metabase: Support COMMON_PREFIX matchtype
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-27 15:03:15 +03:00
Pavel Karpy
27f0758650 [#760] go.mod: Update api-go library
github.com/nspcc-dev/neofs-api-go v1.28.3 => v1.29.0

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-27 15:03:15 +03:00
Alex Vanin
d6439369ab [#768] object/acl: Support all eACL filter fields from the spec
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-08-26 19:01:03 +03:00
Leonard Lyubich
7a5729ea2b [#761] cmd/node: Do not perform bootstrap procedure in relay mode
Storage node should not try to register itself in network in relay mode.

Implement `needBootstrap` method which checks if node need to bootstrap.
Call `bootstrap` method in `bootstrapNode` function only on true return.
Skip re-bootstrap logic in new epoch event handler on false return.
Return an error if `ControlService.SetNetmapStatus` is called on relay
node.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-08-26 18:54:55 +03:00
Leonard Lyubich
05d5b724a9 [#761] cmd/node: Rename reBootstrapEnabled to needBootstrap
`reBootstrapEnabled` state var is not used. It is going to be used to decide
whether to bootstrap node or not.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-08-26 18:54:55 +03:00
Leonard Lyubich
3c78890b97 [#761] cmd/node: Add bootstrap method docs
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-08-26 18:54:55 +03:00
Alex Vanin
cadd94f08f [#766] Fix stylecheck import linter error
Remove redundant imports

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-08-26 17:57:15 +03:00
Alex Vanin
53f031e98c [#766] Fix misspell linter error
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-08-26 17:57:15 +03:00
Alex Vanin
30738549ef [#766] Fix makezero linter error
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-08-26 17:57:15 +03:00
Alex Vanin
b2facf5055 [#766] Add latest go1.17 tests
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-08-26 17:57:15 +03:00
Leonard Lyubich
4b7cc6e293 [#758] cmd/neofs-adm: Implement command to refill storage node's GAS
Add `refill-gas` sub-command to `morph` command which provides the ability
to refill storage node's GAS.

Command performs some actions from `generate-storage-wallet` runner, so
common code is moved to a separate function `refillGas`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-08-25 17:07:23 +03:00
Leonard Lyubich
abfcc7498c [#715] services/policer: Select pseudo-random list of objects to check
In previous implementation of Policer's job queue the same list of objects for
processing was selected at each iteration. This was caused by consistent
return of `engine.List` function.

Use `rand.Shuffle` function to compose pseudo-random list of all objects in
order to approximately evenly distribute objects to work.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-08-25 14:40:12 +03:00
Evgenii Stratonikov
f9d9f33461 neofs-adm: use nnsResolveHash instead of custom code
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-08-24 12:35:52 +03:00
Evgenii Stratonikov
72b8d919fe [#755] neofs-adm: allow to filter containers by ID
`--cid <cid1> --cid <cid2>` as well as `--cid <cid1>,<cid2>` is supported.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-08-24 12:35:52 +03:00
Evgenii Stratonikov
00a299c1a4 [#755] neofs-adm: add contract hash flag to dump-containers
`--container-contract` flag must be used for deployments without NNS.
Our current testnet sidechain is like this, for example.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-08-24 12:35:52 +03:00
Evgenii Stratonikov
5072b703bc [#755] neofs-adm: allow to restore containers
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-08-24 12:35:52 +03:00
Evgenii Stratonikov
a013dcbab5 [#755] neofs-adm: allow to dump active containers
`morph dump-containers` will dump all containers from the
contaner contract. JSON format is chosen to allow manual intervention.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-08-24 12:35:52 +03:00
Evgenii Stratonikov
58e8d6e1fd [#755] neofs-adm: print NNS records during initialization
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-08-24 12:35:52 +03:00
Alex Vanin
da19799c37 [#754] config/testnet: Update neofs-node to v0.23.1
Add pre-defined User-Agent attribute.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-08-24 10:54:33 +03:00
Pavel Karpy
53036276e5 #759] node: Log notary status on startup
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-23 15:26:10 +03:00
Pavel Karpy
85a4682c0c [#759] pkg/innerring: Log notary status on startup
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-23 15:26:10 +03:00
Evgenii Stratonikov
8178c5e69b [#757] neofs-adm: ensure notary contract is enabled
This is the only case we support, it makes sense to fail early.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-08-19 17:46:10 +03:00
Evgenii Stratonikov
33c3f18b4f [#748] neofs-adm: allow to update contracts
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-08-10 12:03:10 +03:00
Alex Vanin
1ea3463604 Release v0.23.1
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-08-06 18:47:42 +03:00
Alex Vanin
118e36c550 [#752] config/mainnet: Add config template
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-08-06 18:47:42 +03:00
Alex Vanin
06af24dd56 [#752] Update neo-go to v0.97.1
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-08-06 18:47:42 +03:00
Alex Vanin
300d8879a6 [#751] innerring: Use CalledByEntry scope in main chain invocations
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-08-06 17:29:27 +03:00
Alex Vanin
43dfccd9b3 [#751] morph/client: Add custom signer scope support
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-08-06 17:29:27 +03:00
ZhangTao1596
f121a73049 [#737] network/cache: handle request canceled in multiclient
Signed-off-by: ZhangTao1596 <zhangtao@ngd.neo.org>
2021-08-05 14:47:41 +03:00
Pavel Karpy
aa0955f15d [#747] neofs-adm: Support ~ for paths in config
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-05 14:44:42 +03:00
Pavel Karpy
b5cadff2c3 [#745] node: Add more debug logs on shutdown
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-05 14:44:05 +03:00
Pavel Karpy
3d981320c6 [#745] gc: Stop GC work on Shard's Close
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-05 14:44:05 +03:00
Alex Vanin
6527f9157c [#744] util/attributes: Support escaped symbols
To encode attributes with semicolon or slash, use
backslash as escaped character.

Example:
  User-Agent:NeoFS\/0.23
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-08-04 16:48:54 +03:00
Pavel Karpy
c3e2738a46 [#743] go.mod: Update neo-go library
github.com/nspcc-dev/neo-go v0.96.1 => v0.97.0

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-04 12:43:33 +03:00
Alex Vanin
d8e47e60a7 [#738] neofs-adm: Use constants and reduce code in dump-config
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-08-03 21:03:36 +03:00
Alex Vanin
ddbfb09560 [#738] neofs-adm: Add command to dump NeoFS network config
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-08-03 21:03:36 +03:00
Alex Vanin
37cc702271 [#738] neofs-adm: Set more network configuration values
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-08-03 21:03:36 +03:00
Alex Vanin
3a34e4a48e [#738] Update go.mod
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-08-03 21:03:36 +03:00
Pavel Karpy
7a52bb31f1 [#740] innerring/notary: Fix reading notary config
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-03 21:00:57 +03:00
Pavel Karpy
e8665f6cef [#730] node/morph: Use disable_cache config param
Do not init caches for eACL, containers and netmap
if `disable_cache` config options is `true`, use
direct RPC calls instead.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-03 09:40:10 +03:00
Pavel Karpy
c423aa432a [#730] node/config/morph: Add disable_cache
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-03 09:40:10 +03:00
Pavel Karpy
cb842096d4 [#720] ir: Delete without_notary defaults
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-03 09:30:11 +03:00
Pavel Karpy
896c749b92 [#720] pkg/innerring: Check Notary availability automatically
Do not read `without_notary` config value from env.
Make morph client constructor return client without
notary support. Enabling notary support should be done
with public `EnableNotarySupport` method separately.

Notary availability is deducted with client. Further,
if notary is presented on chain its support is
enabled at the corresponding client.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-03 09:30:11 +03:00
Pavel Karpy
28aa0f521e [#735] pkg/network: Delete tls multiaddr implementation
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-02 14:28:58 +03:00
Pavel Karpy
c2ab0e19ef [#735] go mod: Update multiaddr library
github.com/multiformats/go-multiaddr v0.3.2 => v0.4.0

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-02 14:28:58 +03:00
Evgenii Stratonikov
e2cef00497 [#732] neofs-adm: read contract path only on init
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-08-02 14:28:23 +03:00
Evgenii Stratonikov
a2f6e07b1d [#732] neofs-adm: read alphabet contract once
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-08-02 14:28:23 +03:00
Evgenii Stratonikov
8e71773c4a [#732] neofs-adm: fetch single accounts during the initialization
Simplify code and perform error checking before the actual work.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-08-02 14:28:23 +03:00
Evgenii Stratonikov
018256def8 [#732] neofs-adm: remove debug output
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-08-02 14:28:23 +03:00
Evgenii Stratonikov
c81008764a [#732] neofs-adm: fetch native hashes once
Retrieve list of native contracts during initialization
in a single query.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-08-02 14:28:23 +03:00
Evgenii Stratonikov
6a40adcfca [#732] neofs-adm: get contract path on initialization
Simplifies code a bit.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-08-02 14:28:23 +03:00
Pavel Karpy
67b17cfb02 [#727] config: Use 0660 as default permissions
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-30 16:42:22 +03:00
Pavel Karpy
7a10d902be [#727] Use util.MkdirAllX instead of os.MkdirAll
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-30 16:42:22 +03:00
Pavel Karpy
51eeae3640 [#727] util: Add MkdirAllX func
Add `MkdirAllX` function that wraps `os.MkdirAll`
and adds `x` permission for user and group.
This is used to make all created dirs openable
for user and group regardless of the read config
permissions.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-30 16:42:22 +03:00
Evgenii Stratonikov
c332188341 [#685] neofs-adm: generate storage node wallets
Initial GAS can be provided both in config and as a CLI argument.
Generating wallet with 0 GAS is currently prohibited.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-07-30 14:09:51 +03:00
Evgenii Stratonikov
90259b5cc7 [#685] neofs-adm: reduce amount of logs
If `init` is run on a dirty network (i.e. with some stages already
done), no transactions are really sent so clean up logs a bit.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-07-30 14:09:51 +03:00
Evgenii Stratonikov
b95c16879d [#686] neofs-adm: implement morph force-new-epoch
Allow to force epoch change.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-07-30 14:08:56 +03:00
Evgenii Stratonikov
9c1fb0b55e [#728] neofs-adm: update to neofs-contract@v0.10.1
Add config parameters to netmap and neofs contracts.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-07-30 12:45:21 +03:00
ZhangTao1596
01806db612 [#723] container/announcement/storage: Map announcements by epoch and container id
Signed-off-by: ZhangTao1596 <zhangtao@ngd.neo.org>
2021-07-30 09:58:30 +03:00
Evgenii Stratonikov
2290109849 [#726] neofs-adm: use NNSIsAvailable from neo-go
There is no need in writing another wrapper.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-07-29 10:50:32 +03:00
Evgenii Stratonikov
9e56012760 [#726] neofs-adm: set alphabet contract addresses in NNS
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-07-29 10:50:32 +03:00
Evgenii Stratonikov
4d65e138f5 [#726] neofs-adm: allow to dump deployed contract hashes
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-07-29 10:50:32 +03:00
Evgenii Stratonikov
9ef2579afa [#726] neofs-adm: remove alphabet contract from contract list
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-07-29 10:50:32 +03:00
Alex Vanin
9491ff2571 [#725] config/testnet: Fix ports
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-28 15:17:45 +03:00
Evgenii Stratonikov
cf5f8a8f78 [#684] neofs-adm: transfer gas to the proxy contract
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-07-26 16:04:05 +03:00
Evgenii Stratonikov
c78350846a [#687] neofs-adm: set aliases for contract hashes in NNS
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-07-26 16:04:05 +03:00
Evgenii Stratonikov
df1b26c708 [#687] neofs-adm: register candidates
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-07-26 16:04:05 +03:00
Evgenii Stratonikov
0efc7b7fee [#687] neofs-adm: deploy NeoFS contracts
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-07-26 16:04:05 +03:00
Evgenii Stratonikov
7c3c1e9183 [#687] *: update neo-go version
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-07-26 16:04:05 +03:00
Evgenii Stratonikov
8ea67ec565 [#687] neofs-adm: check for initialization stage completion
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-07-26 16:04:05 +03:00
Evgenii Stratonikov
c3f7ccaee6 [#687] neofs-adm: set alphabet and notary nodes
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-07-26 16:04:05 +03:00
Evgenii Stratonikov
425c1db5c0 [#687] neofs-adm: transfer funds to consensus wallets
This is the first stage requiring running blockchain.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-07-26 16:04:05 +03:00
Alex Vanin
6f07710693 Release v0.23.0 - Wando (완도, 莞島)
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-23 19:41:15 +03:00
Alex Vanin
311eb3e95a [#719] Update contributors list
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-23 19:41:15 +03:00
Alex Vanin
70bb96a3d5 [#719] config/testnet: Update for RC4(Testnet) launch
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-23 19:41:15 +03:00
Alex Vanin
62d3eeea6e [#719] Update neo-go to v0.96.1
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-23 19:41:15 +03:00
Pavel Karpy
6638136d11 [#711] node/config/test: Adapt test to new config examples
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-23 11:09:02 +03:00
Pavel Karpy
da516c6754 [#711] node/config/example: Make config examples usable
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-23 11:09:02 +03:00
Pavel Karpy
c41ec2e2e9 [#718] ir: Do not require processing hash in main notary disabled mode
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-23 10:59:45 +03:00
Pavel Karpy
e753eebaee Update Neo-Go library to v0.96.0
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-23 10:58:26 +03:00
Alex Vanin
bba470570f [#708] morph/client: Update last epoch block method name
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-22 20:40:09 +03:00
Alex Vanin
3561cf5873 [#712] moprh/client: Log VUB in notary invocations
For easier debug when notary transaction didn't execute.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-22 15:46:48 +03:00
Pavel Karpy
9aa6ab4fc9 [#710] cli: Make --address flag optional
If `--address` was not presented use default wallet
address and do not require it.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-22 15:28:26 +03:00
Alex Vanin
62efa3f098 [#708] innerring: Synchronize initial epoch tick
When Inner Ring node starts, it should sync nearest epoch tick
event based on the block of the latest epoch. Otherwise epoch
ticking can be stopped, because ballots or notary transactions
are valid for limited period of time.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-22 15:20:34 +03:00
Alex Vanin
67b3682348 [#708] morph/timer: Add single tick timer
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-22 15:20:34 +03:00
Alex Vanin
7146afcd28 [#708] morph/client: Add epoch block getter from netmap
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-22 15:20:34 +03:00
Alex Vanin
7e43af3de3 [#708] morph/client: Add BlockCount method
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-22 15:20:34 +03:00
Pavel Karpy
dfbde0004e [#705] innerring/governance: Delete useless sort
Do not sort slice that is already sorted.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-22 15:12:06 +03:00
Pavel Karpy
c8d58d56b9 [#705] innerring/governance: Hide fetching ir keys behind interface
Add to governance processor IRFetcher interface.
Implementation must deduct if notary enabled or not.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-22 15:12:06 +03:00
Pavel Karpy
73ba35b379 [#705] pkg/innerring: Fix disable notary mode
Get innerring list and committee in innerring
indexer from interfaces, not from structs.
Passing IR fetcher to indexer constructor depends
on having Notary contract in sidechain(use NeoFS
role or netmap contract).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-22 15:12:06 +03:00
Pavel Karpy
8e66c67a74 [#705] pkg/innerring: Add IR keys fetchers
Add `IrFetcherWithNotary` and `IrFetcherWithoutNotary`
that can fetch IR keys with and without usage of
notary contract.
Both can be hidden behind
`InnerRingKeys() (keys.PublicKeys, error)` interface.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-22 15:12:06 +03:00
Pavel Karpy
8ea5744326 [#705] pkg/morph/netmap: Fix updateInnerRing method
Add initialization of `updateInnerRing` method name
in config on startup.
Rename var `setInnerRing` => `updateInnerRing`.
Rename method `SetInnerRing` => `UpdateInnerRing`

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-22 15:12:06 +03:00
Pavel Karpy
863633e6a5 [#705] cmd/neofs-node/object: Add fetcher without Notary
Depending on having notary contract in sidechain get
IR list either from NeoFSAlphabet role either from
netmap contract.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-22 15:12:06 +03:00
Pavel Karpy
84361beada [#705] pkg/morph/netmap/wrapper: Add GetInnerRingList method
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-22 15:12:06 +03:00
Pavel Karpy
b3c81116b9 [#705] pkg/morph/netmap: Add innerRingList method
Add `innerRingList` method to `netmap` client
wrapper. Method parses current IR nodes list.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-22 15:12:06 +03:00
Pavel Karpy
00b487e3c3 [#705] morph/notary: Add ProbeNotary method to Client
`ProbeNotary` method checks if native `Notary`
contract is presented on chain.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-22 15:12:06 +03:00
Pavel Karpy
46a66a21ec [#705] pkg/util: Fix commentary error
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-22 15:12:06 +03:00
Pavel Karpy
795c49d16b [#706] ir: Do not require proxy hash in notary disabled mode
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-21 15:32:17 +03:00
Evgenii Stratonikov
a2cb9cbc49 [#684] neofs-adm: add labels to multisig accounts
Also check that correct multisig is generated.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-07-21 15:23:16 +03:00
Evgenii Stratonikov
be6b8ca179 [#684] neofs-adm: add size validation to generate-alphabet
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-07-21 15:23:16 +03:00
Evgenii Stratonikov
459fe40758 [#684] neofs-adm: add tests for generate-alphabet
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-07-21 15:23:16 +03:00
Evgenii Stratonikov
5ca5d9ccf9 [#684] neofs-adm: create wallet right after password input
It is pretty annoying to get an error about non-existent directory
after entering 7 passwords.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-07-21 15:23:16 +03:00
Evgenii Stratonikov
4ebc6f796f [#684] neofs-adm: generate consensus wallets
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-07-21 15:23:16 +03:00
Alex Vanin
96da7ceb4f [#697] governance: Make best effort traversing main chain list of keys
We should go through every key in main chain list to merget lists
as fast as possible. Previously we drop main chain traversing as
soon as we have no more new keys to add. Instead we should try
to go for old keys in the list and add it as more as we can.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-15 10:49:21 +03:00
Alex Vanin
0212f33743 Update changelog for v0.22.3 release
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-13 16:30:57 +03:00
Alex Vanin
d94f21d3c4 config/testnet: Update NeoFS Storage node version
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-13 16:30:57 +03:00
Alex Vanin
aed76f6b5a [#694] pkg/network: Use single function to get or create client instance
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-13 15:17:48 +03:00
Alex Vanin
2c02261709 [#694] Update changelog
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-13 15:17:48 +03:00
Alex Vanin
fd24a99533 [#694] cmd/neofs-node: Reuse single instance of client cache in all components
This will reduce amount of open connections up to 3 times.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-13 15:17:48 +03:00
Alex Vanin
38afb82926 [#694] pkg/network: Use client cache in raw client getter
Every raw client getter invocation produced new connection, that lead
to connection leak.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-13 15:17:48 +03:00
Pavel Karpy
53391f057e [#693] node/reputation: Log keys in hex format
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-13 15:12:21 +03:00
Alex Vanin
e2f7b3f1cc [#690] cmd/neofs-node: Fallback to wallet section if node key is not set
Some users want to specify only wallet section in the SN. It is not
possible if `Key` throws panic on empty value. Instead it should
fallback to wallet section. Panic is suitable if node's key is provided
but invalid.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-13 14:49:31 +03:00
Alex Vanin
e0500d3158 [#689] Update unreleased changelog
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-13 13:11:29 +03:00
Alex Vanin
8112f73023 [#689] innerring: Remove debug output
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-13 13:11:29 +03:00
Alex Vanin
6bf01a0a22 [#683] cmd/neofs-adm: Add CLI flags for morph commands
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-13 11:29:39 +03:00
Pavel Karpy
c20eb15bd5 [#688] pkg/ir/governance: Add unsorted test case
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-12 18:46:09 +03:00
Pavel Karpy
bb3f16d487 [#688] pkg/ir/governance: Add public key sorting
Add sorting of passed public keys in `newAlphabetList`
function before Alphabet list merge to make returned
results more expected.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-12 18:46:09 +03:00
Pavel Karpy
361bff216a [#688] pkg/ir/governance: Check err type in tests
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-12 18:46:09 +03:00
Pavel Karpy
e56a9611db [#681] docs: Clarify creating tag in release instruction
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-12 15:35:35 +03:00
Pavel Karpy
9da777ac8c [#681] ir: Do not require MainNet attributes
If `WITHOUT_MAINNET` environmental variable is
`true`:
- Do not read `NeoFS` and `processing`
script-hashes from envs;
- Do not init Governance processor;
- Do not init NeoFS processor.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-12 15:35:35 +03:00
Stanislav Bogatyrev
af412f7874 Move Docker files into a separate directory
Dock docker files in .docker directory

Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2021-07-09 16:47:01 +03:00
Stanislav Bogatyrev
45cd851c46 [#666] Add Docker images for neofs-adm
Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2021-07-09 16:20:16 +03:00
Pavel Karpy
cf8f640726 [#675] cli/container: Support binary eACL format
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-09 09:36:02 +03:00
Alex Vanin
d189d60925 [#666] cmd/neofs-adm: Initial app structure with config init command
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-08 14:37:38 +03:00
Alex Vanin
24d1725cc8 [#666] innerring: Make GlagoliticLetter enum public for re-use
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-08 14:37:38 +03:00
Pavel Karpy
48c3573e0c [#671] config/testnet: Update NeoFS storage version
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-07 16:00:51 +03:00
Pavel Karpy
d34149a67e Update changelog for v0.22.2 release
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-07 16:00:51 +03:00
Pavel Karpy
937210a2d3 Update API Go library to v1.28.3
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-07 16:00:51 +03:00
Pavel Karpy
ffa5de4963 Fix link in changelog for v0.22.1 release
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-07 16:00:51 +03:00
1203 changed files with 89983 additions and 29543 deletions

19
.docker/Dockerfile.adm Normal file
View file

@ -0,0 +1,19 @@
FROM golang:1.18 as builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository
WORKDIR /src
COPY . /src
RUN make bin/frostfs-adm
# Executable image
FROM alpine AS frostfs-adm
RUN apk add --no-cache bash
WORKDIR /
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /src/bin/frostfs-adm /bin/frostfs-adm
CMD ["frostfs-adm"]

19
.docker/Dockerfile.cli Normal file
View file

@ -0,0 +1,19 @@
FROM golang:1.18 as builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository
WORKDIR /src
COPY . /src
RUN make bin/frostfs-cli
# Executable image
FROM alpine AS frostfs-cli
RUN apk add --no-cache bash
WORKDIR /
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /src/bin/frostfs-cli /bin/frostfs-cli
CMD ["frostfs-cli"]

View file

@ -3,6 +3,6 @@ RUN apk add --no-cache bash ca-certificates
WORKDIR / WORKDIR /
COPY bin/neofs-node /bin/neofs-node COPY bin/frostfs-adm /bin/frostfs-adm
CMD ["neofs-node"] CMD ["frostfs-adm"]

View file

@ -3,6 +3,6 @@ RUN apk add --no-cache bash ca-certificates
WORKDIR / WORKDIR /
COPY bin/neofs-cli /bin/neofs-cli COPY bin/frostfs-cli /bin/frostfs-cli
CMD ["neofs-cli"] CMD ["frostfs-cli"]

View file

@ -3,6 +3,6 @@ RUN apk add --no-cache bash ca-certificates
WORKDIR / WORKDIR /
COPY bin/neofs-ir /bin/neofs-ir COPY bin/frostfs-ir /bin/frostfs-ir
CMD ["neofs-ir"] CMD ["frostfs-ir"]

View file

@ -0,0 +1,8 @@
FROM alpine
RUN apk add --no-cache bash ca-certificates
WORKDIR /
COPY bin/frostfs-node /bin/frostfs-node
CMD ["frostfs-node"]

18
.docker/Dockerfile.ir Normal file
View file

@ -0,0 +1,18 @@
FROM golang:1.18 as builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository
WORKDIR /src
COPY . /src
RUN make bin/frostfs-ir
# Executable image
FROM alpine AS frostfs-ir
RUN apk add --no-cache bash
WORKDIR /
COPY --from=builder /src/bin/frostfs-ir /bin/frostfs-ir
CMD ["frostfs-ir"]

View file

@ -0,0 +1,18 @@
FROM golang:1.18 as builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository
WORKDIR /src
COPY . /src
RUN make bin/frostfs-node
# Executable image
FROM alpine AS frostfs-node
RUN apk add --no-cache bash
WORKDIR /
COPY --from=builder /src/bin/frostfs-node /bin/frostfs-node
CMD ["frostfs-node"]

View file

@ -0,0 +1,19 @@
FROM golang:1.18 as builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository
WORKDIR /src
COPY . /src
RUN make bin/frostfs-node
# Executable image
FROM alpine AS frostfs-node
RUN apk add --no-cache bash
WORKDIR /
COPY --from=builder /src/bin/frostfs-node /bin/frostfs-node
COPY --from=builder /src/config/testnet/config.yml /config.yml
CMD ["frostfs-node", "--config", "/config.yml"]

1
.github/CODEOWNERS vendored Normal file
View file

@ -0,0 +1 @@
* @TrueCloudLab/storage-core @TrueCloudLab/committers

View file

@ -7,39 +7,44 @@ assignees: ''
--- ---
<!--- Provide a general summary of the issue in the Title above --> <!-- Provide a general summary of the issue in the Title above -->
## Expected Behavior ## Expected Behavior
<!--- If you're describing a bug, tell us what should happen --> <!-- If you're describing a bug, tell us what should happen
<!--- If you're suggesting a change/improvement, tell us how it should work --> If you're suggesting a change/improvement, tell us how it should work -->
## Current Behavior ## Current Behavior
<!--- If describing a bug, tell us what happens instead of the expected behavior --> <!-- If describing a bug, tell us what happens instead of the expected behavior
<!--- If suggesting a change/improvement, explain the difference from current behavior --> If suggesting a change/improvement, explain the difference from current behavior -->
## Possible Solution ## Possible Solution
<!--- Not obligatory, but suggest a fix/reason for the bug, --> <!-- Not obligatory, but suggest a fix/reason for the bug,
<!--- or ideas how to implement the addition or change --> or ideas how to implement the addition or change -->
## Steps to Reproduce (for bugs) ## Steps to Reproduce (for bugs)
<!--- Provide a link to a live example, or an unambiguous set of steps to --> <!-- Provide a link to a live example, or an unambiguous set of steps
<!--- reproduce this bug. --> to reproduce this bug. -->
1. 1.
2.
3.
4.
## Context ## Context
<!--- How has this issue affected you? What are you trying to accomplish? --> <!-- How has this issue affected you? What are you trying to accomplish?
<!--- Providing context helps us come up with a solution that is most useful in the real world --> Providing context helps us come up with a solution that is most useful in the real world -->
## Regression ## Regression
<!-- Is this issue a regression? (Yes / No) --> <!-- Is this issue a regression? (Yes / No)
<!-- If Yes, optionally please include version or commit id or PR# that caused this regression, if you have these details. --> If Yes, optionally please include version or commit id or PR# that caused this regression,
if you have these details -->
## Your Environment ## Your Environment
<!--- Include as many relevant details about the environment you experienced the bug in --> <!-- Include as many relevant details about the environment you experienced the bug in -->
* Version used: * Version used:
* Server setup and configuration: * Server setup and configuration:
* Operating System and version (`uname -a`): * Operating System and version (`uname -a`):
## Don't forget to add labels!
- component label (`frostfs-adm`, `frostfs-storage`, ...)
- `goodfirstissue`, `helpwanted` if needed
- does this issue belong to an epic?
- priority (`P0`-`P4`) if already triaged
- quarter label (`202XQY`) if possible

View file

@ -7,14 +7,22 @@ assignees: ''
--- ---
**Is your feature request related to a problem? Please describe.** ## Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] <!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when ... -->
**Describe the solution you'd like** ## Describe the solution you'd like
A clear and concise description of what you want to happen. <!-- A clear and concise description of what you want to happen. -->
**Describe alternatives you've considered** ## Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered. <!-- A clear and concise description of any alternative solutions or features you've considered. -->
**Additional context** ## Additional context
Add any other context or screenshots about the feature request here. <!-- Add any other context or screenshots about the feature request here. -->
## Don't forget to add labels!
- component label (`neofs-adm`, `neofs-storage`, ...)
- issue type (`enhancement`, `refactor`, ...)
- `goodfirstissue`, `helpwanted` if needed
- does this issue belong to an epic?
- priority (`P0`-`P4`) if already triaged
- quarter label (`202XQY`) if possible

187
.github/logo.svg vendored
View file

@ -1,129 +1,70 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?> <?xml version="1.0" encoding="utf-8"?>
<svg <!-- Generator: Adobe Illustrator 25.0.1, SVG Export Plug-In . SVG Version: 6.00 Build 0) -->
xmlns:dc="http://purl.org/dc/elements/1.1/" <svg version="1.1" id="Слой_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
xmlns:cc="http://creativecommons.org/ns#" viewBox="0 0 184.2 51.8" style="enable-background:new 0 0 184.2 51.8;" xml:space="preserve">
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" <style type="text/css">
xmlns:svg="http://www.w3.org/2000/svg" .st0{display:none;}
xmlns="http://www.w3.org/2000/svg" .st1{display:inline;}
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" .st2{fill:#01E397;}
xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" .st3{display:inline;fill:#010032;}
sodipodi:docname="logo_fs.svg" .st4{display:inline;fill:#00E599;}
inkscape:version="1.0 (4035a4fb49, 2020-05-01)" .st5{display:inline;fill:#00AF92;}
id="svg57" .st6{fill:#00C3E5;}
version="1.1" </style>
viewBox="0 0 105 25" <g id="Layer_2">
height="25mm" <g id="Layer_1-2" class="st0">
width="105mm"> <g class="st1">
<defs <path class="st2" d="M146.6,18.3v7.2h10.9V29h-10.9v10.7h-4V14.8h18v3.5H146.6z"/>
id="defs51"> <path class="st2" d="M180,15.7c1.7,0.9,3,2.2,4,3.8l-3,2.7c-0.6-1.3-1.5-2.4-2.6-3.3c-1.3-0.7-2.8-1-4.3-1
<clipPath c-1.4-0.1-2.8,0.3-4,1.1c-0.9,0.5-1.5,1.5-1.4,2.6c0,1,0.5,1.9,1.4,2.4c1.5,0.8,3.2,1.3,4.9,1.5c1.9,0.3,3.7,0.8,5.4,1.6
clipPathUnits="userSpaceOnUse" c1.2,0.5,2.2,1.3,2.9,2.3c0.6,1,1,2.2,0.9,3.4c0,1.4-0.5,2.7-1.3,3.8c-0.9,1.2-2.1,2.1-3.5,2.6c-1.7,0.6-3.4,0.9-5.2,0.8
id="clipPath434"> c-5,0-8.6-1.6-10.7-5l2.9-2.8c0.7,1.4,1.8,2.5,3.1,3.3c1.5,0.7,3.1,1.1,4.7,1c1.5,0.1,2.9-0.2,4.2-0.9c0.9-0.5,1.5-1.5,1.5-2.6
<path c0-0.9-0.5-1.8-1.3-2.2c-1.5-0.7-3.1-1.2-4.8-1.5c-1.9-0.3-3.7-0.8-5.5-1.5c-1.2-0.5-2.2-1.4-3-2.4c-0.6-1-1-2.2-0.9-3.4
d="M 0,0 H 1366 V 768 H 0 Z" c0-1.4,0.4-2.7,1.2-3.8c0.8-1.2,2-2.2,3.3-2.8c1.6-0.7,3.4-1.1,5.2-1C176.1,14.3,178.2,14.8,180,15.7z"/>
id="path432" />
</clipPath>
</defs>
<sodipodi:namedview
inkscape:window-maximized="0"
inkscape:window-y="0"
inkscape:window-x="130"
inkscape:window-height="1040"
inkscape:window-width="1274"
height="50mm"
units="mm"
showgrid="false"
inkscape:document-rotation="0"
inkscape:current-layer="layer1"
inkscape:document-units="mm"
inkscape:cy="344.49897"
inkscape:cx="468.64708"
inkscape:zoom="0.7"
inkscape:pageshadow="2"
inkscape:pageopacity="0.0"
borderopacity="1.0"
bordercolor="#666666"
pagecolor="#ffffff"
id="base" />
<metadata
id="metadata54">
<rdf:RDF>
<cc:Work
rdf:about="">
<dc:format>image/svg+xml</dc:format>
<dc:type
rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
<dc:title></dc:title>
</cc:Work>
</rdf:RDF>
</metadata>
<g
id="layer1"
inkscape:groupmode="layer"
inkscape:label="Layer 1">
<g
id="g424"
transform="matrix(0.35277777,0,0,-0.35277777,63.946468,10.194047)">
<path
d="m 0,0 v -8.093 h 12.287 v -3.94 H 0 V -24.067 H -4.534 V 3.898 H 15.677 V 0 Z"
style="fill:#00e396;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path426" />
</g> </g>
<g <path class="st3" d="M73.3,16.3c1.9,1.9,2.9,4.5,2.7,7.1v15.9h-4V24.8c0-2.6-0.5-4.5-1.6-5.7c-1.2-1.2-2.8-1.8-4.5-1.7
transform="matrix(0.35277777,0,0,-0.35277777,-315.43002,107.34005)" c-1.3,0-2.5,0.3-3.7,0.8c-1.2,0.7-2.2,1.7-2.9,2.9c-0.8,1.5-1.1,3.2-1.1,4.9v13.3h-4V15.1l3.6,1.5v1.7c0.8-1.5,2.1-2.6,3.6-3.3
id="g428"> c1.5-0.8,3.2-1.2,4.9-1.1C68.9,13.8,71.3,14.7,73.3,16.3z"/>
<g <path class="st3" d="M104.4,28.3H85.6c0.1,2.2,1,4.3,2.5,5.9c1.5,1.4,3.5,2.2,5.6,2.1c1.6,0.1,3.2-0.2,4.6-0.9
id="g430" c1.1-0.6,2-1.6,2.5-2.8l3.3,1.8c-0.9,1.7-2.3,3.1-4,4c-2,1-4.2,1.5-6.4,1.4c-3.7,0-6.7-1.1-8.8-3.4s-3.2-5.5-3.2-9.6s1-7.2,3-9.5
clip-path="url(#clipPath434)"> s5-3.4,8.7-3.4c2.1-0.1,4.2,0.5,6.1,1.5c1.6,1,3,2.5,3.8,4.2c0.9,1.8,1.3,3.9,1.3,5.9C104.6,26.4,104.6,27.4,104.4,28.3z
<g M88.1,19.3c-1.4,1.5-2.2,3.4-2.4,5.5h15.1c-0.2-2-1-3.9-2.3-5.5c-1.4-1.3-3.2-2-5.1-1.9C91.5,17.3,89.6,18,88.1,19.3z"/>
id="g436" <path class="st3" d="M131,17.3c2.2,2.3,3.2,5.5,3.2,9.5s-1,7.3-3.2,9.6s-5.1,3.4-8.8,3.4s-6.7-1.1-8.9-3.4s-3.2-5.5-3.2-9.6
transform="translate(1112.874,278.2981)"> s1.1-7.2,3.2-9.5s5.1-3.4,8.9-3.4S128.9,15,131,17.3z M116.2,19.9c-1.5,2-2.2,4.4-2.1,6.9c-0.2,2.5,0.6,5,2.1,7
<path c1.5,1.7,3.7,2.7,6,2.6c2.3,0.1,4.4-0.9,5.9-2.6c1.5-2,2.3-4.5,2.1-7c0.1-2.5-0.6-4.9-2.1-6.9c-1.5-1.7-3.6-2.7-5.9-2.6
d="M 0,0 C 1.822,-0.932 3.354,-2.359 4.597,-4.28 L 1.165,-7.373 c -0.791,1.695 -1.779,2.924 -2.966,3.686 -1.186,0.763 -2.768,1.145 -4.745,1.145 -1.949,0 -3.461,-0.389 -4.534,-1.166 -1.074,-0.777 -1.61,-1.772 -1.61,-2.987 0,-1.13 0.523,-2.027 1.568,-2.69 1.045,-0.664 2.909,-1.236 5.593,-1.716 2.514,-0.452 4.512,-1.024 5.995,-1.716 1.483,-0.693 2.564,-1.554 3.242,-2.585 0.677,-1.031 1.016,-2.309 1.016,-3.834 0,-1.639 -0.466,-3.079 -1.398,-4.322 -0.932,-1.243 -2.239,-2.197 -3.919,-2.86 -1.681,-0.664 -3.623,-0.996 -5.826,-0.996 -5.678,0 -9.689,1.892 -12.033,5.678 l 3.178,3.178 c 0.903,-1.695 2.068,-2.939 3.495,-3.729 1.426,-0.791 3.199,-1.186 5.318,-1.186 2.005,0 3.58,0.345 4.724,1.038 1.144,0.692 1.716,1.674 1.716,2.945 0,1.017 -0.516,1.835 -1.547,2.457 -1.031,0.621 -2.832,1.172 -5.402,1.653 -2.571,0.479 -4.618,1.073 -6.143,1.779 -1.526,0.706 -2.635,1.582 -3.326,2.627 -0.693,1.045 -1.039,2.316 -1.039,3.813 0,1.582 0.438,3.023 1.314,4.322 0.875,1.299 2.14,2.33 3.792,3.093 1.653,0.763 3.58,1.144 5.783,1.144 C -4.018,1.398 -1.822,0.932 0,0" C119.9,17.2,117.7,18.2,116.2,19.9z"/>
style="fill:#00e396;fill-opacity:1;fill-rule:nonzero;stroke:none" <polygon class="st4" points="0,9.1 0,43.7 22.5,51.8 22.5,16.9 46.8,7.9 24.8,0 "/>
id="path438" /> <polygon class="st5" points="24.3,17.9 24.3,36.8 46.8,44.9 46.8,9.6 "/>
</g> </g>
<g <g>
id="g440" <g>
transform="translate(993.0239,277.5454)"> <path class="st6" d="M41.6,17.5H28.2v6.9h10.4v3.3H28.2v10.2h-3.9V14.2h17.2V17.5z"/>
<path <path class="st6" d="M45.8,37.9v-18h3.3l0.4,3.2c0.5-1.2,1.2-2.1,2.1-2.7c0.9-0.6,2.1-0.9,3.5-0.9c0.4,0,0.7,0,1.1,0.1
d="m 0,0 c 2.054,-1.831 3.083,-4.465 3.083,-7.902 v -17.935 h -4.484 v 16.366 c 0,2.914 -0.626,5.024 -1.877,6.332 -1.253,1.308 -2.924,1.962 -5.016,1.962 -1.495,0 -2.896,-0.327 -4.204,-0.981 -1.308,-0.654 -2.381,-1.719 -3.222,-3.194 -0.841,-1.477 -1.261,-3.335 -1.261,-5.576 v -14.909 h -4.484 V 1.328 l 4.086,-1.674 0.118,-1.84 c 0.933,1.681 2.222,2.923 3.867,3.727 1.643,0.803 3.493,1.205 5.548,1.205 C -4.671,2.746 -2.055,1.83 0,0" c0.4,0.1,0.7,0.2,0.9,0.3l-0.5,3.4c-0.3-0.1-0.6-0.2-0.9-0.2C55.4,23,54.9,23,54.4,23c-0.7,0-1.5,0.2-2.2,0.6
style="fill:#000033;fill-opacity:1;fill-rule:nonzero;stroke:none" c-0.7,0.4-1.3,1-1.8,1.8s-0.7,1.8-0.7,3v9.5H45.8z"/>
id="path442" /> <path class="st6" d="M68.6,19.6c1.8,0,3.3,0.4,4.6,1.1c1.3,0.7,2.4,1.8,3.1,3.2s1.1,3.1,1.1,5c0,1.9-0.4,3.6-1.1,5
c-0.8,1.4-1.8,2.5-3.1,3.2c-1.3,0.7-2.9,1.1-4.6,1.1s-3.3-0.4-4.6-1.1c-1.3-0.7-2.4-1.8-3.2-3.2c-0.8-1.4-1.2-3.1-1.2-5
c0-1.9,0.4-3.6,1.2-5s1.8-2.5,3.2-3.2C65.3,19.9,66.8,19.6,68.6,19.6z M68.6,22.6c-1.1,0-2,0.2-2.8,0.7c-0.8,0.5-1.3,1.2-1.7,2.1
s-0.6,2.1-0.6,3.5c0,1.3,0.2,2.5,0.6,3.4s1,1.7,1.7,2.2s1.7,0.7,2.8,0.7c1.1,0,2-0.2,2.7-0.7c0.7-0.5,1.3-1.2,1.7-2.2
s0.6-2.1,0.6-3.4c0-1.4-0.2-2.5-0.6-3.5s-1-1.6-1.7-2.1C70.6,22.8,69.6,22.6,68.6,22.6z"/>
<path class="st6" d="M89.2,38.3c-1.8,0-3.4-0.3-4.9-1c-1.5-0.7-2.7-1.7-3.5-3l2.7-2.3c0.5,1,1.3,1.8,2.3,2.4
c1,0.6,2.2,0.9,3.6,0.9c1.1,0,2-0.2,2.6-0.6c0.6-0.4,1-0.9,1-1.6c0-0.5-0.2-0.9-0.5-1.2s-0.9-0.6-1.7-0.8l-3.8-0.8
c-1.9-0.4-3.3-1-4.1-1.9c-0.8-0.9-1.2-1.9-1.2-3.3c0-1,0.3-1.9,0.9-2.7c0.6-0.8,1.4-1.5,2.5-2s2.5-0.8,4-0.8c1.8,0,3.3,0.3,4.6,1
c1.3,0.6,2.2,1.5,2.9,2.7l-2.7,2.2c-0.5-1-1.1-1.7-2-2.1c-0.9-0.5-1.8-0.7-2.8-0.7c-0.8,0-1.4,0.1-2,0.3c-0.6,0.2-1,0.5-1.3,0.8
c-0.3,0.3-0.4,0.7-0.4,1.2c0,0.5,0.2,0.9,0.5,1.3s1,0.6,1.9,0.8l4.1,0.9c1.7,0.3,2.9,0.9,3.7,1.7c0.7,0.8,1.1,1.8,1.1,2.9
c0,1.2-0.3,2.2-0.9,3c-0.6,0.9-1.5,1.6-2.6,2C92.1,38.1,90.7,38.3,89.2,38.3z"/>
<path class="st6" d="M112.8,19.9v3H99.3v-3H112.8z M106.6,14.6v17.9c0,0.9,0.2,1.5,0.7,1.9c0.5,0.4,1.1,0.6,1.9,0.6
c0.6,0,1.2-0.1,1.7-0.3c0.5-0.2,0.9-0.5,1.3-0.8l0.9,2.8c-0.6,0.5-1.2,0.9-2,1.1c-0.8,0.3-1.7,0.4-2.7,0.4c-1,0-2-0.2-2.8-0.5
s-1.5-0.9-2-1.6c-0.5-0.8-0.7-1.7-0.8-3V15.7L106.6,14.6z"/>
<path d="M137.9,17.5h-13.3v6.9h10.4v3.3h-10.4v10.2h-3.9V14.2h17.2V17.5z"/>
<path d="M150.9,13.8c2.1,0,4,0.4,5.5,1.2c1.6,0.8,2.9,2,4,3.5l-2.6,2.5c-0.9-1.4-1.9-2.4-3.1-3c-1.1-0.6-2.5-0.9-4-0.9
c-1.2,0-2.1,0.2-2.8,0.5c-0.7,0.3-1.3,0.7-1.6,1.2c-0.3,0.5-0.5,1.1-0.5,1.7c0,0.7,0.3,1.4,0.8,1.9c0.5,0.6,1.5,1,2.9,1.3
l4.8,1.1c2.3,0.5,3.9,1.3,4.9,2.3c1,1,1.4,2.3,1.4,3.9c0,1.5-0.4,2.7-1.2,3.8c-0.8,1.1-1.9,1.9-3.3,2.5s-3.1,0.9-5,0.9
c-1.7,0-3.2-0.2-4.5-0.6c-1.3-0.4-2.5-1-3.5-1.8c-1-0.7-1.8-1.6-2.5-2.6l2.7-2.7c0.5,0.8,1.1,1.6,1.9,2.2
c0.8,0.7,1.7,1.2,2.7,1.5c1,0.4,2.2,0.5,3.4,0.5c1.1,0,2.1-0.1,2.9-0.4c0.8-0.3,1.4-0.7,1.8-1.2c0.4-0.5,0.6-1.1,0.6-1.9
c0-0.7-0.2-1.3-0.7-1.8c-0.5-0.5-1.3-0.9-2.6-1.2l-5.2-1.2c-1.4-0.3-2.6-0.8-3.6-1.3c-0.9-0.6-1.6-1.3-2.1-2.1s-0.7-1.8-0.7-2.8
c0-1.3,0.4-2.6,1.1-3.7c0.7-1.1,1.8-2,3.2-2.6C147.3,14.1,148.9,13.8,150.9,13.8z"/>
</g> </g>
<g
id="g444"
transform="translate(1027.9968,264.0386)">
<path
d="m 0,0 h -21.128 c 0.261,-2.84 1.205,-5.044 2.83,-6.613 1.625,-1.57 3.727,-2.355 6.305,-2.355 2.054,0 3.763,0.356 5.128,1.065 1.363,0.71 2.288,1.738 2.774,3.083 l 3.755,-1.961 c -1.121,-1.981 -2.616,-3.495 -4.484,-4.54 -1.868,-1.046 -4.259,-1.569 -7.173,-1.569 -4.223,0 -7.538,1.289 -9.948,3.867 -2.41,2.578 -3.615,6.146 -3.615,10.704 0,4.558 1.149,8.127 3.447,10.705 2.298,2.578 5.557,3.867 9.779,3.867 2.615,0 4.876,-0.58 6.782,-1.738 1.905,-1.158 3.343,-2.728 4.315,-4.707 C -0.262,7.827 0.224,5.605 0.224,3.139 0.224,2.092 0.149,1.046 0,0 m -18.298,10.144 c -1.513,-1.457 -2.438,-3.512 -2.775,-6.165 h 16.982 c -0.3,2.615 -1.159,4.661 -2.578,6.137 -1.42,1.476 -3.307,2.214 -5.661,2.214 -2.466,0 -4.455,-0.728 -5.968,-2.186"
style="fill:#000033;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path446" />
</g>
<g
id="g448"
transform="translate(1057.8818,276.4246)">
<path
d="m 0,0 c 2.41,-2.578 3.615,-6.147 3.615,-10.705 0,-4.558 -1.205,-8.126 -3.615,-10.704 -2.41,-2.578 -5.726,-3.867 -9.948,-3.867 -4.222,0 -7.537,1.289 -9.947,3.867 -2.41,2.578 -3.615,6.146 -3.615,10.704 0,4.558 1.205,8.127 3.615,10.705 2.41,2.578 5.725,3.867 9.947,3.867 C -5.726,3.867 -2.41,2.578 0,0 m -16.617,-2.858 c -1.607,-1.906 -2.41,-4.522 -2.41,-7.847 0,-3.326 0.803,-5.94 2.41,-7.846 1.607,-1.905 3.83,-2.858 6.669,-2.858 2.839,0 5.063,0.953 6.67,2.858 1.606,1.906 2.41,4.52 2.41,7.846 0,3.325 -0.804,5.941 -2.41,7.847 C -4.885,-0.953 -7.109,0 -9.948,0 c -2.839,0 -5.062,-0.953 -6.669,-2.858"
style="fill:#000033;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path450" />
</g>
</g>
</g>
<g
id="g452"
transform="matrix(0.35277777,0,0,-0.35277777,5.8329581,6.5590171)">
<path
d="m 0,0 0.001,-38.946 25.286,-9.076 V -8.753 L 52.626,1.321 27.815,10.207 Z"
style="fill:#00e599;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path454" />
</g>
<g
id="g456"
transform="matrix(0.35277777,0,0,-0.35277777,15.479008,10.041927)">
<path
d="M 0,0 V -21.306 L 25.293,-30.364 25.282,9.347 Z"
style="fill:#00b091;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path458" />
</g> </g>
</g> </g>
</svg> </svg>

Before

Width:  |  Height:  |  Size: 6.5 KiB

After

Width:  |  Height:  |  Size: 5.5 KiB

29
.github/workflows/changelog.yml vendored Normal file
View file

@ -0,0 +1,29 @@
name: CHANGELOG check
on:
pull_request:
branches:
- master
- support/**
jobs:
build:
runs-on: ubuntu-latest
name: Check for updates
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Get changed CHANGELOG
id: changelog-diff
uses: tj-actions/changed-files@v29
with:
files: CHANGELOG.md
- name: Fail if changelog not updated
if: steps.changelog-diff.outputs.any_changed == 'false'
uses: actions/github-script@v3
with:
script: |
core.setFailed('CHANGELOG.md has not been updated')

37
.github/workflows/config-update.yml vendored Normal file
View file

@ -0,0 +1,37 @@
name: Configuration check
on:
pull_request:
branches:
- master
- support/**
jobs:
build:
runs-on: ubuntu-latest
name: config-check
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Get changed config-related files
id: config-diff
uses: tj-actions/changed-files@v29
with:
files: |
config/**
cmd/neofs-node/config/**
- name: Get changed doc files
id: docs-diff
uses: tj-actions/changed-files@v29
with:
files: docs/**
- name: Fail if config files are changed but the documentation is not updated
if: steps.config-diff.outputs.any_changed == 'true' && steps.docs-diff.outputs.any_changed == 'false'
uses: actions/github-script@v3
with:
script: |
core.setFailed('Documentation has not been updated')

View file

@ -4,6 +4,7 @@ on:
pull_request: pull_request:
branches: branches:
- master - master
- support/**
jobs: jobs:
commits_check_job: commits_check_job:

View file

@ -1,14 +1,16 @@
name: neofs-node tests name: frostfs-node tests
on: on:
push: push:
branches: branches:
- master - master
- support/**
paths-ignore: paths-ignore:
- '*.md' - '*.md'
pull_request: pull_request:
branches: branches:
- master - master
- support/**
paths-ignore: paths-ignore:
- '*.md' - '*.md'
@ -17,18 +19,18 @@ jobs:
runs-on: ubuntu-20.04 runs-on: ubuntu-20.04
strategy: strategy:
matrix: matrix:
go: [ '1.16.x' ] go: [ '1.18.x', '1.19.x' ]
steps: steps:
- name: Setup go - name: Setup go
uses: actions/setup-go@v2 uses: actions/setup-go@v3
with: with:
go-version: ${{ matrix.go }} go-version: ${{ matrix.go }}
- name: Check out code - name: Check out code
uses: actions/checkout@v2 uses: actions/checkout@v3
- name: Cache go mod - name: Cache go mod
uses: actions/cache@v2 uses: actions/cache@v3
with: with:
path: ~/go/pkg/mod path: ~/go/pkg/mod
key: ${{ runner.os }}-go-${{ matrix.go }}-${{ hashFiles('**/go.sum') }} key: ${{ runner.os }}-go-${{ matrix.go }}-${{ hashFiles('**/go.sum') }}
@ -46,12 +48,13 @@ jobs:
lint: lint:
runs-on: ubuntu-20.04 runs-on: ubuntu-20.04
steps: steps:
- name: Check out code - uses: actions/setup-go@v3
uses: actions/checkout@v2
- name: golangci-lint
uses: golangci/golangci-lint-action@v2
with: with:
version: v1.30 go-version: 1.19
- uses: actions/checkout@v3
- name: golangci-lint
uses: golangci/golangci-lint-action@v3
with:
version: v1.50.0
args: --timeout=5m args: --timeout=5m
only-new-issues: true only-new-issues: true

48
.gitignore vendored
View file

@ -1,11 +1,51 @@
# IDE
.idea .idea
bin .vscode
# Vendoring
vendor
# tempfiles
.DS_Store
*~
.cache
temp temp
tmp
# binary
bin/
release/
# coverage
coverage.txt
coverage.html
# testing
cmd/test cmd/test
/plugins/ /plugins/
/vendor/
testfile testfile
# misc
.neofs-cli.yml .neofs-cli.yml
.cache # debhelpers
debian/*debhelper*
# logfiles
debian/*.log
# .substvars
debian/*.substvars
# .bash-completion
debian/*.bash-completion
# Install folders and files
debian/frostfs-cli/
debian/frostfs-ir/
debian/files
debian/frostfs-storage/
debian/changelog
man/
debs/

View file

@ -29,26 +29,28 @@ linters:
enable: enable:
# mandatory linters # mandatory linters
- govet - govet
- golint - revive
# some default golangci-lint linters # some default golangci-lint linters
- errcheck - errcheck
- gosimple - gosimple
- godot
- ineffassign - ineffassign
- staticcheck - staticcheck
- typecheck - typecheck
- unused
# extra linters # extra linters
- bidichk
- durationcheck
- exhaustive - exhaustive
- exportloopref
- gofmt - gofmt
- whitespace
- goimports - goimports
- misspell
- predeclared
- reassign
- whitespace
disable-all: true disable-all: true
fast: false fast: false
issues:
# Excluding configuration per-path, per-linter, per-text and per-source
exclude-rules:
- path: policy/grammar\.go # ignore structtag issues there
linters:
- govet

File diff suppressed because it is too large Load diff

View file

@ -3,8 +3,8 @@
First, thank you for contributing! We love and encourage pull requests from First, thank you for contributing! We love and encourage pull requests from
everyone. Please follow the guidelines: everyone. Please follow the guidelines:
- Check the open [issues](https://github.com/nspcc-dev/neofs-node/issues) and - Check the open [issues](https://github.com/TrueCloudLab/frostfs-node/issues) and
[pull requests](https://github.com/nspcc-dev/neofs-node/pulls) for existing [pull requests](https://github.com/TrueCloudLab/frostfs-node/pulls) for existing
discussions. discussions.
- Open an issue first, to discuss a new feature or enhancement. - Open an issue first, to discuss a new feature or enhancement.
@ -23,23 +23,23 @@ everyone. Please follow the guidelines:
## Development Workflow ## Development Workflow
Start by forking the `neofs-node` repository, make changes in a branch and then Start by forking the `frostfs-node` repository, make changes in a branch and then
send a pull request. We encourage pull requests to discuss code changes. Here send a pull request. We encourage pull requests to discuss code changes. Here
are the steps in details: are the steps in details:
### Set up your GitHub Repository ### Set up your GitHub Repository
Fork [NeoFS node upstream](https://github.com/nspcc-dev/neofs-node/fork) source Fork [FrostFS node upstream](https://github.com/TrueCloudLab/frostfs-node/fork) source
repository to your own personal repository. Copy the URL of your fork (you will repository to your own personal repository. Copy the URL of your fork (you will
need it for the `git clone` command below). need it for the `git clone` command below).
```sh ```sh
$ git clone https://github.com/nspcc-dev/neofs-node $ git clone https://github.com/TrueCloudLab/frostfs-node
``` ```
### Set up git remote as ``upstream`` ### Set up git remote as ``upstream``
```sh ```sh
$ cd neofs-node $ cd frostfs-node
$ git remote add upstream https://github.com/nspcc-dev/neofs-node $ git remote add upstream https://github.com/TrueCloudLab/frostfs-node
$ git fetch upstream $ git fetch upstream
$ git merge upstream/master $ git merge upstream/master
... ...
@ -79,7 +79,7 @@ Description
``` ```
``` ```
$ git commit -am '[#123] Add some feature' $ git commit -sam '[#123] Add some feature'
``` ```
### Push to the branch ### Push to the branch
@ -106,7 +106,8 @@ contributors".
To sign your work, just add a line like this at the end of your commit message: To sign your work, just add a line like this at the end of your commit message:
``` ```
Signed-off-by: Samii Sakisaka <samii@nspcc.ru> Signed-off-by: Samii Sakisaka <samii@ivunojikan.co.jp>
``` ```
This can easily be done with the `--signoff` option to `git commit`. This can easily be done with the `--signoff` option to `git commit`.

View file

@ -1,5 +1,7 @@
# Credits # Credits
FrostFS continues the development of NeoFS.
Initial NeoFS research and development (2018-2020) was done by Initial NeoFS research and development (2018-2020) was done by
[NeoSPCC](https://nspcc.ru) team. [NeoSPCC](https://nspcc.ru) team.
@ -18,6 +20,10 @@ In alphabetical order:
In chronological order: In chronological order:
- Pavel Karpy - Pavel Karpy
- Zhang Tao
- Angira Kekteeva
- Sergio Nemirowski
- Tivizi Jing
# Special Thanks # Special Thanks

View file

@ -1,24 +0,0 @@
ARG HUB_IMAGE=nspccdev/neofs
ARG HUB_TAG=latest
FROM ${HUB_IMAGE}-cli:${HUB_TAG} as neofs-cli
FROM ${HUB_IMAGE}-ir:${HUB_TAG} as neofs-ir
FROM ${HUB_IMAGE}-storage:${HUB_TAG} as neofs-storage
# Executable image
FROM alpine AS neofs-aio
RUN apk add --no-cache \
bash \
ca-certificates \
jq \
expect \
iputils
WORKDIR /
COPY --from=neofs-cli /bin/neofs-cli /bin/neofs-cli
COPY --from=neofs-ir /bin/neofs-ir /bin/neofs-ir
COPY --from=neofs-storage /bin/neofs-node /bin/neofs-node
CMD ["neofs-cli"]

View file

@ -1,22 +0,0 @@
FROM golang:1.16-alpine as basebuilder
RUN apk add --update make bash
FROM basebuilder as builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository
WORKDIR /src
COPY . /src
RUN make bin/neofs-cli
# Executable image
FROM alpine AS neofs-cli
RUN apk add --no-cache bash
WORKDIR /
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /src/bin/neofs-cli /bin/neofs-cli
CMD ["neofs-cli"]

View file

@ -1,21 +0,0 @@
FROM golang:1.16-alpine as basebuilder
RUN apk add --update make bash
FROM basebuilder as builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository
WORKDIR /src
COPY . /src
RUN make bin/neofs-ir
# Executable image
FROM alpine AS neofs-ir
RUN apk add --no-cache bash
WORKDIR /
COPY --from=builder /src/bin/neofs-ir /bin/neofs-ir
CMD ["neofs-ir"]

View file

@ -1,21 +0,0 @@
FROM golang:1.16-alpine as basebuilder
RUN apk add --update make bash
FROM basebuilder as builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository
WORKDIR /src
COPY . /src
RUN make bin/neofs-node
# Executable image
FROM alpine AS neofs-node
RUN apk add --no-cache bash
WORKDIR /
COPY --from=builder /src/bin/neofs-node /bin/neofs-node
CMD ["neofs-node"]

View file

@ -1,22 +0,0 @@
FROM golang:1.16-alpine as basebuilder
RUN apk add --update make bash
FROM basebuilder as builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository
WORKDIR /src
COPY . /src
RUN make bin/neofs-node
# Executable image
FROM alpine AS neofs-node
RUN apk add --no-cache bash
WORKDIR /
COPY --from=builder /src/bin/neofs-node /bin/neofs-node
COPY --from=builder /src/config/testnet/config.yml /config.yml
CMD ["neofs-node", "--config", "/config.yml"]

120
Makefile
View file

@ -2,91 +2,113 @@
SHELL = bash SHELL = bash
REPO ?= $(shell go list -m) REPO ?= $(shell go list -m)
VERSION ?= $(shell git describe --tags --dirty --always) VERSION ?= $(shell git describe --tags --dirty --match "v*" --always --abbrev=8 2>/dev/null || cat VERSION 2>/dev/null || echo "develop")
BUILD ?= $(shell date -u --iso=seconds)
DEBUG ?= false
HUB_IMAGE ?= nspccdev/neofs HUB_IMAGE ?= truecloudlab/frostfs
HUB_TAG ?= "$(shell echo ${VERSION} | sed 's/^v//')" HUB_TAG ?= "$(shell echo ${VERSION} | sed 's/^v//')"
GO_VERSION ?= 1.19
LINT_VERSION ?= 1.50.0
ARCH = amd64
BIN = bin BIN = bin
DIRS= $(BIN) RELEASE = release
DIRS = $(BIN) $(RELEASE)
# List of binaries to build. # List of binaries to build.
CMDS = $(notdir $(basename $(wildcard cmd/*))) CMDS = $(notdir $(basename $(wildcard cmd/frostfs-*)))
BINS = $(addprefix $(BIN)/, $(CMDS)) BINS = $(addprefix $(BIN)/, $(CMDS))
.PHONY: help all dep clean fmts fmt imports test lint docker/lint # .deb package versioning
OS_RELEASE = $(shell lsb_release -cs)
PKG_VERSION ?= $(shell echo $(VERSION) | sed "s/^v//" | \
sed -E "s/(.*)-(g[a-fA-F0-9]{6,8})(.*)/\1\3~\2/" | \
sed "s/-/~/")-${OS_RELEASE}
.PHONY: help all images dep clean fmts fmt imports test lint docker/lint
prepare-release debpackage
# To build a specific binary, use it's name prefix with bin/ as a target # To build a specific binary, use it's name prefix with bin/ as a target
# For example `make bin/neofs-node` will build only storage node binary # For example `make bin/frostfs-node` will build only storage node binary
# Just `make` will build all possible binaries # Just `make` will build all possible binaries
all: $(DIRS) $(BINS) all: $(DIRS) $(BINS)
# help target
include help.mk
$(BINS): $(DIRS) dep $(BINS): $(DIRS) dep
@echo "⇒ Build $@" @echo "⇒ Build $@"
CGO_ENABLED=0 \ CGO_ENABLED=0 \
GO111MODULE=on \
go build -v -trimpath \ go build -v -trimpath \
-ldflags "-X $(REPO)/misc.Version=$(VERSION) \ -ldflags "-X $(REPO)/misc.Version=$(VERSION)" \
-X $(REPO)/misc.Build=$(BUILD) \
-X $(REPO)/misc.Debug=$(DEBUG)" \
-o $@ ./cmd/$(notdir $@) -o $@ ./cmd/$(notdir $@)
$(DIRS): $(DIRS):
@echo "⇒ Ensure dir: $@" @echo "⇒ Ensure dir: $@"
@mkdir -p $@ @mkdir -p $@
# Prepare binaries and archives for release
.ONESHELL:
prepare-release: docker/all
@for file in `ls -1 $(BIN)/frostfs-*`; do
cp $$file $(RELEASE)/`basename $$file`-$(ARCH)
strip $(RELEASE)/`basename $$file`-$(ARCH)
tar -czf $(RELEASE)/`basename $$file`-$(ARCH).tar.gz $(RELEASE)/`basename $$file`-$(ARCH)
done
# Pull go dependencies # Pull go dependencies
dep: dep:
@printf "⇒ Download requirements: " @printf "⇒ Download requirements: "
CGO_ENABLED=0 \ CGO_ENABLED=0 \
GO111MODULE=on \
go mod download && echo OK go mod download && echo OK
@printf "⇒ Tidy requirements : " @printf "⇒ Tidy requirements : "
CGO_ENABLED=0 \ CGO_ENABLED=0 \
GO111MODULE=on \
go mod tidy -v && echo OK go mod tidy -v && echo OK
# Regenerate proto files: # Regenerate proto files:
protoc: protoc:
@GOPRIVATE=github.com/nspcc-dev go mod vendor @GOPRIVATE=github.com/TrueCloudLab go mod vendor
# Install specific version for protobuf lib # Install specific version for protobuf lib
@go list -f '{{.Path}}/...@{{.Version}}' -m github.com/golang/protobuf | xargs go get -v @go list -f '{{.Path}}/...@{{.Version}}' -m github.com/golang/protobuf | xargs go install -v
@GOBIN=$(abspath $(BIN)) go install -mod=mod -v github.com/TrueCloudLab/frostfs-api-go/v2/util/protogen
# Protoc generate # Protoc generate
@for f in `find . -type f -name '*.proto' -not -path './vendor/*'`; do \ @for f in `find . -type f -name '*.proto' -not -path './vendor/*'`; do \
echo "⇒ Processing $$f "; \ echo "⇒ Processing $$f "; \
protoc \ protoc \
--proto_path=.:./vendor:/usr/local/include \ --proto_path=.:./vendor:/usr/local/include \
--go_out=plugins=grpc,paths=source_relative:. $$f; \ --plugin=protoc-gen-go-frostfs=$(BIN)/protogen \
--go-frostfs_out=. --go-frostfs_opt=paths=source_relative \
--go_out=. --go_opt=paths=source_relative \
--go-grpc_opt=require_unimplemented_servers=false \
--go-grpc_out=. --go-grpc_opt=paths=source_relative $$f; \
done done
rm -rf vendor rm -rf vendor
# Build all-in-one NeoFS docker image # Build FrostFS component's docker image
image-aio: images
@echo "⇒ Build NeoFS All-In-One Docker image "
@docker build \
--build-arg HUB_IMAGE=$(HUB_IMAGE) \
--build-arg HUB_TAG=$(HUB_TAG) \
--rm \
-f Dockerfile.aio \
-t $(HUB_IMAGE)-aio:$(HUB_TAG) .
# Build NeoFS component's docker image
image-%: image-%:
@echo "⇒ Build NeoFS $* docker image " @echo "⇒ Build FrostFS $* docker image "
@docker build \ @docker build \
--build-arg REPO=$(REPO) \ --build-arg REPO=$(REPO) \
--build-arg VERSION=$(VERSION) \ --build-arg VERSION=$(VERSION) \
--rm \ --rm \
-f Dockerfile.$* \ -f .docker/Dockerfile.$* \
-t $(HUB_IMAGE)-$*:$(HUB_TAG) . -t $(HUB_IMAGE)-$*:$(HUB_TAG) .
# Build all Docker images # Build all Docker images
images: image-storage image-ir image-cli images: image-storage image-ir image-cli image-adm image-storage-testnet
# Build dirty local Docker images # Build dirty local Docker images
dirty-images: image-dirty-storage image-dirty-ir image-dirty-cli dirty-images: image-dirty-storage image-dirty-ir image-dirty-cli image-dirty-adm
# Run `make %` in Golang container
docker/%:
docker run --rm -t \
-v `pwd`:/src \
-w /src \
-u "$$(id -u):$$(id -g)" \
--env HOME=/src \
golang:$(GO_VERSION) make $*
# Run all code formatters # Run all code formatters
fmts: fmt imports fmts: fmt imports
@ -94,17 +116,17 @@ fmts: fmt imports
# Reformat code # Reformat code
fmt: fmt:
@echo "⇒ Processing gofmt check" @echo "⇒ Processing gofmt check"
@GO111MODULE=on gofmt -s -w cmd/ pkg/ misc/ @gofmt -s -w cmd/ pkg/ misc/
# Reformat imports # Reformat imports
imports: imports:
@echo "⇒ Processing goimports check" @echo "⇒ Processing goimports check"
@GO111MODULE=on goimports -w cmd/ pkg/ misc/ @goimports -w cmd/ pkg/ misc/
# Run Unit Test with go test # Run Unit Test with go test
test: test:
@echo "⇒ Running go test" @echo "⇒ Running go test"
@GO111MODULE=on go test ./... @go test ./...
# Run linters # Run linters
lint: lint:
@ -112,26 +134,30 @@ lint:
# Run linters in Docker # Run linters in Docker
docker/lint: docker/lint:
docker run --rm -it \ docker run --rm -t \
-v `pwd`:/src \ -v `pwd`:/src \
-u `stat -c "%u:%g" .` \ -u `stat -c "%u:%g" .` \
--env HOME=/src \ --env HOME=/src \
golangci/golangci-lint:v1.30 bash -c 'cd /src/ && make lint' golangci/golangci-lint:v$(LINT_VERSION) bash -c 'cd /src/ && make lint'
# Print version # Print version
version: version:
@echo $(VERSION) @echo $(VERSION)
# Show this help prompt
help:
@echo ' Usage:'
@echo ''
@echo ' make <target>'
@echo ''
@echo ' Targets:'
@echo ''
@awk '/^#/{ comment = substr($$0,3) } comment && /^[a-zA-Z][a-zA-Z0-9_-]+ ?:/{ print " ", $$1, comment }' $(MAKEFILE_LIST) | column -t -s ':' | grep -v 'IGNORE' | sort -u
clean: clean:
rm -rf vendor rm -rf vendor
rm -rf .cache
rm -rf $(BIN) rm -rf $(BIN)
rm -rf $(RELEASE)
# Package for Debian
debpackage:
dch -b --package frostfs-node \
--controlmaint \
--newversion $(PKG_VERSION) \
--distribution $(OS_RELEASE) \
"Please see CHANGELOG.md for code changes for $(VERSION)"
dpkg-buildpackage --no-sign -b
debclean:
dh clean

View file

@ -1,49 +1,80 @@
<p align="center"> <p align="center">
<img src="./.github/logo.svg" width="500px" alt="NeoFS"> <img src="./.github/logo.svg" width="500px" alt="FrostFS">
</p> </p>
<p align="center"> <p align="center">
<a href="https://fs.neo.org">NeoFS</a> is a decentralized distributed object storage integrated with the <a href="https://neo.org">NEO Blockchain</a>. <a href="https://frostfs.info">FrostFS</a> is a decentralized distributed object storage integrated with the <a href="https://neo.org">NEO Blockchain</a>.
</p> </p>
--- ---
[![Report](https://goreportcard.com/badge/github.com/nspcc-dev/neo-go)](https://goreportcard.com/report/github.com/nspcc-dev/neofs-node) [![Report](https://goreportcard.com/badge/github.com/TrueCloudLab/frostfs-node)](https://goreportcard.com/report/github.com/TrueCloudLab/frostfs-node)
![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/nspcc-dev/neofs-node?sort=semver) ![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/TrueCloudLab/frostfs-node?sort=semver)
![License](https://img.shields.io/github/license/nspcc-dev/neofs-node.svg?style=popout) ![License](https://img.shields.io/github/license/TrueCloudLab/frostfs-node.svg?style=popout)
# Overview # Overview
NeoFS Nodes are organized in peer-to-peer network that takes care of storing and FrostFS Nodes are organized in a peer-to-peer network that takes care of storing
distributing user's data. Any Neo user may participate in the network and get and distributing user's data. Any Neo user may participate in the network and
paid for providing storage resources to other users or store his data in NeoFS get paid for providing storage resources to other users or store their data in
and pay a competitive price for it. FrostFS and pay a competitive price for it.
Users can reliably store object data in the NeoFS network and have a transparent Users can reliably store object data in the FrostFS network and have a transparent
data placement process due to decentralized architecture and flexible storage data placement process due to a decentralized architecture and flexible storage
policies. Each node is responsible for executing the storage policies that the policies. Each node is responsible for executing the storage policies that the
users select for geographical location, reliability level, number of nodes, type users select for geographical location, reliability level, number of nodes, type
of disks, capacity, etc. Thus, NeoFS gives full control over data to users. of disks, capacity, etc. Thus, FrostFS gives full control over data to users.
Deep [Neo Blockchain](https://neo.org) integration allows NeoFS to be used by Deep [Neo Blockchain](https://neo.org) integration allows FrostFS to be used by
dApp directly from dApps directly from
[NeoVM](https://docs.neo.org/docs/en-us/basic/technology/neovm.html) on the [NeoVM](https://docs.neo.org/docs/en-us/basic/technology/neovm.html) on the
[Smart Contract](https://docs.neo.org/docs/en-us/basic/technology/neocontract.html) [Smart Contract](https://docs.neo.org/docs/en-us/intro/glossary.html)
code level. This way dApps are not limited to on-chain storage and can code level. This way dApps are not limited to on-chain storage and can
manipulate large amounts of data without paying a prohibitive price. manipulate large amounts of data without paying a prohibitive price.
NeoFS has native [gRPC](https://grpc.io) API and popular protocol gates such as FrostFS has a native [gRPC API](https://github.com/TrueCloudLab/frostfs-api) and has
[AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html), protocol gateways for popular protocols such as [AWS
[HTTP](https://wikipedia.org/wiki/Hypertext_Transfer_Protocol), S3](https://github.com/TrueCloudLab/frostfs-s3-gw),
[HTTP](https://github.com/TrueCloudLab/frostfs-http-gw),
[FUSE](https://wikipedia.org/wiki/Filesystem_in_Userspace) and [FUSE](https://wikipedia.org/wiki/Filesystem_in_Userspace) and
[sFTP](https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol) allowing [sFTP](https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol) allowing
developers to easily integrate applications without rewriting their code. developers to integrate applications without rewriting their code.
# Supported platforms # Supported platforms
For now we only support GNU/Linux on amd64 CPUs with AVX/AVX2 instructions. More Now, we only support GNU/Linux on amd64 CPUs with AVX/AVX2 instructions. More
platforms will be officially supported after '1.0' release. platforms will be officially supported after release `1.0`.
Latest version of neofs-node works with neofs-contract The latest version of frostfs-node works with frostfs-contract
[v0.9.0](https://github.com/nspcc-dev/neofs-contract/releases/tag/v0.9.0). [v0.16.0](https://github.com/TrueCloudLab/frostfs-contract/releases/tag/v0.16.0).
# Building
To make all binaries you need Go 1.18+ and `make`:
```
make all
```
The resulting binaries will appear in `bin/` folder.
To make a specific binary use:
```
make bin/frostfs-<name>
```
See the list of all available commands in the `cmd` folder.
## Building with Docker
Building can also be performed in a container:
```
make docker/all # build all binaries
make docker/bin/frostfs-<name> # build a specific binary
```
## Docker images
To make docker images suitable for use in [frostfs-dev-env](https://github.com/TrueCloudLab/frostfs-dev-env/) use:
```
make images
```
# Contributing # Contributing
@ -55,7 +86,7 @@ the feature/topic you are going to implement.
# Credits # Credits
NeoFS is maintained by [NeoSPCC](https://nspcc.ru) with the help and FrostFS is maintained by [True Cloud Lab](https://github.com/TrueCloudLab/) with the help and
contributions from community members. contributions from community members.
Please see [CREDITS](CREDITS.md) for details. Please see [CREDITS](CREDITS.md) for details.

1
VERSION Normal file
View file

@ -0,0 +1 @@
v0.35.0

104
cmd/frostfs-adm/README.md Normal file
View file

@ -0,0 +1,104 @@
# FrostFS Admin Tool
## Overview
Admin tool provides an easier way to deploy and maintain private installation
of FrostFS. Private installation provides a set of N3 consensus nodes, FrostFS
Alphabet, and Storage nodes. Admin tool generates consensus keys, initializes
the sidechain, and provides functions to update the network and register new
Storage nodes.
## Build
To build binary locally, use `make bin/frostfs-adm` command.
For clean build inside a docker container, use `make docker/bin/frostfs-adm`.
Build docker image with `make image-adm`.
At FrostFS private install deployment, frostfs-adm requires compiled FrostFS
contracts. Find them in the latest release of
[frostfs-contract repository](https://github.com/TrueCloudLab/frostfs-contract/releases).
## Commands
### Config
Config section provides `init` command that creates a configuration file for
private installation deployment and updates. Config file is optional, all
parameters can be passed by arguments or read from standard input (wallet
passwords).
Config example:
```yaml
rpc-endpoint: https://address:port # sidechain RPC node endpoint
alphabet-wallets: /path # path to consensus node / alphabet wallets storage
network:
max_object_size: 67108864 # max size of a single FrostFS object, bytes
epoch_duration: 240 # duration of a FrostFS epoch in blocks, consider block generation frequency in the sidechain
basic_income_rate: 0 # basic income rate, for private consider 0
fee:
audit: 0 # network audit fee, for private installation consider 0
candidate: 0 # inner ring candidate registration fee, for private installation consider 0
container: 0 # container creation fee, for private installation consider 0
container_alias: 0 # container nice-name registration fee, for private installation consider 0
withdraw: 0 # withdraw fee, for private installation consider 0
credentials: # passwords for consensus node / alphabet wallets
az: password1
buky: password2
vedi: password3
glagoli: password4
dobro: password5
yest: password6
zhivete: password7
```
### Morph
#### Network deployment
- `generate-alphabet` generates a set of wallets for consensus and
Alphabet nodes.
- `init` initializes the sidechain by deploying smart contracts and
setting provided FrostFS network configuration.
- `generate-storage-wallet` generates a wallet for the Storage node that
is ready for deployment. It also transfers a bit of sidechain GAS, so this
wallet can be used for FrostFS bootstrap.
#### Network maintenance
- `set-config` add/update configuration values in the Netmap contract.
- `force-new-epoch` increments FrostFS epoch number and executes new epoch
handlers in FrostFS nodes.
- `refill-gas` transfers sidechain GAS to the specified wallet.
- `update-contracts` updates contracts to a new version.
#### Container migration
If a network has to be redeployed, these commands will migrate all container meta
info. These commands **do not migrate actual objects**.
- `dump-containers` saves all containers and metadata registered in the container
contract to a file.
- `restore-containers` restores previously saved containers by their repeated registration in
the container contract.
- `list-containers` output all containers ids.
#### Network info
- `dump-config` prints FrostFS network configuration.
- `dump-hashes` prints FrostFS contract addresses stored in NNS.
## Private network deployment
Read step-by-step guide of private storage deployment [in docs](./docs/deploy.md).

View file

@ -0,0 +1,211 @@
# Step-by-step private FrostFS deployment
This is a short guide on how to deploy a private FrostFS storage network on bare
metal without docker images. This guide does not cover details on how to start
consensus, Alphabet, or Storage nodes. This guide covers only `frostfs-adm`
related configuration details.
## Prerequisites
To follow this guide you need:
- latest released version of [neo-go](https://github.com/nspcc-dev/neo-go/releases) (v0.97.2 at the moment),
- latest released version of [frostfs-adm](https://github.com/TrueCloudLab/frostfs-node/releases) utility (v0.25.1 at the moment),
- latest released version of compiled [frostfs-contract](https://github.com/TrueCloudLab/frostfs-contract/releases) (v0.11.0 at the moment).
## Step 1: Prepare network configuration
To start a network, you need a set of consensus nodes, the same number of
Alphabet nodes and any number of Storage nodes. While the number of Storage
nodes can be scaled almost infinitely, the number of consensus and Alphabet
nodes can't be changed so easily right now. Consider this before going any further.
It is easier to use`frostfs-adm` with a predefined configuration. First, create
a network configuration file. In this example, there is going to be only one
consensus / Alphabet node in the network.
```
$ frostfs-adm config init --path foo.network.yml
Initial config file saved to foo.network.yml
$ cat foo.network.yml
rpc-endpoint: https://neo.rpc.node:30333
alphabet-wallets: /home/user/deploy/alphabet-wallets
network:
max_object_size: 67108864
epoch_duration: 240
basic_income_rate: 0
fee:
audit: 0
candidate: 0
container: 0
withdraw: 0
credentials:
az: hunter2
```
For private installation, it is recommended to set all **fees** and **basic
income rate** to 0.
As for **epoch duration**, consider consensus node block generation frequency.
With default 15 seconds per block, 240 blocks are going to be a 1-hour epoch.
For **max object size**, 67108864 (64 MiB) or 134217728 (128 MiB) should provide
good chunk distribution in most cases.
With this config, generate wallets (private keys) of consensus nodes. The same
wallets will be used for Alphabet nodes. Make sure, that dir for alphabet
wallets already exists.
```
$ frostfs-adm -c foo.network.yml morph generate-alphabet --size 1
size: 1
alphabet-wallets: /home/user/deploy/alphabet-wallets
wallet[0]: hunter2
```
Do not lose wallet files and network config. Store it in an encrypted backed up
storage.
## Step 2: Launch consensus nodes
Configure blockchain nodes with the generated wallets from the previous step.
Config examples can be found in
[neo-go repository](https://github.com/nspcc-dev/neo-go/tree/master/config).
Gather public keys from **all** generated wallets. We are interested in the first
`simple signature contract` public key.
```
$ neo-go wallet dump-keys -w alphabet-wallets/az.json
NitdS4k4f1Hh5mbLJhAswBK3WC2gQgPN1o (simple signature contract):
02c1cc85f9c856dbe2d02017349bcb7b4e5defa78b8056a09b3240ba2a8c078869
NiMKabp3ddi3xShmLAXhTfbnuWb4cSJT6E (1 out of 1 multisig contract):
02c1cc85f9c856dbe2d02017349bcb7b4e5defa78b8056a09b3240ba2a8c078869
NiMKabp3ddi3xShmLAXhTfbnuWb4cSJT6E (1 out of 1 multisig contract):
02c1cc85f9c856dbe2d02017349bcb7b4e5defa78b8056a09b3240ba2a8c078869
```
Put the list of public keys into `ProtocolConfiguration.StandbyCommittee`
section. Specify the wallet path and the password in `ApplicationConfiguration.P2PNotary`
and `ApplicationConfiguration.UnlockWallet` sections. If config includes
`ProtocolConfiguration.NativeActivations` section, add notary
contract `Notary: [0]`.
```yaml
ProtocolConfiguration:
StandbyCommittee:
- 02c1cc85f9c856dbe2d02017349bcb7b4e5defa78b8056a09b3240ba2a8c078869
NativeActivations:
Notary: [0]
ApplicationConfiguration:
P2PNotary:
Enabled: true
UnlockWallet:
Path: "/home/user/deploy/alphabet-wallets/az.json"
Password: "hunter2"
UnlockWallet:
Path: "/home/user/deploy/alphabet-wallets/az.json"
Password: "hunter2"
```
Then, launch consensus nodes. They should connect to each other and start
producing blocks in consensus. You might want to deploy additional RPC
nodes at this stage because Storage nodes should be connected to the chain too.
It is not recommended to use a consensus node as an RPC node due to security policies
and possible overload issues.
## Step 3: Initialize sidechain
Use archive with compiled FrostFS contracts to initialize the sidechain.
```
$ tar -xzvf frostfs-contract-v0.11.0.tar.gz
$ ./frostfs-adm -c foo.network.yml morph init --contracts ./frostfs-contract-v0.11.0
Stage 1: transfer GAS to alphabet nodes.
Waiting for transactions to persist...
Stage 2: set notary and alphabet nodes in designate contract.
Waiting for transactions to persist...
Stage 3: deploy NNS contract.
Waiting for transactions to persist...
Stage 4: deploy FrostFS contracts.
Waiting for transactions to persist...
Stage 4.1: Transfer GAS to proxy contract.
Waiting for transactions to persist...
Stage 5: register candidates.
Waiting for transactions to persist...
Stage 6: transfer NEO to alphabet contracts.
Waiting for transactions to persist...
Stage 7: set addresses in NNS.
Waiting for transactions to persist...
NNS: Set alphabet0.frostfs -> f692dfb4d43a15b464eb51a7041160fb29c44b6a
NNS: Set audit.frostfs -> 7df847b993affb3852074345a7c2bd622171ee0d
NNS: Set balance.frostfs -> 103519b3067a66307080a66570c0491ee8f68879
NNS: Set container.frostfs -> cae60bdd689d185901e495352d0247752ce50846
NNS: Set frostfsid.frostfs -> c421fb60a3895865a8f24d197d6a80ef686041d2
NNS: Set netmap.frostfs -> 894eb854632f50fb124412ce7951ebc00763525e
NNS: Set proxy.frostfs -> ac6e6fe4b373d0ca0ca4969d1e58fa0988724e7d
NNS: Set reputation.frostfs -> 6eda57c9d93d990573646762d1fea327ce41191f
Waiting for transactions to persist...
```
## Step 4: Launch Alphabet nodes
Configure Alphabet nodes with the wallets generated in step 1. For
`morph.validators` use a list of public keys from
`ProtocolConfiguration.StandbyCommittee`.
```yaml
wallet:
path: "/home/user/deploy/alphabet-wallets/az.json"
password: "hunter2"
account: "NitdS4k4f1Hh5mbLJhAswBK3WC2gQgPN1o"
morph:
validators:
- 02c1cc85f9c856dbe2d02017349bcb7b4e5defa78b8056a09b3240ba2a8c078869
contracts:
alphabet:
amount: 1
```
## Step 4: Launch Storage node
Generate a new wallet for a Storage node.
```
$ frostfs-adm -c foo.network.yml morph generate-storage-wallet --storage-wallet ./sn01.json --initial-gas 10.0
New password >
Waiting for transactions to persist...
$ neo-go wallet dump-keys -w sn01.json
Ngr7p8Z9S22XDH6VkUG9oXobv8zZRAWwwv (simple signature contract):
0355eccb72cd46f09a3e5237eaa0f4949cceb5ecfa5a225bd3bb9fd021c4d75b85
```
Configure the Storage node to use this wallet.
```
node:
wallet:
path: "/home/user/deploy/sn01.json"
address: "Ngr7p8Z9S22XDH6VkUG9oXobv8zZRAWwwv"
password: "foobar"
```
The storage node will be included in the network map in the next FrostFS epoch. To
speed up this process, you can increment epoch counter immediately.
```
$ frostfs-adm -c foo.network.yml morph force-new-epoch
Current epoch: 8, increase to 9.
Waiting for transactions to persist...
```
---
After that, FrostFS Storage is ready to work. You can access it directly or
with protocol gates.

View file

@ -0,0 +1,39 @@
# FrostFS subnetwork creation
This is a short guide on how to create FrostFS subnetworks. This guide
considers that the sidechain and the inner ring (alphabet nodes) have already been
deployed and the sidechain contains a deployed `subnet` contract.
## Prerequisites
To follow this guide, you need:
- neo-go sidechain RPC endpoint;
- latest released version of [frostfs-adm](https://github.com/TrueCloudLab/frostfs-node/releases);
- wallet with FrostFS account.
## Creation
```shell
$ frostfs-adm morph subnet create \
-r <side_chain_RPC_endpoint> \
-w </path/to/owner/wallet> \
--notary
Create subnet request sent successfully. ID: 4223489767.
```
**NOTE:** in notary-enabled environment you should have a sufficient
notary deposit (not expired, with enough GAS balance). Your subnet ID
will differ from the example.
The default account in the wallet that has been passed with `-w` flag is the owner
of the just created subnetwork.
You can check if your subnetwork was created successfully:
```shell
$ frostfs-adm morph subnet get \
-r <side_chain_RPC_endpoint> \
--subnet <subnet_ID>
Owner: NUc734PMJXiqa2J9jRtvskU3kCdyyuSN8Q
```
Your owner will differ from the example.

View file

@ -0,0 +1,137 @@
# Managing Subnetworks
This is a short guide on how to manage FrostFS subnetworks. This guide
considers that the sidechain and the inner ring (alphabet nodes) have already been
deployed, and the sidechain contains a deployed `subnet` contract.
## Prerequisites
- neo-go sidechain RPC endpoint;
- latest released version of [frostfs-adm](https://github.com/TrueCloudLab/frostfs-node/releases);
- [created](subnetwork-creation.md) subnetwork;
- wallet with the account that owns the subnetwork;
- public key of the Storage Node;
- public keys of the node and client administrators;
- owner IDs of the FrostFS users.
## Add node administrator
Node administrators are accounts that can manage (add and delete nodes)
the whitelist of the nodes which can be included to a subnetwork. Only the subnet
owner is allowed to add and remove node administrators from the subnetwork.
```shell
$ frostfs-adm morph subnet admin add \
-r <side_chain_RPC_endpoint> \
-w </path/to/owner/wallet> \
--admin <HEX_admin_public_key> \
--subnet <subnet_ID>
Add admin request sent successfully.
```
## Add node
Adding a node to a subnetwork means that the node becomes able to service
containers that have been created in that subnetwork. Addition only changes
the list of the allowed nodes. Node is not required to be bootstrapped at the
moment of its inclusion.
```shell
$ frostfs-adm morph subnet node add \
-r <side_chain_RPC_endpoint> \
-w </path/to/node_admin/wallet> \
--node <HEX_node_public_key> \
--subnet <subnet_ID>
Add node request sent successfully.
```
**NOTE:** the owner of the subnetwork is also allowed to add nodes.
## Add client administrator
Client administrators are accounts that can manage (add and delete
nodes) the whitelist of the clients that can create containers in the
subnetwork. Only the subnet owner is allowed to add and remove client
administrators from the subnetwork.
```shell
$ frostfs-adm morph subnet admin add \
-r <side_chain_RPC_endpoint> \
-w </path/to/owner/wallet> \
--admin <HEX_admin_public_key> \
--subnet <subnet_ID> \
--client \
--group <group_ID>
Add admin request sent successfully.
```
**NOTE:** you do not need to create a group explicitly, it will be created
right after the first client admin is added. Group ID is a 4-byte
positive integer number.
## Add client
```shell
$ frostfs-adm morph subnet client add \
-r <side_chain_RPC_endpoint> \
-w </path/to/client_admin/wallet> \
--client <client_ownerID> \
--subnet <subnet_ID> \
--group <group_ID>
Add client request sent successfully.
```
**NOTE:** the owner of the subnetwork is also allowed to add clients. This is
the only one command that accepts `ownerID`, not the public key.
Administrator can manage only their group (a group where that administrator
has been added by the subnet owner).
# Bootstrapping Storage Node
After a subnetwork [is created](subnetwork-creation.md) and a node is included into it, the
node could be bootstrapped and service subnetwork containers.
For bootstrapping, you need to specify the ID of the subnetwork in the node's
configuration:
```yaml
...
node:
...
subnet:
entries: # list of IDs of subnets to enter in a text format of FrostFS API protocol (overrides corresponding attributes)
- <subnetwork_ID>
...
...
```
**NOTE:** specifying subnetwork that is denied for the node is not an error:
that configuration value would be ignored. You do not need to specify zero
(with 0 ID) subnetwork: its inclusion is implicit. On the contrary, to exclude
a node from the default zero subnetwork, you need to specify it explicitly:
```yaml
...
node:
...
subnet:
exit_zero: true # toggle entrance to zero subnet (overrides corresponding attribute and occurrence in `entries`)
...
...
```
# Creating container in non-zero subnetwork
Creating containers without using `--subnet` flag is equivalent to
creating container in the zero subnetwork.
To create a container in a private network, your wallet must be added to
the client whitelist by the client admins or the subnet owners:
```shell
$ frostfs-cli container create \
--policy 'REP 1' \
-w </path/to/wallet> \
-r s01.frostfs.devenv:8080 \
--subnet <subnet_ID>
```

View file

@ -0,0 +1,14 @@
package commonflags
const (
ConfigFlag = "config"
ConfigFlagShorthand = "c"
ConfigFlagUsage = "Config file"
ConfigDirFlag = "config-dir"
ConfigDirFlagUsage = "Config directory"
Verbose = "verbose"
VerboseShorthand = "v"
VerboseUsage = "Verbose output"
)

View file

@ -0,0 +1,166 @@
package config
import (
"bytes"
"fmt"
"os"
"path/filepath"
"text/template"
"github.com/TrueCloudLab/frostfs-node/pkg/innerring"
"github.com/nspcc-dev/neo-go/cli/input"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
type configTemplate struct {
Endpoint string
AlphabetDir string
MaxObjectSize int
EpochDuration int
BasicIncomeRate int
AuditFee int
CandidateFee int
ContainerFee int
ContainerAliasFee int
WithdrawFee int
Glagolitics []string
HomomorphicHashDisabled bool
}
const configTxtTemplate = `rpc-endpoint: {{ .Endpoint}}
alphabet-wallets: {{ .AlphabetDir}}
network:
max_object_size: {{ .MaxObjectSize}}
epoch_duration: {{ .EpochDuration}}
basic_income_rate: {{ .BasicIncomeRate}}
homomorphic_hash_disabled: {{ .HomomorphicHashDisabled}}
fee:
audit: {{ .AuditFee}}
candidate: {{ .CandidateFee}}
container: {{ .ContainerFee}}
container_alias: {{ .ContainerAliasFee }}
withdraw: {{ .WithdrawFee}}
# if credentials section is omitted, then frostfs-adm will require manual password input
credentials:
contract: password # wallet for contract group signature{{ range.Glagolitics}}
{{.}}: password{{end}}
`
func initConfig(cmd *cobra.Command, args []string) error {
configPath, err := readConfigPathFromArgs(cmd)
if err != nil {
return nil
}
pathDir := filepath.Dir(configPath)
err = os.MkdirAll(pathDir, 0700)
if err != nil {
return fmt.Errorf("create dir %s: %w", pathDir, err)
}
f, err := os.OpenFile(configPath, os.O_RDWR|os.O_CREATE|os.O_TRUNC|os.O_SYNC, 0600)
if err != nil {
return fmt.Errorf("open %s: %w", configPath, err)
}
defer f.Close()
configText, err := generateConfigExample(pathDir, 7)
if err != nil {
return err
}
_, err = f.WriteString(configText)
if err != nil {
return fmt.Errorf("writing to %s: %w", configPath, err)
}
cmd.Printf("Initial config file saved to %s\n", configPath)
return nil
}
func readConfigPathFromArgs(cmd *cobra.Command) (string, error) {
configPath, err := cmd.Flags().GetString(configPathFlag)
if err != nil {
return "", err
}
if configPath != "" {
return configPath, nil
}
return defaultConfigPath()
}
func defaultConfigPath() (string, error) {
home, err := os.UserHomeDir()
if err != nil {
return "", fmt.Errorf("getting home dir path: %w", err)
}
return filepath.Join(home, ".frostfs", "adm", "config.yml"), nil
}
// generateConfigExample builds .yml representation of the config file. It is
// easier to build it manually with template instead of using viper, because we
// want to order records in specific order in file and, probably, provide
// some comments as well.
func generateConfigExample(appDir string, credSize int) (string, error) {
tmpl := configTemplate{
Endpoint: "https://neo.rpc.node:30333",
MaxObjectSize: 67108864, // 64 MiB
EpochDuration: 240, // 1 hour with 15s per block
BasicIncomeRate: 1_0000_0000, // 0.0001 GAS per GiB (Fixed12)
HomomorphicHashDisabled: false, // object homomorphic hash is enabled
AuditFee: 1_0000, // 0.00000001 GAS per audit (Fixed12)
CandidateFee: 100_0000_0000, // 100.0 GAS (Fixed8)
ContainerFee: 1000, // 0.000000001 * 7 GAS per container (Fixed12)
ContainerAliasFee: 500, // ContainerFee / 2
WithdrawFee: 1_0000_0000, // 1.0 GAS (Fixed8)
Glagolitics: make([]string, 0, credSize),
}
appDir, err := filepath.Abs(appDir)
if err != nil {
return "", fmt.Errorf("making absolute path for %s: %w", appDir, err)
}
tmpl.AlphabetDir = filepath.Join(appDir, "alphabet-wallets")
var i innerring.GlagoliticLetter
for i = 0; i < innerring.GlagoliticLetter(credSize); i++ {
tmpl.Glagolitics = append(tmpl.Glagolitics, i.String())
}
t, err := template.New("config.yml").Parse(configTxtTemplate)
if err != nil {
return "", fmt.Errorf("parsing config template: %w", err)
}
buf := bytes.NewBuffer(nil)
err = t.Execute(buf, tmpl)
if err != nil {
return "", fmt.Errorf("generating config from template: %w", err)
}
return buf.String(), nil
}
func GetPassword(v *viper.Viper, name string) (string, error) {
key := "credentials." + name
if v.IsSet(key) {
return v.GetString(key), nil
}
prompt := "Password for " + name + " wallet > "
return input.ReadPassword(prompt)
}
func GetStoragePassword(v *viper.Viper, name string) (string, error) {
key := "storage." + name
if name != "" && v.IsSet(key) {
return v.GetString(key), nil
}
return input.ReadPassword("New password > ")
}

View file

@ -0,0 +1,46 @@
package config
import (
"bytes"
"path/filepath"
"testing"
"github.com/TrueCloudLab/frostfs-node/pkg/innerring"
"github.com/spf13/viper"
"github.com/stretchr/testify/require"
)
func TestGenerateConfigExample(t *testing.T) {
const (
n = 10
appDir = "/home/example/.frostfs"
)
configText, err := generateConfigExample(appDir, n)
require.NoError(t, err)
v := viper.New()
v.SetConfigType("yml")
require.NoError(t, v.ReadConfig(bytes.NewBufferString(configText)))
require.Equal(t, "https://neo.rpc.node:30333", v.GetString("rpc-endpoint"))
require.Equal(t, filepath.Join(appDir, "alphabet-wallets"), v.GetString("alphabet-wallets"))
require.Equal(t, 67108864, v.GetInt("network.max_object_size"))
require.Equal(t, 240, v.GetInt("network.epoch_duration"))
require.Equal(t, 100000000, v.GetInt("network.basic_income_rate"))
require.Equal(t, 10000, v.GetInt("network.fee.audit"))
require.Equal(t, 10000000000, v.GetInt("network.fee.candidate"))
require.Equal(t, 1000, v.GetInt("network.fee.container"))
require.Equal(t, 100000000, v.GetInt("network.fee.withdraw"))
var i innerring.GlagoliticLetter
for i = 0; i < innerring.GlagoliticLetter(n); i++ {
key := "credentials." + i.String()
require.Equal(t, "password", v.GetString(key))
}
require.Equal(t, "password", v.GetString("credentials.contract"))
key := "credentials." + i.String()
require.Equal(t, "", v.GetString(key))
}

View file

@ -0,0 +1,29 @@
package config
import (
"github.com/spf13/cobra"
)
const configPathFlag = "path"
var (
// RootCmd is a root command of config section.
RootCmd = &cobra.Command{
Use: "config",
Short: "Section for frostfs-adm config related commands",
}
initCmd = &cobra.Command{
Use: "init",
Short: "Initialize basic frostfs-adm configuration file",
Example: `frostfs-adm config init
frostfs-adm config init --path .config/frostfs-adm.yml`,
RunE: initConfig,
}
)
func init() {
RootCmd.AddCommand(initCmd)
initCmd.Flags().String(configPathFlag, "", "Path to config (default ~/.frostfs/adm/config.yml)")
}

View file

@ -0,0 +1,24 @@
package config
import (
"os"
"path/filepath"
"strings"
)
// ResolveHomePath replaces leading `~`
// with home directory.
//
// Does nothing if path does not start
// with contain `~`.
func ResolveHomePath(path string) string {
homeDir, _ := os.UserHomeDir()
if path == "~" {
path = homeDir
} else if strings.HasPrefix(path, "~/") {
path = filepath.Join(homeDir, path[2:])
}
return path
}

View file

@ -0,0 +1,241 @@
package morph
import (
"crypto/elliptic"
"errors"
"fmt"
"math/big"
"github.com/TrueCloudLab/frostfs-contract/nns"
"github.com/TrueCloudLab/frostfs-sdk-go/netmap"
"github.com/nspcc-dev/neo-go/pkg/core/native/noderoles"
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/nspcc-dev/neo-go/pkg/encoding/fixedn"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/gas"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/rolemgmt"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
"github.com/nspcc-dev/neo-go/pkg/vm/vmstate"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
type accBalancePair struct {
scriptHash util.Uint160
balance *big.Int
}
const (
dumpBalancesStorageFlag = "storage"
dumpBalancesAlphabetFlag = "alphabet"
dumpBalancesProxyFlag = "proxy"
dumpBalancesUseScriptHashFlag = "script-hash"
// notaryEnabled signifies whether contracts were deployed in a notary-enabled environment.
// The setting is here to simplify testing and building the command for testnet (notary currently disabled).
// It will be removed eventually.
notaryEnabled = true
)
func dumpBalances(cmd *cobra.Command, _ []string) error {
var (
dumpStorage, _ = cmd.Flags().GetBool(dumpBalancesStorageFlag)
dumpAlphabet, _ = cmd.Flags().GetBool(dumpBalancesAlphabetFlag)
dumpProxy, _ = cmd.Flags().GetBool(dumpBalancesProxyFlag)
nnsCs *state.Contract
nmHash util.Uint160
)
c, err := getN3Client(viper.GetViper())
if err != nil {
return err
}
inv := invoker.New(c, nil)
if !notaryEnabled || dumpStorage || dumpAlphabet || dumpProxy {
nnsCs, err = c.GetContractStateByID(1)
if err != nil {
return fmt.Errorf("can't get NNS contract info: %w", err)
}
nmHash, err = nnsResolveHash(inv, nnsCs.Hash, netmapContract+".frostfs")
if err != nil {
return fmt.Errorf("can't get netmap contract hash: %w", err)
}
}
irList, err := fetchIRNodes(c, nmHash, rolemgmt.Hash)
if err != nil {
return err
}
if err := fetchBalances(inv, gas.Hash, irList); err != nil {
return err
}
printBalances(cmd, "Inner ring nodes balances:", irList)
if dumpStorage {
arr, err := unwrap.Array(inv.Call(nmHash, "netmap"))
if err != nil {
return errors.New("can't fetch the list of storage nodes")
}
snList := make([]accBalancePair, len(arr))
for i := range arr {
node, ok := arr[i].Value().([]stackitem.Item)
if !ok || len(node) == 0 {
return errors.New("can't parse the list of storage nodes")
}
bs, err := node[0].TryBytes()
if err != nil {
return errors.New("can't parse the list of storage nodes")
}
var ni netmap.NodeInfo
if err := ni.Unmarshal(bs); err != nil {
return fmt.Errorf("can't parse the list of storage nodes: %w", err)
}
pub, err := keys.NewPublicKeyFromBytes(ni.PublicKey(), elliptic.P256())
if err != nil {
return fmt.Errorf("can't parse storage node public key: %w", err)
}
snList[i].scriptHash = pub.GetScriptHash()
}
if err := fetchBalances(inv, gas.Hash, snList); err != nil {
return err
}
printBalances(cmd, "\nStorage node balances:", snList)
}
if dumpProxy {
h, err := nnsResolveHash(inv, nnsCs.Hash, proxyContract+".frostfs")
if err != nil {
return fmt.Errorf("can't get hash of the proxy contract: %w", err)
}
proxyList := []accBalancePair{{scriptHash: h}}
if err := fetchBalances(inv, gas.Hash, proxyList); err != nil {
return err
}
printBalances(cmd, "\nProxy contract balance:", proxyList)
}
if dumpAlphabet {
alphaList := make([]accBalancePair, len(irList))
w := io.NewBufBinWriter()
for i := range alphaList {
emit.AppCall(w.BinWriter, nnsCs.Hash, "resolve", callflag.ReadOnly,
getAlphabetNNSDomain(i),
int64(nns.TXT))
}
if w.Err != nil {
panic(w.Err)
}
alphaRes, err := c.InvokeScript(w.Bytes(), nil)
if err != nil {
return fmt.Errorf("can't fetch info from NNS: %w", err)
}
for i := range alphaList {
h, err := parseNNSResolveResult(alphaRes.Stack[i])
if err != nil {
return fmt.Errorf("can't fetch the alphabet contract #%d hash: %w", i, err)
}
alphaList[i].scriptHash = h
}
if err := fetchBalances(inv, gas.Hash, alphaList); err != nil {
return err
}
printBalances(cmd, "\nAlphabet contracts balances:", alphaList)
}
return nil
}
func fetchIRNodes(c Client, nmHash, desigHash util.Uint160) ([]accBalancePair, error) {
var irList []accBalancePair
inv := invoker.New(c, nil)
if notaryEnabled {
height, err := c.GetBlockCount()
if err != nil {
return nil, fmt.Errorf("can't get block height: %w", err)
}
arr, err := getDesignatedByRole(inv, desigHash, noderoles.NeoFSAlphabet, height)
if err != nil {
return nil, errors.New("can't fetch list of IR nodes from the netmap contract")
}
irList = make([]accBalancePair, len(arr))
for i := range arr {
irList[i].scriptHash = arr[i].GetScriptHash()
}
} else {
arr, err := unwrap.ArrayOfBytes(inv.Call(nmHash, "innerRingList"))
if err != nil {
return nil, errors.New("can't fetch list of IR nodes from the netmap contract")
}
irList = make([]accBalancePair, len(arr))
for i := range arr {
pub, err := keys.NewPublicKeyFromBytes(arr[i], elliptic.P256())
if err != nil {
return nil, fmt.Errorf("can't parse IR node public key: %w", err)
}
irList[i].scriptHash = pub.GetScriptHash()
}
}
return irList, nil
}
func printBalances(cmd *cobra.Command, prefix string, accounts []accBalancePair) {
useScriptHash, _ := cmd.Flags().GetBool(dumpBalancesUseScriptHashFlag)
cmd.Println(prefix)
for i := range accounts {
var addr string
if useScriptHash {
addr = accounts[i].scriptHash.StringLE()
} else {
addr = address.Uint160ToString(accounts[i].scriptHash)
}
cmd.Printf("%s: %s\n", addr, fixedn.ToString(accounts[i].balance, 8))
}
}
func fetchBalances(c *invoker.Invoker, gasHash util.Uint160, accounts []accBalancePair) error {
w := io.NewBufBinWriter()
for i := range accounts {
emit.AppCall(w.BinWriter, gasHash, "balanceOf", callflag.ReadStates, accounts[i].scriptHash)
}
if w.Err != nil {
panic(w.Err)
}
res, err := c.Run(w.Bytes())
if err != nil || res.State != vmstate.Halt.String() || len(res.Stack) != len(accounts) {
return errors.New("can't fetch account balances")
}
for i := range accounts {
bal, err := res.Stack[i].TryInteger()
if err != nil {
return fmt.Errorf("can't parse account balance: %w", err)
}
accounts[i].balance = bal
}
return nil
}

View file

@ -0,0 +1,192 @@
package morph
import (
"bytes"
"encoding/binary"
"encoding/hex"
"errors"
"fmt"
"strconv"
"strings"
"text/tabwriter"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
const forceConfigSet = "force"
func dumpNetworkConfig(cmd *cobra.Command, _ []string) error {
c, err := getN3Client(viper.GetViper())
if err != nil {
return fmt.Errorf("can't create N3 client: %w", err)
}
inv := invoker.New(c, nil)
cs, err := c.GetContractStateByID(1)
if err != nil {
return fmt.Errorf("can't get NNS contract info: %w", err)
}
nmHash, err := nnsResolveHash(inv, cs.Hash, netmapContract+".frostfs")
if err != nil {
return fmt.Errorf("can't get netmap contract hash: %w", err)
}
arr, err := unwrap.Array(inv.Call(nmHash, "listConfig"))
if err != nil {
return errors.New("can't fetch list of network config keys from the netmap contract")
}
buf := bytes.NewBuffer(nil)
tw := tabwriter.NewWriter(buf, 0, 2, 2, ' ', 0)
for _, param := range arr {
tuple, ok := param.Value().([]stackitem.Item)
if !ok || len(tuple) != 2 {
return errors.New("invalid ListConfig response from netmap contract")
}
k, err := tuple[0].TryBytes()
if err != nil {
return errors.New("invalid config key from netmap contract")
}
v, err := tuple[1].TryBytes()
if err != nil {
return invalidConfigValueErr(k)
}
switch string(k) {
case netmapAuditFeeKey, netmapBasicIncomeRateKey,
netmapContainerFeeKey, netmapContainerAliasFeeKey,
netmapEigenTrustIterationsKey,
netmapEpochKey, netmapInnerRingCandidateFeeKey,
netmapMaxObjectSizeKey, netmapWithdrawFeeKey:
nbuf := make([]byte, 8)
copy(nbuf[:], v)
n := binary.LittleEndian.Uint64(nbuf)
_, _ = tw.Write([]byte(fmt.Sprintf("%s:\t%d (int)\n", k, n)))
case netmapEigenTrustAlphaKey:
_, _ = tw.Write([]byte(fmt.Sprintf("%s:\t%s (str)\n", k, v)))
case netmapHomomorphicHashDisabledKey, netmapMaintenanceAllowedKey:
vBool, err := tuple[1].TryBool()
if err != nil {
return invalidConfigValueErr(k)
}
_, _ = tw.Write([]byte(fmt.Sprintf("%s:\t%t (bool)\n", k, vBool)))
default:
_, _ = tw.Write([]byte(fmt.Sprintf("%s:\t%s (hex)\n", k, hex.EncodeToString(v))))
}
}
_ = tw.Flush()
cmd.Print(buf.String())
return nil
}
func setConfigCmd(cmd *cobra.Command, args []string) error {
if len(args) == 0 {
return errors.New("empty config pairs")
}
wCtx, err := newInitializeContext(cmd, viper.GetViper())
if err != nil {
return fmt.Errorf("can't initialize context: %w", err)
}
cs, err := wCtx.Client.GetContractStateByID(1)
if err != nil {
return fmt.Errorf("can't get NNS contract info: %w", err)
}
nmHash, err := nnsResolveHash(wCtx.ReadOnlyInvoker, cs.Hash, netmapContract+".frostfs")
if err != nil {
return fmt.Errorf("can't get netmap contract hash: %w", err)
}
forceFlag, _ := cmd.Flags().GetBool(forceConfigSet)
bw := io.NewBufBinWriter()
for _, arg := range args {
k, v, err := parseConfigPair(arg, forceFlag)
if err != nil {
return err
}
// In NeoFS this is done via Notary contract. Here, however, we can form the
// transaction locally. The first `nil` argument is required only for notary
// disabled environment which is not supported by that command.
emit.AppCall(bw.BinWriter, nmHash, "setConfig", callflag.All, nil, k, v)
if bw.Err != nil {
return fmt.Errorf("can't form raw transaction: %w", bw.Err)
}
}
err = wCtx.sendConsensusTx(bw.Bytes())
if err != nil {
return err
}
return wCtx.awaitTx()
}
func parseConfigPair(kvStr string, force bool) (key string, val any, err error) {
k, v, found := strings.Cut(kvStr, "=")
if !found {
return "", nil, fmt.Errorf("invalid parameter format: must be 'key=val', got: %s", kvStr)
}
key = k
valRaw := v
switch key {
case netmapAuditFeeKey, netmapBasicIncomeRateKey,
netmapContainerFeeKey, netmapContainerAliasFeeKey,
netmapEigenTrustIterationsKey,
netmapEpochKey, netmapInnerRingCandidateFeeKey,
netmapMaxObjectSizeKey, netmapWithdrawFeeKey:
val, err = strconv.ParseInt(valRaw, 10, 64)
if err != nil {
err = fmt.Errorf("could not parse %s's value '%s' as int: %w", key, valRaw, err)
}
case netmapEigenTrustAlphaKey:
// just check that it could
// be parsed correctly
_, err = strconv.ParseFloat(v, 64)
if err != nil {
err = fmt.Errorf("could not parse %s's value '%s' as float: %w", key, valRaw, err)
}
val = valRaw
case netmapHomomorphicHashDisabledKey, netmapMaintenanceAllowedKey:
val, err = strconv.ParseBool(valRaw)
if err != nil {
err = fmt.Errorf("could not parse %s's value '%s' as bool: %w", key, valRaw, err)
}
default:
if !force {
return "", nil, fmt.Errorf(
"'%s' key is not well-known, use '--%s' flag if want to set it anyway",
key, forceConfigSet)
}
val = valRaw
}
return
}
func invalidConfigValueErr(key []byte) error {
return fmt.Errorf("invalid %s config value from netmap contract", key)
}

View file

@ -0,0 +1,373 @@
package morph
import (
"encoding/json"
"errors"
"fmt"
"os"
"sort"
cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
"github.com/nspcc-dev/neo-go/pkg/crypto/hash"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
var errInvalidContainerResponse = errors.New("invalid response from container contract")
func getContainerContractHash(cmd *cobra.Command, inv *invoker.Invoker, c Client) (util.Uint160, error) {
s, err := cmd.Flags().GetString(containerContractFlag)
var ch util.Uint160
if err == nil {
ch, err = util.Uint160DecodeStringLE(s)
}
if err != nil {
nnsCs, err := c.GetContractStateByID(1)
if err != nil {
return util.Uint160{}, fmt.Errorf("can't get NNS contract state: %w", err)
}
ch, err = nnsResolveHash(inv, nnsCs.Hash, containerContract+".frostfs")
if err != nil {
return util.Uint160{}, err
}
}
return ch, nil
}
func getContainersList(inv *invoker.Invoker, ch util.Uint160) ([][]byte, error) {
res, err := inv.Call(ch, "list", "")
if err != nil {
return nil, fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
itm, err := unwrap.Item(res, err)
if _, ok := itm.(stackitem.Null); !ok {
return unwrap.ArrayOfBytes(res, err)
}
return nil, nil
}
func dumpContainers(cmd *cobra.Command, _ []string) error {
filename, err := cmd.Flags().GetString(containerDumpFlag)
if err != nil {
return fmt.Errorf("invalid filename: %w", err)
}
c, err := getN3Client(viper.GetViper())
if err != nil {
return fmt.Errorf("can't create N3 client: %w", err)
}
inv := invoker.New(c, nil)
ch, err := getContainerContractHash(cmd, inv, c)
if err != nil {
return fmt.Errorf("unable to get contaract hash: %w", err)
}
cids, err := getContainersList(inv, ch)
if err != nil {
return fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
isOK, err := getCIDFilterFunc(cmd)
if err != nil {
return err
}
var containers []*Container
bw := io.NewBufBinWriter()
for _, id := range cids {
if !isOK(id) {
continue
}
bw.Reset()
emit.AppCall(bw.BinWriter, ch, "get", callflag.All, id)
emit.AppCall(bw.BinWriter, ch, "eACL", callflag.All, id)
res, err := inv.Run(bw.Bytes())
if err != nil {
return fmt.Errorf("can't get container info: %w", err)
}
if len(res.Stack) != 2 {
return fmt.Errorf("%w: expected 2 items on stack", errInvalidContainerResponse)
}
cnt := new(Container)
err = cnt.FromStackItem(res.Stack[0])
if err != nil {
return fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
ea := new(EACL)
err = ea.FromStackItem(res.Stack[1])
if err != nil {
return fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
if len(ea.Value) != 0 {
cnt.EACL = ea
}
containers = append(containers, cnt)
}
out, err := json.Marshal(containers)
if err != nil {
return err
}
return os.WriteFile(filename, out, 0o660)
}
func listContainers(cmd *cobra.Command, _ []string) error {
c, err := getN3Client(viper.GetViper())
if err != nil {
return fmt.Errorf("can't create N3 client: %w", err)
}
inv := invoker.New(c, nil)
ch, err := getContainerContractHash(cmd, inv, c)
if err != nil {
return fmt.Errorf("unable to get contaract hash: %w", err)
}
cids, err := getContainersList(inv, ch)
if err != nil {
return fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
for _, id := range cids {
var idCnr cid.ID
err = idCnr.Decode(id)
if err != nil {
return fmt.Errorf("unable to decode container id: %w", err)
}
cmd.Println(idCnr)
}
return nil
}
func restoreContainers(cmd *cobra.Command, _ []string) error {
filename, err := cmd.Flags().GetString(containerDumpFlag)
if err != nil {
return fmt.Errorf("invalid filename: %w", err)
}
wCtx, err := newInitializeContext(cmd, viper.GetViper())
if err != nil {
return err
}
defer wCtx.close()
nnsCs, err := wCtx.Client.GetContractStateByID(1)
if err != nil {
return fmt.Errorf("can't get NNS contract state: %w", err)
}
ch, err := nnsResolveHash(wCtx.ReadOnlyInvoker, nnsCs.Hash, containerContract+".frostfs")
if err != nil {
return fmt.Errorf("can't fetch container contract hash: %w", err)
}
data, err := os.ReadFile(filename)
if err != nil {
return fmt.Errorf("can't read dump file: %w", err)
}
var containers []Container
err = json.Unmarshal(data, &containers)
if err != nil {
return fmt.Errorf("can't parse dump file: %w", err)
}
isOK, err := getCIDFilterFunc(cmd)
if err != nil {
return err
}
bw := io.NewBufBinWriter()
for _, cnt := range containers {
hv := hash.Sha256(cnt.Value)
if !isOK(hv[:]) {
continue
}
bw.Reset()
emit.AppCall(bw.BinWriter, ch, "get", callflag.All, hv.BytesBE())
res, err := wCtx.Client.InvokeScript(bw.Bytes(), nil)
if err != nil {
return fmt.Errorf("can't check if container is already restored: %w", err)
}
if len(res.Stack) == 0 {
return errors.New("empty stack")
}
old := new(Container)
if err := old.FromStackItem(res.Stack[0]); err != nil {
return fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
if len(old.Value) != 0 {
var id cid.ID
id.SetSHA256(hv)
cmd.Printf("Container %s is already deployed.\n", id)
continue
}
bw.Reset()
emit.AppCall(bw.BinWriter, ch, "put", callflag.All,
cnt.Value, cnt.Signature, cnt.PublicKey, cnt.Token)
if ea := cnt.EACL; ea != nil {
emit.AppCall(bw.BinWriter, ch, "setEACL", callflag.All,
ea.Value, ea.Signature, ea.PublicKey, ea.Token)
}
if bw.Err != nil {
panic(bw.Err)
}
if err := wCtx.sendConsensusTx(bw.Bytes()); err != nil {
return err
}
}
return wCtx.awaitTx()
}
// Container represents container struct in contract storage.
type Container struct {
Value []byte `json:"value"`
Signature []byte `json:"signature"`
PublicKey []byte `json:"public_key"`
Token []byte `json:"token"`
EACL *EACL `json:"eacl"`
}
// EACL represents extended ACL struct in contract storage.
type EACL struct {
Value []byte `json:"value"`
Signature []byte `json:"signature"`
PublicKey []byte `json:"public_key"`
Token []byte `json:"token"`
}
// ToStackItem implements stackitem.Convertible.
func (c *Container) ToStackItem() (stackitem.Item, error) {
return stackitem.NewStruct([]stackitem.Item{
stackitem.NewByteArray(c.Value),
stackitem.NewByteArray(c.Signature),
stackitem.NewByteArray(c.PublicKey),
stackitem.NewByteArray(c.Token),
}), nil
}
// FromStackItem implements stackitem.Convertible.
func (c *Container) FromStackItem(item stackitem.Item) error {
arr, ok := item.Value().([]stackitem.Item)
if !ok || len(arr) != 4 {
return errors.New("invalid stack item type")
}
value, err := arr[0].TryBytes()
if err != nil {
return errors.New("invalid container value")
}
sig, err := arr[1].TryBytes()
if err != nil {
return errors.New("invalid container signature")
}
pub, err := arr[2].TryBytes()
if err != nil {
return errors.New("invalid container public key")
}
tok, err := arr[3].TryBytes()
if err != nil {
return errors.New("invalid container token")
}
c.Value = value
c.Signature = sig
c.PublicKey = pub
c.Token = tok
return nil
}
// ToStackItem implements stackitem.Convertible.
func (c *EACL) ToStackItem() (stackitem.Item, error) {
return stackitem.NewStruct([]stackitem.Item{
stackitem.NewByteArray(c.Value),
stackitem.NewByteArray(c.Signature),
stackitem.NewByteArray(c.PublicKey),
stackitem.NewByteArray(c.Token),
}), nil
}
// FromStackItem implements stackitem.Convertible.
func (c *EACL) FromStackItem(item stackitem.Item) error {
arr, ok := item.Value().([]stackitem.Item)
if !ok || len(arr) != 4 {
return errors.New("invalid stack item type")
}
value, err := arr[0].TryBytes()
if err != nil {
return errors.New("invalid eACL value")
}
sig, err := arr[1].TryBytes()
if err != nil {
return errors.New("invalid eACL signature")
}
pub, err := arr[2].TryBytes()
if err != nil {
return errors.New("invalid eACL public key")
}
tok, err := arr[3].TryBytes()
if err != nil {
return errors.New("invalid eACL token")
}
c.Value = value
c.Signature = sig
c.PublicKey = pub
c.Token = tok
return nil
}
// getCIDFilterFunc returns filtering function for container IDs.
// Raw byte slices are used because it works with structures returned
// from contract.
func getCIDFilterFunc(cmd *cobra.Command) (func([]byte) bool, error) {
rawIDs, err := cmd.Flags().GetStringSlice(containerIDsFlag)
if err != nil {
return nil, err
}
if len(rawIDs) == 0 {
return func([]byte) bool { return true }, nil
}
for i := range rawIDs {
err := new(cid.ID).DecodeString(rawIDs[i])
if err != nil {
return nil, fmt.Errorf("can't parse CID %s: %w", rawIDs[i], err)
}
}
sort.Strings(rawIDs)
return func(rawID []byte) bool {
var v [32]byte
copy(v[:], rawID)
var id cid.ID
id.SetSHA256(v)
idStr := id.EncodeToString()
n := sort.Search(len(rawIDs), func(i int) bool { return rawIDs[i] >= idStr })
return n < len(rawIDs) && rawIDs[n] == idStr
}, nil
}

View file

@ -0,0 +1,224 @@
package morph
import (
"encoding/json"
"fmt"
"os"
"strings"
"github.com/TrueCloudLab/frostfs-contract/nns"
"github.com/nspcc-dev/neo-go/cli/cmdargs"
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/management"
"github.com/nspcc-dev/neo-go/pkg/services/rpcsrv/params"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/opcode"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
const (
contractPathFlag = "contract"
updateFlag = "update"
customZoneFlag = "domain"
)
var deployCmd = &cobra.Command{
Use: "deploy",
Short: "Deploy additional smart-contracts",
Long: `Deploy additional smart-contract which are not related to core.
All contracts are deployed by the committee, so access to the alphabet wallets is required.
Optionally, arguments can be provided to be passed to a contract's _deploy function.
The syntax is the same as for 'neo-go contract testinvokefunction' command.
Compiled contract file name must contain '_contract.nef' suffix.
Contract's manifest file name must be 'config.json'.
NNS name is taken by stripping '_contract.nef' from the NEF file (similar to frostfs contracts).`,
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
RunE: deployContractCmd,
}
func init() {
ff := deployCmd.Flags()
ff.String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
_ = deployCmd.MarkFlagFilename(alphabetWalletsFlag)
ff.StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
ff.String(contractPathFlag, "", "Path to the contract directory")
_ = deployCmd.MarkFlagFilename(contractPathFlag)
ff.Bool(updateFlag, false, "Update an existing contract")
ff.String(customZoneFlag, "frostfs", "Custom zone for NNS")
}
func deployContractCmd(cmd *cobra.Command, args []string) error {
v := viper.GetViper()
c, err := newInitializeContext(cmd, v)
if err != nil {
return fmt.Errorf("initialization error: %w", err)
}
defer c.close()
ctrPath, _ := cmd.Flags().GetString(contractPathFlag)
ctrName, err := probeContractName(ctrPath)
if err != nil {
return err
}
cs, err := readContract(ctrPath, ctrName)
if err != nil {
return err
}
nnsCs, err := c.Client.GetContractStateByID(1)
if err != nil {
return fmt.Errorf("can't fetch NNS contract state: %w", err)
}
callHash := management.Hash
method := deployMethodName
zone, _ := cmd.Flags().GetString(customZoneFlag)
domain := ctrName + "." + zone
isUpdate, _ := cmd.Flags().GetBool(updateFlag)
if isUpdate {
cs.Hash, err = nnsResolveHash(c.ReadOnlyInvoker, nnsCs.Hash, domain)
if err != nil {
return fmt.Errorf("can't fetch contract hash from NNS: %w", err)
}
callHash = cs.Hash
method = updateMethodName
} else {
cs.Hash = state.CreateContractHash(
c.CommitteeAcc.Contract.ScriptHash(),
cs.NEF.Checksum,
cs.Manifest.Name)
}
w := io.NewBufBinWriter()
if err := emitDeploymentArguments(w.BinWriter, args); err != nil {
return err
}
emit.Bytes(w.BinWriter, cs.RawManifest)
emit.Bytes(w.BinWriter, cs.RawNEF)
emit.Int(w.BinWriter, 3)
emit.Opcodes(w.BinWriter, opcode.PACK)
emit.AppCallNoArgs(w.BinWriter, callHash, method, callflag.All)
emit.Opcodes(w.BinWriter, opcode.DROP) // contract state on stack
if !isUpdate {
bw := io.NewBufBinWriter()
emit.Instruction(bw.BinWriter, opcode.INITSSLOT, []byte{1})
emit.AppCall(bw.BinWriter, nnsCs.Hash, "getPrice", callflag.All)
emit.Opcodes(bw.BinWriter, opcode.STSFLD0)
emit.AppCall(bw.BinWriter, nnsCs.Hash, "setPrice", callflag.All, 1)
start := bw.Len()
needRecord := false
ok, err := c.nnsRootRegistered(nnsCs.Hash, zone)
if err != nil {
return err
} else if !ok {
needRecord = true
emit.AppCall(bw.BinWriter, nnsCs.Hash, "register", callflag.All,
zone, c.CommitteeAcc.Contract.ScriptHash(),
"ops@nspcc.ru", int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
emit.AppCall(bw.BinWriter, nnsCs.Hash, "register", callflag.All,
domain, c.CommitteeAcc.Contract.ScriptHash(),
"ops@nspcc.ru", int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
} else {
s, ok, err := c.nnsRegisterDomainScript(nnsCs.Hash, cs.Hash, domain)
if err != nil {
return err
}
needRecord = !ok
if len(s) != 0 {
bw.WriteBytes(s)
}
}
if needRecord {
emit.AppCall(bw.BinWriter, nnsCs.Hash, "deleteRecords", callflag.All, domain, int64(nns.TXT))
emit.AppCall(bw.BinWriter, nnsCs.Hash, "addRecord", callflag.All,
domain, int64(nns.TXT), address.Uint160ToString(cs.Hash))
}
if bw.Err != nil {
panic(fmt.Errorf("BUG: can't create deployment script: %w", w.Err))
} else if bw.Len() != start {
w.WriteBytes(bw.Bytes())
emit.Opcodes(w.BinWriter, opcode.LDSFLD0, opcode.PUSH1, opcode.PACK)
emit.AppCallNoArgs(w.BinWriter, nnsCs.Hash, "setPrice", callflag.All)
if needRecord {
c.Command.Printf("NNS: Set %s -> %s\n", domain, cs.Hash.StringLE())
}
}
}
if w.Err != nil {
panic(fmt.Errorf("BUG: can't create deployment script: %w", w.Err))
}
if err := c.sendCommitteeTx(w.Bytes(), false); err != nil {
return err
}
return c.awaitTx()
}
func emitDeploymentArguments(w *io.BinWriter, args []string) error {
_, ps, err := cmdargs.ParseParams(args, true)
if err != nil {
return err
}
if len(ps) == 0 {
emit.Opcodes(w, opcode.NEWARRAY0)
return nil
}
if len(ps) != 1 {
return fmt.Errorf("at most one argument is expected for deploy, got %d", len(ps))
}
// We could emit this directly, but round-trip through JSON is more robust.
// This a CLI, so optimizing the conversion is not worth the effort.
data, err := json.Marshal(ps)
if err != nil {
return err
}
var pp params.Params
if err := json.Unmarshal(data, &pp); err != nil {
return err
}
return params.ExpandArrayIntoScript(w, pp)
}
func probeContractName(ctrPath string) (string, error) {
ds, err := os.ReadDir(ctrPath)
if err != nil {
return "", fmt.Errorf("can't read directory: %w", err)
}
var ctrName string
for i := range ds {
if strings.HasSuffix(ds[i].Name(), "_contract.nef") {
ctrName = strings.TrimSuffix(ds[i].Name(), "_contract.nef")
break
}
}
if ctrName == "" {
return "", fmt.Errorf("can't find any NEF files in %s", ctrPath)
}
return ctrName, nil
}

View file

@ -0,0 +1,40 @@
package morph
import (
"context"
"errors"
"fmt"
"io"
"net/http"
"strings"
"github.com/google/go-github/v39/github"
"github.com/spf13/cobra"
)
func downloadContractsFromGithub(cmd *cobra.Command) (io.ReadCloser, error) {
gcl := github.NewClient(nil)
release, _, err := gcl.Repositories.GetLatestRelease(context.Background(), "nspcc-dev", "frostfs-contract")
if err != nil {
return nil, fmt.Errorf("can't fetch release info: %w", err)
}
cmd.Printf("Found %s (%s), downloading...\n", release.GetTagName(), release.GetName())
var url string
for _, a := range release.Assets {
if strings.HasPrefix(a.GetName(), "frostfs-contract") {
url = a.GetBrowserDownloadURL()
break
}
}
if url == "" {
return nil, errors.New("can't find contracts archive in release assets")
}
resp, err := http.Get(url)
if err != nil {
return nil, fmt.Errorf("can't fetch contracts archive: %w", err)
}
return resp.Body, nil
}

View file

@ -0,0 +1,250 @@
package morph
import (
"bytes"
"errors"
"fmt"
"strings"
"text/tabwriter"
"github.com/TrueCloudLab/frostfs-contract/nns"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/opcode"
"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
"github.com/nspcc-dev/neo-go/pkg/vm/vmstate"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
const lastGlagoliticLetter = 41
type contractDumpInfo struct {
hash util.Uint160
name string
version string
}
func dumpContractHashes(cmd *cobra.Command, _ []string) error {
c, err := getN3Client(viper.GetViper())
if err != nil {
return fmt.Errorf("can't create N3 client: %w", err)
}
cs, err := c.GetContractStateByID(1)
if err != nil {
return err
}
zone, _ := cmd.Flags().GetString(customZoneFlag)
if zone != "" {
return dumpCustomZoneHashes(cmd, cs.Hash, zone, c)
}
infos := []contractDumpInfo{{name: nnsContract, hash: cs.Hash}}
irSize := 0
for ; irSize < lastGlagoliticLetter; irSize++ {
ok, err := nnsIsAvailable(c, cs.Hash, getAlphabetNNSDomain(irSize))
if err != nil {
return err
} else if ok {
break
}
}
bw := io.NewBufBinWriter()
if irSize != 0 {
bw.Reset()
for i := 0; i < irSize; i++ {
emit.AppCall(bw.BinWriter, cs.Hash, "resolve", callflag.ReadOnly,
getAlphabetNNSDomain(i),
int64(nns.TXT))
}
alphaRes, err := c.InvokeScript(bw.Bytes(), nil)
if err != nil {
return fmt.Errorf("can't fetch info from NNS: %w", err)
}
for i := 0; i < irSize; i++ {
info := contractDumpInfo{name: fmt.Sprintf("alphabet %d", i)}
if h, err := parseNNSResolveResult(alphaRes.Stack[i]); err == nil {
info.hash = h
}
infos = append(infos, info)
}
}
for _, ctrName := range contractList {
bw.Reset()
emit.AppCall(bw.BinWriter, cs.Hash, "resolve", callflag.ReadOnly,
ctrName+".frostfs", int64(nns.TXT))
res, err := c.InvokeScript(bw.Bytes(), nil)
if err != nil {
return fmt.Errorf("can't fetch info from NNS: %w", err)
}
info := contractDumpInfo{name: ctrName}
if len(res.Stack) != 0 {
if h, err := parseNNSResolveResult(res.Stack[0]); err == nil {
info.hash = h
}
}
infos = append(infos, info)
}
fillContractVersion(cmd, c, infos)
printContractInfo(cmd, infos)
return nil
}
func dumpCustomZoneHashes(cmd *cobra.Command, nnsHash util.Uint160, zone string, c Client) error {
const nnsMaxTokens = 100
inv := invoker.New(c, nil)
if !strings.HasPrefix(zone, ".") {
zone = "." + zone
}
var infos []contractDumpInfo
processItem := func(item stackitem.Item) {
bs, err := item.TryBytes()
if err != nil {
cmd.PrintErrf("Invalid NNS record: %v\n", err)
return
}
if !bytes.HasSuffix(bs, []byte(zone)) {
// Related https://github.com/nspcc-dev/neofs-contract/issues/316.
return
}
h, err := nnsResolveHash(inv, nnsHash, string(bs))
if err != nil {
cmd.PrintErrf("Could not resolve name %s: %v\n", string(bs), err)
return
}
infos = append(infos, contractDumpInfo{
hash: h,
name: strings.TrimSuffix(string(bs), zone),
})
}
sessionID, iter, err := unwrap.SessionIterator(inv.Call(nnsHash, "tokens"))
if err != nil {
if errors.Is(err, unwrap.ErrNoSessionID) {
items, err := unwrap.Array(inv.CallAndExpandIterator(nnsHash, "tokens", nnsMaxTokens))
if err != nil {
return fmt.Errorf("can't get a list of NNS domains: %w", err)
}
if len(items) == nnsMaxTokens {
cmd.PrintErrln("Provided RPC endpoint doesn't support sessions, some hashes might be lost.")
}
for i := range items {
processItem(items[i])
}
} else {
return err
}
} else {
defer func() {
_ = inv.TerminateSession(sessionID)
}()
items, err := inv.TraverseIterator(sessionID, &iter, nnsMaxTokens)
for err == nil && len(items) != 0 {
for i := range items {
processItem(items[i])
}
items, err = inv.TraverseIterator(sessionID, &iter, nnsMaxTokens)
}
if err != nil {
return fmt.Errorf("error during NNS domains iteration: %w", err)
}
}
fillContractVersion(cmd, c, infos)
printContractInfo(cmd, infos)
return nil
}
func parseContractVersion(item stackitem.Item) string {
bi, err := item.TryInteger()
if err != nil || bi.Sign() == 0 || !bi.IsInt64() {
return "unknown"
}
v := bi.Int64()
major := v / 1_000_000
minor := (v % 1_000_000) / 1000
patch := v % 1_000
return fmt.Sprintf("v%d.%d.%d", major, minor, patch)
}
func printContractInfo(cmd *cobra.Command, infos []contractDumpInfo) {
if len(infos) == 0 {
return
}
buf := bytes.NewBuffer(nil)
tw := tabwriter.NewWriter(buf, 0, 2, 2, ' ', 0)
for _, info := range infos {
if info.version == "" {
info.version = "unknown"
}
_, _ = tw.Write([]byte(fmt.Sprintf("%s\t(%s):\t%s\n",
info.name, info.version, info.hash.StringLE())))
}
_ = tw.Flush()
cmd.Print(buf.String())
}
func fillContractVersion(cmd *cobra.Command, c Client, infos []contractDumpInfo) {
bw := io.NewBufBinWriter()
sub := io.NewBufBinWriter()
for i := range infos {
if infos[i].hash.Equals(util.Uint160{}) {
emit.Int(bw.BinWriter, 0)
} else {
sub.Reset()
emit.AppCall(sub.BinWriter, infos[i].hash, "version", callflag.NoneFlag)
if sub.Err != nil {
panic(fmt.Errorf("BUG: can't create version script: %w", bw.Err))
}
script := sub.Bytes()
emit.Instruction(bw.BinWriter, opcode.TRY, []byte{byte(3 + len(script) + 2), 0})
bw.BinWriter.WriteBytes(script)
emit.Instruction(bw.BinWriter, opcode.ENDTRY, []byte{2 + 1})
emit.Opcodes(bw.BinWriter, opcode.PUSH0)
}
}
emit.Opcodes(bw.BinWriter, opcode.NOP) // for the last ENDTRY target
if bw.Err != nil {
panic(fmt.Errorf("BUG: can't create version script: %w", bw.Err))
}
res, err := c.InvokeScript(bw.Bytes(), nil)
if err != nil {
cmd.Printf("Can't fetch version from NNS: %v\n", err)
return
}
if res.State == vmstate.Halt.String() {
for i := range res.Stack {
infos[i].version = parseContractVersion(res.Stack[i])
}
}
}

View file

@ -0,0 +1,57 @@
package morph
import (
"errors"
"fmt"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
func forceNewEpochCmd(cmd *cobra.Command, args []string) error {
wCtx, err := newInitializeContext(cmd, viper.GetViper())
if err != nil {
return fmt.Errorf("can't to initialize context: %w", err)
}
cs, err := wCtx.Client.GetContractStateByID(1)
if err != nil {
return fmt.Errorf("can't get NNS contract info: %w", err)
}
nmHash, err := nnsResolveHash(wCtx.ReadOnlyInvoker, cs.Hash, netmapContract+".frostfs")
if err != nil {
return fmt.Errorf("can't get netmap contract hash: %w", err)
}
bw := io.NewBufBinWriter()
if err := emitNewEpochCall(bw, wCtx, nmHash); err != nil {
return err
}
if err := wCtx.sendConsensusTx(bw.Bytes()); err != nil {
return err
}
return wCtx.awaitTx()
}
func emitNewEpochCall(bw *io.BufBinWriter, wCtx *initializeContext, nmHash util.Uint160) error {
curr, err := unwrap.Int64(wCtx.ReadOnlyInvoker.Call(nmHash, "epoch"))
if err != nil {
return errors.New("can't fetch current epoch from the netmap contract")
}
newEpoch := curr + 1
wCtx.Command.Printf("Current epoch: %d, increase to %d.\n", curr, newEpoch)
// In NeoFS this is done via Notary contract. Here, however, we can form the
// transaction locally.
emit.AppCall(bw.BinWriter, nmHash, "newEpoch", callflag.All, newEpoch)
return bw.Err
}

View file

@ -0,0 +1,225 @@
package morph
import (
"errors"
"fmt"
"os"
"path/filepath"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config"
"github.com/TrueCloudLab/frostfs-node/pkg/innerring"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/nspcc-dev/neo-go/pkg/encoding/fixedn"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/gas"
"github.com/nspcc-dev/neo-go/pkg/smartcontract"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/opcode"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
const (
singleAccountName = "single"
committeeAccountName = "committee"
consensusAccountName = "consensus"
)
func generateAlphabetCreds(cmd *cobra.Command, args []string) error {
// alphabet size is not part of the config
size, err := cmd.Flags().GetUint(alphabetSizeFlag)
if err != nil {
return err
}
if size == 0 {
return errors.New("size must be > 0")
}
v := viper.GetViper()
walletDir := config.ResolveHomePath(viper.GetString(alphabetWalletsFlag))
pwds, err := initializeWallets(v, walletDir, int(size))
if err != nil {
return err
}
_, err = initializeContractWallet(v, walletDir)
if err != nil {
return err
}
cmd.Println("size:", size)
cmd.Println("alphabet-wallets:", walletDir)
for i := range pwds {
cmd.Printf("wallet[%d]: %s\n", i, pwds[i])
}
return nil
}
func initializeWallets(v *viper.Viper, walletDir string, size int) ([]string, error) {
wallets := make([]*wallet.Wallet, size)
pubs := make(keys.PublicKeys, size)
passwords := make([]string, size)
for i := range wallets {
password, err := config.GetPassword(v, innerring.GlagoliticLetter(i).String())
if err != nil {
return nil, fmt.Errorf("can't fetch password: %w", err)
}
p := filepath.Join(walletDir, innerring.GlagoliticLetter(i).String()+".json")
f, err := os.OpenFile(p, os.O_CREATE, 0644)
if err != nil {
return nil, fmt.Errorf("can't create wallet file: %w", err)
}
if err := f.Close(); err != nil {
return nil, fmt.Errorf("can't close wallet file: %w", err)
}
w, err := wallet.NewWallet(p)
if err != nil {
return nil, fmt.Errorf("can't create wallet: %w", err)
}
if err := w.CreateAccount(singleAccountName, password); err != nil {
return nil, fmt.Errorf("can't create account: %w", err)
}
passwords[i] = password
wallets[i] = w
pubs[i] = w.Accounts[0].PrivateKey().PublicKey()
}
// Create committee account with N/2+1 multi-signature.
majCount := smartcontract.GetMajorityHonestNodeCount(size)
for i, w := range wallets {
if err := addMultisigAccount(w, majCount, committeeAccountName, passwords[i], pubs); err != nil {
return nil, fmt.Errorf("can't create committee account: %w", err)
}
}
// Create consensus account with 2*N/3+1 multi-signature.
bftCount := smartcontract.GetDefaultHonestNodeCount(size)
for i, w := range wallets {
if err := addMultisigAccount(w, bftCount, consensusAccountName, passwords[i], pubs); err != nil {
return nil, fmt.Errorf("can't create consensus account: %w", err)
}
}
for _, w := range wallets {
if err := w.SavePretty(); err != nil {
return nil, fmt.Errorf("can't save wallet: %w", err)
}
}
return passwords, nil
}
func addMultisigAccount(w *wallet.Wallet, m int, name, password string, pubs keys.PublicKeys) error {
acc := wallet.NewAccountFromPrivateKey(w.Accounts[0].PrivateKey())
acc.Label = name
if err := acc.ConvertMultisig(m, pubs); err != nil {
return err
}
if err := acc.Encrypt(password, keys.NEP2ScryptParams()); err != nil {
return err
}
w.AddAccount(acc)
return nil
}
func generateStorageCreds(cmd *cobra.Command, _ []string) error {
return refillGas(cmd, storageGasConfigFlag, true)
}
func refillGas(cmd *cobra.Command, gasFlag string, createWallet bool) (err error) {
// storage wallet path is not part of the config
storageWalletPath, _ := cmd.Flags().GetString(storageWalletFlag)
// wallet address is not part of the config
walletAddress, _ := cmd.Flags().GetString(walletAddressFlag)
var gasReceiver util.Uint160
if len(walletAddress) != 0 {
gasReceiver, err = address.StringToUint160(walletAddress)
if err != nil {
return fmt.Errorf("invalid wallet address %s: %w", walletAddress, err)
}
} else {
if storageWalletPath == "" {
return fmt.Errorf("missing wallet path (use '--%s <out.json>')", storageWalletFlag)
}
var w *wallet.Wallet
if createWallet {
w, err = wallet.NewWallet(storageWalletPath)
} else {
w, err = wallet.NewWalletFromFile(storageWalletPath)
}
if err != nil {
return fmt.Errorf("can't create wallet: %w", err)
}
if createWallet {
var password string
label, _ := cmd.Flags().GetString(storageWalletLabelFlag)
password, err := config.GetStoragePassword(viper.GetViper(), label)
if err != nil {
return fmt.Errorf("can't fetch password: %w", err)
}
if label == "" {
label = singleAccountName
}
if err := w.CreateAccount(label, password); err != nil {
return fmt.Errorf("can't create account: %w", err)
}
}
gasReceiver = w.Accounts[0].Contract.ScriptHash()
}
gasStr := viper.GetString(gasFlag)
gasAmount, err := parseGASAmount(gasStr)
if err != nil {
return err
}
wCtx, err := newInitializeContext(cmd, viper.GetViper())
if err != nil {
return err
}
bw := io.NewBufBinWriter()
emit.AppCall(bw.BinWriter, gas.Hash, "transfer", callflag.All,
wCtx.CommitteeAcc.Contract.ScriptHash(), gasReceiver, int64(gasAmount), nil)
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
if bw.Err != nil {
return fmt.Errorf("BUG: invalid transfer arguments: %w", bw.Err)
}
if err := wCtx.sendCommitteeTx(bw.Bytes(), false); err != nil {
return err
}
return wCtx.awaitTx()
}
func parseGASAmount(s string) (fixedn.Fixed8, error) {
gasAmount, err := fixedn.Fixed8FromString(s)
if err != nil {
return 0, fmt.Errorf("invalid GAS amount %s: %w", s, err)
}
if gasAmount <= 0 {
return 0, fmt.Errorf("GAS amount must be positive (got %d)", gasAmount)
}
return gasAmount, nil
}

View file

@ -0,0 +1,112 @@
package morph
import (
"bytes"
"io"
"math/rand"
"os"
"path/filepath"
"strconv"
"testing"
"github.com/TrueCloudLab/frostfs-node/pkg/innerring"
"github.com/nspcc-dev/neo-go/cli/input"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/smartcontract"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/viper"
"github.com/stretchr/testify/require"
"golang.org/x/term"
)
const testContractPassword = "grouppass"
func TestGenerateAlphabet(t *testing.T) {
const size = 4
walletDir := t.TempDir()
buf := setupTestTerminal(t)
cmd := generateAlphabetCmd
v := viper.GetViper()
t.Run("zero size", func(t *testing.T) {
buf.Reset()
v.Set(alphabetWalletsFlag, walletDir)
require.NoError(t, cmd.Flags().Set(alphabetSizeFlag, "0"))
buf.WriteString("pass\r")
require.Error(t, generateAlphabetCreds(cmd, nil))
})
t.Run("no password provided", func(t *testing.T) {
buf.Reset()
v.Set(alphabetWalletsFlag, walletDir)
require.NoError(t, cmd.Flags().Set(alphabetSizeFlag, "1"))
require.Error(t, generateAlphabetCreds(cmd, nil))
})
t.Run("missing directory", func(t *testing.T) {
buf.Reset()
dir := filepath.Join(os.TempDir(), "notexist."+strconv.FormatUint(rand.Uint64(), 10))
v.Set(alphabetWalletsFlag, dir)
require.NoError(t, cmd.Flags().Set(alphabetSizeFlag, "1"))
buf.WriteString("pass\r")
require.Error(t, generateAlphabetCreds(cmd, nil))
})
t.Run("no password for contract group wallet", func(t *testing.T) {
buf.Reset()
v.Set(alphabetWalletsFlag, walletDir)
require.NoError(t, cmd.Flags().Set(alphabetSizeFlag, strconv.FormatUint(size, 10)))
for i := uint64(0); i < size; i++ {
buf.WriteString(strconv.FormatUint(i, 10) + "\r")
}
require.Error(t, generateAlphabetCreds(cmd, nil))
})
buf.Reset()
v.Set(alphabetWalletsFlag, walletDir)
require.NoError(t, generateAlphabetCmd.Flags().Set(alphabetSizeFlag, strconv.FormatUint(size, 10)))
for i := uint64(0); i < size; i++ {
buf.WriteString(strconv.FormatUint(i, 10) + "\r")
}
buf.WriteString(testContractPassword + "\r")
require.NoError(t, generateAlphabetCreds(generateAlphabetCmd, nil))
for i := uint64(0); i < size; i++ {
p := filepath.Join(walletDir, innerring.GlagoliticLetter(i).String()+".json")
w, err := wallet.NewWalletFromFile(p)
require.NoError(t, err, "wallet doesn't exist")
require.Equal(t, 3, len(w.Accounts), "not all accounts were created")
for _, a := range w.Accounts {
err := a.Decrypt(strconv.FormatUint(i, 10), keys.NEP2ScryptParams())
require.NoError(t, err, "can't decrypt account")
switch a.Label {
case consensusAccountName:
require.Equal(t, smartcontract.GetDefaultHonestNodeCount(size), len(a.Contract.Parameters))
case committeeAccountName:
require.Equal(t, smartcontract.GetMajorityHonestNodeCount(size), len(a.Contract.Parameters))
default:
require.Equal(t, singleAccountName, a.Label)
}
}
}
t.Run("check contract group wallet", func(t *testing.T) {
p := filepath.Join(walletDir, contractWalletFilename)
w, err := wallet.NewWalletFromFile(p)
require.NoError(t, err, "contract wallet doesn't exist")
require.Equal(t, 1, len(w.Accounts), "contract wallet must have 1 accout")
require.NoError(t, w.Accounts[0].Decrypt(testContractPassword, keys.NEP2ScryptParams()))
})
}
func setupTestTerminal(t *testing.T) *bytes.Buffer {
in := bytes.NewBuffer(nil)
input.Terminal = term.NewTerminal(input.ReadWriter{
Reader: in,
Writer: io.Discard,
}, "")
t.Cleanup(func() { input.Terminal = nil })
return in
}

View file

@ -0,0 +1,106 @@
package morph
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/manifest"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
const (
contractWalletFilename = "contract.json"
contractWalletPasswordKey = "contract"
)
func initializeContractWallet(v *viper.Viper, walletDir string) (*wallet.Wallet, error) {
password, err := config.GetPassword(v, contractWalletPasswordKey)
if err != nil {
return nil, err
}
w, err := wallet.NewWallet(filepath.Join(walletDir, contractWalletFilename))
if err != nil {
return nil, err
}
acc, err := wallet.NewAccount()
if err != nil {
return nil, err
}
err = acc.Encrypt(password, keys.NEP2ScryptParams())
if err != nil {
return nil, err
}
w.AddAccount(acc)
if err := w.SavePretty(); err != nil {
return nil, err
}
return w, nil
}
func openContractWallet(v *viper.Viper, cmd *cobra.Command, walletDir string) (*wallet.Wallet, error) {
p := filepath.Join(walletDir, contractWalletFilename)
w, err := wallet.NewWalletFromFile(p)
if err != nil {
if !os.IsNotExist(err) {
return nil, fmt.Errorf("can't open wallet: %w", err)
}
cmd.Printf("Contract group wallet is missing, initialize at %s\n", p)
return initializeContractWallet(v, walletDir)
}
password, err := config.GetPassword(v, contractWalletPasswordKey)
if err != nil {
return nil, err
}
for i := range w.Accounts {
if err := w.Accounts[i].Decrypt(password, keys.NEP2ScryptParams()); err != nil {
return nil, fmt.Errorf("can't unlock wallet: %w", err)
}
}
return w, nil
}
func (c *initializeContext) addManifestGroup(h util.Uint160, cs *contractState) error {
priv := c.ContractWallet.Accounts[0].PrivateKey()
pub := priv.PublicKey()
sig := priv.Sign(h.BytesBE())
found := false
for i := range cs.Manifest.Groups {
if cs.Manifest.Groups[i].PublicKey.Equal(pub) {
cs.Manifest.Groups[i].Signature = sig
found = true
break
}
}
if !found {
cs.Manifest.Groups = append(cs.Manifest.Groups, manifest.Group{
PublicKey: pub,
Signature: sig,
})
}
data, err := json.Marshal(cs.Manifest)
if err != nil {
return err
}
cs.RawManifest = data
return nil
}

View file

@ -0,0 +1,465 @@
package morph
import (
"errors"
"fmt"
"os"
"path/filepath"
"time"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config"
"github.com/TrueCloudLab/frostfs-node/pkg/innerring"
morphClient "github.com/TrueCloudLab/frostfs-node/pkg/morph/client"
"github.com/nspcc-dev/neo-go/pkg/core/native/nativenames"
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/actor"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/trigger"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm/vmstate"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
type cache struct {
nnsCs *state.Contract
groupKey *keys.PublicKey
}
type initializeContext struct {
clientContext
cache
// CommitteeAcc is used for retrieving the committee address and the verification script.
CommitteeAcc *wallet.Account
// ConsensusAcc is used for retrieving the committee address and the verification script.
ConsensusAcc *wallet.Account
Wallets []*wallet.Wallet
// ContractWallet is a wallet for providing the contract group signature.
ContractWallet *wallet.Wallet
// Accounts contains simple signature accounts in the same order as in Wallets.
Accounts []*wallet.Account
Contracts map[string]*contractState
Command *cobra.Command
ContractPath string
}
func initializeSideChainCmd(cmd *cobra.Command, args []string) error {
initCtx, err := newInitializeContext(cmd, viper.GetViper())
if err != nil {
return fmt.Errorf("initialization error: %w", err)
}
defer initCtx.close()
// 1. Transfer funds to committee accounts.
cmd.Println("Stage 1: transfer GAS to alphabet nodes.")
if err := initCtx.transferFunds(); err != nil {
return err
}
cmd.Println("Stage 2: set notary and alphabet nodes in designate contract.")
if err := initCtx.setNotaryAndAlphabetNodes(); err != nil {
return err
}
// 3. Deploy NNS contract.
cmd.Println("Stage 3: deploy NNS contract.")
if err := initCtx.deployNNS(deployMethodName); err != nil {
return err
}
// 4. Deploy NeoFS contracts.
cmd.Println("Stage 4: deploy NeoFS contracts.")
if err := initCtx.deployContracts(); err != nil {
return err
}
cmd.Println("Stage 4.1: Transfer GAS to proxy contract.")
if err := initCtx.transferGASToProxy(); err != nil {
return err
}
cmd.Println("Stage 5: register candidates.")
if err := initCtx.registerCandidates(); err != nil {
return err
}
cmd.Println("Stage 6: transfer NEO to alphabet contracts.")
if err := initCtx.transferNEOToAlphabetContracts(); err != nil {
return err
}
cmd.Println("Stage 7: set addresses in NNS.")
if err := initCtx.setNNS(); err != nil {
return err
}
return nil
}
func (c *initializeContext) close() {
if local, ok := c.Client.(*localClient); ok {
err := local.dump()
if err != nil {
c.Command.PrintErrf("Can't write dump: %v\n", err)
os.Exit(1)
}
}
}
func newInitializeContext(cmd *cobra.Command, v *viper.Viper) (*initializeContext, error) {
walletDir := config.ResolveHomePath(viper.GetString(alphabetWalletsFlag))
wallets, err := openAlphabetWallets(v, walletDir)
if err != nil {
return nil, err
}
needContracts := cmd.Name() == "update-contracts" || cmd.Name() == "init"
var w *wallet.Wallet
if needContracts {
w, err = openContractWallet(v, cmd, walletDir)
if err != nil {
return nil, err
}
}
var c Client
if v.GetString(localDumpFlag) != "" {
if v.GetString(endpointFlag) != "" {
return nil, fmt.Errorf("`%s` and `%s` flags are mutually exclusive", endpointFlag, localDumpFlag)
}
c, err = newLocalClient(cmd, v, wallets)
} else {
c, err = getN3Client(v)
}
if err != nil {
return nil, fmt.Errorf("can't create N3 client: %w", err)
}
committeeAcc, err := getWalletAccount(wallets[0], committeeAccountName)
if err != nil {
return nil, fmt.Errorf("can't find committee account: %w", err)
}
consensusAcc, err := getWalletAccount(wallets[0], consensusAccountName)
if err != nil {
return nil, fmt.Errorf("can't find consensus account: %w", err)
}
var ctrPath string
if cmd.Name() == "init" {
if viper.GetInt64(epochDurationInitFlag) <= 0 {
return nil, fmt.Errorf("epoch duration must be positive")
}
if viper.GetInt64(maxObjectSizeInitFlag) <= 0 {
return nil, fmt.Errorf("max object size must be positive")
}
}
if needContracts {
ctrPath, err = cmd.Flags().GetString(contractsInitFlag)
if err != nil {
return nil, fmt.Errorf("invalid contracts path: %w", err)
}
}
if err := checkNotaryEnabled(c); err != nil {
return nil, err
}
accounts := make([]*wallet.Account, len(wallets))
for i, w := range wallets {
acc, err := getWalletAccount(w, singleAccountName)
if err != nil {
return nil, fmt.Errorf("wallet %s is invalid (no single account): %w", w.Path(), err)
}
accounts[i] = acc
}
cliCtx, err := defaultClientContext(c, committeeAcc)
if err != nil {
return nil, fmt.Errorf("client context: %w", err)
}
initCtx := &initializeContext{
clientContext: *cliCtx,
ConsensusAcc: consensusAcc,
CommitteeAcc: committeeAcc,
ContractWallet: w,
Wallets: wallets,
Accounts: accounts,
Command: cmd,
Contracts: make(map[string]*contractState),
ContractPath: ctrPath,
}
if needContracts {
err := initCtx.readContracts(fullContractList)
if err != nil {
return nil, err
}
}
return initCtx, nil
}
func openAlphabetWallets(v *viper.Viper, walletDir string) ([]*wallet.Wallet, error) {
walletFiles, err := os.ReadDir(walletDir)
if err != nil {
return nil, fmt.Errorf("can't read alphabet wallets dir: %w", err)
}
var size int
loop:
for i := 0; i < len(walletFiles); i++ {
name := innerring.GlagoliticLetter(i).String() + ".json"
for j := range walletFiles {
if walletFiles[j].Name() == name {
size++
continue loop
}
}
break
}
if size == 0 {
return nil, errors.New("alphabet wallets dir is empty (run `generate-alphabet` command first)")
}
wallets := make([]*wallet.Wallet, size)
for i := 0; i < size; i++ {
letter := innerring.GlagoliticLetter(i).String()
p := filepath.Join(walletDir, letter+".json")
w, err := wallet.NewWalletFromFile(p)
if err != nil {
return nil, fmt.Errorf("can't open wallet: %w", err)
}
password, err := config.GetPassword(v, letter)
if err != nil {
return nil, fmt.Errorf("can't fetch password: %w", err)
}
for i := range w.Accounts {
if err := w.Accounts[i].Decrypt(password, keys.NEP2ScryptParams()); err != nil {
return nil, fmt.Errorf("can't unlock wallet: %w", err)
}
}
wallets[i] = w
}
return wallets, nil
}
func (c *initializeContext) awaitTx() error {
return c.clientContext.awaitTx(c.Command)
}
func (c *initializeContext) nnsContractState() (*state.Contract, error) {
if c.nnsCs != nil {
return c.nnsCs, nil
}
cs, err := c.Client.GetContractStateByID(1)
if err != nil {
return nil, err
}
c.nnsCs = cs
return cs, nil
}
func (c *initializeContext) getSigner(tryGroup bool, acc *wallet.Account) transaction.Signer {
if tryGroup && c.groupKey != nil {
return transaction.Signer{
Account: acc.Contract.ScriptHash(),
Scopes: transaction.CustomGroups,
AllowedGroups: keys.PublicKeys{c.groupKey},
}
}
signer := transaction.Signer{
Account: acc.Contract.ScriptHash(),
Scopes: transaction.Global, // Scope is important, as we have nested call to container contract.
}
if !tryGroup {
return signer
}
nnsCs, err := c.nnsContractState()
if err != nil {
return signer
}
groupKey, err := nnsResolveKey(c.ReadOnlyInvoker, nnsCs.Hash, morphClient.NNSGroupKeyName)
if err == nil {
c.groupKey = groupKey
signer.Scopes = transaction.CustomGroups
signer.AllowedGroups = keys.PublicKeys{groupKey}
}
return signer
}
func (c *clientContext) awaitTx(cmd *cobra.Command) error {
if len(c.SentTxs) == 0 {
return nil
}
if local, ok := c.Client.(*localClient); ok {
if err := local.putTransactions(); err != nil {
return fmt.Errorf("can't persist transactions: %w", err)
}
}
err := awaitTx(cmd, c.Client, c.SentTxs)
c.SentTxs = c.SentTxs[:0]
return err
}
func awaitTx(cmd *cobra.Command, c Client, txs []hashVUBPair) error {
cmd.Println("Waiting for transactions to persist...")
const pollInterval = time.Second
tick := time.NewTicker(pollInterval)
defer tick.Stop()
at := trigger.Application
var retErr error
currBlock, err := c.GetBlockCount()
if err != nil {
return fmt.Errorf("can't fetch current block height: %w", err)
}
loop:
for i := range txs {
res, err := c.GetApplicationLog(txs[i].hash, &at)
if err == nil {
if retErr == nil && len(res.Executions) > 0 && res.Executions[0].VMState != vmstate.Halt {
retErr = fmt.Errorf("tx %d persisted in %s state: %s",
i, res.Executions[0].VMState, res.Executions[0].FaultException)
}
continue loop
}
if txs[i].vub < currBlock {
return fmt.Errorf("tx was not persisted: vub=%d, height=%d", txs[i].vub, currBlock)
}
for range tick.C {
// We must fetch current height before application log, to avoid race condition.
currBlock, err = c.GetBlockCount()
if err != nil {
return fmt.Errorf("can't fetch current block height: %w", err)
}
res, err := c.GetApplicationLog(txs[i].hash, &at)
if err == nil {
if retErr == nil && len(res.Executions) > 0 && res.Executions[0].VMState != vmstate.Halt {
retErr = fmt.Errorf("tx %d persisted in %s state: %s",
i, res.Executions[0].VMState, res.Executions[0].FaultException)
}
continue loop
}
if txs[i].vub < currBlock {
return fmt.Errorf("tx was not persisted: vub=%d, height=%d", txs[i].vub, currBlock)
}
}
}
return retErr
}
// sendCommitteeTx creates transaction from script, signs it by committee nodes and sends it to RPC.
// If tryGroup is false, global scope is used for the signer (useful when
// working with native contracts).
func (c *initializeContext) sendCommitteeTx(script []byte, tryGroup bool) error {
return c.sendMultiTx(script, tryGroup, false)
}
// sendConsensusTx creates transaction from script, signs it by alphabet nodes and sends it to RPC.
// Not that because this is used only after the contracts were initialized and deployed,
// we always try to have a group scope.
func (c *initializeContext) sendConsensusTx(script []byte) error {
return c.sendMultiTx(script, true, true)
}
func (c *initializeContext) sendMultiTx(script []byte, tryGroup bool, withConsensus bool) error {
var act *actor.Actor
var err error
withConsensus = withConsensus && !c.ConsensusAcc.Contract.ScriptHash().Equals(c.CommitteeAcc.ScriptHash())
if tryGroup {
// Even for consensus signatures we need the committee to pay.
signers := make([]actor.SignerAccount, 1, 2)
signers[0] = actor.SignerAccount{
Signer: c.getSigner(tryGroup, c.CommitteeAcc),
Account: c.CommitteeAcc,
}
if withConsensus {
signers = append(signers, actor.SignerAccount{
Signer: c.getSigner(tryGroup, c.ConsensusAcc),
Account: c.ConsensusAcc,
})
}
act, err = actor.New(c.Client, signers)
} else {
if withConsensus {
panic("BUG: should never happen")
}
act, err = c.CommitteeAct, nil
}
if err != nil {
return fmt.Errorf("could not create actor: %w", err)
}
tx, err := act.MakeUnsignedRun(script, []transaction.Attribute{{Type: transaction.HighPriority}})
if err != nil {
return fmt.Errorf("could not perform test invocation: %w", err)
}
if err := c.multiSign(tx, committeeAccountName); err != nil {
return err
}
if withConsensus {
if err := c.multiSign(tx, consensusAccountName); err != nil {
return err
}
}
return c.sendTx(tx, c.Command, false)
}
func getWalletAccount(w *wallet.Wallet, typ string) (*wallet.Account, error) {
for i := range w.Accounts {
if w.Accounts[i].Label == typ {
return w.Accounts[i], nil
}
}
return nil, fmt.Errorf("account for '%s' not found", typ)
}
func checkNotaryEnabled(c Client) error {
ns, err := c.GetNativeContracts()
if err != nil {
return fmt.Errorf("can't get native contract hashes: %w", err)
}
notaryEnabled := false
nativeHashes := make(map[string]util.Uint160, len(ns))
for i := range ns {
if ns[i].Manifest.Name == nativenames.Notary {
notaryEnabled = len(ns[i].UpdateHistory) > 0
}
nativeHashes[ns[i].Manifest.Name] = ns[i].Hash
}
if !notaryEnabled {
return errors.New("notary contract must be enabled")
}
return nil
}

View file

@ -0,0 +1,592 @@
package morph
import (
"archive/tar"
"compress/gzip"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"github.com/TrueCloudLab/frostfs-contract/common"
"github.com/TrueCloudLab/frostfs-contract/nns"
"github.com/TrueCloudLab/frostfs-node/pkg/innerring"
morphClient "github.com/TrueCloudLab/frostfs-node/pkg/morph/client"
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
io2 "github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/actor"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/management"
"github.com/nspcc-dev/neo-go/pkg/smartcontract"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/manifest"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/nef"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/opcode"
"github.com/nspcc-dev/neo-go/pkg/vm/vmstate"
"github.com/spf13/viper"
)
const (
nnsContract = "nns"
frostfsContract = "frostfs" // not deployed in side-chain.
processingContract = "processing" // not deployed in side-chain.
alphabetContract = "alphabet"
auditContract = "audit"
balanceContract = "balance"
containerContract = "container"
frostfsIDContract = "frostfsid"
netmapContract = "netmap"
proxyContract = "proxy"
reputationContract = "reputation"
subnetContract = "subnet"
)
const (
netmapEpochKey = "EpochDuration"
netmapMaxObjectSizeKey = "MaxObjectSize"
netmapAuditFeeKey = "AuditFee"
netmapContainerFeeKey = "ContainerFee"
netmapContainerAliasFeeKey = "ContainerAliasFee"
netmapEigenTrustIterationsKey = "EigenTrustIterations"
netmapEigenTrustAlphaKey = "EigenTrustAlpha"
netmapBasicIncomeRateKey = "BasicIncomeRate"
netmapInnerRingCandidateFeeKey = "InnerRingCandidateFee"
netmapWithdrawFeeKey = "WithdrawFee"
netmapHomomorphicHashDisabledKey = "HomomorphicHashingDisabled"
netmapMaintenanceAllowedKey = "MaintenanceModeAllowed"
defaultEigenTrustIterations = 4
defaultEigenTrustAlpha = "0.1"
)
var (
contractList = []string{
auditContract,
balanceContract,
containerContract,
frostfsIDContract,
netmapContract,
proxyContract,
reputationContract,
subnetContract,
}
fullContractList = append([]string{
frostfsContract,
processingContract,
nnsContract,
alphabetContract,
}, contractList...)
)
type contractState struct {
NEF *nef.File
RawNEF []byte
Manifest *manifest.Manifest
RawManifest []byte
Hash util.Uint160
}
const (
updateMethodName = "update"
deployMethodName = "deploy"
)
func (c *initializeContext) deployNNS(method string) error {
cs := c.getContract(nnsContract)
h := cs.Hash
nnsCs, err := c.nnsContractState()
if err == nil {
if nnsCs.NEF.Checksum == cs.NEF.Checksum {
if method == deployMethodName {
c.Command.Println("NNS contract is already deployed.")
} else {
c.Command.Println("NNS contract is already updated.")
}
return nil
}
h = nnsCs.Hash
}
err = c.addManifestGroup(h, cs)
if err != nil {
return fmt.Errorf("can't sign manifest group: %v", err)
}
params := getContractDeployParameters(cs, nil)
signer := transaction.Signer{
Account: c.CommitteeAcc.Contract.ScriptHash(),
Scopes: transaction.CalledByEntry,
}
invokeHash := management.Hash
if method == updateMethodName {
invokeHash = nnsCs.Hash
}
res, err := invokeFunction(c.Client, invokeHash, method, params, []transaction.Signer{signer})
if err != nil {
return fmt.Errorf("can't deploy NNS contract: %w", err)
}
if res.State != vmstate.Halt.String() {
return fmt.Errorf("can't deploy NNS contract: %s", res.FaultException)
}
tx, err := c.Client.CreateTxFromScript(res.Script, c.CommitteeAcc, res.GasConsumed, 0, []rpcclient.SignerAccount{{
Signer: signer,
Account: c.CommitteeAcc,
}})
if err != nil {
return fmt.Errorf("failed to create deploy tx for %s: %w", nnsContract, err)
}
if err := c.multiSignAndSend(tx, committeeAccountName); err != nil {
return fmt.Errorf("can't send deploy transaction: %w", err)
}
return c.awaitTx()
}
func (c *initializeContext) updateContracts() error {
alphaCs := c.getContract(alphabetContract)
nnsCs, err := c.nnsContractState()
if err != nil {
return err
}
nnsHash := nnsCs.Hash
w := io2.NewBufBinWriter()
var keysParam []any
// Update script size for a single-node committee is close to the maximum allowed size of 65535.
// Because of this we want to reuse alphabet contract NEF and manifest for different updates.
// The generated script is as following.
// 1. Initialize static slot for alphabet NEF.
// 2. Store NEF into the static slot.
// 3. Push parameters for each alphabet contract on stack.
// 4. Add contract group to the manifest.
// 5. For each alphabet contract, invoke `update` using parameters on stack and
// NEF from step 2 and manifest from step 4.
emit.Instruction(w.BinWriter, opcode.INITSSLOT, []byte{1})
emit.Bytes(w.BinWriter, alphaCs.RawNEF)
emit.Opcodes(w.BinWriter, opcode.STSFLD0)
baseGroups := alphaCs.Manifest.Groups
// alphabet contracts should be deployed by individual nodes to get different hashes.
for i, acc := range c.Accounts {
ctrHash, err := nnsResolveHash(c.ReadOnlyInvoker, nnsHash, getAlphabetNNSDomain(i))
if err != nil {
return fmt.Errorf("can't resolve hash for contract update: %w", err)
}
keysParam = append(keysParam, acc.PrivateKey().PublicKey().Bytes())
params := c.getAlphabetDeployItems(i, len(c.Wallets))
emit.Array(w.BinWriter, params...)
alphaCs.Manifest.Groups = baseGroups
err = c.addManifestGroup(ctrHash, alphaCs)
if err != nil {
return fmt.Errorf("can't sign manifest group: %v", err)
}
emit.Bytes(w.BinWriter, alphaCs.RawManifest)
emit.Opcodes(w.BinWriter, opcode.LDSFLD0)
emit.Int(w.BinWriter, 3)
emit.Opcodes(w.BinWriter, opcode.PACK)
emit.AppCallNoArgs(w.BinWriter, ctrHash, updateMethodName, callflag.All)
}
if err := c.sendCommitteeTx(w.Bytes(), false); err != nil {
if !strings.Contains(err.Error(), common.ErrAlreadyUpdated) {
return err
}
c.Command.Println("Alphabet contracts are already updated.")
}
w.Reset()
emit.Instruction(w.BinWriter, opcode.INITSSLOT, []byte{1})
emit.AppCall(w.BinWriter, nnsHash, "getPrice", callflag.All)
emit.Opcodes(w.BinWriter, opcode.STSFLD0)
emit.AppCall(w.BinWriter, nnsHash, "setPrice", callflag.All, 1)
for _, ctrName := range contractList {
cs := c.getContract(ctrName)
method := updateMethodName
ctrHash, err := nnsResolveHash(c.ReadOnlyInvoker, nnsHash, ctrName+".frostfs")
if err != nil {
if errors.Is(err, errMissingNNSRecord) {
// if contract not found we deploy it instead of update
method = deployMethodName
} else {
return fmt.Errorf("can't resolve hash for contract update: %w", err)
}
}
err = c.addManifestGroup(ctrHash, cs)
if err != nil {
return fmt.Errorf("can't sign manifest group: %v", err)
}
invokeHash := management.Hash
if method == updateMethodName {
invokeHash = ctrHash
}
params := getContractDeployParameters(cs, c.getContractDeployData(ctrName, keysParam))
res, err := c.CommitteeAct.MakeCall(invokeHash, method, params...)
if err != nil {
if method != updateMethodName || !strings.Contains(err.Error(), common.ErrAlreadyUpdated) {
return fmt.Errorf("deploy contract: %w", err)
}
c.Command.Printf("%s contract is already updated.\n", ctrName)
continue
}
w.WriteBytes(res.Script)
if method == deployMethodName {
// same actions are done in initializeContext.setNNS, can be unified
domain := ctrName + ".frostfs"
script, ok, err := c.nnsRegisterDomainScript(nnsHash, cs.Hash, domain)
if err != nil {
return err
}
if !ok {
w.WriteBytes(script)
emit.AppCall(w.BinWriter, nnsHash, "deleteRecords", callflag.All, domain, int64(nns.TXT))
emit.AppCall(w.BinWriter, nnsHash, "addRecord", callflag.All,
domain, int64(nns.TXT), cs.Hash.StringLE())
emit.AppCall(w.BinWriter, nnsHash, "addRecord", callflag.All,
domain, int64(nns.TXT), address.Uint160ToString(cs.Hash))
}
c.Command.Printf("NNS: Set %s -> %s\n", domain, cs.Hash.StringLE())
}
}
groupKey := c.ContractWallet.Accounts[0].PrivateKey().PublicKey()
_, _, err = c.emitUpdateNNSGroupScript(w, nnsHash, groupKey)
if err != nil {
return err
}
c.Command.Printf("NNS: Set %s -> %s\n", morphClient.NNSGroupKeyName, hex.EncodeToString(groupKey.Bytes()))
emit.Opcodes(w.BinWriter, opcode.LDSFLD0)
emit.Int(w.BinWriter, 1)
emit.Opcodes(w.BinWriter, opcode.PACK)
emit.AppCallNoArgs(w.BinWriter, nnsHash, "setPrice", callflag.All)
if err := c.sendCommitteeTx(w.Bytes(), false); err != nil {
return err
}
return c.awaitTx()
}
func (c *initializeContext) deployContracts() error {
alphaCs := c.getContract(alphabetContract)
var keysParam []any
baseGroups := alphaCs.Manifest.Groups
// alphabet contracts should be deployed by individual nodes to get different hashes.
for i, acc := range c.Accounts {
ctrHash := state.CreateContractHash(acc.Contract.ScriptHash(), alphaCs.NEF.Checksum, alphaCs.Manifest.Name)
if c.isUpdated(ctrHash, alphaCs) {
c.Command.Printf("Alphabet contract #%d is already deployed.\n", i)
continue
}
alphaCs.Manifest.Groups = baseGroups
err := c.addManifestGroup(ctrHash, alphaCs)
if err != nil {
return fmt.Errorf("can't sign manifest group: %v", err)
}
keysParam = append(keysParam, acc.PrivateKey().PublicKey().Bytes())
params := getContractDeployParameters(alphaCs, c.getAlphabetDeployItems(i, len(c.Wallets)))
act, err := actor.NewSimple(c.Client, acc)
if err != nil {
return fmt.Errorf("could not create actor: %w", err)
}
txHash, vub, err := act.SendCall(management.Hash, deployMethodName, params...)
if err != nil {
return fmt.Errorf("can't deploy alphabet #%d contract: %w", i, err)
}
c.SentTxs = append(c.SentTxs, hashVUBPair{hash: txHash, vub: vub})
}
for _, ctrName := range contractList {
cs := c.getContract(ctrName)
ctrHash := cs.Hash
if c.isUpdated(ctrHash, cs) {
c.Command.Printf("%s contract is already deployed.\n", ctrName)
continue
}
err := c.addManifestGroup(ctrHash, cs)
if err != nil {
return fmt.Errorf("can't sign manifest group: %v", err)
}
params := getContractDeployParameters(cs, c.getContractDeployData(ctrName, keysParam))
res, err := c.CommitteeAct.MakeCall(management.Hash, deployMethodName, params...)
if err != nil {
return fmt.Errorf("can't deploy %s contract: %w", ctrName, err)
}
if err := c.sendCommitteeTx(res.Script, false); err != nil {
return err
}
}
return c.awaitTx()
}
func (c *initializeContext) isUpdated(ctrHash util.Uint160, cs *contractState) bool {
realCs, err := c.Client.GetContractStateByHash(ctrHash)
return err == nil && realCs.NEF.Checksum == cs.NEF.Checksum
}
func (c *initializeContext) getContract(ctrName string) *contractState {
return c.Contracts[ctrName]
}
func (c *initializeContext) readContracts(names []string) error {
var (
fi os.FileInfo
err error
)
if c.ContractPath != "" {
fi, err = os.Stat(c.ContractPath)
if err != nil {
return fmt.Errorf("invalid contracts path: %w", err)
}
}
if c.ContractPath != "" && fi.IsDir() {
for _, ctrName := range names {
cs, err := readContract(filepath.Join(c.ContractPath, ctrName), ctrName)
if err != nil {
return err
}
c.Contracts[ctrName] = cs
}
} else {
var r io.ReadCloser
if c.ContractPath == "" {
c.Command.Println("Contracts flag is missing, latest release will be fetched from Github.")
r, err = downloadContractsFromGithub(c.Command)
} else {
r, err = os.Open(c.ContractPath)
}
if err != nil {
return fmt.Errorf("can't open contracts archive: %w", err)
}
defer r.Close()
m, err := readContractsFromArchive(r, names)
if err != nil {
return err
}
for _, name := range names {
if err := m[name].parse(); err != nil {
return err
}
c.Contracts[name] = m[name]
}
}
for _, ctrName := range names {
if ctrName != alphabetContract {
cs := c.Contracts[ctrName]
cs.Hash = state.CreateContractHash(c.CommitteeAcc.Contract.ScriptHash(),
cs.NEF.Checksum, cs.Manifest.Name)
}
}
return nil
}
func readContract(ctrPath, ctrName string) (*contractState, error) {
rawNef, err := os.ReadFile(filepath.Join(ctrPath, ctrName+"_contract.nef"))
if err != nil {
return nil, fmt.Errorf("can't read NEF file for %s contract: %w", ctrName, err)
}
rawManif, err := os.ReadFile(filepath.Join(ctrPath, "config.json"))
if err != nil {
return nil, fmt.Errorf("can't read manifest file for %s contract: %w", ctrName, err)
}
cs := &contractState{
RawNEF: rawNef,
RawManifest: rawManif,
}
return cs, cs.parse()
}
func (cs *contractState) parse() error {
nf, err := nef.FileFromBytes(cs.RawNEF)
if err != nil {
return fmt.Errorf("can't parse NEF file: %w", err)
}
m := new(manifest.Manifest)
if err := json.Unmarshal(cs.RawManifest, m); err != nil {
return fmt.Errorf("can't parse manifest file: %w", err)
}
cs.NEF = &nf
cs.Manifest = m
return nil
}
func readContractsFromArchive(file io.Reader, names []string) (map[string]*contractState, error) {
m := make(map[string]*contractState, len(names))
for i := range names {
m[names[i]] = new(contractState)
}
gr, err := gzip.NewReader(file)
if err != nil {
return nil, fmt.Errorf("contracts file must be tar.gz archive: %w", err)
}
r := tar.NewReader(gr)
for h, err := r.Next(); ; h, err = r.Next() {
if err != nil {
break
}
dir, _ := filepath.Split(h.Name)
ctrName := filepath.Base(dir)
cs, ok := m[ctrName]
if !ok {
continue
}
switch {
case strings.HasSuffix(h.Name, filepath.Join(ctrName, ctrName+"_contract.nef")):
cs.RawNEF, err = io.ReadAll(r)
if err != nil {
return nil, fmt.Errorf("can't read NEF file for %s contract: %w", ctrName, err)
}
case strings.HasSuffix(h.Name, "config.json"):
cs.RawManifest, err = io.ReadAll(r)
if err != nil {
return nil, fmt.Errorf("can't read manifest file for %s contract: %w", ctrName, err)
}
}
m[ctrName] = cs
}
for ctrName, cs := range m {
if cs.RawNEF == nil {
return nil, fmt.Errorf("NEF for %s contract wasn't found", ctrName)
}
if cs.RawManifest == nil {
return nil, fmt.Errorf("manifest for %s contract wasn't found", ctrName)
}
}
return m, nil
}
func getContractDeployParameters(cs *contractState, deployData []any) []any {
return []any{cs.RawNEF, cs.RawManifest, deployData}
}
func (c *initializeContext) getContractDeployData(ctrName string, keysParam []any) []any {
items := make([]any, 1, 6)
items[0] = false // notaryDisabled is false
switch ctrName {
case frostfsContract:
items = append(items,
c.Contracts[processingContract].Hash,
keysParam,
smartcontract.Parameter{})
case processingContract:
items = append(items, c.Contracts[frostfsContract].Hash)
return items[1:] // no notary info
case auditContract:
items = append(items, c.Contracts[netmapContract].Hash)
case balanceContract:
items = append(items,
c.Contracts[netmapContract].Hash,
c.Contracts[containerContract].Hash)
case containerContract:
// In case if NNS is updated multiple times, we can't calculate
// it's actual hash based on local data, thus query chain.
nnsCs, err := c.Client.GetContractStateByID(1)
if err != nil {
panic("NNS is not yet deployed")
}
items = append(items,
c.Contracts[netmapContract].Hash,
c.Contracts[balanceContract].Hash,
c.Contracts[frostfsIDContract].Hash,
nnsCs.Hash,
"container")
case frostfsIDContract:
items = append(items,
c.Contracts[netmapContract].Hash,
c.Contracts[containerContract].Hash)
case netmapContract:
configParam := []any{
netmapEpochKey, viper.GetInt64(epochDurationInitFlag),
netmapMaxObjectSizeKey, viper.GetInt64(maxObjectSizeInitFlag),
netmapAuditFeeKey, viper.GetInt64(auditFeeInitFlag),
netmapContainerFeeKey, viper.GetInt64(containerFeeInitFlag),
netmapContainerAliasFeeKey, viper.GetInt64(containerAliasFeeInitFlag),
netmapEigenTrustIterationsKey, int64(defaultEigenTrustIterations),
netmapEigenTrustAlphaKey, defaultEigenTrustAlpha,
netmapBasicIncomeRateKey, viper.GetInt64(incomeRateInitFlag),
netmapInnerRingCandidateFeeKey, viper.GetInt64(candidateFeeInitFlag),
netmapWithdrawFeeKey, viper.GetInt64(withdrawFeeInitFlag),
netmapHomomorphicHashDisabledKey, viper.GetBool(homomorphicHashDisabledInitFlag),
netmapMaintenanceAllowedKey, viper.GetBool(maintenanceModeAllowedInitFlag),
}
items = append(items,
c.Contracts[balanceContract].Hash,
c.Contracts[containerContract].Hash,
keysParam,
configParam)
case proxyContract:
items = nil
case reputationContract:
case subnetContract:
default:
panic(fmt.Sprintf("invalid contract name: %s", ctrName))
}
return items
}
func (c *initializeContext) getAlphabetDeployItems(i, n int) []any {
items := make([]any, 6)
items[0] = false
items[1] = c.Contracts[netmapContract].Hash
items[2] = c.Contracts[proxyContract].Hash
items[3] = innerring.GlagoliticLetter(i).String()
items[4] = int64(i)
items[5] = int64(n)
return items
}

View file

@ -0,0 +1,291 @@
package morph
import (
"encoding/hex"
"errors"
"fmt"
"strconv"
"time"
"github.com/TrueCloudLab/frostfs-contract/nns"
morphClient "github.com/TrueCloudLab/frostfs-node/pkg/morph/client"
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/opcode"
"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
"github.com/nspcc-dev/neo-go/pkg/vm/vmstate"
)
const defaultExpirationTime = 10 * 365 * 24 * time.Hour / time.Second
func (c *initializeContext) setNNS() error {
nnsCs, err := c.Client.GetContractStateByID(1)
if err != nil {
return err
}
ok, err := c.nnsRootRegistered(nnsCs.Hash, "frostfs")
if err != nil {
return err
} else if !ok {
bw := io.NewBufBinWriter()
emit.AppCall(bw.BinWriter, nnsCs.Hash, "register", callflag.All,
"frostfs", c.CommitteeAcc.Contract.ScriptHash(),
"ops@nspcc.ru", int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
if err := c.sendCommitteeTx(bw.Bytes(), true); err != nil {
return fmt.Errorf("can't add domain root to NNS: %w", err)
}
if err := c.awaitTx(); err != nil {
return err
}
}
alphaCs := c.getContract(alphabetContract)
for i, acc := range c.Accounts {
alphaCs.Hash = state.CreateContractHash(acc.Contract.ScriptHash(), alphaCs.NEF.Checksum, alphaCs.Manifest.Name)
domain := getAlphabetNNSDomain(i)
if err := c.nnsRegisterDomain(nnsCs.Hash, alphaCs.Hash, domain); err != nil {
return err
}
c.Command.Printf("NNS: Set %s -> %s\n", domain, alphaCs.Hash.StringLE())
}
for _, ctrName := range contractList {
cs := c.getContract(ctrName)
domain := ctrName + ".frostfs"
if err := c.nnsRegisterDomain(nnsCs.Hash, cs.Hash, domain); err != nil {
return err
}
c.Command.Printf("NNS: Set %s -> %s\n", domain, cs.Hash.StringLE())
}
groupKey := c.ContractWallet.Accounts[0].PrivateKey().PublicKey()
err = c.updateNNSGroup(nnsCs.Hash, groupKey)
if err != nil {
return err
}
c.Command.Printf("NNS: Set %s -> %s\n", morphClient.NNSGroupKeyName, hex.EncodeToString(groupKey.Bytes()))
return c.awaitTx()
}
func (c *initializeContext) updateNNSGroup(nnsHash util.Uint160, pub *keys.PublicKey) error {
bw := io.NewBufBinWriter()
needUpdate, needRegister, err := c.emitUpdateNNSGroupScript(bw, nnsHash, pub)
if !needUpdate || err != nil {
return err
}
script := bw.Bytes()
if needRegister {
w := io.NewBufBinWriter()
emit.Instruction(w.BinWriter, opcode.INITSSLOT, []byte{1})
wrapRegisterScriptWithPrice(w, nnsHash, script)
script = w.Bytes()
}
return c.sendCommitteeTx(script, true)
}
// emitUpdateNNSGroupScript emits script for updating group key stored in NNS.
// First return value is true iff the key is already there and nothing should be done.
// Second return value is true iff a domain registration code was emitted.
func (c *initializeContext) emitUpdateNNSGroupScript(bw *io.BufBinWriter, nnsHash util.Uint160, pub *keys.PublicKey) (bool, bool, error) {
isAvail, err := nnsIsAvailable(c.Client, nnsHash, morphClient.NNSGroupKeyName)
if err != nil {
return false, false, err
}
if !isAvail {
currentPub, err := nnsResolveKey(c.ReadOnlyInvoker, nnsHash, morphClient.NNSGroupKeyName)
if err != nil {
return false, false, err
}
if pub.Equal(currentPub) {
return true, false, nil
}
}
if isAvail {
emit.AppCall(bw.BinWriter, nnsHash, "register", callflag.All,
morphClient.NNSGroupKeyName, c.CommitteeAcc.Contract.ScriptHash(),
"ops@nspcc.ru", int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
}
emit.AppCall(bw.BinWriter, nnsHash, "deleteRecords", callflag.All, "group.frostfs", int64(nns.TXT))
emit.AppCall(bw.BinWriter, nnsHash, "addRecord", callflag.All,
"group.frostfs", int64(nns.TXT), hex.EncodeToString(pub.Bytes()))
return false, isAvail, nil
}
func getAlphabetNNSDomain(i int) string {
return alphabetContract + strconv.FormatUint(uint64(i), 10) + ".frostfs"
}
// wrapRegisterScriptWithPrice wraps a given script with `getPrice`/`setPrice` calls for NNS.
// It is intended to be used for a single transaction, and not as a part of other scripts.
// It is assumed that script already contains static slot initialization code, the first one
// (with index 0) is used to store the price.
func wrapRegisterScriptWithPrice(w *io.BufBinWriter, nnsHash util.Uint160, s []byte) {
if len(s) == 0 {
return
}
emit.AppCall(w.BinWriter, nnsHash, "getPrice", callflag.All)
emit.Opcodes(w.BinWriter, opcode.STSFLD0)
emit.AppCall(w.BinWriter, nnsHash, "setPrice", callflag.All, 1)
w.WriteBytes(s)
emit.Opcodes(w.BinWriter, opcode.LDSFLD0, opcode.PUSH1, opcode.PACK)
emit.AppCallNoArgs(w.BinWriter, nnsHash, "setPrice", callflag.All)
if w.Err != nil {
panic(fmt.Errorf("BUG: can't wrap register script: %w", w.Err))
}
}
func (c *initializeContext) nnsRegisterDomainScript(nnsHash, expectedHash util.Uint160, domain string) ([]byte, bool, error) {
ok, err := nnsIsAvailable(c.Client, nnsHash, domain)
if err != nil {
return nil, false, err
}
if ok {
bw := io.NewBufBinWriter()
emit.AppCall(bw.BinWriter, nnsHash, "register", callflag.All,
domain, c.CommitteeAcc.Contract.ScriptHash(),
"ops@nspcc.ru", int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
if bw.Err != nil {
panic(bw.Err)
}
return bw.Bytes(), false, nil
}
s, err := nnsResolveHash(c.ReadOnlyInvoker, nnsHash, domain)
if err != nil {
return nil, false, err
}
return nil, s == expectedHash, nil
}
func (c *initializeContext) nnsRegisterDomain(nnsHash, expectedHash util.Uint160, domain string) error {
script, ok, err := c.nnsRegisterDomainScript(nnsHash, expectedHash, domain)
if ok || err != nil {
return err
}
w := io.NewBufBinWriter()
emit.Instruction(w.BinWriter, opcode.INITSSLOT, []byte{1})
wrapRegisterScriptWithPrice(w, nnsHash, script)
emit.AppCall(w.BinWriter, nnsHash, "deleteRecords", callflag.All, domain, int64(nns.TXT))
emit.AppCall(w.BinWriter, nnsHash, "addRecord", callflag.All,
domain, int64(nns.TXT), expectedHash.StringLE())
emit.AppCall(w.BinWriter, nnsHash, "addRecord", callflag.All,
domain, int64(nns.TXT), address.Uint160ToString(expectedHash))
return c.sendCommitteeTx(w.Bytes(), true)
}
func (c *initializeContext) nnsRootRegistered(nnsHash util.Uint160, zone string) (bool, error) {
res, err := c.CommitteeAct.Call(nnsHash, "isAvailable", "name."+zone)
if err != nil {
return false, err
}
return res.State == vmstate.Halt.String(), nil
}
var errMissingNNSRecord = errors.New("missing NNS record")
// Returns errMissingNNSRecord if invocation fault exception contains "token not found".
func nnsResolveHash(inv *invoker.Invoker, nnsHash util.Uint160, domain string) (util.Uint160, error) {
item, err := nnsResolve(inv, nnsHash, domain)
if err != nil {
return util.Uint160{}, err
}
return parseNNSResolveResult(item)
}
func nnsResolve(inv *invoker.Invoker, nnsHash util.Uint160, domain string) (stackitem.Item, error) {
return unwrap.Item(inv.Call(nnsHash, "resolve", domain, int64(nns.TXT)))
}
func nnsResolveKey(inv *invoker.Invoker, nnsHash util.Uint160, domain string) (*keys.PublicKey, error) {
item, err := nnsResolve(inv, nnsHash, domain)
if err != nil {
return nil, err
}
v, ok := item.Value().(stackitem.Null)
if ok {
return nil, errors.New("NNS record is missing")
}
bs, err := v.TryBytes()
if err != nil {
return nil, errors.New("malformed response")
}
return keys.NewPublicKeyFromString(string(bs))
}
// parseNNSResolveResult parses the result of resolving NNS record.
// It works with multiple formats (corresponding to multiple NNS versions).
// If array of hashes is provided, it returns only the first one.
func parseNNSResolveResult(res stackitem.Item) (util.Uint160, error) {
arr, ok := res.Value().([]stackitem.Item)
if !ok {
arr = []stackitem.Item{res}
}
if _, ok := res.Value().(stackitem.Null); ok || len(arr) == 0 {
return util.Uint160{}, errors.New("NNS record is missing")
}
for i := range arr {
bs, err := arr[i].TryBytes()
if err != nil {
continue
}
// We support several formats for hash encoding, this logic should be maintained in sync
// with nnsResolve from pkg/morph/client/nns.go
h, err := util.Uint160DecodeStringLE(string(bs))
if err == nil {
return h, nil
}
h, err = address.StringToUint160(string(bs))
if err == nil {
return h, nil
}
}
return util.Uint160{}, errors.New("no valid hashes are found")
}
func nnsIsAvailable(c Client, nnsHash util.Uint160, name string) (bool, error) {
switch ct := c.(type) {
case *rpcclient.Client:
return ct.NNSIsAvailable(nnsHash, name)
default:
b, err := unwrap.Bool(invokeFunction(c, nnsHash, "isAvailable", []any{name}, nil))
if err != nil {
return false, fmt.Errorf("`isAvailable`: invalid response: %w", err)
}
return b, nil
}
}

View file

@ -0,0 +1,137 @@
package morph
import (
"errors"
"fmt"
"github.com/nspcc-dev/neo-go/pkg/core/native"
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/neo"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/opcode"
)
// initialAlphabetNEOAmount represents the total amount of GAS distributed between alphabet nodes.
const initialAlphabetNEOAmount = native.NEOTotalSupply
func (c *initializeContext) registerCandidates() error {
neoHash := neo.Hash
cc, err := unwrap.Array(c.ReadOnlyInvoker.Call(neoHash, "getCandidates"))
if err != nil {
return fmt.Errorf("`getCandidates`: %w", err)
}
if len(cc) > 0 {
c.Command.Println("Candidates are already registered.")
return nil
}
regPrice, err := c.getCandidateRegisterPrice()
if err != nil {
return fmt.Errorf("can't fetch registration price: %w", err)
}
w := io.NewBufBinWriter()
emit.AppCall(w.BinWriter, neoHash, "setRegisterPrice", callflag.States, 1)
for _, acc := range c.Accounts {
emit.AppCall(w.BinWriter, neoHash, "registerCandidate", callflag.States, acc.PrivateKey().PublicKey().Bytes())
emit.Opcodes(w.BinWriter, opcode.ASSERT)
}
emit.AppCall(w.BinWriter, neoHash, "setRegisterPrice", callflag.States, regPrice)
if w.Err != nil {
panic(fmt.Sprintf("BUG: %v", w.Err))
}
signers := []rpcclient.SignerAccount{{
Signer: c.getSigner(false, c.CommitteeAcc),
Account: c.CommitteeAcc,
}}
for i := range c.Accounts {
signers = append(signers, rpcclient.SignerAccount{
Signer: transaction.Signer{
Account: c.Accounts[i].Contract.ScriptHash(),
Scopes: transaction.CustomContracts,
AllowedContracts: []util.Uint160{neoHash},
},
Account: c.Accounts[i],
})
}
tx, err := c.Client.CreateTxFromScript(w.Bytes(), c.CommitteeAcc, -1, 0, signers)
if err != nil {
return fmt.Errorf("can't create tx: %w", err)
}
if err := c.multiSign(tx, committeeAccountName); err != nil {
return fmt.Errorf("can't sign a transaction: %w", err)
}
network := c.CommitteeAct.GetNetwork()
for i := range c.Accounts {
if err := c.Accounts[i].SignTx(network, tx); err != nil {
return fmt.Errorf("can't sign a transaction: %w", err)
}
}
return c.sendTx(tx, c.Command, true)
}
func (c *initializeContext) transferNEOToAlphabetContracts() error {
neoHash := neo.Hash
ok, err := c.transferNEOFinished(neoHash)
if ok || err != nil {
return err
}
cs := c.getContract(alphabetContract)
amount := initialAlphabetNEOAmount / len(c.Wallets)
bw := io.NewBufBinWriter()
for _, acc := range c.Accounts {
h := state.CreateContractHash(acc.Contract.ScriptHash(), cs.NEF.Checksum, cs.Manifest.Name)
emit.AppCall(bw.BinWriter, neoHash, "transfer", callflag.All,
c.CommitteeAcc.Contract.ScriptHash(), h, int64(amount), nil)
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
}
if err := c.sendCommitteeTx(bw.Bytes(), false); err != nil {
return err
}
return c.awaitTx()
}
func (c *initializeContext) transferNEOFinished(neoHash util.Uint160) (bool, error) {
bal, err := c.Client.NEP17BalanceOf(neoHash, c.CommitteeAcc.Contract.ScriptHash())
return bal < native.NEOTotalSupply, err
}
var errGetPriceInvalid = errors.New("`getRegisterPrice`: invalid response")
func (c *initializeContext) getCandidateRegisterPrice() (int64, error) {
switch ct := c.Client.(type) {
case *rpcclient.Client:
return ct.GetCandidateRegisterPrice()
default:
neoHash := neo.Hash
res, err := invokeFunction(c.Client, neoHash, "getRegisterPrice", nil, nil)
if err != nil {
return 0, err
}
if len(res.Stack) == 0 {
return 0, errGetPriceInvalid
}
bi, err := res.Stack[0].TryInteger()
if err != nil || !bi.IsInt64() {
return 0, errGetPriceInvalid
}
return bi.Int64(), nil
}
}

View file

@ -0,0 +1,45 @@
package morph
import (
"github.com/nspcc-dev/neo-go/pkg/core/native/noderoles"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/rolemgmt"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
)
func (c *initializeContext) setNotaryAndAlphabetNodes() error {
if ok, err := c.setRolesFinished(); ok || err != nil {
if err == nil {
c.Command.Println("Stage 2: already performed.")
}
return err
}
var pubs []any
for _, acc := range c.Accounts {
pubs = append(pubs, acc.PrivateKey().PublicKey().Bytes())
}
w := io.NewBufBinWriter()
emit.AppCall(w.BinWriter, rolemgmt.Hash, "designateAsRole",
callflag.States|callflag.AllowNotify, int64(noderoles.P2PNotary), pubs)
emit.AppCall(w.BinWriter, rolemgmt.Hash, "designateAsRole",
callflag.States|callflag.AllowNotify, int64(noderoles.NeoFSAlphabet), pubs)
if err := c.sendCommitteeTx(w.Bytes(), false); err != nil {
return err
}
return c.awaitTx()
}
func (c *initializeContext) setRolesFinished() (bool, error) {
height, err := c.Client.GetBlockCount()
if err != nil {
return false, err
}
pubs, err := getDesignatedByRole(c.ReadOnlyInvoker, rolemgmt.Hash, noderoles.NeoFSAlphabet, height)
return len(pubs) == len(c.Wallets), err
}

View file

@ -0,0 +1,121 @@
package morph
import (
"encoding/hex"
"os"
"path/filepath"
"strconv"
"testing"
"github.com/TrueCloudLab/frostfs-node/pkg/innerring"
"github.com/nspcc-dev/neo-go/pkg/config"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/vm"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/viper"
"github.com/stretchr/testify/require"
"gopkg.in/yaml.v3"
)
const (
contractsPath = "../../../../../../frostfs-contract/frostfs-contract-v0.16.0.tar.gz"
protoFileName = "proto.yml"
)
func TestInitialize(t *testing.T) {
// This test needs frostfs-contract tarball, so it is skipped by default.
// It is here for performing local testing after the changes.
t.Skip()
t.Run("1 nodes", func(t *testing.T) {
testInitialize(t, 1)
})
t.Run("4 nodes", func(t *testing.T) {
testInitialize(t, 4)
})
t.Run("7 nodes", func(t *testing.T) {
testInitialize(t, 7)
})
}
func testInitialize(t *testing.T, committeeSize int) {
testdataDir := t.TempDir()
v := viper.GetViper()
generateTestData(t, testdataDir, committeeSize)
v.Set(protoConfigPath, filepath.Join(testdataDir, protoFileName))
// Set to the path or remove the next statement to download from the network.
require.NoError(t, initCmd.Flags().Set(contractsInitFlag, contractsPath))
v.Set(localDumpFlag, filepath.Join(testdataDir, "out"))
v.Set(alphabetWalletsFlag, testdataDir)
v.Set(epochDurationInitFlag, 1)
v.Set(maxObjectSizeInitFlag, 1024)
setTestCredentials(v, committeeSize)
require.NoError(t, initializeSideChainCmd(initCmd, nil))
t.Run("force-new-epoch", func(t *testing.T) {
require.NoError(t, forceNewEpochCmd(forceNewEpoch, nil))
})
t.Run("set-config", func(t *testing.T) {
require.NoError(t, setConfigCmd(setConfig, []string{"MaintenanceModeAllowed=true"}))
})
t.Run("set-policy", func(t *testing.T) {
require.NoError(t, setPolicyCmd(setPolicy, []string{"ExecFeeFactor=1"}))
})
t.Run("remove-node", func(t *testing.T) {
pk, err := keys.NewPrivateKey()
require.NoError(t, err)
pub := hex.EncodeToString(pk.PublicKey().Bytes())
require.NoError(t, removeNodesCmd(removeNodes, []string{pub}))
})
}
func generateTestData(t *testing.T, dir string, size int) {
v := viper.GetViper()
v.Set(alphabetWalletsFlag, dir)
sizeStr := strconv.FormatUint(uint64(size), 10)
require.NoError(t, generateAlphabetCmd.Flags().Set(alphabetSizeFlag, sizeStr))
setTestCredentials(v, size)
require.NoError(t, generateAlphabetCreds(generateAlphabetCmd, nil))
var pubs []string
for i := 0; i < size; i++ {
p := filepath.Join(dir, innerring.GlagoliticLetter(i).String()+".json")
w, err := wallet.NewWalletFromFile(p)
require.NoError(t, err, "wallet doesn't exist")
for _, acc := range w.Accounts {
if acc.Label == singleAccountName {
pub, ok := vm.ParseSignatureContract(acc.Contract.Script)
require.True(t, ok)
pubs = append(pubs, hex.EncodeToString(pub))
continue
}
}
}
cfg := config.Config{}
cfg.ProtocolConfiguration.Magic = 12345
cfg.ProtocolConfiguration.ValidatorsCount = size
cfg.ProtocolConfiguration.SecondsPerBlock = 1
cfg.ProtocolConfiguration.StandbyCommittee = pubs // sorted by glagolic letters
cfg.ProtocolConfiguration.P2PSigExtensions = true
cfg.ProtocolConfiguration.VerifyTransactions = true
cfg.ProtocolConfiguration.VerifyBlocks = true
data, err := yaml.Marshal(cfg)
require.NoError(t, err)
protoPath := filepath.Join(dir, protoFileName)
require.NoError(t, os.WriteFile(protoPath, data, os.ModePerm))
}
func setTestCredentials(v *viper.Viper, size int) {
for i := 0; i < size; i++ {
v.Set("credentials."+innerring.GlagoliticLetter(i).String(), strconv.FormatUint(uint64(i), 10))
}
v.Set("credentials.contract", testContractPassword)
}

View file

@ -0,0 +1,190 @@
package morph
import (
"fmt"
"github.com/nspcc-dev/neo-go/pkg/core/native"
"github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/gas"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/neo"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
scContext "github.com/nspcc-dev/neo-go/pkg/smartcontract/context"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/opcode"
"github.com/nspcc-dev/neo-go/pkg/wallet"
)
const (
gasInitialTotalSupply = 30000000 * native.GASFactor
// initialAlphabetGASAmount represents the amount of GAS given to each alphabet node.
initialAlphabetGASAmount = 10_000 * native.GASFactor
// initialProxyGASAmount represents the amount of GAS given to a proxy contract.
initialProxyGASAmount = 50_000 * native.GASFactor
)
func (c *initializeContext) transferFunds() error {
ok, err := c.transferFundsFinished()
if ok || err != nil {
if err == nil {
c.Command.Println("Stage 1: already performed.")
}
return err
}
var transfers []rpcclient.TransferTarget
for _, acc := range c.Accounts {
to := acc.Contract.ScriptHash()
transfers = append(transfers,
rpcclient.TransferTarget{
Token: gas.Hash,
Address: to,
Amount: initialAlphabetGASAmount,
},
)
}
// It is convenient to have all funds at the committee account.
transfers = append(transfers,
rpcclient.TransferTarget{
Token: gas.Hash,
Address: c.CommitteeAcc.Contract.ScriptHash(),
Amount: (gasInitialTotalSupply - initialAlphabetGASAmount*int64(len(c.Wallets))) / 2,
},
rpcclient.TransferTarget{
Token: neo.Hash,
Address: c.CommitteeAcc.Contract.ScriptHash(),
Amount: native.NEOTotalSupply,
},
)
tx, err := createNEP17MultiTransferTx(c.Client, c.ConsensusAcc, 0, transfers, []rpcclient.SignerAccount{{
Signer: transaction.Signer{
Account: c.ConsensusAcc.Contract.ScriptHash(),
Scopes: transaction.CalledByEntry,
},
Account: c.ConsensusAcc,
}})
if err != nil {
return fmt.Errorf("can't create transfer transaction: %w", err)
}
if err := c.multiSignAndSend(tx, consensusAccountName); err != nil {
return fmt.Errorf("can't send transfer transaction: %w", err)
}
return c.awaitTx()
}
func (c *initializeContext) transferFundsFinished() (bool, error) {
acc := c.Accounts[0]
res, err := c.Client.NEP17BalanceOf(gas.Hash, acc.Contract.ScriptHash())
return res > initialAlphabetGASAmount/2, err
}
func (c *initializeContext) multiSignAndSend(tx *transaction.Transaction, accType string) error {
if err := c.multiSign(tx, accType); err != nil {
return err
}
return c.sendTx(tx, c.Command, false)
}
func (c *initializeContext) multiSign(tx *transaction.Transaction, accType string) error {
network, err := c.Client.GetNetwork()
if err != nil {
// error appears only if client
// has not been initialized
panic(err)
}
// Use parameter context to avoid dealing with signature order.
pc := scContext.NewParameterContext("", network, tx)
h := c.CommitteeAcc.Contract.ScriptHash()
if accType == consensusAccountName {
h = c.ConsensusAcc.Contract.ScriptHash()
}
for _, w := range c.Wallets {
acc, err := getWalletAccount(w, accType)
if err != nil {
return fmt.Errorf("can't find %s wallet account: %w", accType, err)
}
priv := acc.PrivateKey()
sign := priv.SignHashable(uint32(network), tx)
if err := pc.AddSignature(h, acc.Contract, priv.PublicKey(), sign); err != nil {
return fmt.Errorf("can't add signature: %w", err)
}
if len(pc.Items[h].Signatures) == len(acc.Contract.Parameters) {
break
}
}
w, err := pc.GetWitness(h)
if err != nil {
return fmt.Errorf("incomplete signature: %w", err)
}
for i := range tx.Signers {
if tx.Signers[i].Account == h {
if i < len(tx.Scripts) {
tx.Scripts[i] = *w
} else if i == len(tx.Scripts) {
tx.Scripts = append(tx.Scripts, *w)
} else {
panic("BUG: invalid signing order")
}
return nil
}
}
return fmt.Errorf("%s account was not found among transaction signers", accType)
}
func (c *initializeContext) transferGASToProxy() error {
proxyCs := c.getContract(proxyContract)
bal, err := c.Client.NEP17BalanceOf(gas.Hash, proxyCs.Hash)
if err != nil || bal > 0 {
return err
}
tx, err := createNEP17MultiTransferTx(c.Client, c.CommitteeAcc, 0, []rpcclient.TransferTarget{{
Token: gas.Hash,
Address: proxyCs.Hash,
Amount: initialProxyGASAmount,
}}, nil)
if err != nil {
return err
}
if err := c.multiSignAndSend(tx, committeeAccountName); err != nil {
return err
}
return c.awaitTx()
}
func createNEP17MultiTransferTx(c Client, acc *wallet.Account, netFee int64,
recipients []rpcclient.TransferTarget, cosigners []rpcclient.SignerAccount) (*transaction.Transaction, error) {
from := acc.Contract.ScriptHash()
w := io.NewBufBinWriter()
for i := range recipients {
emit.AppCall(w.BinWriter, recipients[i].Token, "transfer", callflag.All,
from, recipients[i].Address, recipients[i].Amount, recipients[i].Data)
emit.Opcodes(w.BinWriter, opcode.ASSERT)
}
if w.Err != nil {
return nil, fmt.Errorf("failed to create transfer script: %w", w.Err)
}
return c.CreateTxFromScript(w.Bytes(), acc, -1, netFee, append([]rpcclient.SignerAccount{{
Signer: transaction.Signer{
Account: from,
Scopes: transaction.CalledByEntry,
},
Account: acc,
}}, cosigners...))
}

View file

@ -0,0 +1,65 @@
package internal
import (
"fmt"
"strconv"
"google.golang.org/protobuf/proto"
)
// StringifySubnetClientGroupID returns string representation of SubnetClientGroupID using MarshalText.
// Returns a string with a message on error.
func StringifySubnetClientGroupID(id *SubnetClientGroupID) string {
text, err := id.MarshalText()
if err != nil {
return fmt.Sprintf("<invalid> %v", err)
}
return string(text)
}
// MarshalText encodes SubnetClientGroupID into text format according to FrostFS API V2 protocol:
// value in base-10 integer string format.
//
// It implements encoding.TextMarshaler.
func (x *SubnetClientGroupID) MarshalText() ([]byte, error) {
num := x.GetValue() // NPE safe, returns zero on nil
return []byte(strconv.FormatUint(uint64(num), 10)), nil
}
// UnmarshalText decodes the SubnetID from the text according to FrostFS API V2 protocol:
// should be base-10 integer string format with bitsize = 32.
//
// Returns strconv.ErrRange if integer overflows uint32.
//
// Must not be called on nil.
//
// Implements encoding.TextUnmarshaler.
func (x *SubnetClientGroupID) UnmarshalText(txt []byte) error {
num, err := strconv.ParseUint(string(txt), 10, 32)
if err != nil {
return fmt.Errorf("invalid numeric value: %w", err)
}
x.SetNumber(uint32(num))
return nil
}
// Marshal encodes the SubnetClientGroupID into a binary format of FrostFS API V2 protocol
// (Protocol Buffers with direct field order).
func (x *SubnetClientGroupID) Marshal() ([]byte, error) {
return proto.Marshal(x)
}
// Unmarshal decodes the SubnetClientGroupID from FrostFS API V2 binary format (see Marshal). Must not be called on nil.
func (x *SubnetClientGroupID) Unmarshal(data []byte) error {
return proto.Unmarshal(data, x)
}
// SetNumber sets SubnetClientGroupID value in uint32 format. Must not be called on nil.
// By default, number is 0.
func (x *SubnetClientGroupID) SetNumber(num uint32) {
x.Value = num
}

View file

@ -0,0 +1,156 @@
// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
// protoc-gen-go v1.26.0
// protoc v3.21.12
// source: cmd/frostfs-adm/internal/modules/morph/internal/types.proto
package internal
import (
protoreflect "google.golang.org/protobuf/reflect/protoreflect"
protoimpl "google.golang.org/protobuf/runtime/protoimpl"
reflect "reflect"
sync "sync"
)
const (
// Verify that this generated code is sufficiently up-to-date.
_ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion)
// Verify that runtime/protoimpl is sufficiently up-to-date.
_ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20)
)
// Client group identifier in the FrostFS subnet.
//
// String representation of a value is base-10 integer.
//
// JSON representation is an object containing single `value` number field.
type SubnetClientGroupID struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
// 4-byte integer identifier of the subnet client group.
Value uint32 `protobuf:"fixed32,1,opt,name=value,proto3" json:"value,omitempty"`
}
func (x *SubnetClientGroupID) Reset() {
*x = SubnetClientGroupID{}
if protoimpl.UnsafeEnabled {
mi := &file_cmd_frostfs_adm_internal_modules_morph_internal_types_proto_msgTypes[0]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
}
func (x *SubnetClientGroupID) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*SubnetClientGroupID) ProtoMessage() {}
func (x *SubnetClientGroupID) ProtoReflect() protoreflect.Message {
mi := &file_cmd_frostfs_adm_internal_modules_morph_internal_types_proto_msgTypes[0]
if protoimpl.UnsafeEnabled && x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use SubnetClientGroupID.ProtoReflect.Descriptor instead.
func (*SubnetClientGroupID) Descriptor() ([]byte, []int) {
return file_cmd_frostfs_adm_internal_modules_morph_internal_types_proto_rawDescGZIP(), []int{0}
}
func (x *SubnetClientGroupID) GetValue() uint32 {
if x != nil {
return x.Value
}
return 0
}
var File_cmd_frostfs_adm_internal_modules_morph_internal_types_proto protoreflect.FileDescriptor
var file_cmd_frostfs_adm_internal_modules_morph_internal_types_proto_rawDesc = []byte{
0x0a, 0x3b, 0x63, 0x6d, 0x64, 0x2f, 0x66, 0x72, 0x6f, 0x73, 0x74, 0x66, 0x73, 0x2d, 0x61, 0x64,
0x6d, 0x2f, 0x69, 0x6e, 0x74, 0x65, 0x72, 0x6e, 0x61, 0x6c, 0x2f, 0x6d, 0x6f, 0x64, 0x75, 0x6c,
0x65, 0x73, 0x2f, 0x6d, 0x6f, 0x72, 0x70, 0x68, 0x2f, 0x69, 0x6e, 0x74, 0x65, 0x72, 0x6e, 0x61,
0x6c, 0x2f, 0x74, 0x79, 0x70, 0x65, 0x73, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x0e, 0x6e,
0x65, 0x6f, 0x2e, 0x66, 0x73, 0x2e, 0x76, 0x32, 0x2e, 0x72, 0x65, 0x66, 0x73, 0x22, 0x2b, 0x0a,
0x13, 0x53, 0x75, 0x62, 0x6e, 0x65, 0x74, 0x43, 0x6c, 0x69, 0x65, 0x6e, 0x74, 0x47, 0x72, 0x6f,
0x75, 0x70, 0x49, 0x44, 0x12, 0x14, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x01, 0x20,
0x01, 0x28, 0x07, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x42, 0x56, 0x5a, 0x54, 0x67, 0x69,
0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x54, 0x72, 0x75, 0x65, 0x43, 0x6c, 0x6f,
0x75, 0x64, 0x4c, 0x61, 0x62, 0x2f, 0x66, 0x72, 0x6f, 0x73, 0x74, 0x66, 0x73, 0x2d, 0x6e, 0x6f,
0x64, 0x65, 0x2f, 0x63, 0x6d, 0x64, 0x2f, 0x66, 0x72, 0x6f, 0x73, 0x74, 0x66, 0x73, 0x2d, 0x61,
0x64, 0x6d, 0x2f, 0x69, 0x6e, 0x74, 0x65, 0x72, 0x6e, 0x61, 0x6c, 0x2f, 0x6d, 0x6f, 0x64, 0x75,
0x6c, 0x65, 0x73, 0x2f, 0x6d, 0x6f, 0x72, 0x70, 0x68, 0x2f, 0x69, 0x6e, 0x74, 0x65, 0x72, 0x6e,
0x61, 0x6c, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33,
}
var (
file_cmd_frostfs_adm_internal_modules_morph_internal_types_proto_rawDescOnce sync.Once
file_cmd_frostfs_adm_internal_modules_morph_internal_types_proto_rawDescData = file_cmd_frostfs_adm_internal_modules_morph_internal_types_proto_rawDesc
)
func file_cmd_frostfs_adm_internal_modules_morph_internal_types_proto_rawDescGZIP() []byte {
file_cmd_frostfs_adm_internal_modules_morph_internal_types_proto_rawDescOnce.Do(func() {
file_cmd_frostfs_adm_internal_modules_morph_internal_types_proto_rawDescData = protoimpl.X.CompressGZIP(file_cmd_frostfs_adm_internal_modules_morph_internal_types_proto_rawDescData)
})
return file_cmd_frostfs_adm_internal_modules_morph_internal_types_proto_rawDescData
}
var file_cmd_frostfs_adm_internal_modules_morph_internal_types_proto_msgTypes = make([]protoimpl.MessageInfo, 1)
var file_cmd_frostfs_adm_internal_modules_morph_internal_types_proto_goTypes = []interface{}{
(*SubnetClientGroupID)(nil), // 0: neo.fs.v2.refs.SubnetClientGroupID
}
var file_cmd_frostfs_adm_internal_modules_morph_internal_types_proto_depIdxs = []int32{
0, // [0:0] is the sub-list for method output_type
0, // [0:0] is the sub-list for method input_type
0, // [0:0] is the sub-list for extension type_name
0, // [0:0] is the sub-list for extension extendee
0, // [0:0] is the sub-list for field type_name
}
func init() { file_cmd_frostfs_adm_internal_modules_morph_internal_types_proto_init() }
func file_cmd_frostfs_adm_internal_modules_morph_internal_types_proto_init() {
if File_cmd_frostfs_adm_internal_modules_morph_internal_types_proto != nil {
return
}
if !protoimpl.UnsafeEnabled {
file_cmd_frostfs_adm_internal_modules_morph_internal_types_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} {
switch v := v.(*SubnetClientGroupID); i {
case 0:
return &v.state
case 1:
return &v.sizeCache
case 2:
return &v.unknownFields
default:
return nil
}
}
}
type x struct{}
out := protoimpl.TypeBuilder{
File: protoimpl.DescBuilder{
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
RawDescriptor: file_cmd_frostfs_adm_internal_modules_morph_internal_types_proto_rawDesc,
NumEnums: 0,
NumMessages: 1,
NumExtensions: 0,
NumServices: 0,
},
GoTypes: file_cmd_frostfs_adm_internal_modules_morph_internal_types_proto_goTypes,
DependencyIndexes: file_cmd_frostfs_adm_internal_modules_morph_internal_types_proto_depIdxs,
MessageInfos: file_cmd_frostfs_adm_internal_modules_morph_internal_types_proto_msgTypes,
}.Build()
File_cmd_frostfs_adm_internal_modules_morph_internal_types_proto = out.File
file_cmd_frostfs_adm_internal_modules_morph_internal_types_proto_rawDesc = nil
file_cmd_frostfs_adm_internal_modules_morph_internal_types_proto_goTypes = nil
file_cmd_frostfs_adm_internal_modules_morph_internal_types_proto_depIdxs = nil
}

View file

@ -0,0 +1,15 @@
syntax = "proto3";
package neo.fs.v2.refs;
option go_package = "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph/internal";
// Client group identifier in the FrostFS subnet.
//
// String representation of a value is base-10 integer.
//
// JSON representation is an object containing single `value` number field.
message SubnetClientGroupID {
// 4-byte integer identifier of the subnet client group.
fixed32 value = 1 [json_name = "value"];
}

View file

@ -0,0 +1,505 @@
package morph
import (
"crypto/elliptic"
"errors"
"fmt"
"os"
"sort"
"time"
"github.com/google/uuid"
"github.com/nspcc-dev/neo-go/pkg/config"
"github.com/nspcc-dev/neo-go/pkg/config/netmode"
"github.com/nspcc-dev/neo-go/pkg/core"
"github.com/nspcc-dev/neo-go/pkg/core/block"
"github.com/nspcc-dev/neo-go/pkg/core/chaindump"
"github.com/nspcc-dev/neo-go/pkg/core/fee"
"github.com/nspcc-dev/neo-go/pkg/core/native/noderoles"
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/core/storage"
"github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/crypto/hash"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/nspcc-dev/neo-go/pkg/encoding/fixedn"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/neorpc/result"
"github.com/nspcc-dev/neo-go/pkg/network/payload"
"github.com/nspcc-dev/neo-go/pkg/rpcclient"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/trigger"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/opcode"
"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
"github.com/nspcc-dev/neo-go/pkg/vm/vmstate"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"go.uber.org/zap"
)
type localClient struct {
bc *core.Blockchain
transactions []*transaction.Transaction
dumpPath string
accounts []*wallet.Account
maxGasInvoke int64
}
func newLocalClient(cmd *cobra.Command, v *viper.Viper, wallets []*wallet.Wallet) (*localClient, error) {
cfg, err := config.LoadFile(v.GetString(protoConfigPath))
if err != nil {
return nil, err
}
bc, err := core.NewBlockchain(storage.NewMemoryStore(), cfg.Blockchain(), zap.NewNop())
if err != nil {
return nil, err
}
m := smartcontract.GetDefaultHonestNodeCount(cfg.ProtocolConfiguration.ValidatorsCount)
accounts := make([]*wallet.Account, len(wallets))
for i := range accounts {
accounts[i], err = getWalletAccount(wallets[i], consensusAccountName)
if err != nil {
return nil, err
}
}
indexMap := make(map[string]int)
for i, pub := range cfg.ProtocolConfiguration.StandbyCommittee {
indexMap[pub] = i
}
sort.Slice(accounts, func(i, j int) bool {
pi := accounts[i].PrivateKey().PublicKey().Bytes()
pj := accounts[j].PrivateKey().PublicKey().Bytes()
return indexMap[string(pi)] < indexMap[string(pj)]
})
sort.Slice(accounts[:cfg.ProtocolConfiguration.ValidatorsCount], func(i, j int) bool {
return accounts[i].PublicKey().Cmp(accounts[j].PublicKey()) == -1
})
go bc.Run()
dumpPath := v.GetString(localDumpFlag)
if cmd.Name() != "init" {
f, err := os.OpenFile(dumpPath, os.O_RDONLY, 0600)
if err != nil {
return nil, fmt.Errorf("can't open local dump: %w", err)
}
defer f.Close()
r := io.NewBinReaderFromIO(f)
var skip uint32
if bc.BlockHeight() != 0 {
skip = bc.BlockHeight() + 1
}
count := r.ReadU32LE() - skip
if err := chaindump.Restore(bc, r, skip, count, nil); err != nil {
return nil, fmt.Errorf("can't restore local dump: %w", err)
}
}
return &localClient{
bc: bc,
dumpPath: dumpPath,
accounts: accounts[:m],
maxGasInvoke: 15_0000_0000,
}, nil
}
func (l *localClient) GetBlockCount() (uint32, error) {
return l.bc.BlockHeight(), nil
}
func (l *localClient) GetContractStateByID(id int32) (*state.Contract, error) {
h, err := l.bc.GetContractScriptHash(id)
if err != nil {
return nil, err
}
return l.GetContractStateByHash(h)
}
func (l *localClient) GetContractStateByHash(h util.Uint160) (*state.Contract, error) {
if cs := l.bc.GetContractState(h); cs != nil {
return cs, nil
}
return nil, storage.ErrKeyNotFound
}
func (l *localClient) GetNativeContracts() ([]state.NativeContract, error) {
return l.bc.GetNatives(), nil
}
func (l *localClient) GetNetwork() (netmode.Magic, error) {
return l.bc.GetConfig().Magic, nil
}
func (l *localClient) GetApplicationLog(h util.Uint256, t *trigger.Type) (*result.ApplicationLog, error) {
aer, err := l.bc.GetAppExecResults(h, *t)
if err != nil {
return nil, err
}
a := result.NewApplicationLog(h, aer, *t)
return &a, nil
}
func (l *localClient) CreateTxFromScript(script []byte, acc *wallet.Account, sysFee int64, netFee int64, cosigners []rpcclient.SignerAccount) (*transaction.Transaction, error) {
signers, accounts, err := getSigners(acc, cosigners)
if err != nil {
return nil, fmt.Errorf("failed to construct tx signers: %w", err)
}
if sysFee < 0 {
res, err := l.InvokeScript(script, signers)
if err != nil {
return nil, fmt.Errorf("can't add system fee to transaction: %w", err)
}
if res.State != "HALT" {
return nil, fmt.Errorf("can't add system fee to transaction: bad vm state: %s due to an error: %s", res.State, res.FaultException)
}
sysFee = res.GasConsumed
}
tx := transaction.New(script, sysFee)
tx.Signers = signers
tx.ValidUntilBlock = l.bc.BlockHeight() + 2
err = l.AddNetworkFee(tx, netFee, accounts...)
if err != nil {
return nil, fmt.Errorf("failed to add network fee: %w", err)
}
return tx, nil
}
func (l *localClient) GetCommittee() (keys.PublicKeys, error) {
// not used by `morph init` command
panic("unexpected call")
}
// InvokeFunction is implemented via `InvokeScript`.
func (l *localClient) InvokeFunction(h util.Uint160, method string, sPrm []smartcontract.Parameter, ss []transaction.Signer) (*result.Invoke, error) {
var err error
pp := make([]any, len(sPrm))
for i, p := range sPrm {
pp[i], err = smartcontract.ExpandParameterToEmitable(p)
if err != nil {
return nil, fmt.Errorf("incorrect parameter type %s: %w", p.Type, err)
}
}
return invokeFunction(l, h, method, pp, ss)
}
func (l *localClient) CalculateNotaryFee(_ uint8) (int64, error) {
// not used by `morph init` command
panic("unexpected call")
}
func (l *localClient) SignAndPushP2PNotaryRequest(_ *transaction.Transaction, _ []byte, _ int64, _ int64, _ uint32, _ *wallet.Account) (*payload.P2PNotaryRequest, error) {
// not used by `morph init` command
panic("unexpected call")
}
func (l *localClient) SignAndPushInvocationTx(_ []byte, _ *wallet.Account, _ int64, _ fixedn.Fixed8, _ []rpcclient.SignerAccount) (util.Uint256, error) {
// not used by `morph init` command
panic("unexpected call")
}
func (l *localClient) TerminateSession(_ uuid.UUID) (bool, error) {
// not used by `morph init` command
panic("unexpected call")
}
func (l *localClient) TraverseIterator(_, _ uuid.UUID, _ int) ([]stackitem.Item, error) {
// not used by `morph init` command
panic("unexpected call")
}
// GetVersion return default version.
func (l *localClient) GetVersion() (*result.Version, error) {
c := l.bc.GetConfig()
return &result.Version{
Protocol: result.Protocol{
AddressVersion: address.NEO3Prefix,
Network: c.Magic,
MillisecondsPerBlock: int(c.TimePerBlock / time.Millisecond),
MaxTraceableBlocks: c.MaxTraceableBlocks,
MaxValidUntilBlockIncrement: c.MaxValidUntilBlockIncrement,
MaxTransactionsPerBlock: c.MaxTransactionsPerBlock,
MemoryPoolMaxTransactions: c.MemPoolSize,
ValidatorsCount: byte(c.ValidatorsCount),
InitialGasDistribution: c.InitialGASSupply,
CommitteeHistory: c.CommitteeHistory,
P2PSigExtensions: c.P2PSigExtensions,
StateRootInHeader: c.StateRootInHeader,
ValidatorsHistory: c.ValidatorsHistory,
},
}, nil
}
func (l *localClient) InvokeContractVerify(contract util.Uint160, params []smartcontract.Parameter, signers []transaction.Signer, witnesses ...transaction.Witness) (*result.Invoke, error) {
// not used by `morph init` command
panic("unexpected call")
}
// CalculateNetworkFee calculates network fee for the given transaction.
// Copied from neo-go with minor corrections (no need to support non-notary mode):
// https://github.com/nspcc-dev/neo-go/blob/v0.99.2/pkg/services/rpcsrv/server.go#L744
func (l *localClient) CalculateNetworkFee(tx *transaction.Transaction) (int64, error) {
hashablePart, err := tx.EncodeHashableFields()
if err != nil {
return 0, fmt.Errorf("failed to compute tx size: %w", err)
}
size := len(hashablePart) + io.GetVarSize(len(tx.Signers))
ef := l.bc.GetBaseExecFee()
var netFee int64
for i, signer := range tx.Signers {
var verificationScript []byte
for _, w := range tx.Scripts {
if w.VerificationScript != nil && hash.Hash160(w.VerificationScript).Equals(signer.Account) {
verificationScript = w.VerificationScript
break
}
}
if verificationScript == nil {
gasConsumed, err := l.bc.VerifyWitness(signer.Account, tx, &tx.Scripts[i], l.maxGasInvoke)
if err != nil {
return 0, fmt.Errorf("invalid signature: %w", err)
}
netFee += gasConsumed
size += io.GetVarSize([]byte{}) + io.GetVarSize(tx.Scripts[i].InvocationScript)
continue
}
fee, sizeDelta := fee.Calculate(ef, verificationScript)
netFee += fee
size += sizeDelta
}
fee := l.bc.FeePerByte()
netFee += int64(size) * fee
return netFee, nil
}
// AddNetworkFee adds network fee for each witness script and optional extra
// network fee to transaction. `accs` is an array signer's accounts.
// Copied from neo-go with minor corrections (no need to support contract signers):
// https://github.com/nspcc-dev/neo-go/blob/6ff11baa1b9e4c71ef0d1de43b92a8c541ca732c/pkg/rpc/client/rpc.go#L960
func (l *localClient) AddNetworkFee(tx *transaction.Transaction, extraFee int64, accs ...*wallet.Account) error {
if len(tx.Signers) != len(accs) {
return errors.New("number of signers must match number of scripts")
}
size := io.GetVarSize(tx)
ef := l.bc.GetBaseExecFee()
for i := range tx.Signers {
netFee, sizeDelta := fee.Calculate(ef, accs[i].Contract.Script)
tx.NetworkFee += netFee
size += sizeDelta
}
tx.NetworkFee += int64(size)*l.bc.FeePerByte() + extraFee
return nil
}
// getSigners returns an array of transaction signers and corresponding accounts from
// given sender and cosigners. If cosigners list already contains sender, the sender
// will be placed at the start of the list.
// Copied from neo-go with minor corrections:
// https://github.com/nspcc-dev/neo-go/blob/6ff11baa1b9e4c71ef0d1de43b92a8c541ca732c/pkg/rpc/client/rpc.go#L735
func getSigners(sender *wallet.Account, cosigners []rpcclient.SignerAccount) ([]transaction.Signer, []*wallet.Account, error) {
var (
signers []transaction.Signer
accounts []*wallet.Account
)
from := sender.Contract.ScriptHash()
s := transaction.Signer{
Account: from,
Scopes: transaction.None,
}
for _, c := range cosigners {
if c.Signer.Account == from {
s = c.Signer
continue
}
signers = append(signers, c.Signer)
accounts = append(accounts, c.Account)
}
signers = append([]transaction.Signer{s}, signers...)
accounts = append([]*wallet.Account{sender}, accounts...)
return signers, accounts, nil
}
func (l *localClient) NEP17BalanceOf(h util.Uint160, acc util.Uint160) (int64, error) {
res, err := invokeFunction(l, h, "balanceOf", []any{acc}, nil)
if err != nil {
return 0, err
}
if res.State != vmstate.Halt.String() || len(res.Stack) == 0 {
return 0, fmt.Errorf("`balance`: invalid response (empty: %t): %s",
len(res.Stack) == 0, res.FaultException)
}
bi, err := res.Stack[0].TryInteger()
if err != nil || !bi.IsInt64() {
return 0, fmt.Errorf("`balance`: invalid response")
}
return bi.Int64(), nil
}
func (l *localClient) InvokeScript(script []byte, signers []transaction.Signer) (*result.Invoke, error) {
lastBlock, err := l.bc.GetBlock(l.bc.CurrentBlockHash())
if err != nil {
return nil, err
}
tx := transaction.New(script, 0)
tx.Signers = signers
tx.ValidUntilBlock = l.bc.BlockHeight() + 2
ic, err := l.bc.GetTestVM(trigger.Application, tx, &block.Block{
Header: block.Header{
Index: lastBlock.Index + 1,
Timestamp: lastBlock.Timestamp + 1,
},
})
if err != nil {
return nil, fmt.Errorf("get test VM: %w", err)
}
ic.VM.GasLimit = 100_0000_0000
ic.VM.LoadScriptWithFlags(script, callflag.All)
var errStr string
if err := ic.VM.Run(); err != nil {
errStr = err.Error()
}
return &result.Invoke{
State: ic.VM.State().String(),
GasConsumed: ic.VM.GasConsumed(),
Script: script,
Stack: ic.VM.Estack().ToArray(),
FaultException: errStr,
}, nil
}
func (l *localClient) SendRawTransaction(tx *transaction.Transaction) (util.Uint256, error) {
// We need to test that transaction was formed correctly to catch as many errors as we can.
bs := tx.Bytes()
_, err := transaction.NewTransactionFromBytes(bs)
if err != nil {
return tx.Hash(), fmt.Errorf("invalid transaction: %w", err)
}
l.transactions = append(l.transactions, tx)
return tx.Hash(), nil
}
func (l *localClient) putTransactions() error {
// 1. Prepare new block.
lastBlock, err := l.bc.GetBlock(l.bc.CurrentBlockHash())
if err != nil {
panic(err)
}
defer func() { l.transactions = l.transactions[:0] }()
b := &block.Block{
Header: block.Header{
NextConsensus: l.accounts[0].Contract.ScriptHash(),
Script: transaction.Witness{
VerificationScript: l.accounts[0].Contract.Script,
},
Timestamp: lastBlock.Timestamp + 1,
},
Transactions: l.transactions,
}
if l.bc.GetConfig().StateRootInHeader {
b.StateRootEnabled = true
b.PrevStateRoot = l.bc.GetStateModule().CurrentLocalStateRoot()
}
b.PrevHash = lastBlock.Hash()
b.Index = lastBlock.Index + 1
b.RebuildMerkleRoot()
// 2. Sign prepared block.
var invocationScript []byte
magic := l.bc.GetConfig().Magic
for _, acc := range l.accounts {
sign := acc.PrivateKey().SignHashable(uint32(magic), b)
invocationScript = append(invocationScript, byte(opcode.PUSHDATA1), 64)
invocationScript = append(invocationScript, sign...)
}
b.Script.InvocationScript = invocationScript
// 3. Persist block.
return l.bc.AddBlock(b)
}
func invokeFunction(c Client, h util.Uint160, method string, parameters []any, signers []transaction.Signer) (*result.Invoke, error) {
w := io.NewBufBinWriter()
emit.Array(w.BinWriter, parameters...)
emit.AppCallNoArgs(w.BinWriter, h, method, callflag.All)
if w.Err != nil {
panic(fmt.Sprintf("BUG: invalid parameters for '%s': %v", method, w.Err))
}
return c.InvokeScript(w.Bytes(), signers)
}
var errGetDesignatedByRoleResponse = errors.New("`getDesignatedByRole`: invalid response")
func getDesignatedByRole(inv *invoker.Invoker, h util.Uint160, role noderoles.Role, u uint32) (keys.PublicKeys, error) {
arr, err := unwrap.Array(inv.Call(h, "getDesignatedByRole", int64(role), int64(u)))
if err != nil {
return nil, errGetDesignatedByRoleResponse
}
pubs := make(keys.PublicKeys, len(arr))
for i := range arr {
bs, err := arr[i].TryBytes()
if err != nil {
return nil, errGetDesignatedByRoleResponse
}
pubs[i], err = keys.NewPublicKeyFromBytes(bs, elliptic.P256())
if err != nil {
return nil, errGetDesignatedByRoleResponse
}
}
return pubs, nil
}
func (l *localClient) dump() (err error) {
defer l.bc.Close()
f, err := os.Create(l.dumpPath)
if err != nil {
return err
}
defer func() {
closeErr := f.Close()
if err == nil && closeErr != nil {
err = closeErr
}
}()
w := io.NewBinWriterFromIO(f)
w.WriteU32LE(l.bc.BlockHeight() + 1)
err = chaindump.Dump(l.bc, w, 0, l.bc.BlockHeight()+1)
return
}

View file

@ -0,0 +1,122 @@
package morph
import (
"context"
"errors"
"fmt"
"time"
"github.com/nspcc-dev/neo-go/pkg/config/netmode"
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/fixedn"
"github.com/nspcc-dev/neo-go/pkg/neorpc/result"
"github.com/nspcc-dev/neo-go/pkg/network/payload"
"github.com/nspcc-dev/neo-go/pkg/rpcclient"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/actor"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/trigger"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
// Client represents N3 client interface capable of test-invoking scripts
// and sending signed transactions to chain.
type Client interface {
invoker.RPCInvoke
GetBlockCount() (uint32, error)
GetContractStateByID(int32) (*state.Contract, error)
GetContractStateByHash(util.Uint160) (*state.Contract, error)
GetNativeContracts() ([]state.NativeContract, error)
GetNetwork() (netmode.Magic, error)
GetApplicationLog(util.Uint256, *trigger.Type) (*result.ApplicationLog, error)
GetVersion() (*result.Version, error)
CreateTxFromScript([]byte, *wallet.Account, int64, int64, []rpcclient.SignerAccount) (*transaction.Transaction, error)
NEP17BalanceOf(util.Uint160, util.Uint160) (int64, error)
SendRawTransaction(*transaction.Transaction) (util.Uint256, error)
GetCommittee() (keys.PublicKeys, error)
CalculateNotaryFee(uint8) (int64, error)
CalculateNetworkFee(tx *transaction.Transaction) (int64, error)
AddNetworkFee(*transaction.Transaction, int64, ...*wallet.Account) error
SignAndPushInvocationTx([]byte, *wallet.Account, int64, fixedn.Fixed8, []rpcclient.SignerAccount) (util.Uint256, error)
SignAndPushP2PNotaryRequest(*transaction.Transaction, []byte, int64, int64, uint32, *wallet.Account) (*payload.P2PNotaryRequest, error)
}
type hashVUBPair struct {
hash util.Uint256
vub uint32
}
type clientContext struct {
Client Client // a raw neo-go client OR a local chain implementation
CommitteeAct *actor.Actor // committee actor with the Global witness scope
ReadOnlyInvoker *invoker.Invoker // R/O contract invoker, does not contain any signer
SentTxs []hashVUBPair
}
func getN3Client(v *viper.Viper) (Client, error) {
// number of opened connections
// by neo-go client per one host
const (
maxConnsPerHost = 10
requestTimeout = time.Second * 10
)
ctx := context.Background()
endpoint := v.GetString(endpointFlag)
if endpoint == "" {
return nil, errors.New("missing endpoint")
}
c, err := rpcclient.New(ctx, endpoint, rpcclient.Options{
MaxConnsPerHost: maxConnsPerHost,
RequestTimeout: requestTimeout,
})
if err != nil {
return nil, err
}
if err := c.Init(); err != nil {
return nil, err
}
return c, nil
}
func defaultClientContext(c Client, committeeAcc *wallet.Account) (*clientContext, error) {
commAct, err := actor.New(c, []actor.SignerAccount{{
Signer: transaction.Signer{
Account: committeeAcc.Contract.ScriptHash(),
Scopes: transaction.Global,
},
Account: committeeAcc,
}})
if err != nil {
return nil, err
}
return &clientContext{
Client: c,
CommitteeAct: commAct,
ReadOnlyInvoker: invoker.New(c, nil),
}, nil
}
func (c *clientContext) sendTx(tx *transaction.Transaction, cmd *cobra.Command, await bool) error {
h, err := c.Client.SendRawTransaction(tx)
if err != nil {
return err
}
if h != tx.Hash() {
return fmt.Errorf("sent and actual tx hashes mismatch:\n\tsent: %v\n\tactual: %v", tx.Hash().StringLE(), h.StringLE())
}
c.SentTxs = append(c.SentTxs, hashVUBPair{hash: h, vub: tx.ValidUntilBlock})
if await {
return c.awaitTx(cmd)
}
return nil
}

View file

@ -0,0 +1,29 @@
package morph
import (
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-node/pkg/morph/client/netmap"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
func listNetmapCandidatesNodes(cmd *cobra.Command, _ []string) {
c, err := getN3Client(viper.GetViper())
commonCmd.ExitOnErr(cmd, "can't create N3 client: %w", err)
inv := invoker.New(c, nil)
cs, err := c.GetContractStateByID(1)
commonCmd.ExitOnErr(cmd, "can't get NNS contract info: %w", err)
nmHash, err := nnsResolveHash(inv, cs.Hash, netmapContract+".frostfs")
commonCmd.ExitOnErr(cmd, "can't get netmap contract hash: %w", err)
res, err := inv.Call(nmHash, "netmapCandidates")
commonCmd.ExitOnErr(cmd, "can't fetch list of network config keys from the netmap contract", err)
nm, err := netmap.DecodeNetMap(res.Stack)
commonCmd.ExitOnErr(cmd, "unable to decode netmap: %w", err)
commonCmd.PrettyPrintNetMap(cmd, *nm, !viper.GetBool(commonflags.Verbose))
}

View file

@ -0,0 +1,121 @@
package morph
import (
"fmt"
"math/big"
"strconv"
"github.com/nspcc-dev/neo-go/cli/input"
"github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/actor"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/gas"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/nep17"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/notary"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
// defaultNotaryDepositLifetime is an amount of blocks notary deposit stays valid.
// https://github.com/nspcc-dev/neo-go/blob/master/pkg/core/native/notary.go#L48
const defaultNotaryDepositLifetime = 5760
func depositNotary(cmd *cobra.Command, _ []string) error {
p, err := cmd.Flags().GetString(storageWalletFlag)
if err != nil {
return err
} else if p == "" {
return fmt.Errorf("missing wallet path (use '--%s <out.json>')", storageWalletFlag)
}
w, err := wallet.NewWalletFromFile(p)
if err != nil {
return fmt.Errorf("can't open wallet: %v", err)
}
accHash := w.GetChangeAddress()
if addr, err := cmd.Flags().GetString(walletAccountFlag); err == nil {
accHash, err = address.StringToUint160(addr)
if err != nil {
return fmt.Errorf("invalid address: %s", addr)
}
}
acc := w.GetAccount(accHash)
if acc == nil {
return fmt.Errorf("can't find account for %s", accHash)
}
prompt := fmt.Sprintf("Enter password for %s >", address.Uint160ToString(accHash))
pass, err := input.ReadPassword(prompt)
if err != nil {
return fmt.Errorf("can't get password: %v", err)
}
err = acc.Decrypt(pass, keys.NEP2ScryptParams())
if err != nil {
return fmt.Errorf("can't unlock account: %v", err)
}
gasStr, err := cmd.Flags().GetString(refillGasAmountFlag)
if err != nil {
return err
}
gasAmount, err := parseGASAmount(gasStr)
if err != nil {
return err
}
till := int64(defaultNotaryDepositLifetime)
tillStr, err := cmd.Flags().GetString(notaryDepositTillFlag)
if err != nil {
return err
}
if tillStr != "" {
till, err = strconv.ParseInt(tillStr, 10, 64)
if err != nil || till <= 0 {
return fmt.Errorf("notary deposit lifetime must be a positive integer")
}
}
c, err := getN3Client(viper.GetViper())
if err != nil {
return err
}
if err := checkNotaryEnabled(c); err != nil {
return err
}
height, err := c.GetBlockCount()
if err != nil {
return fmt.Errorf("can't get current height: %v", err)
}
act, err := actor.New(c, []actor.SignerAccount{{
Signer: transaction.Signer{
Account: acc.Contract.ScriptHash(),
Scopes: transaction.Global,
},
Account: acc,
}})
if err != nil {
return fmt.Errorf("could not create actor: %w", err)
}
gasActor := nep17.New(act, gas.Hash)
txHash, vub, err := gasActor.Transfer(
accHash,
notary.Hash,
big.NewInt(int64(gasAmount)),
[]any{nil, int64(height) + till},
)
if err != nil {
return fmt.Errorf("could not send tx: %w", err)
}
return awaitTx(cmd, c, []hashVUBPair{{hash: txHash, vub: vub}})
}

View file

@ -0,0 +1,54 @@
package morph
import (
"fmt"
"strconv"
"strings"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/policy"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
const (
execFeeParam = "ExecFeeFactor"
storagePriceParam = "StoragePrice"
setFeeParam = "FeePerByte"
)
func setPolicyCmd(cmd *cobra.Command, args []string) error {
wCtx, err := newInitializeContext(cmd, viper.GetViper())
if err != nil {
return fmt.Errorf("can't to initialize context: %w", err)
}
bw := io.NewBufBinWriter()
for i := range args {
k, v, found := strings.Cut(args[i], "=")
if !found {
return fmt.Errorf("invalid parameter format, must be Parameter=Value")
}
switch k {
case execFeeParam, storagePriceParam, setFeeParam:
default:
return fmt.Errorf("parameter must be one of %s, %s and %s", execFeeParam, storagePriceParam, setFeeParam)
}
value, err := strconv.ParseUint(v, 10, 32)
if err != nil {
return fmt.Errorf("can't parse parameter value '%s': %w", args[1], err)
}
emit.AppCall(bw.BinWriter, policy.Hash, "set"+k, callflag.All, int64(value))
}
if err := wCtx.sendCommitteeTx(bw.Bytes(), false); err != nil {
return err
}
return wCtx.awaitTx()
}

View file

@ -0,0 +1,61 @@
package morph
import (
"errors"
"fmt"
netmapcontract "github.com/TrueCloudLab/frostfs-contract/netmap"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
func removeNodesCmd(cmd *cobra.Command, args []string) error {
if len(args) == 0 {
return errors.New("at least one node key must be provided")
}
nodeKeys := make(keys.PublicKeys, len(args))
for i := range args {
var err error
nodeKeys[i], err = keys.NewPublicKeyFromString(args[i])
if err != nil {
return fmt.Errorf("can't parse node public key: %w", err)
}
}
wCtx, err := newInitializeContext(cmd, viper.GetViper())
if err != nil {
return fmt.Errorf("can't initialize context: %w", err)
}
defer wCtx.close()
cs, err := wCtx.Client.GetContractStateByID(1)
if err != nil {
return fmt.Errorf("can't get NNS contract info: %w", err)
}
nmHash, err := nnsResolveHash(wCtx.ReadOnlyInvoker, cs.Hash, netmapContract+".frostfs")
if err != nil {
return fmt.Errorf("can't get netmap contract hash: %w", err)
}
bw := io.NewBufBinWriter()
for i := range nodeKeys {
emit.AppCall(bw.BinWriter, nmHash, "updateStateIR", callflag.All,
int64(netmapcontract.NodeStateOffline), nodeKeys[i].Bytes())
}
if err := emitNewEpochCall(bw, wCtx, nmHash); err != nil {
return err
}
if err := wCtx.sendConsensusTx(bw.Bytes()); err != nil {
return err
}
return wCtx.awaitTx()
}

View file

@ -0,0 +1,339 @@
package morph
import (
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
const (
alphabetWalletsFlag = "alphabet-wallets"
alphabetSizeFlag = "size"
endpointFlag = "rpc-endpoint"
storageWalletFlag = "storage-wallet"
storageWalletLabelFlag = "label"
storageGasCLIFlag = "initial-gas"
storageGasConfigFlag = "storage.initial_gas"
contractsInitFlag = "contracts"
maxObjectSizeInitFlag = "network.max_object_size"
maxObjectSizeCLIFlag = "max-object-size"
epochDurationInitFlag = "network.epoch_duration"
epochDurationCLIFlag = "epoch-duration"
incomeRateInitFlag = "network.basic_income_rate"
incomeRateCLIFlag = "basic-income-rate"
auditFeeInitFlag = "network.fee.audit"
auditFeeCLIFlag = "audit-fee"
containerFeeInitFlag = "network.fee.container"
containerAliasFeeInitFlag = "network.fee.container_alias"
containerFeeCLIFlag = "container-fee"
containerAliasFeeCLIFlag = "container-alias-fee"
candidateFeeInitFlag = "network.fee.candidate"
candidateFeeCLIFlag = "candidate-fee"
homomorphicHashDisabledInitFlag = "network.homomorphic_hash_disabled"
maintenanceModeAllowedInitFlag = "network.maintenance_mode_allowed"
homomorphicHashDisabledCLIFlag = "homomorphic-disabled"
withdrawFeeInitFlag = "network.fee.withdraw"
withdrawFeeCLIFlag = "withdraw-fee"
containerDumpFlag = "dump"
containerContractFlag = "container-contract"
containerIDsFlag = "cid"
refillGasAmountFlag = "gas"
walletAccountFlag = "account"
notaryDepositTillFlag = "till"
localDumpFlag = "local-dump"
protoConfigPath = "protocol"
walletAddressFlag = "wallet-address"
)
var (
// RootCmd is a root command of config section.
RootCmd = &cobra.Command{
Use: "morph",
Short: "Section for morph network configuration commands",
}
generateAlphabetCmd = &cobra.Command{
Use: "generate-alphabet",
Short: "Generate alphabet wallets for consensus nodes of the morph network",
PreRun: func(cmd *cobra.Command, _ []string) {
// PreRun fixes https://github.com/spf13/viper/issues/233
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
},
RunE: generateAlphabetCreds,
}
initCmd = &cobra.Command{
Use: "init",
Short: "Initialize side chain network with smart-contracts and network settings",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
_ = viper.BindPFlag(epochDurationInitFlag, cmd.Flags().Lookup(epochDurationCLIFlag))
_ = viper.BindPFlag(maxObjectSizeInitFlag, cmd.Flags().Lookup(maxObjectSizeCLIFlag))
_ = viper.BindPFlag(incomeRateInitFlag, cmd.Flags().Lookup(incomeRateCLIFlag))
_ = viper.BindPFlag(homomorphicHashDisabledInitFlag, cmd.Flags().Lookup(homomorphicHashDisabledCLIFlag))
_ = viper.BindPFlag(auditFeeInitFlag, cmd.Flags().Lookup(auditFeeCLIFlag))
_ = viper.BindPFlag(candidateFeeInitFlag, cmd.Flags().Lookup(candidateFeeCLIFlag))
_ = viper.BindPFlag(containerFeeInitFlag, cmd.Flags().Lookup(containerFeeCLIFlag))
_ = viper.BindPFlag(containerAliasFeeInitFlag, cmd.Flags().Lookup(containerAliasFeeCLIFlag))
_ = viper.BindPFlag(withdrawFeeInitFlag, cmd.Flags().Lookup(withdrawFeeCLIFlag))
_ = viper.BindPFlag(protoConfigPath, cmd.Flags().Lookup(protoConfigPath))
_ = viper.BindPFlag(localDumpFlag, cmd.Flags().Lookup(localDumpFlag))
},
RunE: initializeSideChainCmd,
}
generateStorageCmd = &cobra.Command{
Use: "generate-storage-wallet",
Short: "Generate storage node wallet for the morph network",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
_ = viper.BindPFlag(storageGasConfigFlag, cmd.Flags().Lookup(storageGasCLIFlag))
},
RunE: generateStorageCreds,
}
refillGasCmd = &cobra.Command{
Use: "refill-gas",
Short: "Refill GAS of storage node's wallet in the morph network",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
_ = viper.BindPFlag(refillGasAmountFlag, cmd.Flags().Lookup(refillGasAmountFlag))
},
RunE: func(cmd *cobra.Command, args []string) error {
return refillGas(cmd, refillGasAmountFlag, false)
},
}
forceNewEpoch = &cobra.Command{
Use: "force-new-epoch",
Short: "Create new FrostFS epoch event in the side chain",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
RunE: forceNewEpochCmd,
}
removeNodes = &cobra.Command{
Use: "remove-nodes key1 [key2 [...]]",
Short: "Remove storage nodes from the netmap",
Long: `Move nodes to the Offline state in the candidates list and tick an epoch to update the netmap`,
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
RunE: removeNodesCmd,
}
setConfig = &cobra.Command{
Use: "set-config key1=val1 [key2=val2 ...]",
DisableFlagsInUseLine: true,
Short: "Add/update global config value in the FrostFS network",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
Args: cobra.MinimumNArgs(1),
RunE: setConfigCmd,
}
setPolicy = &cobra.Command{
Use: "set-policy [ExecFeeFactor=<n1>] [StoragePrice=<n2>] [FeePerByte=<n3>]",
DisableFlagsInUseLine: true,
Short: "Set global policy values",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
RunE: setPolicyCmd,
ValidArgsFunction: func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
return []string{"ExecFeeFactor=", "StoragePrice=", "FeePerByte="}, cobra.ShellCompDirectiveNoSpace
},
}
dumpContractHashesCmd = &cobra.Command{
Use: "dump-hashes",
Short: "Dump deployed contract hashes",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
RunE: dumpContractHashes,
}
dumpNetworkConfigCmd = &cobra.Command{
Use: "dump-config",
Short: "Dump FrostFS network config",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
RunE: dumpNetworkConfig,
}
dumpBalancesCmd = &cobra.Command{
Use: "dump-balances",
Short: "Dump GAS balances",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
RunE: dumpBalances,
}
updateContractsCmd = &cobra.Command{
Use: "update-contracts",
Short: "Update FrostFS contracts",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
RunE: updateContracts,
}
dumpContainersCmd = &cobra.Command{
Use: "dump-containers",
Short: "Dump FrostFS containers to file",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
RunE: dumpContainers,
}
restoreContainersCmd = &cobra.Command{
Use: "restore-containers",
Short: "Restore FrostFS containers from file",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
RunE: restoreContainers,
}
listContainersCmd = &cobra.Command{
Use: "list-containers",
Short: "List FrostFS containers",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
RunE: listContainers,
}
depositNotaryCmd = &cobra.Command{
Use: "deposit-notary",
Short: "Deposit GAS for notary service",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
RunE: depositNotary,
}
netmapCandidatesCmd = &cobra.Command{
Use: "netmap-candidates",
Short: "List netmap candidates nodes",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
},
Run: listNetmapCandidatesNodes,
}
)
func init() {
RootCmd.AddCommand(generateAlphabetCmd)
generateAlphabetCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
generateAlphabetCmd.Flags().Uint(alphabetSizeFlag, 7, "Amount of alphabet wallets to generate")
RootCmd.AddCommand(initCmd)
initCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
initCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
initCmd.Flags().String(contractsInitFlag, "", "Path to archive with compiled FrostFS contracts (default fetched from latest github release)")
initCmd.Flags().Uint(epochDurationCLIFlag, 240, "Amount of side chain blocks in one FrostFS epoch")
initCmd.Flags().Uint(maxObjectSizeCLIFlag, 67108864, "Max single object size in bytes")
initCmd.Flags().Bool(homomorphicHashDisabledCLIFlag, false, "Disable object homomorphic hashing")
// Defaults are taken from neo-preodolenie.
initCmd.Flags().Uint64(containerFeeCLIFlag, 1000, "Container registration fee")
initCmd.Flags().Uint64(containerAliasFeeCLIFlag, 500, "Container alias fee")
initCmd.Flags().String(protoConfigPath, "", "Path to the consensus node configuration")
initCmd.Flags().String(localDumpFlag, "", "Path to the blocks dump file")
RootCmd.AddCommand(deployCmd)
RootCmd.AddCommand(generateStorageCmd)
generateStorageCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
generateStorageCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
generateStorageCmd.Flags().String(storageWalletFlag, "", "Path to new storage node wallet")
generateStorageCmd.Flags().String(storageGasCLIFlag, "", "Initial amount of GAS to transfer")
generateStorageCmd.Flags().StringP(storageWalletLabelFlag, "l", "", "Wallet label")
RootCmd.AddCommand(forceNewEpoch)
forceNewEpoch.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
forceNewEpoch.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
RootCmd.AddCommand(removeNodes)
removeNodes.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
removeNodes.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
RootCmd.AddCommand(setPolicy)
setPolicy.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
setPolicy.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
RootCmd.AddCommand(dumpContractHashesCmd)
dumpContractHashesCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
dumpContractHashesCmd.Flags().String(customZoneFlag, "", "Custom zone to search.")
RootCmd.AddCommand(dumpNetworkConfigCmd)
dumpNetworkConfigCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
RootCmd.AddCommand(setConfig)
setConfig.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
setConfig.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
setConfig.Flags().Bool(forceConfigSet, false, "Force setting not well-known configuration key")
RootCmd.AddCommand(dumpBalancesCmd)
dumpBalancesCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
dumpBalancesCmd.Flags().BoolP(dumpBalancesStorageFlag, "s", false, "Dump balances of storage nodes from the current netmap")
dumpBalancesCmd.Flags().BoolP(dumpBalancesAlphabetFlag, "a", false, "Dump balances of alphabet contracts")
dumpBalancesCmd.Flags().BoolP(dumpBalancesProxyFlag, "p", false, "Dump balances of the proxy contract")
dumpBalancesCmd.Flags().Bool(dumpBalancesUseScriptHashFlag, false, "Use script-hash format for addresses")
RootCmd.AddCommand(updateContractsCmd)
updateContractsCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
updateContractsCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
updateContractsCmd.Flags().String(contractsInitFlag, "", "Path to archive with compiled FrostFS contracts (default fetched from latest github release)")
RootCmd.AddCommand(dumpContainersCmd)
dumpContainersCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
dumpContainersCmd.Flags().String(containerDumpFlag, "", "File where to save dumped containers")
dumpContainersCmd.Flags().String(containerContractFlag, "", "Container contract hash (for networks without NNS)")
dumpContainersCmd.Flags().StringSlice(containerIDsFlag, nil, "Containers to dump")
RootCmd.AddCommand(restoreContainersCmd)
restoreContainersCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
restoreContainersCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
restoreContainersCmd.Flags().String(containerDumpFlag, "", "File to restore containers from")
restoreContainersCmd.Flags().StringSlice(containerIDsFlag, nil, "Containers to restore")
RootCmd.AddCommand(listContainersCmd)
listContainersCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
listContainersCmd.Flags().String(containerContractFlag, "", "Container contract hash (for networks without NNS)")
RootCmd.AddCommand(refillGasCmd)
refillGasCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
refillGasCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
refillGasCmd.Flags().String(storageWalletFlag, "", "Path to storage node wallet")
refillGasCmd.Flags().String(walletAddressFlag, "", "Address of wallet")
refillGasCmd.Flags().String(refillGasAmountFlag, "", "Additional amount of GAS to transfer")
refillGasCmd.MarkFlagsMutuallyExclusive(walletAddressFlag, storageWalletFlag)
RootCmd.AddCommand(cmdSubnet)
RootCmd.AddCommand(depositNotaryCmd)
depositNotaryCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
depositNotaryCmd.Flags().String(storageWalletFlag, "", "Path to storage node wallet")
depositNotaryCmd.Flags().String(walletAccountFlag, "", "Wallet account address")
depositNotaryCmd.Flags().String(refillGasAmountFlag, "", "Amount of GAS to deposit")
depositNotaryCmd.Flags().String(notaryDepositTillFlag, "", "Notary deposit duration in blocks")
RootCmd.AddCommand(netmapCandidatesCmd)
netmapCandidatesCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
}

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,21 @@
package morph
import (
"fmt"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
func updateContracts(cmd *cobra.Command, _ []string) error {
wCtx, err := newInitializeContext(cmd, viper.GetViper())
if err != nil {
return fmt.Errorf("initialization error: %w", err)
}
if err := wCtx.deployNNS(updateMethodName); err != nil {
return err
}
return wCtx.updateContracts()
}

View file

@ -0,0 +1,85 @@
package modules
import (
"os"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/storagecfg"
"github.com/TrueCloudLab/frostfs-node/misc"
"github.com/TrueCloudLab/frostfs-node/pkg/util/autocomplete"
utilConfig "github.com/TrueCloudLab/frostfs-node/pkg/util/config"
"github.com/TrueCloudLab/frostfs-node/pkg/util/gendoc"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
var (
rootCmd = &cobra.Command{
Use: "frostfs-adm",
Short: "FrostFS Administrative Tool",
Long: `FrostFS Administrative Tool provides functions to setup and
manage FrostFS network deployment.`,
RunE: entryPoint,
SilenceUsage: true,
}
)
func init() {
cobra.OnInitialize(func() { initConfig(rootCmd) })
// we need to init viper config to bind viper and cobra configurations for
// rpc endpoint, alphabet wallet dir, key credentials, etc.
// use stdout as default output for cmd.Print()
rootCmd.SetOut(os.Stdout)
rootCmd.PersistentFlags().StringP(commonflags.ConfigFlag, commonflags.ConfigFlagShorthand, "", commonflags.ConfigFlagUsage)
rootCmd.PersistentFlags().String(commonflags.ConfigDirFlag, "", commonflags.ConfigDirFlagUsage)
rootCmd.PersistentFlags().BoolP(commonflags.Verbose, commonflags.VerboseShorthand, false, commonflags.VerboseUsage)
_ = viper.BindPFlag(commonflags.Verbose, rootCmd.PersistentFlags().Lookup(commonflags.Verbose))
rootCmd.Flags().Bool("version", false, "Application version")
rootCmd.AddCommand(config.RootCmd)
rootCmd.AddCommand(morph.RootCmd)
rootCmd.AddCommand(storagecfg.RootCmd)
rootCmd.AddCommand(autocomplete.Command("frostfs-adm"))
rootCmd.AddCommand(gendoc.Command(rootCmd))
}
func Execute() error {
return rootCmd.Execute()
}
func entryPoint(cmd *cobra.Command, args []string) error {
printVersion, _ := cmd.Flags().GetBool("version")
if printVersion {
cmd.Print(misc.BuildInfo("FrostFS Adm"))
return nil
}
return cmd.Usage()
}
func initConfig(cmd *cobra.Command) {
configFile, err := cmd.Flags().GetString(commonflags.ConfigFlag)
if err != nil {
return
}
if configFile != "" {
viper.SetConfigType("yml")
viper.SetConfigFile(configFile)
_ = viper.ReadInConfig() // if config file is set but unavailable, ignore it
}
configDir, err := cmd.Flags().GetString(commonflags.ConfigDirFlag)
if err != nil {
return
}
if configDir != "" {
_ = utilConfig.ReadConfigDir(viper.GetViper(), configDir) // if config files cannot be read, ignore it
}
}

View file

@ -0,0 +1,138 @@
package storagecfg
const configTemplate = `logger:
level: info # logger level: one of "debug", "info" (default), "warn", "error", "dpanic", "panic", "fatal"
node:
wallet:
path: {{ .Wallet.Path }} # path to a NEO wallet; ignored if key is presented
address: {{ .Wallet.Account }} # address of a NEO account in the wallet; ignored if key is presented
password: {{ .Wallet.Password }} # password for a NEO account in the wallet; ignored if key is presented
addresses: # list of addresses announced by Storage node in the Network map
- {{ .AnnouncedAddress }}
attribute_0: UN-LOCODE:{{ .Attribute.Locode }}
relay: {{ .Relay }} # start Storage node in relay mode without bootstrapping into the Network map
subnet:
exit_zero: false # toggle entrance to zero subnet (overrides corresponding attribute and occurrence in entries)
entries: [] # list of IDs of subnets to enter in a text format of FrostFS API protocol (overrides corresponding attributes)
grpc:
num: 1 # total number of listener endpoints
0:
endpoint: {{ .Endpoint }} # endpoint for gRPC server
tls:{{if .TLSCert}}
enabled: true # enable TLS for a gRPC connection (min version is TLS 1.2)
certificate: {{ .TLSCert }} # path to TLS certificate
key: {{ .TLSKey }} # path to TLS key
{{- else }}
enabled: false # disable TLS for a gRPC connection
{{- end}}
control:
authorized_keys: # list of hex-encoded public keys that have rights to use the Control Service
{{- range .AuthorizedKeys }}
- {{.}}{{end}}
grpc:
endpoint: {{.ControlEndpoint}} # endpoint that is listened by the Control Service
morph:
dial_timeout: 20s # timeout for side chain NEO RPC client connection
cache_ttl: 15s # use TTL cache for side chain GET operations
rpc_endpoint: # side chain N3 RPC endpoints
{{- range .MorphRPC }}
- address: wss://{{.}}/ws{{end}}
{{if not .Relay }}
storage:
shard_pool_size: 15 # size of per-shard worker pools used for PUT operations
shard:
default: # section with the default shard parameters
metabase:
perm: 0644 # permissions for metabase files(directories: +x for current user and group)
blobstor:
perm: 0644 # permissions for blobstor files(directories: +x for current user and group)
depth: 2 # max depth of object tree storage in FS
small_object_size: 102400 # 100KiB, size threshold for "small" objects which are stored in key-value DB, not in FS, bytes
compress: true # turn on/off Zstandard compression (level 3) of stored objects
compression_exclude_content_types:
- audio/*
- video/*
blobovnicza:
size: 1073741824 # approximate size limit of single blobovnicza instance, total size will be: size*width^(depth+1), bytes
depth: 1 # max depth of object tree storage in key-value DB
width: 4 # max width of object tree storage in key-value DB
opened_cache_capacity: 50 # maximum number of opened database files
gc:
remover_batch_size: 200 # number of objects to be removed by the garbage collector
remover_sleep_interval: 5m # frequency of the garbage collector invocation
0:
mode: "read-write" # mode of the shard, must be one of the: "read-write" (default), "read-only"
metabase:
path: {{ .MetabasePath }} # path to the metabase
blobstor:
path: {{ .BlobstorPath }} # path to the blobstor
{{end}}`
const (
neofsMainnetAddress = "2cafa46838e8b564468ebd868dcafdd99dce6221"
balanceMainnetAddress = "dc1ec98d9d0c5f9dfade16144defe08cffc5ca55"
neofsTestnetAddress = "b65d8243ac63983206d17e5221af0653a7266fa1"
balanceTestnetAddress = "e0420c216003747626670d1424569c17c79015bf"
)
var n3config = map[string]struct {
MorphRPC []string
RPC []string
NeoFSContract string
BalanceContract string
}{
"testnet": {
MorphRPC: []string{
"rpc01.morph.testnet.fs.neo.org:51331",
"rpc02.morph.testnet.fs.neo.org:51331",
"rpc03.morph.testnet.fs.neo.org:51331",
"rpc04.morph.testnet.fs.neo.org:51331",
"rpc05.morph.testnet.fs.neo.org:51331",
"rpc06.morph.testnet.fs.neo.org:51331",
"rpc07.morph.testnet.fs.neo.org:51331",
},
RPC: []string{
"rpc01.testnet.n3.nspcc.ru:21331",
"rpc02.testnet.n3.nspcc.ru:21331",
"rpc03.testnet.n3.nspcc.ru:21331",
"rpc04.testnet.n3.nspcc.ru:21331",
"rpc05.testnet.n3.nspcc.ru:21331",
"rpc06.testnet.n3.nspcc.ru:21331",
"rpc07.testnet.n3.nspcc.ru:21331",
},
NeoFSContract: neofsTestnetAddress,
BalanceContract: balanceTestnetAddress,
},
"mainnet": {
MorphRPC: []string{
"rpc1.morph.fs.neo.org:40341",
"rpc2.morph.fs.neo.org:40341",
"rpc3.morph.fs.neo.org:40341",
"rpc4.morph.fs.neo.org:40341",
"rpc5.morph.fs.neo.org:40341",
"rpc6.morph.fs.neo.org:40341",
"rpc7.morph.fs.neo.org:40341",
},
RPC: []string{
"rpc1.n3.nspcc.ru:10331",
"rpc2.n3.nspcc.ru:10331",
"rpc3.n3.nspcc.ru:10331",
"rpc4.n3.nspcc.ru:10331",
"rpc5.n3.nspcc.ru:10331",
"rpc6.n3.nspcc.ru:10331",
"rpc7.n3.nspcc.ru:10331",
},
NeoFSContract: neofsMainnetAddress,
BalanceContract: balanceMainnetAddress,
},
}

View file

@ -0,0 +1,417 @@
package storagecfg
import (
"bytes"
"context"
"encoding/hex"
"errors"
"fmt"
"math/rand"
"net"
"net/url"
"os"
"path/filepath"
"strconv"
"strings"
"text/template"
"time"
netutil "github.com/TrueCloudLab/frostfs-node/pkg/network"
"github.com/chzyer/readline"
"github.com/nspcc-dev/neo-go/cli/flags"
"github.com/nspcc-dev/neo-go/cli/input"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/nspcc-dev/neo-go/pkg/encoding/fixedn"
"github.com/nspcc-dev/neo-go/pkg/rpcclient"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/actor"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/gas"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/nep17"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/trigger"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/cobra"
)
const (
walletFlag = "wallet"
accountFlag = "account"
)
const (
defaultControlEndpoint = "localhost:8090"
defaultDataEndpoint = "localhost"
)
// RootCmd is a root command of config section.
var RootCmd = &cobra.Command{
Use: "storage-config [-w wallet] [-a acccount] [<path-to-config>]",
Short: "Section for storage node configuration commands",
Run: storageConfig,
}
func init() {
fs := RootCmd.Flags()
fs.StringP(walletFlag, "w", "", "Path to wallet")
fs.StringP(accountFlag, "a", "", "Wallet account")
}
type config struct {
AnnouncedAddress string
AuthorizedKeys []string
ControlEndpoint string
Endpoint string
TLSCert string
TLSKey string
MorphRPC []string
Attribute struct {
Locode string
}
Wallet struct {
Path string
Account string
Password string
}
Relay bool
BlobstorPath string
MetabasePath string
}
func storageConfig(cmd *cobra.Command, args []string) {
var outPath string
if len(args) != 0 {
outPath = args[0]
} else {
outPath = getPath("File to write config at [./config.yml]: ")
if outPath == "" {
outPath = "./config.yml"
}
}
historyPath := filepath.Join(os.TempDir(), "frostfs-adm.history")
readline.SetHistoryPath(historyPath)
var c config
c.Wallet.Path, _ = cmd.Flags().GetString(walletFlag)
if c.Wallet.Path == "" {
c.Wallet.Path = getPath("Path to the storage node wallet: ")
}
w, err := wallet.NewWalletFromFile(c.Wallet.Path)
fatalOnErr(err)
c.Wallet.Account, _ = cmd.Flags().GetString(accountFlag)
if c.Wallet.Account == "" {
addr := address.Uint160ToString(w.GetChangeAddress())
c.Wallet.Account = getWalletAccount(w, fmt.Sprintf("Wallet account [%s]: ", addr))
if c.Wallet.Account == "" {
c.Wallet.Account = addr
}
}
accH, err := flags.ParseAddress(c.Wallet.Account)
fatalOnErr(err)
acc := w.GetAccount(accH)
if acc == nil {
fatalOnErr(errors.New("can't find account in wallet"))
}
c.Wallet.Password, err = input.ReadPassword(fmt.Sprintf("Account password for %s: ", c.Wallet.Account))
fatalOnErr(err)
err = acc.Decrypt(c.Wallet.Password, keys.NEP2ScryptParams())
fatalOnErr(err)
c.AuthorizedKeys = append(c.AuthorizedKeys, hex.EncodeToString(acc.PrivateKey().PublicKey().Bytes()))
var network string
for {
network = getString("Choose network [mainnet]/testnet: ")
switch network {
case "":
network = "mainnet"
case "testnet", "mainnet":
default:
cmd.Println(`Network must be either "mainnet" or "testnet"`)
continue
}
break
}
c.MorphRPC = n3config[network].MorphRPC
depositGas(cmd, acc, network)
c.Attribute.Locode = getString("UN-LOCODE attribute in [XX YYY] format: ")
var addr, port string
for {
c.AnnouncedAddress = getString("Publicly announced address: ")
validator := netutil.Address{}
err := validator.FromString(c.AnnouncedAddress)
if err != nil {
cmd.Println("Incorrect address format. See https://github.com/TrueCloudLab/frostfs-node/blob/master/pkg/network/address.go for details.")
continue
}
uriAddr, err := url.Parse(validator.URIAddr())
if err != nil {
panic(fmt.Errorf("unexpected error: %w", err))
}
addr = uriAddr.Hostname()
port = uriAddr.Port()
ip, err := net.ResolveIPAddr("ip", addr)
if err != nil {
cmd.Printf("Can't resolve IP address %s: %v\n", addr, err)
continue
}
if !ip.IP.IsGlobalUnicast() {
cmd.Println("IP must be global unicast.")
continue
}
cmd.Printf("Resolved IP address: %s\n", ip.String())
_, err = strconv.ParseUint(port, 10, 16)
if err != nil {
cmd.Println("Port must be an integer.")
continue
}
break
}
defaultAddr := net.JoinHostPort(defaultDataEndpoint, port)
c.Endpoint = getString(fmt.Sprintf("Listening address [%s]: ", defaultAddr))
if c.Endpoint == "" {
c.Endpoint = defaultAddr
}
c.ControlEndpoint = getString(fmt.Sprintf("Listening address (control endpoint) [%s]: ", defaultControlEndpoint))
if c.ControlEndpoint == "" {
c.ControlEndpoint = defaultControlEndpoint
}
c.TLSCert = getPath("TLS Certificate (optional): ")
if c.TLSCert != "" {
c.TLSKey = getPath("TLS Key: ")
}
c.Relay = getConfirmation(false, "Use node as a relay? yes/[no]: ")
if !c.Relay {
p := getPath("Path to the storage directory (all available storage will be used): ")
c.BlobstorPath = filepath.Join(p, "blob")
c.MetabasePath = filepath.Join(p, "meta")
}
out := applyTemplate(c)
fatalOnErr(os.WriteFile(outPath, out, 0644))
cmd.Println("Node is ready for work! Run `frostfs-node -config " + outPath + "`")
}
func getWalletAccount(w *wallet.Wallet, prompt string) string {
addrs := make([]readline.PrefixCompleterInterface, len(w.Accounts))
for i := range w.Accounts {
addrs[i] = readline.PcItem(w.Accounts[i].Address)
}
readline.SetAutoComplete(readline.NewPrefixCompleter(addrs...))
defer readline.SetAutoComplete(nil)
s, err := readline.Line(prompt)
fatalOnErr(err)
return strings.TrimSpace(s) // autocompleter can return a string with a trailing space
}
func getString(prompt string) string {
s, err := readline.Line(prompt)
fatalOnErr(err)
if s != "" {
_ = readline.AddHistory(s)
}
return s
}
type filenameCompleter struct{}
func (filenameCompleter) Do(line []rune, pos int) (newLine [][]rune, length int) {
prefix := string(line[:pos])
dir := filepath.Dir(prefix)
de, err := os.ReadDir(dir)
if err != nil {
return nil, 0
}
for i := range de {
name := filepath.Join(dir, de[i].Name())
if strings.HasPrefix(name, prefix) {
tail := []rune(strings.TrimPrefix(name, prefix))
if de[i].IsDir() {
tail = append(tail, filepath.Separator)
}
newLine = append(newLine, tail)
}
}
if pos != 0 {
return newLine, pos - len([]rune(dir))
}
return newLine, 0
}
func getPath(prompt string) string {
readline.SetAutoComplete(filenameCompleter{})
defer readline.SetAutoComplete(nil)
p, err := readline.Line(prompt)
fatalOnErr(err)
if p == "" {
return p
}
_ = readline.AddHistory(p)
abs, err := filepath.Abs(p)
if err != nil {
fatalOnErr(fmt.Errorf("can't create an absolute path: %w", err))
}
return abs
}
func getConfirmation(def bool, prompt string) bool {
for {
s, err := readline.Line(prompt)
fatalOnErr(err)
switch strings.ToLower(s) {
case "y", "yes":
return true
case "n", "no":
return false
default:
if len(s) == 0 {
return def
}
}
}
}
func applyTemplate(c config) []byte {
tmpl, err := template.New("config").Parse(configTemplate)
fatalOnErr(err)
b := bytes.NewBuffer(nil)
fatalOnErr(tmpl.Execute(b, c))
return b.Bytes()
}
func fatalOnErr(err error) {
if err != nil {
_, _ = fmt.Fprintf(os.Stderr, "Error: %v\n", err)
os.Exit(1)
}
}
func depositGas(cmd *cobra.Command, acc *wallet.Account, network string) {
sideClient := initClient(n3config[network].MorphRPC)
balanceHash, _ := util.Uint160DecodeStringLE(n3config[network].BalanceContract)
sideActor, err := actor.NewSimple(sideClient, acc)
if err != nil {
fatalOnErr(fmt.Errorf("creating actor over side chain client: %w", err))
}
sideGas := nep17.NewReader(sideActor, balanceHash)
accSH := acc.Contract.ScriptHash()
balance, err := sideGas.BalanceOf(accSH)
if err != nil {
fatalOnErr(fmt.Errorf("side chain balance: %w", err))
}
ok := getConfirmation(false, fmt.Sprintf("Current NeoFS balance is %s, make a deposit? y/[n]: ",
fixedn.ToString(balance, 12)))
if !ok {
return
}
amountStr := getString("Enter amount in GAS: ")
amount, err := fixedn.FromString(amountStr, 8)
if err != nil {
fatalOnErr(fmt.Errorf("invalid amount: %w", err))
}
mainClient := initClient(n3config[network].RPC)
neofsHash, _ := util.Uint160DecodeStringLE(n3config[network].NeoFSContract)
mainActor, err := actor.NewSimple(mainClient, acc)
if err != nil {
fatalOnErr(fmt.Errorf("creating actor over main chain client: %w", err))
}
mainGas := nep17.New(mainActor, gas.Hash)
txHash, _, err := mainGas.Transfer(accSH, neofsHash, amount, nil)
if err != nil {
fatalOnErr(fmt.Errorf("sending TX to the NeoFS contract: %w", err))
}
cmd.Print("Waiting for transactions to persist.")
tick := time.NewTicker(time.Second / 2)
defer tick.Stop()
timer := time.NewTimer(time.Second * 20)
defer timer.Stop()
at := trigger.Application
loop:
for {
select {
case <-tick.C:
_, err := mainClient.GetApplicationLog(txHash, &at)
if err == nil {
cmd.Print("\n")
break loop
}
cmd.Print(".")
case <-timer.C:
cmd.Printf("\nTimeout while waiting for transaction to persist.\n")
if getConfirmation(false, "Continue configuration? yes/[no]: ") {
return
}
os.Exit(1)
}
}
}
func initClient(rpc []string) *rpcclient.Client {
var c *rpcclient.Client
var err error
shuffled := make([]string, len(rpc))
copy(shuffled, rpc)
rand.Shuffle(len(shuffled), func(i, j int) { shuffled[i], shuffled[j] = shuffled[j], shuffled[i] })
for _, endpoint := range shuffled {
c, err = rpcclient.New(context.Background(), "https://"+endpoint, rpcclient.Options{
DialTimeout: time.Second * 2,
RequestTimeout: time.Second * 5,
})
if err != nil {
continue
}
if err = c.Init(); err != nil {
continue
}
return c
}
fatalOnErr(fmt.Errorf("can't create N3 client: %w", err))
panic("unreachable")
}

13
cmd/frostfs-adm/main.go Normal file
View file

@ -0,0 +1,13 @@
package main
import (
"os"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules"
)
func main() {
if err := modules.Execute(); err != nil {
os.Exit(1)
}
}

View file

@ -0,0 +1,75 @@
# How FrostFS CLI uses session mechanism of the FrostFS
## Overview
FrostFS sessions implement a mechanism for issuing a power of attorney by one
party to another. A trusted party can provide a so-called session token as
proof of the right to act on behalf of another member of the network. The
client of operations carried out with such a token will be the user who opened
the session. The token contains information which limits power of attorney like
action context or lifetime.
The client confirms trust in a third party by signing its public (session) key
with his private key. Any operation signed using private session key with
attached session token is treated as performed by the original client.
## Types
FrostFS CLI supports two ways to execute operation within a session depending on
whether the user of the command application is an original user (1) or a trusted
one (2).
### Dynamic
For case (1) CLI user can only open dynamic sessions. Protocol call
`SessionService.Create` is used for this purpose. As a result of the call, a
private session key will be generated on the server, thus making the remote
server trusted. This type of session is useful when the client needs to
transfer part of the responsibility for the formation of strict system elements
to the trusted server. At the moment, the approach is applicable only to
creating objects.
```shell
$ frostfs-cli session create --rpc-endpoint <server_ip> --out ./blank_token
```
After this example command remote node holds session private key while its
public part is written into the session token encoded into the output file.
Later this token can be attached to the operations which support dynamic
sessions. Then the token will be finally formed and signed by CLI itself.
### Static
For case (2) CLI user can act on behalf of the person who issued the session
token to him. Unlike (1) the token must be fully prepared on the side of the
original client, and the CLI uses it only for reading. Ready token MUST have:
- correct context (object, container, etc.)
- valid lifetime
- public session key corresponding to the CLI key
- valid client signature
To sign the session token, exec:
```shell
$ frostfs-cli --wallet <client_wallet> util sign session-token --from ./blank_token --to ./token
```
Once the token is signed, it MUST NOT be modified.
## Commands
### Object
Here are sub-commands of `object` command which support only dynamic sessions (1):
- `put`
- `delete`
- `lock`
These commands accept blank token of the dynamically opened session or open
session internally if it has not been opened yet.
All other `object` sub-commands support only static sessions (2).
### Container
List of commands supporting sessions (static only):
- `create`
- `delete`
- `set-eacl`

View file

@ -0,0 +1,34 @@
# Extended headers
## Overview
Extended headers are used for request/response. They may contain any
user-defined headers to be interpreted on application level. Key name must be a
unique valid UTF-8 string. Value can't be empty. Requests or Responses with
duplicated header names or headers with empty values are considered invalid.
## Existing headers
There are some "well-known" headers starting with `__FROSTFS__` prefix that
affect system behaviour. For backward compatibility, the same set of
"well-known" headers may also use `__NEOFS__` prefix:
* `__FROSTFS__NETMAP_EPOCH` - netmap epoch to use for object placement calculation. The `value` is string
encoded `uint64` in decimal presentation. If set to '0' or omitted, the
current epoch only will be used.
* `__FROSTFS__NETMAP_LOOKUP_DEPTH` - if object can't be found using current epoch's netmap, this header limits
how many past epochs the node can look up through. Depth is applied to a current epoch or the value
of `__FROSTFS__NETMAP_EPOCH` attribute. The `value` is string encoded `uint64` in decimal presentation.
If set to '0' or not set, only the current epoch is used.
## `frostfs-cli` commands with `--xhdr`
List of commands with support of extended headers:
* `container list-objects`
* `object delete/get/hash/head/lock/put/range/search`
* `storagegroup delete/get/list/put`
Example:
```shell
$ frostfs-cli object put -r s01.frostfs.devenv:8080 -w wallet.json --cid CID --file FILE --xhdr "__FROSTFS__NETMAP_EPOCH=777"
```

View file

@ -0,0 +1,891 @@
package internal
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"github.com/TrueCloudLab/frostfs-sdk-go/accounting"
"github.com/TrueCloudLab/frostfs-sdk-go/client"
containerSDK "github.com/TrueCloudLab/frostfs-sdk-go/container"
cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
"github.com/TrueCloudLab/frostfs-sdk-go/eacl"
"github.com/TrueCloudLab/frostfs-sdk-go/netmap"
"github.com/TrueCloudLab/frostfs-sdk-go/object"
oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
"github.com/TrueCloudLab/frostfs-sdk-go/version"
)
// BalanceOfPrm groups parameters of BalanceOf operation.
type BalanceOfPrm struct {
commonPrm
client.PrmBalanceGet
}
// BalanceOfRes groups the resulting values of BalanceOf operation.
type BalanceOfRes struct {
cliRes *client.ResBalanceGet
}
// Balance returns the current balance.
func (x BalanceOfRes) Balance() accounting.Decimal {
return x.cliRes.Amount()
}
// BalanceOf requests the current balance of a FrostFS user.
//
// Returns any error which prevented the operation from completing correctly in error return.
func BalanceOf(prm BalanceOfPrm) (res BalanceOfRes, err error) {
res.cliRes, err = prm.cli.BalanceGet(context.Background(), prm.PrmBalanceGet)
return
}
// ListContainersPrm groups parameters of ListContainers operation.
type ListContainersPrm struct {
commonPrm
client.PrmContainerList
}
// ListContainersRes groups the resulting values of ListContainers operation.
type ListContainersRes struct {
cliRes *client.ResContainerList
}
// IDList returns list of identifiers of user's containers.
func (x ListContainersRes) IDList() []cid.ID {
return x.cliRes.Containers()
}
// ListContainers requests a list of FrostFS user's containers.
//
// Returns any error which prevented the operation from completing correctly in error return.
func ListContainers(prm ListContainersPrm) (res ListContainersRes, err error) {
res.cliRes, err = prm.cli.ContainerList(context.Background(), prm.PrmContainerList)
return
}
// PutContainerPrm groups parameters of PutContainer operation.
type PutContainerPrm struct {
commonPrm
client.PrmContainerPut
}
// PutContainerRes groups the resulting values of PutContainer operation.
type PutContainerRes struct {
cnr cid.ID
}
// ID returns identifier of the created container.
func (x PutContainerRes) ID() cid.ID {
return x.cnr
}
// PutContainer sends a request to save the container in FrostFS.
//
// Operation is asynchronous and not guaranteed even in the absence of errors.
// The required time is also not predictable.
//
// Success can be verified by reading by identifier.
//
// Returns any error which prevented the operation from completing correctly in error return.
func PutContainer(prm PutContainerPrm) (res PutContainerRes, err error) {
cliRes, err := prm.cli.ContainerPut(context.Background(), prm.PrmContainerPut)
if err == nil {
res.cnr = cliRes.ID()
}
return
}
// GetContainerPrm groups parameters of GetContainer operation.
type GetContainerPrm struct {
commonPrm
cliPrm client.PrmContainerGet
}
// SetContainer sets identifier of the container to be read.
func (x *GetContainerPrm) SetContainer(id cid.ID) {
x.cliPrm.SetContainer(id)
}
// GetContainerRes groups the resulting values of GetContainer operation.
type GetContainerRes struct {
cliRes *client.ResContainerGet
}
// Container returns structured of the requested container.
func (x GetContainerRes) Container() containerSDK.Container {
return x.cliRes.Container()
}
// GetContainer reads a container from FrostFS by ID.
//
// Returns any error which prevented the operation from completing correctly in error return.
func GetContainer(prm GetContainerPrm) (res GetContainerRes, err error) {
res.cliRes, err = prm.cli.ContainerGet(context.Background(), prm.cliPrm)
return
}
// IsACLExtendable checks if ACL of the container referenced by the given identifier
// can be extended. Client connection MUST BE correctly established in advance.
func IsACLExtendable(c *client.Client, cnr cid.ID) (bool, error) {
var prm GetContainerPrm
prm.SetClient(c)
prm.SetContainer(cnr)
res, err := GetContainer(prm)
if err != nil {
return false, fmt.Errorf("get container from the FrostFS: %w", err)
}
return res.Container().BasicACL().Extendable(), nil
}
// DeleteContainerPrm groups parameters of DeleteContainerPrm operation.
type DeleteContainerPrm struct {
commonPrm
client.PrmContainerDelete
}
// DeleteContainerRes groups the resulting values of DeleteContainer operation.
type DeleteContainerRes struct{}
// DeleteContainer sends a request to remove a container from FrostFS by ID.
//
// Operation is asynchronous and not guaranteed even in the absence of errors.
// The required time is also not predictable.
//
// Success can be verified by reading by identifier.
//
// Returns any error which prevented the operation from completing correctly in error return.
func DeleteContainer(prm DeleteContainerPrm) (res DeleteContainerRes, err error) {
_, err = prm.cli.ContainerDelete(context.Background(), prm.PrmContainerDelete)
return
}
// EACLPrm groups parameters of EACL operation.
type EACLPrm struct {
commonPrm
client.PrmContainerEACL
}
// EACLRes groups the resulting values of EACL operation.
type EACLRes struct {
cliRes *client.ResContainerEACL
}
// EACL returns requested eACL table.
func (x EACLRes) EACL() eacl.Table {
return x.cliRes.Table()
}
// EACL reads eACL table from FrostFS by container ID.
//
// Returns any error which prevented the operation from completing correctly in error return.
func EACL(prm EACLPrm) (res EACLRes, err error) {
res.cliRes, err = prm.cli.ContainerEACL(context.Background(), prm.PrmContainerEACL)
return
}
// SetEACLPrm groups parameters of SetEACL operation.
type SetEACLPrm struct {
commonPrm
client.PrmContainerSetEACL
}
// SetEACLRes groups the resulting values of SetEACL operation.
type SetEACLRes struct{}
// SetEACL requests to save an eACL table in FrostFS.
//
// Operation is asynchronous and no guaranteed even in the absence of errors.
// The required time is also not predictable.
//
// Success can be verified by reading by container identifier.
//
// Returns any error which prevented the operation from completing correctly in error return.
func SetEACL(prm SetEACLPrm) (res SetEACLRes, err error) {
_, err = prm.cli.ContainerSetEACL(context.Background(), prm.PrmContainerSetEACL)
return
}
// NetworkInfoPrm groups parameters of NetworkInfo operation.
type NetworkInfoPrm struct {
commonPrm
client.PrmNetworkInfo
}
// NetworkInfoRes groups the resulting values of NetworkInfo operation.
type NetworkInfoRes struct {
cliRes *client.ResNetworkInfo
}
// NetworkInfo returns structured information about the FrostFS network.
func (x NetworkInfoRes) NetworkInfo() netmap.NetworkInfo {
return x.cliRes.Info()
}
// NetworkInfo reads information about the FrostFS network.
//
// Returns any error which prevented the operation from completing correctly in error return.
func NetworkInfo(prm NetworkInfoPrm) (res NetworkInfoRes, err error) {
res.cliRes, err = prm.cli.NetworkInfo(context.Background(), prm.PrmNetworkInfo)
return
}
// NodeInfoPrm groups parameters of NodeInfo operation.
type NodeInfoPrm struct {
commonPrm
client.PrmEndpointInfo
}
// NodeInfoRes groups the resulting values of NodeInfo operation.
type NodeInfoRes struct {
cliRes *client.ResEndpointInfo
}
// NodeInfo returns information about the node from netmap.
func (x NodeInfoRes) NodeInfo() netmap.NodeInfo {
return x.cliRes.NodeInfo()
}
// LatestVersion returns the latest FrostFS API version in use.
func (x NodeInfoRes) LatestVersion() version.Version {
return x.cliRes.LatestVersion()
}
// NodeInfo requests information about the remote server from FrostFS netmap.
//
// Returns any error which prevented the operation from completing correctly in error return.
func NodeInfo(prm NodeInfoPrm) (res NodeInfoRes, err error) {
res.cliRes, err = prm.cli.EndpointInfo(context.Background(), prm.PrmEndpointInfo)
return
}
// NetMapSnapshotPrm groups parameters of NetMapSnapshot operation.
type NetMapSnapshotPrm struct {
commonPrm
}
// NetMapSnapshotRes groups the resulting values of NetMapSnapshot operation.
type NetMapSnapshotRes struct {
cliRes *client.ResNetMapSnapshot
}
// NetMap returns current local snapshot of the FrostFS network map.
func (x NetMapSnapshotRes) NetMap() netmap.NetMap {
return x.cliRes.NetMap()
}
// NetMapSnapshot requests current network view of the remote server.
//
// Returns any error which prevented the operation from completing correctly in error return.
func NetMapSnapshot(prm NetMapSnapshotPrm) (res NetMapSnapshotRes, err error) {
res.cliRes, err = prm.cli.NetMapSnapshot(context.Background(), client.PrmNetMapSnapshot{})
return
}
// CreateSessionPrm groups parameters of CreateSession operation.
type CreateSessionPrm struct {
commonPrm
client.PrmSessionCreate
}
// CreateSessionRes groups the resulting values of CreateSession operation.
type CreateSessionRes struct {
cliRes *client.ResSessionCreate
}
// ID returns session identifier.
func (x CreateSessionRes) ID() []byte {
return x.cliRes.ID()
}
// SessionKey returns public session key in a binary format.
func (x CreateSessionRes) SessionKey() []byte {
return x.cliRes.PublicKey()
}
// CreateSession opens a new unlimited session with the remote node.
//
// Returns any error which prevented the operation from completing correctly in error return.
func CreateSession(prm CreateSessionPrm) (res CreateSessionRes, err error) {
res.cliRes, err = prm.cli.SessionCreate(context.Background(), prm.PrmSessionCreate)
return
}
// PutObjectPrm groups parameters of PutObject operation.
type PutObjectPrm struct {
commonObjectPrm
hdr *object.Object
rdr io.Reader
headerCallback func(*object.Object)
}
// SetHeader sets object header.
func (x *PutObjectPrm) SetHeader(hdr *object.Object) {
x.hdr = hdr
}
// SetPayloadReader sets reader of the object payload.
func (x *PutObjectPrm) SetPayloadReader(rdr io.Reader) {
x.rdr = rdr
}
// SetHeaderCallback sets callback which is called on the object after the header is received
// but before the payload is written.
func (x *PutObjectPrm) SetHeaderCallback(f func(*object.Object)) {
x.headerCallback = f
}
// PutObjectRes groups the resulting values of PutObject operation.
type PutObjectRes struct {
id oid.ID
}
// ID returns identifier of the created object.
func (x PutObjectRes) ID() oid.ID {
return x.id
}
// PutObject saves the object in FrostFS network.
//
// Returns any error which prevented the operation from completing correctly in error return.
func PutObject(prm PutObjectPrm) (*PutObjectRes, error) {
var putPrm client.PrmObjectPutInit
if prm.sessionToken != nil {
putPrm.WithinSession(*prm.sessionToken)
}
if prm.bearerToken != nil {
putPrm.WithBearerToken(*prm.bearerToken)
}
if prm.local {
putPrm.MarkLocal()
}
putPrm.WithXHeaders(prm.xHeaders...)
wrt, err := prm.cli.ObjectPutInit(context.Background(), putPrm)
if err != nil {
return nil, fmt.Errorf("init object writing: %w", err)
}
if wrt.WriteHeader(*prm.hdr) {
if prm.headerCallback != nil {
prm.headerCallback(prm.hdr)
}
sz := prm.hdr.PayloadSize()
if data := prm.hdr.Payload(); len(data) > 0 {
if prm.rdr != nil {
prm.rdr = io.MultiReader(bytes.NewReader(data), prm.rdr)
} else {
prm.rdr = bytes.NewReader(data)
sz = uint64(len(data))
}
}
if prm.rdr != nil {
const defaultBufferSizePut = 3 << 20 // Maximum chunk size is 3 MiB in the SDK.
if sz == 0 || sz > defaultBufferSizePut {
sz = defaultBufferSizePut
}
buf := make([]byte, sz)
var n int
for {
n, err = prm.rdr.Read(buf)
if n > 0 {
if !wrt.WritePayloadChunk(buf[:n]) {
break
}
continue
}
if errors.Is(err, io.EOF) {
break
}
return nil, fmt.Errorf("read payload: %w", err)
}
}
}
cliRes, err := wrt.Close()
if err != nil { // here err already carries both status and client errors
return nil, fmt.Errorf("client failure: %w", err)
}
return &PutObjectRes{
id: cliRes.StoredObjectID(),
}, nil
}
// DeleteObjectPrm groups parameters of DeleteObject operation.
type DeleteObjectPrm struct {
commonObjectPrm
objectAddressPrm
}
// DeleteObjectRes groups the resulting values of DeleteObject operation.
type DeleteObjectRes struct {
tomb oid.ID
}
// Tombstone returns the ID of the created object with tombstone.
func (x DeleteObjectRes) Tombstone() oid.ID {
return x.tomb
}
// DeleteObject marks an object to be removed from FrostFS through tombstone placement.
//
// Returns any error which prevented the operation from completing correctly in error return.
func DeleteObject(prm DeleteObjectPrm) (*DeleteObjectRes, error) {
var delPrm client.PrmObjectDelete
delPrm.FromContainer(prm.objAddr.Container())
delPrm.ByID(prm.objAddr.Object())
if prm.sessionToken != nil {
delPrm.WithinSession(*prm.sessionToken)
}
if prm.bearerToken != nil {
delPrm.WithBearerToken(*prm.bearerToken)
}
delPrm.WithXHeaders(prm.xHeaders...)
cliRes, err := prm.cli.ObjectDelete(context.Background(), delPrm)
if err != nil {
return nil, fmt.Errorf("remove object via client: %w", err)
}
return &DeleteObjectRes{
tomb: cliRes.Tombstone(),
}, nil
}
// GetObjectPrm groups parameters of GetObject operation.
type GetObjectPrm struct {
commonObjectPrm
objectAddressPrm
rawPrm
payloadWriterPrm
headerCallback func(*object.Object)
}
// SetHeaderCallback sets callback which is called on the object after the header is received
// but before the payload is written.
func (p *GetObjectPrm) SetHeaderCallback(f func(*object.Object)) {
p.headerCallback = f
}
// GetObjectRes groups the resulting values of GetObject operation.
type GetObjectRes struct {
hdr *object.Object
}
// Header returns the header of the request object.
func (x GetObjectRes) Header() *object.Object {
return x.hdr
}
// GetObject reads an object by address.
//
// Interrupts on any writer error. If successful, payload is written to the writer.
//
// Returns any error which prevented the operation from completing correctly in error return.
// For raw reading, returns *object.SplitInfoError error if object is virtual.
func GetObject(prm GetObjectPrm) (*GetObjectRes, error) {
var getPrm client.PrmObjectGet
getPrm.FromContainer(prm.objAddr.Container())
getPrm.ByID(prm.objAddr.Object())
if prm.sessionToken != nil {
getPrm.WithinSession(*prm.sessionToken)
}
if prm.bearerToken != nil {
getPrm.WithBearerToken(*prm.bearerToken)
}
if prm.raw {
getPrm.MarkRaw()
}
if prm.local {
getPrm.MarkLocal()
}
getPrm.WithXHeaders(prm.xHeaders...)
rdr, err := prm.cli.ObjectGetInit(context.Background(), getPrm)
if err != nil {
return nil, fmt.Errorf("init object reading on client: %w", err)
}
var hdr object.Object
if !rdr.ReadHeader(&hdr) {
_, err = rdr.Close()
return nil, fmt.Errorf("read object header: %w", err)
}
if prm.headerCallback != nil {
prm.headerCallback(&hdr)
}
_, err = io.Copy(prm.wrt, rdr)
if err != nil {
return nil, fmt.Errorf("copy payload: %w", err)
}
return &GetObjectRes{
hdr: &hdr,
}, nil
}
// HeadObjectPrm groups parameters of HeadObject operation.
type HeadObjectPrm struct {
commonObjectPrm
objectAddressPrm
rawPrm
mainOnly bool
}
// SetMainOnlyFlag sets flag to get only main fields of an object header in terms of FrostFS API.
func (x *HeadObjectPrm) SetMainOnlyFlag(v bool) {
x.mainOnly = v
}
// HeadObjectRes groups the resulting values of HeadObject operation.
type HeadObjectRes struct {
hdr *object.Object
}
// Header returns the requested object header.
func (x HeadObjectRes) Header() *object.Object {
return x.hdr
}
// HeadObject reads an object header by address.
//
// Returns any error which prevented the operation from completing correctly in error return.
// For raw reading, returns *object.SplitInfoError error if object is virtual.
func HeadObject(prm HeadObjectPrm) (*HeadObjectRes, error) {
var cliPrm client.PrmObjectHead
cliPrm.FromContainer(prm.objAddr.Container())
cliPrm.ByID(prm.objAddr.Object())
if prm.sessionToken != nil {
cliPrm.WithinSession(*prm.sessionToken)
}
if prm.bearerToken != nil {
cliPrm.WithBearerToken(*prm.bearerToken)
}
if prm.raw {
cliPrm.MarkRaw()
}
if prm.local {
cliPrm.MarkLocal()
}
cliPrm.WithXHeaders(prm.xHeaders...)
res, err := prm.cli.ObjectHead(context.Background(), cliPrm)
if err != nil {
return nil, fmt.Errorf("read object header via client: %w", err)
}
var hdr object.Object
if !res.ReadHeader(&hdr) {
return nil, fmt.Errorf("missing header in response")
}
return &HeadObjectRes{
hdr: &hdr,
}, nil
}
// SearchObjectsPrm groups parameters of SearchObjects operation.
type SearchObjectsPrm struct {
commonObjectPrm
containerIDPrm
filters object.SearchFilters
}
// SetFilters sets search filters.
func (x *SearchObjectsPrm) SetFilters(filters object.SearchFilters) {
x.filters = filters
}
// SearchObjectsRes groups the resulting values of SearchObjects operation.
type SearchObjectsRes struct {
ids []oid.ID
}
// IDList returns identifiers of the matched objects.
func (x SearchObjectsRes) IDList() []oid.ID {
return x.ids
}
// SearchObjects selects objects from the container which match the filters.
//
// Returns any error which prevented the operation from completing correctly in error return.
func SearchObjects(prm SearchObjectsPrm) (*SearchObjectsRes, error) {
var cliPrm client.PrmObjectSearch
cliPrm.InContainer(prm.cnrID)
cliPrm.SetFilters(prm.filters)
if prm.sessionToken != nil {
cliPrm.WithinSession(*prm.sessionToken)
}
if prm.bearerToken != nil {
cliPrm.WithBearerToken(*prm.bearerToken)
}
if prm.local {
cliPrm.MarkLocal()
}
cliPrm.WithXHeaders(prm.xHeaders...)
rdr, err := prm.cli.ObjectSearchInit(context.Background(), cliPrm)
if err != nil {
return nil, fmt.Errorf("init object search: %w", err)
}
buf := make([]oid.ID, 10)
var list []oid.ID
var n int
var ok bool
for {
n, ok = rdr.Read(buf)
for i := 0; i < n; i++ {
list = append(list, buf[i])
}
if !ok {
break
}
}
_, err = rdr.Close()
if err != nil {
return nil, fmt.Errorf("read object list: %w", err)
}
return &SearchObjectsRes{
ids: list,
}, nil
}
// HashPayloadRangesPrm groups parameters of HashPayloadRanges operation.
type HashPayloadRangesPrm struct {
commonObjectPrm
objectAddressPrm
tz bool
rngs []*object.Range
salt []byte
}
// TZ sets flag to request Tillich-Zemor hashes.
func (x *HashPayloadRangesPrm) TZ() {
x.tz = true
}
// SetRanges sets a list of payload ranges to hash.
func (x *HashPayloadRangesPrm) SetRanges(rngs []*object.Range) {
x.rngs = rngs
}
// SetSalt sets data for each range to be XOR'ed with.
func (x *HashPayloadRangesPrm) SetSalt(salt []byte) {
x.salt = salt
}
// HashPayloadRangesRes groups the resulting values of HashPayloadRanges operation.
type HashPayloadRangesRes struct {
cliRes *client.ResObjectHash
}
// HashList returns a list of hashes of the payload ranges keeping order.
func (x HashPayloadRangesRes) HashList() [][]byte {
return x.cliRes.Checksums()
}
// HashPayloadRanges requests hashes (by default SHA256) of the object payload ranges.
//
// Returns any error which prevented the operation from completing correctly in error return.
// Returns an error if number of received hashes differs with the number of requested ranges.
func HashPayloadRanges(prm HashPayloadRangesPrm) (*HashPayloadRangesRes, error) {
var cliPrm client.PrmObjectHash
cliPrm.FromContainer(prm.objAddr.Container())
cliPrm.ByID(prm.objAddr.Object())
if prm.local {
cliPrm.MarkLocal()
}
cliPrm.UseSalt(prm.salt)
rngs := make([]uint64, 2*len(prm.rngs))
for i := range prm.rngs {
rngs[2*i] = prm.rngs[i].GetOffset()
rngs[2*i+1] = prm.rngs[i].GetLength()
}
cliPrm.SetRangeList(rngs...)
if prm.tz {
cliPrm.TillichZemorAlgo()
}
if prm.sessionToken != nil {
cliPrm.WithinSession(*prm.sessionToken)
}
if prm.bearerToken != nil {
cliPrm.WithBearerToken(*prm.bearerToken)
}
cliPrm.WithXHeaders(prm.xHeaders...)
res, err := prm.cli.ObjectHash(context.Background(), cliPrm)
if err != nil {
return nil, fmt.Errorf("read payload hashes via client: %w", err)
}
return &HashPayloadRangesRes{
cliRes: res,
}, nil
}
// PayloadRangePrm groups parameters of PayloadRange operation.
type PayloadRangePrm struct {
commonObjectPrm
objectAddressPrm
rawPrm
payloadWriterPrm
rng *object.Range
}
// SetRange sets payload range to read.
func (x *PayloadRangePrm) SetRange(rng *object.Range) {
x.rng = rng
}
// PayloadRangeRes groups the resulting values of PayloadRange operation.
type PayloadRangeRes struct{}
// PayloadRange reads object payload range from FrostFS and writes it to the specified writer.
//
// Interrupts on any writer error.
//
// Returns any error which prevented the operation from completing correctly in error return.
// For raw reading, returns *object.SplitInfoError error if object is virtual.
func PayloadRange(prm PayloadRangePrm) (*PayloadRangeRes, error) {
var cliPrm client.PrmObjectRange
cliPrm.FromContainer(prm.objAddr.Container())
cliPrm.ByID(prm.objAddr.Object())
if prm.sessionToken != nil {
cliPrm.WithinSession(*prm.sessionToken)
}
if prm.bearerToken != nil {
cliPrm.WithBearerToken(*prm.bearerToken)
}
if prm.raw {
cliPrm.MarkRaw()
}
if prm.local {
cliPrm.MarkLocal()
}
cliPrm.SetOffset(prm.rng.GetOffset())
cliPrm.SetLength(prm.rng.GetLength())
cliPrm.WithXHeaders(prm.xHeaders...)
rdr, err := prm.cli.ObjectRangeInit(context.Background(), cliPrm)
if err != nil {
return nil, fmt.Errorf("init payload reading: %w", err)
}
_, err = io.Copy(prm.wrt, rdr)
if err != nil {
return nil, fmt.Errorf("copy payload: %w", err)
}
return new(PayloadRangeRes), nil
}
// SyncContainerPrm groups parameters of SyncContainerSettings operation.
type SyncContainerPrm struct {
commonPrm
c *containerSDK.Container
}
// SetContainer sets a container that is required to be synced.
func (s *SyncContainerPrm) SetContainer(c *containerSDK.Container) {
s.c = c
}
// SyncContainerRes groups resulting values of SyncContainerSettings
// operation.
type SyncContainerRes struct{}
// SyncContainerSettings reads global network config from FrostFS and
// syncs container settings with it.
//
// Interrupts on any writer error.
//
// Panics if a container passed as a parameter is nil.
func SyncContainerSettings(prm SyncContainerPrm) (*SyncContainerRes, error) {
if prm.c == nil {
panic("sync container settings with the network: nil container")
}
err := client.SyncContainerWithNetwork(context.Background(), prm.c, prm.cli)
if err != nil {
return nil, err
}
return new(SyncContainerRes), nil
}

View file

@ -0,0 +1,15 @@
// Package internal provides functionality for FrostFS CLI application
// communication with FrostFS network.
//
// The base client for accessing remote nodes via FrostFS API is a FrostFS SDK
// Go API client. However, although it encapsulates a useful piece of business
// logic (e.g. the signature mechanism), the FrostFS CLI application does not
// fully use the client's flexible interface.
//
// In this regard, this package provides functions over base API client
// necessary for the application. This allows you to concentrate the entire
// spectrum of the client's use in one place (this will be convenient both when
// updating the base client and for evaluating the UX of SDK library). So it is
// expected that all application packages will be limited to this package for
// the development of functionality requiring FrostFS API communication.
package internal

View file

@ -0,0 +1,92 @@
package internal
import (
"io"
"github.com/TrueCloudLab/frostfs-sdk-go/bearer"
"github.com/TrueCloudLab/frostfs-sdk-go/client"
cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
"github.com/TrueCloudLab/frostfs-sdk-go/session"
)
// here are small structures with public setters to share between parameter structures
type commonPrm struct {
cli *client.Client
}
// SetClient sets the base client for FrostFS API communication.
func (x *commonPrm) SetClient(cli *client.Client) {
x.cli = cli
}
type containerIDPrm struct {
cnrID cid.ID
}
// SetContainerID sets the container identifier.
func (x *containerIDPrm) SetContainerID(id cid.ID) {
x.cnrID = id
}
type bearerTokenPrm struct {
bearerToken *bearer.Token
}
// SetBearerToken sets the bearer token to be attached to the request.
func (x *bearerTokenPrm) SetBearerToken(tok *bearer.Token) {
x.bearerToken = tok
}
type objectAddressPrm struct {
objAddr oid.Address
}
func (x *objectAddressPrm) SetAddress(addr oid.Address) {
x.objAddr = addr
}
type rawPrm struct {
raw bool
}
// SetRawFlag sets flag of raw request.
func (x *rawPrm) SetRawFlag(raw bool) {
x.raw = raw
}
type payloadWriterPrm struct {
wrt io.Writer
}
// SetPayloadWriter sets the writer of the object payload.
func (x *payloadWriterPrm) SetPayloadWriter(wrt io.Writer) {
x.wrt = wrt
}
type commonObjectPrm struct {
commonPrm
bearerTokenPrm
sessionToken *session.Object
local bool
xHeaders []string
}
// SetTTL sets request TTL value.
func (x *commonObjectPrm) SetTTL(ttl uint32) {
x.local = ttl < 2
}
// SetXHeaders sets request X-Headers.
func (x *commonObjectPrm) SetXHeaders(hs []string) {
x.xHeaders = hs
}
// SetSessionToken sets the token of the session within which the request should be sent.
func (x *commonObjectPrm) SetSessionToken(tok *session.Object) {
x.sessionToken = tok
}

View file

@ -0,0 +1,96 @@
package internal
import (
"context"
"crypto/ecdsa"
"crypto/elliptic"
"crypto/rand"
"errors"
"fmt"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-node/pkg/network"
"github.com/TrueCloudLab/frostfs-sdk-go/client"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
var errInvalidEndpoint = errors.New("provided RPC endpoint is incorrect")
// GetSDKClientByFlag returns default frostfs-sdk-go client using the specified flag for the address.
// On error, outputs to stderr of cmd and exits with non-zero code.
func GetSDKClientByFlag(cmd *cobra.Command, key *ecdsa.PrivateKey, endpointFlag string) *client.Client {
cli, err := getSDKClientByFlag(cmd, key, endpointFlag)
if err != nil {
commonCmd.ExitOnErr(cmd, "can't create API client: %w", err)
}
return cli
}
func getSDKClientByFlag(cmd *cobra.Command, key *ecdsa.PrivateKey, endpointFlag string) (*client.Client, error) {
var addr network.Address
err := addr.FromString(viper.GetString(endpointFlag))
if err != nil {
return nil, fmt.Errorf("%v: %w", errInvalidEndpoint, err)
}
return GetSDKClient(cmd, key, addr)
}
// GetSDKClient returns default frostfs-sdk-go client.
func GetSDKClient(cmd *cobra.Command, key *ecdsa.PrivateKey, addr network.Address) (*client.Client, error) {
var (
c client.Client
prmInit client.PrmInit
prmDial client.PrmDial
)
prmInit.SetDefaultPrivateKey(*key)
prmInit.ResolveNeoFSFailures()
prmDial.SetServerURI(addr.URIAddr())
if timeout := viper.GetDuration(commonflags.Timeout); timeout > 0 {
// In CLI we can only set a timeout for the whole operation.
// By also setting stream timeout we ensure that no operation hands
// for too long.
prmDial.SetTimeout(timeout)
prmDial.SetStreamTimeout(timeout)
common.PrintVerbose(cmd, "Set request timeout to %s.", timeout)
}
c.Init(prmInit)
if err := c.Dial(prmDial); err != nil {
return nil, fmt.Errorf("can't init SDK client: %w", err)
}
return &c, nil
}
// GetCurrentEpoch returns current epoch.
func GetCurrentEpoch(ctx context.Context, cmd *cobra.Command, endpoint string) (uint64, error) {
var addr network.Address
if err := addr.FromString(endpoint); err != nil {
return 0, fmt.Errorf("can't parse RPC endpoint: %w", err)
}
key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
if err != nil {
return 0, fmt.Errorf("can't generate key to sign query: %w", err)
}
c, err := GetSDKClient(cmd, key, addr)
if err != nil {
return 0, err
}
ni, err := c.NetworkInfo(ctx, client.PrmNetworkInfo{})
if err != nil {
return 0, err
}
return ni.Info().CurrentEpoch(), nil
}

View file

@ -0,0 +1,50 @@
package common
import (
"errors"
"os"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-node/pkg/core/version"
"github.com/TrueCloudLab/frostfs-sdk-go/eacl"
versionSDK "github.com/TrueCloudLab/frostfs-sdk-go/version"
"github.com/spf13/cobra"
)
var errUnsupportedEACLFormat = errors.New("unsupported eACL format")
// ReadEACL reads extended ACL table from eaclPath.
func ReadEACL(cmd *cobra.Command, eaclPath string) *eacl.Table {
_, err := os.Stat(eaclPath) // check if `eaclPath` is an existing file
if err != nil {
commonCmd.ExitOnErr(cmd, "", errors.New("incorrect path to file with EACL"))
}
PrintVerbose(cmd, "Reading EACL from file: %s", eaclPath)
data, err := os.ReadFile(eaclPath)
commonCmd.ExitOnErr(cmd, "can't read file with EACL: %w", err)
table := eacl.NewTable()
if err = table.UnmarshalJSON(data); err == nil {
validateAndFixEACLVersion(table)
PrintVerbose(cmd, "Parsed JSON encoded EACL table")
return table
}
if err = table.Unmarshal(data); err == nil {
validateAndFixEACLVersion(table)
PrintVerbose(cmd, "Parsed binary encoded EACL table")
return table
}
commonCmd.ExitOnErr(cmd, "", errUnsupportedEACLFormat)
return nil
}
func validateAndFixEACLVersion(table *eacl.Table) {
if !version.IsValid(table.Version()) {
table.SetVersion(versionSDK.Current())
}
}

View file

@ -0,0 +1,28 @@
package common
import (
"fmt"
"strconv"
"github.com/spf13/cobra"
)
// ParseEpoch parses epoch argument. Second return value is true if
// the specified epoch is relative, and false otherwise.
func ParseEpoch(cmd *cobra.Command, flag string) (uint64, bool, error) {
s, _ := cmd.Flags().GetString(flag)
if len(s) == 0 {
return 0, false, nil
}
relative := s[0] == '+'
if relative {
s = s[1:]
}
epoch, err := strconv.ParseUint(s, 10, 64)
if err != nil {
return 0, relative, fmt.Errorf("can't parse epoch for %s argument: %w", flag, err)
}
return epoch, relative, nil
}

View file

@ -0,0 +1,23 @@
package common
import (
"bytes"
"encoding/json"
"github.com/spf13/cobra"
)
// PrettyPrintJSON prints m as an indented JSON to the cmd output.
func PrettyPrintJSON(cmd *cobra.Command, m json.Marshaler, entity string) {
data, err := m.MarshalJSON()
if err != nil {
PrintVerbose(cmd, "Can't convert %s to json: %w", entity, err)
return
}
buf := new(bytes.Buffer)
if err := json.Indent(buf, data, "", " "); err != nil {
PrintVerbose(cmd, "Can't pretty print json: %w", err)
return
}
cmd.Println(buf)
}

View file

@ -0,0 +1,67 @@
package common
import (
"encoding/json"
"errors"
"fmt"
"os"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-sdk-go/bearer"
"github.com/spf13/cobra"
)
// ReadBearerToken reads bearer token from the path provided in a specified flag.
func ReadBearerToken(cmd *cobra.Command, flagname string) *bearer.Token {
path, err := cmd.Flags().GetString(flagname)
commonCmd.ExitOnErr(cmd, "", err)
if len(path) == 0 {
return nil
}
PrintVerbose(cmd, "Reading bearer token from file [%s]...", path)
var tok bearer.Token
err = ReadBinaryOrJSON(cmd, &tok, path)
commonCmd.ExitOnErr(cmd, "invalid bearer token: %v", err)
return &tok
}
// BinaryOrJSON is an interface of entities which provide json.Unmarshaler
// and FrostFS binary decoder.
type BinaryOrJSON interface {
Unmarshal([]byte) error
json.Unmarshaler
}
// ReadBinaryOrJSON reads file data using provided path and decodes
// BinaryOrJSON from the data.
func ReadBinaryOrJSON(cmd *cobra.Command, dst BinaryOrJSON, fPath string) error {
PrintVerbose(cmd, "Reading file [%s]...", fPath)
// try to read session token from file
data, err := os.ReadFile(fPath)
if err != nil {
return fmt.Errorf("read file <%s>: %w", fPath, err)
}
PrintVerbose(cmd, "Trying to decode binary...")
err = dst.Unmarshal(data)
if err != nil {
PrintVerbose(cmd, "Failed to decode binary: %v", err)
PrintVerbose(cmd, "Trying to decode JSON...")
err = dst.UnmarshalJSON(data)
if err != nil {
PrintVerbose(cmd, "Failed to decode JSON: %v", err)
return errors.New("invalid format")
}
}
return nil
}

View file

@ -0,0 +1,46 @@
package common
import (
"encoding/hex"
"strconv"
"time"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/TrueCloudLab/frostfs-sdk-go/checksum"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
// PrintVerbose prints to the stdout if the commonflags.Verbose flag is on.
func PrintVerbose(cmd *cobra.Command, format string, a ...any) {
if viper.GetBool(commonflags.Verbose) {
cmd.Printf(format+"\n", a...)
}
}
// PrettyPrintUnixTime interprets s as unix timestamp and prints it as
// a date. Is s is invalid, "malformed" is returned.
func PrettyPrintUnixTime(s string) string {
unixTime, err := strconv.ParseInt(s, 10, 64)
if err != nil {
return "malformed"
}
timestamp := time.Unix(unixTime, 0)
return timestamp.String()
}
// PrintChecksum prints checksum.
func PrintChecksum(cmd *cobra.Command, name string, recv func() (checksum.Checksum, bool)) {
var strVal string
cs, csSet := recv()
if csSet {
strVal = hex.EncodeToString(cs.Value())
} else {
strVal = "<empty>"
}
cmd.Printf("%s: %s\n", name, strVal)
}

View file

@ -0,0 +1,33 @@
package commonflags
import (
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
const (
TTL = "ttl"
TTLShorthand = ""
TTLDefault = 2
TTLUsage = "TTL value in request meta header"
XHeadersKey = "xhdr"
XHeadersShorthand = "x"
XHeadersUsage = "Request X-Headers in form of Key=Value"
)
// InitAPI inits common flags for storage node services.
func InitAPI(cmd *cobra.Command) {
ff := cmd.Flags()
ff.StringSliceP(XHeadersKey, XHeadersShorthand, []string{}, XHeadersUsage)
ff.Uint32P(TTL, TTLShorthand, TTLDefault, TTLUsage)
}
// BindAPI binds API flags of storage node services to the viper.
func BindAPI(cmd *cobra.Command) {
ff := cmd.Flags()
_ = viper.BindPFlag(TTL, ff.Lookup(TTL))
_ = viper.BindPFlag(XHeadersKey, ff.Lookup(XHeadersKey))
}

View file

@ -0,0 +1,9 @@
package commonflags
const (
// ExpireAt is a flag for setting last epoch of an object or a token.
ExpireAt = "expire-at"
// Lifetime is a flag for setting the lifetime of an object or a token,
// starting from the current epoch.
Lifetime = "lifetime"
)

View file

@ -0,0 +1,84 @@
package commonflags
import (
"time"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
// Common CLI flag keys, shorthands, default
// values and their usage descriptions.
const (
GenerateKey = "generate-key"
generateKeyShorthand = "g"
generateKeyDefault = false
generateKeyUsage = "Generate new private key"
WalletPath = "wallet"
WalletPathShorthand = "w"
WalletPathDefault = ""
WalletPathUsage = "Path to the wallet or binary key"
Account = "address"
AccountShorthand = ""
AccountDefault = ""
AccountUsage = "Address of wallet account"
RPC = "rpc-endpoint"
RPCShorthand = "r"
RPCDefault = ""
RPCUsage = "Remote node address (as 'multiaddr' or '<host>:<port>')"
Timeout = "timeout"
TimeoutShorthand = "t"
TimeoutDefault = 15 * time.Second
TimeoutUsage = "Timeout for an operation"
Verbose = "verbose"
VerboseShorthand = "v"
VerboseUsage = "Verbose output"
ForceFlag = "force"
ForceFlagShorthand = "f"
CIDFlag = "cid"
CIDFlagUsage = "Container ID."
OIDFlag = "oid"
OIDFlagUsage = "Object ID."
)
// Init adds common flags to the command:
// - GenerateKey,
// - WalletPath,
// - Account,
// - RPC,
// - Timeout.
func Init(cmd *cobra.Command) {
InitWithoutRPC(cmd)
ff := cmd.Flags()
ff.StringP(RPC, RPCShorthand, RPCDefault, RPCUsage)
ff.DurationP(Timeout, TimeoutShorthand, TimeoutDefault, TimeoutUsage)
}
// InitWithoutRPC is similar to Init but doesn't create the RPC flag.
func InitWithoutRPC(cmd *cobra.Command) {
ff := cmd.Flags()
ff.BoolP(GenerateKey, generateKeyShorthand, generateKeyDefault, generateKeyUsage)
ff.StringP(WalletPath, WalletPathShorthand, WalletPathDefault, WalletPathUsage)
ff.StringP(Account, AccountShorthand, AccountDefault, AccountUsage)
}
// Bind binds common command flags to the viper.
func Bind(cmd *cobra.Command) {
ff := cmd.Flags()
_ = viper.BindPFlag(GenerateKey, ff.Lookup(GenerateKey))
_ = viper.BindPFlag(WalletPath, ff.Lookup(WalletPath))
_ = viper.BindPFlag(Account, ff.Lookup(Account))
_ = viper.BindPFlag(RPC, ff.Lookup(RPC))
_ = viper.BindPFlag(Timeout, ff.Lookup(Timeout))
}

View file

@ -0,0 +1,3 @@
package commonflags
const JSON = "json"

View file

@ -0,0 +1,19 @@
package commonflags
import (
"fmt"
"github.com/spf13/cobra"
)
const SessionToken = "session"
// InitSession registers SessionToken flag representing file path to the token of
// the session with the given name. Supports FrostFS-binary and JSON files.
func InitSession(cmd *cobra.Command, name string) {
cmd.Flags().String(
SessionToken,
"",
fmt.Sprintf("Filepath to a JSON- or binary-encoded token of the %s session", name),
)
}

View file

@ -0,0 +1,126 @@
package key
import (
"bytes"
"io"
"os"
"path/filepath"
"testing"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/nspcc-dev/neo-go/cli/input"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"github.com/stretchr/testify/require"
"golang.org/x/term"
)
var testCmd = &cobra.Command{
Use: "test",
Short: "test",
Run: func(cmd *cobra.Command, args []string) {},
}
func Test_getOrGenerate(t *testing.T) {
dir := t.TempDir()
wallPath := filepath.Join(dir, "wallet.json")
w, err := wallet.NewWallet(wallPath)
require.NoError(t, err)
badWallPath := filepath.Join(dir, "bad_wallet.json")
require.NoError(t, os.WriteFile(badWallPath, []byte("bad content"), os.ModePerm))
acc1, err := wallet.NewAccount()
require.NoError(t, err)
require.NoError(t, acc1.Encrypt("pass", keys.NEP2ScryptParams()))
w.AddAccount(acc1)
acc2, err := wallet.NewAccount()
require.NoError(t, err)
require.NoError(t, acc2.Encrypt("pass", keys.NEP2ScryptParams()))
acc2.Default = true
w.AddAccount(acc2)
require.NoError(t, w.Save())
keyPath := filepath.Join(dir, "binary.key")
rawKey, err := keys.NewPrivateKey()
require.NoError(t, err)
require.NoError(t, os.WriteFile(keyPath, rawKey.Bytes(), os.ModePerm))
wifKey, err := keys.NewPrivateKey()
require.NoError(t, err)
nep2Key, err := keys.NewPrivateKey()
require.NoError(t, err)
nep2, err := keys.NEP2Encrypt(nep2Key, "pass", keys.NEP2ScryptParams())
require.NoError(t, err)
in := bytes.NewBuffer(nil)
input.Terminal = term.NewTerminal(input.ReadWriter{
Reader: in,
Writer: io.Discard,
}, "")
checkKeyError(t, filepath.Join(dir, "badfile"), ErrFs)
checkKeyError(t, badWallPath, ErrInvalidKey)
t.Run("wallet", func(t *testing.T) {
checkKeyError(t, wallPath, ErrInvalidPassword)
in.WriteString("invalid\r")
checkKeyError(t, wallPath, ErrInvalidPassword)
in.WriteString("pass\r")
checkKey(t, wallPath, acc2.PrivateKey()) // default account
viper.Set(commonflags.Account, acc1.Address)
in.WriteString("pass\r")
checkKey(t, wallPath, acc1.PrivateKey())
viper.Set(commonflags.Account, "not an address")
checkKeyError(t, wallPath, ErrInvalidAddress)
acc, err := wallet.NewAccount()
require.NoError(t, err)
viper.Set(commonflags.Account, acc.Address)
checkKeyError(t, wallPath, ErrInvalidAddress)
})
t.Run("WIF", func(t *testing.T) {
checkKeyError(t, wifKey.WIF(), ErrFs)
})
t.Run("NEP-2", func(t *testing.T) {
checkKeyError(t, nep2, ErrFs)
})
t.Run("raw key", func(t *testing.T) {
checkKey(t, keyPath, rawKey)
})
t.Run("generate", func(t *testing.T) {
viper.Set(commonflags.GenerateKey, true)
actual, err := getOrGenerate(testCmd)
require.NoError(t, err)
require.NotNil(t, actual)
for _, p := range []*keys.PrivateKey{nep2Key, rawKey, wifKey, acc1.PrivateKey(), acc2.PrivateKey()} {
require.NotEqual(t, p, actual, "expected new key to be generated")
}
})
}
func checkKeyError(t *testing.T, desc string, err error) {
viper.Set(commonflags.WalletPath, desc)
_, actualErr := getOrGenerate(testCmd)
require.ErrorIs(t, actualErr, err)
}
func checkKey(t *testing.T, desc string, expected *keys.PrivateKey) {
viper.Set(commonflags.WalletPath, desc)
actual, err := getOrGenerate(testCmd)
require.NoError(t, err)
require.Equal(t, &expected.PrivateKey, actual)
}

View file

@ -0,0 +1,62 @@
package key
import (
"crypto/ecdsa"
"errors"
"fmt"
"os"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
var errCantGenerateKey = errors.New("can't generate new private key")
// Get returns private key from wallet or binary file.
// Ideally we want to touch file-system on the last step.
// This function assumes that all flags were bind to viper in a `PersistentPreRun`.
func Get(cmd *cobra.Command) *ecdsa.PrivateKey {
pk, err := get(cmd)
commonCmd.ExitOnErr(cmd, "can't fetch private key: %w", err)
return pk
}
func get(cmd *cobra.Command) (*ecdsa.PrivateKey, error) {
keyDesc := viper.GetString(commonflags.WalletPath)
data, err := os.ReadFile(keyDesc)
if err != nil {
return nil, fmt.Errorf("%w: %v", ErrFs, err)
}
priv, err := keys.NewPrivateKeyFromBytes(data)
if err != nil {
w, err := wallet.NewWalletFromFile(keyDesc)
if err == nil {
return FromWallet(cmd, w, viper.GetString(commonflags.Account))
}
return nil, fmt.Errorf("%w: %v", ErrInvalidKey, err)
}
return &priv.PrivateKey, nil
}
// GetOrGenerate is similar to get but generates a new key if commonflags.GenerateKey is set.
func GetOrGenerate(cmd *cobra.Command) *ecdsa.PrivateKey {
pk, err := getOrGenerate(cmd)
commonCmd.ExitOnErr(cmd, "can't fetch private key: %w", err)
return pk
}
func getOrGenerate(cmd *cobra.Command) (*ecdsa.PrivateKey, error) {
if viper.GetBool(commonflags.GenerateKey) {
priv, err := keys.NewPrivateKey()
if err != nil {
return nil, fmt.Errorf("%w: %v", errCantGenerateKey, err)
}
return &priv.PrivateKey, nil
}
return get(cmd)
}

View file

@ -0,0 +1,70 @@
package key
import (
"crypto/ecdsa"
"errors"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"github.com/nspcc-dev/neo-go/cli/flags"
"github.com/nspcc-dev/neo-go/cli/input"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
// Key-related errors.
var (
ErrFs = errors.New("unable to read file from given path")
ErrInvalidKey = errors.New("provided key is incorrect, only wallet or binary key supported")
ErrInvalidAddress = errors.New("--address option must be specified and valid")
ErrInvalidPassword = errors.New("invalid password for the encrypted key")
)
// FromWallet returns private key of the wallet account.
func FromWallet(cmd *cobra.Command, w *wallet.Wallet, addrStr string) (*ecdsa.PrivateKey, error) {
var (
addr util.Uint160
err error
)
if addrStr == "" {
common.PrintVerbose(cmd, "Using default wallet address")
addr = w.GetChangeAddress()
} else {
addr, err = flags.ParseAddress(addrStr)
if err != nil {
common.PrintVerbose(cmd, "Can't parse address: %s", addrStr)
return nil, ErrInvalidAddress
}
}
acc := w.GetAccount(addr)
if acc == nil {
common.PrintVerbose(cmd, "Can't find wallet account for %s", addrStr)
return nil, ErrInvalidAddress
}
pass, err := getPassword()
if err != nil {
common.PrintVerbose(cmd, "Can't read password: %v", err)
return nil, ErrInvalidPassword
}
if err := acc.Decrypt(pass, keys.NEP2ScryptParams()); err != nil {
common.PrintVerbose(cmd, "Can't decrypt account: %v", err)
return nil, ErrInvalidPassword
}
return &acc.PrivateKey().PrivateKey, nil
}
func getPassword() (string, error) {
// this check allows empty passwords
if viper.IsSet("password") {
return viper.GetString("password"), nil
}
return input.ReadPassword("Enter password > ")
}

7
cmd/frostfs-cli/main.go Normal file
View file

@ -0,0 +1,7 @@
package main
import cmd "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules"
func main() {
cmd.Execute()
}

View file

@ -0,0 +1,70 @@
package accounting
import (
"math/big"
internalclient "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-node/pkg/util/precision"
"github.com/TrueCloudLab/frostfs-sdk-go/accounting"
"github.com/TrueCloudLab/frostfs-sdk-go/user"
"github.com/nspcc-dev/neo-go/pkg/encoding/fixedn"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
const (
ownerFlag = "owner"
)
var accountingBalanceCmd = &cobra.Command{
Use: "balance",
Short: "Get internal balance of FrostFS account",
Long: `Get internal balance of FrostFS account`,
Run: func(cmd *cobra.Command, args []string) {
var idUser user.ID
pk := key.GetOrGenerate(cmd)
balanceOwner, _ := cmd.Flags().GetString(ownerFlag)
if balanceOwner == "" {
user.IDFromKey(&idUser, pk.PublicKey)
} else {
commonCmd.ExitOnErr(cmd, "can't decode owner ID wallet address: %w", idUser.DecodeString(balanceOwner))
}
cli := internalclient.GetSDKClientByFlag(cmd, pk, commonflags.RPC)
var prm internalclient.BalanceOfPrm
prm.SetClient(cli)
prm.SetAccount(idUser)
res, err := internalclient.BalanceOf(prm)
commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
// print to stdout
prettyPrintDecimal(cmd, res.Balance())
},
}
func initAccountingBalanceCmd() {
ff := accountingBalanceCmd.Flags()
ff.StringP(commonflags.WalletPath, commonflags.WalletPathShorthand, commonflags.WalletPathDefault, commonflags.WalletPathUsage)
ff.StringP(commonflags.Account, commonflags.AccountShorthand, commonflags.AccountDefault, commonflags.AccountUsage)
ff.StringP(commonflags.RPC, commonflags.RPCShorthand, commonflags.RPCDefault, commonflags.RPCUsage)
ff.String(ownerFlag, "", "owner of balance account (omit to use owner from private key)")
}
func prettyPrintDecimal(cmd *cobra.Command, decimal accounting.Decimal) {
if viper.GetBool(commonflags.Verbose) {
cmd.Println("value:", decimal.Value())
cmd.Println("precision:", decimal.Precision())
} else {
amountF8 := precision.Convert(decimal.Precision(), 8, big.NewInt(decimal.Value()))
cmd.Println(fixedn.ToString(amountF8, 8))
}
}

View file

@ -0,0 +1,27 @@
package accounting
import (
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
// Cmd represents the accounting command.
var Cmd = &cobra.Command{
Use: "accounting",
Short: "Operations with accounts and balances",
Long: `Operations with accounts and balances`,
PersistentPreRun: func(cmd *cobra.Command, args []string) {
flags := cmd.Flags()
_ = viper.BindPFlag(commonflags.WalletPath, flags.Lookup(commonflags.WalletPath))
_ = viper.BindPFlag(commonflags.Account, flags.Lookup(commonflags.Account))
_ = viper.BindPFlag(commonflags.RPC, flags.Lookup(commonflags.RPC))
},
}
func init() {
Cmd.AddCommand(accountingBalanceCmd)
initAccountingBalanceCmd()
}

View file

@ -0,0 +1,28 @@
package basic
import (
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-sdk-go/container/acl"
"github.com/spf13/cobra"
)
var printACLCmd = &cobra.Command{
Use: "print",
Short: "Pretty print basic ACL from the HEX representation",
Example: `frostfs-cli acl basic print 0x1C8C8CCC`,
Long: `Pretty print basic ACL from the HEX representation.
Few roles have exclusive default access to set of operation, even if particular bit deny it.
Container have access to the operations of the data replication mechanism:
Get, Head, Put, Search, Hash.
InnerRing members are allowed to data audit ops only:
Get, Head, Hash, Search.`,
Run: printACL,
Args: cobra.ExactArgs(1),
}
func printACL(cmd *cobra.Command, args []string) {
var bacl acl.Basic
commonCmd.ExitOnErr(cmd, "unable to parse basic acl: %w", bacl.DecodeString(args[0]))
util.PrettyPrintTableBACL(cmd, &bacl)
}

View file

@ -0,0 +1,14 @@
package basic
import (
"github.com/spf13/cobra"
)
var Cmd = &cobra.Command{
Use: "basic",
Short: "Operations with Basic Access Control Lists",
}
func init() {
Cmd.AddCommand(printACLCmd)
}

View file

@ -0,0 +1,127 @@
package extended
import (
"bytes"
"encoding/json"
"os"
"strings"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
"github.com/TrueCloudLab/frostfs-sdk-go/eacl"
"github.com/spf13/cobra"
)
var createCmd = &cobra.Command{
Use: "create",
Short: "Create extended ACL from the text representation",
Long: `Create extended ACL from the text representation.
Rule consist of these blocks: <action> <operation> [<filter1> ...] [<target1> ...]
Action is 'allow' or 'deny'.
Operation is an object service verb: 'get', 'head', 'put', 'search', 'delete', 'getrange', or 'getrangehash'.
Filter consists of <typ>:<key><match><value>
Typ is 'obj' for object applied filter or 'req' for request applied filter.
Key is a valid unicode string corresponding to object or request header key.
Well-known system object headers start with '$Object:' prefix.
User defined headers start without prefix.
Read more about filter keys at github.com/TrueCloudLab/frostfs-api/blob/master/proto-docs/acl.md#message-eaclrecordfilter
Match is '=' for matching and '!=' for non-matching filter.
Value is a valid unicode string corresponding to object or request header value.
Target is
'user' for container owner,
'system' for Storage nodes in container and Inner Ring nodes,
'others' for all other request senders,
'pubkey:<key1>,<key2>,...' for exact request sender, where <key> is a hex-encoded 33-byte public key.
When both '--rule' and '--file' arguments are used, '--rule' records will be placed higher in resulting extended ACL table.
`,
Example: `frostfs-cli acl extended create --cid EutHBsdT1YCzHxjCfQHnLPL1vFrkSyLSio4vkphfnEk -f rules.txt --out table.json
frostfs-cli acl extended create --cid EutHBsdT1YCzHxjCfQHnLPL1vFrkSyLSio4vkphfnEk -r 'allow get obj:Key=Value others' -r 'deny put others'`,
Run: createEACL,
}
func init() {
createCmd.Flags().StringArrayP("rule", "r", nil, "Extended ACL table record to apply")
createCmd.Flags().StringP("file", "f", "", "Read list of extended ACL table records from text file")
createCmd.Flags().StringP("out", "o", "", "Save JSON formatted extended ACL table in file")
createCmd.Flags().StringP(commonflags.CIDFlag, "", "", commonflags.CIDFlagUsage)
_ = cobra.MarkFlagFilename(createCmd.Flags(), "file")
_ = cobra.MarkFlagFilename(createCmd.Flags(), "out")
}
func createEACL(cmd *cobra.Command, _ []string) {
rules, _ := cmd.Flags().GetStringArray("rule")
fileArg, _ := cmd.Flags().GetString("file")
outArg, _ := cmd.Flags().GetString("out")
cidArg, _ := cmd.Flags().GetString(commonflags.CIDFlag)
var containerID cid.ID
if cidArg != "" {
if err := containerID.DecodeString(cidArg); err != nil {
cmd.PrintErrf("invalid container ID: %v\n", err)
os.Exit(1)
}
}
rulesFile, err := getRulesFromFile(fileArg)
if err != nil {
cmd.PrintErrf("can't read rules from file: %v\n", err)
os.Exit(1)
}
rules = append(rules, rulesFile...)
if len(rules) == 0 {
cmd.PrintErrln("no extended ACL rules has been provided")
os.Exit(1)
}
tb := eacl.NewTable()
commonCmd.ExitOnErr(cmd, "unable to parse provided rules: %w", util.ParseEACLRules(tb, rules))
tb.SetCID(containerID)
data, err := tb.MarshalJSON()
if err != nil {
cmd.PrintErrln(err)
os.Exit(1)
}
buf := new(bytes.Buffer)
err = json.Indent(buf, data, "", " ")
if err != nil {
cmd.PrintErrln(err)
os.Exit(1)
}
if len(outArg) == 0 {
cmd.Println(buf)
return
}
err = os.WriteFile(outArg, buf.Bytes(), 0644)
if err != nil {
cmd.PrintErrln(err)
os.Exit(1)
}
}
func getRulesFromFile(filename string) ([]string, error) {
if len(filename) == 0 {
return nil, nil
}
data, err := os.ReadFile(filename)
if err != nil {
return nil, err
}
return strings.Split(strings.TrimSpace(string(data)), "\n"), nil
}

View file

@ -0,0 +1,90 @@
package extended
import (
"testing"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
"github.com/TrueCloudLab/frostfs-sdk-go/eacl"
"github.com/stretchr/testify/require"
)
func TestParseTable(t *testing.T) {
tests := [...]struct {
name string // test name
rule string // input extended ACL rule
jsonRecord string // produced record after successfull parsing
}{
{
name: "valid rule with multiple filters",
rule: "deny get obj:a=b req:c=d others",
jsonRecord: `{"operation":"GET","action":"DENY","filters":[{"headerType":"OBJECT","matchType":"STRING_EQUAL","key":"a","value":"b"},{"headerType":"REQUEST","matchType":"STRING_EQUAL","key":"c","value":"d"}],"targets":[{"role":"OTHERS","keys":[]}]}`,
},
{
name: "valid rule without filters",
rule: "allow put user",
jsonRecord: `{"operation":"PUT","action":"ALLOW","filters":[],"targets":[{"role":"USER","keys":[]}]}`,
},
{
name: "valid rule with public key",
rule: "deny getrange pubkey:036410abb260bbbda89f61c0cad65a4fa15ac5cb83b3c3abf8aee403856fcf65ed",
jsonRecord: `{"operation":"GETRANGE","action":"DENY","filters":[],"targets":[{"role":"ROLE_UNSPECIFIED","keys":["A2QQq7Jgu72on2HAytZaT6FaxcuDs8Or+K7kA4Vvz2Xt"]}]}`,
},
{
name: "missing action",
rule: "get obj:a=b others",
},
{
name: "invalid action",
rule: "permit get obj:a=b others",
},
{
name: "missing op",
rule: "deny obj:a=b others",
},
{
name: "invalid op action",
rule: "deny look obj:a=b others",
},
{
name: "invalid filter type",
rule: "deny get invalid:a=b others",
},
{
name: "invalid target group",
rule: "deny get obj:a=b helpers",
},
{
name: "invalid public key",
rule: "deny get obj:a=b pubkey:0123",
},
}
eaclTable := eacl.NewTable()
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
err := util.ParseEACLRule(eaclTable, test.rule)
ok := len(test.jsonRecord) > 0
require.Equal(t, ok, err == nil, err)
if ok {
expectedRecord := eacl.NewRecord()
err = expectedRecord.UnmarshalJSON([]byte(test.jsonRecord))
require.NoError(t, err)
actualRecord := eaclTable.Records()[len(eaclTable.Records())-1]
equalRecords(t, expectedRecord, &actualRecord)
}
})
}
}
func equalRecords(t *testing.T, r1, r2 *eacl.Record) {
d1, err := r1.Marshal()
require.NoError(t, err)
d2, err := r2.Marshal()
require.NoError(t, err)
require.Equal(t, d1, d2)
}

View file

@ -0,0 +1,38 @@
package extended
import (
"os"
"strings"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-sdk-go/eacl"
"github.com/spf13/cobra"
)
var printEACLCmd = &cobra.Command{
Use: "print",
Short: "Pretty print extended ACL from the file(in text or json format) or for given container.",
Run: printEACL,
}
func init() {
flags := printEACLCmd.Flags()
flags.StringP("file", "f", "",
"Read list of extended ACL table records from text or json file")
_ = printEACLCmd.MarkFlagRequired("file")
}
func printEACL(cmd *cobra.Command, _ []string) {
file, _ := cmd.Flags().GetString("file")
eaclTable := new(eacl.Table)
data, err := os.ReadFile(file)
commonCmd.ExitOnErr(cmd, "can't read file with EACL: %w", err)
if strings.HasSuffix(file, ".json") {
commonCmd.ExitOnErr(cmd, "unable to parse json: %w", eaclTable.UnmarshalJSON(data))
} else {
rules := strings.Split(strings.TrimSpace(string(data)), "\n")
commonCmd.ExitOnErr(cmd, "can't parse file with EACL: %w", util.ParseEACLRules(eaclTable, rules))
}
util.PrettyPrintTableEACL(cmd, eaclTable)
}

View file

@ -0,0 +1,15 @@
package extended
import (
"github.com/spf13/cobra"
)
var Cmd = &cobra.Command{
Use: "extended",
Short: "Operations with Extended Access Control Lists",
}
func init() {
Cmd.AddCommand(createCmd)
Cmd.AddCommand(printEACLCmd)
}

Some files were not shown because too many files have changed in this diff Show more