Compare commits

...

807 commits

Author SHA1 Message Date
79ba34714a [#79] cli: Fix panic when setting domain for container
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-03-01 11:11:11 +03:00
6d4f48f37a [TrueCloudLab#78] .github: Fix CODEOWNERS
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-28 16:44:50 +03:00
e9f3c24229 [#65] Use strings.Cut instead of strings.Split* where possible
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-28 13:39:14 +03:00
88e3868f47 [#37] cli: Add nns-name and nns-zone for container create
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-28 13:37:23 +03:00
6925fb4c59 [TrueCloudLab/hrw#2] node: Use typed HRW methods
Update HRW lib and use typed HRW methods to sort shards and nodes

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-02-28 13:36:25 +03:00
c3a7039801 [TrueCloudLab/hrw#2] node: Optimize shard hash
Compute shard hash only once

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-02-28 13:36:25 +03:00
a1ab25b33e [#72] .github: Fix CODEOWNERS
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-22 16:29:58 +03:00
73bb590cb1 [#64] node: Use pool_size_local and separate pool for local puts
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-02-22 13:43:19 +03:00
7eaf159a8b [#63] adm: Fix contract wallet creation
Create contract wallet only by init and update-config command.

Close #63

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-02-22 10:08:37 +03:00
cb5468abb8 [#66] node: Replace interface{} with any
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-02-21 16:47:07 +03:00
3d873237d5 [#44] Update Changelog
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-02-21 10:00:28 +03:00
633c5a35de [#44] adm: Support multiple configs
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-02-21 10:00:28 +03:00
5f06232d34 [#44] cli: Support multiple configs
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-02-21 10:00:28 +03:00
bed5a36235 [#44] ir: Support multiple configs
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-02-21 10:00:28 +03:00
87e69b9349 [#44] node: Support multiple configs
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-02-21 10:00:28 +03:00
Pavel Karpy
337049b2ce [#56] node: Allow reading expired locked object
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-21 09:56:57 +03:00
Pavel Karpy
3beef10f89 [#61] node: Do not fetch missing objects
If an object is missing in a `meta`, shard should not look for it in
a `blobstor`.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 14:47:38 +03:00
5303736acd [#62] Fix CHANGELOG.md
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-02-20 14:25:21 +03:00
22f3c7d080 [#1868] Reload config for pprof and metrics on SIGHUP
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-20 13:53:27 +03:00
2b755ddb12 [#2260] node: Use a separate client cache for PUT service
Currently, under a mixed load one failed PUT can lead to closing
connection for all concurrent GETs. For PUT it does no harm: we have
many other nodes to choose from. For GET we are limited by `REP N`
factor, so in case of failover we can close the connection with the only
node posessing an object, which leads to failing the whole operation.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
0b61a3c961 [#2260] network/cache: Ignore clients only on Dial errors
The problem is that accidental timeout errors can make us to ignore
other nodes for some time. The primary purpose of the whole ignore
mechanism is not to degrade in case of failover. For this case,
closing connection and limiting the amount of dials is enough.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
bf1e59bb83 [#2260] network/cache: Ignore context cancelled errors
Timeouts on client side should node affect inter-node communication.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
2567f8020e [#2260] services/object: Do not assemble object with TTL=1
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
d1d123d180 [#2234] writecache: Fix possible panic in initFlushMarks
In case we have many small objects in the write-cache, `indices` should
not be reused between iterations.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
315141dc2c [#2252] fstree: Allow concurrent writes
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
Pavel Karpy
b422ac9f94 [#2164] node: Fix multi-client error reporting
Missing `ReportError` method did not allow casing multi-client interface to
`errorReporter` interface and dropping broken connections.
`replicationClient` embeds that interface, and it is widely used across
node's code. Embedded interface does not allow casting its parent structure
to `errorReporter` and breaks multi client error reporting logic.
Multi-client scheme is extremely hard to maintain, it makes unpredictable
casts and does not allow tracking code flow, so it will be refactored in the
future anyway.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 13:53:27 +03:00
Pavel Karpy
95ee905861 [#2244] node: Fix subscriptions lock
Subscribing without async listening could lead to a dead-lock in the
`neo-go` client.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 13:53:27 +03:00
Pavel Karpy
07ec51ea60 [#2244] node: Add object address to WC's operations
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 13:53:27 +03:00
Pavel Karpy
dbbbef9ddb [#2244] node: Update expired storage ID by WC
Previously, node could get an "infinite" small object: it could be expired
and thus could not be flushed (update its storage ID) to metabase => could
not be marked as flushed => node never removes such object and repeat all
the cycle one more time. If object exists and is not marked with GC (meta
returns `ErrObjectIsExpired`, not `ObjectNotFound` and not
`ObjectAlreadyRemoved`), its ID is safe to update _in the same_ bbolt
transaction.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 13:53:27 +03:00
351fdd9fa2 [#2246] node: Allow to configure tombsone lifetime
Currently, DELETE service sets tombstone expiration epoch to
`current epoch + 5`. This works less than ideal in private networks
where an epoch can be e.g. 10 minutes. In this case, after a node is
unavailable for more than 1 hour, already deleted objects have a chance
to reappear.

After this commit tombstone lifetime can be configured.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
6fd88a036f [#2241] metrics: Fix request count metrics names
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
2272c55c4d [#2238] engine: Add test for component initialization failures
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
5cb2c5ae62 [#2238] engine: Add test for component initialization failures
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
59748b7ae8 [#2238] neofs-node: Gracefully handle shard initialization errors
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
427fe276f2 [#2238] shard: Try closing all components
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
c53903ccd0 [#2238] engine: Make Open and Init similar
1. Both could initialize shards in parallel.
2. Both should close shards after an error.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
e0309e398c [#2239] writecache: Fix possible deadlock
LRU `Peek`/`Contains` take LRU mutex _inside_ of a `View` transaction.
`View` transaction itself takes `mmapLock` [1], which is lifted after tx
finishes (in `tx.Commit()` -> `tx.close()` -> `tx.db.removeTx`)

When we evict items from LRU cache mutex order is different:
first we take LRU mutex and then execute `Batch` which _does_ take
`mmapLock` in case we need to remap. Thus the deadlock.

[1] 8f4a7e1f92/db.go (L708)

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
58367e4df6 [#2232] pilorama: Merge in-queue batches
To achieve high performance we must choose proper values for both
batch size and delay. For user operations we want to set low delay.
However it would prevent tree synchronization operations to form big
enough batches. For these operations, batching gives the most benefit
not only in terms of on-CPU execution cost, but also by speeding up
transaction persist (`fsync`).
In this commit we try merging batches that are already
_triggered_, but not yet _started to execute_. This way we can still
query batches for execution after the provided delay while also allowing
multiple formed batches to execute faster.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
236c4af615 [#2224] adm: Use native neo-go sessions in dump-hashes
If we had lots of domains in one zone, `dump-hashes` for all others
can miss some domains, because we need to restrict ourselves with _some_
number.
In this commit we use neo-go sessions by default, with a proper
failback to in-script iterator unwrapping.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
Pavel Karpy
40822adb51 [#2213] node: Do not return object expired object
"Object is expired" means that object is presented in `meta` but it is not
`ObjectNotFound` error. Previous implementation made `shard` search for an
object without `meta` which was an error.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 13:53:27 +03:00
Roman Khimov
ad93d4db7c CHANGELOG: add more fancy glyphs
How could you forget adding it?

Signed-off-by: Roman Khimov <roman@nspcc.ru>
2023-02-20 13:53:27 +03:00
Roman Khimov
a6f071d66f CHANGELOG: fix whitespacing errors
Signed-off-by: Roman Khimov <roman@nspcc.ru>
2023-02-20 13:53:27 +03:00
9afe86ba3e [#2212] morph: Fix subscription restoration
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
c43b2dbac9 [#1465] Add log entry for morph components shutdown action
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-17 12:13:00 +03:00
85cf1f47ac [#1465] node: Prevent process from killing by systemd when shutting down
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-17 12:13:00 +03:00
362f24953a [#47] shard: Switch container size metric from physical to logical capacity
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-02-17 12:03:42 +03:00
Pavel Karpy
901d62567d [#57] node: Broadcast link objects
It boosts object assembling by an _average_ container node.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-17 11:58:27 +03:00
Aleksey Pastukhov
269a4e9b50 [#53] Fix dirty in version
Signed-off-by: Aleksey Pastukhov <a.pastukhov@yadro.com>
2023-02-16 11:10:43 +03:00
3e5bc394b5 [#48] adm: Add initialize test for 1 node
Single node is used in dev-env, worth testing.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-16 09:07:18 +03:00
1d3669232e [#48] adm: Allow using nonzero magic with local client
neo-go actor API uses `getVersion` call which returned incorrect magic.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-16 09:07:18 +03:00
204cd3a11c [#31] fstree: Optimize treePath
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-10 12:49:31 +03:00
dee4498c1e [#31] fstree: Do not check for a file existence twice
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-10 12:49:31 +03:00
abbecf49d6 [#31] fstree: Speedup string-to-address conversion
```
name                  old time/op    new time/op    delta
_addressFromString-8    1.25µs ±30%    1.02µs ± 6%  -18.49%  (p=0.000 n=9+9)

name                  old alloc/op   new alloc/op   delta
_addressFromString-8      352B ± 0%      256B ± 0%  -27.27%  (p=0.000 n=9+10)

name                  old allocs/op  new allocs/op  delta
_addressFromString-8      6.00 ± 0%      4.00 ± 0%  -33.33%  (p=0.000
n=10+10)
```

Also, assure compiler that `s` doesn't escape:
Before this commit:
```
./fstree.go:74:24: leaking param: s
./fstree.go:90:6: moved to heap: addr
```

After this commit:
```
./fstree.go:74:24: s does not escape
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-10 12:49:31 +03:00
ab21d90cfb [#1794] shard: Add increasing case for the payload size metric
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-02-09 13:30:23 +03:00
5ffa826897 [#1] Update Changelog
Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-02-06 17:41:14 +03:00
cb016d53a6 [#1] Fix comments and error messages
Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-02-06 17:41:14 +03:00
c761a95eef [#1] Fix project name in control service
Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-02-06 17:41:14 +03:00
f825cfac78 [#1] Fix project name in comments here and there
Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-02-06 17:41:14 +03:00
c6645ef775 [#1] Documentation rebranding
Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-02-06 17:41:14 +03:00
488eece25f [#1] Fix Debian package references to old upstream
Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-02-06 17:41:14 +03:00
1cedd446bb [#1] Fix sample configs
- Update sample configs to match unit tests
- Remove Docker container for N3 testnet
  Will return with updates when FrostFS enters N3

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-02-06 17:41:14 +03:00
1858f11146 [#1] Fix viper env prefix in cli tools
Changing env prefix and corresponding example config files.

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-02-06 17:41:14 +03:00
Pavel Karpy
73bc1b0b68 [#38] node: Fix linter warnings
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-06 17:27:54 +03:00
515c60bdf4 [#1889] adm: Add command morph netmap-candidates
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-06 17:26:34 +03:00
ee24815748 [#1889] Move flag --config in cmd/frostfs-adm/internal/commonflags/flags.go
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-06 17:26:34 +03:00
2b09564355 [#1889] Move netmap.go and exit.go from cli to cmd/internal/common
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-06 17:26:34 +03:00
5a9d6a09d8 [#8] cli: Set flag mode required for control shards set-mode
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-06 16:45:03 +03:00
2540598779 [#36] Packaging: allow any version
Also updated changelog

Signed-off-by: Dmitriy Zabolotskiy <d.zabolotskiy@yadro.com>
2023-01-31 16:33:32 +03:00
5d64a354cb [#1] Fix naming in FrostFS Adm help output
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-01-31 11:24:50 +03:00
d7e9e2ef9e [#1] Fix naming in FrostFS Lens help output
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-01-31 11:24:50 +03:00
d31d8c5335 [#1] Fix naming in FrostFS CLI help output
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-01-31 11:24:50 +03:00
406ff1360f [#1] Fix version output for all compiled binaries
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-01-31 11:24:50 +03:00
Pavel Karpy
89a0266f5e [#1794] metrics: Track physical object capacity per shard
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-01-26 20:06:28 +03:00
Evgenii Stratonikov
9513f163aa [#2116] metrics: Track physical object capacity in the container
Currently we track based on `PayloadSize`, because it is already stored
in the metabase and it is easier to calculate without slowing down the
whole system.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-01-26 20:06:28 +03:00
d65a95a2c6 [#28] pilorama: Remove LogMove struct
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
46c62be7e8 [#28] Fix linter issues
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
072a7d61ab [#2176] neofs-node: Do not cache full container list
We rarely need to list all containers: as one example
we need it for tree service synchronization once per epoch.
Given that cache TTL has the order of block time it makes no sense
to cache the list of all containers.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
8078af3424 [#2176] neofs-node: Do not invalidate old cache items
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
e5a40b90b6 [#2176] neofs-node: Peek during cache invalidation
`Get` needs write mutex and makes our item move to top.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
93eb72ef44 [#2176] neofs-node: Remove unused field from morphContainerWriter
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
8ee590794f [#1962] cli: common.PrintVerbose prints via cobra.Command.Printf
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
cff4184cd3 [#2128] Add doc for extended headers
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
c72576e72f [#2208] engine: Log time-consuming shard operations
Currently the only way to tell whether `evacuate/set-mode` is finished
is to set a very big timeout and _hope_ that the operation will finish.
In this commit we add INFO logs for such operations which should
simplify the life of an administrator.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
87f0e3ea25 [#2208] fstree: Rename file after write
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
792319a044 [#2208] fstree: Remove file if there was an error during write
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
67c97c6804 [#2210] services/tree: Drop messages not in queue
Currently, under high load clients are blocked on channel send
and the number of goroutines can increase indefinitely.
In this commit we drop replication messages if send/recv queue is full
and rely on a background synchronization.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
25d5995cef [#2210] pilorama: Allocate bucket name outside of batches
1. Reduce allocations inside transactions.
2. Do not encode container ID to string: it allocates a lot and takes more
space.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
165a600624 [#2210] pilorama: Reduce the amount of keys per node
Under high load we are limited by the _amount_ of keys we need to update
in a single transaction. In this commit we try storing all state
with a single key.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Pavel Karpy
64a5294b27 [#2200] shard: Do not fetch big objects from blobovniczas
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Pavel Karpy
91757329ae [#2200] shard: Fix blobstor obj fetching
In the previous implementation any non-nil error that preceded object
fetching from blobstor led to iterating over every storage (in other words,
no storage ID information was taken into account). Now storage ID is
skipped only if metabase (storage ID source) returns any error.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Pavel Karpy
cf1a91a758 [#2206] blobovnicza: Use Latin letters in the code
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Pavel Karpy
c33ad3c474 [#2164] node: Use reconnect_interval from config
Not always the default one.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
0d8366f475 [#2207] object/acl: Return status error for expired session token
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
6451f019d2 [#2203] shard: Do not panic in Close after unsuccessful Init
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Evgenii Stratonikov
6efa93be0a [#1621] services/tree: Return Apply result asyncronously
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Evgenii Stratonikov
ac81c70c09 [#1621] pilorama: Batch related operations
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
9009612a82 [#2198] blobovniczatree: Properly handle concurrent active blobovnicza update
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
cedbd380f2 [#2197] pilorama: Close database in degraded mode
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Pavel Karpy
1d21b1e3e8 [#1978] node: Do not drop clients on split errors
After the reconnection interval feature there was an bug related to the big
objects collecting: split error is returned from a client directly, not
via API status and was considered as a connection error.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
554b85411f [#2190] services/object: Log service error with INFO level
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
b0ad1b9ed2 [#2193] pilorama: Use do in TreeMove
It should be similar to a `TreeAddByPath`. `applyOperation` is used for
`Apply` when the operation can be inserted in the middle of a log.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
ba393e3e91 [#2188] engine: Fix panic during setting shard mode
Under load changing shard mode can lead to it being removed from the
list during some other PUT.
```
Dec 28 07:01:26 az neofs-node[364505]: panic: runtime error: invalid memory address or nil pointer dereference
Dec 28 07:01:26 az neofs-node[364505]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xc9fbb1]
Dec 28 07:01:26 az neofs-node[364505]: goroutine 11791912 [running]:
Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).putToShard(0xc000435490, {0xc0003f7a28?, 0xc0001192c0?}, 0x2, {0x0, 0x>
Dec 28 07:01:26 az neofs-node[364505]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:91 +0x1b1
Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).put.func1(0xc000435490?, {0xc0003f7a28?, 0xc0001192c0?})
Dec 28 07:01:26 az neofs-node[364505]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:71 +0x19c
Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).iterateOverSortedShards(0x1?, {{0x62, 0x23, 0xfe, 0x60, 0x67, 0xd5, 0x>
Dec 28 07:01:26 az neofs-node[364505]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/shards.go:225 +0xc8
Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).put(0xc000435490, {0x1?})
Dec 28 07:01:26 az neofs-node[364505]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:66 +0x2a9
Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).Put.func1()
Dec 28 07:01:26 az neofs-node[364505]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:43 +0x2a
Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).execIfNotBlocked(0x8?, 0x38?)
Dec 28 07:01:26 az neofs-node[364505]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/control.go:147 +0xcf
Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).Put(0xc4df775a80?, {0x0?})
Dec 28 07:01:26 az neofs-node[364505]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:42 +0x65
Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.Put(0xc06d928b80?, 0xc06b1b8dc8?)
Dec 28 07:01:26 az neofs-node[364505]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:158 +0x19
Dec 28 07:01:26 az neofs-node[364505]: main.engineWithoutNotifications.Put({0x20301b?}, 0x20301b?)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
118783c4cf [#26] .github: Fix CODEOWNERS
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:23 +03:00
3d57f4c961 [#2179] test: Fix test TestEvacuateNetwork/multiple_shards,_evacuate_many
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-01-24 13:37:49 +03:00
d4d2a8c865 [#11] Rename NeoFS to FrostFS in docs
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-12 08:55:47 +03:00
3d1d2ee7b1 [#11] Regenerate proto files
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-12 08:55:47 +03:00
054bc4a727 [#11] Rename contract-related NeoFS occurences
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-12 08:55:47 +03:00
9cb4b4cc17 [#11] Rename neofsid contract to frostfsid
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-12 08:55:47 +03:00
023396e6a4 [#11] adm: Remove NeoFS mentions
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-12 08:55:47 +03:00
19d180b510 [#11] Rename neofs contract to frostfs
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-12 08:55:47 +03:00
e355442532 [#10] .docker: Update go version to 1.18
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-01-10 12:27:25 +03:00
cc71e6902a [#7] Rebranding leftovers
- logo update
- docker image build fixes
- README and other texts updates

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-01-09 11:41:51 +03:00
fdb0affc31 [#5] frost-node: Used generic cache
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
9936b112b8 [#5] blobstor: Use generic LRU cache
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
f0be0befc5 [#5] services/object_manager: Use generic LRU cache
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
1b3374ac7f [#5] services/tree: User generic LRU cache
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
d0a0432a51 [#5] innerring: Use generic LRU cache
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
4155c1bdff [#5] writecache: Use generic LRU cache
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
8f61cc1dcc [#5] policer: Use generic LRU client
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
55b403e0ee [#5] morph/client: Use generic LRU cache
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
d8fb9c85eb [#5] go.mod: Bump supported go version to 1.18
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
6f8dd816fb [#2189] go.mod: Update nats-io
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
26560b6b8d [#2184] go.mod: Update dependencies
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
0272218eb9 [#2184] compression: Properly calculate upper bound
If the data is not compressible allocating `len(data)` will lead to a
slice reallocation. For a compressible data the results for small size
are flaky and we allocate a bit more. However, it feels right to use a
provided function if we need to pick any size at all.

```
name                                                           old time/op    new time/op    delta
Compression/size=128/zeroed_slice-8                              2.23µs ±12%    2.06µs ± 6%   -7.35%  (p=0.009 n=10+10)
Compression/size=128/not_so_random_slice_(block_=_123)-8         19.0µs ±10%    15.8µs ±16%  -17.09%  (p=0.000 n=9+10)
Compression/size=128/random_slice-8                              17.6µs ±15%    16.1µs ±16%     ~     (p=0.075 n=10+10)
Compression/size=1024/zeroed_slice-8                             3.05µs ±11%    2.84µs ±10%     ~     (p=0.089 n=10+10)
Compression/size=1024/not_so_random_slice_(block_=_123)-8        18.1µs ± 6%    18.2µs ±12%     ~     (p=0.971 n=10+10)
Compression/size=1024/random_slice-8                             48.6µs ± 6%    45.6µs ± 5%   -6.07%  (p=0.006 n=10+9)
Compression/size=32768/zeroed_slice-8                            26.8µs ± 3%    28.7µs ± 8%   +7.23%  (p=0.001 n=10+10)
Compression/size=32768/not_so_random_slice_(block_=_123)-8       44.3µs ± 8%    43.7µs ±13%     ~     (p=0.762 n=8+10)
Compression/size=32768/random_slice-8                            97.3µs ±32%    68.9µs ±15%  -29.13%  (p=0.000 n=10+10)
Compression/size=33554432/zeroed_slice-8                         29.8ms ± 9%    30.3ms ±17%     ~     (p=1.000 n=9+9)
Compression/size=33554432/not_so_random_slice_(block_=_123)-8    33.1ms ±14%    30.3ms ±11%   -8.61%  (p=0.043 n=10+10)
Compression/size=33554432/random_slice-8                         41.7ms ± 3%    30.1ms ± 8%  -27.72%  (p=0.000 n=9+10)

name                                                           old alloc/op   new alloc/op   delta
Compression/size=128/zeroed_slice-8                                128B ± 0%      144B ± 0%  +12.50%  (p=0.000 n=10+10)
Compression/size=128/not_so_random_slice_(block_=_123)-8           384B ± 0%      144B ± 0%  -62.50%  (p=0.000 n=10+10)
Compression/size=128/random_slice-8                                384B ± 0%      144B ± 0%  -62.50%  (p=0.000 n=10+10)
Compression/size=1024/zeroed_slice-8                             1.02kB ± 0%    1.15kB ± 0%  +12.50%  (p=0.000 n=10+10)
Compression/size=1024/not_so_random_slice_(block_=_123)-8        1.02kB ± 0%    1.15kB ± 0%  +12.50%  (p=0.000 n=10+10)
Compression/size=1024/random_slice-8                             2.56kB ± 0%    1.15kB ± 0%  -55.00%  (p=0.000 n=10+10)
Compression/size=32768/zeroed_slice-8                            32.8kB ± 0%    41.0kB ± 0%  +25.00%  (p=0.000 n=10+10)
Compression/size=32768/not_so_random_slice_(block_=_123)-8       32.8kB ± 0%    41.0kB ± 0%  +25.00%  (p=0.000 n=10+10)
Compression/size=32768/random_slice-8                            81.9kB ± 0%    41.0kB ± 0%  -50.00%  (p=0.000 n=10+10)
Compression/size=33554432/zeroed_slice-8                         33.6MB ± 0%    33.6MB ± 0%   +0.02%  (p=0.000 n=9+9)
Compression/size=33554432/not_so_random_slice_(block_=_123)-8    33.6MB ± 0%    33.6MB ± 0%   +0.02%  (p=0.000 n=8+10)
Compression/size=33554432/random_slice-8                         75.5MB ± 0%    33.6MB ± 0%  -55.55%  (p=0.000 n=10+10)

name                                                           old allocs/op  new allocs/op  delta
Compression/size=128/zeroed_slice-8                                1.00 ± 0%      1.00 ± 0%     ~     (all equal)
Compression/size=128/not_so_random_slice_(block_=_123)-8           2.00 ± 0%      1.00 ± 0%  -50.00%  (p=0.000 n=10+10)
Compression/size=128/random_slice-8                                2.00 ± 0%      1.00 ± 0%  -50.00%  (p=0.000 n=10+10)
Compression/size=1024/zeroed_slice-8                               1.00 ± 0%      1.00 ± 0%     ~     (all equal)
Compression/size=1024/not_so_random_slice_(block_=_123)-8          1.00 ± 0%      1.00 ± 0%     ~     (all equal)
Compression/size=1024/random_slice-8                               2.00 ± 0%      1.00 ± 0%  -50.00%  (p=0.000 n=10+10)
Compression/size=32768/zeroed_slice-8                              1.00 ± 0%      1.00 ± 0%     ~     (all equal)
Compression/size=32768/not_so_random_slice_(block_=_123)-8         1.00 ± 0%      1.00 ± 0%     ~     (all equal)
Compression/size=32768/random_slice-8                              2.00 ± 0%      1.00 ± 0%  -50.00%  (p=0.000 n=10+10)
Compression/size=33554432/zeroed_slice-8                           1.00 ± 0%      1.00 ± 0%     ~     (all equal)
Compression/size=33554432/not_so_random_slice_(block_=_123)-8      1.00 ± 0%      1.00 ± 0%     ~     (all equal)
Compression/size=33554432/random_slice-8                           2.00 ± 0%      1.00 ± 0%  -50.00%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
874be99076 [#2184] go.mod: Update neo-go to v0.100.1
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
71c64ae253 Release v0.35.0
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-12-30 11:07:35 +03:00
Mikhail Petrov
cfdb53a788 [#614] Add logo for dark theme
Signed-off-by: Mikhail Petrov <mike@nspcc.ru>
2022-12-30 11:07:35 +03:00
a2fe912d1a [#2164] neofs-node: Use a separate client cache for replicator
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
741482c26f [#2164] neofs-node: Allow to set reconnect interval
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
6f5edac730 [#2164] network/cache: Do not reconnect to failed clients immediately
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
f3caf6acfe [#2164] network/cache: Separate mutex for addr
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
0ace28e43d [#2175] blobovniczatree: Close all non-active blobovniczas
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
c1cf418956 [#2175] blobovniczatree: Make function parameters more descriptive
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
b4e90cdf51 [#2165] pilorama: Optimize TreeApply when used for synchronization
Because synchronization _most likely_ will have apply already existing
operations, it is much faster to check their presence in a read
transaction. However, always doing this will degrade the perfomance
for normal `Apply`. And, let's be honest, it is already not good.
Thus we add a separate parameter which specifies whether this logic is
enabled.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
f9fcd85363 [#2165] services/tree: Remember starting height for the synchronization
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
06137dbf8e [#2165] services/tree: Do not export synchronizeAllTrees
It is used only privately.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
c299b98afe [#2165] services/tree: Parallelize synchronization
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
de9957e076 [#2165] services/tree: Always synchronize all containers
In case of split-brain we must synchronize everything.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
d3054e577a [#2165] services/tree: Allow to set custom synchronization interval
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
6a4e5e6f0a [#2144] node: Try node's private key if dynamic token fetching failed
`GETRANGEHASH` request spawns `GETRANGE` requests if an object could not be
found locally. If the original request contains session, it can be static
and, therefore, fetching session key can not be performed successfully.
As the best effort a node could request object's range with its own key.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
86a4fba571 [#2144] node: Clarify KeyStorage.GetKey method
Actualize the doc, fix API status error return.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
b7a99c757a [#2097] cli: Clarify help for --expire-at parameter for commands object lock/put and bearer create
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-12-30 11:07:35 +03:00
e406036629 [#2166] go.mod: Update dependencies
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
21717262ec [#2016] shard: Check meta first on Get
`meta` should prevent returning removed objects (`GCMark` and `TS` relations
are `meta` abstractions).

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
74ec71446f [#2167] shard: Do not use write-cache by default in Head
Both `meta` and `write-cache` are expected to have a fast underlying disk,
so it does not seem like an optimisation. Moreover, `write-cache`'s `Head`
is a `Get` with payload cutting, it _must_ use more memory for no reason
(`meta` was created for such requests). Also, `write-cache` does not allow
performing any "meta" relations checks (such as locking, tombstoning).

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
1608fd1c07 [#2167] write-cache: Add "write-cache" to its logs
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
eea2892109 [#1956] node: Lock shard's mode on its methods switch
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
33d279a3f2 [#2152] cli: Do not search for LOCK objects when delete container when session provided
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
04b5ec759b [#2139] object/put: Use sync.Pool for temporary payloads
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
9e0decd12d [#2162] services/tree: Close connection after the syncronization
There was a goroutine leak here.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
e1c3bdbfa6 [#1621] pilorama: Remove Timestamp field from nodeInfo
It is already present in `Meta`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
1044adbe94 [#1621] pilorama: Improve memory allocation
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
2539d466a6 [#1621] pilorama: Seek after cursor invalidation
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
e9ba8931f8 [#1621] pilorama: Simplify bucket creation
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
fe7ddfdc6a [#1621] pilorama: Compare memory forests properly
Node children are not sorted and could occur in any order.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-30 11:07:35 +03:00
Pavel Karpy
306609030a [#2159] node: Add tree replication timeout configuration
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
edb1428248 [#2022] Add metric readonly to get shards mode
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-12-30 11:07:35 +03:00
e5c304536b [#2161] pilorama: Do not apply already existing operations
Speeds up synchronization a bit.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
3bb5a320d7 [#2154] services/tree: Do not log an error when synchronizing container of 1 node
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
387d1e2977 [#2127] services/tree: Randomize node order for synchronization
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
21c58c92a9 [#2145] meta: Do allow force inhuming a locked object
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
a30310b2ca [#2145] cli: Do not print "rpc error:" twice
That error part is already included in GRPC's library.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
b207dc424f [#2158] policer: Reduce default cache size
We use cache to avoid policing the same object multiple times in a short
time span (< 30 seconds). If we have 200_000 objects in a blobstor, it is a bit useless
-- if it takes 1 second to process an object and we have `replicator.pool_size: 20`
in config, the next iteration will happen in 10_000 second which is much
larger than 30 second. However we still consume a lot of memory, so it
makes sense to use saner default.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
0244f2e5ce [#2156] core/netmap: Change node addresses iteration order
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Leonard Lyubich
a68ff67ed8 [#2106] cli: Verify container owner in container delete command
In NeoFS containers can be removed on behalf of its owner only. To
improve user experience, there is a need to add ownership check to the
removal command of the NeoFS CLI.

Check container ownership in `container delete` command `Run` function.
The check can be skipped by `--force` option.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
73ef5b18c7 [#2092] go.mod: Update SDK
Includes a fix that adds error messages' text, it affects incomplete object
put error message.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
b413094704 [#2095] node: Fix collecting child objects
Stop child objects collection if the last returned object (the most "left"
object in the collected chain) starts exactly from the `GETRANGE`'s `from`
value.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-12-30 11:07:35 +03:00
Pavel Karpy
350eecfa13 [#2095] node: Do not allow GETRANGE requests with zero length
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-12-30 11:07:35 +03:00
Pavel Karpy
d6196c3971 [#2095] cli: Do not panic on object range
Also, includes range parsing error messages enhancement.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-12-30 11:07:35 +03:00
0b78af467e [#2140] engine: Fix error handling in TreeMove
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
923f84722a Move to frostfs-node
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-28 15:04:29 +03:00
Evgenii Stratonikov
42554a9298 [#2068] writecache: Remove deleted objects from the writecache
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-02 11:52:05 +03:00
Evgenii Stratonikov
4a49ea0855 [#2068] writecache: Allow to open FSTree in read-only mode
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-02 11:52:05 +03:00
Evgenii Stratonikov
857d2dc3f5 [#2068] writecache: Optimize initial flush existence checking
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-02 11:52:05 +03:00
Evgenii Stratonikov
63f604e948 [#2068] blobstor: Allow to provide storage ID in Exists
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-02 11:52:05 +03:00
Evgenii Stratonikov
6ad2b5d5b8 [#2068] blobovnicza: Add Exists method
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-02 11:52:05 +03:00
9eccf3bbf5 [#1486] Update changelog
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-12-02 11:43:33 +03:00
b0eeb0dfcf [#1486] node: Use endless notary deposit for side chain
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-12-02 11:43:33 +03:00
ba74244d1b [#1486] ir: Use endless notary deposit for side chain
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-12-02 11:43:33 +03:00
b6a40241f8 [#1486] morph: Add method for endless notary deposit
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-12-02 11:43:33 +03:00
Pavel Karpy
d54022eacc [#2047] node: Do not send chunk twice on request forwarding
That could happen if a node forwards request to a node that closed the
connection during the original object stream.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-12-02 11:27:48 +03:00
Evgenii Stratonikov
bd25db5d4a [#1984] metrics: Use separate metrics for success/failed requests
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-01 14:18:10 +03:00
Evgenii Stratonikov
e21c472dc7 [#1984] services/object: Increase put_req_count after the request is processed
As it is specified in metrics description.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-01 14:18:10 +03:00
afabd6be91 [#2038] neofs-cli: Check the sufficiency of the number of nodes in the selector for replicas
Perform this check on container creation.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-12-01 11:32:27 +03:00
Pavel Karpy
50d28b72c3 [#2075] node, ir: Log notary state with info
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 19:26:33 +03:00
Pavel Karpy
0f41c09207 [#2075] node: Do not make notary requests on shutdown
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 19:26:33 +03:00
Pavel Karpy
b8c30b88f6 [#2075] morph: Do not do notary deposit with zero balance
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 19:26:33 +03:00
Pavel Karpy
c01d4ecb50 [#2080] morph: Close morph clients
Could be related to "websocket users limit reached" on the `neo-go` server
side when an SN/IR is rebooting repeatedly.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 19:26:33 +03:00
Pavel Karpy
3d0768a1d3 [#2061] node: Unify meta.Get benchmarks
Make them get exactly one (different) object per a bench iteration.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 18:29:14 +03:00
Pavel Karpy
bc905f169d [#2061] meta: Add parallel bench for Get
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 18:29:14 +03:00
Pavel Karpy
761e82fecd [#2079] cli: Do not panic in object hash
Sign RPC requests with the provided key.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 16:58:52 +03:00
Pavel Karpy
960a2d0629 [#2081] ir: Set default key in IR's SDK clients
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 16:58:52 +03:00
8d15c14be6 [#2078] adm: Pack parameters for setPrice invocation
Contract arguments have to be packed.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-11-30 16:58:52 +03:00
Pavel Karpy
51963abce7 [#1972] node: Fix errors comments in the Put service
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 16:58:52 +03:00
Evgenii Stratonikov
7335a52f29 [#1732] pilorama: Improve logical error handling
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:58:52 +03:00
Evgenii Stratonikov
ae7b473768 [#2064] blobovniczatree: Remove index too big log
There is no need to log about a situation which is expected.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:53:18 +03:00
8a77b4638a [#2012] Add commands neofs-cli acl basic/extended print to show ACL table in human readable format
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-11-30 16:45:37 +03:00
Evgenii Stratonikov
59db66cdb6 [#2091] neofs-adm: Sign blocks properly in tests
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:45:13 +03:00
Evgenii Stratonikov
6882887bdd [#2091] neofs-adm: Do not query hashes via network
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:45:13 +03:00
Evgenii Stratonikov
c1ea6fd854 [#2091] morph/client: Use notary.FakeContractAccount wrapper
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:45:13 +03:00
Evgenii Stratonikov
b93be8869b [#2091] morph/client: Simplify code
1. Replace `mn` function with a `sigCount`.
2. Use `notary.FakeMultisigAccount` for account creation.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:45:13 +03:00
Evgenii Stratonikov
cfefebd5b3 [#2091] neofs-adm: Do not use deprecated methods
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:45:13 +03:00
Evgenii Stratonikov
6af52c46d8 [#2091] morph/client: Do not use deprecated methods
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:45:13 +03:00
51e886dd67 [#2090] neofs-cli: Remove --header from object get
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-11-30 11:46:02 +03:00
79130f781e [#2089] neofs-cli: Remove -g option from neofs-cli control ... and neofs-cli container create commands
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-11-30 09:14:50 +03:00
Evgenii Stratonikov
dd76ceadf1 [#2060] go.mod: Retract v1.* versions
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-29 09:29:42 +03:00
Pavel Karpy
ca3596dd05 [#2104] cli: OID signature in output
OID signature should always be present in an object; it does not relate
to the object split.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-29 08:52:09 +03:00
Evgenii Stratonikov
816c74d185 [#2075] morph/client: Ignore error if a transaction already exists
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
71853348b9 [#2063] morph/client: Support new hash format in morph nns client
Signed-off-by: Vladimir Domnich <v.domnich@yadro.com>
2022-11-19 11:01:04 +03:00
Pavel Karpy
ed4351aab0 [#2074] write-cache: Do not flush same object twice
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
dd225906a0 [#2074] write-cache: Remove unused variables
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
9f7ac6bf9f [#2069] innerring: Do not panic in Head
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
98e6dc5ce8 [#409] debian: Refactor storage service paths
Separate User data and Service data:
 - /var/lib/neofs/storage for service persistence
 - /srv/neofs for user data

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
660c38d07e [#2062] services/policer: Use a proper key for object cache
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
b1025bdb42 [#2057] meta: Fail write operations in R/O mode
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
fdeea1dfac [#2057] meta: Fix concurrent mode changes
Includes:
1. mode change read lock operation in every exported method that r/w the
underlying database;
2. returning `ErrDegradedMode` logical error if any exported method is
called in degraded (without a metabase) mode.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
3d6defd3e8 [#2057] meta: Do not lock the whole meta on GET
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
fa231b8c56 [#2057] blobstor: Block operations on a mode change
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
1779664644 [#2058] services/policer: Fix panic in shardPolicyWorker
```
2022/11/15 08:40:56 worker exits from a panic: runtime error: index out of range [0] with length 0
2022/11/15 08:40:56 worker exits from panic: goroutine 1188 [running]:
github.com/panjf2000/ants/v2.(*goWorker).run.func1.1()
	github.com/panjf2000/ants/v2@v2.4.0/worker.go:58 +0x10c
panic({0x1042b60, 0xc0015ae018})
	runtime/panic.go:1038 +0x215
github.com/nspcc-dev/neofs-node/pkg/services/policer.(*Policer).shardPolicyWorker.func1()
	github.com/nspcc-dev/neofs-node/pkg/services/policer/process.go:65 +0x366
github.com/panjf2000/ants/v2.(*goWorker).run.func1()
	github.com/panjf2000/ants/v2@v2.4.0/worker.go:68 +0x97
created by github.com/panjf2000/ants/v2.(*goWorker).run
	github.com/panjf2000/ants/v2@v2.4.0/worker.go:48 +0x68
```

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
b673d9e472 [#2053] engine: Do not switch mode because of logical errors
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
9a20498f34 [#1940] Removing all trees by container ID if tree ID is empty in pilorama.Forest.TreeDrop
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
426fe97990 [#2026] neofs-adm: Make contract update idempotent
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
634792077e [#1502] node: Store lock object on every container node
Includes extending listing methods in the Storage Engine with object types.
It allows tuning replication/policer algorithms: container nodes do
not remove `LOCK` objects as redundant and try to fulfill `LOCK` placement
on the ohter container nodes.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
fe09cd9c70 [#1502] core: Add AddressWithType
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
3b61cb4f49 [#1502] engine: Check all shards for LOCK'ing before inhuming
It allows keeping all the locked objects safe after metabase
resynchronization. Currently, all `LOCK` objects are broadcast to all nodes
in a container, it guarantees `LOCK` object presence in a regular situation.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
34e8d2ba56 [#1502] shard: Add IsLocked method
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
9a039ba582 [#1502] meta: Add IsLocked method
It gets an object and returns its locking status.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
7ef0303e13 [#2003] neofs-node: Allow to configure replicator pool size
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
659011c143 [#2048] network/cache: Optimize ClientCache
1. Remove a layer of indirection for mutex, `ClientCache` is already
   used by pointer.
2. Fix duplication of a `AllowExternal` field.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
597ed18269 [#2048] neofs-node: Use a separate client cache for client operations
Background workers can prevent user operations to complete because of
locking in cache.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
d5a14041e0 [#2040] node: Do not attach tokens in the assembly process
A container node is expected to have full "get" access to assemble the
object.
A non-container node is expected to forward any request to a container node.
Any token is expected to be issued for an original request sender not for a
node so any new request is invalid by design with that token.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
fd61bdadcb [#2040] node: Attach original meta to the spawned requests
Do not lose meta information of the original requests: cache session and
bearer tokens of the original request b/w a new generated ones. Middle
request wrappers should not contain any meta information, since it is
useless (e.g. ACL service checks only the original tokens).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
d65604ad30 [#1985] blobstor: Allow to report multiple errors to caller
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
f2d7e65e39 [#2035] engine: Allow moving to degraded from background workers
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
c85bea15ef [#1978] cli: Add children to the static session on DELETE
If an external session is provided and is not opened by CLI itself, add
children objects to it too. It fixes "not found" errors when removing a big
object with a predefined session (from a file).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
f48b1de54b [#2029] cli: Fix panic caused by flag redefinition
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
09a59fef56 [#2029] cli: Allow attaching static session to object hash
All the other object commands already have it.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
481b48b942 [#2028] node: Check session token's NBF and IAT
ACL service did not check "Not Valid Before" and "Issued At" claims.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
aadd2ad050 [#2028] node: Do not wrap malformed request errors
After presenting request statuses on the API level, all the errors are
unwrapped before sending to the caller side. It led to a losing invalid
request's context.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
2522d924b9 [#2037] services/object: Fix concurrent map writes in traverser
```
fatal error: concurrent map writes

goroutine 4337 [running]:
github.com/nspcc-dev/neofs-node/pkg/services/object/put.(*traversal).submitProcessed(...)
        github.com/nspcc-dev/neofs-node/pkg/services/object/put/distributed.go:78
github.com/nspcc-dev/neofs-node/pkg/services/object/put.(*distributedTarget).iteratePlacement.func1()
        github.com/nspcc-dev/neofs-node/pkg/services/object/put/distributed.go:198 +0x265
github.com/panjf2000/ants/v2.(*goWorker).run.func1()
        github.com/panjf2000/ants/v2@v2.4.0/worker.go:68 +0x97
created by github.com/panjf2000/ants/v2.(*goWorker).run
        github.com/panjf2000/ants/v2@v2.4.0/worker.go:48 +0x65
```

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
37f813604f [#2000] cli: Provide a bearer token to spawned HEAD by DELETE
If a `neofs-cli object delete` operation is performing using a bearer token,
add it to the new `HEAD` requests that collects children OIDs.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
b0e94b6a6b [#1906] writecache: Do not require read-only mode in Flush
It was needed before we started to flush during transition to
`degraded` mode. Now it is confusing.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
aa478f1def [#2024] services/object: Unify status errors
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
3875fef542 [#2024] services/object: Cover corner cases for children OutOfRange
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
b9fb0d6050 [#1972] node: Fix object format unit tests
Includes:
1. Unused func removal;
2. Err check of the `Sign` method.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
aab398f4f5 [#1972] node: Do not save objects if node not in a container
Do not use node's local storage if it is clear that an object will be
removed anyway as a redundant. It requires moving the changing local storage
logic from the validation step to the local target implementation.
It allows performing any relations checks (e.g. object locking) only if a
node is considered as a valid container member and is expected to store
(stored previously) all the helper objects (e.g. `LOCK`, `TOMBSTONE`, etc).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
a77392e9ce [#1972] cli: Fix lifetime flag in the lock command
That part of the code was refactored incorrectly.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
2849e465f9 [#1699] meta: Do not return SplitInfoError on Delete
It is not an error: removing virtual object is expected and should be just
skipped. Getting a virtual object with `raw` flag is considered as an
impossible action, all the virtual objects removals will be handled via
their children's removals implicitly.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
a3e7365cbd [#1732] pilorama: Fill parent mark correctly
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
134f2ba02e [#1732] pilorama: Fix backwards log insertion
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
2ef38cfbc4 [#1996] engine: Ignore pilorama.ErrTreeNotFound for write operations
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
a455ec18c3 [#2007] services/object: Allocate memory on-demand in GET_RANGE
For big objects we want to get OutOfRange error before all the memory is
allocated.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
ff5526038d [#2007] services/object: Fix comment
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
d8d3588e1b [#1996] engine: Always select proper shard for a tree
Currently there is a possibility for modifying operations to fail
because of I/O errors and a new tree to be created on another shard.
This commit adds existence check for modifying operations.
Read operations remain as they are, not to slow things.
`TreeDrop` is an exception, because this is a tree removal and trying
multiple shards is not an unwanted behaviour.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-03 15:29:23 +03:00
Pavel Karpy
08efe6eb11 [#2010] github: Run CI on support branches
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-02 14:29:17 +03:00
Evgenii Stratonikov
777fd32d4f [#1818] writecache: Increase error counter on background errors
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-02 14:24:02 +03:00
Evgenii Stratonikov
bffb0f894c [#1818] writecache: Update storage ID during flush
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-02 14:24:02 +03:00
Evgenii Stratonikov
5cf75404dc [#1818] metabase: Add UpdateStorageID operation
By default writecache puts the whole object to update storage ID.
This logic comes from the times when we needed to put objects
in the metabase by the writecache itself. Now this is done by the
blobstor at unmarshaling objects during flush only to update storage ID
is an overkill.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-02 14:24:02 +03:00
Evgenii Stratonikov
b64b14eb54 [#1818] writecache: Reuse FSTree flushing code between flushes
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-02 14:24:02 +03:00
Evgenii Stratonikov
a56927e3d4 [#1818] writecache: Remove unused variable
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-02 14:24:02 +03:00
bb52857b2b [#1338] neofs-cli: Add support to store/restore/delete binary objects
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-11-01 15:30:00 +03:00
Pavel Karpy
1f82c583e3 [#1971] cli: Unify CID and OID flags provision
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-01 15:07:19 +03:00
Pavel Karpy
7daa57d4d2 [#1991] cli: Refine container placement description
Not to confuse a user by mixing a replication vector number with its copy
number.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-01 15:00:00 +03:00
51e3810285 [#1689] Add new command morph list-containers in neofs-adm
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-11-01 09:43:02 +03:00
Evgenii Stratonikov
98a152256b [#1992] writecache: Allow to open in NOSYNC mode
Applicable only to FSTree as we cannot handle corrupted databases
properly yet.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-01 09:42:26 +03:00
Evgenii Stratonikov
b6930f2219 [#1992] neofs-node: Allow to open fstree in NOSYNC mode
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-01 09:42:26 +03:00
Evgenii Stratonikov
148da5fdbb [#1994] docs: Update storage node configuration
Reflect the reality after a not so recent refactoring.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-01 09:42:26 +03:00
Evgenii Stratonikov
f564430b90 [#1992] fstree: Allow working in SYNC mode
Make O_SYNC the default and allow to opt-out explicitly.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-01 09:42:26 +03:00
Pavel Karpy
694d888219 Release v0.34.0 - Marado (마라도, 馬羅島)
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-31 15:07:53 +03:00
Pavel Karpy
d739e06289 [#1980] make: Update linter version in docker command
To be in sync with GitHub actions.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-31 15:07:53 +03:00
Pavel Karpy
cd2faf29a9 [#1980] go.mod: Update neofs-sdk to rc7
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-31 15:07:53 +03:00
Evgenii Stratonikov
34501685b7 [#1969] local_object_storage: Move ErrObjectIsExpired to another package
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-31 11:41:24 +03:00
Evgenii Stratonikov
56de2f1363 [#1969] local_object_storage: Simplify logic error construction
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-31 11:41:24 +03:00
Evgenii Stratonikov
fcdbf5e509 [#1969] local_object_storage: Add a type for logical errors
All logic errors are wrapped in `logicerr.Logical` type and do not
affect shard error counter.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-31 11:41:24 +03:00
Leonard Lyubich
98034005f1 [#1900] node: Fix loosing the "maintenance" status
In previous implementation node lost maintenance status after successful
switching to it. For example, after some period of time node sent
bootstrap requests with the "online" state instead of "maintenance".

Make `startMaintenance` method to set maintenance status in the
`networkState`.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-28 18:30:59 +03:00
Leonard Lyubich
db92e96e40 [#1900] node: Refactor bootstrap methods
In previous implementation bootstrapping state was chosen according to
bool flag which was not convenient.

Create separate method to boostrap with "online" and the current state.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-28 18:30:59 +03:00
Leonard Lyubich
e8c5f03c30 [#1905] shard: Don't log read-only errors of write-cache
There is no need to log `writecache.ErrReadOnly` errors in `Delete`
method of the `Shard`.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-28 18:30:45 +03:00
Leonard Lyubich
b1fa084756 [#1905] shard: Decrease severity level of write-cache failure logs
In previous implementation `Shard.Delete` logged writecache's removal
failures in `error` level. There is a need to decrease severity of these
log records since they aren't critical and don't require individual
review.

Change level of the message to `info`.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-28 18:30:45 +03:00
Leonard Lyubich
98ac525272 [#1978] cli/object: Gather all related object in delete session
Object removal session should reflect all objects related to the
removing one.

Make `OpenSessionViaClient` to gather the split members of the original
object in order to spread the session to them.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-28 13:49:41 +03:00
Stanislav Bogatyrev
7653a1f626 [#409] Fix package building doc typos
Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2022-10-28 12:58:32 +03:00
Stanislav Bogatyrev
43fe156f7c [#409] Fix more lintian warnings
Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2022-10-28 12:58:32 +03:00
c640374d98 [#409] Return OS release suffix to package version
Signed-off-by: Dmitriy Zabolotskiy <d.zabolotskiy@yadro.com>
2022-10-28 12:58:32 +03:00
1e0b1919ed [#409] Fix lintian warnings
Signed-off-by: Dmitriy Zabolotskiy <d.zabolotskiy@yadro.com>
2022-10-28 12:58:32 +03:00
Stanislav Bogatyrev
bbfc1d9263 [#409] Fix neofs-locode-db dependency
And update CHANGELOG.md =)

Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2022-10-28 12:58:32 +03:00
Stanislav Bogatyrev
e1a085ffd5 [#409] Remove docker target
It would not work this way for non-root users. Let's postpone.

Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2022-10-28 12:58:32 +03:00
Stanislav Bogatyrev
ff67e903ca [#409] Add default package changelog
Make it work with simple `dpkg-buildpackage`

Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2022-10-28 12:58:32 +03:00
92e9782c44 [#409] Debian packaging
Debian package building, including:

 - users creation if not exist;
 - dirs creation and ownership/ACLs set;
 - man/doc/examples installation;
 - shell completion generation;
 - locode DB binary download

Using:
	make debpackage

Signed-off-by: Dmitriy Zabolotskiy <d.zabolotskiy@yadro.com>
2022-10-28 12:58:32 +03:00
Evgenii Stratonikov
846ff515e6 [#1812] policer: Do not remove copies if there are maintenance nodes
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-26 19:13:17 +03:00
Evgenii Stratonikov
280e56f4bb [#1893] neofs-node: Do not fail unless all gRPC endpoints are unavailable
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-26 12:48:38 +03:00
Evgenii Stratonikov
1e6588e761 [#1974] shard: Do not panic in degraded mode
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-26 12:41:12 +03:00
Evgenii Stratonikov
7395ab8ef7 [#1944] metabase: Assume static buckets are created on Init
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-26 12:41:12 +03:00
Evgenii Stratonikov
e4bc3d0e9d [#1944] metabase: Recreate static buckets instead of resetting
From the `Bucket.ForEach` doc:
```
The provided function must not modify the bucket; this will result in undefined behavior.
```

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-26 12:41:12 +03:00
Leonard Lyubich
3c6daa2995 [#1883] cli/shards: Exclude degraded-read-write mode from help
The mode is not expected to be used by the user. It is supported only
for developers.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-26 08:24:40 +03:00
Leonard Lyubich
eb206d6e59 [#1883] config/example: Actualize desc of shard modes
Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-26 08:24:40 +03:00
Leonard Lyubich
c4a2a283ae [#1883] cli/shards: Make supporting new modes easier
Add new shard modes as a map entry to automatically parse them in
`set-mode` command. The change also automatically adds new modes to help
message.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-26 08:24:40 +03:00
Evgenii Stratonikov
3b939d190c [#1957] engine: Move shard to read-only if cannot move to degraded
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-26 08:20:53 +03:00
Leonard Lyubich
7f54462e60 [#1920] cli: Print used RPC timeout in verbose exec
Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-26 08:20:36 +03:00
Pavel Karpy
9a0824766c [#1713] node: Add Neo RPC switch interval to the config
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-25 17:38:48 +03:00
Pavel Karpy
0631b38da1 [#1713] ir: Add Neo RPC switch interval to the config
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-25 17:38:48 +03:00
Pavel Karpy
d240e2dc87 [#1713] morph: Allow switch interval configuration
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-25 17:38:48 +03:00
Pavel Karpy
d09ec2e3fe [#1615] node/config: Remove unused default constants
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-25 17:38:48 +03:00
Pavel Karpy
17f7d0a2ee [#1615] morph: Switch to a more prioritized RPC node
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-25 17:38:48 +03:00
Pavel Karpy
7c0aa69d11 [#1615] morph: Fix the switch comment message
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-25 17:38:48 +03:00
Evgenii Stratonikov
713fdab177 [#1907] shard: Return from Close after GC has stopped
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-25 11:54:45 +03:00
Evgenii Stratonikov
4fe1aad30f [#1966] neofs-adm: Fix signature creation for multiactor scenario
`SignTx` from neo-go appended verification script only if `Scripts`
array was not completely filled.

```
can't persist transactions: transaction b9cd02fb81721c7316c908687301eeaf8ef384e381476632c24a77bdf9ffacd5 failed to verify: witness #1: unknown verification contract
```

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-25 11:22:00 +03:00
Evgenii Stratonikov
2dcf8d51be [#1966] neofs-adm: Set VerifyBlocks: true and fix NetworkFee calculation
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-25 11:22:00 +03:00
Pavel Karpy
41f93dcc1d [#1953] blobstor: Log any object removal operation
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-24 21:52:18 +03:00
Pavel Karpy
942cc38146 [#1953] engine: Add refactored storage helpers
After the refactor there are new storage characteristics: a type and
a general storage id (that could be stringified).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-24 21:52:18 +03:00
Pavel Karpy
0371d15b2f [#1953] shard: Fix debug log messages
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-24 21:52:18 +03:00
Pavel Karpy
f8180447a1 [#1938] meta: Make version error messages more descriptive
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-24 21:46:18 +03:00
Evgenii Stratonikov
f396ffcc0f [#1959] CHANGELOG.md: Update neofs-adm remarks
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
6c9423cfb0 [#1959] neofs-adm: Add tests for more commands
Also, fix `restore-containers`, but it is hard to test it now.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
7c3ac7150d [#1959] neofs-adm: Remove newTempDir function
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
2404a267b9 [#1959] neofs-adm: Move testdata generation to a separate function
Later we could make it a separate tests an run neofs-adm tests on CI.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
6c21e2cc28 [#1959] neofs-adm: Fix set-config for >4 nodes
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
c87c3d3721 [#1959] neofs-adm: Refactor tests
Make it easier to test both common deployment scenarios.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
02bea52f40 [#1959] neofs-adm: Allow to test all commands with a local dump
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
8648b102db [#1959] neofs-adm: Dump all blocks in file
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
23622ed283 [#1959] neofs-adm: Check transaction validity in tests
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
942155be6e [#1959] neofs-adm: Fix double multisig
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
2484c1d35e [#1959] neofs-adm: Verify blocks and transactions in tests
Also update to the actual contracts version.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
926830bb9c [#1961] network/cache: Do not return nil client
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 17:56:34 +03:00
Leonard Lyubich
a4573d5026 [#1933] cli: Document sessions
Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-24 17:45:22 +03:00
Leonard Lyubich
7b418c36b4 [#1930] services/session: Log calling Create RPC
There is a need to check if session is opened during system
testing/debug.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-24 17:45:22 +03:00
Leonard Lyubich
8bba490c30 [#1933] cli: Support binary sessions
There is a need to support NeoFS-binary sessions along with JSON ones in
NeoFS CLI.

Provide generic `common.ReadBinaryOrJSON` functions which tries to
decode NeoFS-binary structure and falls back to JSON format. Use this
function in all places with token reading.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-24 17:45:22 +03:00
Evgenii Stratonikov
04b67f3ba5 [#1949] neofs-adm: Fix epoch tick for >4 node committee
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 13:36:08 +03:00
Evgenii Stratonikov
8796807040 [#1936] network/cache: Handle dial errors
After an SDK update `Dial` can return an error.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 13:28:53 +03:00
Evgenii Stratonikov
a189eca5d5 [#1908] docs: Describe SIGHUP behaviour
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 13:28:26 +03:00
Evgenii Stratonikov
1beafea0b5 [#1869] shard: Add logs for SetMode operations on reload
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 13:28:26 +03:00
Evgenii Stratonikov
87be4f1629 [#1869] shard: Restore shard mode on failed reloads
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 13:28:26 +03:00
Evgenii Stratonikov
c785e11b20 [#1869] shard: Allow to reload metabase on SIGHUP
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 13:28:26 +03:00
Evgenii Stratonikov
f769fc83fc [#1869] shard: Embed gcCfg as raw struct
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 13:28:26 +03:00
Leonard Lyubich
60e9de8d63 [#1916] control: Check maintenance allowance on Control server
In previous implementation turning to maintenance mode using NeoFS CLI
required NeoFS API endpoint. This was not convenient from the user
perspective. It's worth to move networks settings' check to the server
side.

Add `force_maintenance` field to `SetNetmapStatusRequest.Body` message
of Control API. Add `force` flag to `neofs-cli control set-status`
command which sets corresponding field in the requests body if status is
`maintenance`. Force flag is ignored for any other status.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-24 09:20:24 +04:00
Leonard Lyubich
c50603494b [#1916] cli/control: Pre-check maintenance mode in set-status
Node's maintenance state can be set only if the network allows it.

Make `neofs-cli control set-status` to read current network settings and
check is maintenance state is allowed. Fail the execution if the mode is
not allowed.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-24 09:20:24 +04:00
Pavel Karpy
e1be0180f6 [#1329] tree: Sync trees when a node first time appears in a container
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-21 18:47:56 +03:00
Leonard Lyubich
217b030d20 [#1946] cli/internal: Fix test by wallet non-closing
Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-21 15:21:59 +03:00
Leonard Lyubich
810087d8b9 [#1946] Bump neo-go module to v0.99.4
Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-21 15:21:59 +03:00
Evgenii Stratonikov
de055f27c7 [#1942] neofs-adm: Fix multisig checks in tests
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-21 10:16:51 +03:00
Pavel Karpy
7d456cb4d0 [#1902] Update CHANGELOG
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
49c38d642d [#1902] engine: Search for the tree IDs in every shard
Iterate over every shard and search for the container's trees. Final result
is a concatenation of shards' results. It is considered that one fixed tree
is placed on one fixed shard but the different trees of a fixed container
could be placed on different shards.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
aa37078570 [#1902] node: Sync all the trees on bootstrap
Do not limit synchronization by "system" and "version" tree IDs.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
1766ca2039 [#1902] tree: Allow synchronize all the container trees
Add `SynchronizeAllTrees` method of the Tree service. It allows fetching
tree IDs and sync all of them. Share common logic b/w the new method and
the `SynchronizeTree`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
1805c6606d [#1332] cli: Add tree list
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
6d4beea187 [#1902] tree: Extend grpc service with ListTrees method
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
24e9e3f3bf [#1902] engine, shard: Implement TreeList method
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
2e199c7ab1 [#1902] shard: Fix pilorama disabled err message
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
19850ef157 [#1902] pilorama: Add TreeList method
To both `bolt` and `memory` forests; extend `Forest` interface.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
3aa9938b8f [#1902] Update protoc to v3.21.7
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Evgenii Stratonikov
9ec01bb9c1 [#1931] control: Allow to clear errors in SetShardMode RPC
It hasn't been working since the initial implementation 7fb15fa1d0.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-20 15:51:31 +03:00
Evgenii Stratonikov
a95fad833e [#1909] docs: Add shard modes description
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-20 08:53:58 +03:00
Evgenii Stratonikov
0d14ef69f0 [#1922] neofs-node: Allow to go online after maintenance
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-19 10:14:56 +03:00
Evgenii Stratonikov
1980ed968a [#1926] neofs-node: Use writecache.max_object_size from the config
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-18 16:44:28 +03:00
Evgenii Stratonikov
13e818e2ca [#1910] .github: Update golangci-lint version
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-18 15:08:26 +03:00
Evgenii Stratonikov
0d65888005 [#1910] .golangci.yml: Add predeclared linker
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-18 15:08:26 +03:00
Evgenii Stratonikov
1cb892c579 [#1910] .golangci.yml: Add misspell linker
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-18 15:08:26 +03:00
Evgenii Stratonikov
9e46799cd4 [#1910] .golangci.yml: Add more useful linkers
Steal them from the neo-go repo.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-18 15:08:26 +03:00
Evgenii Stratonikov
d772e35aba [#1910] .golangci.yml: Add godot linker
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-18 15:08:26 +03:00
Evgenii Stratonikov
347912ea0b [#1910] .golangci.yml: Move unused to a list of default linters
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-18 15:08:26 +03:00
Evgenii Stratonikov
0a1530afa0 [#1917] neofs-cli: Support timeout in control commands
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-18 10:58:10 +03:00
Pavel Karpy
4c63be6629 [#1329] Update CHANGELOG
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-17 20:53:34 +03:00
Pavel Karpy
bc67f77d86 [#1329] tree: Fix sync applying
Add the node position in a container and the container size to the CID
descriptor that is passed to the `TreeApply`. Previously, `checkValid` does
not allow any log operarations.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-17 20:53:34 +03:00
Pavel Karpy
c16a31ef3e [#1329] node: Set undefined netmap status by default
It allows general usage of `controlNetmapStatus` getter: does not matter of
handler (`NewEpoch` handler of initial setter).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-17 20:53:34 +03:00
Pavel Karpy
f4a3fa2977 [#1329] tree: Sync tree on startup
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-17 20:53:34 +03:00
Pavel Karpy
f5f560d903 [#1329] tree: Do not sync trees that are from external containers
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-17 20:53:34 +03:00
Pavel Karpy
ea9a5690cb [#1918] Fix v0.33.0 CHANGELOG record
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-17 20:42:32 +03:00
Evgenii Stratonikov
e2970bf892 Release v0.33.0
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-17 19:09:42 +03:00
Evgenii Stratonikov
b40dd10b7a [#1913] go.mod: Update neofs-api-go to v2.14.0
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-17 19:09:42 +03:00
Evgenii Stratonikov
f524c812c1 [#1913] go.mod: Update neo-contract to v0.16.0
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-17 19:09:42 +03:00
Leonard Lyubich
d96139b201 [#1900] node: Do not override maintenance state on bootstrap
In previous implementation node always sent bootstrap requests with
"online" network state. This provoked switching to online of the nodes
under maintenance.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-17 18:59:22 +03:00
Pavel Karpy
f76516a883 [#1860] meta: Fix 0,1 -> 2+ version migration
In the 2nd version, there was a database format change: buckets have changed
their keys, so it becomes impossible to check the version in the 1 -> 2+
migrations because of different buckets that store info about the version.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-17 13:42:05 +03:00
Leonard Lyubich
6eb82c4cbf [#1890] docs: Describe maintenance mode's behavior
Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-17 13:40:56 +03:00
Evgenii Stratonikov
8145e79d7f [#1903] neofs-adm: Expand update parameters in MakeCall
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-17 13:37:48 +03:00
anastasia prasolova
bcf3df354b Add CODEOWNERS file
Signed-off-by: anastasia prasolova <anastasia@nspcc.ru>
2022-10-16 21:09:07 +03:00
Leonard Lyubich
1406d096a2 [#1680] service/object: Fail all operations in maintenance mode
Storage node should not provide NeoFS Object API service when it is
under maintenance.

Declare `Common` service that unifies behavior of all object operations.
The implementation pre-checks if node is under maintenance and returns
`apistatus.NodeUnderMaintenance` if so. Use `Common` service as a first
logical processor in object service pipeline.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-14 13:54:32 +04:00
Leonard Lyubich
05420173cc [#1894] services/object: Ignore unrelated sessions in client
In some scenarios original session can be unrelated to the objects which
are read internally by the node. For example, node requests child
objects when removing the parent one.

Tune internal NeoFS API client used by node's Object API server to
ignore unrelated sessions in `GetObject` / `HeadObject` / `PayloadRange`
ops.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-14 12:37:31 +03:00
Evgenii Stratonikov
2050a623ac [#1896] neofs-node: Set MAINTENANCE status on update
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-14 11:46:19 +03:00
Pavel Karpy
ade19077ee [#1332] Update Changelog
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-13 20:01:48 +03:00
Pavel Karpy
beb1998ed1 [#1332] cli: Print missing nodes message verbosely
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-13 20:01:48 +03:00
Pavel Karpy
6e4f7180fe [#1332] cli: Implement tree add-by-path command
It is `TreeService.AddNodeByPath` method implementation.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-13 20:01:48 +03:00
Pavel Karpy
3d8349d7f8 [#1332] cli: Implement tree get-by-path command
It is `TreeService.GetNodeByPath` method implementation.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-13 20:01:48 +03:00
Pavel Karpy
2c02e66939 [#1332] cli: Implement tree add command
It is `TreeService.Add` method implementation.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-13 20:01:48 +03:00
Pavel Karpy
13c4a9f4b8 [#1332] tree: Make SignMessage public
It will allow reusing signing routine in other components
(e.g. `neofs-cli`).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-13 20:01:48 +03:00
Evgenii Stratonikov
80d3c7f9d6 [#1892] neofs-adm: Support MaintenanceModeAllowed network setting
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-13 17:45:47 +03:00
Evgenii Stratonikov
f2793060c5 [#1892] neofs-cli: Display maintenance status in netmap nodeinfo
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-13 17:45:47 +03:00
Pavel Karpy
31c623636d [#1863] node: Fix shard id in the object counter metrics
If shard ID is stored in metabase (it is not the first time boot), read it,
set it, use it (not a generated one) in the metrics writer.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-13 13:06:41 +03:00
8714fc42b5 [#1765] Use hex format to print storage node ID
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-10-13 12:55:21 +03:00
Pavel Karpy
5a2daadd37 [#1770] Update Changelog
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-12 18:11:05 +03:00
Pavel Karpy
b6806ea6b9 [#1770] node: Support logger config rereading
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-12 18:11:05 +03:00
Pavel Karpy
8c75cb1dad [#1770] node: Validate logger config section
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-12 18:11:05 +03:00
Pavel Karpy
d7c7022bbd [#1770] logger: Support runtime level reloading
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-12 18:11:05 +03:00
Pavel Karpy
f037022a7a [#1770] logger: Refactor Logger component
Make it store its internal `zap.Logger`'s level. Also, make all the
components to accept internal `logger.Logger` instead of `zap.Logger`; it
will simplify future refactor.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-12 18:11:05 +03:00
Evgenii Stratonikov
4baf00aa21 [#1884] services/object: Fallback to GET in GET_RANGE
Current spec allows denying GET_RANGE requests from other storage nodes.
However, GET should always be allowed and it is enough to perform
GET_RANGE locally

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-12 17:05:51 +03:00
Leonard Lyubich
dde4d4df2a [#1878] services/object: Fix child check in GET
In previous implementation `ObjectService.Get` RPC handler failed with
`parent address in child object differs` while assembling the "big"
object. This was caused by the child check which required parent
reference to be set in all child objects. The check was impracticable
because not all elements of the split-chain have a link to the parent.

Make `execCtx.isChild` to return `true` if parameterized object has no
parent header in its own header.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-12 16:56:37 +03:00
Leonard Lyubich
9a006ac14f [#1865] cli/object: Do not open remote sessions in reading ops
It does not make sense to open remote sessions with the storage node in
`get`, `head`, `search`, `range` and `hash` sub-commands of `neofs-cli
object` command.

Do not use NeoFS API `SessionService` in mentioned commands. Decode
object session from JSON file specified `--session` flag. Perform some
sanity checks instantly on CLI side.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-12 13:22:19 +03:00
Evgenii Stratonikov
49eab6318c [#1867] control: Fix degraded-read-only mode parsing
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-12 11:20:48 +03:00
Evgenii Stratonikov
c0199dee93 [#1867] services/control: Interpret empty list of IDs as all shards
In neofs-cli the flag is still required, but `all` can be used to
process all shards.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-12 11:20:48 +03:00
Evgenii Stratonikov
b632260995 [#1867] neofs-cli: Support multiple shard IDs in control subcommands
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-12 11:20:48 +03:00
Evgenii Stratonikov
19c0a74e94 [#1867] services/control: Allow to provide multiple shard IDs to some commands
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-12 11:20:48 +03:00
74d2f2c8d3 [#1854] cli: Unify help messages
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-10-12 09:53:11 +03:00
Pavel Karpy
9e2edfedc6 [#1833] node: Update local node info on nil updates
If the contract returns a netmap that does not contain the node, update
local `NodeInfo`. It fixes `neofs-cli netmap nodeinfo` command that printed
"state: online" previously.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-12 09:52:51 +03:00
Evgenii Stratonikov
c5fdb7bedf [#1876] .github: Add an action monitoring config updates
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-12 09:33:28 +03:00
Leonard Lyubich
050ad2762c [#1680] replicator: Consider NODE_UNDER_MAINTENANCE as OK
Node response with `NODE_UNDER_MAINTENANCE` status signals that the node
was switched to maintenance mode. There is a delay between the actual
switch and the reflection in the network map of up to one epoch. To
speed up the reaction to the maintenance, it is required to recognize
such node responses in the Policer.

Make `Policer.processNodes` to exclude elements with shortage decreasing
on `NODE_UNDER_MAINTENANCE` status response.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-11 12:54:27 +03:00
Leonard Lyubich
e99e25b52f [#1680] replicator: Consider nodes under maintenance as OK
Nodes under maintenance SHOULD not respond to object requests. Based on
this, storage node's Policer SHOULD consider such nodes as problem ones.
However, to prevent spam with the new replicas, on the contrary, Policer
should consider them normal.

Make `Policer.processNodes` to exclude elements if `IsMaintenance()`
with shortage decreasing.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-11 12:54:27 +03:00
Leonard Lyubich
df5d7bf729 [#1680] replicator: Work with netmap.NodeInfo in TaskResult
Make `replicator.TaskResult` to accept `netmap.NodeInfo` type instead of
uint64 in order to clarify the meaning and prevent passing the random
numbers.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-11 12:54:27 +03:00
Leonard Lyubich
e6f8904040 [#1680] policer: Refactor tracking the processed nodes
Add clear methods with docs. Use the methods instead of direct map
and bool instructions.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-11 12:54:27 +03:00
feaa9eace7 [#1768] Add healthcheck method to the Tree service
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-10-11 12:53:45 +03:00
Leonard Lyubich
807c0a1321 [#1859] services/object: Do not session check relation in PUT
It doesn't make sense to check object relation in session check of
`ObjectService.Put` RPC which has been spawned by `ObjectService.Delete`
with session. Session issuer can't predict identifier of the tombstone
object to be created.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-10 20:09:47 +03:00
Evgenii Stratonikov
2d43892fc9 [#1840] neofs-node: Use blobstor paths to identify shard
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-10 11:14:55 +03:00
Evgenii Stratonikov
4b005d3178 [#1840] blobstor: Return info about all components
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-10 11:14:55 +03:00
Evgenii Stratonikov
6d7ffefec5 [#1840] blobstor/test: Add tests for auxiliary functions
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-10 11:14:55 +03:00
Evgenii Stratonikov
328691c94f [#1840] blobstor: Return Path from components
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-10 11:14:55 +03:00
Evgenii Stratonikov
9b241e4a17 [#1840] neofs-node: Allow to use mode: disabled in config
Currently, when removing shard special care must be taken with respect
to shard numbering. `mode: disabled` allows to leave shard configuration
in place while also ignoring it during initialization. This makes
disk replacement much more convenient.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-10 11:14:55 +03:00
Pavel Karpy
1360273fec [#1856] node: Fix config rereading
Config was reread from the file only once in two SIGHUPs.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-10 10:35:52 +03:00
Evgenii Stratonikov
b2aa9947c2 [#1829] engine: Delete split objects properly
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-07 16:35:46 +03:00
5e493b7f1c [#1704] Add command container nodes to output list of nodes for container, grouped by replica (#1704)
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-10-07 14:58:01 +03:00
Evgenii Stratonikov
6557f5d249 [#1839] engine: Handle Inhume errors properly
If shard is in read-only or degraded mode, there is no need to increase
error counter.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-07 14:25:52 +03:00
Evgenii Stratonikov
7356ee91ff [#1837] services/object: Optimize uniqueIDWriter
Avoid encoding object ID to string.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-07 13:29:15 +03:00
Evgenii Stratonikov
2ac42b70ce [#1826] services/object: Parallelize object search
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-07 13:29:15 +03:00
Evgenii Stratonikov
90bfe0bad9 [#1826] neofs-cli: Add --timeout flag
Allow to specify it everywhere `--rpc-endpoint` flag is present.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-07 13:29:15 +03:00
Evgenii Stratonikov
0cb8e7f6f1 [#1837] services/object: Fix log messages
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-07 13:29:15 +03:00
Evgenii Stratonikov
ca8dc872b2 [#1846] neofs-node: Make morph.cache_ttl equal to block time by default
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-07 10:34:55 +03:00
Leonard Lyubich
e54b52ec03 [#1420] object/acl: Fix correlation of object session to request
In previous implementation of `neofs-node` app object session was not
checked for substitution of the object related to it. Also, for access
checks, the session object was substituted instead of the one from the
request. This, on the one hand, made it possible to inherit the session
from the parent object for authorization for certain actions. On the
other hand, it covered the mentioned object substitution, which is a
critical vulnerability.

Next changes are applied to processing of all Object service requests:
 - check if object session relates to the requested object
 - use requested object in access checks.

Disclosed problem of object context inheritance will be solved within

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-07 10:34:38 +03:00
Evgenii Stratonikov
5834f9807e [#1847] services/policer: Provide container ID in logs
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-07 09:58:16 +03:00
Evgenii Stratonikov
c20c2e3b39 [#1845] morph/client: Use 0 as OfflineState in update peer
As the documentation prescribes.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-06 12:26:25 +03:00
Pavel Karpy
01ddb3f8e6 [#1770] node: Refactor application configuration
Split all the fields in `cfg` structure on:
1. `applicationConfiguration`;
2. `internals`; // shared entities for an application work, such as
`context.Context`
3. `shared`; // holder for the shared entities b/w;
4. `cfgXXX`; // configuration for internal services.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-05 14:25:33 +04:00
Leonard Lyubich
713aea06fa [#1681] node: Block only Object service ops under maintenance
In previous implementation node blocked any operation of local object
storage in maintenance mode. There is a need to perform some storage
operations like data evacuation or restoration.

Do not call block storage engine in maintenance mode. Make all Object
service operations to return `apistatus.NodeUnderMaintenance` error from
each local op.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
082602b668 [#1680] Update changelog
Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
a7668618c9 [#1680] cli/netmap: Support MAINTENANCE mode
Make `netmap snapshot` command to print `MAINTENANCE` state of the nodes
with `IsMaintenance()` flag set.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
eb1fba5182 [#1680] morph/netmap: Adopt to recent contract changes
After recent Netmap contract changes all read methods which return
network map (either candidates or snapshots) encode node descriptors
into same structure.

Decode `netmap.Node` contract-side structure from the call results.
Replace node state with the value from the `netmap.Node.State` field.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
d6c01199c8 [#1680] node: Return MAINTENANCE status in corresponding mode
Make storage node to return `NODE_UNDER_MAINTENANCE` status
error on each local object operation if the node is in `MAINTENANCE`
mode.

Pass `apistatus.NodeUnderMaintenance` to `StorageEngine.BlockExecution`
during `ControlService.SetNetmapStatus` RPC processing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
9fcc80ea38 [#1680] node/control: Send request to switch to MAINTENANCE state
After recent changes `MAINTENANCE` state is reflected in the Sidechain.
Storage node should switch its state to "maintenance" during serving the
`ControlService.SetNetmapStatus` RPC with correspoding status in the
request.

Call `UpdatePeerState` operation of Netmap contract's client in
`control.NodeState` provider on Storage node app side. The op is
executed if `BlockExecution` on local object storage is succeeded.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
6a2ec21641 [#1681] ir/netmap: Check MAINTENANCE state in UpdateState
Storage node can be requested to be switched into `MAINTENANCE` state.
Inner Ring should accept such requests only if network configuration
allows it.

Make `Processor` of Netmap contract's notifications to depend on
`state.NetworkSettings`. Make `Processor.processUpdatePeer` to call
`MaintenanceModeAllowed` if notification event relates to `MAINTENANCE`
mode`. Share singe `state.NetworkSettings` provider in Inner Ring
application.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
bdb8243a5a [#1681] morph/netmap: Support MAINTENANCE state notification
After recent changes Netmap contract can send `UpdateState` notification
event with `MAINTENANCE` node's state. There is a need to provide
functionality to work with the status.

Provide `UpdatePeer.Maintenance` method. Support new state in
`ParseUpdatePeer` and `ParseUpdatePeerNotary` functions.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
0903b2af93 [#1680] morph/netmap: Allow to set MAINTENANCE state
After recent changes in NeoFS protocol storage nodes can be in
`MAINTENANCE` state. There is a need to support this state in
`UpdateState` operation.

Add `UpdatePeerPrm.SetMaintenance` method which makes node to be
switched into `MAINTENANCE` mode after the `UpdatePeerState` operation.

New functionality is going to be used in Storage node application for
Control API serving.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
6b99f2a0df [#1680] ir/netmap: Correlate node's state with network settings
Inner Ring should allow registering of storage nodes with `MAINTENANCE`
state in the NeoFS network only if its configuration allows this status.

Make `networkSettings.MaintenanceModeAllowed` to call
`MaintenanceModeAllowed` method of underlying Netmap contract's client
in order to assert state allowance.

From now nodes will be accepted to the network with `MAINTENANCE` state
only with the appropriate network configuration.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
876e132d22 [#1680] cli/netmap: Print MaintenanceModeAllowed config
After recent changes network configuration provided by NeoFS storage
nodes contains `MaintenanceModeAllowed` flag. There is
a need to support this value in NeoFS CLI application.

Print `MaintenanceModeAllowed` flag in `netmap netinfo` command.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
7a3a827d35 [#1680] node/netmap: Support MaintenanceModeAllowed config in RPC
After recent changes network configuration stored in the Netmap contract
of the NeoFS Sidechain contains `MaintenanceModeAllowed` flag. There is
a need to support this value in Storage node application.

Make `NetmapService.NetworkInfo` RPC server of the storage node to set
`MaintenanceModeAllowed` flag according to corresponding value in the
Sidechain.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
0826310b2a [#1681] morph/netmap: Support MaintenanceModeAllowed config
After recent changes in the NeoFS API protocol network configuration
contains `MaintenanceModeAllowed` boolean flag. There is a need to
support the config value in all NeoFS applications.

Provide `Client.MaintenanceModeAllowed` method which read the config
from the Sidechain. Extend `NetworkConfiguration` structure with
`MaintenanceModeAllowed` field.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
d2d4191868 [#1680] morph/netmap: Support HomomorphicHashingDisabled config
`NetworkConfiguration` represents NeoFS network configuration stored in
the Sidechain. In previous implementation the configuration missed flag
of disabled homomorphic hashing.

Add `NetworkConfiguration.HomomorphicHashingDisabled` boolean field.
Decode the field in `Client.ReadNetworkConfiguration` method. Print this
value in `netmap netinfo` command of NeoFS CLI.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
e0194dbde5 [#1681] morph/netmap: Refactor reading the boolean configurations
`readBoolConfig` method is going to be reused for reading other
configuration values. All boolean settings are `false` by default, so it
makes sense to return default value on missing key directly from
`readBoolConfig`.

Handle `ErrConfigNotFound` case in `readBoolConfig` method. Change
`HomomorphicHashDisabled` method to call `readBoolConfig` directly.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
f64ae55806 [#1681] ir/netmap: Require MAINTENANCE mode to be allowed by network
There is a need to prevent limitless abuse of MAINTENANCE status of the
storage nodes. To do this, configuration of the NeoFS network is going
to be extended with the flag which allows the state. Until this is done,
it makes sense to prepare a site for this in the code.

Define `state.NetworkSettings` interface as an abstraction of global
network configuration within the `state` package. Make
`NetMapCandidateValidator` to depend on `NetworkSettings` and provide
corresponding field setter. Change `VerifyAndUpdate` method's behavior
to return an error for candidates with MAINTENANCE state if this state
is disallowed by the network configuration. Provide `NetworkSettings`
from the wrapper over Netmap contract's client on Inner Ring application
side. The provider is implemented to statically disallow MAINTENANCE
mode in order to save previous behavior.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
42fb40e841 [#1680] morph/netmap: Pre-refactor processing of node states
New network status of storage nodes is going to be introduced. To
simplify the addition, it would be useful to prepare the code for this.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
4c94faac67 [#1797] node/control: Verify states in SetNetmapStatus RPC server
In previous implementation storage node interpreted all status values
sent in `SetNetmapStatus` RPC as `OFFLINE` except `ONLINE` and
`MAINTENANCE`. This could lead to incorrect processing of new values,
and also didn't allow detection of problems with sending garbage values.

Make implementation of `NodeState` interface used by Control API server
to deny requests with statuses other than protocol-declared enum.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
8858840751 [#1796] ir/netmap: Allow to call AddPeer with ONLINE state only
In previous implementation Inner Ring allowed storage nodes with any
state to register in the network. According to the current design, only
nodes with ONLINE state are allowed to enter the network map.

Create new `state` sub-package of `nodevalidation` package of Inner Ring
application. Define `state.NetMapCandidateValidator` type and provide
`NodeValidator` interface required by the Inner Ring's processor of
`Netmap` contract's notification events. Embed new validator into the
one used by the Inner Ring application.

From now all `AddPeer` notifications with node state other than `ONLINE`
will be denied.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
8bf82d738b [#1704] cli: Add force option to the command container create
Validate policy before container creation

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-10-05 09:14:13 +03:00
Pavel Karpy
4eb0ed11f8 [#1809] node: Do not boot up if metabase is outdated
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 12:32:10 +03:00
Evgenii Stratonikov
8b3b16fe62 [#1825] writecache: Flush cache when moving to the DEGRADED mode
Degraded mode allows us to operate without an SSD,
thus writecache should be unavailable in this mode.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-04 12:13:09 +03:00
Evgenii Stratonikov
236414df49 [#1817] network: Allow to use network addresses from the iterator
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-04 12:12:18 +03:00
Evgenii Stratonikov
76893bdc50 [#1817] go.mod: Update neofs-sdk-go
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-04 12:12:18 +03:00
Evgenii Stratonikov
2e3ef817f4 [#1819] engine: Increase error counter for PUT errors
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-04 10:11:52 +03:00
Evgenii Stratonikov
b89e71fa78 [#1819] common: Add ErrNoSpace
Add a common error for this case because it is not an error
which should increase error counter. Single error simplifies checks on
the call-site.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-04 10:11:52 +03:00
Evgenii Stratonikov
af56574849 [#1819] engine: Fix error counter in Inhume
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-04 10:11:52 +03:00
Pavel Karpy
1e35c12cc1 [#1770] node: Validate config before apply it
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
8ebe95747e [#1770] node: Do not lock on shard's Close call
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
ab3ef7110e [#1770] node: Reread config files on SIGHUP
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
6fc3268ebf [#1770] node: Support configuration reread on SIGHUP
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
887afeaddb [#1770] engine: Do not lock on shard init
Init can take a lot of time. Because the mutex is taken, all new operations
are blocked.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
fbd5bc8c38 [#1770] engine: Support configuration reload
Currently, it only supports changing the compound of the shards.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
91b56ad3e8 [#1770] node: Read storage config in a separate struct
It will allow rereading config values and will simplify distinguishing them
from the custom values in the `cfg` structure.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
5c69e19016 [#1770] fstree: Depth parameter int -> uint64
Negative values have no sense. On the other hand it differs from the
blobovnicza's configuration and prevents unification.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
4aa4694152 [#1770] node: Do not init local storage in reading config routine
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
2d7166f8d0 [#1770] shard: Move NewEpoch event routing on SE level
It will allow dynamic shard management. Closing a shard does not allow
removing event handlers.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
9374823950 [#1770] node: Do not create meta dir on config read
It is created in `Open` anyway.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
76cfcc242c [#1820] neofs-adm: Add wallet-address flag in refill command
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-10-03 10:11:38 +03:00
Evgenii Stratonikov
1edc048870 [#1697] services/object: Return proper error if session token is missing
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-30 16:13:01 +03:00
5284ac53f9 [#1821] neofs-cli: Fix description of cli commands
1. nodeinfo - updated description of node.
2. eacl - fixed misprint.

Signed-off-by: Vladimir Domnich <v.domnich@yadro.com>
2022-09-28 16:09:24 +03:00
Evgenii Stratonikov
0a411908ee [#1806] writecache: Allow to ignore read errors during flush
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
f2045c10d7 [#1806] shard: Check each component mode when setting mode
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
1c62f1b2c4 [#1806] neofs-cli: Add control flush-cache command
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
51b8f26a31 [#1806] services/control: Allow to flush write-cache
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
3d882e9f47 [#1806] engine: Allow to flush write-cache
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
0b4c867ef1 [#1806] writecache: Allow to start flush manually
Allow user to initiate flushing objects from a writecache.
We need this in 2 cases:
1. During writecache storage schema update, it should be flushed with
   the old version of node and started clean with a new one.
2. During SSD replacement, to avoid data loss.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
55148404ee [#1806] writecache: Use interfaces for blobstor and metabase
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
36ab1a2472 [#1731] neofs-cli: Make --endpoint flag optional
Allow to provide it using config file.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
Leonard Lyubich
93742d37b7 [#1793] node/netmap: Change interface of the latest network map reader
Replace `ProcessCurrentNetMap` method of `NodeState` interface with
`ReadCurrentNetMap` one with two changes:
 * Replace network map type from NeoFS SDK package with the
   protocol-generated message. This replaces all the business logic to
   the application layer.
 * Support error return. This allows to cover problem node states.

Return an error from `NodeState.ReadCurrentNetMap` method implemeted
through `atomic.Value` if `Store` method has not been called yet.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-27 09:03:17 +03:00
Leonard Lyubich
485a5418d2 [#1793] node: Serve NetmapService.NetmapSnapshot RPC
There is no more need to serve the same request on Control API.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-27 09:03:17 +03:00
Evgenii Stratonikov
59de20fbba [#1764] neofs-node: Allow to check configuration
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-27 08:57:20 +03:00
Evgenii Stratonikov
0fb5c51ac9 [#1764] neofs-node: Validate config before usage
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-27 08:57:20 +03:00
Evgenii Stratonikov
6f45cc81fc [#1764] neofs-node: Use constants for storage types
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-27 08:57:20 +03:00
Evgenii Stratonikov
9113793688 [#1764] neofs-node: Allow to return error from IterateShards
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-27 08:57:20 +03:00
Evgenii Stratonikov
898689ec14 [#1731] services/replicator: Unify Task interface with other parameters
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-24 13:47:48 +03:00
Evgenii Stratonikov
4e043a801c [#1731] services/control: Replicate object over network in EvacuateShard RPC
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-24 13:47:48 +03:00
Evgenii Stratonikov
a49137349b [#1731] engine: Allow to use user handler for evacuated objects
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-24 13:47:48 +03:00
Pavel Karpy
93ae3f0b19 [#1808] .github: Add changelog workflow
It checks whether the CHANGELOG.md was updated in a just-created PR.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-23 10:20:25 +04:00
Evgenii Stratonikov
7cc68cf4d4 [#1810] blobovniczatree: Add a test for blobovnicva usage
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-23 10:19:26 +04:00
c7f85994e5 [nspcc-dev#1692] cli: Remove --generate-key option in neofs-cli container delete
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-09-20 07:19:46 +04:00
bb02913c39 [nspcc-dev#1128] cli: Remove WIF and NEP2 support in --wallet argument
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-09-19 14:21:37 +03:00
Evgenii Stratonikov
3df98ce7ba [#1731] engine: Return the amount of actually moved objects in Evacuate
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-19 11:33:52 +03:00
Evgenii Stratonikov
8fc88487db [#1731] neofs-cli: Add control shards evacuate command
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-19 11:33:52 +03:00
Evgenii Stratonikov
091d7d30f6 [#1731] services/control: Allow to evacuate data from shard
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-19 11:33:52 +03:00
Evgenii Stratonikov
a51b76056e [#1731] engine: Add Evacuate command
Make it possible to move all data from 1 shard to other shards.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-19 11:33:52 +03:00
Evgenii Stratonikov
7377979e12 [#1731] engine: Move single shard PUT to a separate function
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-19 11:33:52 +03:00
Evgenii Stratonikov
4208f7c0cf [#1741] neofs-adm: Use ValidUntilBlock to wait for tx persist
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-15 14:27:16 +03:00
Evgenii Stratonikov
5321f8ef9c [#1786] engine: Unify parameter setters
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-15 10:28:48 +03:00
Evgenii Stratonikov
b064fb24d8 [#1616] engine: Do not use batches in delete
Use a simple loop at the callsite. This way we remove as much as we can.
Also, `Delete` metrics is more meaningful now.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-15 10:28:48 +03:00
Evgenii Stratonikov
cda8f9df2e [#1786] services/control: Remove WithDeletedObjectHandler option
Use storage engine directly instead. It is already provided in the
options.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-15 10:28:48 +03:00
Evgenii Stratonikov
ae1dab29bc [#1482] metabase: Encode database keys in binary
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-15 10:28:26 +03:00
Leonard Lyubich
d6fef68a62 Release v0.32.0
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-14 17:52:49 +04:00
Leonard Lyubich
d895a272ad docs: Improve release instructions
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-14 17:52:49 +04:00
Leonard Lyubich
ae655b74f0 *: Exec make fmts
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-14 17:52:49 +04:00
Evgenii Stratonikov
4b8b575ca4 [#1782] go.mod: Update neofs-sdk-go
Add fixes for key parameter in `object.Get` operation.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-14 13:18:19 +04:00
Pavel Karpy
532c41ca05 [#1658] meta: Do not check object expiration in counters
Do not perform operations that produce unused results in `syncCounter`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy
431e331373 [#1658] shard: Update metric counters
Use meta's operation results to change the metrics. Support typed object
counters.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy
ad47e2a985 [#1658] meta: Add logic counter
- Meta now supports (and requires) inc/dec labeled counters
- The new logic counter is incremented on `Put` operations and is
decremented on `Inhume` and `Delete` operations that are performed on
_stored_ objects only
- Allow force counters sync. "Force" mode should be used on metabase resync
and should not be used on a regular meta start

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy
d872862710 [#1658] meta: Force counters resync process
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy
beb7f2a048 [#1634] node: Change default epoch in tests
Do not treat objects with expiration as expired by default.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy
edef26a4fd [#1658] engine: Update metrics interfaces
It now supports typed object counter metrics.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy
175e2b9fa0 [#1658] metrics: Add object type label to the counters
It will allow to use labeled object counters, e.g.: "phy" for all
the physically stored objects, "logic" for only available objects. Also, it
will allow to add typed (regular, TS, SG, LOCK) counters if needed.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Leonard Lyubich
a789159e5b [#1189] config/example: Mention persistent_sessions.path default
In current implementation storage node app uses in-memory session
storage if `persistent_sessions.path` value is empty/missing in the
config.

Clarify default behavior in `config/example/node.yaml`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-13 21:29:04 +04:00
Leonard Lyubich
d25b7e177b [#1189] node/config: Fix PersistentSessionsConfig.Path method's docs
Method never returns `PersistentStatePathDefault` value, and this is
correct.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-13 21:29:04 +04:00
Pavel Karpy
4083a62679 [#1628] tree: Pre-allocate errors in ACL checks
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 10:33:50 +03:00
Pavel Karpy
5adf089c1d [#1628] tree: Log unacceptable bearer attachment
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 10:33:50 +03:00
Pavel Karpy
2ffcd02ac3 [#1628] tree: Document ACL checks in tree service
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 10:33:50 +03:00
Pavel Karpy
8d0906c6ab [#1628] tree: Skip eACL filters for tree requests
Do not call `CalculateAction` for the eACL checks since it requires object
headers that are meaningless in the tree context.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 10:33:50 +03:00
Pavel Karpy
876e014b5d [#1628] tree: Make ACL checks the same way as for object requests
1. Do not require a request to be signed by the container owner if a
bearer token is missing
2. Do not check the system role since public requests are not expected to
be signed by IR or a container node (unlike the object requests)

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 10:33:50 +03:00
Pavel Karpy
4f18893d9b [#1628] node: Move common EACLSource interface to core pkg
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 10:33:50 +03:00
Pavel Karpy
4afb928ab6 [#1674] *: Expire entities after the expiration epoch
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-12 19:56:59 +04:00
Evgenii Stratonikov
f1572a674b [#1775] go.mod: Update neofs-sdk-go
Make use of fixes for the placement policy parser.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-12 19:10:49 +04:00
Evgenii Stratonikov
d25ec52459 [#1778] neofs-node: Panic if config type is invalid
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-12 17:06:00 +03:00
Evgenii Stratonikov
60b2930417 [#1778] neofs-node: Fix blobovnicza config type
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-12 17:06:00 +03:00
Evgenii Stratonikov
9da5d784cb [#1630] neofs-node: Remove trees on container removal event
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-12 09:54:15 +03:00
Evgenii Stratonikov
87bd49563e [#1761] pilorama: Use batch size of 1 for tests
Improve tests speed by a lot and use more iterations in `ApplyRandom` test.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-12 09:54:15 +03:00
Evgenii Stratonikov
a2bb3a2a96 [#1630] pilorama: Support dropping trees
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-12 09:54:15 +03:00
Leonard Lyubich
699b534416 [#1763] node/netmap: Write log message about parsed NewEpoch event
There is a need to have the ability to track NeoFS timeline on storage
nodes. Epochs tick on notifications receipt, so the most obvious way to
know about received epochs is logging the events.

Wrap `morphEvent.ParseNewEpoch` event parser into function which writes
log message about new epoch number.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-12 09:53:05 +04:00
Leonard Lyubich
3a206b5d7a [#1763] morph/event: Write log message about new notification event
Sometimes it is useful to track sidechain events processed by the node.
For example, we need some easy way to look up for log messages about new
epoch notifications.

Make `subscriber.Subscriber` to log names of all events received from
the sidechain notification channel.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-12 09:53:05 +04:00
Leonard Lyubich
8e8265d5ea [#1707] blobovnicza/get: Iterate over all buckets regardless of config
Re-configuration of Blobovnicza's object size limit must not affect
already stored objects. In previous implementation `Blobovnicza` didn't
see objects stored in buckets which became too big after size limit
re-configuration. For example, lets consider 1st configuration with 64KB
size limit for stored objects. Lets assume that we stored object of 64KB
size, and re-configured `Blobovnicza` with 32KB limit. After reboot
object should be still available, but actually it isn't. This is caused
by `Get` operation algorithm which iterates over configured size ranges
only, and doesn't process any other existing size bucket. By the way,
increasing of the object size limit didn't lead to the problem even in
previous implementation.

Make `Blobovnicza.Get` method to iterate over all size buckets
regardless of current configuration. This covers the described scenario.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-08 12:24:28 +04:00
Pavel Karpy
8e6e89aca3 [#1711] amd: Cache committee actor in the init context
Also simplify the code using `invoker.Invoker`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-08 10:09:44 +03:00
Pavel Karpy
60aa53651b [#1711] adm: Append witnesses only if they are missing
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-08 10:09:44 +03:00
Pavel Karpy
1e57565f6c [#1711] adm: Implement local client's methods
Implement methods that `actor.Actor` requires:
1. `InvokeFunction` -- wrapper over `InvokeScript`
2. `GetVersion` returns default struct
3. `CalculateNetworkFee` copied and simplified from Neo-go server side.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-08 10:09:44 +03:00
Pavel Karpy
4bd8608b37 [#1711] *: Simplify code using neo-go actors
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-08 10:09:44 +03:00
Evgenii Stratonikov
4354359aed [#1746] network: Set timeout for streaming operations
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 16:00:51 +04:00
Evgenii Stratonikov
0140ac354b [#1759] services/tree: Make logs more descriptive
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 12:57:33 +04:00
Evgenii Stratonikov
74c861342e [#1753] services/tree: Add parent ID to the GetNodeByPath response
Currently, you need to use `GetSubTree` to get parent, which seems an
overkill.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 10:52:37 +03:00
Evgenii Stratonikov
8a47f9ba12 [#1753] services/tree: Remove MaxGetSubTreeCount constant
It was used only in `GetOpLog` and ignored on the server side.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 10:52:37 +03:00
Evgenii Stratonikov
9840936a4f [#1753] services/tree: Do not restrict depth in GetSubTree
Previously, the depth was restricted because with BFS the amount of
nodes we have in memory blows up exponentially. With DFS is is linear,
so we can process trees of arbitrary depth.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 10:52:37 +03:00
c4c2840b52 [#1632] neofs-adm: Update autogenerated storage node config
Replace `diable_cache` with `cache_ttl` value.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-09-07 10:34:46 +03:00
944665dd5d [#1632] neofs-node: Update docs
Replace `diable_cache` with `cache_ttl` value

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-09-07 10:34:46 +03:00
Evgenii Stratonikov
e4dcc4d6a9 [#1749] neofs-adm: Register contract hashes in NNS in 2 formats
NNS proposal describes string N3 address format, however we must also
have hex-string for backwards compatibility.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 09:58:05 +03:00
Evgenii Stratonikov
fe0c6db67d [#1749] neofs-adm: Delete NNS records before updating
In case we already have the record and it is invalid, we should
overwrite it instead of having several conflicting records.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 09:58:05 +03:00
Evgenii Stratonikov
0818d8d43a [#1751] .github: Add label checklist to templates
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 09:57:50 +03:00
Pavel Karpy
fa18100489 [#1714] lens: Add open*COMPONENT* funcs
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-06 18:09:18 +04:00
Pavel Karpy
01d7c007aa [#1714] Update CHANGELOG
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-06 18:09:18 +04:00
Pavel Karpy
2132b78aba [#1714] lens: Add meta subcommand
Includes:
1. `inspect`
2. `list-garbage`
3. `list-graveyard`

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-06 18:09:18 +04:00
Pavel Karpy
adff08ad02 [#1714] lens: Separate commands
Separate `inspect` and `list` on `write-cache` and `blobovnicza`
subcommands.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-06 18:09:18 +04:00
Pavel Karpy
387953c1e0 [#1756] make: Fix commit abbrev length in version
Make it uniform across all our repos.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-06 10:53:38 +03:00
Evgenii Stratonikov
fef4f6d155 [#1748] neofs-adm: Allow to dump hashes from a custom zone
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 21:31:32 +03:00
Evgenii Stratonikov
d4ac5bdb97 [#1748] neofs-adm: Allow to parse hashes in multiple formats
NEO NNS proposal uses addresses. We should eventually use the same,
but must stay compatible now.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 21:31:32 +03:00
Evgenii Stratonikov
4ccacb89e8 [#1748] neofs-adm: Set custom contract hashes as N3 address
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 21:31:32 +03:00
Evgenii Stratonikov
11e6f03aa2 [#1748] neofs-adm: Refactor NNS price handling
Make `nnsRegisterDomainScript` simpler.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 21:31:32 +03:00
Evgenii Stratonikov
9290a7e1fe [#1748] neofs-adm: Remove addRecord from nnsRegisterDomainScript
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 21:31:32 +03:00
Evgenii Stratonikov
eb5e1121c5 [#1748] neofs-adm: Move dump-hashes command to a separate file
I have just lost 3 minutes trying to find it.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 21:31:32 +03:00
Leonard Lyubich
e26e70ffcf [#1652] cli/container: Mention new flag in the CHANGELOG
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 16:22:16 +04:00
Leonard Lyubich
33143b18a9 [#1652] cli/container: Invert pre-check flag of set-eacl command
Flag `--pre-check` of `set-eacl` command found to be in demand in most
cases. based on this, it makes sense to add its action to the default
behavior.

Pre-check container extensibility by default. Rename flag to
`--no-precheck` and invert its action.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 16:22:16 +04:00
Leonard Lyubich
46b815c863 [#1652] cli/container: Move eaclPathFrom flag var into the struct
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 16:22:16 +04:00
Leonard Lyubich
491a908af1 [#1652] cli/container: Pre-check ACL extensibility in set-eacl
Container ACL in NeoFS can be extended only for container in which the
corresponding option is enabled. In previous implementation command
`set-eacl` could hang up on modifying eACL of the non-existent or
non-extendable container. To improve UX, there is a need to pre-check
the availability of `SETEACL` operation.

Add boolean `precheck` flag to `set-eacl` cmd which reads the container
before the actual transaction formation. If flag is set, command fails
on non-extendable container ACL.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 16:22:16 +04:00
Pavel Karpy
17059b34ea [#1743] morph: Fix non-notary calls
Some methods add "IR" suffix to its names in notary enabled envs
because of contract logic. It was broken due to incorrect notary state
reading (tryNotary != notary is enabled).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-02 12:11:57 +03:00
Evgenii Stratonikov
e6cf0b3d42 [#1745] writecache: Set flush parameters based on max size
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
c04126f35e [#1745] writecache: Remove IsErrNotFound
We specify the error in the doc-comment, and it is the same for all our
components.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
9e41e85295 [#1745] writecache: Actualize docs
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
e9c6ee2623 [#1745] writecache: Rename constants for default values
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
177e8e01b1 [#1745] neofs-node: Remove memcache_capacity from the configuration
It is unused since ddaed283e9 .

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
bda084f331 [#1745] writecache: Simplify object counters
Remove unused option and additional pointers to db/fstree.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
20abdaeed4 [#1745] writecache: Set flush mark only on success
Set flush mark in the inside the flush worker because writing to the blobstor
can fail. Because each evicted object must be deleted, it is reasonable
to do this in the evict callback.

The evict callback is protected by LRU mutex and thus potentially interferes
with `Get` and `Iterate` methods. This problem will be addressed in the
future.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 11:59:53 +03:00
Pavel Karpy
82839cb1c9 [#1711] morph: Add nolint for AddNetworkFee
Waits for `v0.99.3` neo-go unreleased yet.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-02 10:41:18 +03:00
Pavel Karpy
62b293b7ab [#1711] adm: Add error context
To make all the calls in the command to be in the same form.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-02 10:41:18 +03:00
Pavel Karpy
a99474c40e [#1711] adm: Drop deprecated neo-go calls
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-02 10:41:18 +03:00
Pavel Karpy
6d277a57aa [#1711] morph: Simplify code using neo-go actors
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-02 10:41:18 +03:00
Pavel Karpy
4582b8f0d4 [#1711] morph: Drop deprecated neo-go calls
Does not include notary actor and does not drop the deprecated
`AddNetworkFee`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-02 10:41:18 +03:00
Leonard Lyubich
ae6ebccd9c [#1632] node/container: Don't add new container to missed cache record
If container listing cache on node's side is missing (for particular
owner), then updating it as a reaction to successful container creation
leads to potentially invalid cache value for a period of time equivalent
to cache TTL.

Immediately return from `ttlContainerLister.update` method if owner's
container list isn't cached.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 11:29:55 +04:00
Leonard Lyubich
fa3124fc33 [#1632] node: Configure TTL of Sidechain caches
From now cache TTL can be parameterized in the `neofs-node` app using
`cache_ttl` config key. `disable_cache` value is no longer supported.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 11:29:55 +04:00
Leonard Lyubich
5ce8315cd8 [#1632] node/container: Update LIST cache on successful writes
In previous implementation storage node responded with the outdated
container list after successful creation/removal up until cache
invalidation due to TTL. In order to decrease the probability of
outdated responses node should update its cache on event receipts.

Implement `ttlContainerLister.update` method which actualizes cached
list of the owner's containers. Make node to call `update` method
on `PutSuccess`/`DeleteSuccess` notifications from the `Container`
contract.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 11:29:55 +04:00
Leonard Lyubich
d15a7d8d3d [#1632] node/container: Update GET cache on successful removal
In previous implementation storage node responded with the removed
container up until cache invalidation due to TTL. In order to avoid
false-positive responses node should update its cache on `DeleteSuccess`
events.

Make node to call `handleRemoval` method of the container cache which
leads to subsequent `apistatus.ErrContainerNotFound` errors.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 11:29:55 +04:00
Leonard Lyubich
a529149e6f [#1632] node: Cache Sidechain errors along with the values
In previous implementation failed requests to the Sidechain weren't
cached. It makes sense to cache errors along with the values in order to
decrease potential load spikes onto Sidechain nodes.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 11:29:55 +04:00
Leonard Lyubich
7c1babb7d6 [#1632] node: Subscribe on the successful container creations/removals
There is a need to sync container-related caching mechanism with the
actual Sidechain changes. To do this, node should be able to listen
incoming notifications about container ops.

Define `PutSuccess` / `DeleteSuccess` notification event's parsers.
Subscribe to these events in node app. As initial implementation node
will log event receipts. Later handling is going to be practically
complicated.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 11:29:55 +04:00
Evgenii Stratonikov
d8a00c365a [#1691] blobovniczatree: Fix active blobovnicza caching
Maintain an invariant that any blobovnicza is present either
in `opened` or in `active` map. Otherwise, the logic becomes too
complicate because it is not obvious when we should close the blobovnicza.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-31 19:11:48 +03:00
Evgenii Stratonikov
dfac4e1c0b [#1691] blobovniczatree: Do not fill cache during initialization
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-31 19:11:48 +03:00
Evgenii Stratonikov
df88afeef3 [#1738] neofs-adm: Clean tx hash list after awaiting
Do not query application log for the same transaction more than once.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 16:53:00 +03:00
Evgenii Stratonikov
ae52d53609 [#1698] pilorama: Add a test for the empty FileName
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 16:52:51 +03:00
Evgenii Stratonikov
180e5e938f [#1735] go.mod: Update API and SDK
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 12:27:46 +03:00
Evgenii Stratonikov
78bf17c0b0 [#1686] blobstor/*: Remove unneeded tests
The blobovniczatree test also checks the amount of objects we can put,
so leave it here.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
b9a2055e1c [#1686] blobstor/*: Return ErrReadOnly for modifying operations
This check should occur on the shard level, but because
blobstor components expose `Open(readOnly bool)` interface,
it is reasonable to expect an error here.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
6c2d3b020f [#1686] fstree: Return proper error for concurrent Delete
If the file doesn't exist, return `apistatus.ObjectNotFound`.
First check is still there as a shortcut.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
0b95a21701 [#1686] blobstor: Add generic tests
This tests check that each blobstor component behaves similarly when
same methods are being used. It is intended to serve as a specification
for all future components.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
b2d4cc556e [#1686] blobovnicza: Remove IterateObjects
It is used only once, makes sense to inline.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
5f95498364 [#1686] blobstor: Add generic tests
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
9792e6a4ef [#1686] metabase: Add generic tests
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
482a7e7f2f [#1686] pilorama: Add generic tests
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
b8b9d25f9d [#1686] local_object_storage: Add generic tests
Use them for writecache as a simple example.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
bc20851cd1 [#1686] shard: Open all components in case metabase has failed
All components must be opened first, before any other operation is
performed.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Pavel Karpy
c7c1c257e1 [#1658] node: Init object counter on meta's Init
Includes:
1. Renaming counter key to distinguish logical and physical objects
2. Version update dropping since changes could be done in a compatible way

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-25 19:20:33 +03:00
Pavel Karpy
cd6f8e051a [#1658] Update CHANGELOG
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-25 19:20:33 +03:00
Pavel Karpy
9c7b3ce799 [#1658] node: Read metrics from meta on startup
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-25 19:20:33 +03:00
Pavel Karpy
745f72fff0 [#1658] node: Support object counter metrics
Increment shard's object counter on successful `Put` calls and decrement on
`Delete`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-25 19:20:33 +03:00
Pavel Karpy
067ab44e0b [#1658] metrics: Add object counter metric
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-25 19:20:33 +03:00
Pavel Karpy
626766db08 [#1658] meta: Add object counter
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-25 19:20:33 +03:00
Evgenii Stratonikov
61f0d85834 [#1727] neofs-adm: Allow to register custom TLD for contracts
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-25 12:31:00 +03:00
Evgenii Stratonikov
ddbe129cb2 [#1729] neofs-adm: Change default domain expiration time
It is a noop for current contract versions, but will matter in the next
release.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-25 12:31:00 +03:00
Evgenii Stratonikov
5827a926c9 [#1729] go.mod: Update neofs-contract to v0.15.5
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-25 12:31:00 +03:00
Evgenii Stratonikov
19585c63ed [#1730] go.mod: Update neofs-sdk-go
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-24 17:17:42 +03:00
e68c80df0a [#1728] Don't describe default value twice
Cobra will mention default value automatically.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-08-24 16:44:44 +03:00
31b6af5667 [#1728] Update neofs-adm morph deploy command description
Add more details about expected file names in contract dir.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-08-24 16:44:44 +03:00
Pavel Karpy
b54f34d7df [#1683] adm: Do not deploy custom contract with a group
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-24 13:52:55 +03:00
Pavel Karpy
bbd6b9780f [#1683] adm: Minimize registration price
Register NNS domain in one TX:
1. Set minimal (`1`) registration price;
2. Register domain;
3. Return registration price back.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-24 13:52:55 +03:00
Evgenii Stratonikov
d57c57010f [#1726] neofs-cli: Truncate file before writing the object
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-23 18:07:12 +03:00
Evgenii Stratonikov
40f1cf36e1 [#1719] neofs-cli: Save object with proper permissions
`curl` uses 0644, for example.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-23 18:07:12 +03:00
Evgenii Stratonikov
871fae8321 [#1705] go.mod: Update neo-go to v0.99.2
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-23 11:59:23 +03:00
Evgenii Stratonikov
e4370c4129 [#1702] neofs-adm: Set HighPriority attribute for committee transactions
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-23 11:59:23 +03:00
Pavel Karpy
0720d96c9d [#1687] go.mod: Update neofs-sdk-go
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-22 18:59:57 +03:00
Pavel Karpy
d40ed00438 [#1687] go.mod: Update neofs-api-go
It contains the signature fix for go `v1.19`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-22 18:59:57 +03:00
Pavel Karpy
56792d5eaf [#1687] .github: Update actions to the latest version
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-22 18:59:57 +03:00
Pavel Karpy
99903e1aba [#1687] Support v1.19 Golang
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-22 18:59:57 +03:00
Pavel Karpy
30341f2192 [#1687] *: Perform go fmt using go v1.19
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-22 18:59:57 +03:00
Pavel Karpy
37ab26bfa9 [#1296] cli: "ID" -> "OID"
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-22 14:29:50 +04:00
Evgenii Stratonikov
e8c6dce466 [#1523] blobstor: Rename compression.CConfig to Config
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
0042c1d4a6 [#1584] blobovniczatree: move public methods to different files
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
26b4a258e0 [#1523] neofs-node: Refactor configuration
1. Move compression parameters to the `shard` section.
2. Allow to use multiple sub-storage components in the blobstor.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
13cdbde2e2 [#1523] blobstor: Remove unused types
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
0c9e4e6a35 [#1523] blobstor: Unify request dispatch logic
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
266458fe5c [#1523] blobstor: Unify fstree and blobovnicza interfaces
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
5aa3defc67 [#1523] writecache: Remove cloneBytes
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
ddaed283e9 [#1523] writecache: Simplify logic
1. Remove in-memory cache. It doesn't persist objects and if we want
   more speed, `NoSync` option can be used for the bolt DB.
2. Put to the metabase in a synchronous fashion. This considerably
   simplifies overall logic and plays nicely with the metabase bolt DB
   batch settings.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
4176b2a1bc [#1523] local_object_storage: Unify parameters for the Iterate operation
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
73f8bb3e5f [#1523] shard: Store generic storage ID in metabase
Allow to extend blobstor with more storage sub-systems. Currently
objects stored in the FSTree have empty byte slice descriptor and object
from blobovnicza tree have the same id as earlier. Each such change in
the identifier formation should be accompanied with metabase version
increase.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
9eb018672c [#1523] blobstor: Unify parameters for Put operation
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
7d6df543d7 [#1523] blobstor: Export single Get, GetRange and Delete methods
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
ca15083a50 [#1523] local_object_storage: Unify parameters for the Exists operation
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
d75d030a90 [#1523] local_object_storage: Unify parameters for the Delete operation
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
36c88f0dc8 [#1523] local_object_storage: Unify parameters for the GetRange operation
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
d9a2f280c9 [#1523] local_object_storage: Unify parameters for the Get operation
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
b621f5983a [#1523] local_object_storage: Move blobovnicza tree to a separate package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Pavel Karpy
5139dc9864 [#1706] cli: Do not duplicate payload on SG put
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-19 09:49:47 +04:00
Evgenii Stratonikov
4e361e2ab4 [#1647] writecache: Open DB in sync mode
Close #1647.

Initially, `Sync: false` was provided because we can already lose
objects cached in memory. However, future changes in writecache will
remove inmemory cache and speed up it via other means.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-16 15:27:18 +04:00
Evgenii Stratonikov
89cba2cc7b [#1684] .github: Update golangci-lint version
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-15 10:57:31 +03:00
Evgenii Stratonikov
f8b106ac85 [#1684] *: Fix linter warnings
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-15 10:57:31 +03:00
Leonard Lyubich
67af4c89a8 [#1649] cli: Refactor --owner flag definition in container list cmd
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-08-15 10:18:51 +03:00
Leonard Lyubich
7a26b2a57a [#1649] cli: Add option to print attributes in list
Define `--with-attr` flag of `container list` which makes the command to
request and print user attributes for each found element.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-08-15 10:18:51 +03:00
Leonard Lyubich
2740bf7ee4 [#1649] cli: Add option to print attributes in list-objects
Define `--with-attr` flag of `container list-objects` which makes the
command to request and print user attributes for each object from the
container.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-08-15 10:18:51 +03:00
e07921fc46 Use only v-prefixed tags for version calculation
This allows us to use any custom tags along with tags that denote releases

Signed-off-by: Vladimir Domnich <v.domnich@yadro.com>
2022-08-10 10:34:07 +03:00
Pavel Karpy
da2975a2f9 [#1664] write-cache: Fix panic on Delete operation
If an object is found in the Write-cache and is placed at the end of
the in-memory cache, the memory counter update operation tries to
dereference the index that is out of the sliced array. Moreover, even if
panic does not appear, the counter is updated with the wrong value.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-09 20:18:26 +03:00
a.y.volkov
81684b6f04 [#1665] neofs-lens: Use stdout as default output
Signed-off-by: a.y.volkov <a.y.volkov@yadro.com>
2022-08-09 13:23:32 +03:00
Pavel Karpy
cd71de69a0 Release v0.31.0
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 17:04:33 +03:00
Pavel Karpy
5ff1df285b Release v0.30.2
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 17:04:33 +03:00
Evgenii Stratonikov
058538768b [#1640] Release v0.30.1
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 17:04:33 +03:00
Pavel Karpy
a5da665e69 [#1656] meta: Fix MatchCommonPrefix operation
Return all the objects on the empty common prefix search without search
optimizations that breaks boltDB logic.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 16:32:00 +03:00
Pavel Karpy
156ba85326 [#1634] node: Do not return expired objects
If an object has not been marked for removal by the GC in the current epoch
yet but has already expired, respond with `ErrObjectNotFound` api status.
Also, optimize shard iteration: a node must stop any iteration if the object
 is found but gonna be removed soon.
All the checks are performed by the Metabase.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 16:31:49 +03:00
Pavel Karpy
9aba0ba512 [#1634] meta: Add epoch state
It allows performing expiration checks on the stored objects.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 16:31:49 +03:00
Pavel Karpy
a97dee008c [#1648] morph: Change endpoint priority order
The lowest value means the highest priority.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 16:11:24 +03:00
Pavel Karpy
2467be0117 [#1648] morph: Init endpoints before first client creation
Sort the endpoint by their priority before the first WS client creation to
start the morph client with the highest priority endpoint.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 16:11:24 +03:00
Pavel Karpy
1c1b5043fc [#1653] morph: Handle chain notifications via pool
Handling notification in a synchronous manner may lead to a blocking state
if a handler uses neo-go client.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 17:06:53 +04:00
Pavel Karpy
8f44335925 [#1651] cli: Fix CID setting in eACL creation
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 17:04:42 +04:00
Evgenii Stratonikov
9a5f9d6f0e [#1643] innerring: Exit if we cannot bind to the control endpoint
Return listen errors in a synchronous fashion.
Another solution would be to use buffered channel, but this is not
scalable: for each new similar runner we would need to extend the
buffer.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-04 15:41:41 +03:00
Pavel Karpy
0a60524a9c [#1654] ir, node: Drop deprecated profiler and metrics config sections
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 15:38:35 +03:00
Pavel Karpy
713cfa5610 [#1655] cli: Do not force specifying session lifetime
We have the default value which is also printed in the help messages but any
call that does not specify that flag leads to an error.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 15:37:50 +03:00
Pavel Karpy
c2918fce3a [#1645] node: Support EACL_NOT_FOUND status
Remove internal `ErrEACLNotFound` error.
Also, update `neofs-api-go` and `neofs-sdk-go` libraries.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-01 20:45:36 +03:00
Pavel Karpy
8c8ae56960 [#1645] go.mod: Update dependencies
- `neofs-api-go` to `v2.13.1`;
- `neofs-sdk-go` to `7a99cc916c8e`.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-01 20:45:36 +03:00
Evgenii Stratonikov
8ffc2fdf5e [#1642] engine: Do not increase error counter if the pilorama is disabled
After a4adb79db new logical error could be returned. Do not increase
error counter in this case.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-30 18:39:22 +03:00
Evgenii Stratonikov
4558f30575 [#1644] services/tree: Do not use deprecated gRPC options
Get rid of `grpc.WithInsecure`.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-30 18:39:08 +03:00
Evgenii Stratonikov
8d0884e74f [#1644] storagelog: Fix doc comment
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-30 18:39:08 +03:00
Evgenii Stratonikov
7ecf54c6df [#1631] neofs-adm: Set group for updated alphabet contracts
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 12:09:48 +03:00
Evgenii Stratonikov
d3eac7626d [#1631] neofs-adm: Set group for a proper contract
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 12:09:48 +03:00
Evgenii Stratonikov
5efcd34680 [#1639] go.mod: Update neofs-sdk-go to v1.0.0-rc.6
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 12:07:53 +03:00
Evgenii Stratonikov
5d02613c33 [#1633] neofs-adm: Allow to set custom NNS zone for deployed contracts
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 12:02:40 +03:00
Evgenii Stratonikov
6146df4998 [#1633] neofs-adm: Allow to provide arguments to deployed contracts
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 12:02:40 +03:00
Evgenii Stratonikov
dd30703a6b [#1633] neofs-adm: Allow to update arbitrary contracts
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 12:02:40 +03:00
Evgenii Stratonikov
fc06b6e89a [#1633] neofs-adm: Allow to deploy arbitrary contracts
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 12:02:40 +03:00
Evgenii Stratonikov
38558a3238 [#1633] neofs-adm: Start chain only on happy path
If the error is returned, it will not be closed properly.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 12:02:40 +03:00
Evgenii Stratonikov
642d7328ab [#1633] neofs-adm: Fix dump generation in init
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 12:02:40 +03:00
Evgenii Stratonikov
4abe5a7245 shard: add more checks for GetRange parameters
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 11:22:04 +03:00
Evgenii Stratonikov
72586f17d4 shard: fix GetRange for objects stored in the write-cache
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 11:22:04 +03:00
Evgenii Stratonikov
07465849a4 [#1637] go.mod: Update neo-go to v0.99.1
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-28 20:11:45 +03:00
Alex Vanin
71fd86f220 [#1636] ir: Listen balance contract in balance processor
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-07-28 18:47:20 +03:00
Pavel Karpy
8162b27264 [#1627] node: Add tree service to the config framework
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-27 12:15:56 +03:00
Pavel Karpy
a0fae0443f [#1627] node: Add missing tree service to the config files
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-27 12:15:56 +03:00
Evgenii Stratonikov
54fe7667fb [#1625] go.mod: Update dependencies
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-25 19:00:07 +03:00
Evgenii Stratonikov
2a1c5557f2 [#1624] Release v0.30.0
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-25 16:56:02 +03:00
Evgenii Stratonikov
10c855efef [#1624] go.mod: Update dependencies
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-25 16:56:02 +03:00
Evgenii Stratonikov
91361c4fe2 [#1624] CHANGELOG.md: Fix v0.29.0 release name
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-25 16:56:02 +03:00
Evgenii Stratonikov
25a0f539ea [#1624] config: Fix tree section example
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-25 16:56:02 +03:00
Pavel Karpy
589a54805d [#1618] node: Use OID/CID from the request in eACL checks
Also, try to fetch object header info from the local storage to find as much
object info as possible for the requests which do not assume returning
object header as a response.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-25 09:41:11 +03:00
Evgenii Stratonikov
6049022f7e [#1612] neofs-cli: Remove RunE functions from commands
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-22 16:18:09 +03:00
Evgenii Stratonikov
a52e7c2c99 [#1612] neofs-cli: Unify expiration flags
Use `expire-at` everywhere expiration epoch is expected.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-22 16:18:09 +03:00
Evgenii Stratonikov
175e9da3a7 [#1612] go.mod: Update cobra dependency
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-22 16:18:09 +03:00
Evgenii Stratonikov
7e717e80ef [#1623] services/tree: Fix error message
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-22 15:13:03 +03:00
Pavel Karpy
2f4e465cf4 [#1620] node: Remove debug attribute printing
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-22 13:50:36 +03:00
Pavel Karpy
ee9312dcfb [#1619] example: Fix IR RPC
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-22 12:37:25 +03:00
Pavel Karpy
fb1fac02e9 [#1619] ir: Fix RPC endpoint iteration
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-22 12:37:25 +03:00
Evgenii Stratonikov
1691364653 [#1559] local_object_storage: Fix tests and some data races
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
50e28f22f9 [#1559] shard: Change Degraded mode string representation
It is a flag, but is a `degraded-read-write` mode for a user.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Leonard Lyubich
fabe717d32 [#1549] shard: Turn to read-only mode on metabase failure
If metabase can't be opened in the default mode, try opening shard
first in `ReadOnly` mode and then in `DegradedReadOnly`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
4944490ffb [#1559] local_object_storage: Move shard to the DegradedReadOnly mode
`Degraded` mode can be set by the administrator if needed.
Modifying operations in this mode can lead node into an inconsistent state
because metabase checks such as lock checking are not performed.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
3df62769c0 [#1559] local_object_storage: Allow to set mode for all components
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
1e786233bf [#1559] local_object_storage: Provide readOnly flag to Open
We should be able to reopen storage in readonly in runtime.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Leonard Lyubich
e38b0aa4ba [#1549] engine: Disable shard on blobovnicza init failure
There is a need to support working w/o shard if it has problems with
blobovnicza tree.

Make `BlobStor.Init` to return new `ErrInitBlobovniczas` error. Remove
shard from storage engine's shard set if it returned this error from
`Init` call. So if some of the shards (but not all) return this error,
the node will be able to continue working without them.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
40a56c6b42 [#1559] engine: Do not count logical errors as storage ones
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
c8911d72d0 [#1559] shard: Do not consult metabase in a degraded mode
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
339864b720 [#1559] local_object_storage: Move shard.Mode to a separate package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
d8ba954aff [#1559] shard: Use Set prefix for parameter setting
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
7b882b26d8 [#1559] shard: Remove public functions
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
34d319fed2 [#1559] metabase: Use Set prefix for parameter setting
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
f58234aa2f [#1559] metabase: Remove public functions
Reduce public interface of this package. Later each result will contain
an additional status, so it makes more sense to use the same functions
and result processing everywhere.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
30c7925b3c [#1609] morph/client: Retry connecting to failed endpoint during the switch
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 16:08:42 +03:00
Evgenii Stratonikov
7410827db8 [#1609] config: Allow to prioritize N3 endpoints
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 16:08:42 +03:00
Evgenii Stratonikov
aed83d1660 [#1609] config: Remove mainchain section
It is deprecated and unused, there is no need to have it in the
documentation.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 16:08:42 +03:00
Pavel Karpy
a4ed91f9b5 [#1614] node: Update health metric on its change
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-21 16:07:47 +03:00
Pavel Karpy
581a9901c9 [#1614] metrics: Add health metrics
Also, rename metrics structure since it collects not only storage metrics
now.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-21 16:07:47 +03:00
Evgenii Stratonikov
2455b72844 [#1607] CHANGELOG.md: add tree service info
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
b549cc314c [#1607] services/tree: allow to customize some parameters
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
2334e94fdb [#1607] neofs-node: Initialize storage before other services
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
a4adb79db7 [#1607] pilorama: Enable tree service explicitly
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
d62723f038 [#1505] pilorama: Provide timeout to bbolt.Open
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
26041f18bf [#1505] pilorama: Allow to customize database parameters
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
f80e52fbea [#1334] services/tree: Document *.proto files
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
3904b0e017 [#1333] neofs-cli: Add control synchronize-tree command
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
bfdd68dcb3 [#1333] services/control: Allow to synchronize local trees
Do not check that a node indeed belongs to the container, because the
synchronization will fail in this case anyway.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
0dc1a4e336 [#1333] neofs-node: Initialize tree service before the control one
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
c7b10598f9 [#1333] engine: Increase error counter for pilorama errors
1. Modifying operations are not expected to fail, unless the shard is
   read-only.
2. `Get*` operations should increase error counter too, unless the
   error is `ErrTreeNotFound`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
5e843a73f9 [#1333] services/control: Return pilorama info in ListShards RPC
Do not return backend type from the service for now, because memory
backend is expected to vanish.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
8f4ee1aded [#1333] local_object_storage: Support ReadOnly mode in pilorama
The tricky part here is the engine itself: we stop iteration on
`ErrReadOnly` because it is better to synchronize the shard later than
to have partial trees stored in 2 shards.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
735931c842 [#1481] pilorama: Fix TreeApply
Current implementation prevents invalid operations to become valid at
some later point (consider adding a child to the non-existent parent and
then adding the parent). This seems to diverge from the paper algorithm
and complicates implementation. Make it simpler.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
5ffbeb76e6 [#1456] services/tree: Wait some time before reconnecting after failure
In case node is down or failing for some reason, we can expect `Dial` to
fail. In case we actively try to replicate and `Dial` always takes 2
seconds, replication-related channels quickly become full. That affects
latency of all other write operations.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
39f47f61c6 [#1445] services/tree: Cache the list of container nodes
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
a2edfec0c3 [#1446] services/tree: Use grpc.WithInsecure only for nodes without TLS
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
ad3038d16d [#1444] pilorama: Fix TreeMove in bbolt backend
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
33d8fb187a [#1427] services/tree: Parallelize replicator
Before this commit the replication channel was quickly filled under
heavy load. This lead to the continuously increasing latency for all
write operations. Now it looks better.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
8027b7bb6b [#1444] pilorama: Optimize internal encoding/decoding
```
name                      old time/op    new time/op    delta
ApplySequential/bbolt-8     55.5µs ± 4%    55.5µs ± 3%     ~     (p=1.000 n=10+7)
ApplyReorderLast/bbolt-8     108µs ± 6%     112µs ± 8%     ~     (p=0.077 n=9+9)

name                      old alloc/op   new alloc/op   delta
ApplySequential/bbolt-8     28.8kB ± 3%    27.7kB ± 6%   -3.79%  (p=0.005 n=10+10)
ApplyReorderLast/bbolt-8    41.4kB ± 5%    38.9kB ± 5%   -6.19%  (p=0.001 n=10+9)

name                      old allocs/op  new allocs/op  delta
ApplySequential/bbolt-8        262 ± 2%       235 ±10%  -10.41%  (p=0.000 n=10+10)
ApplyReorderLast/bbolt-8       684 ± 6%       616 ± 7%  -10.04%  (p=0.000 n=10+9)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
b04f712773 [#1446] services/tree: Cache connections to the container nodes
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
4437cd7113 [#1442] pilorama: Generate timestamp based on node position in the container
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
3caa982283 [#1442] services/tree: Fix log message for failed Apply
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
01d2e06a9b [#1401] services/tree: Marshal public key once
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
1f5a650b05 [#1401] services/tree: Retransmit queries to container nodes
Also fix a bug with replicator using the multiaddress instead of
<host>:<port> format expected by gRPC library.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
3312924b82 [#1431] pilorama: Use Batch for write transactions
Helps a lot in case of concurrent request flow.

```
name                      old time/op    new time/op    delta
ApplySequential/bbolt-8     78.0µs ± 9%    59.8µs ± 4%  -23.39%  (p=0.000 n=10+9)
ApplyReorderLast/bbolt-8     143µs ± 5%     113µs ±15%  -21.06%  (p=0.000 n=10+10)

name                      old alloc/op   new alloc/op   delta
ApplySequential/bbolt-8     56.9kB ± 8%    28.9kB ± 3%  -49.22%  (p=0.000 n=10+10)
ApplyReorderLast/bbolt-8    87.3kB ± 3%    40.9kB ±10%  -53.16%  (p=0.000 n=10+10)

name                      old allocs/op  new allocs/op  delta
ApplySequential/bbolt-8        224 ±11%       262 ± 5%  +16.93%  (p=0.000 n=9+10)
ApplyReorderLast/bbolt-8       518 ± 4%       674 ±11%  +30.09%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
f0a67f948d [#1431] pilorama: Cache attributes in the index
Currently to find a node by path we iterate over all the children on
each level. This is far from optimal and scales badly with the number of
nodes on a single level. Thus we introduce "indexed attributes" for
which an additional information is stored and which can be use in
`*ByPath` operations. Currently this set only includes `FileName`
attribute but this may change in future.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
f5d35571d0 [#1431] engine: Add benchmark for Select vs TreeGetByPath
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
f4bc1c601a [#1329] services/tree: Synchronize from the last stored height
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
835170e452 [#1329] pilorama: Allow to benchmark all tree backends
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
536857ea5a [#1329] services/tree: Implement GetOpLog RPC
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
886baf3136 [#1426] services/tree: Do not replicate to a local node
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
7703dd5d7f [#1419] pilorama: Create new nodes in path if needed
Consider a node `{FileName: "dir", Attribute: "xxx"}`. In case we add
a new node by path `["dir", "file.txt"]`, create a new intermediate node
with a single attribute.

`GetByPath` now also considers only nodes with a single attribute while building a path.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
ad48918a97 [#1406] pilorama: Return parent from TreeGetMeta
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Denis Kirillov
b30f14978d [#1404] services/tree: Fix nodeId in GetSubTree
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
45321b3f6d [#1328] services/tree: Fix grpc import path
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
16e3421825 [#1328] services/tree: Implement access control
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
94b9e13431 [#1328] services/tree: Fix proto field numbers
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
36e24f7f78 [#1342] neofs-node: Use the default endpoint for tree service
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
8844d9b2db [#1344] pilorama: Document errors for Get* methods
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
d8ad68d613 [#1344] engine: Log errors in Tree* operations
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
910db42748 [#1344] pilorama: Use require.ErrorIs in tests
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
aea855e8f3 [#1326] services/tree: Implement GetSubTree RPC
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
62154da17c [#1324] services/tree: Implement Object Tree Service
Object Tree Service allows changing trees assotiated with
the container in runtime.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
46f4ce2773 [#1324] engine: Implement Forest interface for storage engine
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
8cf71b7f1c [#1324] local_object_storage: Implement tree service backend
In this commit we implement algorithm for CRDT trees from
https://martin.klepmann.com/papers/move-op.pdf

Each tree is identified by the ID of a container it belongs to
and the tree name itself. Essentially, it is a sequence of operations
which should be applied in chronological order to get a usual tree
representation.

There are 2 backends for now: bbolt database and in-memory.
In-memory backend is here for debugging and will eventually act
as a memory-cache for the on-disk database.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
198beae703 [#1613] neofs-cli: Print lock object ID
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-19 17:15:15 +03:00
Evgenii Stratonikov
5c8d725447 [#1610] neofs-adm: Fix generated storage config
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-19 16:19:41 +03:00
Evgenii Stratonikov
11c9df6f00 [#1611] shard: Print shard ID in component logs
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-19 15:57:53 +03:00
Pavel Karpy
9f7a22e2aa [#1561] object: Return OUT_OF_RANGE status
Replace `ErrRangeOutOfBounds` error from `pkg/core/object` package with
`ObjectOutOfRange` from `apistatus` package. That error is returned by
storage node's server as NeoFS API statuses.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-19 13:29:45 +03:00
Evgenii Stratonikov
ccba07fe19 [#1608] neofs-adm: Register candidates in a single transaction
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-19 13:28:17 +03:00
Evgenii Stratonikov
59d08b8dae [#1608] neofs-adm: Print info about already performed steps
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-19 13:28:17 +03:00
Evgenii Stratonikov
5e1636e50c [#1604] docs: Describe storage node configuration
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-19 11:12:52 +03:00
Evgenii Stratonikov
eeef83a74c [#1604] config: Change yaml example a bit
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-19 11:12:52 +03:00
Evgenii Stratonikov
e5c0338f8c [#1602] neofs-ir: Fix defaults for pprof and prometheus
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-18 19:41:22 +03:00
Evgenii Stratonikov
3a0b06d5da [#1602] config: Rename metrics and profiler sections
Depracate old names and remove them in the next release.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-18 19:41:22 +03:00
Evgenii Stratonikov
ac46d1a11f [#1602] config: Enable metrics and profiler services with a flag
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-18 19:41:22 +03:00
Evgenii Stratonikov
12dc5c3395 [#1563] config: Replace 127.0.0.1 with localhost
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-18 19:41:22 +03:00
Evgenii Stratonikov
4a8dfd0ce4 [#1606] Remove deprecated functions
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-18 12:41:06 +03:00
Evgenii Stratonikov
6cc180391e [#1606] go.mod: Update neo-go
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-18 12:41:06 +03:00
Pavel Karpy
cb0bb7207c [#1461] shard: Add a separate ErrLockObjectRemoval
Do not return `meta.ErrLockObjectRemoval` from shard's methods, add shard's
own error for that.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
558cc1193a [#1461] engine: Clarify force removal
Document force removal behaviour in all the Storage engine parts.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
b1e54dec9f [#1461] cli: Add context to GetCurrentEpoch helper
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
51afcc1182 [#1461] engine, policer: Force remove objects w/o container
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
18ec5d7c8e [#1461] meta: Return error on lock object removal
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
b5c56d459a [#1461] engine: Add force lock removal tests
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
63c00e785d [#1461] shard: Fix option naming
`WitDeletedLockCallback` => `WithDeletedLockCallback`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
5122be34e7 [#1461] shard: Add lock tests
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
eaf96bccf7 [#1461] meta: Add lock tests
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
c6cf8e5c0b [#1461] engine: Fix LOCK test
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
9c5ef3bab8 [#1461] node: Allow force LOCK removal
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
25826bd96d [#1461] node: Do not accept lock object w/o expiration
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
de31bb27b6 [#1461] cli: Require LOCK object to be expired
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
595fc60d72 [#1461] cli: Reorganize common functions
Make `ParseEpoch` and `GetCurrentEpoch` funcs public and move the first one
to the `common` package.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
fed9e6679d [#1461] node: Unlock locked object on its lock removal
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Evgenii Stratonikov
72c044e2eb [#1599] engine: Parallelize shard initialization
Shard is intended to be used as a separate failure domain,
which usually resides on a separate disk. Thus, sequential
initialization is bound by IO and this change speeds up thing a bit.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-15 13:44:54 +03:00
Evgenii Stratonikov
6eb5260562 [#1500] neofs-cli: Check if container contains LOCK objects before removal
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-15 13:25:56 +03:00
Pavel Karpy
a421344727 [#1605] node: Stream forwarder requests
Do not hold objects in memory in the forwarded Get/GetRange requests.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-15 11:16:35 +03:00
Pavel Karpy
89ebc278b8 [#1596] *: Remove .gitkeep files
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-14 15:10:57 +03:00
Evgenii Stratonikov
c4b86cf1f1 [#1592] Remove debug builds
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-14 10:39:51 +03:00
Evgenii Stratonikov
b8508585a5 [#1592] cmd: Ignore error when fetching a defined flag
There is an error only if the flag is not defined, such errors should be
caught during debugging.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-14 10:39:51 +03:00
Evgenii Stratonikov
f7d5045876 [#1592] neofs-lens: Add --version flag
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-14 10:39:51 +03:00
Evgenii Stratonikov
5569ff82ef [#1587] Do not print build time in version
This makes our build more reproducible.
Also print `Component` and `GoVersion`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-14 10:39:51 +03:00
Pavel Karpy
b3272e7cf1 [#1367] adm: Support homomorphic hashing config
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
4a316ceae9 [#1365] morph: Do not return errors if config key is missing
Return default values instead of casting errors in `HomomorphicHashDisabled`
method.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
9bd1951ca4 [#1365] adm: Add homomorphic hash disabling option
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
89118e9da0 [#1365] node: Calculate object homomorphic hash flexibly
Do not calculate and do not write homomorphic hash for containers that were
configured to store objects without hash.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
ae4740f99f [#1365] ir: Check homomorphic hash flexibly in audit
Do not perform that check if it was turned off for the container being
checked.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
8a61d33c65 [#1365] ir: Check homomorphic hash setting on ContainerPut
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
fa9c910648 [#1365] morph: Add HomomorphicHashDisabled config getter
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
ae86d2907c [#1365] cli: Calculate homomorphic hash flexibly
Do not use homomorphic hash in storage group for containers that have
`homomorphic_hashing_disabled` set to `true`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
68e583f143 [#1365] cli: Sync container with network config
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
49ba1ef1b6 [#1365] cli: Add SyncContainerSettings func to internal client
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Evgenii Stratonikov
1119014e53 [#1579] config: Move storage.default to shard.default
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-13 12:31:42 +03:00
Evgenii Stratonikov
0ab168a00a [#1564] .github: Remove assign-to-project action
It doesn't work now and needs to be done differently.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-13 11:35:48 +03:00
Evgenii Stratonikov
f4a3fc5f11 [#1594] neofs-adm: Support better completion for set-policy
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-13 11:00:35 +03:00
Pavel Karpy
466b9b4273 [#1593] adm: Support changing network config
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-12 21:03:43 +03:00
Pavel Karpy
1e034c8d48 [#1532] node: Support signature status codes
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-12 18:22:59 +03:00
Pavel Karpy
dac4a838fa [#1490] Update the CHANGELOG
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-12 17:35:46 +03:00
Pavel Karpy
96a24b5721 [#1490] cli: Require SG members to be unique
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-12 17:35:46 +03:00
Pavel Karpy
5d5d21e3d0 [#1490] node: Require SG members to be unique
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-12 17:35:46 +03:00
Evgenii Stratonikov
0504c3e0c6 [#1266] object/acl: Check bearer token container ID
If the container ID is not nil and not equal to the container ID in the
request, consider bearer token invalid.

See also nspcc-dev/neofs-api#207.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-12 12:25:02 +03:00
1111 changed files with 37210 additions and 16636 deletions

View file

@ -1,19 +1,19 @@
FROM golang:1.17 as builder
FROM golang:1.18 as builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository
WORKDIR /src
COPY . /src
RUN make bin/neofs-adm
RUN make bin/frostfs-adm
# Executable image
FROM alpine AS neofs-adm
FROM alpine AS frostfs-adm
RUN apk add --no-cache bash
WORKDIR /
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /src/bin/neofs-adm /bin/neofs-adm
COPY --from=builder /src/bin/frostfs-adm /bin/frostfs-adm
CMD ["neofs-adm"]
CMD ["frostfs-adm"]

View file

@ -1,19 +1,19 @@
FROM golang:1.17 as builder
FROM golang:1.18 as builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository
WORKDIR /src
COPY . /src
RUN make bin/neofs-cli
RUN make bin/frostfs-cli
# Executable image
FROM alpine AS neofs-cli
FROM alpine AS frostfs-cli
RUN apk add --no-cache bash
WORKDIR /
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /src/bin/neofs-cli /bin/neofs-cli
COPY --from=builder /src/bin/frostfs-cli /bin/frostfs-cli
CMD ["neofs-cli"]
CMD ["frostfs-cli"]

View file

@ -3,6 +3,6 @@ RUN apk add --no-cache bash ca-certificates
WORKDIR /
COPY bin/neofs-adm /bin/neofs-adm
COPY bin/frostfs-adm /bin/frostfs-adm
CMD ["neofs-adm"]
CMD ["frostfs-adm"]

View file

@ -3,6 +3,6 @@ RUN apk add --no-cache bash ca-certificates
WORKDIR /
COPY bin/neofs-cli /bin/neofs-cli
COPY bin/frostfs-cli /bin/frostfs-cli
CMD ["neofs-cli"]
CMD ["frostfs-cli"]

View file

@ -3,6 +3,6 @@ RUN apk add --no-cache bash ca-certificates
WORKDIR /
COPY bin/neofs-ir /bin/neofs-ir
COPY bin/frostfs-ir /bin/frostfs-ir
CMD ["neofs-ir"]
CMD ["frostfs-ir"]

View file

@ -3,6 +3,6 @@ RUN apk add --no-cache bash ca-certificates
WORKDIR /
COPY bin/neofs-node /bin/neofs-node
COPY bin/frostfs-node /bin/frostfs-node
CMD ["neofs-node"]
CMD ["frostfs-node"]

View file

@ -1,18 +1,18 @@
FROM golang:1.17 as builder
FROM golang:1.18 as builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository
WORKDIR /src
COPY . /src
RUN make bin/neofs-ir
RUN make bin/frostfs-ir
# Executable image
FROM alpine AS neofs-ir
FROM alpine AS frostfs-ir
RUN apk add --no-cache bash
WORKDIR /
COPY --from=builder /src/bin/neofs-ir /bin/neofs-ir
COPY --from=builder /src/bin/frostfs-ir /bin/frostfs-ir
CMD ["neofs-ir"]
CMD ["frostfs-ir"]

View file

@ -1,18 +1,18 @@
FROM golang:1.17 as builder
FROM golang:1.18 as builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository
WORKDIR /src
COPY . /src
RUN make bin/neofs-node
RUN make bin/frostfs-node
# Executable image
FROM alpine AS neofs-node
FROM alpine AS frostfs-node
RUN apk add --no-cache bash
WORKDIR /
COPY --from=builder /src/bin/neofs-node /bin/neofs-node
COPY --from=builder /src/bin/frostfs-node /bin/frostfs-node
CMD ["neofs-node"]
CMD ["frostfs-node"]

View file

@ -1,19 +1,19 @@
FROM golang:1.17 as builder
FROM golang:1.18 as builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository
WORKDIR /src
COPY . /src
RUN make bin/neofs-node
RUN make bin/frostfs-node
# Executable image
FROM alpine AS neofs-node
FROM alpine AS frostfs-node
RUN apk add --no-cache bash
WORKDIR /
COPY --from=builder /src/bin/neofs-node /bin/neofs-node
COPY --from=builder /src/bin/frostfs-node /bin/frostfs-node
COPY --from=builder /src/config/testnet/config.yml /config.yml
CMD ["neofs-node", "--config", "/config.yml"]
CMD ["frostfs-node", "--config", "/config.yml"]

1
.github/CODEOWNERS vendored Normal file
View file

@ -0,0 +1 @@
* @TrueCloudLab/storage-core @TrueCloudLab/committers

View file

@ -41,3 +41,10 @@ assignees: ''
* Version used:
* Server setup and configuration:
* Operating System and version (`uname -a`):
## Don't forget to add labels!
- component label (`frostfs-adm`, `frostfs-storage`, ...)
- `goodfirstissue`, `helpwanted` if needed
- does this issue belong to an epic?
- priority (`P0`-`P4`) if already triaged
- quarter label (`202XQY`) if possible

View file

@ -18,3 +18,11 @@ assignees: ''
## Additional context
<!-- Add any other context or screenshots about the feature request here. -->
## Don't forget to add labels!
- component label (`neofs-adm`, `neofs-storage`, ...)
- issue type (`enhancement`, `refactor`, ...)
- `goodfirstissue`, `helpwanted` if needed
- does this issue belong to an epic?
- priority (`P0`-`P4`) if already triaged
- quarter label (`202XQY`) if possible

189
.github/logo.svg vendored
View file

@ -1,129 +1,70 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<svg
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:cc="http://creativecommons.org/ns#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:svg="http://www.w3.org/2000/svg"
xmlns="http://www.w3.org/2000/svg"
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
sodipodi:docname="logo_fs.svg"
inkscape:version="1.0 (4035a4fb49, 2020-05-01)"
id="svg57"
version="1.1"
viewBox="0 0 105 25"
height="25mm"
width="105mm">
<defs
id="defs51">
<clipPath
clipPathUnits="userSpaceOnUse"
id="clipPath434">
<path
d="M 0,0 H 1366 V 768 H 0 Z"
id="path432" />
</clipPath>
</defs>
<sodipodi:namedview
inkscape:window-maximized="0"
inkscape:window-y="0"
inkscape:window-x="130"
inkscape:window-height="1040"
inkscape:window-width="1274"
height="50mm"
units="mm"
showgrid="false"
inkscape:document-rotation="0"
inkscape:current-layer="layer1"
inkscape:document-units="mm"
inkscape:cy="344.49897"
inkscape:cx="468.64708"
inkscape:zoom="0.7"
inkscape:pageshadow="2"
inkscape:pageopacity="0.0"
borderopacity="1.0"
bordercolor="#666666"
pagecolor="#ffffff"
id="base" />
<metadata
id="metadata54">
<rdf:RDF>
<cc:Work
rdf:about="">
<dc:format>image/svg+xml</dc:format>
<dc:type
rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
<dc:title></dc:title>
</cc:Work>
</rdf:RDF>
</metadata>
<g
id="layer1"
inkscape:groupmode="layer"
inkscape:label="Layer 1">
<g
id="g424"
transform="matrix(0.35277777,0,0,-0.35277777,63.946468,10.194047)">
<path
d="m 0,0 v -8.093 h 12.287 v -3.94 H 0 V -24.067 H -4.534 V 3.898 H 15.677 V 0 Z"
style="fill:#00e396;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path426" />
<?xml version="1.0" encoding="utf-8"?>
<!-- Generator: Adobe Illustrator 25.0.1, SVG Export Plug-In . SVG Version: 6.00 Build 0) -->
<svg version="1.1" id="Слой_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
viewBox="0 0 184.2 51.8" style="enable-background:new 0 0 184.2 51.8;" xml:space="preserve">
<style type="text/css">
.st0{display:none;}
.st1{display:inline;}
.st2{fill:#01E397;}
.st3{display:inline;fill:#010032;}
.st4{display:inline;fill:#00E599;}
.st5{display:inline;fill:#00AF92;}
.st6{fill:#00C3E5;}
</style>
<g id="Layer_2">
<g id="Layer_1-2" class="st0">
<g class="st1">
<path class="st2" d="M146.6,18.3v7.2h10.9V29h-10.9v10.7h-4V14.8h18v3.5H146.6z"/>
<path class="st2" d="M180,15.7c1.7,0.9,3,2.2,4,3.8l-3,2.7c-0.6-1.3-1.5-2.4-2.6-3.3c-1.3-0.7-2.8-1-4.3-1
c-1.4-0.1-2.8,0.3-4,1.1c-0.9,0.5-1.5,1.5-1.4,2.6c0,1,0.5,1.9,1.4,2.4c1.5,0.8,3.2,1.3,4.9,1.5c1.9,0.3,3.7,0.8,5.4,1.6
c1.2,0.5,2.2,1.3,2.9,2.3c0.6,1,1,2.2,0.9,3.4c0,1.4-0.5,2.7-1.3,3.8c-0.9,1.2-2.1,2.1-3.5,2.6c-1.7,0.6-3.4,0.9-5.2,0.8
c-5,0-8.6-1.6-10.7-5l2.9-2.8c0.7,1.4,1.8,2.5,3.1,3.3c1.5,0.7,3.1,1.1,4.7,1c1.5,0.1,2.9-0.2,4.2-0.9c0.9-0.5,1.5-1.5,1.5-2.6
c0-0.9-0.5-1.8-1.3-2.2c-1.5-0.7-3.1-1.2-4.8-1.5c-1.9-0.3-3.7-0.8-5.5-1.5c-1.2-0.5-2.2-1.4-3-2.4c-0.6-1-1-2.2-0.9-3.4
c0-1.4,0.4-2.7,1.2-3.8c0.8-1.2,2-2.2,3.3-2.8c1.6-0.7,3.4-1.1,5.2-1C176.1,14.3,178.2,14.8,180,15.7z"/>
</g>
<g
transform="matrix(0.35277777,0,0,-0.35277777,-315.43002,107.34005)"
id="g428">
<g
id="g430"
clip-path="url(#clipPath434)">
<g
id="g436"
transform="translate(1112.874,278.2981)">
<path
d="M 0,0 C 1.822,-0.932 3.354,-2.359 4.597,-4.28 L 1.165,-7.373 c -0.791,1.695 -1.779,2.924 -2.966,3.686 -1.186,0.763 -2.768,1.145 -4.745,1.145 -1.949,0 -3.461,-0.389 -4.534,-1.166 -1.074,-0.777 -1.61,-1.772 -1.61,-2.987 0,-1.13 0.523,-2.027 1.568,-2.69 1.045,-0.664 2.909,-1.236 5.593,-1.716 2.514,-0.452 4.512,-1.024 5.995,-1.716 1.483,-0.693 2.564,-1.554 3.242,-2.585 0.677,-1.031 1.016,-2.309 1.016,-3.834 0,-1.639 -0.466,-3.079 -1.398,-4.322 -0.932,-1.243 -2.239,-2.197 -3.919,-2.86 -1.681,-0.664 -3.623,-0.996 -5.826,-0.996 -5.678,0 -9.689,1.892 -12.033,5.678 l 3.178,3.178 c 0.903,-1.695 2.068,-2.939 3.495,-3.729 1.426,-0.791 3.199,-1.186 5.318,-1.186 2.005,0 3.58,0.345 4.724,1.038 1.144,0.692 1.716,1.674 1.716,2.945 0,1.017 -0.516,1.835 -1.547,2.457 -1.031,0.621 -2.832,1.172 -5.402,1.653 -2.571,0.479 -4.618,1.073 -6.143,1.779 -1.526,0.706 -2.635,1.582 -3.326,2.627 -0.693,1.045 -1.039,2.316 -1.039,3.813 0,1.582 0.438,3.023 1.314,4.322 0.875,1.299 2.14,2.33 3.792,3.093 1.653,0.763 3.58,1.144 5.783,1.144 C -4.018,1.398 -1.822,0.932 0,0"
style="fill:#00e396;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path438" />
<path class="st3" d="M73.3,16.3c1.9,1.9,2.9,4.5,2.7,7.1v15.9h-4V24.8c0-2.6-0.5-4.5-1.6-5.7c-1.2-1.2-2.8-1.8-4.5-1.7
c-1.3,0-2.5,0.3-3.7,0.8c-1.2,0.7-2.2,1.7-2.9,2.9c-0.8,1.5-1.1,3.2-1.1,4.9v13.3h-4V15.1l3.6,1.5v1.7c0.8-1.5,2.1-2.6,3.6-3.3
c1.5-0.8,3.2-1.2,4.9-1.1C68.9,13.8,71.3,14.7,73.3,16.3z"/>
<path class="st3" d="M104.4,28.3H85.6c0.1,2.2,1,4.3,2.5,5.9c1.5,1.4,3.5,2.2,5.6,2.1c1.6,0.1,3.2-0.2,4.6-0.9
c1.1-0.6,2-1.6,2.5-2.8l3.3,1.8c-0.9,1.7-2.3,3.1-4,4c-2,1-4.2,1.5-6.4,1.4c-3.7,0-6.7-1.1-8.8-3.4s-3.2-5.5-3.2-9.6s1-7.2,3-9.5
s5-3.4,8.7-3.4c2.1-0.1,4.2,0.5,6.1,1.5c1.6,1,3,2.5,3.8,4.2c0.9,1.8,1.3,3.9,1.3,5.9C104.6,26.4,104.6,27.4,104.4,28.3z
M88.1,19.3c-1.4,1.5-2.2,3.4-2.4,5.5h15.1c-0.2-2-1-3.9-2.3-5.5c-1.4-1.3-3.2-2-5.1-1.9C91.5,17.3,89.6,18,88.1,19.3z"/>
<path class="st3" d="M131,17.3c2.2,2.3,3.2,5.5,3.2,9.5s-1,7.3-3.2,9.6s-5.1,3.4-8.8,3.4s-6.7-1.1-8.9-3.4s-3.2-5.5-3.2-9.6
s1.1-7.2,3.2-9.5s5.1-3.4,8.9-3.4S128.9,15,131,17.3z M116.2,19.9c-1.5,2-2.2,4.4-2.1,6.9c-0.2,2.5,0.6,5,2.1,7
c1.5,1.7,3.7,2.7,6,2.6c2.3,0.1,4.4-0.9,5.9-2.6c1.5-2,2.3-4.5,2.1-7c0.1-2.5-0.6-4.9-2.1-6.9c-1.5-1.7-3.6-2.7-5.9-2.6
C119.9,17.2,117.7,18.2,116.2,19.9z"/>
<polygon class="st4" points="0,9.1 0,43.7 22.5,51.8 22.5,16.9 46.8,7.9 24.8,0 "/>
<polygon class="st5" points="24.3,17.9 24.3,36.8 46.8,44.9 46.8,9.6 "/>
</g>
<g
id="g440"
transform="translate(993.0239,277.5454)">
<path
d="m 0,0 c 2.054,-1.831 3.083,-4.465 3.083,-7.902 v -17.935 h -4.484 v 16.366 c 0,2.914 -0.626,5.024 -1.877,6.332 -1.253,1.308 -2.924,1.962 -5.016,1.962 -1.495,0 -2.896,-0.327 -4.204,-0.981 -1.308,-0.654 -2.381,-1.719 -3.222,-3.194 -0.841,-1.477 -1.261,-3.335 -1.261,-5.576 v -14.909 h -4.484 V 1.328 l 4.086,-1.674 0.118,-1.84 c 0.933,1.681 2.222,2.923 3.867,3.727 1.643,0.803 3.493,1.205 5.548,1.205 C -4.671,2.746 -2.055,1.83 0,0"
style="fill:#000033;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path442" />
</g>
<g
id="g444"
transform="translate(1027.9968,264.0386)">
<path
d="m 0,0 h -21.128 c 0.261,-2.84 1.205,-5.044 2.83,-6.613 1.625,-1.57 3.727,-2.355 6.305,-2.355 2.054,0 3.763,0.356 5.128,1.065 1.363,0.71 2.288,1.738 2.774,3.083 l 3.755,-1.961 c -1.121,-1.981 -2.616,-3.495 -4.484,-4.54 -1.868,-1.046 -4.259,-1.569 -7.173,-1.569 -4.223,0 -7.538,1.289 -9.948,3.867 -2.41,2.578 -3.615,6.146 -3.615,10.704 0,4.558 1.149,8.127 3.447,10.705 2.298,2.578 5.557,3.867 9.779,3.867 2.615,0 4.876,-0.58 6.782,-1.738 1.905,-1.158 3.343,-2.728 4.315,-4.707 C -0.262,7.827 0.224,5.605 0.224,3.139 0.224,2.092 0.149,1.046 0,0 m -18.298,10.144 c -1.513,-1.457 -2.438,-3.512 -2.775,-6.165 h 16.982 c -0.3,2.615 -1.159,4.661 -2.578,6.137 -1.42,1.476 -3.307,2.214 -5.661,2.214 -2.466,0 -4.455,-0.728 -5.968,-2.186"
style="fill:#000033;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path446" />
</g>
<g
id="g448"
transform="translate(1057.8818,276.4246)">
<path
d="m 0,0 c 2.41,-2.578 3.615,-6.147 3.615,-10.705 0,-4.558 -1.205,-8.126 -3.615,-10.704 -2.41,-2.578 -5.726,-3.867 -9.948,-3.867 -4.222,0 -7.537,1.289 -9.947,3.867 -2.41,2.578 -3.615,6.146 -3.615,10.704 0,4.558 1.205,8.127 3.615,10.705 2.41,2.578 5.725,3.867 9.947,3.867 C -5.726,3.867 -2.41,2.578 0,0 m -16.617,-2.858 c -1.607,-1.906 -2.41,-4.522 -2.41,-7.847 0,-3.326 0.803,-5.94 2.41,-7.846 1.607,-1.905 3.83,-2.858 6.669,-2.858 2.839,0 5.063,0.953 6.67,2.858 1.606,1.906 2.41,4.52 2.41,7.846 0,3.325 -0.804,5.941 -2.41,7.847 C -4.885,-0.953 -7.109,0 -9.948,0 c -2.839,0 -5.062,-0.953 -6.669,-2.858"
style="fill:#000033;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path450" />
</g>
</g>
</g>
<g
id="g452"
transform="matrix(0.35277777,0,0,-0.35277777,5.8329581,6.5590171)">
<path
d="m 0,0 0.001,-38.946 25.286,-9.076 V -8.753 L 52.626,1.321 27.815,10.207 Z"
style="fill:#00e599;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path454" />
</g>
<g
id="g456"
transform="matrix(0.35277777,0,0,-0.35277777,15.479008,10.041927)">
<path
d="M 0,0 V -21.306 L 25.293,-30.364 25.282,9.347 Z"
style="fill:#00b091;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path458" />
<g>
<g>
<path class="st6" d="M41.6,17.5H28.2v6.9h10.4v3.3H28.2v10.2h-3.9V14.2h17.2V17.5z"/>
<path class="st6" d="M45.8,37.9v-18h3.3l0.4,3.2c0.5-1.2,1.2-2.1,2.1-2.7c0.9-0.6,2.1-0.9,3.5-0.9c0.4,0,0.7,0,1.1,0.1
c0.4,0.1,0.7,0.2,0.9,0.3l-0.5,3.4c-0.3-0.1-0.6-0.2-0.9-0.2C55.4,23,54.9,23,54.4,23c-0.7,0-1.5,0.2-2.2,0.6
c-0.7,0.4-1.3,1-1.8,1.8s-0.7,1.8-0.7,3v9.5H45.8z"/>
<path class="st6" d="M68.6,19.6c1.8,0,3.3,0.4,4.6,1.1c1.3,0.7,2.4,1.8,3.1,3.2s1.1,3.1,1.1,5c0,1.9-0.4,3.6-1.1,5
c-0.8,1.4-1.8,2.5-3.1,3.2c-1.3,0.7-2.9,1.1-4.6,1.1s-3.3-0.4-4.6-1.1c-1.3-0.7-2.4-1.8-3.2-3.2c-0.8-1.4-1.2-3.1-1.2-5
c0-1.9,0.4-3.6,1.2-5s1.8-2.5,3.2-3.2C65.3,19.9,66.8,19.6,68.6,19.6z M68.6,22.6c-1.1,0-2,0.2-2.8,0.7c-0.8,0.5-1.3,1.2-1.7,2.1
s-0.6,2.1-0.6,3.5c0,1.3,0.2,2.5,0.6,3.4s1,1.7,1.7,2.2s1.7,0.7,2.8,0.7c1.1,0,2-0.2,2.7-0.7c0.7-0.5,1.3-1.2,1.7-2.2
s0.6-2.1,0.6-3.4c0-1.4-0.2-2.5-0.6-3.5s-1-1.6-1.7-2.1C70.6,22.8,69.6,22.6,68.6,22.6z"/>
<path class="st6" d="M89.2,38.3c-1.8,0-3.4-0.3-4.9-1c-1.5-0.7-2.7-1.7-3.5-3l2.7-2.3c0.5,1,1.3,1.8,2.3,2.4
c1,0.6,2.2,0.9,3.6,0.9c1.1,0,2-0.2,2.6-0.6c0.6-0.4,1-0.9,1-1.6c0-0.5-0.2-0.9-0.5-1.2s-0.9-0.6-1.7-0.8l-3.8-0.8
c-1.9-0.4-3.3-1-4.1-1.9c-0.8-0.9-1.2-1.9-1.2-3.3c0-1,0.3-1.9,0.9-2.7c0.6-0.8,1.4-1.5,2.5-2s2.5-0.8,4-0.8c1.8,0,3.3,0.3,4.6,1
c1.3,0.6,2.2,1.5,2.9,2.7l-2.7,2.2c-0.5-1-1.1-1.7-2-2.1c-0.9-0.5-1.8-0.7-2.8-0.7c-0.8,0-1.4,0.1-2,0.3c-0.6,0.2-1,0.5-1.3,0.8
c-0.3,0.3-0.4,0.7-0.4,1.2c0,0.5,0.2,0.9,0.5,1.3s1,0.6,1.9,0.8l4.1,0.9c1.7,0.3,2.9,0.9,3.7,1.7c0.7,0.8,1.1,1.8,1.1,2.9
c0,1.2-0.3,2.2-0.9,3c-0.6,0.9-1.5,1.6-2.6,2C92.1,38.1,90.7,38.3,89.2,38.3z"/>
<path class="st6" d="M112.8,19.9v3H99.3v-3H112.8z M106.6,14.6v17.9c0,0.9,0.2,1.5,0.7,1.9c0.5,0.4,1.1,0.6,1.9,0.6
c0.6,0,1.2-0.1,1.7-0.3c0.5-0.2,0.9-0.5,1.3-0.8l0.9,2.8c-0.6,0.5-1.2,0.9-2,1.1c-0.8,0.3-1.7,0.4-2.7,0.4c-1,0-2-0.2-2.8-0.5
s-1.5-0.9-2-1.6c-0.5-0.8-0.7-1.7-0.8-3V15.7L106.6,14.6z"/>
<path d="M137.9,17.5h-13.3v6.9h10.4v3.3h-10.4v10.2h-3.9V14.2h17.2V17.5z"/>
<path d="M150.9,13.8c2.1,0,4,0.4,5.5,1.2c1.6,0.8,2.9,2,4,3.5l-2.6,2.5c-0.9-1.4-1.9-2.4-3.1-3c-1.1-0.6-2.5-0.9-4-0.9
c-1.2,0-2.1,0.2-2.8,0.5c-0.7,0.3-1.3,0.7-1.6,1.2c-0.3,0.5-0.5,1.1-0.5,1.7c0,0.7,0.3,1.4,0.8,1.9c0.5,0.6,1.5,1,2.9,1.3
l4.8,1.1c2.3,0.5,3.9,1.3,4.9,2.3c1,1,1.4,2.3,1.4,3.9c0,1.5-0.4,2.7-1.2,3.8c-0.8,1.1-1.9,1.9-3.3,2.5s-3.1,0.9-5,0.9
c-1.7,0-3.2-0.2-4.5-0.6c-1.3-0.4-2.5-1-3.5-1.8c-1-0.7-1.8-1.6-2.5-2.6l2.7-2.7c0.5,0.8,1.1,1.6,1.9,2.2
c0.8,0.7,1.7,1.2,2.7,1.5c1,0.4,2.2,0.5,3.4,0.5c1.1,0,2.1-0.1,2.9-0.4c0.8-0.3,1.4-0.7,1.8-1.2c0.4-0.5,0.6-1.1,0.6-1.9
c0-0.7-0.2-1.3-0.7-1.8c-0.5-0.5-1.3-0.9-2.6-1.2l-5.2-1.2c-1.4-0.3-2.6-0.8-3.6-1.3c-0.9-0.6-1.6-1.3-2.1-2.1s-0.7-1.8-0.7-2.8
c0-1.3,0.4-2.6,1.1-3.7c0.7-1.1,1.8-2,3.2-2.6C147.3,14.1,148.9,13.8,150.9,13.8z"/>
</g>
</g>
</g>
</svg>

Before

Width:  |  Height:  |  Size: 6.5 KiB

After

Width:  |  Height:  |  Size: 5.5 KiB

29
.github/workflows/changelog.yml vendored Normal file
View file

@ -0,0 +1,29 @@
name: CHANGELOG check
on:
pull_request:
branches:
- master
- support/**
jobs:
build:
runs-on: ubuntu-latest
name: Check for updates
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Get changed CHANGELOG
id: changelog-diff
uses: tj-actions/changed-files@v29
with:
files: CHANGELOG.md
- name: Fail if changelog not updated
if: steps.changelog-diff.outputs.any_changed == 'false'
uses: actions/github-script@v3
with:
script: |
core.setFailed('CHANGELOG.md has not been updated')

37
.github/workflows/config-update.yml vendored Normal file
View file

@ -0,0 +1,37 @@
name: Configuration check
on:
pull_request:
branches:
- master
- support/**
jobs:
build:
runs-on: ubuntu-latest
name: config-check
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Get changed config-related files
id: config-diff
uses: tj-actions/changed-files@v29
with:
files: |
config/**
cmd/neofs-node/config/**
- name: Get changed doc files
id: docs-diff
uses: tj-actions/changed-files@v29
with:
files: docs/**
- name: Fail if config files are changed but the documentation is not updated
if: steps.config-diff.outputs.any_changed == 'true' && steps.docs-diff.outputs.any_changed == 'false'
uses: actions/github-script@v3
with:
script: |
core.setFailed('Documentation has not been updated')

View file

@ -4,6 +4,7 @@ on:
pull_request:
branches:
- master
- support/**
jobs:
commits_check_job:

View file

@ -1,14 +1,16 @@
name: neofs-node tests
name: frostfs-node tests
on:
push:
branches:
- master
- support/**
paths-ignore:
- '*.md'
pull_request:
branches:
- master
- support/**
paths-ignore:
- '*.md'
@ -17,18 +19,18 @@ jobs:
runs-on: ubuntu-20.04
strategy:
matrix:
go: [ '1.17.x', '1.18.x' ]
go: [ '1.18.x', '1.19.x' ]
steps:
- name: Setup go
uses: actions/setup-go@v2
uses: actions/setup-go@v3
with:
go-version: ${{ matrix.go }}
- name: Check out code
uses: actions/checkout@v2
uses: actions/checkout@v3
- name: Cache go mod
uses: actions/cache@v2
uses: actions/cache@v3
with:
path: ~/go/pkg/mod
key: ${{ runner.os }}-go-${{ matrix.go }}-${{ hashFiles('**/go.sum') }}
@ -46,12 +48,13 @@ jobs:
lint:
runs-on: ubuntu-20.04
steps:
- name: Check out code
uses: actions/checkout@v2
- name: golangci-lint
uses: golangci/golangci-lint-action@v2
- uses: actions/setup-go@v3
with:
version: v1.45.2
go-version: 1.19
- uses: actions/checkout@v3
- name: golangci-lint
uses: golangci/golangci-lint-action@v3
with:
version: v1.50.0
args: --timeout=5m
only-new-issues: true

View file

@ -1,19 +0,0 @@
name: Add issues to NeoFS Core project
on:
issues:
types:
- opened
- transferred
- labeled
jobs:
add-to-project:
name: Add issue to project
runs-on: ubuntu-latest
steps:
- uses: actions/add-to-project@main
with:
project-url: https://github.com/orgs/nspcc-dev/projects/1
github-token: ${{ secrets.GITHUB_TOKEN }}
labeled: triage

21
.gitignore vendored
View file

@ -28,3 +28,24 @@ testfile
# misc
.neofs-cli.yml
# debhelpers
debian/*debhelper*
# logfiles
debian/*.log
# .substvars
debian/*.substvars
# .bash-completion
debian/*.bash-completion
# Install folders and files
debian/frostfs-cli/
debian/frostfs-ir/
debian/files
debian/frostfs-storage/
debian/changelog
man/
debs/

View file

@ -34,16 +34,23 @@ linters:
# some default golangci-lint linters
- errcheck
- gosimple
- godot
- ineffassign
- staticcheck
- typecheck
- unused
# extra linters
- bidichk
- durationcheck
- exhaustive
- exportloopref
- gofmt
- whitespace
- goimports
- unused
- misspell
- predeclared
- reassign
- whitespace
disable-all: true
fast: false

View file

@ -1,19 +1,451 @@
# Changelog
Changelog for NeoFS Node
Changelog for FrostFS Node
## [Unreleased]
### Added
- Separate batching for replicated operations over the same container in pilorama (#1621)
- Doc for extended headers (#2128)
- New `frostfs_node_object_container_size` metric for tracking size of reqular objects in a container (#2116)
- New `frostfs_node_object_payload_size` metric for tracking size of reqular objects on a single shard (#1794)
- Add command `frostfs-adm morph netmap-candidates` (#1889)
- `object.delete.tombstone_lifetime` config parameter to set tombstone lifetime in the DELETE service (#2246)
- Reload config for pprof and metrics on SIGHUP in `neofs-node` (#1868)
- Multiple configs support (#44)
- Parameters `nns-name` and `nns-zone` for command `frostfs-cli container create` (#37)
### Changed
- Change `frostfs_node_engine_container_size` to counting sizes of logical objects
- `common.PrintVerbose` prints via `cobra.Command.Printf` (#1962)
- Env prefix in configuration changed to `FROSTFS_*` (#43)
- Link object is broadcast throughout the whole container now (#57)
- Pilorama now can merge multiple batches into one (#2231)
- Storage engine now can start even when some shard components are unavailable (#2238)
- `neofs-cli` buffer for object put increased from 4 KiB to 3 MiB (#2243)
- Expired locked object is available for reading (#56)
### Fixed
- Increase payload size metric on shards' `put` operation (#1794)
- Big object removal with non-local parts (#1978)
- Disable pilorama when moving to degraded mode (#2197)
- Fetching blobovnicza objects that not found in write-cache (#2206)
- Do not search for the small objects in FSTree (#2206)
- Correct status error for expired session token (#2207)
- Set flag `mode` required for `frostfs-cli control shards set-mode` (#8)
- Fix `dirty` suffix in debian package version (#53)
- Prevent node process from killing by systemd when shutting down (#1465)
- Restore subscriptions correctly on morph client switch (#2212)
- Expired objects could be returned if not marked with GC yet (#2213)
- `neofs-adm morph dump-hashes` now properly iterates over custom domain (#2224)
- Possible deadlock in write-cache (#2239)
- Fix `*_req_count` and `*_req_count_success` metric values (#2241)
- Storage ID update by write-cache (#2244)
- `neo-go` client deadlock on subscription restoration (#2244)
- Possible panic during write-cache initialization (#2234)
- Do not fetch an object if `meta` is missing it (#61)
- Create contract wallet only by `init` and `update-config` command (#63)
- Actually use `object.put.pool_size_local` and independent pool for local puts (#64).
### Removed
### Updated
- `neo-go` to `v0.100.1`
- `github.com/klauspost/compress` to `v1.15.13`
- `github.com/multiformats/go-multiaddr` to `v0.8.0`
- `golang.org/x/term` to `v0.3.0`
- `google.golang.org/grpc` to `v1.51.0`
- `github.com/nats-io/nats.go` to `v1.22.1`
- `github.com/TrueCloudLab/hrw` to `v.1.1.1`
- Minimum go version to v1.18
### Updating from v0.35.0
You need to change configuration environment variables to `FROSTFS_*` if you use any.
New config field `object.delete.tombstone_lifetime` allows to set tombstone lifetime
more appropriate for a specific deployment.
## [0.35.0] - 2022-12-28 - Sindo (신도, 信島)
### Added
- `morph list-containers` in `neofs-adm` (#1689)
- `--binary` flag in `neofs-cli object put/get/delete` commands (#1338)
- `session` flag support to `neofs-cli object hash` (#2029)
- Shard can now change mode when encountering background disk errors (#2035)
- Background workers and object service now use separate client caches (#2048)
- `replicator.pool_size` config field to tune replicator pool size (#2049)
- Fix NNS hash parsing in morph client (#2063)
- `neofs-cli neofs-cli acl basic/extended print` commands (#2012)
- `neofs_node_object_*_req_count_success` prometheus metrics for tracking successfully executed requests (#1984)
- Metric 'readonly' to get shards mode (#2022)
- Tree service replication timeout (#2159)
- `apiclient.reconnect_timeout` setting allowing to ignore failed clients for some time (#2164)
### Changed
- `object lock` command reads CID and OID the same way other commands do (#1971)
- `LOCK` object are stored on every container node (#1502)
- `neofs-cli container get-eacl` print ACL table in json format only with arg `--json' (#2012)
- Side chain notary deposits use max uint32 as till parameter (#1486)
- Allow object removal without linking object (#2100)
- `neofs-cli container delete` command pre-checks container ownership (#2106)
- Policer cache size is now 1024 (#2158)
- Tree service now synchronizes with container nodes in a random order (#2127)
- Pilorama no longer tries to apply already applied operations (#2161)
- Use `sync.Pool` in Object.PUT service (#2139)
- Shard uses metabase for `HEAD` requests by default, not write-cache (#2167)
- Clarify help for `--expire-at` parameter for commands `object lock/put` and `bearer create` (#2097)
- Node spawns `GETRANGE` requests signed with the node's key if session key was not found for `RANGEHASH` (#2144)
- Full list of container is no longer cached (#2176)
### Fixed
- Open FSTree in sync mode by default (#1992)
- `neofs-cli container nodes`'s output (#1991)
- Increase error counter for write-cache flush errors (#1818)
- Correctly select the shard for applying tree service operations (#1996)
- Do not panic and return correct errors for bad inputs in `GET_RANGE` (#2007, #2024)
- Physical child object removal by GC (#1699)
- Broadcasting helper objects (#1972)
- `neofs-cli lock object`'s `lifetime` flag handling (#1972)
- Do not move write-cache in read-only mode for flushing (#1906)
- Child object collection on CLI side with a bearer token (#2000)
- Fix concurrent map writes in `Object.Put` service (#2037)
- Malformed request errors' reasons in the responses (#2028)
- Session token's IAT and NBF checks in ACL service (#2028)
- Losing meta information on request forwarding (#2040)
- Assembly process triggered by a request with a bearer token (#2040)
- Losing locking context after metabase resync (#1502)
- Removing all trees by container ID if tree ID is empty in `pilorama.Forest.TreeDrop` (#1940)
- Concurrent mode changes in the metabase and blobstor (#2057)
- Panic in IR when performing HEAD requests (#2069)
- Write-cache flush duplication (#2074)
- Ignore error if a transaction already exists in a morph client (#2075)
- ObjectID signature output in the CLI (#2104)
- Pack arguments of `setPrice` invocation during contract update (#2078)
- `neofs-cli object hash` panic (#2079)
- Closing `neo-go` WS clients on shutdown and switch processes (#2080)
- Making notary deposits with a zero GAS balance (#2080)
- Notary requests on shutdown (#2075)
- `neofs-cli container create ` check the sufficiency of the number of nodes in the selector for replicas (#2038)
- Data duplication during request forwarding (#2047)
- Tree service panic on `TreeMove` operation (#2140)
- Panic in `GETRANGE` with zero length (#2095)
- Spawning useless `GETRANGE` with zero length for a big object (#2101)
- Incomplete object put errors do contain the deepest error's message (#2092)
- Prioritize internal addresses for clients (#2156)
- Force object removal via control service (#2145)
- Synchronizing a tree now longer reports an error for a single-node container (#2154)
- Prevent leaking goroutines in the tree service (#2162)
- Do not search for LOCK objects when delete container when session provided (#2152)
- Race conditions on shard's mode switch (#1956)
- Returning expired/removed objects from write-cache (#2016)
### Removed
- `-g` option from `neofs-cli control ...` and `neofs-cli container create` commands (#2089)
- `--header` from `neofs-cli object get` (#2090)
### Updated
- `neo-go` to `v0.100.0`
- `spf13/cobra` to `v1.6.1`
- `spf13/viper` to `v1.8.0`
- `google.golang.org/grpc` to `v1.50.1`
### Updating from v0.34.0
Pass CID and OID parameters via the `--cid` and `--oid` flags, not as the command arguments.
Replicator pool size can now be fine-tuned with `replicator.pool_size` config field.
The default value is taken from `object.put.pool_size_remote` as in earlier versions.
Added `neofs_node_object_*_req_count_success` metrics for tracking successfully executed requests.
`neofs-cli container delete` command now requires given account or session issuer
to match the container owner. Use `--force` (`-f`) flag to bypass this requirement.
Tree service network replication can now be fine-tuned with `tree.replication_timeout` config field.
## [0.34.0] - 2022-10-31 - Marado (마라도, 馬羅島)
### Added
- `--timeout` flag in `neofs-cli control` commands (#1917)
- Document shard modes of operation (#1909)
- `tree list` CLI command (#1332)
- `TreeService.GetTrees` RPC (#1902)
- All trees synchronization on bootstrap (#1902)
- `--force` flag to `neofs-cli control set-status` command (#1916)
- Logging `SessionService.Create` RPC on the server for debug (#1930)
- Debian packages can now be built with `make debpackage` (#409)
### Changed
- Path to a metabase can now be reloaded with a SIGHUP (#1869)
### Fixed
- `writecache.max_object_size` is now correctly handled (#1925)
- Correctly handle setting ONLINE netmap status after maintenance (#1922)
- Correctly reset shard errors in `ControlService.SetShardMode` RPC (#1931)
- Setting node's network state to `MAINTENANCE` while network settings forbid it (#1916)
- Do not panic during API client creation (#1936)
- Correctly sign new epoch transaction in neofs-adm for a committee of more than 4 nodes (#1949)
- Inability to provide session to NeoFS CLI in a NeoFS-binary format (#1933)
- `neofs-adm` now works correctly with a committee of more than 4 nodes (#1949, #1959)
- Closing a shard now waits until GC background workers stop (#1964)
- Make it possible to use `shard.ContainerSize` in read-only mode (#1975)
- Storage node now starts if at least one gRPC endpoint is available (#1893)
- Panic in API multy client (#1961)
- Blobstor object removal log messages (#1953)
- Missing object relatives in object removal session opened by NeoFS CLI (#1978)
- Bringing a node back online during maintenance (#1900)
### Updated
- `neo-go` to `v0.99.4`
- `protoc` to `v3.21.7`
- `neofs-sdk` to `v1.0.0-rc.7`
### Updating from v0.33.0
Now storage node serves Control API `SetNemapStatus` request with `MAINTENANCE`
status only if the mode is allowed in the network settings. To force starting the local
maintenance on the node, provide `--force` flag to the `neofs-cli control set-status`
command.
## [0.33.0] - 2022-10-17 - Anmado (안마도, 鞍馬島)
### Added
- Serving `NetmapService.NetmapSnapshot` RPC (#1793)
- `netmap snapshot` command of NeoFS CLI (#1793)
- `apiclient.allow_external` config flag to fallback to node external addresses (#1817)
- Support `MAINTENANCE` state of the storage nodes (#1680, #1681)
- Changelog updates CI step (#1808)
- Validate storage node configuration before node startup (#1805)
- `neofs-node -check` command to check the configuration file (#1805)
- `flush-cache` control service command to flush write-cache (#1806)
- `wallet-address` flag in `neofs-adm morph refill-gas` command (#1820)
- Validate policy before container creation (#1704)
- `--timeout` flag in `neofs-cli` subcommands (#1837)
- `container nodes` command to output list of nodes for container, grouped by replica (#1704)
- Configuration flag to ignore shard in `neofs-node` (#1840)
- Add new RPC `TreeService.Healthcheck`
- Fallback to `GET` if `GET_RANGE` from one storage nodes to another is denied by basic ACL (#1884)
- List of shards and logger level runtime reconfiguration (#1770)
- `neofs-adm morph set-config` now supports well-known `MaintenanceModeAllowed` key (#1892)
- `add`, `get-by-path` and `add-by-path` tree service CLI commands (#1332)
- Tree synchronisation on startup (#1329)
- Morph client returns to the highest priority endpoint after the switch (#1615)
### Changed
- Allow to evacuate shard data with `EvacuateShard` control RPC (#1800)
- Flush write-cache when moving shard to DEGRADED mode (#1825)
- Make `morph.cache_ttl` default value equal to morph block time (#1846)
- Policer marks nodes under maintenance as OK without requests (#1680)
- Unify help messages in CLI (#1854)
- `evacuate`, `set-mode` and `flush-cache` control subcommands now accept a list of shard ids (#1867)
- Reading `object` commands of NeoFS CLI don't open remote sessions (#1865)
- Use hex format to print storage node ID (#1765)
### Fixed
- Description of command `netmap nodeinfo` (#1821)
- Proper status for object.Delete if session token is missing (#1697)
- Fail startup if metabase has an old version (#1809)
- Storage nodes could enter the network with any state (#1796)
- Missing check of new state value in `ControlService.SetNetmapStatus` (#1797)
- Correlation of object session to request (#1420)
- Do not increase error counter in `engine.Inhume` if shard is read-only (#1839)
- `control drop-objects` can remove split objects (#1830)
- Node's status in `neofs-cli netmap nodeinfo` command (#1833)
- Child check in object assembly process of `ObjectService.Get` handler (#1878)
- Shard ID in the object counter metrics (#1863)
- Metabase migration from the first version (#1860)
### Removed
- Remove WIF and NEP2 support in `neofs-cli`'s --wallet flag (#1128)
- Remove --generate-key option in `neofs-cli container delete` (#1692)
- Serving `ControlService.NetmapSnapshot` RPC (#1793)
- `control netmap-snapshot` command of NeoFS CLI (#1793)
### Updated
## [0.29.0] - 2022-07-07
- `neofs-contract` to `v0.16.0`
- `neofs-api-go` to `v2.14.0`
### Updating from v0.32.0
Replace using the `control netmap-snapshot` command with `netmap snapshot` one in NeoFS CLI.
Node can now specify additional addresses in `ExternalAddr` attribute. To allow a node to dial
other nodes external address, use `apiclient.allow_external` config setting.
Add `--force` option to skip placement validity check for container creation.
Pass `maintenance` state to `neofs-cli control set-status` to enter maintenance mode.
If network allows maintenance state (*), it will be reflected in the network map.
Storage nodes under maintenance are not excluded from the network map, but don't
serve object operations. (*) can be fetched from network configuration via
`neofs-cli netmap netinfo` command.
To allow maintenance mode during neofs-adm deployments, set
`network.maintenance_mode_allowed` parameter in config.
When issuing an object session token for root (virtual, "big") objects,
additionally include all members of the split-chain. If session context
includes root object only, it is not spread to physical ("small") objects.
`neofs-node` configuration now supports `mode: disabled` flag for a shard.
This can be used to temporarily ignore shards without completely removing them
from the config file.
## [0.32.0] - 2022-09-14 - Pungdo (풍도, 楓島)
### Added
- Objects counter metric (#1712)
- `meta` subcommand to `neofs-lens` (#1714)
- Storage node metrics with global and per-shard object counters (#1658)
- Removal of trees on container removal (#1630)
- Logging new epoch events on storage node (#1763)
- Timeout for streaming RPC (#1746)
- `neofs-adm` is now able to dump hashes from a custom zone (#1748)
- Empty filename support in the Tree Service (#1698)
- Flag to `neofs-cli container list-objects` command for attribute printing (#1649)
### Changed
- `neofs-cli object put`'s object ID output has changed from "ID" to "OID" (#1296)
- `neofs-cli container set-eacl` command now pre-checks container ACL's extensibility (#1652)
- Access control in Tree service (#1628)
- Tree service doesn't restrict depth in `rpc GetSubTree` (#1753)
- `neofs-adm` registers contract hashes in both hex and string address formats (#1749)
- Container list cache synchronization with the Sidechain (#1632)
- Blobstor components are unified (#1584, #1686, #1523)
### Fixed
- Panic on write-cache's `Delete` operation (#1664)
- Payload duplication in `neofs-cli storagegroup put` (#1706)
- Contract calls in notary disabled environments (#1743)
- `Blobovnicza.Get` op now iterates over all size buckets (#1707)
- Object expiration time (#1670)
- Parser of the placement policy (#1775)
- Tree service timeout logs (#1759)
- Object flushing on writecache side (#1745)
- Active blobovniczas caching (#1691)
- `neofs-adm` TX waiting (#1738)
- `neofs-adm` registers contracts with a minimal GAS payment (#1683)
- Permissions of the file created by `neofs-cli` (#1719)
- `neofs-adm` creates TX with a high priority attribute (#1702)
- Storage node's restart after a hard reboot (#1647)
### Updated
- `neo-go` to `v0.99.2`
- `nspcc-dev/neofs-contract` to `v0.15.5`
- `prometheus/client_golang` to `v1.13.0`
- `google.golang.org/protobuf` to `v1.28.1`
### Updating from v0.31.0
Storage Node now collects object count prometheus metrics: `neofs_node_object_counter`.
Provide `--no-precheck` flag to `neofs-cli container set-eacl` for unconditional sending of a request
(previous default behavior).
## [0.31.0] - 2022-08-04 - Baengnyeongdo (백령도, 白翎島)
### Added
- `neofs-adm` allows deploying arbitrary contracts (#1629)
### Changed
- Priority order in the Morph client (#1648)
### Fixed
- Losing request context in eACL response checks (#1595)
- Do not return expired objects that have not been handled by the GC yet (#1634)
- Setting CID field in `neofs-cli acl extended create` (#1650)
- `neofs-ir` no longer hangs if it cannot bind to the control endpoint (#1643)
- Do not require `lifetime` flag in `session create` CLI command (#1655)
- Using deprecated gRPC options (#1644)
- Increasing metabase error counter on disabled pilorama (#1642)
- Deadlock in the morph client related to synchronous notification handling (#1653)
- Slow metabase `COMMON_PREFIX` search for empty prefix (#1656)
### Removed
- Deprecated `profiler` and `metrics` configuration sections (#1654)
### Updated
- `chzyer/realine` to `v1.5.1`
- `google/uuid` to `v1.3.0`
- `nats-io/nats.go` to `v1.16.0`
- `prometheus/client_golang` to `v1.12.2`
- `spf13/cast` to `v1.5.0`
- `spf13/viper` to `v1.12.0`
- `go.uber.org/zap` to `v1.21.0`
- `google.golang.org/grpc` to `v1.48.0`
### Updating from v0.30.0
1. Change `morph.endpoint.client` priority values using the following rule:
the higher the priority the lower the value (non-specified or `0` values are
interpreted as the highest priority -- `1`).
2. Deprecated `profiler` and `metrics` configuration sections are dropped,
use `pprof` and `prometheus` instead.
## [0.30.2] - 2022-08-01
### Added
- `EACL_NOT_FOUND` status code support (#1645).
## [0.30.1] - 2022-07-29
### Fixed
- `GetRange` operation now works correctly with objects stored in write-cache (#1638)
- Losing request context in eACL response checks (#1595)
- Wrong balance contract in innerring processor (#1636)
- `neofs-adm` now sets groups in manifest for all contracts properly (#1631)
### Updated
- `neo-go` to `v0.99.1`
- `neofs-sdk-go` to `v1.0.0-rc.6`
## [0.30.0] - 2022-07-22 - Saengildo (생일도, 生日島)
### Added
- Profiler and metrics services now should be enabled with a separate flag
- Experimental support for the tree-service, disabled by default (#1607)
- Homomorphic hashes calculation can be disabled across the whole network (#1365)
- Improve `neofs-adm` auto-completion (#1594)
### Changed
- Require SG members to be unique (#1490)
- `neofs-cli` now doesn't remove container with LOCK objects without `--force` flag (#1500)
- LOCK objects are now required to have an expiration epoch (#1461)
- `morph` sections in IR and storage node configuration now accept an address and a priority of an endpoint (#1609)
- Morph client now retries connecting to the failed endpoint too (#1609)
- Redirecting `GET` and `GETRANGE` requests now does not store full object copy in memory (#1605)
- `neofs-adm` now registers candidates during initialization in a single transaction (#1608)
### Fixed
- Invalid smart contract address in balance contract listener (#1636)
- Shard now can start in degraded mode if the metabase is unavailable (#1559)
- Shard can now be disabled completely on init failure (#1559)
- Storage group members are now required to be unique (#1490)
- Print shard ID in component logs (#1611)
### Updated
- `neofs-contract` to `v0.15.3`
- `neo-go` to the pre-release version
- `github.com/spf13/cobra` to v1.5.0
### Updating from v0.29.0
1. Change morph endpoints from simple string to a pair of `address` and `priority`. The second can be omitted.
For inner ring node this resides in `morph.endpoint.client` section,
for storage node -- in `morph.rpc_endpoint` section. See `config/example` for an example.
2. Move `storage.default` section to `storage.shard.default`.
3. Rename `metrics` and `profiler` sections to `prometheus` and `pprof` respectively, though old versions are supported.
In addition, these sections must now be explicitly enabled with `enabled: true` flag.
## [0.29.0] - 2022-07-07 - Yeonpyeongdo (연평도, 延坪島)
Support WalletConnect signature scheme.
@ -1138,8 +1570,15 @@ NeoFS-API v2.0 support and updated brand-new storage node application.
## [0.10.0] - 2020-07-10
First public review release.
[Unreleased]: https://github.com/nspcc-dev/neofs-node/compare/v0.29.0...master
[Unreleased]: https://github.com/nspcc-dev/neofs-node/compare/v0.35.0...master
[0.35.0]: https://github.com/nspcc-dev/neofs-node/compare/v0.34.0...v0.35.0
[0.34.0]: https://github.com/nspcc-dev/neofs-node/compare/v0.33.0...v0.34.0
[0.33.0]: https://github.com/nspcc-dev/neofs-node/compare/v0.32.0...v0.33.0
[0.32.0]: https://github.com/nspcc-dev/neofs-node/compare/v0.31.0...v0.32.0
[0.31.0]: https://github.com/nspcc-dev/neofs-node/compare/v0.30.2...v0.31.0
[0.30.2]: https://github.com/nspcc-dev/neofs-node/compare/v0.30.1...v0.30.2
[0.30.1]: https://github.com/nspcc-dev/neofs-node/compare/v0.30.0...v0.30.1
[0.30.0]: https://github.com/nspcc-dev/neofs-node/compare/v0.29.0...v0.30.0
[0.29.0]: https://github.com/nspcc-dev/neofs-node/compare/v0.28.3...v0.29.0
[0.28.3]: https://github.com/nspcc-dev/neofs-node/compare/v0.28.2...v0.28.3
[0.28.2]: https://github.com/nspcc-dev/neofs-node/compare/v0.28.1...v0.28.2

View file

@ -3,8 +3,8 @@
First, thank you for contributing! We love and encourage pull requests from
everyone. Please follow the guidelines:
- Check the open [issues](https://github.com/nspcc-dev/neofs-node/issues) and
[pull requests](https://github.com/nspcc-dev/neofs-node/pulls) for existing
- Check the open [issues](https://github.com/TrueCloudLab/frostfs-node/issues) and
[pull requests](https://github.com/TrueCloudLab/frostfs-node/pulls) for existing
discussions.
- Open an issue first, to discuss a new feature or enhancement.
@ -23,23 +23,23 @@ everyone. Please follow the guidelines:
## Development Workflow
Start by forking the `neofs-node` repository, make changes in a branch and then
Start by forking the `frostfs-node` repository, make changes in a branch and then
send a pull request. We encourage pull requests to discuss code changes. Here
are the steps in details:
### Set up your GitHub Repository
Fork [NeoFS node upstream](https://github.com/nspcc-dev/neofs-node/fork) source
Fork [FrostFS node upstream](https://github.com/TrueCloudLab/frostfs-node/fork) source
repository to your own personal repository. Copy the URL of your fork (you will
need it for the `git clone` command below).
```sh
$ git clone https://github.com/nspcc-dev/neofs-node
$ git clone https://github.com/TrueCloudLab/frostfs-node
```
### Set up git remote as ``upstream``
```sh
$ cd neofs-node
$ git remote add upstream https://github.com/nspcc-dev/neofs-node
$ cd frostfs-node
$ git remote add upstream https://github.com/TrueCloudLab/frostfs-node
$ git fetch upstream
$ git merge upstream/master
...
@ -79,7 +79,7 @@ Description
```
```
$ git commit -am '[#123] Add some feature'
$ git commit -sam '[#123] Add some feature'
```
### Push to the branch
@ -106,7 +106,8 @@ contributors".
To sign your work, just add a line like this at the end of your commit message:
```
Signed-off-by: Samii Sakisaka <samii@nspcc.ru>
Signed-off-by: Samii Sakisaka <samii@ivunojikan.co.jp>
```
This can easily be done with the `--signoff` option to `git commit`.

View file

@ -1,5 +1,7 @@
# Credits
FrostFS continues the development of NeoFS.
Initial NeoFS research and development (2018-2020) was done by
[NeoSPCC](https://nspcc.ru) team.

View file

@ -2,15 +2,13 @@
SHELL = bash
REPO ?= $(shell go list -m)
VERSION ?= $(shell git describe --tags --dirty --always 2>/dev/null || cat VERSION 2>/dev/null || echo "develop")
BUILD ?= $(shell date -u --iso=seconds)
DEBUG ?= false
VERSION ?= $(shell git describe --tags --dirty --match "v*" --always --abbrev=8 2>/dev/null || cat VERSION 2>/dev/null || echo "develop")
HUB_IMAGE ?= nspccdev/neofs
HUB_IMAGE ?= truecloudlab/frostfs
HUB_TAG ?= "$(shell echo ${VERSION} | sed 's/^v//')"
GO_VERSION ?= 1.17
LINT_VERSION ?= 1.46.2
GO_VERSION ?= 1.19
LINT_VERSION ?= 1.50.0
ARCH = amd64
BIN = bin
@ -18,13 +16,20 @@ RELEASE = release
DIRS = $(BIN) $(RELEASE)
# List of binaries to build.
CMDS = $(notdir $(basename $(wildcard cmd/*)))
CMDS = $(notdir $(basename $(wildcard cmd/frostfs-*)))
BINS = $(addprefix $(BIN)/, $(CMDS))
.PHONY: help all images dep clean fmts fmt imports test lint docker/lint prepare-release
# .deb package versioning
OS_RELEASE = $(shell lsb_release -cs)
PKG_VERSION ?= $(shell echo $(VERSION) | sed "s/^v//" | \
sed -E "s/(.*)-(g[a-fA-F0-9]{6,8})(.*)/\1\3~\2/" | \
sed "s/-/~/")-${OS_RELEASE}
.PHONY: help all images dep clean fmts fmt imports test lint docker/lint
prepare-release debpackage
# To build a specific binary, use it's name prefix with bin/ as a target
# For example `make bin/neofs-node` will build only storage node binary
# For example `make bin/frostfs-node` will build only storage node binary
# Just `make` will build all possible binaries
all: $(DIRS) $(BINS)
@ -35,9 +40,7 @@ $(BINS): $(DIRS) dep
@echo "⇒ Build $@"
CGO_ENABLED=0 \
go build -v -trimpath \
-ldflags "-X $(REPO)/misc.Version=$(VERSION) \
-X $(REPO)/misc.Build=$(BUILD) \
-X $(REPO)/misc.Debug=$(DEBUG)" \
-ldflags "-X $(REPO)/misc.Version=$(VERSION)" \
-o $@ ./cmd/$(notdir $@)
$(DIRS):
@ -47,7 +50,7 @@ $(DIRS):
# Prepare binaries and archives for release
.ONESHELL:
prepare-release: docker/all
@for file in `ls -1 $(BIN)/neofs-*`; do
@for file in `ls -1 $(BIN)/frostfs-*`; do
cp $$file $(RELEASE)/`basename $$file`-$(ARCH)
strip $(RELEASE)/`basename $$file`-$(ARCH)
tar -czf $(RELEASE)/`basename $$file`-$(ARCH).tar.gz $(RELEASE)/`basename $$file`-$(ARCH)
@ -64,26 +67,26 @@ dep:
# Regenerate proto files:
protoc:
@GOPRIVATE=github.com/nspcc-dev go mod vendor
@GOPRIVATE=github.com/TrueCloudLab go mod vendor
# Install specific version for protobuf lib
@go list -f '{{.Path}}/...@{{.Version}}' -m github.com/golang/protobuf | xargs go install -v
@GOBIN=$(abspath $(BIN)) go install -mod=mod -v github.com/nspcc-dev/neofs-api-go/v2/util/protogen
@GOBIN=$(abspath $(BIN)) go install -mod=mod -v github.com/TrueCloudLab/frostfs-api-go/v2/util/protogen
# Protoc generate
@for f in `find . -type f -name '*.proto' -not -path './vendor/*'`; do \
echo "⇒ Processing $$f "; \
protoc \
--proto_path=.:./vendor:/usr/local/include \
--plugin=protoc-gen-go-neofs=$(BIN)/protogen \
--go-neofs_out=. --go-neofs_opt=paths=source_relative \
--plugin=protoc-gen-go-frostfs=$(BIN)/protogen \
--go-frostfs_out=. --go-frostfs_opt=paths=source_relative \
--go_out=. --go_opt=paths=source_relative \
--go-grpc_opt=require_unimplemented_servers=false \
--go-grpc_out=. --go-grpc_opt=paths=source_relative $$f; \
done
rm -rf vendor
# Build NeoFS component's docker image
# Build FrostFS component's docker image
image-%:
@echo "⇒ Build NeoFS $* docker image "
@echo "⇒ Build FrostFS $* docker image "
@docker build \
--build-arg REPO=$(REPO) \
--build-arg VERSION=$(VERSION) \
@ -146,3 +149,15 @@ clean:
rm -rf .cache
rm -rf $(BIN)
rm -rf $(RELEASE)
# Package for Debian
debpackage:
dch -b --package frostfs-node \
--controlmaint \
--newversion $(PKG_VERSION) \
--distribution $(OS_RELEASE) \
"Please see CHANGELOG.md for code changes for $(VERSION)"
dpkg-buildpackage --no-sign -b
debclean:
dh clean

View file

@ -1,39 +1,40 @@
<p align="center">
<img src="./.github/logo.svg" width="500px" alt="NeoFS">
<img src="./.github/logo.svg" width="500px" alt="FrostFS">
</p>
<p align="center">
<a href="https://fs.neo.org">NeoFS</a> is a decentralized distributed object storage integrated with the <a href="https://neo.org">NEO Blockchain</a>.
<a href="https://frostfs.info">FrostFS</a> is a decentralized distributed object storage integrated with the <a href="https://neo.org">NEO Blockchain</a>.
</p>
---
[![Report](https://goreportcard.com/badge/github.com/nspcc-dev/neofs-node)](https://goreportcard.com/report/github.com/nspcc-dev/neofs-node)
![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/nspcc-dev/neofs-node?sort=semver)
![License](https://img.shields.io/github/license/nspcc-dev/neofs-node.svg?style=popout)
[![Report](https://goreportcard.com/badge/github.com/TrueCloudLab/frostfs-node)](https://goreportcard.com/report/github.com/TrueCloudLab/frostfs-node)
![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/TrueCloudLab/frostfs-node?sort=semver)
![License](https://img.shields.io/github/license/TrueCloudLab/frostfs-node.svg?style=popout)
# Overview
NeoFS Nodes are organized in a peer-to-peer network that takes care of storing
FrostFS Nodes are organized in a peer-to-peer network that takes care of storing
and distributing user's data. Any Neo user may participate in the network and
get paid for providing storage resources to other users or store their data in
NeoFS and pay a competitive price for it.
FrostFS and pay a competitive price for it.
Users can reliably store object data in the NeoFS network and have a transparent
Users can reliably store object data in the FrostFS network and have a transparent
data placement process due to a decentralized architecture and flexible storage
policies. Each node is responsible for executing the storage policies that the
users select for geographical location, reliability level, number of nodes, type
of disks, capacity, etc. Thus, NeoFS gives full control over data to users.
of disks, capacity, etc. Thus, FrostFS gives full control over data to users.
Deep [Neo Blockchain](https://neo.org) integration allows NeoFS to be used by
Deep [Neo Blockchain](https://neo.org) integration allows FrostFS to be used by
dApps directly from
[NeoVM](https://docs.neo.org/docs/en-us/basic/technology/neovm.html) on the
[Smart Contract](https://docs.neo.org/docs/en-us/intro/glossary.html)
code level. This way dApps are not limited to on-chain storage and can
manipulate large amounts of data without paying a prohibitive price.
NeoFS has a native [gRPC API](https://github.com/nspcc-dev/neofs-api) and has
FrostFS has a native [gRPC API](https://github.com/TrueCloudLab/frostfs-api) and has
protocol gateways for popular protocols such as [AWS
S3](https://github.com/nspcc-dev/neofs-s3-gw),
[HTTP](https://github.com/nspcc-dev/neofs-http-gw),
S3](https://github.com/TrueCloudLab/frostfs-s3-gw),
[HTTP](https://github.com/TrueCloudLab/frostfs-http-gw),
[FUSE](https://wikipedia.org/wiki/Filesystem_in_Userspace) and
[sFTP](https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol) allowing
developers to integrate applications without rewriting their code.
@ -43,12 +44,12 @@ developers to integrate applications without rewriting their code.
Now, we only support GNU/Linux on amd64 CPUs with AVX/AVX2 instructions. More
platforms will be officially supported after release `1.0`.
The latest version of neofs-node works with neofs-contract
[v0.13.0](https://github.com/nspcc-dev/neofs-contract/releases/tag/v0.13.0).
The latest version of frostfs-node works with frostfs-contract
[v0.16.0](https://github.com/TrueCloudLab/frostfs-contract/releases/tag/v0.16.0).
# Building
To make all binaries you need Go 1.17+ and `make`:
To make all binaries you need Go 1.18+ and `make`:
```
make all
```
@ -56,7 +57,7 @@ The resulting binaries will appear in `bin/` folder.
To make a specific binary use:
```
make bin/neofs-<name>
make bin/frostfs-<name>
```
See the list of all available commands in the `cmd` folder.
@ -65,12 +66,12 @@ See the list of all available commands in the `cmd` folder.
Building can also be performed in a container:
```
make docker/all # build all binaries
make docker/bin/neofs-<name> # build a specific binary
make docker/bin/frostfs-<name> # build a specific binary
```
## Docker images
To make docker images suitable for use in [neofs-dev-env](https://github.com/nspcc-dev/neofs-dev-env/) use:
To make docker images suitable for use in [frostfs-dev-env](https://github.com/TrueCloudLab/frostfs-dev-env/) use:
```
make images
```
@ -85,7 +86,7 @@ the feature/topic you are going to implement.
# Credits
NeoFS is maintained by [NeoSPCC](https://nspcc.ru) with the help and
FrostFS is maintained by [True Cloud Lab](https://github.com/TrueCloudLab/) with the help and
contributions from community members.
Please see [CREDITS](CREDITS.md) for details.

View file

@ -1 +1 @@
v0.28.3
v0.35.0

View file

@ -1,24 +1,24 @@
# NeoFS Admin Tool
# FrostFS Admin Tool
## Overview
Admin tool provides an easier way to deploy and maintain private installation
of NeoFS. Private installation provides a set of N3 consensus nodes, NeoFS
of FrostFS. Private installation provides a set of N3 consensus nodes, FrostFS
Alphabet, and Storage nodes. Admin tool generates consensus keys, initializes
the sidechain, and provides functions to update the network and register new
Storage nodes.
## Build
To build binary locally, use `make bin/neofs-adm` command.
To build binary locally, use `make bin/frostfs-adm` command.
For clean build inside a docker container, use `make docker/bin/neofs-adm`.
For clean build inside a docker container, use `make docker/bin/frostfs-adm`.
Build docker image with `make image-adm`.
At NeoFS private install deployment, neofs-adm requires compiled NeoFS
At FrostFS private install deployment, frostfs-adm requires compiled FrostFS
contracts. Find them in the latest release of
[neofs-contract repository](https://github.com/nspcc-dev/neofs-contract/releases).
[frostfs-contract repository](https://github.com/TrueCloudLab/frostfs-contract/releases).
## Commands
@ -35,8 +35,8 @@ Config example:
rpc-endpoint: https://address:port # sidechain RPC node endpoint
alphabet-wallets: /path # path to consensus node / alphabet wallets storage
network:
max_object_size: 67108864 # max size of a single NeoFS object, bytes
epoch_duration: 240 # duration of a NeoFS epoch in blocks, consider block generation frequency in the sidechain
max_object_size: 67108864 # max size of a single FrostFS object, bytes
epoch_duration: 240 # duration of a FrostFS epoch in blocks, consider block generation frequency in the sidechain
basic_income_rate: 0 # basic income rate, for private consider 0
fee:
audit: 0 # network audit fee, for private installation consider 0
@ -62,16 +62,18 @@ credentials: # passwords for consensus node / alphabet wallets
Alphabet nodes.
- `init` initializes the sidechain by deploying smart contracts and
setting provided NeoFS network configuration.
setting provided FrostFS network configuration.
- `generate-storage-wallet` generates a wallet for the Storage node that
is ready for deployment. It also transfers a bit of sidechain GAS, so this
wallet can be used for NeoFS bootstrap.
wallet can be used for FrostFS bootstrap.
#### Network maintenance
- `force-new-epoch` increments NeoFS epoch number and executes new epoch
handlers in NeoFS nodes.
- `set-config` add/update configuration values in the Netmap contract.
- `force-new-epoch` increments FrostFS epoch number and executes new epoch
handlers in FrostFS nodes.
- `refill-gas` transfers sidechain GAS to the specified wallet.
@ -88,11 +90,13 @@ info. These commands **do not migrate actual objects**.
- `restore-containers` restores previously saved containers by their repeated registration in
the container contract.
- `list-containers` output all containers ids.
#### Network info
- `dump-config` prints NeoFS network configuration.
- `dump-config` prints FrostFS network configuration.
- `dump-hashes` prints NeoFS contract addresses stored in NNS.
- `dump-hashes` prints FrostFS contract addresses stored in NNS.
## Private network deployment

View file

@ -1,16 +1,16 @@
# Step-by-step private NeoFS deployment
# Step-by-step private FrostFS deployment
This is a short guide on how to deploy a private NeoFS storage network on bare
This is a short guide on how to deploy a private FrostFS storage network on bare
metal without docker images. This guide does not cover details on how to start
consensus, Alphabet, or Storage nodes. This guide covers only `neofs-adm`
consensus, Alphabet, or Storage nodes. This guide covers only `frostfs-adm`
related configuration details.
## Prerequisites
To follow this guide you need:
- latest released version of [neo-go](https://github.com/nspcc-dev/neo-go/releases) (v0.97.2 at the moment),
- latest released version of [neofs-adm](https://github.com/nspcc-dev/neofs-node/releases) utility (v0.25.1 at the moment),
- latest released version of compiled [neofs-contract](https://github.com/nspcc-dev/neofs-contract/releases) (v0.11.0 at the moment).
- latest released version of [frostfs-adm](https://github.com/TrueCloudLab/frostfs-node/releases) utility (v0.25.1 at the moment),
- latest released version of compiled [frostfs-contract](https://github.com/TrueCloudLab/frostfs-contract/releases) (v0.11.0 at the moment).
## Step 1: Prepare network configuration
@ -19,12 +19,12 @@ Alphabet nodes and any number of Storage nodes. While the number of Storage
nodes can be scaled almost infinitely, the number of consensus and Alphabet
nodes can't be changed so easily right now. Consider this before going any further.
It is easier to use`neofs-adm` with a predefined configuration. First, create
It is easier to use`frostfs-adm` with a predefined configuration. First, create
a network configuration file. In this example, there is going to be only one
consensus / Alphabet node in the network.
```
$ neofs-adm config init --path foo.network.yml
$ frostfs-adm config init --path foo.network.yml
Initial config file saved to foo.network.yml
$ cat foo.network.yml
@ -57,7 +57,7 @@ wallets will be used for Alphabet nodes. Make sure, that dir for alphabet
wallets already exists.
```
$ neofs-adm -c foo.network.yml morph generate-alphabet --size 1
$ frostfs-adm -c foo.network.yml morph generate-alphabet --size 1
size: 1
alphabet-wallets: /home/user/deploy/alphabet-wallets
wallet[0]: hunter2
@ -118,19 +118,19 @@ and possible overload issues.
## Step 3: Initialize sidechain
Use archive with compiled NeoFS contracts to initialize the sidechain.
Use archive with compiled FrostFS contracts to initialize the sidechain.
```
$ tar -xzvf neofs-contract-v0.11.0.tar.gz
$ tar -xzvf frostfs-contract-v0.11.0.tar.gz
$ ./neofs-adm -c foo.network.yml morph init --contracts ./neofs-contract-v0.11.0
$ ./frostfs-adm -c foo.network.yml morph init --contracts ./frostfs-contract-v0.11.0
Stage 1: transfer GAS to alphabet nodes.
Waiting for transactions to persist...
Stage 2: set notary and alphabet nodes in designate contract.
Waiting for transactions to persist...
Stage 3: deploy NNS contract.
Waiting for transactions to persist...
Stage 4: deploy NeoFS contracts.
Stage 4: deploy FrostFS contracts.
Waiting for transactions to persist...
Stage 4.1: Transfer GAS to proxy contract.
Waiting for transactions to persist...
@ -140,14 +140,14 @@ Stage 6: transfer NEO to alphabet contracts.
Waiting for transactions to persist...
Stage 7: set addresses in NNS.
Waiting for transactions to persist...
NNS: Set alphabet0.neofs -> f692dfb4d43a15b464eb51a7041160fb29c44b6a
NNS: Set audit.neofs -> 7df847b993affb3852074345a7c2bd622171ee0d
NNS: Set balance.neofs -> 103519b3067a66307080a66570c0491ee8f68879
NNS: Set container.neofs -> cae60bdd689d185901e495352d0247752ce50846
NNS: Set neofsid.neofs -> c421fb60a3895865a8f24d197d6a80ef686041d2
NNS: Set netmap.neofs -> 894eb854632f50fb124412ce7951ebc00763525e
NNS: Set proxy.neofs -> ac6e6fe4b373d0ca0ca4969d1e58fa0988724e7d
NNS: Set reputation.neofs -> 6eda57c9d93d990573646762d1fea327ce41191f
NNS: Set alphabet0.frostfs -> f692dfb4d43a15b464eb51a7041160fb29c44b6a
NNS: Set audit.frostfs -> 7df847b993affb3852074345a7c2bd622171ee0d
NNS: Set balance.frostfs -> 103519b3067a66307080a66570c0491ee8f68879
NNS: Set container.frostfs -> cae60bdd689d185901e495352d0247752ce50846
NNS: Set frostfsid.frostfs -> c421fb60a3895865a8f24d197d6a80ef686041d2
NNS: Set netmap.frostfs -> 894eb854632f50fb124412ce7951ebc00763525e
NNS: Set proxy.frostfs -> ac6e6fe4b373d0ca0ca4969d1e58fa0988724e7d
NNS: Set reputation.frostfs -> 6eda57c9d93d990573646762d1fea327ce41191f
Waiting for transactions to persist...
```
@ -177,7 +177,7 @@ contracts:
Generate a new wallet for a Storage node.
```
$ neofs-adm -c foo.network.yml morph generate-storage-wallet --storage-wallet ./sn01.json --initial-gas 10.0
$ frostfs-adm -c foo.network.yml morph generate-storage-wallet --storage-wallet ./sn01.json --initial-gas 10.0
New password >
Waiting for transactions to persist...
@ -196,16 +196,16 @@ node:
password: "foobar"
```
The storage node will be included in the network map in the next NeoFS epoch. To
The storage node will be included in the network map in the next FrostFS epoch. To
speed up this process, you can increment epoch counter immediately.
```
$ neofs-adm -c foo.network.yml morph force-new-epoch
$ frostfs-adm -c foo.network.yml morph force-new-epoch
Current epoch: 8, increase to 9.
Waiting for transactions to persist...
```
---
After that, NeoFS Storage is ready to work. You can access it directly or
After that, FrostFS Storage is ready to work. You can access it directly or
with protocol gates.

View file

@ -1,6 +1,6 @@
# NeoFS subnetwork creation
# FrostFS subnetwork creation
This is a short guide on how to create NeoFS subnetworks. This guide
This is a short guide on how to create FrostFS subnetworks. This guide
considers that the sidechain and the inner ring (alphabet nodes) have already been
deployed and the sidechain contains a deployed `subnet` contract.
@ -8,13 +8,13 @@ deployed and the sidechain contains a deployed `subnet` contract.
To follow this guide, you need:
- neo-go sidechain RPC endpoint;
- latest released version of [neofs-adm](https://github.com/nspcc-dev/neofs-node/releases);
- wallet with NeoFS account.
- latest released version of [frostfs-adm](https://github.com/TrueCloudLab/frostfs-node/releases);
- wallet with FrostFS account.
## Creation
```shell
$ neofs-adm morph subnet create \
$ frostfs-adm morph subnet create \
-r <side_chain_RPC_endpoint> \
-w </path/to/owner/wallet> \
--notary
@ -31,7 +31,7 @@ of the just created subnetwork.
You can check if your subnetwork was created successfully:
```shell
$ neofs-adm morph subnet get \
$ frostfs-adm morph subnet get \
-r <side_chain_RPC_endpoint> \
--subnet <subnet_ID>
Owner: NUc734PMJXiqa2J9jRtvskU3kCdyyuSN8Q

View file

@ -1,18 +1,18 @@
# Managing Subnetworks
This is a short guide on how to manage NeoFS subnetworks. This guide
This is a short guide on how to manage FrostFS subnetworks. This guide
considers that the sidechain and the inner ring (alphabet nodes) have already been
deployed, and the sidechain contains a deployed `subnet` contract.
## Prerequisites
- neo-go sidechain RPC endpoint;
- latest released version of [neofs-adm](https://github.com/nspcc-dev/neofs-node/releases);
- latest released version of [frostfs-adm](https://github.com/TrueCloudLab/frostfs-node/releases);
- [created](subnetwork-creation.md) subnetwork;
- wallet with the account that owns the subnetwork;
- public key of the Storage Node;
- public keys of the node and client administrators;
- owner IDs of the NeoFS users.
- owner IDs of the FrostFS users.
## Add node administrator
@ -21,7 +21,7 @@ the whitelist of the nodes which can be included to a subnetwork. Only the subne
owner is allowed to add and remove node administrators from the subnetwork.
```shell
$ neofs-adm morph subnet admin add \
$ frostfs-adm morph subnet admin add \
-r <side_chain_RPC_endpoint> \
-w </path/to/owner/wallet> \
--admin <HEX_admin_public_key> \
@ -37,7 +37,7 @@ the list of the allowed nodes. Node is not required to be bootstrapped at the
moment of its inclusion.
```shell
$ neofs-adm morph subnet node add \
$ frostfs-adm morph subnet node add \
-r <side_chain_RPC_endpoint> \
-w </path/to/node_admin/wallet> \
--node <HEX_node_public_key> \
@ -55,7 +55,7 @@ subnetwork. Only the subnet owner is allowed to add and remove client
administrators from the subnetwork.
```shell
$ neofs-adm morph subnet admin add \
$ frostfs-adm morph subnet admin add \
-r <side_chain_RPC_endpoint> \
-w </path/to/owner/wallet> \
--admin <HEX_admin_public_key> \
@ -72,7 +72,7 @@ positive integer number.
## Add client
```shell
$ neofs-adm morph subnet client add \
$ frostfs-adm morph subnet client add \
-r <side_chain_RPC_endpoint> \
-w </path/to/client_admin/wallet> \
--client <client_ownerID> \
@ -99,7 +99,7 @@ configuration:
node:
...
subnet:
entries: # list of IDs of subnets to enter in a text format of NeoFS API protocol (overrides corresponding attributes)
entries: # list of IDs of subnets to enter in a text format of FrostFS API protocol (overrides corresponding attributes)
- <subnetwork_ID>
...
...
@ -129,9 +129,9 @@ To create a container in a private network, your wallet must be added to
the client whitelist by the client admins or the subnet owners:
```shell
$ neofs-cli container create \
$ frostfs-cli container create \
--policy 'REP 1' \
-w </path/to/wallet> \
-r s01.neofs.devenv:8080 \
-r s01.frostfs.devenv:8080 \
--subnet <subnet_ID>
```

View file

@ -0,0 +1,14 @@
package commonflags
const (
ConfigFlag = "config"
ConfigFlagShorthand = "c"
ConfigFlagUsage = "Config file"
ConfigDirFlag = "config-dir"
ConfigDirFlagUsage = "Config directory"
Verbose = "verbose"
VerboseShorthand = "v"
VerboseUsage = "Verbose output"
)

View file

@ -7,8 +7,8 @@ import (
"path/filepath"
"text/template"
"github.com/TrueCloudLab/frostfs-node/pkg/innerring"
"github.com/nspcc-dev/neo-go/cli/input"
"github.com/nspcc-dev/neofs-node/pkg/innerring"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
@ -25,6 +25,7 @@ type configTemplate struct {
ContainerAliasFee int
WithdrawFee int
Glagolitics []string
HomomorphicHashDisabled bool
}
const configTxtTemplate = `rpc-endpoint: {{ .Endpoint}}
@ -33,13 +34,14 @@ network:
max_object_size: {{ .MaxObjectSize}}
epoch_duration: {{ .EpochDuration}}
basic_income_rate: {{ .BasicIncomeRate}}
homomorphic_hash_disabled: {{ .HomomorphicHashDisabled}}
fee:
audit: {{ .AuditFee}}
candidate: {{ .CandidateFee}}
container: {{ .ContainerFee}}
container_alias: {{ .ContainerAliasFee }}
withdraw: {{ .WithdrawFee}}
# if credentials section is omitted, then neofs-adm will require manual password input
# if credentials section is omitted, then frostfs-adm will require manual password input
credentials:
contract: password # wallet for contract group signature{{ range.Glagolitics}}
{{.}}: password{{end}}
@ -97,7 +99,7 @@ func defaultConfigPath() (string, error) {
return "", fmt.Errorf("getting home dir path: %w", err)
}
return filepath.Join(home, ".neofs", "adm", "config.yml"), nil
return filepath.Join(home, ".frostfs", "adm", "config.yml"), nil
}
// generateConfigExample builds .yml representation of the config file. It is
@ -110,6 +112,7 @@ func generateConfigExample(appDir string, credSize int) (string, error) {
MaxObjectSize: 67108864, // 64 MiB
EpochDuration: 240, // 1 hour with 15s per block
BasicIncomeRate: 1_0000_0000, // 0.0001 GAS per GiB (Fixed12)
HomomorphicHashDisabled: false, // object homomorphic hash is enabled
AuditFee: 1_0000, // 0.00000001 GAS per audit (Fixed12)
CandidateFee: 100_0000_0000, // 100.0 GAS (Fixed8)
ContainerFee: 1000, // 0.000000001 * 7 GAS per container (Fixed12)

View file

@ -5,7 +5,7 @@ import (
"path/filepath"
"testing"
"github.com/nspcc-dev/neofs-node/pkg/innerring"
"github.com/TrueCloudLab/frostfs-node/pkg/innerring"
"github.com/spf13/viper"
"github.com/stretchr/testify/require"
)
@ -13,7 +13,7 @@ import (
func TestGenerateConfigExample(t *testing.T) {
const (
n = 10
appDir = "/home/example/.neofs"
appDir = "/home/example/.frostfs"
)
configText, err := generateConfigExample(appDir, n)

View file

@ -0,0 +1,29 @@
package config
import (
"github.com/spf13/cobra"
)
const configPathFlag = "path"
var (
// RootCmd is a root command of config section.
RootCmd = &cobra.Command{
Use: "config",
Short: "Section for frostfs-adm config related commands",
}
initCmd = &cobra.Command{
Use: "init",
Short: "Initialize basic frostfs-adm configuration file",
Example: `frostfs-adm config init
frostfs-adm config init --path .config/frostfs-adm.yml`,
RunE: initConfig,
}
)
func init() {
RootCmd.AddCommand(initCmd)
initCmd.Flags().String(configPathFlag, "", "Path to config (default ~/.frostfs/adm/config.yml)")
}

View file

@ -6,20 +6,23 @@ import (
"fmt"
"math/big"
"github.com/nspcc-dev/neo-go/pkg/core/native/nativenames"
"github.com/TrueCloudLab/frostfs-contract/nns"
"github.com/TrueCloudLab/frostfs-sdk-go/netmap"
"github.com/nspcc-dev/neo-go/pkg/core/native/noderoles"
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/nspcc-dev/neo-go/pkg/encoding/fixedn"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/gas"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/rolemgmt"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
"github.com/nspcc-dev/neofs-contract/nns"
"github.com/nspcc-dev/neofs-sdk-go/netmap"
"github.com/nspcc-dev/neo-go/pkg/vm/vmstate"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
@ -55,20 +58,7 @@ func dumpBalances(cmd *cobra.Command, _ []string) error {
return err
}
ns, err := getNativeHashes(c)
if err != nil {
return fmt.Errorf("can't fetch the list of native contracts: %w", err)
}
gasHash, ok := ns[nativenames.Gas]
if !ok {
return fmt.Errorf("can't find the %s native contract hash", nativenames.Gas)
}
desigHash, ok := ns[nativenames.Designation]
if !ok {
return fmt.Errorf("can't find the %s native contract hash", nativenames.Designation)
}
inv := invoker.New(c, nil)
if !notaryEnabled || dumpStorage || dumpAlphabet || dumpProxy {
nnsCs, err = c.GetContractStateByID(1)
@ -76,29 +66,25 @@ func dumpBalances(cmd *cobra.Command, _ []string) error {
return fmt.Errorf("can't get NNS contract info: %w", err)
}
nmHash, err = nnsResolveHash(c, nnsCs.Hash, netmapContract+".neofs")
nmHash, err = nnsResolveHash(inv, nnsCs.Hash, netmapContract+".frostfs")
if err != nil {
return fmt.Errorf("can't get netmap contract hash: %w", err)
}
}
irList, err := fetchIRNodes(c, nmHash, desigHash)
irList, err := fetchIRNodes(c, nmHash, rolemgmt.Hash)
if err != nil {
return err
}
if err := fetchBalances(c, gasHash, irList); err != nil {
if err := fetchBalances(inv, gas.Hash, irList); err != nil {
return err
}
printBalances(cmd, "Inner ring nodes balances:", irList)
if dumpStorage {
res, err := invokeFunction(c, nmHash, "netmap", []interface{}{}, nil)
if err != nil || res.State != vm.HaltState.String() || len(res.Stack) == 0 {
return errors.New("can't fetch the list of storage nodes")
}
arr, ok := res.Stack[0].Value().([]stackitem.Item)
if !ok {
arr, err := unwrap.Array(inv.Call(nmHash, "netmap"))
if err != nil {
return errors.New("can't fetch the list of storage nodes")
}
@ -123,20 +109,20 @@ func dumpBalances(cmd *cobra.Command, _ []string) error {
snList[i].scriptHash = pub.GetScriptHash()
}
if err := fetchBalances(c, gasHash, snList); err != nil {
if err := fetchBalances(inv, gas.Hash, snList); err != nil {
return err
}
printBalances(cmd, "\nStorage node balances:", snList)
}
if dumpProxy {
h, err := nnsResolveHash(c, nnsCs.Hash, proxyContract+".neofs")
h, err := nnsResolveHash(inv, nnsCs.Hash, proxyContract+".frostfs")
if err != nil {
return fmt.Errorf("can't get hash of the proxy contract: %w", err)
}
proxyList := []accBalancePair{{scriptHash: h}}
if err := fetchBalances(c, gasHash, proxyList); err != nil {
if err := fetchBalances(inv, gas.Hash, proxyList); err != nil {
return err
}
printBalances(cmd, "\nProxy contract balance:", proxyList)
@ -168,7 +154,7 @@ func dumpBalances(cmd *cobra.Command, _ []string) error {
alphaList[i].scriptHash = h
}
if err := fetchBalances(c, gasHash, alphaList); err != nil {
if err := fetchBalances(inv, gas.Hash, alphaList); err != nil {
return err
}
printBalances(cmd, "\nAlphabet contracts balances:", alphaList)
@ -180,13 +166,15 @@ func dumpBalances(cmd *cobra.Command, _ []string) error {
func fetchIRNodes(c Client, nmHash, desigHash util.Uint160) ([]accBalancePair, error) {
var irList []accBalancePair
inv := invoker.New(c, nil)
if notaryEnabled {
height, err := c.GetBlockCount()
if err != nil {
return nil, fmt.Errorf("can't get block height: %w", err)
}
arr, err := getDesignatedByRole(c, desigHash, noderoles.NeoFSAlphabet, height)
arr, err := getDesignatedByRole(inv, desigHash, noderoles.NeoFSAlphabet, height)
if err != nil {
return nil, errors.New("can't fetch list of IR nodes from the netmap contract")
}
@ -196,27 +184,14 @@ func fetchIRNodes(c Client, nmHash, desigHash util.Uint160) ([]accBalancePair, e
irList[i].scriptHash = arr[i].GetScriptHash()
}
} else {
res, err := invokeFunction(c, nmHash, "innerRingList", []interface{}{}, nil)
if err != nil || res.State != vm.HaltState.String() || len(res.Stack) == 0 {
arr, err := unwrap.ArrayOfBytes(inv.Call(nmHash, "innerRingList"))
if err != nil {
return nil, errors.New("can't fetch list of IR nodes from the netmap contract")
}
arr, ok := res.Stack[0].Value().([]stackitem.Item)
if !ok || len(arr) == 0 {
return nil, errors.New("can't fetch list of IR nodes: invalid response")
}
irList = make([]accBalancePair, len(arr))
for i := range arr {
node, ok := arr[i].Value().([]stackitem.Item)
if !ok || len(arr) == 0 {
return nil, errors.New("can't fetch list of IR nodes: invalid response")
}
bs, err := node[0].TryBytes()
if err != nil {
return nil, fmt.Errorf("can't fetch list of IR nodes: %w", err)
}
pub, err := keys.NewPublicKeyFromBytes(bs, elliptic.P256())
pub, err := keys.NewPublicKeyFromBytes(arr[i], elliptic.P256())
if err != nil {
return nil, fmt.Errorf("can't parse IR node public key: %w", err)
}
@ -241,7 +216,7 @@ func printBalances(cmd *cobra.Command, prefix string, accounts []accBalancePair)
}
}
func fetchBalances(c Client, gasHash util.Uint160, accounts []accBalancePair) error {
func fetchBalances(c *invoker.Invoker, gasHash util.Uint160, accounts []accBalancePair) error {
w := io.NewBufBinWriter()
for i := range accounts {
emit.AppCall(w.BinWriter, gasHash, "balanceOf", callflag.ReadStates, accounts[i].scriptHash)
@ -250,8 +225,8 @@ func fetchBalances(c Client, gasHash util.Uint160, accounts []accBalancePair) er
panic(w.Err)
}
res, err := c.InvokeScript(w.Bytes(), nil)
if err != nil || res.State != vm.HaltState.String() || len(res.Stack) != len(accounts) {
res, err := c.Run(w.Bytes())
if err != nil || res.State != vmstate.Halt.String() || len(res.Stack) != len(accounts) {
return errors.New("can't fetch account balances")
}

View file

@ -0,0 +1,192 @@
package morph
import (
"bytes"
"encoding/binary"
"encoding/hex"
"errors"
"fmt"
"strconv"
"strings"
"text/tabwriter"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
const forceConfigSet = "force"
func dumpNetworkConfig(cmd *cobra.Command, _ []string) error {
c, err := getN3Client(viper.GetViper())
if err != nil {
return fmt.Errorf("can't create N3 client: %w", err)
}
inv := invoker.New(c, nil)
cs, err := c.GetContractStateByID(1)
if err != nil {
return fmt.Errorf("can't get NNS contract info: %w", err)
}
nmHash, err := nnsResolveHash(inv, cs.Hash, netmapContract+".frostfs")
if err != nil {
return fmt.Errorf("can't get netmap contract hash: %w", err)
}
arr, err := unwrap.Array(inv.Call(nmHash, "listConfig"))
if err != nil {
return errors.New("can't fetch list of network config keys from the netmap contract")
}
buf := bytes.NewBuffer(nil)
tw := tabwriter.NewWriter(buf, 0, 2, 2, ' ', 0)
for _, param := range arr {
tuple, ok := param.Value().([]stackitem.Item)
if !ok || len(tuple) != 2 {
return errors.New("invalid ListConfig response from netmap contract")
}
k, err := tuple[0].TryBytes()
if err != nil {
return errors.New("invalid config key from netmap contract")
}
v, err := tuple[1].TryBytes()
if err != nil {
return invalidConfigValueErr(k)
}
switch string(k) {
case netmapAuditFeeKey, netmapBasicIncomeRateKey,
netmapContainerFeeKey, netmapContainerAliasFeeKey,
netmapEigenTrustIterationsKey,
netmapEpochKey, netmapInnerRingCandidateFeeKey,
netmapMaxObjectSizeKey, netmapWithdrawFeeKey:
nbuf := make([]byte, 8)
copy(nbuf[:], v)
n := binary.LittleEndian.Uint64(nbuf)
_, _ = tw.Write([]byte(fmt.Sprintf("%s:\t%d (int)\n", k, n)))
case netmapEigenTrustAlphaKey:
_, _ = tw.Write([]byte(fmt.Sprintf("%s:\t%s (str)\n", k, v)))
case netmapHomomorphicHashDisabledKey, netmapMaintenanceAllowedKey:
vBool, err := tuple[1].TryBool()
if err != nil {
return invalidConfigValueErr(k)
}
_, _ = tw.Write([]byte(fmt.Sprintf("%s:\t%t (bool)\n", k, vBool)))
default:
_, _ = tw.Write([]byte(fmt.Sprintf("%s:\t%s (hex)\n", k, hex.EncodeToString(v))))
}
}
_ = tw.Flush()
cmd.Print(buf.String())
return nil
}
func setConfigCmd(cmd *cobra.Command, args []string) error {
if len(args) == 0 {
return errors.New("empty config pairs")
}
wCtx, err := newInitializeContext(cmd, viper.GetViper())
if err != nil {
return fmt.Errorf("can't initialize context: %w", err)
}
cs, err := wCtx.Client.GetContractStateByID(1)
if err != nil {
return fmt.Errorf("can't get NNS contract info: %w", err)
}
nmHash, err := nnsResolveHash(wCtx.ReadOnlyInvoker, cs.Hash, netmapContract+".frostfs")
if err != nil {
return fmt.Errorf("can't get netmap contract hash: %w", err)
}
forceFlag, _ := cmd.Flags().GetBool(forceConfigSet)
bw := io.NewBufBinWriter()
for _, arg := range args {
k, v, err := parseConfigPair(arg, forceFlag)
if err != nil {
return err
}
// In NeoFS this is done via Notary contract. Here, however, we can form the
// transaction locally. The first `nil` argument is required only for notary
// disabled environment which is not supported by that command.
emit.AppCall(bw.BinWriter, nmHash, "setConfig", callflag.All, nil, k, v)
if bw.Err != nil {
return fmt.Errorf("can't form raw transaction: %w", bw.Err)
}
}
err = wCtx.sendConsensusTx(bw.Bytes())
if err != nil {
return err
}
return wCtx.awaitTx()
}
func parseConfigPair(kvStr string, force bool) (key string, val any, err error) {
k, v, found := strings.Cut(kvStr, "=")
if !found {
return "", nil, fmt.Errorf("invalid parameter format: must be 'key=val', got: %s", kvStr)
}
key = k
valRaw := v
switch key {
case netmapAuditFeeKey, netmapBasicIncomeRateKey,
netmapContainerFeeKey, netmapContainerAliasFeeKey,
netmapEigenTrustIterationsKey,
netmapEpochKey, netmapInnerRingCandidateFeeKey,
netmapMaxObjectSizeKey, netmapWithdrawFeeKey:
val, err = strconv.ParseInt(valRaw, 10, 64)
if err != nil {
err = fmt.Errorf("could not parse %s's value '%s' as int: %w", key, valRaw, err)
}
case netmapEigenTrustAlphaKey:
// just check that it could
// be parsed correctly
_, err = strconv.ParseFloat(v, 64)
if err != nil {
err = fmt.Errorf("could not parse %s's value '%s' as float: %w", key, valRaw, err)
}
val = valRaw
case netmapHomomorphicHashDisabledKey, netmapMaintenanceAllowedKey:
val, err = strconv.ParseBool(valRaw)
if err != nil {
err = fmt.Errorf("could not parse %s's value '%s' as bool: %w", key, valRaw, err)
}
default:
if !force {
return "", nil, fmt.Errorf(
"'%s' key is not well-known, use '--%s' flag if want to set it anyway",
key, forceConfigSet)
}
val = valRaw
}
return
}
func invalidConfigValueErr(key []byte) error {
return fmt.Errorf("invalid %s config value from netmap contract", key)
}

View file

@ -7,19 +7,52 @@ import (
"os"
"sort"
cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
"github.com/nspcc-dev/neo-go/pkg/crypto/hash"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
cid "github.com/nspcc-dev/neofs-sdk-go/container/id"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
var errInvalidContainerResponse = errors.New("invalid response from container contract")
func getContainerContractHash(cmd *cobra.Command, inv *invoker.Invoker, c Client) (util.Uint160, error) {
s, err := cmd.Flags().GetString(containerContractFlag)
var ch util.Uint160
if err == nil {
ch, err = util.Uint160DecodeStringLE(s)
}
if err != nil {
nnsCs, err := c.GetContractStateByID(1)
if err != nil {
return util.Uint160{}, fmt.Errorf("can't get NNS contract state: %w", err)
}
ch, err = nnsResolveHash(inv, nnsCs.Hash, containerContract+".frostfs")
if err != nil {
return util.Uint160{}, err
}
}
return ch, nil
}
func getContainersList(inv *invoker.Invoker, ch util.Uint160) ([][]byte, error) {
res, err := inv.Call(ch, "list", "")
if err != nil {
return nil, fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
itm, err := unwrap.Item(res, err)
if _, ok := itm.(stackitem.Null); !ok {
return unwrap.ArrayOfBytes(res, err)
}
return nil, nil
}
func dumpContainers(cmd *cobra.Command, _ []string) error {
filename, err := cmd.Flags().GetString(containerDumpFlag)
if err != nil {
@ -31,41 +64,18 @@ func dumpContainers(cmd *cobra.Command, _ []string) error {
return fmt.Errorf("can't create N3 client: %w", err)
}
nnsCs, err := c.GetContractStateByID(1)
inv := invoker.New(c, nil)
ch, err := getContainerContractHash(cmd, inv, c)
if err != nil {
return fmt.Errorf("can't get NNS contract state: %w", err)
return fmt.Errorf("unable to get contaract hash: %w", err)
}
var ch util.Uint160
s, err := cmd.Flags().GetString(containerContractFlag)
if err == nil {
ch, err = util.Uint160DecodeStringLE(s)
}
if err != nil {
ch, err = nnsResolveHash(c, nnsCs.Hash, containerContract+".neofs")
if err != nil {
return err
}
}
res, err := invokeFunction(c, ch, "list", []interface{}{""}, nil)
cids, err := getContainersList(inv, ch)
if err != nil {
return fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
var cids [][]byte
arr, ok := res.Stack[0].Value().([]stackitem.Item)
if !ok {
return fmt.Errorf("%w: not a struct", errInvalidContainerResponse)
}
for _, item := range arr {
id, err := item.TryBytes()
if err != nil {
return fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
cids = append(cids, id)
}
isOK, err := getCIDFilterFunc(cmd)
if err != nil {
return err
@ -80,7 +90,7 @@ func dumpContainers(cmd *cobra.Command, _ []string) error {
bw.Reset()
emit.AppCall(bw.BinWriter, ch, "get", callflag.All, id)
emit.AppCall(bw.BinWriter, ch, "eACL", callflag.All, id)
res, err := c.InvokeScript(bw.Bytes(), nil)
res, err := inv.Run(bw.Bytes())
if err != nil {
return fmt.Errorf("can't get container info: %w", err)
}
@ -113,6 +123,35 @@ func dumpContainers(cmd *cobra.Command, _ []string) error {
return os.WriteFile(filename, out, 0o660)
}
func listContainers(cmd *cobra.Command, _ []string) error {
c, err := getN3Client(viper.GetViper())
if err != nil {
return fmt.Errorf("can't create N3 client: %w", err)
}
inv := invoker.New(c, nil)
ch, err := getContainerContractHash(cmd, inv, c)
if err != nil {
return fmt.Errorf("unable to get contaract hash: %w", err)
}
cids, err := getContainersList(inv, ch)
if err != nil {
return fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
for _, id := range cids {
var idCnr cid.ID
err = idCnr.Decode(id)
if err != nil {
return fmt.Errorf("unable to decode container id: %w", err)
}
cmd.Println(idCnr)
}
return nil
}
func restoreContainers(cmd *cobra.Command, _ []string) error {
filename, err := cmd.Flags().GetString(containerDumpFlag)
if err != nil {
@ -123,13 +162,14 @@ func restoreContainers(cmd *cobra.Command, _ []string) error {
if err != nil {
return err
}
defer wCtx.close()
nnsCs, err := wCtx.Client.GetContractStateByID(1)
if err != nil {
return fmt.Errorf("can't get NNS contract state: %w", err)
}
ch, err := nnsResolveHash(wCtx.Client, nnsCs.Hash, containerContract+".neofs")
ch, err := nnsResolveHash(wCtx.ReadOnlyInvoker, nnsCs.Hash, containerContract+".frostfs")
if err != nil {
return fmt.Errorf("can't fetch container contract hash: %w", err)
}
@ -188,7 +228,7 @@ func restoreContainers(cmd *cobra.Command, _ []string) error {
panic(bw.Err)
}
if err := wCtx.sendCommitteeTx(bw.Bytes(), -1, true); err != nil {
if err := wCtx.sendConsensusTx(bw.Bytes()); err != nil {
return err
}
}

View file

@ -0,0 +1,224 @@
package morph
import (
"encoding/json"
"fmt"
"os"
"strings"
"github.com/TrueCloudLab/frostfs-contract/nns"
"github.com/nspcc-dev/neo-go/cli/cmdargs"
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/management"
"github.com/nspcc-dev/neo-go/pkg/services/rpcsrv/params"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/opcode"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
const (
contractPathFlag = "contract"
updateFlag = "update"
customZoneFlag = "domain"
)
var deployCmd = &cobra.Command{
Use: "deploy",
Short: "Deploy additional smart-contracts",
Long: `Deploy additional smart-contract which are not related to core.
All contracts are deployed by the committee, so access to the alphabet wallets is required.
Optionally, arguments can be provided to be passed to a contract's _deploy function.
The syntax is the same as for 'neo-go contract testinvokefunction' command.
Compiled contract file name must contain '_contract.nef' suffix.
Contract's manifest file name must be 'config.json'.
NNS name is taken by stripping '_contract.nef' from the NEF file (similar to frostfs contracts).`,
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
RunE: deployContractCmd,
}
func init() {
ff := deployCmd.Flags()
ff.String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
_ = deployCmd.MarkFlagFilename(alphabetWalletsFlag)
ff.StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
ff.String(contractPathFlag, "", "Path to the contract directory")
_ = deployCmd.MarkFlagFilename(contractPathFlag)
ff.Bool(updateFlag, false, "Update an existing contract")
ff.String(customZoneFlag, "frostfs", "Custom zone for NNS")
}
func deployContractCmd(cmd *cobra.Command, args []string) error {
v := viper.GetViper()
c, err := newInitializeContext(cmd, v)
if err != nil {
return fmt.Errorf("initialization error: %w", err)
}
defer c.close()
ctrPath, _ := cmd.Flags().GetString(contractPathFlag)
ctrName, err := probeContractName(ctrPath)
if err != nil {
return err
}
cs, err := readContract(ctrPath, ctrName)
if err != nil {
return err
}
nnsCs, err := c.Client.GetContractStateByID(1)
if err != nil {
return fmt.Errorf("can't fetch NNS contract state: %w", err)
}
callHash := management.Hash
method := deployMethodName
zone, _ := cmd.Flags().GetString(customZoneFlag)
domain := ctrName + "." + zone
isUpdate, _ := cmd.Flags().GetBool(updateFlag)
if isUpdate {
cs.Hash, err = nnsResolveHash(c.ReadOnlyInvoker, nnsCs.Hash, domain)
if err != nil {
return fmt.Errorf("can't fetch contract hash from NNS: %w", err)
}
callHash = cs.Hash
method = updateMethodName
} else {
cs.Hash = state.CreateContractHash(
c.CommitteeAcc.Contract.ScriptHash(),
cs.NEF.Checksum,
cs.Manifest.Name)
}
w := io.NewBufBinWriter()
if err := emitDeploymentArguments(w.BinWriter, args); err != nil {
return err
}
emit.Bytes(w.BinWriter, cs.RawManifest)
emit.Bytes(w.BinWriter, cs.RawNEF)
emit.Int(w.BinWriter, 3)
emit.Opcodes(w.BinWriter, opcode.PACK)
emit.AppCallNoArgs(w.BinWriter, callHash, method, callflag.All)
emit.Opcodes(w.BinWriter, opcode.DROP) // contract state on stack
if !isUpdate {
bw := io.NewBufBinWriter()
emit.Instruction(bw.BinWriter, opcode.INITSSLOT, []byte{1})
emit.AppCall(bw.BinWriter, nnsCs.Hash, "getPrice", callflag.All)
emit.Opcodes(bw.BinWriter, opcode.STSFLD0)
emit.AppCall(bw.BinWriter, nnsCs.Hash, "setPrice", callflag.All, 1)
start := bw.Len()
needRecord := false
ok, err := c.nnsRootRegistered(nnsCs.Hash, zone)
if err != nil {
return err
} else if !ok {
needRecord = true
emit.AppCall(bw.BinWriter, nnsCs.Hash, "register", callflag.All,
zone, c.CommitteeAcc.Contract.ScriptHash(),
"ops@nspcc.ru", int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
emit.AppCall(bw.BinWriter, nnsCs.Hash, "register", callflag.All,
domain, c.CommitteeAcc.Contract.ScriptHash(),
"ops@nspcc.ru", int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
} else {
s, ok, err := c.nnsRegisterDomainScript(nnsCs.Hash, cs.Hash, domain)
if err != nil {
return err
}
needRecord = !ok
if len(s) != 0 {
bw.WriteBytes(s)
}
}
if needRecord {
emit.AppCall(bw.BinWriter, nnsCs.Hash, "deleteRecords", callflag.All, domain, int64(nns.TXT))
emit.AppCall(bw.BinWriter, nnsCs.Hash, "addRecord", callflag.All,
domain, int64(nns.TXT), address.Uint160ToString(cs.Hash))
}
if bw.Err != nil {
panic(fmt.Errorf("BUG: can't create deployment script: %w", w.Err))
} else if bw.Len() != start {
w.WriteBytes(bw.Bytes())
emit.Opcodes(w.BinWriter, opcode.LDSFLD0, opcode.PUSH1, opcode.PACK)
emit.AppCallNoArgs(w.BinWriter, nnsCs.Hash, "setPrice", callflag.All)
if needRecord {
c.Command.Printf("NNS: Set %s -> %s\n", domain, cs.Hash.StringLE())
}
}
}
if w.Err != nil {
panic(fmt.Errorf("BUG: can't create deployment script: %w", w.Err))
}
if err := c.sendCommitteeTx(w.Bytes(), false); err != nil {
return err
}
return c.awaitTx()
}
func emitDeploymentArguments(w *io.BinWriter, args []string) error {
_, ps, err := cmdargs.ParseParams(args, true)
if err != nil {
return err
}
if len(ps) == 0 {
emit.Opcodes(w, opcode.NEWARRAY0)
return nil
}
if len(ps) != 1 {
return fmt.Errorf("at most one argument is expected for deploy, got %d", len(ps))
}
// We could emit this directly, but round-trip through JSON is more robust.
// This a CLI, so optimizing the conversion is not worth the effort.
data, err := json.Marshal(ps)
if err != nil {
return err
}
var pp params.Params
if err := json.Unmarshal(data, &pp); err != nil {
return err
}
return params.ExpandArrayIntoScript(w, pp)
}
func probeContractName(ctrPath string) (string, error) {
ds, err := os.ReadDir(ctrPath)
if err != nil {
return "", fmt.Errorf("can't read directory: %w", err)
}
var ctrName string
for i := range ds {
if strings.HasSuffix(ds[i].Name(), "_contract.nef") {
ctrName = strings.TrimSuffix(ds[i].Name(), "_contract.nef")
break
}
}
if ctrName == "" {
return "", fmt.Errorf("can't find any NEF files in %s", ctrPath)
}
return ctrName, nil
}

View file

@ -14,7 +14,7 @@ import (
func downloadContractsFromGithub(cmd *cobra.Command) (io.ReadCloser, error) {
gcl := github.NewClient(nil)
release, _, err := gcl.Repositories.GetLatestRelease(context.Background(), "nspcc-dev", "neofs-contract")
release, _, err := gcl.Repositories.GetLatestRelease(context.Background(), "nspcc-dev", "frostfs-contract")
if err != nil {
return nil, fmt.Errorf("can't fetch release info: %w", err)
}
@ -23,7 +23,7 @@ func downloadContractsFromGithub(cmd *cobra.Command) (io.ReadCloser, error) {
var url string
for _, a := range release.Assets {
if strings.HasPrefix(a.GetName(), "neofs-contract") {
if strings.HasPrefix(a.GetName(), "frostfs-contract") {
url = a.GetBrowserDownloadURL()
break
}

View file

@ -0,0 +1,250 @@
package morph
import (
"bytes"
"errors"
"fmt"
"strings"
"text/tabwriter"
"github.com/TrueCloudLab/frostfs-contract/nns"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/opcode"
"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
"github.com/nspcc-dev/neo-go/pkg/vm/vmstate"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
const lastGlagoliticLetter = 41
type contractDumpInfo struct {
hash util.Uint160
name string
version string
}
func dumpContractHashes(cmd *cobra.Command, _ []string) error {
c, err := getN3Client(viper.GetViper())
if err != nil {
return fmt.Errorf("can't create N3 client: %w", err)
}
cs, err := c.GetContractStateByID(1)
if err != nil {
return err
}
zone, _ := cmd.Flags().GetString(customZoneFlag)
if zone != "" {
return dumpCustomZoneHashes(cmd, cs.Hash, zone, c)
}
infos := []contractDumpInfo{{name: nnsContract, hash: cs.Hash}}
irSize := 0
for ; irSize < lastGlagoliticLetter; irSize++ {
ok, err := nnsIsAvailable(c, cs.Hash, getAlphabetNNSDomain(irSize))
if err != nil {
return err
} else if ok {
break
}
}
bw := io.NewBufBinWriter()
if irSize != 0 {
bw.Reset()
for i := 0; i < irSize; i++ {
emit.AppCall(bw.BinWriter, cs.Hash, "resolve", callflag.ReadOnly,
getAlphabetNNSDomain(i),
int64(nns.TXT))
}
alphaRes, err := c.InvokeScript(bw.Bytes(), nil)
if err != nil {
return fmt.Errorf("can't fetch info from NNS: %w", err)
}
for i := 0; i < irSize; i++ {
info := contractDumpInfo{name: fmt.Sprintf("alphabet %d", i)}
if h, err := parseNNSResolveResult(alphaRes.Stack[i]); err == nil {
info.hash = h
}
infos = append(infos, info)
}
}
for _, ctrName := range contractList {
bw.Reset()
emit.AppCall(bw.BinWriter, cs.Hash, "resolve", callflag.ReadOnly,
ctrName+".frostfs", int64(nns.TXT))
res, err := c.InvokeScript(bw.Bytes(), nil)
if err != nil {
return fmt.Errorf("can't fetch info from NNS: %w", err)
}
info := contractDumpInfo{name: ctrName}
if len(res.Stack) != 0 {
if h, err := parseNNSResolveResult(res.Stack[0]); err == nil {
info.hash = h
}
}
infos = append(infos, info)
}
fillContractVersion(cmd, c, infos)
printContractInfo(cmd, infos)
return nil
}
func dumpCustomZoneHashes(cmd *cobra.Command, nnsHash util.Uint160, zone string, c Client) error {
const nnsMaxTokens = 100
inv := invoker.New(c, nil)
if !strings.HasPrefix(zone, ".") {
zone = "." + zone
}
var infos []contractDumpInfo
processItem := func(item stackitem.Item) {
bs, err := item.TryBytes()
if err != nil {
cmd.PrintErrf("Invalid NNS record: %v\n", err)
return
}
if !bytes.HasSuffix(bs, []byte(zone)) {
// Related https://github.com/nspcc-dev/neofs-contract/issues/316.
return
}
h, err := nnsResolveHash(inv, nnsHash, string(bs))
if err != nil {
cmd.PrintErrf("Could not resolve name %s: %v\n", string(bs), err)
return
}
infos = append(infos, contractDumpInfo{
hash: h,
name: strings.TrimSuffix(string(bs), zone),
})
}
sessionID, iter, err := unwrap.SessionIterator(inv.Call(nnsHash, "tokens"))
if err != nil {
if errors.Is(err, unwrap.ErrNoSessionID) {
items, err := unwrap.Array(inv.CallAndExpandIterator(nnsHash, "tokens", nnsMaxTokens))
if err != nil {
return fmt.Errorf("can't get a list of NNS domains: %w", err)
}
if len(items) == nnsMaxTokens {
cmd.PrintErrln("Provided RPC endpoint doesn't support sessions, some hashes might be lost.")
}
for i := range items {
processItem(items[i])
}
} else {
return err
}
} else {
defer func() {
_ = inv.TerminateSession(sessionID)
}()
items, err := inv.TraverseIterator(sessionID, &iter, nnsMaxTokens)
for err == nil && len(items) != 0 {
for i := range items {
processItem(items[i])
}
items, err = inv.TraverseIterator(sessionID, &iter, nnsMaxTokens)
}
if err != nil {
return fmt.Errorf("error during NNS domains iteration: %w", err)
}
}
fillContractVersion(cmd, c, infos)
printContractInfo(cmd, infos)
return nil
}
func parseContractVersion(item stackitem.Item) string {
bi, err := item.TryInteger()
if err != nil || bi.Sign() == 0 || !bi.IsInt64() {
return "unknown"
}
v := bi.Int64()
major := v / 1_000_000
minor := (v % 1_000_000) / 1000
patch := v % 1_000
return fmt.Sprintf("v%d.%d.%d", major, minor, patch)
}
func printContractInfo(cmd *cobra.Command, infos []contractDumpInfo) {
if len(infos) == 0 {
return
}
buf := bytes.NewBuffer(nil)
tw := tabwriter.NewWriter(buf, 0, 2, 2, ' ', 0)
for _, info := range infos {
if info.version == "" {
info.version = "unknown"
}
_, _ = tw.Write([]byte(fmt.Sprintf("%s\t(%s):\t%s\n",
info.name, info.version, info.hash.StringLE())))
}
_ = tw.Flush()
cmd.Print(buf.String())
}
func fillContractVersion(cmd *cobra.Command, c Client, infos []contractDumpInfo) {
bw := io.NewBufBinWriter()
sub := io.NewBufBinWriter()
for i := range infos {
if infos[i].hash.Equals(util.Uint160{}) {
emit.Int(bw.BinWriter, 0)
} else {
sub.Reset()
emit.AppCall(sub.BinWriter, infos[i].hash, "version", callflag.NoneFlag)
if sub.Err != nil {
panic(fmt.Errorf("BUG: can't create version script: %w", bw.Err))
}
script := sub.Bytes()
emit.Instruction(bw.BinWriter, opcode.TRY, []byte{byte(3 + len(script) + 2), 0})
bw.BinWriter.WriteBytes(script)
emit.Instruction(bw.BinWriter, opcode.ENDTRY, []byte{2 + 1})
emit.Opcodes(bw.BinWriter, opcode.PUSH0)
}
}
emit.Opcodes(bw.BinWriter, opcode.NOP) // for the last ENDTRY target
if bw.Err != nil {
panic(fmt.Errorf("BUG: can't create version script: %w", bw.Err))
}
res, err := c.InvokeScript(bw.Bytes(), nil)
if err != nil {
cmd.Printf("Can't fetch version from NNS: %v\n", err)
return
}
if res.State == vmstate.Halt.String() {
for i := range res.Stack {
infos[i].version = parseContractVersion(res.Stack[i])
}
}
}

View file

@ -5,9 +5,9 @@ import (
"fmt"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/spf13/cobra"
"github.com/spf13/viper"
@ -24,7 +24,7 @@ func forceNewEpochCmd(cmd *cobra.Command, args []string) error {
return fmt.Errorf("can't get NNS contract info: %w", err)
}
nmHash, err := nnsResolveHash(wCtx.Client, cs.Hash, netmapContract+".neofs")
nmHash, err := nnsResolveHash(wCtx.ReadOnlyInvoker, cs.Hash, netmapContract+".frostfs")
if err != nil {
return fmt.Errorf("can't get netmap contract hash: %w", err)
}
@ -34,7 +34,7 @@ func forceNewEpochCmd(cmd *cobra.Command, args []string) error {
return err
}
if err := wCtx.sendCommitteeTx(bw.Bytes(), -1, true); err != nil {
if err := wCtx.sendConsensusTx(bw.Bytes()); err != nil {
return err
}
@ -42,18 +42,13 @@ func forceNewEpochCmd(cmd *cobra.Command, args []string) error {
}
func emitNewEpochCall(bw *io.BufBinWriter, wCtx *initializeContext, nmHash util.Uint160) error {
res, err := invokeFunction(wCtx.Client, nmHash, "epoch", nil, nil)
if err != nil || res.State != vm.HaltState.String() || len(res.Stack) == 0 {
curr, err := unwrap.Int64(wCtx.ReadOnlyInvoker.Call(nmHash, "epoch"))
if err != nil {
return errors.New("can't fetch current epoch from the netmap contract")
}
bi, err := res.Stack[0].TryInteger()
if err != nil {
return fmt.Errorf("can't parse current epoch: %w", err)
}
newEpoch := bi.Int64() + 1
wCtx.Command.Printf("Current epoch: %s, increase to %d.\n", bi, newEpoch)
newEpoch := curr + 1
wCtx.Command.Printf("Current epoch: %d, increase to %d.\n", curr, newEpoch)
// In NeoFS this is done via Notary contract. Here, however, we can form the
// transaction locally.

View file

@ -6,17 +6,19 @@ import (
"os"
"path/filepath"
"github.com/nspcc-dev/neo-go/pkg/core/native/nativenames"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config"
"github.com/TrueCloudLab/frostfs-node/pkg/innerring"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/nspcc-dev/neo-go/pkg/encoding/fixedn"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/gas"
"github.com/nspcc-dev/neo-go/pkg/smartcontract"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/opcode"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/nspcc-dev/neofs-node/cmd/neofs-adm/internal/modules/config"
"github.com/nspcc-dev/neofs-node/pkg/innerring"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
@ -44,11 +46,10 @@ func generateAlphabetCreds(cmd *cobra.Command, args []string) error {
return err
}
w, err := initializeContractWallet(v, walletDir)
_, err = initializeContractWallet(v, walletDir)
if err != nil {
return err
}
w.Close()
cmd.Println("size:", size)
cmd.Println("alphabet-wallets:", walletDir)
@ -111,7 +112,6 @@ func initializeWallets(v *viper.Viper, walletDir string, size int) ([]string, er
if err := w.SavePretty(); err != nil {
return nil, fmt.Errorf("can't save wallet: %w", err)
}
w.Close()
}
return passwords, nil
@ -135,12 +135,20 @@ func generateStorageCreds(cmd *cobra.Command, _ []string) error {
return refillGas(cmd, storageGasConfigFlag, true)
}
func refillGas(cmd *cobra.Command, gasFlag string, createWallet bool) error {
func refillGas(cmd *cobra.Command, gasFlag string, createWallet bool) (err error) {
// storage wallet path is not part of the config
storageWalletPath, err := cmd.Flags().GetString(storageWalletFlag)
storageWalletPath, _ := cmd.Flags().GetString(storageWalletFlag)
// wallet address is not part of the config
walletAddress, _ := cmd.Flags().GetString(walletAddressFlag)
var gasReceiver util.Uint160
if len(walletAddress) != 0 {
gasReceiver, err = address.StringToUint160(walletAddress)
if err != nil {
return err
return fmt.Errorf("invalid wallet address %s: %w", walletAddress, err)
}
} else {
if storageWalletPath == "" {
return fmt.Errorf("missing wallet path (use '--%s <out.json>')", storageWalletFlag)
}
@ -175,6 +183,9 @@ func refillGas(cmd *cobra.Command, gasFlag string, createWallet bool) error {
}
}
gasReceiver = w.Accounts[0].Contract.ScriptHash()
}
gasStr := viper.GetString(gasFlag)
gasAmount, err := parseGASAmount(gasStr)
@ -187,17 +198,15 @@ func refillGas(cmd *cobra.Command, gasFlag string, createWallet bool) error {
return err
}
gasHash := wCtx.nativeHash(nativenames.Gas)
bw := io.NewBufBinWriter()
emit.AppCall(bw.BinWriter, gasHash, "transfer", callflag.All,
wCtx.CommitteeAcc.Contract.ScriptHash(), w.Accounts[0].Contract.ScriptHash(), int64(gasAmount), nil)
emit.AppCall(bw.BinWriter, gas.Hash, "transfer", callflag.All,
wCtx.CommitteeAcc.Contract.ScriptHash(), gasReceiver, int64(gasAmount), nil)
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
if bw.Err != nil {
return fmt.Errorf("BUG: invalid transfer arguments: %w", bw.Err)
}
if err := wCtx.sendCommitteeTx(bw.Bytes(), -1, false); err != nil {
if err := wCtx.sendCommitteeTx(bw.Bytes(), false); err != nil {
return err
}

View file

@ -9,19 +9,22 @@ import (
"strconv"
"testing"
"github.com/TrueCloudLab/frostfs-node/pkg/innerring"
"github.com/nspcc-dev/neo-go/cli/input"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/smartcontract"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/nspcc-dev/neofs-node/pkg/innerring"
"github.com/spf13/viper"
"github.com/stretchr/testify/require"
"golang.org/x/term"
)
const testContractPassword = "grouppass"
func TestGenerateAlphabet(t *testing.T) {
const size = 4
walletDir := newTempDir(t)
walletDir := t.TempDir()
buf := setupTestTerminal(t)
cmd := generateAlphabetCmd
@ -58,7 +61,15 @@ func TestGenerateAlphabet(t *testing.T) {
require.Error(t, generateAlphabetCreds(cmd, nil))
})
testGenerateAlphabet(t, buf, v, size, walletDir)
buf.Reset()
v.Set(alphabetWalletsFlag, walletDir)
require.NoError(t, generateAlphabetCmd.Flags().Set(alphabetSizeFlag, strconv.FormatUint(size, 10)))
for i := uint64(0); i < size; i++ {
buf.WriteString(strconv.FormatUint(i, 10) + "\r")
}
buf.WriteString(testContractPassword + "\r")
require.NoError(t, generateAlphabetCreds(generateAlphabetCmd, nil))
for i := uint64(0); i < size; i++ {
p := filepath.Join(walletDir, innerring.GlagoliticLetter(i).String()+".json")
@ -70,9 +81,9 @@ func TestGenerateAlphabet(t *testing.T) {
require.NoError(t, err, "can't decrypt account")
switch a.Label {
case consensusAccountName:
require.Equal(t, size/2+1, len(a.Contract.Parameters))
require.Equal(t, smartcontract.GetDefaultHonestNodeCount(size), len(a.Contract.Parameters))
case committeeAccountName:
require.Equal(t, size*2/3+1, len(a.Contract.Parameters))
require.Equal(t, smartcontract.GetMajorityHonestNodeCount(size), len(a.Contract.Parameters))
default:
require.Equal(t, singleAccountName, a.Label)
}
@ -99,26 +110,3 @@ func setupTestTerminal(t *testing.T) *bytes.Buffer {
return in
}
func newTempDir(t *testing.T) string {
dir := filepath.Join(os.TempDir(), "neofs-adm.test."+strconv.FormatUint(rand.Uint64(), 10))
require.NoError(t, os.Mkdir(dir, os.ModePerm))
t.Cleanup(func() {
require.NoError(t, os.RemoveAll(dir))
})
return dir
}
const testContractPassword = "grouppass"
func testGenerateAlphabet(t *testing.T, buf *bytes.Buffer, v *viper.Viper, size uint64, walletDir string) {
buf.Reset()
v.Set(alphabetWalletsFlag, walletDir)
require.NoError(t, generateAlphabetCmd.Flags().Set(alphabetSizeFlag, strconv.FormatUint(size, 10)))
for i := uint64(0); i < size; i++ {
buf.WriteString(strconv.FormatUint(i, 10) + "\r")
}
buf.WriteString(testContractPassword + "\r")
require.NoError(t, generateAlphabetCreds(generateAlphabetCmd, nil))
}

View file

@ -6,11 +6,11 @@ import (
"os"
"path/filepath"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/manifest"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/nspcc-dev/neofs-node/cmd/neofs-adm/internal/modules/config"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)

View file

@ -7,18 +7,18 @@ import (
"path/filepath"
"time"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config"
"github.com/TrueCloudLab/frostfs-node/pkg/innerring"
morphClient "github.com/TrueCloudLab/frostfs-node/pkg/morph/client"
"github.com/nspcc-dev/neo-go/pkg/core/native/nativenames"
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/rpc/client"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/actor"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/trigger"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm"
"github.com/nspcc-dev/neo-go/pkg/vm/vmstate"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/nspcc-dev/neofs-node/cmd/neofs-adm/internal/modules/config"
"github.com/nspcc-dev/neofs-node/pkg/innerring"
morphClient "github.com/nspcc-dev/neofs-node/pkg/morph/client"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
@ -43,7 +43,6 @@ type initializeContext struct {
Contracts map[string]*contractState
Command *cobra.Command
ContractPath string
Natives map[string]util.Uint160
}
func initializeSideChainCmd(cmd *cobra.Command, args []string) error {
@ -116,20 +115,22 @@ func newInitializeContext(cmd *cobra.Command, v *viper.Viper) (*initializeContex
return nil, err
}
w, err := openContractWallet(v, cmd, walletDir)
needContracts := cmd.Name() == "update-contracts" || cmd.Name() == "init"
var w *wallet.Wallet
if needContracts {
w, err = openContractWallet(v, cmd, walletDir)
if err != nil {
return nil, err
}
}
var c Client
if v.GetString(localDumpFlag) != "" {
if cmd.Name() != "init" {
return nil, errors.New("dump creation is only supported for `init` command")
}
if v.GetString(endpointFlag) != "" {
return nil, fmt.Errorf("`%s` and `%s` flags are mutually exclusive", endpointFlag, localDumpFlag)
}
c, err = newLocalClient(v, wallets)
c, err = newLocalClient(cmd, v, wallets)
} else {
c, err = getN3Client(v)
}
@ -158,7 +159,6 @@ func newInitializeContext(cmd *cobra.Command, v *viper.Viper) (*initializeContex
}
}
needContracts := cmd.Name() == "update-contracts" || cmd.Name() == "init"
if needContracts {
ctrPath, err = cmd.Flags().GetString(contractsInitFlag)
if err != nil {
@ -166,8 +166,7 @@ func newInitializeContext(cmd *cobra.Command, v *viper.Viper) (*initializeContex
}
}
nativeHashes, err := getNativeHashes(c)
if err != nil {
if err := checkNotaryEnabled(c); err != nil {
return nil, err
}
@ -180,8 +179,13 @@ func newInitializeContext(cmd *cobra.Command, v *viper.Viper) (*initializeContex
accounts[i] = acc
}
cliCtx, err := defaultClientContext(c, committeeAcc)
if err != nil {
return nil, fmt.Errorf("client context: %w", err)
}
initCtx := &initializeContext{
clientContext: *defaultClientContext(c),
clientContext: *cliCtx,
ConsensusAcc: consensusAcc,
CommitteeAcc: committeeAcc,
ContractWallet: w,
@ -190,7 +194,6 @@ func newInitializeContext(cmd *cobra.Command, v *viper.Viper) (*initializeContex
Command: cmd,
Contracts: make(map[string]*contractState),
ContractPath: ctrPath,
Natives: nativeHashes,
}
if needContracts {
@ -203,10 +206,6 @@ func newInitializeContext(cmd *cobra.Command, v *viper.Viper) (*initializeContex
return initCtx, nil
}
func (c *initializeContext) nativeHash(name string) util.Uint160 {
return c.Natives[name]
}
func openAlphabetWallets(v *viper.Viper, walletDir string) ([]*wallet.Wallet, error) {
walletFiles, err := os.ReadDir(walletDir)
if err != nil {
@ -273,17 +272,17 @@ func (c *initializeContext) nnsContractState() (*state.Contract, error) {
return cs, nil
}
func (c *initializeContext) getSigner(tryGroup bool) transaction.Signer {
func (c *initializeContext) getSigner(tryGroup bool, acc *wallet.Account) transaction.Signer {
if tryGroup && c.groupKey != nil {
return transaction.Signer{
Account: c.CommitteeAcc.Contract.ScriptHash(),
Account: acc.Contract.ScriptHash(),
Scopes: transaction.CustomGroups,
AllowedGroups: keys.PublicKeys{c.groupKey},
}
}
signer := transaction.Signer{
Account: c.CommitteeAcc.Contract.ScriptHash(),
Account: acc.Contract.ScriptHash(),
Scopes: transaction.Global, // Scope is important, as we have nested call to container contract.
}
@ -296,7 +295,7 @@ func (c *initializeContext) getSigner(tryGroup bool) transaction.Signer {
return signer
}
groupKey, err := nnsResolveKey(c.Client, nnsCs.Hash, morphClient.NNSGroupKeyName)
groupKey, err := nnsResolveKey(c.ReadOnlyInvoker, nnsCs.Hash, morphClient.NNSGroupKeyName)
if err == nil {
c.groupKey = groupKey
@ -307,7 +306,7 @@ func (c *initializeContext) getSigner(tryGroup bool) transaction.Signer {
}
func (c *clientContext) awaitTx(cmd *cobra.Command) error {
if len(c.Hashes) == 0 {
if len(c.SentTxs) == 0 {
return nil
}
@ -317,41 +316,58 @@ func (c *clientContext) awaitTx(cmd *cobra.Command) error {
}
}
err := awaitTx(cmd, c.Client, c.SentTxs)
c.SentTxs = c.SentTxs[:0]
return err
}
func awaitTx(cmd *cobra.Command, c Client, txs []hashVUBPair) error {
cmd.Println("Waiting for transactions to persist...")
tick := time.NewTicker(c.PollInterval)
defer tick.Stop()
const pollInterval = time.Second
timer := time.NewTimer(c.WaitDuration)
defer timer.Stop()
tick := time.NewTicker(pollInterval)
defer tick.Stop()
at := trigger.Application
var retErr error
currBlock, err := c.GetBlockCount()
if err != nil {
return fmt.Errorf("can't fetch current block height: %w", err)
}
loop:
for i := range c.Hashes {
res, err := c.Client.GetApplicationLog(c.Hashes[i], &at)
for i := range txs {
res, err := c.GetApplicationLog(txs[i].hash, &at)
if err == nil {
if retErr == nil && len(res.Executions) > 0 && res.Executions[0].VMState != vm.HaltState {
if retErr == nil && len(res.Executions) > 0 && res.Executions[0].VMState != vmstate.Halt {
retErr = fmt.Errorf("tx %d persisted in %s state: %s",
i, res.Executions[0].VMState, res.Executions[0].FaultException)
}
continue loop
}
for {
select {
case <-tick.C:
res, err := c.Client.GetApplicationLog(c.Hashes[i], &at)
if txs[i].vub < currBlock {
return fmt.Errorf("tx was not persisted: vub=%d, height=%d", txs[i].vub, currBlock)
}
for range tick.C {
// We must fetch current height before application log, to avoid race condition.
currBlock, err = c.GetBlockCount()
if err != nil {
return fmt.Errorf("can't fetch current block height: %w", err)
}
res, err := c.GetApplicationLog(txs[i].hash, &at)
if err == nil {
if retErr == nil && len(res.Executions) > 0 && res.Executions[0].VMState != vm.HaltState {
if retErr == nil && len(res.Executions) > 0 && res.Executions[0].VMState != vmstate.Halt {
retErr = fmt.Errorf("tx %d persisted in %s state: %s",
i, res.Executions[0].VMState, res.Executions[0].FaultException)
}
continue loop
}
case <-timer.C:
return errors.New("timeout while waiting for transaction to persist")
if txs[i].vub < currBlock {
return fmt.Errorf("tx was not persisted: vub=%d, height=%d", txs[i].vub, currBlock)
}
}
}
@ -359,41 +375,61 @@ loop:
return retErr
}
// sendCommitteeTx creates transaction from script and sends it to RPC.
// If sysFee is -1, it is calculated automatically. If tryGroup is false,
// global scope is used for the signer (useful when working with native contracts).
func (c *initializeContext) sendCommitteeTx(script []byte, sysFee int64, tryGroup bool) error {
tx, err := c.Client.CreateTxFromScript(script, c.CommitteeAcc, sysFee, 0, []client.SignerAccount{{
Signer: c.getSigner(tryGroup),
Account: c.CommitteeAcc,
}})
if err != nil {
return fmt.Errorf("can't create tx: %w", err)
}
return c.multiSignAndSend(tx, committeeAccountName)
// sendCommitteeTx creates transaction from script, signs it by committee nodes and sends it to RPC.
// If tryGroup is false, global scope is used for the signer (useful when
// working with native contracts).
func (c *initializeContext) sendCommitteeTx(script []byte, tryGroup bool) error {
return c.sendMultiTx(script, tryGroup, false)
}
// sendSingleTx creates transaction signed by a simple account and pushes in onto the chain.
// It neither waits until tx persists nor checks the execution result.
func (c *initializeContext) sendSingleTx(script []byte, sysFee int64, acc *wallet.Account) error {
tx, err := c.Client.CreateTxFromScript(script, acc, sysFee, 0, []client.SignerAccount{{
Signer: transaction.Signer{
Account: acc.Contract.ScriptHash(),
Scopes: transaction.CalledByEntry,
},
Account: acc,
}})
// sendConsensusTx creates transaction from script, signs it by alphabet nodes and sends it to RPC.
// Not that because this is used only after the contracts were initialized and deployed,
// we always try to have a group scope.
func (c *initializeContext) sendConsensusTx(script []byte) error {
return c.sendMultiTx(script, true, true)
}
func (c *initializeContext) sendMultiTx(script []byte, tryGroup bool, withConsensus bool) error {
var act *actor.Actor
var err error
withConsensus = withConsensus && !c.ConsensusAcc.Contract.ScriptHash().Equals(c.CommitteeAcc.ScriptHash())
if tryGroup {
// Even for consensus signatures we need the committee to pay.
signers := make([]actor.SignerAccount, 1, 2)
signers[0] = actor.SignerAccount{
Signer: c.getSigner(tryGroup, c.CommitteeAcc),
Account: c.CommitteeAcc,
}
if withConsensus {
signers = append(signers, actor.SignerAccount{
Signer: c.getSigner(tryGroup, c.ConsensusAcc),
Account: c.ConsensusAcc,
})
}
act, err = actor.New(c.Client, signers)
} else {
if withConsensus {
panic("BUG: should never happen")
}
act, err = c.CommitteeAct, nil
}
if err != nil {
return err
return fmt.Errorf("could not create actor: %w", err)
}
magic, err := c.Client.GetNetwork()
tx, err := act.MakeUnsignedRun(script, []transaction.Attribute{{Type: transaction.HighPriority}})
if err != nil {
return fmt.Errorf("can't fetch network magic: %w", err)
return fmt.Errorf("could not perform test invocation: %w", err)
}
if err := c.multiSign(tx, committeeAccountName); err != nil {
return err
}
if withConsensus {
if err := c.multiSign(tx, consensusAccountName); err != nil {
return err
}
if err := acc.SignTx(magic, tx); err != nil {
return fmt.Errorf("can't sign tx: %w", err)
}
return c.sendTx(tx, c.Command, false)
@ -408,10 +444,10 @@ func getWalletAccount(w *wallet.Wallet, typ string) (*wallet.Account, error) {
return nil, fmt.Errorf("account for '%s' not found", typ)
}
func getNativeHashes(c Client) (map[string]util.Uint160, error) {
func checkNotaryEnabled(c Client) error {
ns, err := c.GetNativeContracts()
if err != nil {
return nil, fmt.Errorf("can't get native contract hashes: %w", err)
return fmt.Errorf("can't get native contract hashes: %w", err)
}
notaryEnabled := false
@ -423,7 +459,7 @@ func getNativeHashes(c Client) (map[string]util.Uint160, error) {
nativeHashes[ns[i].Manifest.Name] = ns[i].Hash
}
if !notaryEnabled {
return nil, errors.New("notary contract must be enabled")
return errors.New("notary contract must be enabled")
}
return nativeHashes, nil
return nil
}

View file

@ -12,34 +12,37 @@ import (
"path/filepath"
"strings"
"github.com/nspcc-dev/neo-go/pkg/core/native"
"github.com/nspcc-dev/neo-go/pkg/core/native/nativenames"
"github.com/TrueCloudLab/frostfs-contract/common"
"github.com/TrueCloudLab/frostfs-contract/nns"
"github.com/TrueCloudLab/frostfs-node/pkg/innerring"
morphClient "github.com/TrueCloudLab/frostfs-node/pkg/morph/client"
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
io2 "github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpc/client"
"github.com/nspcc-dev/neo-go/pkg/rpcclient"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/actor"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/management"
"github.com/nspcc-dev/neo-go/pkg/smartcontract"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/manifest"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/nef"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/opcode"
"github.com/nspcc-dev/neofs-node/pkg/innerring"
morphClient "github.com/nspcc-dev/neofs-node/pkg/morph/client"
"github.com/nspcc-dev/neo-go/pkg/vm/vmstate"
"github.com/spf13/viper"
)
const (
nnsContract = "nns"
neofsContract = "neofs" // not deployed in side-chain.
frostfsContract = "frostfs" // not deployed in side-chain.
processingContract = "processing" // not deployed in side-chain.
alphabetContract = "alphabet"
auditContract = "audit"
balanceContract = "balance"
containerContract = "container"
neofsIDContract = "neofsid"
frostfsIDContract = "frostfsid"
netmapContract = "netmap"
proxyContract = "proxy"
reputationContract = "reputation"
@ -57,6 +60,8 @@ const (
netmapBasicIncomeRateKey = "BasicIncomeRate"
netmapInnerRingCandidateFeeKey = "InnerRingCandidateFee"
netmapWithdrawFeeKey = "WithdrawFee"
netmapHomomorphicHashDisabledKey = "HomomorphicHashingDisabled"
netmapMaintenanceAllowedKey = "MaintenanceModeAllowed"
defaultEigenTrustIterations = 4
defaultEigenTrustAlpha = "0.1"
@ -67,7 +72,7 @@ var (
auditContract,
balanceContract,
containerContract,
neofsIDContract,
frostfsIDContract,
netmapContract,
proxyContract,
reputationContract,
@ -75,7 +80,7 @@ var (
}
fullContractList = append([]string{
neofsContract,
frostfsContract,
processingContract,
nnsContract,
alphabetContract,
@ -102,7 +107,11 @@ func (c *initializeContext) deployNNS(method string) error {
nnsCs, err := c.nnsContractState()
if err == nil {
if nnsCs.NEF.Checksum == cs.NEF.Checksum {
if method == deployMethodName {
c.Command.Println("NNS contract is already deployed.")
} else {
c.Command.Println("NNS contract is already updated.")
}
return nil
}
h = nnsCs.Hash
@ -119,7 +128,7 @@ func (c *initializeContext) deployNNS(method string) error {
Scopes: transaction.CalledByEntry,
}
invokeHash := c.nativeHash(nativenames.Management)
invokeHash := management.Hash
if method == updateMethodName {
invokeHash = nnsCs.Hash
}
@ -128,11 +137,11 @@ func (c *initializeContext) deployNNS(method string) error {
if err != nil {
return fmt.Errorf("can't deploy NNS contract: %w", err)
}
if res.State != vm.HaltState.String() {
if res.State != vmstate.Halt.String() {
return fmt.Errorf("can't deploy NNS contract: %s", res.FaultException)
}
tx, err := c.Client.CreateTxFromScript(res.Script, c.CommitteeAcc, res.GasConsumed, 0, []client.SignerAccount{{
tx, err := c.Client.CreateTxFromScript(res.Script, c.CommitteeAcc, res.GasConsumed, 0, []rpcclient.SignerAccount{{
Signer: signer,
Account: c.CommitteeAcc,
}})
@ -148,7 +157,6 @@ func (c *initializeContext) deployNNS(method string) error {
}
func (c *initializeContext) updateContracts() error {
mgmtHash := c.nativeHash(nativenames.Management)
alphaCs := c.getContract(alphabetContract)
nnsCs, err := c.nnsContractState()
@ -157,28 +165,28 @@ func (c *initializeContext) updateContracts() error {
}
nnsHash := nnsCs.Hash
totalGasCost := int64(0)
w := io2.NewBufBinWriter()
var keysParam []interface{}
var keysParam []any
// Update script size for a single-node committee is close to the maximum allowed size of 65535.
// Because of this we want to reuse alphabet contract NEF and manifest for different updates.
// The generated script is as following.
// 1. Initialize static slots for alphabet NEF and manifest.
// 2. Store NEF and manifest into static slots.
// 1. Initialize static slot for alphabet NEF.
// 2. Store NEF into the static slot.
// 3. Push parameters for each alphabet contract on stack.
// 4. For each alphabet contract, invoke `update` using parameters on stack and
// NEF and manifest from step 2.
// 5. Update other contracts as usual.
emit.Instruction(w.BinWriter, opcode.INITSSLOT, []byte{2})
emit.Bytes(w.BinWriter, alphaCs.RawManifest)
// 4. Add contract group to the manifest.
// 5. For each alphabet contract, invoke `update` using parameters on stack and
// NEF from step 2 and manifest from step 4.
emit.Instruction(w.BinWriter, opcode.INITSSLOT, []byte{1})
emit.Bytes(w.BinWriter, alphaCs.RawNEF)
emit.Opcodes(w.BinWriter, opcode.STSFLD0, opcode.STSFLD1)
emit.Opcodes(w.BinWriter, opcode.STSFLD0)
baseGroups := alphaCs.Manifest.Groups
// alphabet contracts should be deployed by individual nodes to get different hashes.
for i, acc := range c.Accounts {
ctrHash, err := nnsResolveHash(c.Client, nnsHash, getAlphabetNNSDomain(i))
ctrHash, err := nnsResolveHash(c.ReadOnlyInvoker, nnsHash, getAlphabetNNSDomain(i))
if err != nil {
return fmt.Errorf("can't resolve hash for contract update: %w", err)
}
@ -187,32 +195,39 @@ func (c *initializeContext) updateContracts() error {
params := c.getAlphabetDeployItems(i, len(c.Wallets))
emit.Array(w.BinWriter, params...)
emit.Opcodes(w.BinWriter, opcode.LDSFLD1, opcode.LDSFLD0)
alphaCs.Manifest.Groups = baseGroups
err = c.addManifestGroup(ctrHash, alphaCs)
if err != nil {
return fmt.Errorf("can't sign manifest group: %v", err)
}
emit.Bytes(w.BinWriter, alphaCs.RawManifest)
emit.Opcodes(w.BinWriter, opcode.LDSFLD0)
emit.Int(w.BinWriter, 3)
emit.Opcodes(w.BinWriter, opcode.PACK)
emit.AppCallNoArgs(w.BinWriter, ctrHash, updateMethodName, callflag.All)
}
res, err := c.Client.InvokeScript(w.Bytes(), []transaction.Signer{{
Account: c.CommitteeAcc.Contract.ScriptHash(),
Scopes: transaction.Global,
}})
if err != nil {
return fmt.Errorf("can't update alphabet contracts: %w", err)
if err := c.sendCommitteeTx(w.Bytes(), false); err != nil {
if !strings.Contains(err.Error(), common.ErrAlreadyUpdated) {
return err
}
if res.State != vm.HaltState.String() {
return fmt.Errorf("can't update alphabet contracts: %s", res.FaultException)
c.Command.Println("Alphabet contracts are already updated.")
}
w.Reset()
totalGasCost += res.GasConsumed
w.WriteBytes(res.Script)
emit.Instruction(w.BinWriter, opcode.INITSSLOT, []byte{1})
emit.AppCall(w.BinWriter, nnsHash, "getPrice", callflag.All)
emit.Opcodes(w.BinWriter, opcode.STSFLD0)
emit.AppCall(w.BinWriter, nnsHash, "setPrice", callflag.All, 1)
for _, ctrName := range contractList {
cs := c.getContract(ctrName)
method := updateMethodName
ctrHash, err := nnsResolveHash(c.Client, nnsHash, ctrName+".neofs")
ctrHash, err := nnsResolveHash(c.ReadOnlyInvoker, nnsHash, ctrName+".frostfs")
if err != nil {
if errors.Is(err, errMissingNNSRecord) {
// if contract not found we deploy it instead of update
@ -222,67 +237,69 @@ func (c *initializeContext) updateContracts() error {
}
}
err = c.addManifestGroup(ctrHash, alphaCs)
err = c.addManifestGroup(ctrHash, cs)
if err != nil {
return fmt.Errorf("can't sign manifest group: %v", err)
}
invokeHash := mgmtHash
invokeHash := management.Hash
if method == updateMethodName {
invokeHash = ctrHash
}
params := getContractDeployParameters(cs, c.getContractDeployData(ctrName, keysParam))
signer := transaction.Signer{
Account: c.CommitteeAcc.Contract.ScriptHash(),
Scopes: transaction.Global,
}
res, err := invokeFunction(c.Client, invokeHash, method, params, []transaction.Signer{signer})
res, err := c.CommitteeAct.MakeCall(invokeHash, method, params...)
if err != nil {
return fmt.Errorf("can't deploy %s contract: %w", ctrName, err)
if method != updateMethodName || !strings.Contains(err.Error(), common.ErrAlreadyUpdated) {
return fmt.Errorf("deploy contract: %w", err)
}
if res.State != vm.HaltState.String() {
return fmt.Errorf("can't deploy %s contract: %s", ctrName, res.FaultException)
c.Command.Printf("%s contract is already updated.\n", ctrName)
continue
}
totalGasCost += res.GasConsumed
w.WriteBytes(res.Script)
if method == deployMethodName {
// same actions are done in initializeContext.setNNS, can be unified
domain := ctrName + ".neofs"
script, err := c.nnsRegisterDomainScript(nnsHash, cs.Hash, domain)
domain := ctrName + ".frostfs"
script, ok, err := c.nnsRegisterDomainScript(nnsHash, cs.Hash, domain)
if err != nil {
return err
}
if script != nil {
totalGasCost += defaultRegisterSysfee + native.GASFactor
if !ok {
w.WriteBytes(script)
emit.AppCall(w.BinWriter, nnsHash, "deleteRecords", callflag.All, domain, int64(nns.TXT))
emit.AppCall(w.BinWriter, nnsHash, "addRecord", callflag.All,
domain, int64(nns.TXT), cs.Hash.StringLE())
emit.AppCall(w.BinWriter, nnsHash, "addRecord", callflag.All,
domain, int64(nns.TXT), address.Uint160ToString(cs.Hash))
}
c.Command.Printf("NNS: Set %s -> %s\n", domain, cs.Hash.StringLE())
}
}
groupKey := c.ContractWallet.Accounts[0].PrivateKey().PublicKey()
sysFee, err := c.emitUpdateNNSGroupScript(w, nnsHash, groupKey)
_, _, err = c.emitUpdateNNSGroupScript(w, nnsHash, groupKey)
if err != nil {
return err
}
c.Command.Printf("NNS: Set %s -> %s\n", morphClient.NNSGroupKeyName, hex.EncodeToString(groupKey.Bytes()))
totalGasCost += sysFee
if err := c.sendCommitteeTx(w.Bytes(), totalGasCost, false); err != nil {
emit.Opcodes(w.BinWriter, opcode.LDSFLD0)
emit.Int(w.BinWriter, 1)
emit.Opcodes(w.BinWriter, opcode.PACK)
emit.AppCallNoArgs(w.BinWriter, nnsHash, "setPrice", callflag.All)
if err := c.sendCommitteeTx(w.Bytes(), false); err != nil {
return err
}
return c.awaitTx()
}
func (c *initializeContext) deployContracts() error {
mgmtHash := c.nativeHash(nativenames.Management)
alphaCs := c.getContract(alphabetContract)
var keysParam []interface{}
var keysParam []any
baseGroups := alphaCs.Manifest.Groups
@ -303,20 +320,17 @@ func (c *initializeContext) deployContracts() error {
keysParam = append(keysParam, acc.PrivateKey().PublicKey().Bytes())
params := getContractDeployParameters(alphaCs, c.getAlphabetDeployItems(i, len(c.Wallets)))
res, err := invokeFunction(c.Client, mgmtHash, deployMethodName, params, []transaction.Signer{{
Account: acc.Contract.ScriptHash(),
Scopes: transaction.CalledByEntry,
}})
act, err := actor.NewSimple(c.Client, acc)
if err != nil {
return fmt.Errorf("could not create actor: %w", err)
}
txHash, vub, err := act.SendCall(management.Hash, deployMethodName, params...)
if err != nil {
return fmt.Errorf("can't deploy alphabet #%d contract: %w", i, err)
}
if res.State != vm.HaltState.String() {
return fmt.Errorf("can't deploy alpabet #%d contract: %s", i, res.FaultException)
}
if err := c.sendSingleTx(res.Script, res.GasConsumed, acc); err != nil {
return err
}
c.SentTxs = append(c.SentTxs, hashVUBPair{hash: txHash, vub: vub})
}
for _, ctrName := range contractList {
@ -334,20 +348,12 @@ func (c *initializeContext) deployContracts() error {
}
params := getContractDeployParameters(cs, c.getContractDeployData(ctrName, keysParam))
signer := transaction.Signer{
Account: c.CommitteeAcc.Contract.ScriptHash(),
Scopes: transaction.Global,
}
res, err := invokeFunction(c.Client, mgmtHash, deployMethodName, params, []transaction.Signer{signer})
res, err := c.CommitteeAct.MakeCall(management.Hash, deployMethodName, params...)
if err != nil {
return fmt.Errorf("can't deploy %s contract: %w", ctrName, err)
}
if res.State != vm.HaltState.String() {
return fmt.Errorf("can't deploy %s contract: %s", ctrName, res.FaultException)
}
if err := c.sendCommitteeTx(res.Script, res.GasConsumed, false); err != nil {
if err := c.sendCommitteeTx(res.Script, false); err != nil {
return err
}
}
@ -378,14 +384,9 @@ func (c *initializeContext) readContracts(names []string) error {
if c.ContractPath != "" && fi.IsDir() {
for _, ctrName := range names {
cs := new(contractState)
cs.RawNEF, err = os.ReadFile(filepath.Join(c.ContractPath, ctrName, ctrName+"_contract.nef"))
cs, err := readContract(filepath.Join(c.ContractPath, ctrName), ctrName)
if err != nil {
return fmt.Errorf("can't read NEF file for %s contract: %w", ctrName, err)
}
cs.RawManifest, err = os.ReadFile(filepath.Join(c.ContractPath, ctrName, "config.json"))
if err != nil {
return fmt.Errorf("can't read manifest file for %s contract: %w", ctrName, err)
return err
}
c.Contracts[ctrName] = cs
}
@ -407,17 +408,16 @@ func (c *initializeContext) readContracts(names []string) error {
return err
}
for _, name := range names {
if err := m[name].parse(); err != nil {
return err
}
c.Contracts[name] = m[name]
}
}
for _, ctrName := range names {
cs := c.Contracts[ctrName]
if err := cs.parse(); err != nil {
return err
}
if ctrName != alphabetContract {
cs := c.Contracts[ctrName]
cs.Hash = state.CreateContractHash(c.CommitteeAcc.Contract.ScriptHash(),
cs.NEF.Checksum, cs.Manifest.Name)
}
@ -425,6 +425,24 @@ func (c *initializeContext) readContracts(names []string) error {
return nil
}
func readContract(ctrPath, ctrName string) (*contractState, error) {
rawNef, err := os.ReadFile(filepath.Join(ctrPath, ctrName+"_contract.nef"))
if err != nil {
return nil, fmt.Errorf("can't read NEF file for %s contract: %w", ctrName, err)
}
rawManif, err := os.ReadFile(filepath.Join(ctrPath, "config.json"))
if err != nil {
return nil, fmt.Errorf("can't read manifest file for %s contract: %w", ctrName, err)
}
cs := &contractState{
RawNEF: rawNef,
RawManifest: rawManif,
}
return cs, cs.parse()
}
func (cs *contractState) parse() error {
nf, err := nef.FileFromBytes(cs.RawNEF)
if err != nil {
@ -492,22 +510,22 @@ func readContractsFromArchive(file io.Reader, names []string) (map[string]*contr
return m, nil
}
func getContractDeployParameters(cs *contractState, deployData []interface{}) []interface{} {
return []interface{}{cs.RawNEF, cs.RawManifest, deployData}
func getContractDeployParameters(cs *contractState, deployData []any) []any {
return []any{cs.RawNEF, cs.RawManifest, deployData}
}
func (c *initializeContext) getContractDeployData(ctrName string, keysParam []interface{}) []interface{} {
items := make([]interface{}, 1, 6)
func (c *initializeContext) getContractDeployData(ctrName string, keysParam []any) []any {
items := make([]any, 1, 6)
items[0] = false // notaryDisabled is false
switch ctrName {
case neofsContract:
case frostfsContract:
items = append(items,
c.Contracts[processingContract].Hash,
keysParam,
smartcontract.Parameter{})
case processingContract:
items = append(items, c.Contracts[neofsContract].Hash)
items = append(items, c.Contracts[frostfsContract].Hash)
return items[1:] // no notary info
case auditContract:
items = append(items, c.Contracts[netmapContract].Hash)
@ -525,15 +543,15 @@ func (c *initializeContext) getContractDeployData(ctrName string, keysParam []in
items = append(items,
c.Contracts[netmapContract].Hash,
c.Contracts[balanceContract].Hash,
c.Contracts[neofsIDContract].Hash,
c.Contracts[frostfsIDContract].Hash,
nnsCs.Hash,
"container")
case neofsIDContract:
case frostfsIDContract:
items = append(items,
c.Contracts[netmapContract].Hash,
c.Contracts[containerContract].Hash)
case netmapContract:
configParam := []interface{}{
configParam := []any{
netmapEpochKey, viper.GetInt64(epochDurationInitFlag),
netmapMaxObjectSizeKey, viper.GetInt64(maxObjectSizeInitFlag),
netmapAuditFeeKey, viper.GetInt64(auditFeeInitFlag),
@ -544,6 +562,8 @@ func (c *initializeContext) getContractDeployData(ctrName string, keysParam []in
netmapBasicIncomeRateKey, viper.GetInt64(incomeRateInitFlag),
netmapInnerRingCandidateFeeKey, viper.GetInt64(candidateFeeInitFlag),
netmapWithdrawFeeKey, viper.GetInt64(withdrawFeeInitFlag),
netmapHomomorphicHashDisabledKey, viper.GetBool(homomorphicHashDisabledInitFlag),
netmapMaintenanceAllowedKey, viper.GetBool(maintenanceModeAllowedInitFlag),
}
items = append(items,
c.Contracts[balanceContract].Hash,
@ -560,8 +580,8 @@ func (c *initializeContext) getContractDeployData(ctrName string, keysParam []in
return items
}
func (c *initializeContext) getAlphabetDeployItems(i, n int) []interface{} {
items := make([]interface{}, 6)
func (c *initializeContext) getAlphabetDeployItems(i, n int) []any {
items := make([]any, 6)
items[0] = false
items[1] = c.Contracts[netmapContract].Hash
items[2] = c.Contracts[proxyContract].Hash

View file

@ -0,0 +1,291 @@
package morph
import (
"encoding/hex"
"errors"
"fmt"
"strconv"
"time"
"github.com/TrueCloudLab/frostfs-contract/nns"
morphClient "github.com/TrueCloudLab/frostfs-node/pkg/morph/client"
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/opcode"
"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
"github.com/nspcc-dev/neo-go/pkg/vm/vmstate"
)
const defaultExpirationTime = 10 * 365 * 24 * time.Hour / time.Second
func (c *initializeContext) setNNS() error {
nnsCs, err := c.Client.GetContractStateByID(1)
if err != nil {
return err
}
ok, err := c.nnsRootRegistered(nnsCs.Hash, "frostfs")
if err != nil {
return err
} else if !ok {
bw := io.NewBufBinWriter()
emit.AppCall(bw.BinWriter, nnsCs.Hash, "register", callflag.All,
"frostfs", c.CommitteeAcc.Contract.ScriptHash(),
"ops@nspcc.ru", int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
if err := c.sendCommitteeTx(bw.Bytes(), true); err != nil {
return fmt.Errorf("can't add domain root to NNS: %w", err)
}
if err := c.awaitTx(); err != nil {
return err
}
}
alphaCs := c.getContract(alphabetContract)
for i, acc := range c.Accounts {
alphaCs.Hash = state.CreateContractHash(acc.Contract.ScriptHash(), alphaCs.NEF.Checksum, alphaCs.Manifest.Name)
domain := getAlphabetNNSDomain(i)
if err := c.nnsRegisterDomain(nnsCs.Hash, alphaCs.Hash, domain); err != nil {
return err
}
c.Command.Printf("NNS: Set %s -> %s\n", domain, alphaCs.Hash.StringLE())
}
for _, ctrName := range contractList {
cs := c.getContract(ctrName)
domain := ctrName + ".frostfs"
if err := c.nnsRegisterDomain(nnsCs.Hash, cs.Hash, domain); err != nil {
return err
}
c.Command.Printf("NNS: Set %s -> %s\n", domain, cs.Hash.StringLE())
}
groupKey := c.ContractWallet.Accounts[0].PrivateKey().PublicKey()
err = c.updateNNSGroup(nnsCs.Hash, groupKey)
if err != nil {
return err
}
c.Command.Printf("NNS: Set %s -> %s\n", morphClient.NNSGroupKeyName, hex.EncodeToString(groupKey.Bytes()))
return c.awaitTx()
}
func (c *initializeContext) updateNNSGroup(nnsHash util.Uint160, pub *keys.PublicKey) error {
bw := io.NewBufBinWriter()
needUpdate, needRegister, err := c.emitUpdateNNSGroupScript(bw, nnsHash, pub)
if !needUpdate || err != nil {
return err
}
script := bw.Bytes()
if needRegister {
w := io.NewBufBinWriter()
emit.Instruction(w.BinWriter, opcode.INITSSLOT, []byte{1})
wrapRegisterScriptWithPrice(w, nnsHash, script)
script = w.Bytes()
}
return c.sendCommitteeTx(script, true)
}
// emitUpdateNNSGroupScript emits script for updating group key stored in NNS.
// First return value is true iff the key is already there and nothing should be done.
// Second return value is true iff a domain registration code was emitted.
func (c *initializeContext) emitUpdateNNSGroupScript(bw *io.BufBinWriter, nnsHash util.Uint160, pub *keys.PublicKey) (bool, bool, error) {
isAvail, err := nnsIsAvailable(c.Client, nnsHash, morphClient.NNSGroupKeyName)
if err != nil {
return false, false, err
}
if !isAvail {
currentPub, err := nnsResolveKey(c.ReadOnlyInvoker, nnsHash, morphClient.NNSGroupKeyName)
if err != nil {
return false, false, err
}
if pub.Equal(currentPub) {
return true, false, nil
}
}
if isAvail {
emit.AppCall(bw.BinWriter, nnsHash, "register", callflag.All,
morphClient.NNSGroupKeyName, c.CommitteeAcc.Contract.ScriptHash(),
"ops@nspcc.ru", int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
}
emit.AppCall(bw.BinWriter, nnsHash, "deleteRecords", callflag.All, "group.frostfs", int64(nns.TXT))
emit.AppCall(bw.BinWriter, nnsHash, "addRecord", callflag.All,
"group.frostfs", int64(nns.TXT), hex.EncodeToString(pub.Bytes()))
return false, isAvail, nil
}
func getAlphabetNNSDomain(i int) string {
return alphabetContract + strconv.FormatUint(uint64(i), 10) + ".frostfs"
}
// wrapRegisterScriptWithPrice wraps a given script with `getPrice`/`setPrice` calls for NNS.
// It is intended to be used for a single transaction, and not as a part of other scripts.
// It is assumed that script already contains static slot initialization code, the first one
// (with index 0) is used to store the price.
func wrapRegisterScriptWithPrice(w *io.BufBinWriter, nnsHash util.Uint160, s []byte) {
if len(s) == 0 {
return
}
emit.AppCall(w.BinWriter, nnsHash, "getPrice", callflag.All)
emit.Opcodes(w.BinWriter, opcode.STSFLD0)
emit.AppCall(w.BinWriter, nnsHash, "setPrice", callflag.All, 1)
w.WriteBytes(s)
emit.Opcodes(w.BinWriter, opcode.LDSFLD0, opcode.PUSH1, opcode.PACK)
emit.AppCallNoArgs(w.BinWriter, nnsHash, "setPrice", callflag.All)
if w.Err != nil {
panic(fmt.Errorf("BUG: can't wrap register script: %w", w.Err))
}
}
func (c *initializeContext) nnsRegisterDomainScript(nnsHash, expectedHash util.Uint160, domain string) ([]byte, bool, error) {
ok, err := nnsIsAvailable(c.Client, nnsHash, domain)
if err != nil {
return nil, false, err
}
if ok {
bw := io.NewBufBinWriter()
emit.AppCall(bw.BinWriter, nnsHash, "register", callflag.All,
domain, c.CommitteeAcc.Contract.ScriptHash(),
"ops@nspcc.ru", int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
if bw.Err != nil {
panic(bw.Err)
}
return bw.Bytes(), false, nil
}
s, err := nnsResolveHash(c.ReadOnlyInvoker, nnsHash, domain)
if err != nil {
return nil, false, err
}
return nil, s == expectedHash, nil
}
func (c *initializeContext) nnsRegisterDomain(nnsHash, expectedHash util.Uint160, domain string) error {
script, ok, err := c.nnsRegisterDomainScript(nnsHash, expectedHash, domain)
if ok || err != nil {
return err
}
w := io.NewBufBinWriter()
emit.Instruction(w.BinWriter, opcode.INITSSLOT, []byte{1})
wrapRegisterScriptWithPrice(w, nnsHash, script)
emit.AppCall(w.BinWriter, nnsHash, "deleteRecords", callflag.All, domain, int64(nns.TXT))
emit.AppCall(w.BinWriter, nnsHash, "addRecord", callflag.All,
domain, int64(nns.TXT), expectedHash.StringLE())
emit.AppCall(w.BinWriter, nnsHash, "addRecord", callflag.All,
domain, int64(nns.TXT), address.Uint160ToString(expectedHash))
return c.sendCommitteeTx(w.Bytes(), true)
}
func (c *initializeContext) nnsRootRegistered(nnsHash util.Uint160, zone string) (bool, error) {
res, err := c.CommitteeAct.Call(nnsHash, "isAvailable", "name."+zone)
if err != nil {
return false, err
}
return res.State == vmstate.Halt.String(), nil
}
var errMissingNNSRecord = errors.New("missing NNS record")
// Returns errMissingNNSRecord if invocation fault exception contains "token not found".
func nnsResolveHash(inv *invoker.Invoker, nnsHash util.Uint160, domain string) (util.Uint160, error) {
item, err := nnsResolve(inv, nnsHash, domain)
if err != nil {
return util.Uint160{}, err
}
return parseNNSResolveResult(item)
}
func nnsResolve(inv *invoker.Invoker, nnsHash util.Uint160, domain string) (stackitem.Item, error) {
return unwrap.Item(inv.Call(nnsHash, "resolve", domain, int64(nns.TXT)))
}
func nnsResolveKey(inv *invoker.Invoker, nnsHash util.Uint160, domain string) (*keys.PublicKey, error) {
item, err := nnsResolve(inv, nnsHash, domain)
if err != nil {
return nil, err
}
v, ok := item.Value().(stackitem.Null)
if ok {
return nil, errors.New("NNS record is missing")
}
bs, err := v.TryBytes()
if err != nil {
return nil, errors.New("malformed response")
}
return keys.NewPublicKeyFromString(string(bs))
}
// parseNNSResolveResult parses the result of resolving NNS record.
// It works with multiple formats (corresponding to multiple NNS versions).
// If array of hashes is provided, it returns only the first one.
func parseNNSResolveResult(res stackitem.Item) (util.Uint160, error) {
arr, ok := res.Value().([]stackitem.Item)
if !ok {
arr = []stackitem.Item{res}
}
if _, ok := res.Value().(stackitem.Null); ok || len(arr) == 0 {
return util.Uint160{}, errors.New("NNS record is missing")
}
for i := range arr {
bs, err := arr[i].TryBytes()
if err != nil {
continue
}
// We support several formats for hash encoding, this logic should be maintained in sync
// with nnsResolve from pkg/morph/client/nns.go
h, err := util.Uint160DecodeStringLE(string(bs))
if err == nil {
return h, nil
}
h, err = address.StringToUint160(string(bs))
if err == nil {
return h, nil
}
}
return util.Uint160{}, errors.New("no valid hashes are found")
}
func nnsIsAvailable(c Client, nnsHash util.Uint160, name string) (bool, error) {
switch ct := c.(type) {
case *rpcclient.Client:
return ct.NNSIsAvailable(nnsHash, name)
default:
b, err := unwrap.Bool(invokeFunction(c, nnsHash, "isAvailable", []any{name}, nil))
if err != nil {
return false, fmt.Errorf("`isAvailable`: invalid response: %w", err)
}
return b, nil
}
}

View file

@ -5,56 +5,85 @@ import (
"fmt"
"github.com/nspcc-dev/neo-go/pkg/core/native"
"github.com/nspcc-dev/neo-go/pkg/core/native/nativenames"
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpc/client"
"github.com/nspcc-dev/neo-go/pkg/rpcclient"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/neo"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/opcode"
"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
)
// initialAlphabetNEOAmount represents the total amount of GAS distributed between alphabet nodes.
const initialAlphabetNEOAmount = native.NEOTotalSupply
func (c *initializeContext) registerCandidates() error {
neoHash := c.nativeHash(nativenames.Neo)
neoHash := neo.Hash
res, err := invokeFunction(c.Client, neoHash, "getCandidates", nil, nil)
cc, err := unwrap.Array(c.ReadOnlyInvoker.Call(neoHash, "getCandidates"))
if err != nil {
return err
return fmt.Errorf("`getCandidates`: %w", err)
}
if res.State == vm.HaltState.String() && len(res.Stack) > 0 {
arr, ok := res.Stack[0].Value().([]stackitem.Item)
if ok && len(arr) > 0 {
if len(cc) > 0 {
c.Command.Println("Candidates are already registered.")
return nil
}
}
regPrice, err := c.getCandidateRegisterPrice()
if err != nil {
return fmt.Errorf("can't fetch registration price: %w", err)
}
sysGas := regPrice + native.GASFactor // + 1 GAS
for _, acc := range c.Accounts {
w := io.NewBufBinWriter()
emit.AppCall(w.BinWriter, neoHash, "setRegisterPrice", callflag.States, 1)
for _, acc := range c.Accounts {
emit.AppCall(w.BinWriter, neoHash, "registerCandidate", callflag.States, acc.PrivateKey().PublicKey().Bytes())
emit.Opcodes(w.BinWriter, opcode.ASSERT)
}
emit.AppCall(w.BinWriter, neoHash, "setRegisterPrice", callflag.States, regPrice)
if w.Err != nil {
panic(fmt.Sprintf("BUG: %v", w.Err))
}
if err := c.sendSingleTx(w.Bytes(), sysGas, acc); err != nil {
return err
signers := []rpcclient.SignerAccount{{
Signer: c.getSigner(false, c.CommitteeAcc),
Account: c.CommitteeAcc,
}}
for i := range c.Accounts {
signers = append(signers, rpcclient.SignerAccount{
Signer: transaction.Signer{
Account: c.Accounts[i].Contract.ScriptHash(),
Scopes: transaction.CustomContracts,
AllowedContracts: []util.Uint160{neoHash},
},
Account: c.Accounts[i],
})
}
tx, err := c.Client.CreateTxFromScript(w.Bytes(), c.CommitteeAcc, -1, 0, signers)
if err != nil {
return fmt.Errorf("can't create tx: %w", err)
}
if err := c.multiSign(tx, committeeAccountName); err != nil {
return fmt.Errorf("can't sign a transaction: %w", err)
}
network := c.CommitteeAct.GetNetwork()
for i := range c.Accounts {
if err := c.Accounts[i].SignTx(network, tx); err != nil {
return fmt.Errorf("can't sign a transaction: %w", err)
}
}
return c.awaitTx()
return c.sendTx(tx, c.Command, true)
}
func (c *initializeContext) transferNEOToAlphabetContracts() error {
neoHash := c.nativeHash(nativenames.Neo)
neoHash := neo.Hash
ok, err := c.transferNEOFinished(neoHash)
if ok || err != nil {
@ -72,7 +101,7 @@ func (c *initializeContext) transferNEOToAlphabetContracts() error {
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
}
if err := c.sendCommitteeTx(bw.Bytes(), -1, false); err != nil {
if err := c.sendCommitteeTx(bw.Bytes(), false); err != nil {
return err
}
@ -88,10 +117,10 @@ var errGetPriceInvalid = errors.New("`getRegisterPrice`: invalid response")
func (c *initializeContext) getCandidateRegisterPrice() (int64, error) {
switch ct := c.Client.(type) {
case *client.Client:
case *rpcclient.Client:
return ct.GetCandidateRegisterPrice()
default:
neoHash := c.nativeHash(nativenames.Neo)
neoHash := neo.Hash
res, err := invokeFunction(c.Client, neoHash, "getRegisterPrice", nil, nil)
if err != nil {
return 0, err

View file

@ -1,9 +1,9 @@
package morph
import (
"github.com/nspcc-dev/neo-go/pkg/core/native/nativenames"
"github.com/nspcc-dev/neo-go/pkg/core/native/noderoles"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/rolemgmt"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
)
@ -16,20 +16,18 @@ func (c *initializeContext) setNotaryAndAlphabetNodes() error {
return err
}
designateHash := c.nativeHash(nativenames.Designation)
var pubs []interface{}
var pubs []any
for _, acc := range c.Accounts {
pubs = append(pubs, acc.PrivateKey().PublicKey().Bytes())
}
w := io.NewBufBinWriter()
emit.AppCall(w.BinWriter, designateHash, "designateAsRole",
emit.AppCall(w.BinWriter, rolemgmt.Hash, "designateAsRole",
callflag.States|callflag.AllowNotify, int64(noderoles.P2PNotary), pubs)
emit.AppCall(w.BinWriter, designateHash, "designateAsRole",
emit.AppCall(w.BinWriter, rolemgmt.Hash, "designateAsRole",
callflag.States|callflag.AllowNotify, int64(noderoles.NeoFSAlphabet), pubs)
if err := c.sendCommitteeTx(w.Bytes(), -1, false); err != nil {
if err := c.sendCommitteeTx(w.Bytes(), false); err != nil {
return err
}
@ -42,7 +40,6 @@ func (c *initializeContext) setRolesFinished() (bool, error) {
return false, err
}
h := c.nativeHash(nativenames.Designation)
pubs, err := getDesignatedByRole(c.Client, h, noderoles.NeoFSAlphabet, height)
pubs, err := getDesignatedByRole(c.ReadOnlyInvoker, rolemgmt.Hash, noderoles.NeoFSAlphabet, height)
return len(pubs) == len(c.Wallets), err
}

View file

@ -0,0 +1,121 @@
package morph
import (
"encoding/hex"
"os"
"path/filepath"
"strconv"
"testing"
"github.com/TrueCloudLab/frostfs-node/pkg/innerring"
"github.com/nspcc-dev/neo-go/pkg/config"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/vm"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/viper"
"github.com/stretchr/testify/require"
"gopkg.in/yaml.v3"
)
const (
contractsPath = "../../../../../../frostfs-contract/frostfs-contract-v0.16.0.tar.gz"
protoFileName = "proto.yml"
)
func TestInitialize(t *testing.T) {
// This test needs frostfs-contract tarball, so it is skipped by default.
// It is here for performing local testing after the changes.
t.Skip()
t.Run("1 nodes", func(t *testing.T) {
testInitialize(t, 1)
})
t.Run("4 nodes", func(t *testing.T) {
testInitialize(t, 4)
})
t.Run("7 nodes", func(t *testing.T) {
testInitialize(t, 7)
})
}
func testInitialize(t *testing.T, committeeSize int) {
testdataDir := t.TempDir()
v := viper.GetViper()
generateTestData(t, testdataDir, committeeSize)
v.Set(protoConfigPath, filepath.Join(testdataDir, protoFileName))
// Set to the path or remove the next statement to download from the network.
require.NoError(t, initCmd.Flags().Set(contractsInitFlag, contractsPath))
v.Set(localDumpFlag, filepath.Join(testdataDir, "out"))
v.Set(alphabetWalletsFlag, testdataDir)
v.Set(epochDurationInitFlag, 1)
v.Set(maxObjectSizeInitFlag, 1024)
setTestCredentials(v, committeeSize)
require.NoError(t, initializeSideChainCmd(initCmd, nil))
t.Run("force-new-epoch", func(t *testing.T) {
require.NoError(t, forceNewEpochCmd(forceNewEpoch, nil))
})
t.Run("set-config", func(t *testing.T) {
require.NoError(t, setConfigCmd(setConfig, []string{"MaintenanceModeAllowed=true"}))
})
t.Run("set-policy", func(t *testing.T) {
require.NoError(t, setPolicyCmd(setPolicy, []string{"ExecFeeFactor=1"}))
})
t.Run("remove-node", func(t *testing.T) {
pk, err := keys.NewPrivateKey()
require.NoError(t, err)
pub := hex.EncodeToString(pk.PublicKey().Bytes())
require.NoError(t, removeNodesCmd(removeNodes, []string{pub}))
})
}
func generateTestData(t *testing.T, dir string, size int) {
v := viper.GetViper()
v.Set(alphabetWalletsFlag, dir)
sizeStr := strconv.FormatUint(uint64(size), 10)
require.NoError(t, generateAlphabetCmd.Flags().Set(alphabetSizeFlag, sizeStr))
setTestCredentials(v, size)
require.NoError(t, generateAlphabetCreds(generateAlphabetCmd, nil))
var pubs []string
for i := 0; i < size; i++ {
p := filepath.Join(dir, innerring.GlagoliticLetter(i).String()+".json")
w, err := wallet.NewWalletFromFile(p)
require.NoError(t, err, "wallet doesn't exist")
for _, acc := range w.Accounts {
if acc.Label == singleAccountName {
pub, ok := vm.ParseSignatureContract(acc.Contract.Script)
require.True(t, ok)
pubs = append(pubs, hex.EncodeToString(pub))
continue
}
}
}
cfg := config.Config{}
cfg.ProtocolConfiguration.Magic = 12345
cfg.ProtocolConfiguration.ValidatorsCount = size
cfg.ProtocolConfiguration.SecondsPerBlock = 1
cfg.ProtocolConfiguration.StandbyCommittee = pubs // sorted by glagolic letters
cfg.ProtocolConfiguration.P2PSigExtensions = true
cfg.ProtocolConfiguration.VerifyTransactions = true
cfg.ProtocolConfiguration.VerifyBlocks = true
data, err := yaml.Marshal(cfg)
require.NoError(t, err)
protoPath := filepath.Join(dir, protoFileName)
require.NoError(t, os.WriteFile(protoPath, data, os.ModePerm))
}
func setTestCredentials(v *viper.Viper, size int) {
for i := 0; i < size; i++ {
v.Set("credentials."+innerring.GlagoliticLetter(i).String(), strconv.FormatUint(uint64(i), 10))
}
v.Set("credentials.contract", testContractPassword)
}

View file

@ -4,10 +4,11 @@ import (
"fmt"
"github.com/nspcc-dev/neo-go/pkg/core/native"
"github.com/nspcc-dev/neo-go/pkg/core/native/nativenames"
"github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpc/client"
"github.com/nspcc-dev/neo-go/pkg/rpcclient"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/gas"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/neo"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
scContext "github.com/nspcc-dev/neo-go/pkg/smartcontract/context"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
@ -32,15 +33,12 @@ func (c *initializeContext) transferFunds() error {
return err
}
gasHash := c.nativeHash(nativenames.Gas)
neoHash := c.nativeHash(nativenames.Neo)
var transfers []client.TransferTarget
var transfers []rpcclient.TransferTarget
for _, acc := range c.Accounts {
to := acc.Contract.ScriptHash()
transfers = append(transfers,
client.TransferTarget{
Token: gasHash,
rpcclient.TransferTarget{
Token: gas.Hash,
Address: to,
Amount: initialAlphabetGASAmount,
},
@ -49,19 +47,19 @@ func (c *initializeContext) transferFunds() error {
// It is convenient to have all funds at the committee account.
transfers = append(transfers,
client.TransferTarget{
Token: gasHash,
rpcclient.TransferTarget{
Token: gas.Hash,
Address: c.CommitteeAcc.Contract.ScriptHash(),
Amount: (gasInitialTotalSupply - initialAlphabetGASAmount*int64(len(c.Wallets))) / 2,
},
client.TransferTarget{
Token: neoHash,
rpcclient.TransferTarget{
Token: neo.Hash,
Address: c.CommitteeAcc.Contract.ScriptHash(),
Amount: native.NEOTotalSupply,
},
)
tx, err := createNEP17MultiTransferTx(c.Client, c.ConsensusAcc, 0, transfers, []client.SignerAccount{{
tx, err := createNEP17MultiTransferTx(c.Client, c.ConsensusAcc, 0, transfers, []rpcclient.SignerAccount{{
Signer: transaction.Signer{
Account: c.ConsensusAcc.Contract.ScriptHash(),
Scopes: transaction.CalledByEntry,
@ -80,10 +78,9 @@ func (c *initializeContext) transferFunds() error {
}
func (c *initializeContext) transferFundsFinished() (bool, error) {
gasHash := c.nativeHash(nativenames.Gas)
acc := c.Accounts[0]
res, err := c.Client.NEP17BalanceOf(gasHash, acc.Contract.ScriptHash())
res, err := c.Client.NEP17BalanceOf(gas.Hash, acc.Contract.ScriptHash())
return res > initialAlphabetGASAmount/2, err
}
@ -125,26 +122,37 @@ func (c *initializeContext) multiSign(tx *transaction.Transaction, accType strin
}
}
w, err := pc.GetWitness(tx.Signers[0].Account)
w, err := pc.GetWitness(h)
if err != nil {
return fmt.Errorf("incomplete signature: %w", err)
}
tx.Scripts = append(tx.Scripts, *w)
for i := range tx.Signers {
if tx.Signers[i].Account == h {
if i < len(tx.Scripts) {
tx.Scripts[i] = *w
} else if i == len(tx.Scripts) {
tx.Scripts = append(tx.Scripts, *w)
} else {
panic("BUG: invalid signing order")
}
return nil
}
}
return fmt.Errorf("%s account was not found among transaction signers", accType)
}
func (c *initializeContext) transferGASToProxy() error {
gasHash := c.nativeHash(nativenames.Gas)
proxyCs := c.getContract(proxyContract)
bal, err := c.Client.NEP17BalanceOf(gasHash, proxyCs.Hash)
bal, err := c.Client.NEP17BalanceOf(gas.Hash, proxyCs.Hash)
if err != nil || bal > 0 {
return err
}
tx, err := createNEP17MultiTransferTx(c.Client, c.CommitteeAcc, 0, []client.TransferTarget{{
Token: gasHash,
tx, err := createNEP17MultiTransferTx(c.Client, c.CommitteeAcc, 0, []rpcclient.TransferTarget{{
Token: gas.Hash,
Address: proxyCs.Hash,
Amount: initialProxyGASAmount,
}}, nil)
@ -160,7 +168,7 @@ func (c *initializeContext) transferGASToProxy() error {
}
func createNEP17MultiTransferTx(c Client, acc *wallet.Account, netFee int64,
recipients []client.TransferTarget, cosigners []client.SignerAccount) (*transaction.Transaction, error) {
recipients []rpcclient.TransferTarget, cosigners []rpcclient.SignerAccount) (*transaction.Transaction, error) {
from := acc.Contract.ScriptHash()
w := io.NewBufBinWriter()
@ -172,7 +180,7 @@ func createNEP17MultiTransferTx(c Client, acc *wallet.Account, netFee int64,
if w.Err != nil {
return nil, fmt.Errorf("failed to create transfer script: %w", w.Err)
}
return c.CreateTxFromScript(w.Bytes(), acc, -1, netFee, append([]client.SignerAccount{{
return c.CreateTxFromScript(w.Bytes(), acc, -1, netFee, append([]rpcclient.SignerAccount{{
Signer: transaction.Signer{
Account: from,
Scopes: transaction.CalledByEntry,

View file

@ -18,7 +18,7 @@ func StringifySubnetClientGroupID(id *SubnetClientGroupID) string {
return string(text)
}
// MarshalText encodes SubnetClientGroupID into text format according to NeoFS API V2 protocol:
// MarshalText encodes SubnetClientGroupID into text format according to FrostFS API V2 protocol:
// value in base-10 integer string format.
//
// It implements encoding.TextMarshaler.
@ -28,7 +28,7 @@ func (x *SubnetClientGroupID) MarshalText() ([]byte, error) {
return []byte(strconv.FormatUint(uint64(num), 10)), nil
}
// UnmarshalText decodes the SubnetID from the text according to NeoFS API V2 protocol:
// UnmarshalText decodes the SubnetID from the text according to FrostFS API V2 protocol:
// should be base-10 integer string format with bitsize = 32.
//
// Returns strconv.ErrRange if integer overflows uint32.
@ -47,13 +47,13 @@ func (x *SubnetClientGroupID) UnmarshalText(txt []byte) error {
return nil
}
// Marshal encodes the SubnetClientGroupID into a binary format of NeoFS API V2 protocol
// Marshal encodes the SubnetClientGroupID into a binary format of FrostFS API V2 protocol
// (Protocol Buffers with direct field order).
func (x *SubnetClientGroupID) Marshal() ([]byte, error) {
return proto.Marshal(x)
}
// Unmarshal decodes the SubnetClientGroupID from NeoFS API V2 binary format (see Marshal). Must not be called on nil.
// Unmarshal decodes the SubnetClientGroupID from FrostFS API V2 binary format (see Marshal). Must not be called on nil.
func (x *SubnetClientGroupID) Unmarshal(data []byte) error {
return proto.Unmarshal(data, x)
}

Binary file not shown.

View file

@ -2,9 +2,9 @@ syntax = "proto3";
package neo.fs.v2.refs;
option go_package = "github.com/nspcc-dev/neofs-node/cmd/neofs-adm/internal/modules/morph/internal";
option go_package = "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph/internal";
// Client group identifier in the NeoFS subnet.
// Client group identifier in the FrostFS subnet.
//
// String representation of a value is base-10 integer.
//

View file

@ -1,13 +1,14 @@
package morph
import (
"bytes"
"crypto/elliptic"
"errors"
"fmt"
"os"
"sort"
"time"
"github.com/google/uuid"
"github.com/nspcc-dev/neo-go/pkg/config"
"github.com/nspcc-dev/neo-go/pkg/config/netmode"
"github.com/nspcc-dev/neo-go/pkg/core"
@ -18,21 +19,26 @@ import (
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/core/storage"
"github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/crypto/hash"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/nspcc-dev/neo-go/pkg/encoding/fixedn"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/neorpc/result"
"github.com/nspcc-dev/neo-go/pkg/network/payload"
"github.com/nspcc-dev/neo-go/pkg/rpc/client"
"github.com/nspcc-dev/neo-go/pkg/rpc/response/result"
"github.com/nspcc-dev/neo-go/pkg/rpcclient"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/trigger"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/opcode"
"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
"github.com/nspcc-dev/neo-go/pkg/vm/vmstate"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"go.uber.org/zap"
)
@ -42,19 +48,19 @@ type localClient struct {
transactions []*transaction.Transaction
dumpPath string
accounts []*wallet.Account
maxGasInvoke int64
}
func newLocalClient(v *viper.Viper, wallets []*wallet.Wallet) (*localClient, error) {
func newLocalClient(cmd *cobra.Command, v *viper.Viper, wallets []*wallet.Wallet) (*localClient, error) {
cfg, err := config.LoadFile(v.GetString(protoConfigPath))
if err != nil {
return nil, err
}
bc, err := core.NewBlockchain(storage.NewMemoryStore(), cfg.ProtocolConfiguration, zap.NewNop())
bc, err := core.NewBlockchain(storage.NewMemoryStore(), cfg.Blockchain(), zap.NewNop())
if err != nil {
return nil, err
}
defer func() { go bc.Run() }()
m := smartcontract.GetDefaultHonestNodeCount(cfg.ProtocolConfiguration.ValidatorsCount)
accounts := make([]*wallet.Account, len(wallets))
@ -76,15 +82,37 @@ func newLocalClient(v *viper.Viper, wallets []*wallet.Wallet) (*localClient, err
return indexMap[string(pi)] < indexMap[string(pj)]
})
sort.Slice(accounts[:cfg.ProtocolConfiguration.ValidatorsCount], func(i, j int) bool {
pi := accounts[i].PrivateKey().PublicKey().Bytes()
pj := accounts[j].PrivateKey().PublicKey().Bytes()
return bytes.Compare(pi, pj) == -1
return accounts[i].PublicKey().Cmp(accounts[j].PublicKey()) == -1
})
go bc.Run()
dumpPath := v.GetString(localDumpFlag)
if cmd.Name() != "init" {
f, err := os.OpenFile(dumpPath, os.O_RDONLY, 0600)
if err != nil {
return nil, fmt.Errorf("can't open local dump: %w", err)
}
defer f.Close()
r := io.NewBinReaderFromIO(f)
var skip uint32
if bc.BlockHeight() != 0 {
skip = bc.BlockHeight() + 1
}
count := r.ReadU32LE() - skip
if err := chaindump.Restore(bc, r, skip, count, nil); err != nil {
return nil, fmt.Errorf("can't restore local dump: %w", err)
}
}
return &localClient{
bc: bc,
dumpPath: v.GetString(localDumpFlag),
dumpPath: dumpPath,
accounts: accounts[:m],
maxGasInvoke: 15_0000_0000,
}, nil
}
@ -125,7 +153,7 @@ func (l *localClient) GetApplicationLog(h util.Uint256, t *trigger.Type) (*resul
return &a, nil
}
func (l *localClient) CreateTxFromScript(script []byte, acc *wallet.Account, sysFee int64, netFee int64, cosigners []client.SignerAccount) (*transaction.Transaction, error) {
func (l *localClient) CreateTxFromScript(script []byte, acc *wallet.Account, sysFee int64, netFee int64, cosigners []rpcclient.SignerAccount) (*transaction.Transaction, error) {
signers, accounts, err := getSigners(acc, cosigners)
if err != nil {
return nil, fmt.Errorf("failed to construct tx signers: %w", err)
@ -158,9 +186,19 @@ func (l *localClient) GetCommittee() (keys.PublicKeys, error) {
panic("unexpected call")
}
func (l *localClient) InvokeFunction(_ util.Uint160, _ string, _ []smartcontract.Parameter, _ []transaction.Signer) (*result.Invoke, error) {
// not used by `morph init` command
panic("unexpected call")
// InvokeFunction is implemented via `InvokeScript`.
func (l *localClient) InvokeFunction(h util.Uint160, method string, sPrm []smartcontract.Parameter, ss []transaction.Signer) (*result.Invoke, error) {
var err error
pp := make([]any, len(sPrm))
for i, p := range sPrm {
pp[i], err = smartcontract.ExpandParameterToEmitable(p)
if err != nil {
return nil, fmt.Errorf("incorrect parameter type %s: %w", p.Type, err)
}
}
return invokeFunction(l, h, method, pp, ss)
}
func (l *localClient) CalculateNotaryFee(_ uint8) (int64, error) {
@ -173,11 +211,90 @@ func (l *localClient) SignAndPushP2PNotaryRequest(_ *transaction.Transaction, _
panic("unexpected call")
}
func (l *localClient) SignAndPushInvocationTx(_ []byte, _ *wallet.Account, _ int64, _ fixedn.Fixed8, _ []client.SignerAccount) (util.Uint256, error) {
func (l *localClient) SignAndPushInvocationTx(_ []byte, _ *wallet.Account, _ int64, _ fixedn.Fixed8, _ []rpcclient.SignerAccount) (util.Uint256, error) {
// not used by `morph init` command
panic("unexpected call")
}
func (l *localClient) TerminateSession(_ uuid.UUID) (bool, error) {
// not used by `morph init` command
panic("unexpected call")
}
func (l *localClient) TraverseIterator(_, _ uuid.UUID, _ int) ([]stackitem.Item, error) {
// not used by `morph init` command
panic("unexpected call")
}
// GetVersion return default version.
func (l *localClient) GetVersion() (*result.Version, error) {
c := l.bc.GetConfig()
return &result.Version{
Protocol: result.Protocol{
AddressVersion: address.NEO3Prefix,
Network: c.Magic,
MillisecondsPerBlock: int(c.TimePerBlock / time.Millisecond),
MaxTraceableBlocks: c.MaxTraceableBlocks,
MaxValidUntilBlockIncrement: c.MaxValidUntilBlockIncrement,
MaxTransactionsPerBlock: c.MaxTransactionsPerBlock,
MemoryPoolMaxTransactions: c.MemPoolSize,
ValidatorsCount: byte(c.ValidatorsCount),
InitialGasDistribution: c.InitialGASSupply,
CommitteeHistory: c.CommitteeHistory,
P2PSigExtensions: c.P2PSigExtensions,
StateRootInHeader: c.StateRootInHeader,
ValidatorsHistory: c.ValidatorsHistory,
},
}, nil
}
func (l *localClient) InvokeContractVerify(contract util.Uint160, params []smartcontract.Parameter, signers []transaction.Signer, witnesses ...transaction.Witness) (*result.Invoke, error) {
// not used by `morph init` command
panic("unexpected call")
}
// CalculateNetworkFee calculates network fee for the given transaction.
// Copied from neo-go with minor corrections (no need to support non-notary mode):
// https://github.com/nspcc-dev/neo-go/blob/v0.99.2/pkg/services/rpcsrv/server.go#L744
func (l *localClient) CalculateNetworkFee(tx *transaction.Transaction) (int64, error) {
hashablePart, err := tx.EncodeHashableFields()
if err != nil {
return 0, fmt.Errorf("failed to compute tx size: %w", err)
}
size := len(hashablePart) + io.GetVarSize(len(tx.Signers))
ef := l.bc.GetBaseExecFee()
var netFee int64
for i, signer := range tx.Signers {
var verificationScript []byte
for _, w := range tx.Scripts {
if w.VerificationScript != nil && hash.Hash160(w.VerificationScript).Equals(signer.Account) {
verificationScript = w.VerificationScript
break
}
}
if verificationScript == nil {
gasConsumed, err := l.bc.VerifyWitness(signer.Account, tx, &tx.Scripts[i], l.maxGasInvoke)
if err != nil {
return 0, fmt.Errorf("invalid signature: %w", err)
}
netFee += gasConsumed
size += io.GetVarSize([]byte{}) + io.GetVarSize(tx.Scripts[i].InvocationScript)
continue
}
fee, sizeDelta := fee.Calculate(ef, verificationScript)
netFee += fee
size += sizeDelta
}
fee := l.bc.FeePerByte()
netFee += int64(size) * fee
return netFee, nil
}
// AddNetworkFee adds network fee for each witness script and optional extra
// network fee to transaction. `accs` is an array signer's accounts.
// Copied from neo-go with minor corrections (no need to support contract signers):
@ -204,7 +321,7 @@ func (l *localClient) AddNetworkFee(tx *transaction.Transaction, extraFee int64,
// will be placed at the start of the list.
// Copied from neo-go with minor corrections:
// https://github.com/nspcc-dev/neo-go/blob/6ff11baa1b9e4c71ef0d1de43b92a8c541ca732c/pkg/rpc/client/rpc.go#L735
func getSigners(sender *wallet.Account, cosigners []client.SignerAccount) ([]transaction.Signer, []*wallet.Account, error) {
func getSigners(sender *wallet.Account, cosigners []rpcclient.SignerAccount) ([]transaction.Signer, []*wallet.Account, error) {
var (
signers []transaction.Signer
accounts []*wallet.Account
@ -229,11 +346,11 @@ func getSigners(sender *wallet.Account, cosigners []client.SignerAccount) ([]tra
}
func (l *localClient) NEP17BalanceOf(h util.Uint160, acc util.Uint160) (int64, error) {
res, err := invokeFunction(l, h, "balanceOf", []interface{}{acc}, nil)
res, err := invokeFunction(l, h, "balanceOf", []any{acc}, nil)
if err != nil {
return 0, err
}
if res.State != vm.HaltState.String() || len(res.Stack) == 0 {
if res.State != vmstate.Halt.String() || len(res.Stack) == 0 {
return 0, fmt.Errorf("`balance`: invalid response (empty: %t): %s",
len(res.Stack) == 0, res.FaultException)
}
@ -254,12 +371,15 @@ func (l *localClient) InvokeScript(script []byte, signers []transaction.Signer)
tx.Signers = signers
tx.ValidUntilBlock = l.bc.BlockHeight() + 2
ic := l.bc.GetTestVM(trigger.Application, tx, &block.Block{
ic, err := l.bc.GetTestVM(trigger.Application, tx, &block.Block{
Header: block.Header{
Index: lastBlock.Index + 1,
Timestamp: lastBlock.Timestamp + 1,
},
})
if err != nil {
return nil, fmt.Errorf("get test VM: %w", err)
}
ic.VM.GasLimit = 100_0000_0000
ic.VM.LoadScriptWithFlags(script, callflag.All)
@ -278,6 +398,13 @@ func (l *localClient) InvokeScript(script []byte, signers []transaction.Signer)
}
func (l *localClient) SendRawTransaction(tx *transaction.Transaction) (util.Uint256, error) {
// We need to test that transaction was formed correctly to catch as many errors as we can.
bs := tx.Bytes()
_, err := transaction.NewTransactionFromBytes(bs)
if err != nil {
return tx.Hash(), fmt.Errorf("invalid transaction: %w", err)
}
l.transactions = append(l.transactions, tx)
return tx.Hash(), nil
}
@ -324,7 +451,7 @@ func (l *localClient) putTransactions() error {
return l.bc.AddBlock(b)
}
func invokeFunction(c Client, h util.Uint160, method string, parameters []interface{}, signers []transaction.Signer) (*result.Invoke, error) {
func invokeFunction(c Client, h util.Uint160, method string, parameters []any, signers []transaction.Signer) (*result.Invoke, error) {
w := io.NewBufBinWriter()
emit.Array(w.BinWriter, parameters...)
emit.AppCallNoArgs(w.BinWriter, h, method, callflag.All)
@ -336,16 +463,9 @@ func invokeFunction(c Client, h util.Uint160, method string, parameters []interf
var errGetDesignatedByRoleResponse = errors.New("`getDesignatedByRole`: invalid response")
func getDesignatedByRole(c Client, h util.Uint160, role noderoles.Role, u uint32) (keys.PublicKeys, error) {
res, err := invokeFunction(c, h, "getDesignatedByRole", []interface{}{int64(role), int64(u)}, nil)
func getDesignatedByRole(inv *invoker.Invoker, h util.Uint160, role noderoles.Role, u uint32) (keys.PublicKeys, error) {
arr, err := unwrap.Array(inv.Call(h, "getDesignatedByRole", int64(role), int64(u)))
if err != nil {
return nil, err
}
if res.State != vm.HaltState.String() || len(res.Stack) == 0 {
return nil, errGetDesignatedByRoleResponse
}
arr, ok := res.Stack[0].Value().([]stackitem.Item)
if !ok {
return nil, errGetDesignatedByRoleResponse
}
@ -379,6 +499,7 @@ func (l *localClient) dump() (err error) {
}()
w := io.NewBinWriterFromIO(f)
err = chaindump.Dump(l.bc, w, 0, l.bc.BlockHeight())
w.WriteU32LE(l.bc.BlockHeight() + 1)
err = chaindump.Dump(l.bc, w, 0, l.bc.BlockHeight()+1)
return
}

View file

@ -11,9 +11,11 @@ import (
"github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/fixedn"
"github.com/nspcc-dev/neo-go/pkg/neorpc/result"
"github.com/nspcc-dev/neo-go/pkg/network/payload"
"github.com/nspcc-dev/neo-go/pkg/rpc/client"
"github.com/nspcc-dev/neo-go/pkg/rpc/response/result"
"github.com/nspcc-dev/neo-go/pkg/rpcclient"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/actor"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/trigger"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/wallet"
@ -24,28 +26,36 @@ import (
// Client represents N3 client interface capable of test-invoking scripts
// and sending signed transactions to chain.
type Client interface {
invoker.RPCInvoke
GetBlockCount() (uint32, error)
GetContractStateByID(int32) (*state.Contract, error)
GetContractStateByHash(util.Uint160) (*state.Contract, error)
GetNativeContracts() ([]state.NativeContract, error)
GetNetwork() (netmode.Magic, error)
GetApplicationLog(util.Uint256, *trigger.Type) (*result.ApplicationLog, error)
CreateTxFromScript([]byte, *wallet.Account, int64, int64, []client.SignerAccount) (*transaction.Transaction, error)
GetVersion() (*result.Version, error)
CreateTxFromScript([]byte, *wallet.Account, int64, int64, []rpcclient.SignerAccount) (*transaction.Transaction, error)
NEP17BalanceOf(util.Uint160, util.Uint160) (int64, error)
InvokeScript([]byte, []transaction.Signer) (*result.Invoke, error)
SendRawTransaction(*transaction.Transaction) (util.Uint256, error)
GetCommittee() (keys.PublicKeys, error)
CalculateNotaryFee(uint8) (int64, error)
CalculateNetworkFee(tx *transaction.Transaction) (int64, error)
AddNetworkFee(*transaction.Transaction, int64, ...*wallet.Account) error
SignAndPushInvocationTx([]byte, *wallet.Account, int64, fixedn.Fixed8, []client.SignerAccount) (util.Uint256, error)
SignAndPushInvocationTx([]byte, *wallet.Account, int64, fixedn.Fixed8, []rpcclient.SignerAccount) (util.Uint256, error)
SignAndPushP2PNotaryRequest(*transaction.Transaction, []byte, int64, int64, uint32, *wallet.Account) (*payload.P2PNotaryRequest, error)
}
type hashVUBPair struct {
hash util.Uint256
vub uint32
}
type clientContext struct {
Client Client
Hashes []util.Uint256
WaitDuration time.Duration
PollInterval time.Duration
Client Client // a raw neo-go client OR a local chain implementation
CommitteeAct *actor.Actor // committee actor with the Global witness scope
ReadOnlyInvoker *invoker.Invoker // R/O contract invoker, does not contain any signer
SentTxs []hashVUBPair
}
func getN3Client(v *viper.Viper) (Client, error) {
@ -61,7 +71,7 @@ func getN3Client(v *viper.Viper) (Client, error) {
if endpoint == "" {
return nil, errors.New("missing endpoint")
}
c, err := client.New(ctx, endpoint, client.Options{
c, err := rpcclient.New(ctx, endpoint, rpcclient.Options{
MaxConnsPerHost: maxConnsPerHost,
RequestTimeout: requestTimeout,
})
@ -74,12 +84,23 @@ func getN3Client(v *viper.Viper) (Client, error) {
return c, nil
}
func defaultClientContext(c Client) *clientContext {
func defaultClientContext(c Client, committeeAcc *wallet.Account) (*clientContext, error) {
commAct, err := actor.New(c, []actor.SignerAccount{{
Signer: transaction.Signer{
Account: committeeAcc.Contract.ScriptHash(),
Scopes: transaction.Global,
},
Account: committeeAcc,
}})
if err != nil {
return nil, err
}
return &clientContext{
Client: c,
WaitDuration: time.Second * 30,
PollInterval: time.Second,
}
CommitteeAct: commAct,
ReadOnlyInvoker: invoker.New(c, nil),
}, nil
}
func (c *clientContext) sendTx(tx *transaction.Transaction, cmd *cobra.Command, await bool) error {
@ -92,7 +113,7 @@ func (c *clientContext) sendTx(tx *transaction.Transaction, cmd *cobra.Command,
return fmt.Errorf("sent and actual tx hashes mismatch:\n\tsent: %v\n\tactual: %v", tx.Hash().StringLE(), h.StringLE())
}
c.Hashes = append(c.Hashes, h)
c.SentTxs = append(c.SentTxs, hashVUBPair{hash: h, vub: tx.ValidUntilBlock})
if await {
return c.awaitTx(cmd)

View file

@ -0,0 +1,29 @@
package morph
import (
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-node/pkg/morph/client/netmap"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
func listNetmapCandidatesNodes(cmd *cobra.Command, _ []string) {
c, err := getN3Client(viper.GetViper())
commonCmd.ExitOnErr(cmd, "can't create N3 client: %w", err)
inv := invoker.New(c, nil)
cs, err := c.GetContractStateByID(1)
commonCmd.ExitOnErr(cmd, "can't get NNS contract info: %w", err)
nmHash, err := nnsResolveHash(inv, cs.Hash, netmapContract+".frostfs")
commonCmd.ExitOnErr(cmd, "can't get netmap contract hash: %w", err)
res, err := inv.Call(nmHash, "netmapCandidates")
commonCmd.ExitOnErr(cmd, "can't fetch list of network config keys from the netmap contract", err)
nm, err := netmap.DecodeNetMap(res.Stack)
commonCmd.ExitOnErr(cmd, "unable to decode netmap: %w", err)
commonCmd.PrettyPrintNetMap(cmd, *nm, !viper.GetBool(commonflags.Verbose))
}

View file

@ -2,18 +2,17 @@ package morph
import (
"fmt"
"math/big"
"strconv"
"github.com/nspcc-dev/neo-go/cli/input"
"github.com/nspcc-dev/neo-go/pkg/core/native/nativenames"
"github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpc/client"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/opcode"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/actor"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/gas"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/nep17"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/notary"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/cobra"
"github.com/spf13/viper"
@ -86,18 +85,8 @@ func depositNotary(cmd *cobra.Command, _ []string) error {
return err
}
nhs, err := getNativeHashes(c)
if err != nil {
return fmt.Errorf("can't get native contract hashes: %w", err)
}
gasHash, ok := nhs[nativenames.Gas]
if !ok {
return fmt.Errorf("can't retrieve %s contract hash", nativenames.Gas)
}
notaryHash, ok := nhs[nativenames.Notary]
if !ok {
return fmt.Errorf("can't retrieve %s contract hash", nativenames.Notary)
if err := checkNotaryEnabled(c); err != nil {
return err
}
height, err := c.GetBlockCount()
@ -105,15 +94,7 @@ func depositNotary(cmd *cobra.Command, _ []string) error {
return fmt.Errorf("can't get current height: %v", err)
}
bw := io.NewBufBinWriter()
emit.AppCall(bw.BinWriter, gasHash, "transfer", callflag.All,
accHash, notaryHash.BytesBE(), int64(gasAmount), []interface{}{nil, int64(height) + till})
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
if bw.Err != nil {
return fmt.Errorf("BUG: invalid transfer arguments: %w", bw.Err)
}
tx, err := c.CreateTxFromScript(bw.Bytes(), acc, -1, 0, []client.SignerAccount{{
act, err := actor.New(c, []actor.SignerAccount{{
Signer: transaction.Signer{
Account: acc.Contract.ScriptHash(),
Scopes: transaction.Global,
@ -121,21 +102,20 @@ func depositNotary(cmd *cobra.Command, _ []string) error {
Account: acc,
}})
if err != nil {
return fmt.Errorf("can't create tx: %w", err)
return fmt.Errorf("could not create actor: %w", err)
}
mn, err := c.GetNetwork()
gasActor := nep17.New(act, gas.Hash)
txHash, vub, err := gasActor.Transfer(
accHash,
notary.Hash,
big.NewInt(int64(gasAmount)),
[]any{nil, int64(height) + till},
)
if err != nil {
// error appears only if client
// has not been initialized
panic(err)
return fmt.Errorf("could not send tx: %w", err)
}
err = acc.SignTx(mn, tx)
if err != nil {
return fmt.Errorf("can't sign tx: %w", err)
}
cc := defaultClientContext(c)
return cc.sendTx(tx, cmd, true)
return awaitTx(cmd, c, []hashVUBPair{{hash: txHash, vub: vub}})
}

View file

@ -5,8 +5,8 @@ import (
"strconv"
"strings"
"github.com/nspcc-dev/neo-go/pkg/core/native/nativenames"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/policy"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/spf13/cobra"
@ -25,30 +25,28 @@ func setPolicyCmd(cmd *cobra.Command, args []string) error {
return fmt.Errorf("can't to initialize context: %w", err)
}
policyHash := wCtx.nativeHash(nativenames.Policy)
bw := io.NewBufBinWriter()
for i := range args {
kv := strings.SplitN(args[i], "=", 2)
if len(kv) != 2 {
k, v, found := strings.Cut(args[i], "=")
if !found {
return fmt.Errorf("invalid parameter format, must be Parameter=Value")
}
switch kv[0] {
switch k {
case execFeeParam, storagePriceParam, setFeeParam:
default:
return fmt.Errorf("parameter must be one of %s, %s and %s", execFeeParam, storagePriceParam, setFeeParam)
}
value, err := strconv.ParseUint(kv[1], 10, 32)
value, err := strconv.ParseUint(v, 10, 32)
if err != nil {
return fmt.Errorf("can't parse parameter value '%s': %w", args[1], err)
}
emit.AppCall(bw.BinWriter, policyHash, "set"+kv[0], callflag.All, int64(value))
emit.AppCall(bw.BinWriter, policy.Hash, "set"+k, callflag.All, int64(value))
}
if err := wCtx.sendCommitteeTx(bw.Bytes(), -1, false); err != nil {
if err := wCtx.sendCommitteeTx(bw.Bytes(), false); err != nil {
return err
}

View file

@ -4,11 +4,11 @@ import (
"errors"
"fmt"
netmapcontract "github.com/TrueCloudLab/frostfs-contract/netmap"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
netmapcontract "github.com/nspcc-dev/neofs-contract/netmap"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
@ -31,13 +31,14 @@ func removeNodesCmd(cmd *cobra.Command, args []string) error {
if err != nil {
return fmt.Errorf("can't initialize context: %w", err)
}
defer wCtx.close()
cs, err := wCtx.Client.GetContractStateByID(1)
if err != nil {
return fmt.Errorf("can't get NNS contract info: %w", err)
}
nmHash, err := nnsResolveHash(wCtx.Client, cs.Hash, netmapContract+".neofs")
nmHash, err := nnsResolveHash(wCtx.ReadOnlyInvoker, cs.Hash, netmapContract+".frostfs")
if err != nil {
return fmt.Errorf("can't get netmap contract hash: %w", err)
}
@ -45,14 +46,14 @@ func removeNodesCmd(cmd *cobra.Command, args []string) error {
bw := io.NewBufBinWriter()
for i := range nodeKeys {
emit.AppCall(bw.BinWriter, nmHash, "updateStateIR", callflag.All,
int64(netmapcontract.OfflineState), nodeKeys[i].Bytes())
int64(netmapcontract.NodeStateOffline), nodeKeys[i].Bytes())
}
if err := emitNewEpochCall(bw, wCtx, nmHash); err != nil {
return err
}
if err := wCtx.sendCommitteeTx(bw.Bytes(), -1, true); err != nil {
if err := wCtx.sendConsensusTx(bw.Bytes()); err != nil {
return err
}

View file

@ -28,6 +28,9 @@ const (
containerAliasFeeCLIFlag = "container-alias-fee"
candidateFeeInitFlag = "network.fee.candidate"
candidateFeeCLIFlag = "candidate-fee"
homomorphicHashDisabledInitFlag = "network.homomorphic_hash_disabled"
maintenanceModeAllowedInitFlag = "network.maintenance_mode_allowed"
homomorphicHashDisabledCLIFlag = "homomorphic-disabled"
withdrawFeeInitFlag = "network.fee.withdraw"
withdrawFeeCLIFlag = "withdraw-fee"
containerDumpFlag = "dump"
@ -38,6 +41,7 @@ const (
notaryDepositTillFlag = "till"
localDumpFlag = "local-dump"
protoConfigPath = "protocol"
walletAddressFlag = "wallet-address"
)
var (
@ -66,6 +70,7 @@ var (
_ = viper.BindPFlag(epochDurationInitFlag, cmd.Flags().Lookup(epochDurationCLIFlag))
_ = viper.BindPFlag(maxObjectSizeInitFlag, cmd.Flags().Lookup(maxObjectSizeCLIFlag))
_ = viper.BindPFlag(incomeRateInitFlag, cmd.Flags().Lookup(incomeRateCLIFlag))
_ = viper.BindPFlag(homomorphicHashDisabledInitFlag, cmd.Flags().Lookup(homomorphicHashDisabledCLIFlag))
_ = viper.BindPFlag(auditFeeInitFlag, cmd.Flags().Lookup(auditFeeCLIFlag))
_ = viper.BindPFlag(candidateFeeInitFlag, cmd.Flags().Lookup(candidateFeeCLIFlag))
_ = viper.BindPFlag(containerFeeInitFlag, cmd.Flags().Lookup(containerFeeCLIFlag))
@ -103,7 +108,7 @@ var (
forceNewEpoch = &cobra.Command{
Use: "force-new-epoch",
Short: "Create new NeoFS epoch event in the side chain",
Short: "Create new FrostFS epoch event in the side chain",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
@ -122,6 +127,18 @@ var (
RunE: removeNodesCmd,
}
setConfig = &cobra.Command{
Use: "set-config key1=val1 [key2=val2 ...]",
DisableFlagsInUseLine: true,
Short: "Add/update global config value in the FrostFS network",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
Args: cobra.MinimumNArgs(1),
RunE: setConfigCmd,
}
setPolicy = &cobra.Command{
Use: "set-policy [ExecFeeFactor=<n1>] [StoragePrice=<n2>] [FeePerByte=<n3>]",
DisableFlagsInUseLine: true,
@ -131,6 +148,9 @@ var (
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
RunE: setPolicyCmd,
ValidArgsFunction: func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
return []string{"ExecFeeFactor=", "StoragePrice=", "FeePerByte="}, cobra.ShellCompDirectiveNoSpace
},
}
dumpContractHashesCmd = &cobra.Command{
@ -144,7 +164,7 @@ var (
dumpNetworkConfigCmd = &cobra.Command{
Use: "dump-config",
Short: "Dump NeoFS network config",
Short: "Dump FrostFS network config",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
@ -162,7 +182,7 @@ var (
updateContractsCmd = &cobra.Command{
Use: "update-contracts",
Short: "Update NeoFS contracts",
Short: "Update FrostFS contracts",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
@ -172,7 +192,7 @@ var (
dumpContainersCmd = &cobra.Command{
Use: "dump-containers",
Short: "Dump NeoFS containers to file",
Short: "Dump FrostFS containers to file",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
@ -181,7 +201,7 @@ var (
restoreContainersCmd = &cobra.Command{
Use: "restore-containers",
Short: "Restore NeoFS containers from file",
Short: "Restore FrostFS containers from file",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
@ -189,6 +209,15 @@ var (
RunE: restoreContainers,
}
listContainersCmd = &cobra.Command{
Use: "list-containers",
Short: "List FrostFS containers",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
RunE: listContainers,
}
depositNotaryCmd = &cobra.Command{
Use: "deposit-notary",
Short: "Deposit GAS for notary service",
@ -197,86 +226,114 @@ var (
},
RunE: depositNotary,
}
netmapCandidatesCmd = &cobra.Command{
Use: "netmap-candidates",
Short: "List netmap candidates nodes",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
},
Run: listNetmapCandidatesNodes,
}
)
func init() {
RootCmd.AddCommand(generateAlphabetCmd)
generateAlphabetCmd.Flags().String(alphabetWalletsFlag, "", "path to alphabet wallets dir")
generateAlphabetCmd.Flags().Uint(alphabetSizeFlag, 7, "amount of alphabet wallets to generate")
generateAlphabetCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
generateAlphabetCmd.Flags().Uint(alphabetSizeFlag, 7, "Amount of alphabet wallets to generate")
RootCmd.AddCommand(initCmd)
initCmd.Flags().String(alphabetWalletsFlag, "", "path to alphabet wallets dir")
initCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
initCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
initCmd.Flags().String(contractsInitFlag, "", "path to archive with compiled NeoFS contracts (default fetched from latest github release)")
initCmd.Flags().Uint(epochDurationCLIFlag, 240, "amount of side chain blocks in one NeoFS epoch")
initCmd.Flags().Uint(maxObjectSizeCLIFlag, 67108864, "max single object size in bytes")
initCmd.Flags().String(contractsInitFlag, "", "Path to archive with compiled FrostFS contracts (default fetched from latest github release)")
initCmd.Flags().Uint(epochDurationCLIFlag, 240, "Amount of side chain blocks in one FrostFS epoch")
initCmd.Flags().Uint(maxObjectSizeCLIFlag, 67108864, "Max single object size in bytes")
initCmd.Flags().Bool(homomorphicHashDisabledCLIFlag, false, "Disable object homomorphic hashing")
// Defaults are taken from neo-preodolenie.
initCmd.Flags().Uint64(containerFeeCLIFlag, 1000, "container registration fee")
initCmd.Flags().Uint64(containerAliasFeeCLIFlag, 500, "container alias fee")
initCmd.Flags().String(protoConfigPath, "", "path to the consensus node configuration")
initCmd.Flags().String(localDumpFlag, "", "path to the blocks dump file")
initCmd.Flags().Uint64(containerFeeCLIFlag, 1000, "Container registration fee")
initCmd.Flags().Uint64(containerAliasFeeCLIFlag, 500, "Container alias fee")
initCmd.Flags().String(protoConfigPath, "", "Path to the consensus node configuration")
initCmd.Flags().String(localDumpFlag, "", "Path to the blocks dump file")
RootCmd.AddCommand(deployCmd)
RootCmd.AddCommand(generateStorageCmd)
generateStorageCmd.Flags().String(alphabetWalletsFlag, "", "path to alphabet wallets dir")
generateStorageCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
generateStorageCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
generateStorageCmd.Flags().String(storageWalletFlag, "", "path to new storage node wallet")
generateStorageCmd.Flags().String(storageGasCLIFlag, "", "initial amount of GAS to transfer")
generateStorageCmd.Flags().StringP(storageWalletLabelFlag, "l", "", "wallet label")
generateStorageCmd.Flags().String(storageWalletFlag, "", "Path to new storage node wallet")
generateStorageCmd.Flags().String(storageGasCLIFlag, "", "Initial amount of GAS to transfer")
generateStorageCmd.Flags().StringP(storageWalletLabelFlag, "l", "", "Wallet label")
RootCmd.AddCommand(forceNewEpoch)
forceNewEpoch.Flags().String(alphabetWalletsFlag, "", "path to alphabet wallets dir")
forceNewEpoch.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
forceNewEpoch.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
RootCmd.AddCommand(removeNodes)
removeNodes.Flags().String(alphabetWalletsFlag, "", "path to alphabet wallets dir")
removeNodes.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
removeNodes.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
RootCmd.AddCommand(setPolicy)
setPolicy.Flags().String(alphabetWalletsFlag, "", "path to alphabet wallets dir")
setPolicy.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
setPolicy.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
RootCmd.AddCommand(dumpContractHashesCmd)
dumpContractHashesCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
dumpContractHashesCmd.Flags().String(customZoneFlag, "", "Custom zone to search.")
RootCmd.AddCommand(dumpNetworkConfigCmd)
dumpNetworkConfigCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
RootCmd.AddCommand(setConfig)
setConfig.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
setConfig.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
setConfig.Flags().Bool(forceConfigSet, false, "Force setting not well-known configuration key")
RootCmd.AddCommand(dumpBalancesCmd)
dumpBalancesCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
dumpBalancesCmd.Flags().BoolP(dumpBalancesStorageFlag, "s", false, "dump balances of storage nodes from the current netmap")
dumpBalancesCmd.Flags().BoolP(dumpBalancesAlphabetFlag, "a", false, "dump balances of alphabet contracts")
dumpBalancesCmd.Flags().BoolP(dumpBalancesProxyFlag, "p", false, "dump balances of the proxy contract")
dumpBalancesCmd.Flags().Bool(dumpBalancesUseScriptHashFlag, false, "use script-hash format for addresses")
dumpBalancesCmd.Flags().BoolP(dumpBalancesStorageFlag, "s", false, "Dump balances of storage nodes from the current netmap")
dumpBalancesCmd.Flags().BoolP(dumpBalancesAlphabetFlag, "a", false, "Dump balances of alphabet contracts")
dumpBalancesCmd.Flags().BoolP(dumpBalancesProxyFlag, "p", false, "Dump balances of the proxy contract")
dumpBalancesCmd.Flags().Bool(dumpBalancesUseScriptHashFlag, false, "Use script-hash format for addresses")
RootCmd.AddCommand(updateContractsCmd)
updateContractsCmd.Flags().String(alphabetWalletsFlag, "", "path to alphabet wallets dir")
updateContractsCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
updateContractsCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
updateContractsCmd.Flags().String(contractsInitFlag, "", "path to archive with compiled NeoFS contracts (default fetched from latest github release)")
updateContractsCmd.Flags().String(contractsInitFlag, "", "Path to archive with compiled FrostFS contracts (default fetched from latest github release)")
RootCmd.AddCommand(dumpContainersCmd)
dumpContainersCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
dumpContainersCmd.Flags().String(containerDumpFlag, "", "file where to save dumped containers")
dumpContainersCmd.Flags().String(containerContractFlag, "", "container contract hash (for networks without NNS)")
dumpContainersCmd.Flags().StringSlice(containerIDsFlag, nil, "containers to dump")
dumpContainersCmd.Flags().String(containerDumpFlag, "", "File where to save dumped containers")
dumpContainersCmd.Flags().String(containerContractFlag, "", "Container contract hash (for networks without NNS)")
dumpContainersCmd.Flags().StringSlice(containerIDsFlag, nil, "Containers to dump")
RootCmd.AddCommand(restoreContainersCmd)
restoreContainersCmd.Flags().String(alphabetWalletsFlag, "", "path to alphabet wallets dir")
restoreContainersCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
restoreContainersCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
restoreContainersCmd.Flags().String(containerDumpFlag, "", "file to restore containers from")
restoreContainersCmd.Flags().StringSlice(containerIDsFlag, nil, "containers to restore")
restoreContainersCmd.Flags().String(containerDumpFlag, "", "File to restore containers from")
restoreContainersCmd.Flags().StringSlice(containerIDsFlag, nil, "Containers to restore")
RootCmd.AddCommand(listContainersCmd)
listContainersCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
listContainersCmd.Flags().String(containerContractFlag, "", "Container contract hash (for networks without NNS)")
RootCmd.AddCommand(refillGasCmd)
refillGasCmd.Flags().String(alphabetWalletsFlag, "", "path to alphabet wallets dir")
refillGasCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
refillGasCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
refillGasCmd.Flags().String(storageWalletFlag, "", "path to storage node wallet")
refillGasCmd.Flags().String(refillGasAmountFlag, "", "additional amount of GAS to transfer")
refillGasCmd.Flags().String(storageWalletFlag, "", "Path to storage node wallet")
refillGasCmd.Flags().String(walletAddressFlag, "", "Address of wallet")
refillGasCmd.Flags().String(refillGasAmountFlag, "", "Additional amount of GAS to transfer")
refillGasCmd.MarkFlagsMutuallyExclusive(walletAddressFlag, storageWalletFlag)
RootCmd.AddCommand(cmdSubnet)
RootCmd.AddCommand(depositNotaryCmd)
depositNotaryCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
depositNotaryCmd.Flags().String(storageWalletFlag, "", "path to storage node wallet")
depositNotaryCmd.Flags().String(walletAccountFlag, "", "wallet account address")
depositNotaryCmd.Flags().String(refillGasAmountFlag, "", "amount of GAS to deposit")
depositNotaryCmd.Flags().String(notaryDepositTillFlag, "", "notary deposit duration in blocks")
depositNotaryCmd.Flags().String(storageWalletFlag, "", "Path to storage node wallet")
depositNotaryCmd.Flags().String(walletAccountFlag, "", "Wallet account address")
depositNotaryCmd.Flags().String(refillGasAmountFlag, "", "Amount of GAS to deposit")
depositNotaryCmd.Flags().String(notaryDepositTillFlag, "", "Notary deposit duration in blocks")
RootCmd.AddCommand(netmapCandidatesCmd)
netmapCandidatesCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
}

View file

@ -5,6 +5,12 @@ import (
"errors"
"fmt"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph/internal"
"github.com/TrueCloudLab/frostfs-node/pkg/morph/client"
"github.com/TrueCloudLab/frostfs-node/pkg/util/rand"
"github.com/TrueCloudLab/frostfs-sdk-go/subnet"
subnetid "github.com/TrueCloudLab/frostfs-sdk-go/subnet/id"
"github.com/TrueCloudLab/frostfs-sdk-go/user"
"github.com/nspcc-dev/neo-go/cli/flags"
"github.com/nspcc-dev/neo-go/cli/input"
"github.com/nspcc-dev/neo-go/pkg/core/native/nativenames"
@ -12,19 +18,14 @@ import (
"github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/crypto/hash"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
neogocli "github.com/nspcc-dev/neo-go/pkg/rpc/client"
"github.com/nspcc-dev/neo-go/pkg/rpc/response/result"
"github.com/nspcc-dev/neo-go/pkg/neorpc/result"
"github.com/nspcc-dev/neo-go/pkg/rpcclient"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/nspcc-dev/neo-go/pkg/smartcontract"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm/opcode"
"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/nspcc-dev/neofs-node/cmd/neofs-adm/internal/modules/morph/internal"
"github.com/nspcc-dev/neofs-node/pkg/morph/client"
"github.com/nspcc-dev/neofs-node/pkg/util/rand"
"github.com/nspcc-dev/neofs-sdk-go/subnet"
subnetid "github.com/nspcc-dev/neofs-sdk-go/subnet/id"
"github.com/nspcc-dev/neofs-sdk-go/user"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
@ -38,7 +39,7 @@ func viperBindFlags(cmd *cobra.Command, flags ...string) {
// subnet command section.
var cmdSubnet = &cobra.Command{
Use: "subnet",
Short: "NeoFS subnet management",
Short: "FrostFS subnet management",
PreRun: func(cmd *cobra.Command, _ []string) {
viperBindFlags(cmd,
endpointFlag,
@ -48,14 +49,10 @@ var cmdSubnet = &cobra.Command{
// shared flags of cmdSubnet sub-commands.
const (
// subnet identifier
flagSubnet = "subnet"
// subnet client group ID
flagSubnetGroup = "group"
// filepath to wallet
flagSubnetWallet = "wallet"
// address in the wallet, optional
flagSubnetAddress = "address"
flagSubnet = "subnet" // subnet identifier
flagSubnetGroup = "group" // subnet client group ID
flagSubnetWallet = "wallet" // filepath to wallet
flagSubnetAddress = "address" // address in the wallet, optional
)
// reads wallet from the filepath configured in flagSubnetWallet flag,
@ -115,7 +112,7 @@ func readSubnetKey(key *keys.PrivateKey) error {
// create subnet command.
var cmdSubnetCreate = &cobra.Command{
Use: "create",
Short: "Create NeoFS subnet",
Short: "Create FrostFS subnet",
PreRun: func(cmd *cobra.Command, _ []string) {
viperBindFlags(cmd,
flagSubnetWallet,
@ -170,7 +167,7 @@ var cmdSubnetCreate = &cobra.Command{
// cmdSubnetRemove flags.
const (
// subnet ID to be removed
// subnet ID to be removed.
flagSubnetRemoveID = flagSubnet
)
@ -180,7 +177,7 @@ var errZeroSubnet = errors.New("zero subnet")
// remove subnet command.
var cmdSubnetRemove = &cobra.Command{
Use: "remove",
Short: "Remove NeoFS subnet",
Short: "Remove FrostFS subnet",
PreRun: func(cmd *cobra.Command, _ []string) {
viperBindFlags(cmd,
flagSubnetWallet,
@ -222,14 +219,14 @@ var cmdSubnetRemove = &cobra.Command{
// cmdSubnetGet flags.
const (
// subnet ID to be read
// subnet ID to be read.
flagSubnetGetID = flagSubnet
)
// get subnet command.
var cmdSubnetGet = &cobra.Command{
Use: "get",
Short: "Read information about the NeoFS subnet",
Short: "Read information about the FrostFS subnet",
PreRun: func(cmd *cobra.Command, _ []string) {
viperBindFlags(cmd,
flagSubnetGetID,
@ -285,18 +282,15 @@ var cmdSubnetGet = &cobra.Command{
// cmdSubnetAdmin subnet flags.
const (
// subnet ID to be managed
flagSubnetAdminSubnet = flagSubnet
// admin public key
flagSubnetAdminID = "admin"
// manage client admins instead of node ones
flagSubnetAdminClient = "client"
flagSubnetAdminSubnet = flagSubnet // subnet ID to be managed
flagSubnetAdminID = "admin" // admin public key
flagSubnetAdminClient = "client" // manage client admins instead of node ones
)
// command to manage subnet admins.
var cmdSubnetAdmin = &cobra.Command{
Use: "admin",
Short: "Manage administrators of the NeoFS subnet",
Short: "Manage administrators of the FrostFS subnet",
PreRun: func(cmd *cobra.Command, args []string) {
viperBindFlags(cmd,
flagSubnetWallet,
@ -309,8 +303,7 @@ var cmdSubnetAdmin = &cobra.Command{
// cmdSubnetAdminAdd flags.
const (
// client group ID
flagSubnetAdminAddGroup = flagSubnetGroup
flagSubnetAdminAddGroup = flagSubnetGroup // client group ID
)
// common executor cmdSubnetAdminAdd and cmdSubnetAdminRemove commands.
@ -347,7 +340,7 @@ func manageSubnetAdmins(cmd *cobra.Command, rm bool) error {
}
// prepare call parameters
prm := make([]interface{}, 0, 3)
prm := make([]any, 0, 3)
prm = append(prm, id.Marshal())
var method string
@ -404,7 +397,7 @@ func manageSubnetAdmins(cmd *cobra.Command, rm bool) error {
// command to add subnet admin.
var cmdSubnetAdminAdd = &cobra.Command{
Use: "add",
Short: "Add admin to the NeoFS subnet",
Short: "Add admin to the FrostFS subnet",
PreRun: func(cmd *cobra.Command, _ []string) {
viperBindFlags(cmd,
flagSubnetAdminAddGroup,
@ -419,7 +412,7 @@ var cmdSubnetAdminAdd = &cobra.Command{
// command to remove subnet admin.
var cmdSubnetAdminRemove = &cobra.Command{
Use: "remove",
Short: "Remove admin of the NeoFS subnet",
Short: "Remove admin of the FrostFS subnet",
PreRun: func(cmd *cobra.Command, _ []string) {
viperBindFlags(cmd,
flagSubnetAdminClient,
@ -432,18 +425,15 @@ var cmdSubnetAdminRemove = &cobra.Command{
// cmdSubnetClient flags.
const (
// ID of the subnet to be managed
flagSubnetClientSubnet = flagSubnet
// client's NeoFS ID
flagSubnetClientID = flagSubnetAdminClient
// ID of the subnet client group
flagSubnetClientGroup = flagSubnetGroup
flagSubnetClientSubnet = flagSubnet // ID of the subnet to be managed
flagSubnetClientID = flagSubnetAdminClient // client's NeoFS ID
flagSubnetClientGroup = flagSubnetGroup // ID of the subnet client group
)
// command to manage subnet clients.
var cmdSubnetClient = &cobra.Command{
Use: "client",
Short: "Manage clients of the NeoFS subnet",
Short: "Manage clients of the FrostFS subnet",
PreRun: func(cmd *cobra.Command, _ []string) {
viperBindFlags(cmd,
flagSubnetWallet,
@ -526,7 +516,7 @@ func manageSubnetClients(cmd *cobra.Command, rm bool) error {
// command to add subnet client.
var cmdSubnetClientAdd = &cobra.Command{
Use: "add",
Short: "Add client to the NeoFS subnet",
Short: "Add client to the FrostFS subnet",
RunE: func(cmd *cobra.Command, _ []string) error {
return manageSubnetClients(cmd, false)
},
@ -535,7 +525,7 @@ var cmdSubnetClientAdd = &cobra.Command{
// command to remove subnet client.
var cmdSubnetClientRemove = &cobra.Command{
Use: "remove",
Short: "Remove client of the NeoFS subnet",
Short: "Remove client of the FrostFS subnet",
RunE: func(cmd *cobra.Command, _ []string) error {
return manageSubnetClients(cmd, true)
},
@ -543,10 +533,8 @@ var cmdSubnetClientRemove = &cobra.Command{
// cmdSubnetNode flags.
const (
// node ID
flagSubnetNode = "node"
// ID of the subnet to be managed
flagSubnetNodeSubnet = flagSubnet
flagSubnetNode = "node" // node ID
flagSubnetNodeSubnet = flagSubnet // ID of the subnet to be managed
)
// common executor cmdSubnetNodeAdd and cmdSubnetNodeRemove commands.
@ -610,7 +598,7 @@ func manageSubnetNodes(cmd *cobra.Command, rm bool) error {
// command to manage subnet nodes.
var cmdSubnetNode = &cobra.Command{
Use: "node",
Short: "Manage nodes of the NeoFS subnet",
Short: "Manage nodes of the FrostFS subnet",
PreRun: func(cmd *cobra.Command, _ []string) {
viperBindFlags(cmd,
flagSubnetWallet,
@ -623,7 +611,7 @@ var cmdSubnetNode = &cobra.Command{
// command to add subnet node.
var cmdSubnetNodeAdd = &cobra.Command{
Use: "add",
Short: "Add node to the NeoFS subnet",
Short: "Add node to the FrostFS subnet",
RunE: func(cmd *cobra.Command, _ []string) error {
return manageSubnetNodes(cmd, false)
},
@ -632,7 +620,7 @@ var cmdSubnetNodeAdd = &cobra.Command{
// command to remove subnet node.
var cmdSubnetNodeRemove = &cobra.Command{
Use: "remove",
Short: "Remove node from the NeoFS subnet",
Short: "Remove node from the FrostFS subnet",
RunE: func(cmd *cobra.Command, _ []string) error {
return manageSubnetNodes(cmd, true)
},
@ -705,7 +693,7 @@ func init() {
_ = cmdSubnetClient.MarkFlagRequired(flagSubnetClientSubnet)
clientFlags.String(flagSubnetClientGroup, "", "ID of the client group to work with")
_ = cmdSubnetClient.MarkFlagRequired(flagSubnetClientGroup)
clientFlags.String(flagSubnetClientID, "", "Client's user ID in NeoFS system in text format")
clientFlags.String(flagSubnetClientID, "", "Client's user ID in FrostFS system in text format")
_ = cmdSubnetClient.MarkFlagRequired(flagSubnetClientID)
clientFlags.StringP(flagSubnetWallet, "w", "", "Path to file with wallet")
_ = cmdSubnetClient.MarkFlagRequired(flagSubnetWallet)
@ -754,7 +742,7 @@ func init() {
)
}
func testInvokeMethod(key keys.PrivateKey, method string, args ...interface{}) ([]stackitem.Item, error) {
func testInvokeMethod(key keys.PrivateKey, method string, args ...any) ([]stackitem.Item, error) {
c, err := getN3Client(viper.GetViper())
if err != nil {
return nil, fmt.Errorf("morph client creation: %w", err)
@ -765,11 +753,6 @@ func testInvokeMethod(key keys.PrivateKey, method string, args ...interface{}) (
return nil, fmt.Errorf("NNS contract resolving: %w", err)
}
subnetHash, err := nnsResolveHash(c, nnsCs.Hash, subnetContract+".neofs")
if err != nil {
return nil, fmt.Errorf("subnet hash resolving: %w", err)
}
cosigner := []transaction.Signer{
{
Account: key.PublicKey().GetScriptHash(),
@ -777,7 +760,14 @@ func testInvokeMethod(key keys.PrivateKey, method string, args ...interface{}) (
},
}
res, err := invokeFunction(c, subnetHash, method, args, cosigner)
inv := invoker.New(c, cosigner)
subnetHash, err := nnsResolveHash(inv, nnsCs.Hash, subnetContract+".frostfs")
if err != nil {
return nil, fmt.Errorf("subnet hash resolving: %w", err)
}
res, err := inv.Call(subnetHash, method, args...)
if err != nil {
return nil, fmt.Errorf("invocation parameters prepararion: %w", err)
}
@ -790,7 +780,7 @@ func testInvokeMethod(key keys.PrivateKey, method string, args ...interface{}) (
return res.Stack, nil
}
func invokeMethod(key keys.PrivateKey, tryNotary bool, method string, args ...interface{}) error {
func invokeMethod(key keys.PrivateKey, tryNotary bool, method string, args ...any) error {
c, err := getN3Client(viper.GetViper())
if err != nil {
return fmt.Errorf("morph client creation: %w", err)
@ -831,17 +821,12 @@ func invokeMethod(key keys.PrivateKey, tryNotary bool, method string, args ...in
return nil
}
func invokeNonNotary(c Client, key keys.PrivateKey, method string, args ...interface{}) error {
func invokeNonNotary(c Client, key keys.PrivateKey, method string, args ...any) error {
nnsCs, err := c.GetContractStateByID(1)
if err != nil {
return fmt.Errorf("NNS contract resolving: %w", err)
}
subnetHash, err := nnsResolveHash(c, nnsCs.Hash, subnetContract+".neofs")
if err != nil {
return fmt.Errorf("subnet hash resolving: %w", err)
}
acc := wallet.NewAccountFromPrivateKey(&key)
cosigner := []transaction.Signer{
@ -851,14 +836,21 @@ func invokeNonNotary(c Client, key keys.PrivateKey, method string, args ...inter
},
}
cosignerAcc := []neogocli.SignerAccount{
cosignerAcc := []rpcclient.SignerAccount{
{
Signer: cosigner[0],
Account: acc,
},
}
test, err := invokeFunction(c, subnetHash, method, args, cosigner)
inv := invoker.New(c, cosigner)
subnetHash, err := nnsResolveHash(inv, nnsCs.Hash, subnetContract+".frostfs")
if err != nil {
return fmt.Errorf("subnet hash resolving: %w", err)
}
test, err := inv.Call(subnetHash, method, args...)
if err != nil {
return fmt.Errorf("test invocation: %w", err)
}
@ -876,17 +868,12 @@ func invokeNonNotary(c Client, key keys.PrivateKey, method string, args ...inter
return nil
}
func invokeNotary(c Client, key keys.PrivateKey, method string, notaryHash util.Uint160, args ...interface{}) error {
func invokeNotary(c Client, key keys.PrivateKey, method string, notaryHash util.Uint160, args ...any) error {
nnsCs, err := c.GetContractStateByID(1)
if err != nil {
return fmt.Errorf("NNS contract resolving: %w", err)
}
subnetHash, err := nnsResolveHash(c, nnsCs.Hash, subnetContract+".neofs")
if err != nil {
return fmt.Errorf("subnet hash resolving: %w", err)
}
alphabet, err := c.GetCommittee()
if err != nil {
return fmt.Errorf("alphabet list: %w", err)
@ -902,8 +889,15 @@ func invokeNotary(c Client, key keys.PrivateKey, method string, notaryHash util.
return fmt.Errorf("cosigners collecting: %w", err)
}
inv := invoker.New(c, cosigners)
subnetHash, err := nnsResolveHash(inv, nnsCs.Hash, subnetContract+".frostfs")
if err != nil {
return fmt.Errorf("subnet hash resolving: %w", err)
}
// make test invocation of the method
test, err := invokeFunction(c, subnetHash, method, args, cosigners)
test, err := inv.Call(subnetHash, method, args...)
if err != nil {
return fmt.Errorf("test invocation: %w", err)
}
@ -974,7 +968,7 @@ func invokeNotary(c Client, key keys.PrivateKey, method string, notaryHash util.
func notaryCosigners(c Client, notaryHash util.Uint160, nnsCs *state.Contract,
key keys.PrivateKey, alphabetAccount util.Uint160) ([]transaction.Signer, error) {
proxyHash, err := nnsResolveHash(c, nnsCs.Hash, proxyContract+".neofs")
proxyHash, err := nnsResolveHash(invoker.New(c, nil), nnsCs.Hash, proxyContract+".frostfs")
if err != nil {
return nil, fmt.Errorf("proxy hash resolving: %w", err)
}

View file

@ -0,0 +1,85 @@
package modules
import (
"os"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/storagecfg"
"github.com/TrueCloudLab/frostfs-node/misc"
"github.com/TrueCloudLab/frostfs-node/pkg/util/autocomplete"
utilConfig "github.com/TrueCloudLab/frostfs-node/pkg/util/config"
"github.com/TrueCloudLab/frostfs-node/pkg/util/gendoc"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
var (
rootCmd = &cobra.Command{
Use: "frostfs-adm",
Short: "FrostFS Administrative Tool",
Long: `FrostFS Administrative Tool provides functions to setup and
manage FrostFS network deployment.`,
RunE: entryPoint,
SilenceUsage: true,
}
)
func init() {
cobra.OnInitialize(func() { initConfig(rootCmd) })
// we need to init viper config to bind viper and cobra configurations for
// rpc endpoint, alphabet wallet dir, key credentials, etc.
// use stdout as default output for cmd.Print()
rootCmd.SetOut(os.Stdout)
rootCmd.PersistentFlags().StringP(commonflags.ConfigFlag, commonflags.ConfigFlagShorthand, "", commonflags.ConfigFlagUsage)
rootCmd.PersistentFlags().String(commonflags.ConfigDirFlag, "", commonflags.ConfigDirFlagUsage)
rootCmd.PersistentFlags().BoolP(commonflags.Verbose, commonflags.VerboseShorthand, false, commonflags.VerboseUsage)
_ = viper.BindPFlag(commonflags.Verbose, rootCmd.PersistentFlags().Lookup(commonflags.Verbose))
rootCmd.Flags().Bool("version", false, "Application version")
rootCmd.AddCommand(config.RootCmd)
rootCmd.AddCommand(morph.RootCmd)
rootCmd.AddCommand(storagecfg.RootCmd)
rootCmd.AddCommand(autocomplete.Command("frostfs-adm"))
rootCmd.AddCommand(gendoc.Command(rootCmd))
}
func Execute() error {
return rootCmd.Execute()
}
func entryPoint(cmd *cobra.Command, args []string) error {
printVersion, _ := cmd.Flags().GetBool("version")
if printVersion {
cmd.Print(misc.BuildInfo("FrostFS Adm"))
return nil
}
return cmd.Usage()
}
func initConfig(cmd *cobra.Command) {
configFile, err := cmd.Flags().GetString(commonflags.ConfigFlag)
if err != nil {
return
}
if configFile != "" {
viper.SetConfigType("yml")
viper.SetConfigFile(configFile)
_ = viper.ReadInConfig() // if config file is set but unavailable, ignore it
}
configDir, err := cmd.Flags().GetString(commonflags.ConfigDirFlag)
if err != nil {
return
}
if configDir != "" {
_ = utilConfig.ReadConfigDir(viper.GetViper(), configDir) // if config files cannot be read, ignore it
}
}

View file

@ -14,7 +14,7 @@ node:
relay: {{ .Relay }} # start Storage node in relay mode without bootstrapping into the Network map
subnet:
exit_zero: false # toggle entrance to zero subnet (overrides corresponding attribute and occurrence in entries)
entries: [] # list of IDs of subnets to enter in a text format of NeoFS API protocol (overrides corresponding attributes)
entries: [] # list of IDs of subnets to enter in a text format of FrostFS API protocol (overrides corresponding attributes)
grpc:
num: 1 # total number of listener endpoints
@ -37,15 +37,15 @@ control:
morph:
dial_timeout: 20s # timeout for side chain NEO RPC client connection
disable_cache: false # use TTL cache for side chain GET operations
cache_ttl: 15s # use TTL cache for side chain GET operations
rpc_endpoint: # side chain N3 RPC endpoints
{{- range .MorphRPC }}
- wss://{{.}}/ws{{end}}
- address: wss://{{.}}/ws{{end}}
{{if not .Relay }}
storage:
shard_pool_size: 15 # size of per-shard worker pools used for PUT operations
shard_num: 1 # total number of shards
shard:
default: # section with the default shard parameters
metabase:
perm: 0644 # permissions for metabase files(directories: +x for current user and group)
@ -68,8 +68,6 @@ storage:
gc:
remover_batch_size: 200 # number of objects to be removed by the garbage collector
remover_sleep_interval: 5m # frequency of the garbage collector invocation
shard:
0:
mode: "read-write" # mode of the shard, must be one of the: "read-write" (default), "read-only"

View file

@ -6,7 +6,6 @@ import (
"encoding/hex"
"errors"
"fmt"
"math/big"
"math/rand"
"net"
"net/url"
@ -17,20 +16,20 @@ import (
"text/template"
"time"
netutil "github.com/TrueCloudLab/frostfs-node/pkg/network"
"github.com/chzyer/readline"
"github.com/nspcc-dev/neo-go/cli/flags"
"github.com/nspcc-dev/neo-go/cli/input"
"github.com/nspcc-dev/neo-go/pkg/core/native/nativenames"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/nspcc-dev/neo-go/pkg/encoding/fixedn"
"github.com/nspcc-dev/neo-go/pkg/rpc/client"
"github.com/nspcc-dev/neo-go/pkg/smartcontract"
"github.com/nspcc-dev/neo-go/pkg/rpcclient"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/actor"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/gas"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/nep17"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/trigger"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm"
"github.com/nspcc-dev/neo-go/pkg/wallet"
netutil "github.com/nspcc-dev/neofs-node/pkg/network"
"github.com/spf13/cobra"
)
@ -41,8 +40,8 @@ const (
)
const (
defaultControlEndpoint = "127.0.0.1:8090"
defaultDataEndpoint = "127.0.0.1"
defaultControlEndpoint = "localhost:8090"
defaultDataEndpoint = "localhost"
)
// RootCmd is a root command of config section.
@ -55,8 +54,8 @@ var RootCmd = &cobra.Command{
func init() {
fs := RootCmd.Flags()
fs.StringP(walletFlag, "w", "", "path to wallet")
fs.StringP(accountFlag, "a", "", "wallet account")
fs.StringP(walletFlag, "w", "", "Path to wallet")
fs.StringP(accountFlag, "a", "", "Wallet account")
}
type config struct {
@ -91,7 +90,7 @@ func storageConfig(cmd *cobra.Command, args []string) {
}
}
historyPath := filepath.Join(os.TempDir(), "neofs-adm.history")
historyPath := filepath.Join(os.TempDir(), "frostfs-adm.history")
readline.SetHistoryPath(historyPath)
var c config
@ -154,7 +153,7 @@ func storageConfig(cmd *cobra.Command, args []string) {
validator := netutil.Address{}
err := validator.FromString(c.AnnouncedAddress)
if err != nil {
cmd.Println("Incorrect address format. See https://github.com/nspcc-dev/neofs-node/blob/master/pkg/network/address.go for details.")
cmd.Println("Incorrect address format. See https://github.com/TrueCloudLab/frostfs-node/blob/master/pkg/network/address.go for details.")
continue
}
uriAddr, err := url.Parse(validator.URIAddr())
@ -210,7 +209,7 @@ func storageConfig(cmd *cobra.Command, args []string) {
out := applyTemplate(c)
fatalOnErr(os.WriteFile(outPath, out, 0644))
cmd.Println("Node is ready for work! Run `neofs-node -config " + outPath + "`")
cmd.Println("Node is ready for work! Run `frostfs-node -config " + outPath + "`")
}
func getWalletAccount(w *wallet.Wallet, prompt string) string {
@ -322,23 +321,17 @@ func depositGas(cmd *cobra.Command, acc *wallet.Account, network string) {
sideClient := initClient(n3config[network].MorphRPC)
balanceHash, _ := util.Uint160DecodeStringLE(n3config[network].BalanceContract)
res, err := sideClient.InvokeFunction(balanceHash, "balanceOf", []smartcontract.Parameter{{
Type: smartcontract.Hash160Type,
Value: acc.Contract.ScriptHash(),
}}, nil)
fatalOnErr(err)
if res.State != vm.HaltState.String() {
fatalOnErr(fmt.Errorf("invalid response from balance contract: %s", res.FaultException))
sideActor, err := actor.NewSimple(sideClient, acc)
if err != nil {
fatalOnErr(fmt.Errorf("creating actor over side chain client: %w", err))
}
var balance *big.Int
if len(res.Stack) != 0 {
balance, _ = res.Stack[0].TryInteger()
}
sideGas := nep17.NewReader(sideActor, balanceHash)
accSH := acc.Contract.ScriptHash()
if balance == nil {
fatalOnErr(errors.New("invalid response from balance contract"))
balance, err := sideGas.BalanceOf(accSH)
if err != nil {
fatalOnErr(fmt.Errorf("side chain balance: %w", err))
}
ok := getConfirmation(false, fmt.Sprintf("Current NeoFS balance is %s, make a deposit? y/[n]: ",
@ -356,14 +349,17 @@ func depositGas(cmd *cobra.Command, acc *wallet.Account, network string) {
mainClient := initClient(n3config[network].RPC)
neofsHash, _ := util.Uint160DecodeStringLE(n3config[network].NeoFSContract)
gasHash, err := mainClient.GetNativeContractHash(nativenames.Gas)
fatalOnErr(err)
mainActor, err := actor.NewSimple(mainClient, acc)
if err != nil {
fatalOnErr(fmt.Errorf("creating actor over main chain client: %w", err))
}
tx, err := mainClient.CreateNEP17TransferTx(acc, neofsHash, gasHash, amount.Int64(), 0, nil, nil)
fatalOnErr(err)
mainGas := nep17.New(mainActor, gas.Hash)
txHash, err := mainClient.SignAndPushTx(tx, acc, nil)
fatalOnErr(err)
txHash, _, err := mainGas.Transfer(accSH, neofsHash, amount, nil)
if err != nil {
fatalOnErr(fmt.Errorf("sending TX to the NeoFS contract: %w", err))
}
cmd.Print("Waiting for transactions to persist.")
tick := time.NewTicker(time.Second / 2)
@ -394,8 +390,8 @@ loop:
}
}
func initClient(rpc []string) *client.Client {
var c *client.Client
func initClient(rpc []string) *rpcclient.Client {
var c *rpcclient.Client
var err error
shuffled := make([]string, len(rpc))
@ -403,7 +399,7 @@ func initClient(rpc []string) *client.Client {
rand.Shuffle(len(shuffled), func(i, j int) { shuffled[i], shuffled[j] = shuffled[j], shuffled[i] })
for _, endpoint := range shuffled {
c, err = client.New(context.Background(), "https://"+endpoint, client.Options{
c, err = rpcclient.New(context.Background(), "https://"+endpoint, rpcclient.Options{
DialTimeout: time.Second * 2,
RequestTimeout: time.Second * 5,
})

View file

@ -3,7 +3,7 @@ package main
import (
"os"
"github.com/nspcc-dev/neofs-node/cmd/neofs-adm/internal/modules"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules"
)
func main() {

View file

@ -0,0 +1,75 @@
# How FrostFS CLI uses session mechanism of the FrostFS
## Overview
FrostFS sessions implement a mechanism for issuing a power of attorney by one
party to another. A trusted party can provide a so-called session token as
proof of the right to act on behalf of another member of the network. The
client of operations carried out with such a token will be the user who opened
the session. The token contains information which limits power of attorney like
action context or lifetime.
The client confirms trust in a third party by signing its public (session) key
with his private key. Any operation signed using private session key with
attached session token is treated as performed by the original client.
## Types
FrostFS CLI supports two ways to execute operation within a session depending on
whether the user of the command application is an original user (1) or a trusted
one (2).
### Dynamic
For case (1) CLI user can only open dynamic sessions. Protocol call
`SessionService.Create` is used for this purpose. As a result of the call, a
private session key will be generated on the server, thus making the remote
server trusted. This type of session is useful when the client needs to
transfer part of the responsibility for the formation of strict system elements
to the trusted server. At the moment, the approach is applicable only to
creating objects.
```shell
$ frostfs-cli session create --rpc-endpoint <server_ip> --out ./blank_token
```
After this example command remote node holds session private key while its
public part is written into the session token encoded into the output file.
Later this token can be attached to the operations which support dynamic
sessions. Then the token will be finally formed and signed by CLI itself.
### Static
For case (2) CLI user can act on behalf of the person who issued the session
token to him. Unlike (1) the token must be fully prepared on the side of the
original client, and the CLI uses it only for reading. Ready token MUST have:
- correct context (object, container, etc.)
- valid lifetime
- public session key corresponding to the CLI key
- valid client signature
To sign the session token, exec:
```shell
$ frostfs-cli --wallet <client_wallet> util sign session-token --from ./blank_token --to ./token
```
Once the token is signed, it MUST NOT be modified.
## Commands
### Object
Here are sub-commands of `object` command which support only dynamic sessions (1):
- `put`
- `delete`
- `lock`
These commands accept blank token of the dynamically opened session or open
session internally if it has not been opened yet.
All other `object` sub-commands support only static sessions (2).
### Container
List of commands supporting sessions (static only):
- `create`
- `delete`
- `set-eacl`

View file

@ -0,0 +1,34 @@
# Extended headers
## Overview
Extended headers are used for request/response. They may contain any
user-defined headers to be interpreted on application level. Key name must be a
unique valid UTF-8 string. Value can't be empty. Requests or Responses with
duplicated header names or headers with empty values are considered invalid.
## Existing headers
There are some "well-known" headers starting with `__FROSTFS__` prefix that
affect system behaviour. For backward compatibility, the same set of
"well-known" headers may also use `__NEOFS__` prefix:
* `__FROSTFS__NETMAP_EPOCH` - netmap epoch to use for object placement calculation. The `value` is string
encoded `uint64` in decimal presentation. If set to '0' or omitted, the
current epoch only will be used.
* `__FROSTFS__NETMAP_LOOKUP_DEPTH` - if object can't be found using current epoch's netmap, this header limits
how many past epochs the node can look up through. Depth is applied to a current epoch or the value
of `__FROSTFS__NETMAP_EPOCH` attribute. The `value` is string encoded `uint64` in decimal presentation.
If set to '0' or not set, only the current epoch is used.
## `frostfs-cli` commands with `--xhdr`
List of commands with support of extended headers:
* `container list-objects`
* `object delete/get/hash/head/lock/put/range/search`
* `storagegroup delete/get/list/put`
Example:
```shell
$ frostfs-cli object put -r s01.frostfs.devenv:8080 -w wallet.json --cid CID --file FILE --xhdr "__FROSTFS__NETMAP_EPOCH=777"
```

View file

@ -7,15 +7,15 @@ import (
"fmt"
"io"
"github.com/nspcc-dev/neofs-sdk-go/accounting"
"github.com/nspcc-dev/neofs-sdk-go/client"
"github.com/nspcc-dev/neofs-sdk-go/container"
cid "github.com/nspcc-dev/neofs-sdk-go/container/id"
"github.com/nspcc-dev/neofs-sdk-go/eacl"
"github.com/nspcc-dev/neofs-sdk-go/netmap"
"github.com/nspcc-dev/neofs-sdk-go/object"
oid "github.com/nspcc-dev/neofs-sdk-go/object/id"
"github.com/nspcc-dev/neofs-sdk-go/version"
"github.com/TrueCloudLab/frostfs-sdk-go/accounting"
"github.com/TrueCloudLab/frostfs-sdk-go/client"
containerSDK "github.com/TrueCloudLab/frostfs-sdk-go/container"
cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
"github.com/TrueCloudLab/frostfs-sdk-go/eacl"
"github.com/TrueCloudLab/frostfs-sdk-go/netmap"
"github.com/TrueCloudLab/frostfs-sdk-go/object"
oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
"github.com/TrueCloudLab/frostfs-sdk-go/version"
)
// BalanceOfPrm groups parameters of BalanceOf operation.
@ -30,11 +30,11 @@ type BalanceOfRes struct {
}
// Balance returns the current balance.
func (x BalanceOfRes) Balance() *accounting.Decimal {
func (x BalanceOfRes) Balance() accounting.Decimal {
return x.cliRes.Amount()
}
// BalanceOf requests the current balance of a NeoFS user.
// BalanceOf requests the current balance of a FrostFS user.
//
// Returns any error which prevented the operation from completing correctly in error return.
func BalanceOf(prm BalanceOfPrm) (res BalanceOfRes, err error) {
@ -59,7 +59,7 @@ func (x ListContainersRes) IDList() []cid.ID {
return x.cliRes.Containers()
}
// ListContainers requests a list of NeoFS user's containers.
// ListContainers requests a list of FrostFS user's containers.
//
// Returns any error which prevented the operation from completing correctly in error return.
func ListContainers(prm ListContainersPrm) (res ListContainersRes, err error) {
@ -84,7 +84,7 @@ func (x PutContainerRes) ID() cid.ID {
return x.cnr
}
// PutContainer sends a request to save the container in NeoFS.
// PutContainer sends a request to save the container in FrostFS.
//
// Operation is asynchronous and not guaranteed even in the absence of errors.
// The required time is also not predictable.
@ -95,12 +95,7 @@ func (x PutContainerRes) ID() cid.ID {
func PutContainer(prm PutContainerPrm) (res PutContainerRes, err error) {
cliRes, err := prm.cli.ContainerPut(context.Background(), prm.PrmContainerPut)
if err == nil {
cnr := cliRes.ID()
if cnr == nil {
err = errors.New("missing container ID in response")
} else {
res.cnr = *cnr
}
res.cnr = cliRes.ID()
}
return
@ -123,11 +118,11 @@ type GetContainerRes struct {
}
// Container returns structured of the requested container.
func (x GetContainerRes) Container() container.Container {
func (x GetContainerRes) Container() containerSDK.Container {
return x.cliRes.Container()
}
// GetContainer reads a container from NeoFS by ID.
// GetContainer reads a container from FrostFS by ID.
//
// Returns any error which prevented the operation from completing correctly in error return.
func GetContainer(prm GetContainerPrm) (res GetContainerRes, err error) {
@ -136,6 +131,21 @@ func GetContainer(prm GetContainerPrm) (res GetContainerRes, err error) {
return
}
// IsACLExtendable checks if ACL of the container referenced by the given identifier
// can be extended. Client connection MUST BE correctly established in advance.
func IsACLExtendable(c *client.Client, cnr cid.ID) (bool, error) {
var prm GetContainerPrm
prm.SetClient(c)
prm.SetContainer(cnr)
res, err := GetContainer(prm)
if err != nil {
return false, fmt.Errorf("get container from the FrostFS: %w", err)
}
return res.Container().BasicACL().Extendable(), nil
}
// DeleteContainerPrm groups parameters of DeleteContainerPrm operation.
type DeleteContainerPrm struct {
commonPrm
@ -145,7 +155,7 @@ type DeleteContainerPrm struct {
// DeleteContainerRes groups the resulting values of DeleteContainer operation.
type DeleteContainerRes struct{}
// DeleteContainer sends a request to remove a container from NeoFS by ID.
// DeleteContainer sends a request to remove a container from FrostFS by ID.
//
// Operation is asynchronous and not guaranteed even in the absence of errors.
// The required time is also not predictable.
@ -171,11 +181,11 @@ type EACLRes struct {
}
// EACL returns requested eACL table.
func (x EACLRes) EACL() *eacl.Table {
func (x EACLRes) EACL() eacl.Table {
return x.cliRes.Table()
}
// EACL reads eACL table from NeoFS by container ID.
// EACL reads eACL table from FrostFS by container ID.
//
// Returns any error which prevented the operation from completing correctly in error return.
func EACL(prm EACLPrm) (res EACLRes, err error) {
@ -193,7 +203,7 @@ type SetEACLPrm struct {
// SetEACLRes groups the resulting values of SetEACL operation.
type SetEACLRes struct{}
// SetEACL requests to save an eACL table in NeoFS.
// SetEACL requests to save an eACL table in FrostFS.
//
// Operation is asynchronous and no guaranteed even in the absence of errors.
// The required time is also not predictable.
@ -218,12 +228,12 @@ type NetworkInfoRes struct {
cliRes *client.ResNetworkInfo
}
// NetworkInfo returns structured information about the NeoFS network.
func (x NetworkInfoRes) NetworkInfo() *netmap.NetworkInfo {
// NetworkInfo returns structured information about the FrostFS network.
func (x NetworkInfoRes) NetworkInfo() netmap.NetworkInfo {
return x.cliRes.Info()
}
// NetworkInfo reads information about the NeoFS network.
// NetworkInfo reads information about the FrostFS network.
//
// Returns any error which prevented the operation from completing correctly in error return.
func NetworkInfo(prm NetworkInfoPrm) (res NetworkInfoRes, err error) {
@ -245,15 +255,15 @@ type NodeInfoRes struct {
// NodeInfo returns information about the node from netmap.
func (x NodeInfoRes) NodeInfo() netmap.NodeInfo {
return *x.cliRes.NodeInfo()
return x.cliRes.NodeInfo()
}
// LatestVersion returns the latest NeoFS API version in use.
func (x NodeInfoRes) LatestVersion() *version.Version {
// LatestVersion returns the latest FrostFS API version in use.
func (x NodeInfoRes) LatestVersion() version.Version {
return x.cliRes.LatestVersion()
}
// NodeInfo requests information about the remote server from NeoFS netmap.
// NodeInfo requests information about the remote server from FrostFS netmap.
//
// Returns any error which prevented the operation from completing correctly in error return.
func NodeInfo(prm NodeInfoPrm) (res NodeInfoRes, err error) {
@ -262,6 +272,29 @@ func NodeInfo(prm NodeInfoPrm) (res NodeInfoRes, err error) {
return
}
// NetMapSnapshotPrm groups parameters of NetMapSnapshot operation.
type NetMapSnapshotPrm struct {
commonPrm
}
// NetMapSnapshotRes groups the resulting values of NetMapSnapshot operation.
type NetMapSnapshotRes struct {
cliRes *client.ResNetMapSnapshot
}
// NetMap returns current local snapshot of the FrostFS network map.
func (x NetMapSnapshotRes) NetMap() netmap.NetMap {
return x.cliRes.NetMap()
}
// NetMapSnapshot requests current network view of the remote server.
//
// Returns any error which prevented the operation from completing correctly in error return.
func NetMapSnapshot(prm NetMapSnapshotPrm) (res NetMapSnapshotRes, err error) {
res.cliRes, err = prm.cli.NetMapSnapshot(context.Background(), client.PrmNetMapSnapshot{})
return
}
// CreateSessionPrm groups parameters of CreateSession operation.
type CreateSessionPrm struct {
commonPrm
@ -329,31 +362,31 @@ func (x PutObjectRes) ID() oid.ID {
return x.id
}
// PutObject saves the object in NeoFS network.
// PutObject saves the object in FrostFS network.
//
// Returns any error which prevented the operation from completing correctly in error return.
func PutObject(prm PutObjectPrm) (*PutObjectRes, error) {
var putPrm client.PrmObjectPutInit
if prm.sessionToken != nil {
putPrm.WithinSession(*prm.sessionToken)
}
if prm.bearerToken != nil {
putPrm.WithBearerToken(*prm.bearerToken)
}
if prm.local {
putPrm.MarkLocal()
}
putPrm.WithXHeaders(prm.xHeaders...)
wrt, err := prm.cli.ObjectPutInit(context.Background(), putPrm)
if err != nil {
return nil, fmt.Errorf("init object writing: %w", err)
}
if prm.sessionToken != nil {
wrt.WithinSession(*prm.sessionToken)
}
if prm.bearerToken != nil {
wrt.WithBearerToken(*prm.bearerToken)
}
if prm.local {
wrt.MarkLocal()
}
wrt.WithXHeaders(prm.xHeaders...)
if wrt.WriteHeader(*prm.hdr) {
if prm.headerCallback != nil {
prm.headerCallback(prm.hdr)
@ -371,8 +404,7 @@ func PutObject(prm PutObjectPrm) (*PutObjectRes, error) {
}
if prm.rdr != nil {
// TODO: (neofs-node#1198) explore better values or configure it
const defaultBufferSizePut = 4096
const defaultBufferSizePut = 3 << 20 // Maximum chunk size is 3 MiB in the SDK.
if sz == 0 || sz > defaultBufferSizePut {
sz = defaultBufferSizePut
@ -406,13 +438,9 @@ func PutObject(prm PutObjectPrm) (*PutObjectRes, error) {
return nil, fmt.Errorf("client failure: %w", err)
}
var res PutObjectRes
if !cliRes.ReadStoredObjectID(&res.id) {
return nil, errors.New("missing ID of the stored object")
}
return &res, nil
return &PutObjectRes{
id: cliRes.StoredObjectID(),
}, nil
}
// DeleteObjectPrm groups parameters of DeleteObject operation.
@ -431,7 +459,7 @@ func (x DeleteObjectRes) Tombstone() oid.ID {
return x.tomb
}
// DeleteObject marks an object to be removed from NeoFS through tombstone placement.
// DeleteObject marks an object to be removed from FrostFS through tombstone placement.
//
// Returns any error which prevented the operation from completing correctly in error return.
func DeleteObject(prm DeleteObjectPrm) (*DeleteObjectRes, error) {
@ -454,14 +482,8 @@ func DeleteObject(prm DeleteObjectPrm) (*DeleteObjectRes, error) {
return nil, fmt.Errorf("remove object via client: %w", err)
}
var id oid.ID
if !cliRes.ReadTombstoneID(&id) {
return nil, errors.New("object removed but tombstone ID is missing")
}
return &DeleteObjectRes{
tomb: id,
tomb: cliRes.Tombstone(),
}, nil
}
@ -553,7 +575,7 @@ type HeadObjectPrm struct {
mainOnly bool
}
// SetMainOnlyFlag sets flag to get only main fields of an object header in terms of NeoFS API.
// SetMainOnlyFlag sets flag to get only main fields of an object header in terms of FrostFS API.
func (x *HeadObjectPrm) SetMainOnlyFlag(v bool) {
x.mainOnly = v
}
@ -789,7 +811,7 @@ func (x *PayloadRangePrm) SetRange(rng *object.Range) {
// PayloadRangeRes groups the resulting values of PayloadRange operation.
type PayloadRangeRes struct{}
// PayloadRange reads object payload range from NeoFS and writes it to the specified writer.
// PayloadRange reads object payload range from FrostFS and writes it to the specified writer.
//
// Interrupts on any writer error.
//
@ -833,3 +855,37 @@ func PayloadRange(prm PayloadRangePrm) (*PayloadRangeRes, error) {
return new(PayloadRangeRes), nil
}
// SyncContainerPrm groups parameters of SyncContainerSettings operation.
type SyncContainerPrm struct {
commonPrm
c *containerSDK.Container
}
// SetContainer sets a container that is required to be synced.
func (s *SyncContainerPrm) SetContainer(c *containerSDK.Container) {
s.c = c
}
// SyncContainerRes groups resulting values of SyncContainerSettings
// operation.
type SyncContainerRes struct{}
// SyncContainerSettings reads global network config from FrostFS and
// syncs container settings with it.
//
// Interrupts on any writer error.
//
// Panics if a container passed as a parameter is nil.
func SyncContainerSettings(prm SyncContainerPrm) (*SyncContainerRes, error) {
if prm.c == nil {
panic("sync container settings with the network: nil container")
}
err := client.SyncContainerWithNetwork(context.Background(), prm.c, prm.cli)
if err != nil {
return nil, err
}
return new(SyncContainerRes), nil
}

View file

@ -0,0 +1,15 @@
// Package internal provides functionality for FrostFS CLI application
// communication with FrostFS network.
//
// The base client for accessing remote nodes via FrostFS API is a FrostFS SDK
// Go API client. However, although it encapsulates a useful piece of business
// logic (e.g. the signature mechanism), the FrostFS CLI application does not
// fully use the client's flexible interface.
//
// In this regard, this package provides functions over base API client
// necessary for the application. This allows you to concentrate the entire
// spectrum of the client's use in one place (this will be convenient both when
// updating the base client and for evaluating the UX of SDK library). So it is
// expected that all application packages will be limited to this package for
// the development of functionality requiring FrostFS API communication.
package internal

View file

@ -3,11 +3,11 @@ package internal
import (
"io"
"github.com/nspcc-dev/neofs-sdk-go/bearer"
"github.com/nspcc-dev/neofs-sdk-go/client"
cid "github.com/nspcc-dev/neofs-sdk-go/container/id"
oid "github.com/nspcc-dev/neofs-sdk-go/object/id"
"github.com/nspcc-dev/neofs-sdk-go/session"
"github.com/TrueCloudLab/frostfs-sdk-go/bearer"
"github.com/TrueCloudLab/frostfs-sdk-go/client"
cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
"github.com/TrueCloudLab/frostfs-sdk-go/session"
)
// here are small structures with public setters to share between parameter structures
@ -16,7 +16,7 @@ type commonPrm struct {
cli *client.Client
}
// SetClient sets the base client for NeoFS API communication.
// SetClient sets the base client for FrostFS API communication.
func (x *commonPrm) SetClient(cli *client.Client) {
x.cli = cli
}

View file

@ -0,0 +1,96 @@
package internal
import (
"context"
"crypto/ecdsa"
"crypto/elliptic"
"crypto/rand"
"errors"
"fmt"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-node/pkg/network"
"github.com/TrueCloudLab/frostfs-sdk-go/client"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
var errInvalidEndpoint = errors.New("provided RPC endpoint is incorrect")
// GetSDKClientByFlag returns default frostfs-sdk-go client using the specified flag for the address.
// On error, outputs to stderr of cmd and exits with non-zero code.
func GetSDKClientByFlag(cmd *cobra.Command, key *ecdsa.PrivateKey, endpointFlag string) *client.Client {
cli, err := getSDKClientByFlag(cmd, key, endpointFlag)
if err != nil {
commonCmd.ExitOnErr(cmd, "can't create API client: %w", err)
}
return cli
}
func getSDKClientByFlag(cmd *cobra.Command, key *ecdsa.PrivateKey, endpointFlag string) (*client.Client, error) {
var addr network.Address
err := addr.FromString(viper.GetString(endpointFlag))
if err != nil {
return nil, fmt.Errorf("%v: %w", errInvalidEndpoint, err)
}
return GetSDKClient(cmd, key, addr)
}
// GetSDKClient returns default frostfs-sdk-go client.
func GetSDKClient(cmd *cobra.Command, key *ecdsa.PrivateKey, addr network.Address) (*client.Client, error) {
var (
c client.Client
prmInit client.PrmInit
prmDial client.PrmDial
)
prmInit.SetDefaultPrivateKey(*key)
prmInit.ResolveNeoFSFailures()
prmDial.SetServerURI(addr.URIAddr())
if timeout := viper.GetDuration(commonflags.Timeout); timeout > 0 {
// In CLI we can only set a timeout for the whole operation.
// By also setting stream timeout we ensure that no operation hands
// for too long.
prmDial.SetTimeout(timeout)
prmDial.SetStreamTimeout(timeout)
common.PrintVerbose(cmd, "Set request timeout to %s.", timeout)
}
c.Init(prmInit)
if err := c.Dial(prmDial); err != nil {
return nil, fmt.Errorf("can't init SDK client: %w", err)
}
return &c, nil
}
// GetCurrentEpoch returns current epoch.
func GetCurrentEpoch(ctx context.Context, cmd *cobra.Command, endpoint string) (uint64, error) {
var addr network.Address
if err := addr.FromString(endpoint); err != nil {
return 0, fmt.Errorf("can't parse RPC endpoint: %w", err)
}
key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
if err != nil {
return 0, fmt.Errorf("can't generate key to sign query: %w", err)
}
c, err := GetSDKClient(cmd, key, addr)
if err != nil {
return 0, err
}
ni, err := c.NetworkInfo(ctx, client.PrmNetworkInfo{})
if err != nil {
return 0, err
}
return ni.Info().CurrentEpoch(), nil
}

View file

@ -4,9 +4,10 @@ import (
"errors"
"os"
"github.com/nspcc-dev/neofs-node/pkg/core/version"
"github.com/nspcc-dev/neofs-sdk-go/eacl"
versionSDK "github.com/nspcc-dev/neofs-sdk-go/version"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-node/pkg/core/version"
"github.com/TrueCloudLab/frostfs-sdk-go/eacl"
versionSDK "github.com/TrueCloudLab/frostfs-sdk-go/version"
"github.com/spf13/cobra"
)
@ -16,29 +17,29 @@ var errUnsupportedEACLFormat = errors.New("unsupported eACL format")
func ReadEACL(cmd *cobra.Command, eaclPath string) *eacl.Table {
_, err := os.Stat(eaclPath) // check if `eaclPath` is an existing file
if err != nil {
ExitOnErr(cmd, "", errors.New("incorrect path to file with EACL"))
commonCmd.ExitOnErr(cmd, "", errors.New("incorrect path to file with EACL"))
}
PrintVerbose("Reading EACL from file: %s", eaclPath)
PrintVerbose(cmd, "Reading EACL from file: %s", eaclPath)
data, err := os.ReadFile(eaclPath)
ExitOnErr(cmd, "can't read file with EACL: %w", err)
commonCmd.ExitOnErr(cmd, "can't read file with EACL: %w", err)
table := eacl.NewTable()
if err = table.UnmarshalJSON(data); err == nil {
validateAndFixEACLVersion(table)
PrintVerbose("Parsed JSON encoded EACL table")
PrintVerbose(cmd, "Parsed JSON encoded EACL table")
return table
}
if err = table.Unmarshal(data); err == nil {
validateAndFixEACLVersion(table)
PrintVerbose("Parsed binary encoded EACL table")
PrintVerbose(cmd, "Parsed binary encoded EACL table")
return table
}
ExitOnErr(cmd, "", errUnsupportedEACLFormat)
commonCmd.ExitOnErr(cmd, "", errUnsupportedEACLFormat)
return nil
}

View file

@ -0,0 +1,28 @@
package common
import (
"fmt"
"strconv"
"github.com/spf13/cobra"
)
// ParseEpoch parses epoch argument. Second return value is true if
// the specified epoch is relative, and false otherwise.
func ParseEpoch(cmd *cobra.Command, flag string) (uint64, bool, error) {
s, _ := cmd.Flags().GetString(flag)
if len(s) == 0 {
return 0, false, nil
}
relative := s[0] == '+'
if relative {
s = s[1:]
}
epoch, err := strconv.ParseUint(s, 10, 64)
if err != nil {
return 0, relative, fmt.Errorf("can't parse epoch for %s argument: %w", flag, err)
}
return epoch, relative, nil
}

View file

@ -11,12 +11,12 @@ import (
func PrettyPrintJSON(cmd *cobra.Command, m json.Marshaler, entity string) {
data, err := m.MarshalJSON()
if err != nil {
PrintVerbose("Can't convert %s to json: %w", entity, err)
PrintVerbose(cmd, "Can't convert %s to json: %w", entity, err)
return
}
buf := new(bytes.Buffer)
if err := json.Indent(buf, data, "", " "); err != nil {
PrintVerbose("Can't pretty print json: %w", err)
PrintVerbose(cmd, "Can't pretty print json: %w", err)
return
}
cmd.Println(buf)

View file

@ -0,0 +1,67 @@
package common
import (
"encoding/json"
"errors"
"fmt"
"os"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-sdk-go/bearer"
"github.com/spf13/cobra"
)
// ReadBearerToken reads bearer token from the path provided in a specified flag.
func ReadBearerToken(cmd *cobra.Command, flagname string) *bearer.Token {
path, err := cmd.Flags().GetString(flagname)
commonCmd.ExitOnErr(cmd, "", err)
if len(path) == 0 {
return nil
}
PrintVerbose(cmd, "Reading bearer token from file [%s]...", path)
var tok bearer.Token
err = ReadBinaryOrJSON(cmd, &tok, path)
commonCmd.ExitOnErr(cmd, "invalid bearer token: %v", err)
return &tok
}
// BinaryOrJSON is an interface of entities which provide json.Unmarshaler
// and FrostFS binary decoder.
type BinaryOrJSON interface {
Unmarshal([]byte) error
json.Unmarshaler
}
// ReadBinaryOrJSON reads file data using provided path and decodes
// BinaryOrJSON from the data.
func ReadBinaryOrJSON(cmd *cobra.Command, dst BinaryOrJSON, fPath string) error {
PrintVerbose(cmd, "Reading file [%s]...", fPath)
// try to read session token from file
data, err := os.ReadFile(fPath)
if err != nil {
return fmt.Errorf("read file <%s>: %w", fPath, err)
}
PrintVerbose(cmd, "Trying to decode binary...")
err = dst.Unmarshal(data)
if err != nil {
PrintVerbose(cmd, "Failed to decode binary: %v", err)
PrintVerbose(cmd, "Trying to decode JSON...")
err = dst.UnmarshalJSON(data)
if err != nil {
PrintVerbose(cmd, "Failed to decode JSON: %v", err)
return errors.New("invalid format")
}
}
return nil
}

View file

@ -2,20 +2,19 @@ package common
import (
"encoding/hex"
"fmt"
"strconv"
"time"
"github.com/nspcc-dev/neofs-node/cmd/neofs-cli/internal/commonflags"
"github.com/nspcc-dev/neofs-sdk-go/checksum"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/TrueCloudLab/frostfs-sdk-go/checksum"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
// PrintVerbose prints to the stdout if the commonflags.Verbose flag is on.
func PrintVerbose(format string, a ...interface{}) {
func PrintVerbose(cmd *cobra.Command, format string, a ...any) {
if viper.GetBool(commonflags.Verbose) {
fmt.Printf(format+"\n", a...)
cmd.Printf(format+"\n", a...)
}
}

View file

@ -0,0 +1,9 @@
package commonflags
const (
// ExpireAt is a flag for setting last epoch of an object or a token.
ExpireAt = "expire-at"
// Lifetime is a flag for setting the lifetime of an object or a token,
// starting from the current epoch.
Lifetime = "lifetime"
)

View file

@ -1,6 +1,8 @@
package commonflags
import (
"time"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
@ -11,38 +13,54 @@ const (
GenerateKey = "generate-key"
generateKeyShorthand = "g"
generateKeyDefault = false
generateKeyUsage = "generate new private key"
generateKeyUsage = "Generate new private key"
WalletPath = "wallet"
WalletPathShorthand = "w"
WalletPathDefault = ""
WalletPathUsage = "WIF (NEP-2) string or path to the wallet or binary key"
WalletPathUsage = "Path to the wallet or binary key"
Account = "address"
AccountShorthand = ""
AccountDefault = ""
AccountUsage = "address of wallet account"
AccountUsage = "Address of wallet account"
RPC = "rpc-endpoint"
RPCShorthand = "r"
RPCDefault = ""
RPCUsage = "remote node address (as 'multiaddr' or '<host>:<port>')"
RPCUsage = "Remote node address (as 'multiaddr' or '<host>:<port>')"
Timeout = "timeout"
TimeoutShorthand = "t"
TimeoutDefault = 15 * time.Second
TimeoutUsage = "Timeout for an operation"
Verbose = "verbose"
VerboseShorthand = "v"
VerboseUsage = "verbose output"
VerboseUsage = "Verbose output"
ForceFlag = "force"
ForceFlagShorthand = "f"
CIDFlag = "cid"
CIDFlagUsage = "Container ID."
OIDFlag = "oid"
OIDFlagUsage = "Object ID."
)
// Init adds common flags to the command:
// - GenerateKey
// - WalletPath
// - Account
// - RPC
// - GenerateKey,
// - WalletPath,
// - Account,
// - RPC,
// - Timeout.
func Init(cmd *cobra.Command) {
InitWithoutRPC(cmd)
ff := cmd.Flags()
ff.StringP(RPC, RPCShorthand, RPCDefault, RPCUsage)
ff.DurationP(Timeout, TimeoutShorthand, TimeoutDefault, TimeoutUsage)
}
// InitWithoutRPC is similar to Init but doesn't create the RPC flag.
@ -62,4 +80,5 @@ func Bind(cmd *cobra.Command) {
_ = viper.BindPFlag(WalletPath, ff.Lookup(WalletPath))
_ = viper.BindPFlag(Account, ff.Lookup(Account))
_ = viper.BindPFlag(RPC, ff.Lookup(RPC))
_ = viper.BindPFlag(Timeout, ff.Lookup(Timeout))
}

View file

@ -0,0 +1,19 @@
package commonflags
import (
"fmt"
"github.com/spf13/cobra"
)
const SessionToken = "session"
// InitSession registers SessionToken flag representing file path to the token of
// the session with the given name. Supports FrostFS-binary and JSON files.
func InitSession(cmd *cobra.Command, name string) {
cmd.Flags().String(
SessionToken,
"",
fmt.Sprintf("Filepath to a JSON- or binary-encoded token of the %s session", name),
)
}

View file

@ -7,15 +7,22 @@ import (
"path/filepath"
"testing"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/nspcc-dev/neo-go/cli/input"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/nspcc-dev/neofs-node/cmd/neofs-cli/internal/commonflags"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"github.com/stretchr/testify/require"
"golang.org/x/term"
)
var testCmd = &cobra.Command{
Use: "test",
Short: "test",
Run: func(cmd *cobra.Command, args []string) {},
}
func Test_getOrGenerate(t *testing.T) {
dir := t.TempDir()
@ -23,6 +30,9 @@ func Test_getOrGenerate(t *testing.T) {
w, err := wallet.NewWallet(wallPath)
require.NoError(t, err)
badWallPath := filepath.Join(dir, "bad_wallet.json")
require.NoError(t, os.WriteFile(badWallPath, []byte("bad content"), os.ModePerm))
acc1, err := wallet.NewAccount()
require.NoError(t, err)
require.NoError(t, acc1.Encrypt("pass", keys.NEP2ScryptParams()))
@ -34,7 +44,6 @@ func Test_getOrGenerate(t *testing.T) {
acc2.Default = true
w.AddAccount(acc2)
require.NoError(t, w.Save())
w.Close()
keyPath := filepath.Join(dir, "binary.key")
rawKey, err := keys.NewPrivateKey()
@ -55,7 +64,8 @@ func Test_getOrGenerate(t *testing.T) {
Writer: io.Discard,
}, "")
checkKeyError(t, filepath.Join(dir, "badfile"), ErrInvalidKey)
checkKeyError(t, filepath.Join(dir, "badfile"), ErrFs)
checkKeyError(t, badWallPath, ErrInvalidKey)
t.Run("wallet", func(t *testing.T) {
checkKeyError(t, wallPath, ErrInvalidPassword)
@ -80,27 +90,11 @@ func Test_getOrGenerate(t *testing.T) {
})
t.Run("WIF", func(t *testing.T) {
checkKey(t, wifKey.WIF(), wifKey)
checkKeyError(t, wifKey.WIF(), ErrFs)
})
t.Run("NEP-2", func(t *testing.T) {
checkKeyError(t, nep2, ErrInvalidPassword)
in.WriteString("invalid\r")
checkKeyError(t, nep2, ErrInvalidPassword)
in.WriteString("pass\r")
checkKey(t, nep2, nep2Key)
t.Run("password from config", func(t *testing.T) {
viper.Set("password", "invalid")
in.WriteString("pass\r")
checkKeyError(t, nep2, ErrInvalidPassword)
viper.Set("password", "pass")
in.WriteString("invalid\r")
checkKey(t, nep2, nep2Key)
})
checkKeyError(t, nep2, ErrFs)
})
t.Run("raw key", func(t *testing.T) {
@ -109,7 +103,7 @@ func Test_getOrGenerate(t *testing.T) {
t.Run("generate", func(t *testing.T) {
viper.Set(commonflags.GenerateKey, true)
actual, err := getOrGenerate()
actual, err := getOrGenerate(testCmd)
require.NoError(t, err)
require.NotNil(t, actual)
for _, p := range []*keys.PrivateKey{nep2Key, rawKey, wifKey, acc1.PrivateKey(), acc2.PrivateKey()} {
@ -120,13 +114,13 @@ func Test_getOrGenerate(t *testing.T) {
func checkKeyError(t *testing.T, desc string, err error) {
viper.Set(commonflags.WalletPath, desc)
_, actualErr := getOrGenerate()
_, actualErr := getOrGenerate(testCmd)
require.ErrorIs(t, actualErr, err)
}
func checkKey(t *testing.T, desc string, expected *keys.PrivateKey) {
viper.Set(commonflags.WalletPath, desc)
actual, err := getOrGenerate()
actual, err := getOrGenerate(testCmd)
require.NoError(t, err)
require.Equal(t, &expected.PrivateKey, actual)
}

View file

@ -0,0 +1,62 @@
package key
import (
"crypto/ecdsa"
"errors"
"fmt"
"os"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
var errCantGenerateKey = errors.New("can't generate new private key")
// Get returns private key from wallet or binary file.
// Ideally we want to touch file-system on the last step.
// This function assumes that all flags were bind to viper in a `PersistentPreRun`.
func Get(cmd *cobra.Command) *ecdsa.PrivateKey {
pk, err := get(cmd)
commonCmd.ExitOnErr(cmd, "can't fetch private key: %w", err)
return pk
}
func get(cmd *cobra.Command) (*ecdsa.PrivateKey, error) {
keyDesc := viper.GetString(commonflags.WalletPath)
data, err := os.ReadFile(keyDesc)
if err != nil {
return nil, fmt.Errorf("%w: %v", ErrFs, err)
}
priv, err := keys.NewPrivateKeyFromBytes(data)
if err != nil {
w, err := wallet.NewWalletFromFile(keyDesc)
if err == nil {
return FromWallet(cmd, w, viper.GetString(commonflags.Account))
}
return nil, fmt.Errorf("%w: %v", ErrInvalidKey, err)
}
return &priv.PrivateKey, nil
}
// GetOrGenerate is similar to get but generates a new key if commonflags.GenerateKey is set.
func GetOrGenerate(cmd *cobra.Command) *ecdsa.PrivateKey {
pk, err := getOrGenerate(cmd)
commonCmd.ExitOnErr(cmd, "can't fetch private key: %w", err)
return pk
}
func getOrGenerate(cmd *cobra.Command) (*ecdsa.PrivateKey, error) {
if viper.GetBool(commonflags.GenerateKey) {
priv, err := keys.NewPrivateKey()
if err != nil {
return nil, fmt.Errorf("%w: %v", errCantGenerateKey, err)
}
return &priv.PrivateKey, nil
}
return get(cmd)
}

View file

@ -3,55 +3,57 @@ package key
import (
"crypto/ecdsa"
"errors"
"fmt"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"github.com/nspcc-dev/neo-go/cli/flags"
"github.com/nspcc-dev/neo-go/cli/input"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
// Key-related errors.
var (
ErrInvalidKey = errors.New("provided key is incorrect")
ErrFs = errors.New("unable to read file from given path")
ErrInvalidKey = errors.New("provided key is incorrect, only wallet or binary key supported")
ErrInvalidAddress = errors.New("--address option must be specified and valid")
ErrInvalidPassword = errors.New("invalid password for the encrypted key")
)
// FromWallet returns private key of the wallet account.
func FromWallet(w *wallet.Wallet, addrStr string) (*ecdsa.PrivateKey, error) {
func FromWallet(cmd *cobra.Command, w *wallet.Wallet, addrStr string) (*ecdsa.PrivateKey, error) {
var (
addr util.Uint160
err error
)
if addrStr == "" {
printVerbose("Using default wallet address")
common.PrintVerbose(cmd, "Using default wallet address")
addr = w.GetChangeAddress()
} else {
addr, err = flags.ParseAddress(addrStr)
if err != nil {
printVerbose("Can't parse address: %s", addrStr)
common.PrintVerbose(cmd, "Can't parse address: %s", addrStr)
return nil, ErrInvalidAddress
}
}
acc := w.GetAccount(addr)
if acc == nil {
printVerbose("Can't find wallet account for %s", addrStr)
common.PrintVerbose(cmd, "Can't find wallet account for %s", addrStr)
return nil, ErrInvalidAddress
}
pass, err := getPassword()
if err != nil {
printVerbose("Can't read password: %v", err)
common.PrintVerbose(cmd, "Can't read password: %v", err)
return nil, ErrInvalidPassword
}
if err := acc.Decrypt(pass, keys.NEP2ScryptParams()); err != nil {
printVerbose("Can't decrypt account: %v", err)
common.PrintVerbose(cmd, "Can't decrypt account: %v", err)
return nil, ErrInvalidPassword
}
@ -66,9 +68,3 @@ func getPassword() (string, error) {
return input.ReadPassword("Enter password > ")
}
func printVerbose(format string, a ...interface{}) {
if viper.GetBool("verbose") {
fmt.Printf(format+"\n", a...)
}
}

7
cmd/frostfs-cli/main.go Normal file
View file

@ -0,0 +1,7 @@
package main
import cmd "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules"
func main() {
cmd.Execute()
}

View file

@ -3,14 +3,14 @@ package accounting
import (
"math/big"
internalclient "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-node/pkg/util/precision"
"github.com/TrueCloudLab/frostfs-sdk-go/accounting"
"github.com/TrueCloudLab/frostfs-sdk-go/user"
"github.com/nspcc-dev/neo-go/pkg/encoding/fixedn"
internalclient "github.com/nspcc-dev/neofs-node/cmd/neofs-cli/internal/client"
"github.com/nspcc-dev/neofs-node/cmd/neofs-cli/internal/common"
"github.com/nspcc-dev/neofs-node/cmd/neofs-cli/internal/commonflags"
"github.com/nspcc-dev/neofs-node/cmd/neofs-cli/internal/key"
"github.com/nspcc-dev/neofs-node/pkg/util/precision"
"github.com/nspcc-dev/neofs-sdk-go/accounting"
"github.com/nspcc-dev/neofs-sdk-go/user"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
@ -21,8 +21,8 @@ const (
var accountingBalanceCmd = &cobra.Command{
Use: "balance",
Short: "Get internal balance of NeoFS account",
Long: `Get internal balance of NeoFS account`,
Short: "Get internal balance of FrostFS account",
Long: `Get internal balance of FrostFS account`,
Run: func(cmd *cobra.Command, args []string) {
var idUser user.ID
@ -32,7 +32,7 @@ var accountingBalanceCmd = &cobra.Command{
if balanceOwner == "" {
user.IDFromKey(&idUser, pk.PublicKey)
} else {
common.ExitOnErr(cmd, "can't decode owner ID wallet address: %w", idUser.DecodeString(balanceOwner))
commonCmd.ExitOnErr(cmd, "can't decode owner ID wallet address: %w", idUser.DecodeString(balanceOwner))
}
cli := internalclient.GetSDKClientByFlag(cmd, pk, commonflags.RPC)
@ -42,7 +42,7 @@ var accountingBalanceCmd = &cobra.Command{
prm.SetAccount(idUser)
res, err := internalclient.BalanceOf(prm)
common.ExitOnErr(cmd, "rpc error: %w", err)
commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
// print to stdout
prettyPrintDecimal(cmd, res.Balance())
@ -58,11 +58,7 @@ func initAccountingBalanceCmd() {
ff.String(ownerFlag, "", "owner of balance account (omit to use owner from private key)")
}
func prettyPrintDecimal(cmd *cobra.Command, decimal *accounting.Decimal) {
if decimal == nil {
return
}
func prettyPrintDecimal(cmd *cobra.Command, decimal accounting.Decimal) {
if viper.GetBool(commonflags.Verbose) {
cmd.Println("value:", decimal.Value())
cmd.Println("precision:", decimal.Precision())

View file

@ -1,12 +1,12 @@
package accounting
import (
"github.com/nspcc-dev/neofs-node/cmd/neofs-cli/internal/commonflags"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
// Cmd represents the accounting command
// Cmd represents the accounting command.
var Cmd = &cobra.Command{
Use: "accounting",
Short: "Operations with accounts and balances",

View file

@ -0,0 +1,28 @@
package basic
import (
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-sdk-go/container/acl"
"github.com/spf13/cobra"
)
var printACLCmd = &cobra.Command{
Use: "print",
Short: "Pretty print basic ACL from the HEX representation",
Example: `frostfs-cli acl basic print 0x1C8C8CCC`,
Long: `Pretty print basic ACL from the HEX representation.
Few roles have exclusive default access to set of operation, even if particular bit deny it.
Container have access to the operations of the data replication mechanism:
Get, Head, Put, Search, Hash.
InnerRing members are allowed to data audit ops only:
Get, Head, Hash, Search.`,
Run: printACL,
Args: cobra.ExactArgs(1),
}
func printACL(cmd *cobra.Command, args []string) {
var bacl acl.Basic
commonCmd.ExitOnErr(cmd, "unable to parse basic acl: %w", bacl.DecodeString(args[0]))
util.PrettyPrintTableBACL(cmd, &bacl)
}

View file

@ -0,0 +1,14 @@
package basic
import (
"github.com/spf13/cobra"
)
var Cmd = &cobra.Command{
Use: "basic",
Short: "Operations with Basic Access Control Lists",
}
func init() {
Cmd.AddCommand(printACLCmd)
}

View file

@ -0,0 +1,127 @@
package extended
import (
"bytes"
"encoding/json"
"os"
"strings"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
"github.com/TrueCloudLab/frostfs-sdk-go/eacl"
"github.com/spf13/cobra"
)
var createCmd = &cobra.Command{
Use: "create",
Short: "Create extended ACL from the text representation",
Long: `Create extended ACL from the text representation.
Rule consist of these blocks: <action> <operation> [<filter1> ...] [<target1> ...]
Action is 'allow' or 'deny'.
Operation is an object service verb: 'get', 'head', 'put', 'search', 'delete', 'getrange', or 'getrangehash'.
Filter consists of <typ>:<key><match><value>
Typ is 'obj' for object applied filter or 'req' for request applied filter.
Key is a valid unicode string corresponding to object or request header key.
Well-known system object headers start with '$Object:' prefix.
User defined headers start without prefix.
Read more about filter keys at github.com/TrueCloudLab/frostfs-api/blob/master/proto-docs/acl.md#message-eaclrecordfilter
Match is '=' for matching and '!=' for non-matching filter.
Value is a valid unicode string corresponding to object or request header value.
Target is
'user' for container owner,
'system' for Storage nodes in container and Inner Ring nodes,
'others' for all other request senders,
'pubkey:<key1>,<key2>,...' for exact request sender, where <key> is a hex-encoded 33-byte public key.
When both '--rule' and '--file' arguments are used, '--rule' records will be placed higher in resulting extended ACL table.
`,
Example: `frostfs-cli acl extended create --cid EutHBsdT1YCzHxjCfQHnLPL1vFrkSyLSio4vkphfnEk -f rules.txt --out table.json
frostfs-cli acl extended create --cid EutHBsdT1YCzHxjCfQHnLPL1vFrkSyLSio4vkphfnEk -r 'allow get obj:Key=Value others' -r 'deny put others'`,
Run: createEACL,
}
func init() {
createCmd.Flags().StringArrayP("rule", "r", nil, "Extended ACL table record to apply")
createCmd.Flags().StringP("file", "f", "", "Read list of extended ACL table records from text file")
createCmd.Flags().StringP("out", "o", "", "Save JSON formatted extended ACL table in file")
createCmd.Flags().StringP(commonflags.CIDFlag, "", "", commonflags.CIDFlagUsage)
_ = cobra.MarkFlagFilename(createCmd.Flags(), "file")
_ = cobra.MarkFlagFilename(createCmd.Flags(), "out")
}
func createEACL(cmd *cobra.Command, _ []string) {
rules, _ := cmd.Flags().GetStringArray("rule")
fileArg, _ := cmd.Flags().GetString("file")
outArg, _ := cmd.Flags().GetString("out")
cidArg, _ := cmd.Flags().GetString(commonflags.CIDFlag)
var containerID cid.ID
if cidArg != "" {
if err := containerID.DecodeString(cidArg); err != nil {
cmd.PrintErrf("invalid container ID: %v\n", err)
os.Exit(1)
}
}
rulesFile, err := getRulesFromFile(fileArg)
if err != nil {
cmd.PrintErrf("can't read rules from file: %v\n", err)
os.Exit(1)
}
rules = append(rules, rulesFile...)
if len(rules) == 0 {
cmd.PrintErrln("no extended ACL rules has been provided")
os.Exit(1)
}
tb := eacl.NewTable()
commonCmd.ExitOnErr(cmd, "unable to parse provided rules: %w", util.ParseEACLRules(tb, rules))
tb.SetCID(containerID)
data, err := tb.MarshalJSON()
if err != nil {
cmd.PrintErrln(err)
os.Exit(1)
}
buf := new(bytes.Buffer)
err = json.Indent(buf, data, "", " ")
if err != nil {
cmd.PrintErrln(err)
os.Exit(1)
}
if len(outArg) == 0 {
cmd.Println(buf)
return
}
err = os.WriteFile(outArg, buf.Bytes(), 0644)
if err != nil {
cmd.PrintErrln(err)
os.Exit(1)
}
}
func getRulesFromFile(filename string) ([]string, error) {
if len(filename) == 0 {
return nil, nil
}
data, err := os.ReadFile(filename)
if err != nil {
return nil, err
}
return strings.Split(strings.TrimSpace(string(data)), "\n"), nil
}

View file

@ -1,10 +1,10 @@
package extended
import (
"strings"
"testing"
"github.com/nspcc-dev/neofs-sdk-go/eacl"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
"github.com/TrueCloudLab/frostfs-sdk-go/eacl"
"github.com/stretchr/testify/require"
)
@ -63,8 +63,7 @@ func TestParseTable(t *testing.T) {
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
ss := strings.Split(test.rule, " ")
err := parseTable(eaclTable, ss)
err := util.ParseEACLRule(eaclTable, test.rule)
ok := len(test.jsonRecord) > 0
require.Equal(t, ok, err == nil, err)
if ok {

View file

@ -0,0 +1,38 @@
package extended
import (
"os"
"strings"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-sdk-go/eacl"
"github.com/spf13/cobra"
)
var printEACLCmd = &cobra.Command{
Use: "print",
Short: "Pretty print extended ACL from the file(in text or json format) or for given container.",
Run: printEACL,
}
func init() {
flags := printEACLCmd.Flags()
flags.StringP("file", "f", "",
"Read list of extended ACL table records from text or json file")
_ = printEACLCmd.MarkFlagRequired("file")
}
func printEACL(cmd *cobra.Command, _ []string) {
file, _ := cmd.Flags().GetString("file")
eaclTable := new(eacl.Table)
data, err := os.ReadFile(file)
commonCmd.ExitOnErr(cmd, "can't read file with EACL: %w", err)
if strings.HasSuffix(file, ".json") {
commonCmd.ExitOnErr(cmd, "unable to parse json: %w", eaclTable.UnmarshalJSON(data))
} else {
rules := strings.Split(strings.TrimSpace(string(data)), "\n")
commonCmd.ExitOnErr(cmd, "can't parse file with EACL: %w", util.ParseEACLRules(eaclTable, rules))
}
util.PrettyPrintTableEACL(cmd, eaclTable)
}

View file

@ -1,6 +1,8 @@
package extended
import "github.com/spf13/cobra"
import (
"github.com/spf13/cobra"
)
var Cmd = &cobra.Command{
Use: "extended",
@ -9,4 +11,5 @@ var Cmd = &cobra.Command{
func init() {
Cmd.AddCommand(createCmd)
Cmd.AddCommand(printEACLCmd)
}

View file

@ -1,7 +1,8 @@
package acl
import (
"github.com/nspcc-dev/neofs-node/cmd/neofs-cli/modules/acl/extended"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/acl/basic"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/acl/extended"
"github.com/spf13/cobra"
)
@ -12,4 +13,5 @@ var Cmd = &cobra.Command{
func init() {
Cmd.AddCommand(extended.Cmd)
Cmd.AddCommand(basic.Cmd)
}

View file

@ -0,0 +1,126 @@
package bearer
import (
"context"
"encoding/json"
"fmt"
"os"
"time"
internalclient "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-sdk-go/bearer"
eaclSDK "github.com/TrueCloudLab/frostfs-sdk-go/eacl"
"github.com/TrueCloudLab/frostfs-sdk-go/user"
"github.com/spf13/cobra"
)
const (
eaclFlag = "eacl"
issuedAtFlag = "issued-at"
notValidBeforeFlag = "not-valid-before"
ownerFlag = "owner"
outFlag = "out"
jsonFlag = commonflags.JSON
)
var createCmd = &cobra.Command{
Use: "create",
Short: "Create bearer token",
Long: `Create bearer token.
All epoch flags can be specified relative to the current epoch with the +n syntax.
In this case --` + commonflags.RPC + ` flag should be specified and the epoch in bearer token
is set to current epoch + n.
`,
Run: createToken,
}
func init() {
createCmd.Flags().StringP(eaclFlag, "e", "", "Path to the extended ACL table")
createCmd.Flags().StringP(issuedAtFlag, "i", "", "Epoch to issue token at")
createCmd.Flags().StringP(notValidBeforeFlag, "n", "", "Not valid before epoch")
createCmd.Flags().StringP(commonflags.ExpireAt, "x", "", "The last active epoch for the token")
createCmd.Flags().StringP(ownerFlag, "o", "", "Token owner")
createCmd.Flags().String(outFlag, "", "File to write token to")
createCmd.Flags().Bool(jsonFlag, false, "Output token in JSON")
createCmd.Flags().StringP(commonflags.RPC, commonflags.RPCShorthand, commonflags.RPCDefault, commonflags.RPCUsage)
_ = cobra.MarkFlagFilename(createCmd.Flags(), eaclFlag)
_ = cobra.MarkFlagRequired(createCmd.Flags(), issuedAtFlag)
_ = cobra.MarkFlagRequired(createCmd.Flags(), notValidBeforeFlag)
_ = cobra.MarkFlagRequired(createCmd.Flags(), commonflags.ExpireAt)
_ = cobra.MarkFlagRequired(createCmd.Flags(), ownerFlag)
_ = cobra.MarkFlagRequired(createCmd.Flags(), outFlag)
}
func createToken(cmd *cobra.Command, _ []string) {
iat, iatRelative, err := common.ParseEpoch(cmd, issuedAtFlag)
commonCmd.ExitOnErr(cmd, "can't parse --"+issuedAtFlag+" flag: %w", err)
exp, expRelative, err := common.ParseEpoch(cmd, commonflags.ExpireAt)
commonCmd.ExitOnErr(cmd, "can't parse --"+commonflags.ExpireAt+" flag: %w", err)
nvb, nvbRelative, err := common.ParseEpoch(cmd, notValidBeforeFlag)
commonCmd.ExitOnErr(cmd, "can't parse --"+notValidBeforeFlag+" flag: %w", err)
if iatRelative || expRelative || nvbRelative {
ctx, cancel := context.WithTimeout(context.Background(), time.Second*30)
defer cancel()
endpoint, _ := cmd.Flags().GetString(commonflags.RPC)
currEpoch, err := internalclient.GetCurrentEpoch(ctx, cmd, endpoint)
commonCmd.ExitOnErr(cmd, "can't fetch current epoch: %w", err)
if iatRelative {
iat += currEpoch
}
if expRelative {
exp += currEpoch
}
if nvbRelative {
nvb += currEpoch
}
}
if exp < nvb {
commonCmd.ExitOnErr(cmd, "",
fmt.Errorf("expiration epoch is less than not-valid-before epoch: %d < %d", exp, nvb))
}
ownerStr, _ := cmd.Flags().GetString(ownerFlag)
var ownerID user.ID
commonCmd.ExitOnErr(cmd, "can't parse recipient: %w", ownerID.DecodeString(ownerStr))
var b bearer.Token
b.SetExp(exp)
b.SetNbf(nvb)
b.SetIat(iat)
b.ForUser(ownerID)
eaclPath, _ := cmd.Flags().GetString(eaclFlag)
if eaclPath != "" {
table := eaclSDK.NewTable()
raw, err := os.ReadFile(eaclPath)
commonCmd.ExitOnErr(cmd, "can't read extended ACL file: %w", err)
commonCmd.ExitOnErr(cmd, "can't parse extended ACL: %w", json.Unmarshal(raw, table))
b.SetEACLTable(*table)
}
var data []byte
toJSON, _ := cmd.Flags().GetBool(jsonFlag)
if toJSON {
data, err = json.Marshal(b)
commonCmd.ExitOnErr(cmd, "can't mashal token to JSON: %w", err)
} else {
data = b.Marshal()
}
out, _ := cmd.Flags().GetString(outFlag)
err = os.WriteFile(out, data, 0644)
commonCmd.ExitOnErr(cmd, "can't write token to file: %w", err)
}

View file

@ -0,0 +1,9 @@
package cmd
import (
"github.com/TrueCloudLab/frostfs-node/pkg/util/autocomplete"
)
func init() {
rootCmd.AddCommand(autocomplete.Command("frostfs-cli"))
}

Some files were not shown because too many files have changed in this diff Show more