Commit graph

1598 commits

Author SHA1 Message Date
be8f499b91 [#717] Fix mixing CID and container ID
All container ID outputs that may be used in scripts
were replaced with CID.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-12-19 12:36:29 +03:00
7d7cf05575 [#851] ape: Initialize and use policy contract interface
* Replace inmemory policy contract mock by initialized
  policy contract interface.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-12-18 11:06:35 +00:00
61da7dca24 [#835] node: Fix appCfg concurrent access
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-14 16:38:59 +03:00
f4877e7b42 [#835] grpc: Try to reconnect if endpoint listen failed
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-14 16:38:59 +03:00
4a64b07703 [#869] cli: Pass only CID in requests for control API
* Fix add-rule, list-rules, remove-rule, get-rule commands:
  do not convert container ID to native-schema resource format
  and pass it to control API.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-12-14 11:01:20 +00:00
ef07c1a3c9 [#552] Add sysd notifications to storage service
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-12-13 17:51:41 +03:00
eca7ac9f0d [#552] Add sdnotify package
To avoid using third-party dependencies.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-12-13 17:49:26 +03:00
9b2dce5763 [#552] Add status notification to systemd
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-12-13 15:02:39 +03:00
05f8f49289 [#552] gofumpt changes
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-12-13 15:02:25 +03:00
11add38e87 [#857] golangci: Add protogetter linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-12 16:27:02 +03:00
d2746a7d67 [#857] Makefile: Update linter version
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-12 16:27:02 +03:00
0f45e3d344 [#804] ape: Implement boltdb storage for local overrides
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-12-07 19:08:41 +03:00
39060382a1 [#842] control: Recieve target in gRPC methods for APE managing
* Introduce Target type and pass it to all gRPC methods
  for APE chain managing instead CID.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-12-07 14:21:55 +00:00
db49ad16cc [#826] blobovniczatree: Do not create DB's on init
Blobovniczas will be created on write requests.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
8bbfb2df43 [#661] blobovniczatree: Pass object size limit from config
If actual small object size value lower than default
object size limit, then unnecessary buckets created.
If actual small object size value greated than default
object size limit, then error happens.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
f1c7905263 [#661] blobovniczatree: Make Rebuild concurrent
Different DBs can be rebuild concurrently.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
da4fee2d0b [#698] blobovniczatree: Init blobovniczas concurrently
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
484eb59893 [#661] blobovniczatree: Use .db extension for db files
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:32 +03:00
b892feeaf6 [#845] adm: Relax notary-enabled check
Starting from v0.104.0 `NativeActivations` config field is no longer
present and Notary activation height is always 0.

https://github.com/nspcc-dev/neo-go/pull/3212/
TrueCloudLab/frostfs-dev-env#59

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-12-06 11:08:04 +00:00
f1db468d48 [#840] adm: Update FrostFS ID deploy arguments
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-12-04 17:39:41 +03:00
a982c3df18 [#824] cli: Support passing chain ID in add-rule command
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-30 13:13:46 +00:00
7f6852bbd2 [#639] node: Refactor TTL cache
Migrate from internal to external TTL implementation

Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-11-30 12:54:51 +00:00
306f12e6c5 [#828] adm: Fix policy contract deploy
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-29 06:23:56 +00:00
5521737f0b [#808] cli: Use EnableTraverseRunHooks in cobra
Adopt EnableTraverseRunHooks to get rid of tracing boilerplate in
multiple commands. Now adding `--trace` flag is sufficient for a command
to support tracing. Finally, it looks how it _should_.

Refs TrueCloudLab/frostfs-node#406

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-11-27 09:58:19 +00:00
e81a58b8da [#808] cli: Use MarkFlagsOneRequired after cobra update
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-11-27 09:58:19 +00:00
c516c7c5f4 [#821] node: Pass user.ID by value
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-23 10:21:07 +03:00
07390ad4e3 [#715] node: Unify config parameter names
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-22 17:13:50 +03:00
02454df14a [#809] client: Refactor PrmInit, PrmDial usage
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-17 13:37:03 +00:00
5cfb758e4e [#806] morph: Remove container list cache
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-17 10:40:08 +03:00
29fe8c41f3 [#655] storage: Drop ErrorHandler
The only one usage was for logging.
Now logging performed by storage anyway.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-16 17:27:38 +03:00
4d5be5ccb5 [#811] ape: Update policy-engine module version and rebase
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-16 11:31:37 +03:00
fd9128d051 [#800] node: eACL -> APE converter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-15 11:55:55 +03:00
364f835b7e [#740] logs: Add Loki
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-11-14 19:01:05 +00:00
c1ec6e33b4 [#793] adm: Always use committee as FrostFS ID owner
Committee should be able to authorize everything, there are no other
usecases for the frostfs-adm currently. Also, it somewhat eases
configuration, because committee hash depends on the protocol
configuration.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-11-14 19:00:32 +00:00
f871f5cc6c [#793] adm: Support new FrostFS ID contract
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-11-14 19:00:32 +00:00
b62008daca [#506] node: Invalidate list cache after container add/removal
`update` already has problems mentioned in its doc-comment and the code
itself is not straightforward. Invalidating cache altogether seems like
a better option because we don't construct cache output ourselves (thus, no
"impossible" results).

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-11-14 19:00:11 +00:00
f13f5d3b0f [#506] node: Use DeletionInfo() method to get deleted container owner
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-11-14 19:00:11 +00:00
2393d13e4d [#787] morph: Return VUB for IR service calls
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-13 17:13:03 +03:00
5466e88444 [#787] cli: Add vub for control ir commands
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-13 17:13:03 +03:00
78cfb6aea8 [#796] cli: Fix object nodes command
Tombstone objects must be present on all container nodes.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-09 13:41:36 +03:00
3ed3e2715b [#xx] adm: Drop notaryDisabled deploy parameter
Refs TrueCloudLab/frostfs-contract#50

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-11-08 13:38:04 +00:00
66848d3288 [#770] cli: Add methods to work with APE rules via control svc
* Add methods to frostfs-cli
* Implement rpc in control service

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-08 13:34:03 +00:00
8e11ef46b8 [#770] object: Introduce ape chain checker for object svc
* Introduce Request type converted from RequestInfo type
  to implement policy-engine's Request interface
* Implement basic ape checker to check if a request is
  permitted to be performed
* Make put handlers use APE checker instead EACL

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-08 13:34:03 +00:00
5ec73fe8a0 [#770] node: Introduce ape chain source
* Provide methods to access rule chains with access
  policy engine (APE) chain source
* Initialize apeChainSource within object service
  initialization
* Share apeChainSource with control service
* Implement dummy apeChainSource instance based on
  in-memory implementation

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-08 13:34:03 +00:00
1e21733ea5 [#684] cli: Add skipped count to evacuation status output
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-07 12:17:11 +00:00
74c91eeef5 [#777] client: Refactor PrmContainerList, PrmObjectSearch usage
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-06 06:50:11 +00:00
cae50ecb21 [#716] adm: Add dump policy
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-11-06 06:43:56 +00:00
1ab567870a [#779] adm: Support deploying policy contract
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-11-02 09:23:41 +00:00
0b0e5dab24 [#756] adm: Add polling interval increase
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-01 14:24:28 +03:00
c7a7229484 [#764] metrics: Fix epoch metric
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-01 10:57:31 +00:00
a26483fc30 [#749] morph: Fix panic when closing morph client
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-11-01 10:48:10 +00:00
c80b46fad3 [#754] blobstor: Estimate compressability
Now it is possible to enable compressability estimation.
If data is likely uncompressable, it should reduce CPU time and memory.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-01 11:24:32 +03:00
79088baa06 [#772] node: Apply gofumpt
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-31 17:03:03 +03:00
261d281154 [#762] go.mod: Update sdk-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-31 11:22:29 +00:00
4239f1e817 [#750] adm: Drop deprecated rpcclient.TransferTarget
We do not use `nep17` wrapper, because transfers of different tokens are
possible in a single transaction.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-25 07:57:05 +00:00
7f35f2fb1d [#750] adm: Drop deprecated CreateTxFromScript()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-25 07:57:05 +00:00
b0d303f3ed [#750] adm: Drop unused methods from Client
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-25 07:57:05 +00:00
a788c24e6d [#750] adm: Drop deprecated AddNetworkFee()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-25 07:57:05 +00:00
4368243bed [#750] adm: Drop deprecated NEP17BalanceOf()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-25 07:57:05 +00:00
00a0045d9a [#750] adm: Drop deprecated GetContract*()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-25 07:57:05 +00:00
7f8ccc105b [#750] adm: Drop deprecated GetNetwork()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-25 07:57:05 +00:00
efb37b0e65 [#750] adm: Fix invalid tests
Introduced in a9d04ba86f.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-25 07:57:05 +00:00
b0cf100427 [#49] node: React on SIGHUP only when node in READY state
Add more info in logs when node is going to shut down,
but initialization process still in progress.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-24 15:55:29 +00:00
58b6224dd8 [#747] client: Refactor PrmObjectPutInit usage
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-10-20 11:55:40 +00:00
189dbb01be [#733] frostfs-cli: Add control ir remove-container
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-19 16:22:18 +03:00
d5c10612f4 [#735] policer: Register metrics
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-12 09:31:36 +03:00
3a997d1207 [#680] metrics: Initialize log metrics together with services
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-11 17:08:03 +03:00
994f48f8bb [#680] metrics: Export log and morph with script
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-11 17:08:03 +03:00
4caa934eea [#729] containersvc: Remove load announcement
IR code was removed in 8879c6ea.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-09 19:01:13 +00:00
bd5bf8b1a9 [#721] netmap: Drop already bootstraped check
Because of this check, under certain conditions,
the node could be removed from the network map,
although the node was functioning normally.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-05 11:47:06 +03:00
f3278d76a9 [#721] netmap: Send bootstrap at each epoch tick
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-05 11:46:56 +03:00
627b302745 [#709] node: Put in log info about listening endpoints
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-02 13:31:10 +00:00
b8c3c2486d [#333] Sort containers by ID
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-09-25 11:30:36 +03:00
c14c9a023c [#333] Sort objects by ID in SearchObjects
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-09-25 11:30:36 +03:00
d9b93b12c1 [#333] Sort shards by shard_ID in cli output
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-09-25 10:04:29 +03:00
58239d1b2c [#683] cli: Add context to policy parsing errors
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-12 09:47:21 +00:00
3c76884182 [#682] cli: Unify array of ranges type
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-12 10:42:34 +03:00
f435ab1b26 [#682] go.mod: Update sdk-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-12 10:42:34 +03:00
aa9f8dce3d [#677] client: Refactor PrmAnnounceSpace/EndpointInfo/NetworkInfo usage
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-09-08 09:42:28 +03:00
8a81af5a3b [#653] Add context parameter to Open functions
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-09-07 18:03:29 +03:00
d9de9e2bbb [#675] client: Refactor PrmObjectDelete usage
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-09-06 08:05:47 +00:00
806cc13d9f [#658] client: Refactor PrmObjectGet/Head/Range usage
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-30 17:13:23 +00:00
7654847f79 [#659] adm: Remove nspcc.ru from the default email
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-30 08:29:26 +00:00
ae81d6660a [#529] objectcore: Fix object content validation
There are old objects where the owner of the object
may not match the one who issued the token.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-29 10:33:06 +03:00
4ea0df77d0 [#574] policer: Check if the container was really removed
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-28 14:21:38 +00:00
554ff2c06b [#574] core: Extend Source interface with DeletionInfo method
* Introduce common method EverExisted
* Define DeletionInfo for struct that must implement Source
* Refactor tree srv

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-28 14:21:38 +00:00
238b8f10a0 [#630] cli: Fix SDK SetEACLPrm usage for PrmContainerSetEACL
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-22 14:25:39 +00:00
dc3bc08c07 [#631] lens: Fix db type flag name
Typo from 1a0cb0f34a.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-21 17:18:05 +00:00
23be3eb627 [#574] tree: Check if container is really removed
* Use DeletionInfo method from morph client to check if
  the container has been really removed from neo-go

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-08-21 12:50:20 +03:00
84ea075587 [#625] cli: Fix SDK EACLPrm usage for PrmContainerEACL
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-21 10:36:47 +03:00
354a92ea2c [#602] blobovnicza: Add leaf width implementation
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-21 10:27:32 +03:00
d3904ec599 [#602] config: Add blobovnicza leaf width parameter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-21 10:27:32 +03:00
127c676786 [#607] *: Use keys.PublicKeys.Copy() where possible
Semantic patch:
```
@@
var dst identifier
var src identifier
var keys identifier
@@
 import keys "github.com/nspcc-dev/neo-go/pkg/crypto/keys"

-dst := make(keys.PublicKeys, len(src))
-copy(dst, src)
+dst := src.Copy()
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-17 16:50:02 +00:00
e604a3d749 [#607] *: Use zap.Stringer() where possible
Semantic patch:
```
@@
var f expression
var t expression
var a expression
@@
 f(
    ...,
-    zap.String(t, a.String()),
+    zap.Stringer(t, a),
    ...,
)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-17 16:50:02 +00:00
58c8722c81 [#585] fstree: Add optional file counter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-16 14:47:44 +03:00
21800e9fcc [#162] core: Move literals to constants
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-08-11 15:48:42 +03:00
abdb0910cc [#600] adm/tests: Add missing WaitGroup.Add()
```
panic: sync: negative WaitGroup counter
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-11 09:50:07 +00:00
b44a8dd46c [#597] *: Fix linter warnings
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-11 09:55:51 +03:00
20af34ecdb [#591] cli: fix SDK PrmContainerDelete usage for DeleteContainerPrm
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-10 16:40:35 +00:00
dd988a5912 [#592] adm: Allow to tick epoch concurrently
Epoch can be ticked by IR in the background. Because the only usecase
for this is to apply some changes, the right behaviour is ignoring an
error, not retrying.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 16:40:00 +00:00