Commit graph

172 commits

Author SHA1 Message Date
5ec73fe8a0 [#770] node: Introduce ape chain source
* Provide methods to access rule chains with access
  policy engine (APE) chain source
* Initialize apeChainSource within object service
  initialization
* Share apeChainSource with control service
* Implement dummy apeChainSource instance based on
  in-memory implementation

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-08 13:34:03 +00:00
3a2c319b87 [#770] control: Generate gRPC methods to manipulate APE chains
* Define new types and gRPC methods to manipulate APE chains
  in control service.
* Stub gRPC handlers for the generated methods.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-08 13:34:03 +00:00
226e84d782 [#684] node: Add skipped objects count to evacuation result
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-07 12:17:11 +00:00
523fb3ca51 [#684] proto: Add skipped count to evacuation status response
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-07 12:17:11 +00:00
05b508f79a [#772] proto: Fix file ending
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-31 17:03:04 +03:00
79088baa06 [#772] node: Apply gofumpt
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-31 17:03:03 +03:00
b0cf100427 [#49] node: React on SIGHUP only when node in READY state
Add more info in logs when node is going to shut down,
but initialization process still in progress.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-24 15:55:29 +00:00
189dbb01be [#733] frostfs-cli: Add control ir remove-container
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-19 16:22:18 +03:00
1e8b4b8a17 [#557] services: Regenerate stable marshalers
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-02 13:37:40 +00:00
f9730f090d [#92] Refactor policer and add some unit tests
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-03 07:05:31 +00:00
8a4e250dae [#468] *: replace outdated TODO crypto-related links
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 12:13:20 +00:00
9119199f6e [#397] cli: Fix evacuation method names
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-26 11:28:30 +03:00
f7c0b50d70 [#329] cli: Add async evacuate commands
Add start, stop evacuate and evacuate status commands.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-19 08:43:52 +00:00
e4889e06ba [#329] node: Make evacuate async
Now it's possible to run evacuate shard in async.
Also only one evacuate process can be in progress.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-19 08:43:52 +00:00
100b1b5128 [#329] node: Add async evacuate proto methods
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-19 08:43:52 +00:00
90799497d3 [#114] Add remove-node IR control command
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-10 14:31:44 +00:00
eca5c210dd [#299] evacuate: Add context cancel checks
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 15:55:30 +03:00
f73ac6e02d [#290] control: Use generics for response wrappers
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 10:57:24 +03:00
ff25521204 [#270] Add IR epoch tick control call
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-04-28 07:57:00 +00:00
f07e2d4812 [#285] lint: Fix revive/unused-parameter
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-26 17:35:04 +03:00
f07d4158f5 [#249] node: Drop subnet from IR and morph
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-17 09:46:02 +03:00
8466894fdf [#250] control: remove DumpShard and RestoreShard RPC
We have `Evacuate` with a cleaner interface.
Also, remove them from CLI and engine.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-14 12:28:49 +00:00
d62c6e4ce6 [#242] node: Add tracing spans
Add tracing spans for PUT requests.
Add tracing spans for DELETE requests.
Add tracing spans for SELECT requests.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-14 10:25:53 +00:00
0920d848d0 [#135] get-object: Add tracing spans
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-12 06:52:00 +00:00
ccf8463e69 [#223] controlsvc: Drop unnecessary nolint
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 14:16:34 +03:00
0b9622c418 [#191] control: Add Doctor RPC
Doctor RPC performs complex operations on the storage engine.
Currently only duplicate removal is supported.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-07 17:25:50 +00:00
4ec69cbbf8 [#203] control: Fix test
Fix shard compare loop.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
23575e1ac0 [#210] policier: Resolve contextcheck linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:55:52 +03:00
44b86bac5a [#148] linter: Add contextcheck linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:54:41 +03:00
20de74a505 Rename package name
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-07 16:38:26 +03:00
cb5468abb8 [#66] node: Replace interface{} with any
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-02-21 16:47:07 +03:00
c761a95eef [#1] Fix project name in control service
Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-02-06 17:41:14 +03:00
3d1d2ee7b1 [#11] Regenerate proto files
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-12 08:55:47 +03:00
Pavel Karpy
923f84722a Move to frostfs-node
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-28 15:04:29 +03:00
Leonard Lyubich
60e9de8d63 [#1916] control: Check maintenance allowance on Control server
In previous implementation turning to maintenance mode using NeoFS CLI
required NeoFS API endpoint. This was not convenient from the user
perspective. It's worth to move networks settings' check to the server
side.

Add `force_maintenance` field to `SetNetmapStatusRequest.Body` message
of Control API. Add `force` flag to `neofs-cli control set-status`
command which sets corresponding field in the requests body if status is
`maintenance`. Force flag is ignored for any other status.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-24 09:20:24 +04:00
Pavel Karpy
3aa9938b8f [#1902] Update protoc to v3.21.7
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Evgenii Stratonikov
9ec01bb9c1 [#1931] control: Allow to clear errors in SetShardMode RPC
It hasn't been working since the initial implementation 7fb15fa1d0.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-20 15:51:31 +03:00
Evgenii Stratonikov
49eab6318c [#1867] control: Fix degraded-read-only mode parsing
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-12 11:20:48 +03:00
Evgenii Stratonikov
c0199dee93 [#1867] services/control: Interpret empty list of IDs as all shards
In neofs-cli the flag is still required, but `all` can be used to
process all shards.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-12 11:20:48 +03:00
Evgenii Stratonikov
19c0a74e94 [#1867] services/control: Allow to provide multiple shard IDs to some commands
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-12 11:20:48 +03:00
Leonard Lyubich
df5d7bf729 [#1680] replicator: Work with netmap.NodeInfo in TaskResult
Make `replicator.TaskResult` to accept `netmap.NodeInfo` type instead of
uint64 in order to clarify the meaning and prevent passing the random
numbers.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-11 12:54:27 +03:00
Evgenii Stratonikov
4b005d3178 [#1840] blobstor: Return info about all components
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-10 11:14:55 +03:00
Evgenii Stratonikov
51b8f26a31 [#1806] services/control: Allow to flush write-cache
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
Leonard Lyubich
485a5418d2 [#1793] node: Serve NetmapService.NetmapSnapshot RPC
There is no more need to serve the same request on Control API.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-27 09:03:17 +03:00
Evgenii Stratonikov
898689ec14 [#1731] services/replicator: Unify Task interface with other parameters
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-24 13:47:48 +03:00
Evgenii Stratonikov
4e043a801c [#1731] services/control: Replicate object over network in EvacuateShard RPC
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-24 13:47:48 +03:00
Evgenii Stratonikov
091d7d30f6 [#1731] services/control: Allow to evacuate data from shard
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-19 11:33:52 +03:00
Evgenii Stratonikov
b064fb24d8 [#1616] engine: Do not use batches in delete
Use a simple loop at the callsite. This way we remove as much as we can.
Also, `Delete` metrics is more meaningful now.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-15 10:28:48 +03:00
Evgenii Stratonikov
cda8f9df2e [#1786] services/control: Remove WithDeletedObjectHandler option
Use storage engine directly instead. It is already provided in the
options.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-15 10:28:48 +03:00
Leonard Lyubich
ae655b74f0 *: Exec make fmts
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-14 17:52:49 +04:00
Pavel Karpy
30341f2192 [#1687] *: Perform go fmt using go v1.19
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-22 18:59:57 +03:00
Evgenii Stratonikov
10c855efef [#1624] go.mod: Update dependencies
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-25 16:56:02 +03:00
Evgenii Stratonikov
4944490ffb [#1559] local_object_storage: Move shard to the DegradedReadOnly mode
`Degraded` mode can be set by the administrator if needed.
Modifying operations in this mode can lead node into an inconsistent state
because metabase checks such as lock checking are not performed.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
339864b720 [#1559] local_object_storage: Move shard.Mode to a separate package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
bfdd68dcb3 [#1333] services/control: Allow to synchronize local trees
Do not check that a node indeed belongs to the container, because the
synchronization will fail in this case anyway.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
5e843a73f9 [#1333] services/control: Return pilorama info in ListShards RPC
Do not return backend type from the service for now, because memory
backend is expected to vanish.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
0408acc40e services/control: Autogenerate stable marshalers
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-20 16:52:58 +03:00
Leonard Lyubich
21d2f8f861 [#1513] Upgrade NeoFS SDK Go with changed netmap package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-17 15:53:18 +03:00
Pavel Karpy
6e752f36dc [#1418] shard: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Leonard Lyubich
1c30414a6c [#1454] Upgrade NeoFS SDK Go module with new IDs
Core changes:
 * avoid package-colliding variable naming
 * avoid using pointers to IDs where unnecessary
 * avoid using `idSDK` import alias pattern
 * use `EncodeToString` for protocol string calculation and `String` for
  printing

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-01 17:41:45 +03:00
Leonard Lyubich
aeb9884218 [#1389] crypto: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-19 10:12:14 +03:00
Evgenii Stratonikov
0f376a5d83 [#1388] services/control: Extend tests for ShardInfo marshaling
It is nice to have different paths for different components and also
check that the information returned is different for different shards.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-17 13:48:07 +03:00
Elizaveta Chichindaeva
cc7a723d77 [#1320] English Check
Signed-off-by: Elizaveta Chichindaeva <elizaveta@nspcc.ru>
2022-05-11 10:40:02 +03:00
Evgenii Stratonikov
6472a170eb [#1143] shard: Introduce explicit Degraded mode
`Degraded` mode is set automatically after error counter is over the
threshold. `ReadOnly` mode can still be set by an administrator.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
Alex Vanin
7f3195b197 [#1238] Adopt neofs-cli for non pointer slices in SDK
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-16 15:29:31 +03:00
Pavel Karpy
cf17686321 [#1111] control: Rename aliaces
`objectSDKAddress` -> `addressSDK`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-05 10:06:14 +03:00
Pavel Karpy
1667ec9e6d [#1131] *: Adopt SDK changes
`object.Address` has been moved to `object/address`
`object.ID` has been moved to `object/id`

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-02-08 09:45:38 +03:00
Evgenii Stratonikov
ed50cf6207 [#1118] services/control: return error counter in ListShards
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-03 15:14:27 +03:00
Evgenii Stratonikov
7fb15fa1d0 [#1118] services/control: allow to reset error counter in SetShardMode
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-03 15:14:27 +03:00
Evgenii Stratonikov
18cfd8b042 [#1086] services/control: implement RestoreShard RPC
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:14 +03:00
Evgenii Stratonikov
2ec4a3c897 [#1086] neofs-cli: add command for shard dump
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:14 +03:00
Evgenii Stratonikov
0e60b1d6c9 [#1086] services/control: implement DumpShard RPC
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:14 +03:00
Evgenii Stratonikov
aa53418119 [#1086] services/control: fix comments
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:14 +03:00
Pavel Karpy
97fd330fae [#1059] control: Fix ListShards wrapper name
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
77808c7b41 [#1059] control: Provide shard's real mode
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
88bb8e2df9 [#1059] control: Add SetShardMode implementation to ctrl svc
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
fa78a37787 [#1059] control: Add unit test for SetShardMode
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
29c69f37eb [#1059] control: Add SetShardMode RPC wrappers
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
25665cb0fd [#1059] control: Add marshalling and setters to SetShardMode
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
4533784e11 [#1059] control: Recompile proto files
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
41dd385b7b [#1059] control: Add SetShardMode rpc to proto files
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
876b0c53de [#1048] control: Rename Blobstore to Blobstor
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
f5a9735e1c [#1048] cli: Add shard list operation
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
0e5410603e [#1048] control: Add ListShards implementation to ctrl svc
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
4e989e7133 [#1048] control: Add unit test for ListShards
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
f88a12eaa7 [#1048] control: Add ListShards RPC wrappers
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
a6b3e16975 [#1048] control: Add marshalling and setters to ListShards
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
a42b3d37f6 [#1048] control: Recompile proto files
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
20f11c88ca [#1048] control: Add ListShards rpc to proto files
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Leonard Lyubich
e8f8e58e90 [#979] adm: Add subnet-related commands
Add `subnet` command which contains all subnet-related commands. Add
sub-commands:
  * `create` for creation;
  * `remove` for removal;
  * `get` for reading;
  * `admin` for admin management;
  * `client` for client management.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Leonard Lyubich
7f5fb130c0 [#961] *: Support NeoFS API status returns
Upgrade NeoFS API Go library to version with status returns. Make all API
clients to pull out and return errors from failed statuses. Make signature
service to respond with status if client version supports it.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-24 09:26:40 +03:00
Evgenii Stratonikov
95893927aa *: replace neofs-api-go with neofs-sdk-go
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-12 17:29:09 +03:00
Leonard Lyubich
8f8b638b95 [#922] control: Add MAINTENANCE netmap status
Add `MAINTENANCE` value to `NetmapStatus` enum in Control API. The status is
going to be used to toggle maintenance mode of the storage node.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
be05bed0b3 [#860] Re-compile protobuf files after make protoc target update
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-29 17:51:33 +03:00
Leonard Lyubich
8eadf18e01 [#607] services/control: Fix incorrect allocation in NetmapSnapshot
Fix make with length + subsequent append.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
1e52e86bbc [#607] node/control: Make group address in NodeInfo message
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
455fd952dd [#414] ir: Serve ControlService
Serve `ControlService` instance on configured endpoint (do not serve if not
specified). Read allowed keys from config.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-15 15:21:52 +03:00
Leonard Lyubich
dcfe9a6504 [#414] ir/control: Implement service server
Implement `ControlServiceServer` on `Server` type. The `Server` requires all
requests to be signed with keys from the so-called whitelist. To obtain
health status, it uses the abstraction in the form of `HealthChecker`
interface.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-15 15:21:52 +03:00
Leonard Lyubich
93803b1a90 [#414] ir: Define Control service
Define `ControlService` for IR similar to the one from storage node. Add
`HealthStatus` RPC which returns health status of the IR application.
Implement getters, setters and methods to sign/verify the messages.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-15 15:21:52 +03:00
Alex Vanin
89b147ebe7 [#521] Recompile protobuf files after transition from pkg/errors
Autogenerated files were accidentally modified. This commit returns
them back to the original.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-05-19 19:09:35 +03:00