Commit graph

292 commits

Author SHA1 Message Date
e890f1b4b1 [#1307] object: Implement Patch method
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-16 14:13:09 +00:00
a4a1c3f18b [#1307] go.mod: Bump frostfs-sdk-go/frostfs-api-go/v2 versions
* Also, resolve dependencies and conflicts for object service
  by creating stub for `Patch` method.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-16 14:13:09 +00:00
68029d756e [#1302] writecache: Allow to specify custom page size
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-09 12:12:07 +03:00
08b1f18bca [#1296] writecache: Add count limit
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-09 06:30:32 +00:00
0c2b6f3dac [#1216] ape: Make services use bearer chains fed router
* Refactor object and tree service - they should instantiate
  chain router cheking the bearer token. If there are no bearer
  token rules, then defaul chain router is used.
* Fix unit-tests.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-05 18:26:48 +00:00
78b1d9b18d [#1226] blobovniczatree: Drop leaf width limitation
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-04 12:22:06 +03:00
40c9ddb6ba [#1226] blobovniczatree: Drop init in advance option
To make blobovniczatree unlimited.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-04 12:22:06 +03:00
91bed3b0ba [#1219] Remove Container.SetEACL method
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-02 13:05:40 +00:00
56eeb630b6 [#1217] Fix grammar mistakes and misspelling
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-07-01 19:14:25 +03:00
4f7d76c9ef [#1206] audit: Drop not required events
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-06-27 10:54:31 +00:00
a0e5fc733e [#1145] node/ir: Handle double SIGHUP correctly
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-06-25 12:07:14 +03:00
df894fbac7 [#451] frostfs-node: Add cache metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-06-25 08:52:37 +00:00
81ea91de52 [#451] metrics: Move to internal
`metrics` don't look like something others want to import.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-06-25 08:52:37 +00:00
a83eeddb1d [#60] control: Add GetNetmapStatus method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-06-20 16:28:42 +03:00
ecd1ed7a5e [#1184] node: Add audit middleware for grpc services
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-06-19 16:05:53 +03:00
7b8937ec35 [#1184] config: Add audit.enabled parameter for node
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-06-19 16:05:53 +03:00
dbd3b238f7 [#1170] node: Support morph mTLS
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-06-14 09:30:46 +03:00
239323eeef [#1157] tree: Make tree service use Bearer token's APE overrides
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-06-07 12:11:11 +00:00
04a3f891fd [#1157] object: Make APE checker use Bearer-token's APE overrides
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-06-07 12:11:11 +00:00
a849236b68 [#1161] node: Remove notification functionality
It is unused and will be reworked in future.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-06-07 12:10:51 +00:00
542d3adcb2 [#1105] apemanager: Implement apemanager service
* Introduce grpc server for apemanager service and
  its implementation in `pkg/services/apemanager`.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-27 09:34:21 +00:00
2697d4d1fe [#1143] node: Fix frostfsid cache
* Fix `subjectFromSubjectExtended` converter.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-20 18:49:52 +03:00
436c9f5558 [#1129] policer: Restore EC object
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-05-17 14:36:18 +03:00
cbe9757490 [#1129] policer: Pull required EC chunks
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-05-16 16:28:49 +03:00
d45d086acd [#1129] policer: Add EC chunk replication
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-05-16 16:28:48 +03:00
b4cfc80579 [#1133] node: Improve tests for shards.default config section
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-05-15 09:48:21 +03:00
0144117cc9 [#1125] objectSvc: Add EC header APE check
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-05-08 16:25:55 +03:00
854200a874 [#1115] node: Remove unused const
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-05-07 19:11:21 +03:00
9cc51f86b7 [#1117] node: Introduce cache for frostfsid contract client
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-07 10:01:21 +00:00
52bebe9452 [#1110] node: Use single handler for new epoch event
Bootstrap logic depends on the netmap status, which in turn depends on
the node info. Updating them in a single thread makes things more
predictable.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-27 15:17:45 +03:00
4b514f5ba0 [#1110] node: Log maintenance stop only if it was enabled
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-27 15:17:45 +03:00
c8e2ca2ab4 [#1110] node: Rename handleLocalNodeInfo()
It is used in "handler" only once, what we really do is set the
variable. And we have another "local" node info in `cfgNodeInfo`, this
one is not really local (node info), more like (local node) info, so use
setContractNodeInfo to distinguish it from the local view on the node
info.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-27 15:17:45 +03:00
3b9d12b11c [#1110] node: Fix comment about nodeInfo type
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-27 15:17:45 +03:00
411a8d0245 [#1004] blobovnicza: Use TTL for blobovnicza tree cache
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-04-26 19:54:29 +03:00
112a7c690f [#1103] node: Implement Get\Head requests for EC object
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-04-24 18:15:53 +03:00
10ee865e98 [#1096] tree: Make verifyClient fill ape request with user claim tags
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-04-16 15:12:46 +03:00
c21d72ac23 [#1096] object: Make ape middleware fill request with user claim tags
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-04-16 15:12:44 +03:00
6a46c6d229 [#1090] tree: Make workaround for APE checks
* Make `verifyClient` method perform APE check if a container
  was created with zero-filled basic ACL.
* Object verbs are used in APE, until tree verbs are introduced.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-04-15 07:45:45 +00:00
57466594fb [#1024] shard: Resync metabase concurrently
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-10 10:21:10 +03:00
9aa533e59a [#1072] node, ir: Add new config option kludge_compatibility_mode
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-04-04 11:05:25 +03:00
4080b99310 [#1061] node: Set TTL for morph rule cache from morph config
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-04-02 12:40:56 +03:00
5d58b44bc8 [#1044] node: Drop unused methods from APE implementation
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-18 18:44:10 +03:00
6bf77cabd4 [#1044] ape: Add morph chain cache
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-18 18:44:10 +03:00
d1d53d2bb6 [#963] node: Add logging for waitNotaryDeposit
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-03-13 11:59:51 +03:00
7a4b6057ce Revert "[#963] node: Go on initialization even deposit notary is hung"
This reverts commit b4cb54e7ed.

Signed-off-by: aarifullin <a.arifullin@yadro.com>
2024-03-13 11:53:59 +03:00
b4cb54e7ed [#963] node: Go on initialization even deposit notary is hung
* Make makeAndWaitNotaryDeposit run asynchronously as worker
  during application boot-up.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-03-12 10:18:07 +00:00
d433b49265 [#973] node: Resolve perfsprint linter
`fmt.Errorf can be replaced with errors.New` and `fmt.Sprintf can be replaced with string addition`

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-11 17:55:50 +03:00
66a26b7775 [#973] node: Resolve revive: unused-parameter linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-11 17:11:49 +03:00
702351a5d1 [#983] blobstor: Allow to specify wait before drop time
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-05 09:23:57 +03:00
ae5bb87e70 Revert "[#866] Use TTL for blobovnicza tree cache"
This reverts commit d9cbb16bd3.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-03-01 19:29:33 +03:00
d6534fd755 [#1016] frostfs-node: Fix gopls issues
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-01 12:13:43 +03:00
962e5a9c19 [#736] logger: Add journald support
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-12 06:17:22 +00:00
db67c21d55 [#947] engine: Evacuate trees to remote nodes
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-09 11:33:15 +03:00
805862f4b7 [#956] node: Allow to reload goroutine pool sizes
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-08 15:49:43 +00:00
426cf58b98 [#956] node: Remove pool sizes from config struct
They are available through the pool methods and unused outside of the
function that sets them.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-08 15:49:43 +00:00
cbfeb72466 [#956] policer: Remove WithMaxCapacity option
We already provide the pool and this argument is used only for
preallocation. No functional changes.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-08 15:49:43 +00:00
a6c9a337cd [#965] morph: Get rid of container.List invocations
ContainersOf() is better in almost every aspect, besides creating a
session when the containers number is between 1024 and 2048 (prefetch
script does limited unwrapping). Making List() private helps to ensure
it is no longer used and can be safely removed in future.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-07 08:56:27 +00:00
5be2af881a [#934] container: Make container APE middleware read namespaces
* Those methods that can access already existing containers and thus
  can get container properties should read namespace from Zone
  property. If Zone is not set, take a namespace for root.
* Otherwise, define namespaces by owner ID via frostfs-id contract.
* Improve unit-tests, consider more cases.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-02-01 17:38:24 +00:00
e3573de6db [#930] gc: Stop internal activity by context
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-01-31 08:30:34 +00:00
1fe7736d92 [#925] morph: Introduce switch rpc guarded rpc actor
* Introduce switch rpc guarded rpc actor in morph client
  to prevent using invalidated rpc actor when RPC switch
  happens.
* Initialize NewContractStorage with SwitchRPCGuardedActor.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-01-25 13:24:13 +00:00
c8baf76fae [#872] object: Introduce APE middlewar for object service
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-01-12 18:41:35 +03:00
52ffa9f164 [#891] getSvc: Refactor Get service V2 creation
Use arguments for mandatory fields.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-12 13:35:38 +03:00
394f086fe2 [#891] getSvc: Fix get range hash implementation
Get range can perform GET request, so this request must be done
from container node to not to get access denied error.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-12 13:35:38 +03:00
d75e7e9a21 [#864] engine: Drop container size metric if container deleted
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-10 10:44:54 +03:00
764f70634d [#881] containerSvc: Add APE validation
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-27 11:05:34 +03:00
8180a0664f [#887] node: Drop badger writecache implementation
Badger implementation isn't tested and works not well,
but requires human resources to maintain.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-22 13:00:54 +03:00
825f65f79e [#873] node: Start metrics and pprof as soon as possible
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-21 15:09:51 +00:00
d69d318cb0 [#878] node: Drain internal error's channel
This fixes shutdown panic:
1. Some morph connection gets error and passes it to internalErr channel.
2. Storage node starts to shutdow and closes internalErr channel.
3. Other morph connection gets error and tries to pass it to internalErr channel.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-19 16:38:03 +00:00
d9cbb16bd3 [#866] Use TTL for blobovnicza tree cache
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-12-19 16:36:28 +00:00
7d7cf05575 [#851] ape: Initialize and use policy contract interface
* Replace inmemory policy contract mock by initialized
  policy contract interface.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-12-18 11:06:35 +00:00
61da7dca24 [#835] node: Fix appCfg concurrent access
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-14 16:38:59 +03:00
f4877e7b42 [#835] grpc: Try to reconnect if endpoint listen failed
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-14 16:38:59 +03:00
ef07c1a3c9 [#552] Add sysd notifications to storage service
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-12-13 17:51:41 +03:00
eca7ac9f0d [#552] Add sdnotify package
To avoid using third-party dependencies.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-12-13 17:49:26 +03:00
9b2dce5763 [#552] Add status notification to systemd
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-12-13 15:02:39 +03:00
05f8f49289 [#552] gofumpt changes
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-12-13 15:02:25 +03:00
d2746a7d67 [#857] Makefile: Update linter version
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-12 16:27:02 +03:00
0f45e3d344 [#804] ape: Implement boltdb storage for local overrides
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-12-07 19:08:41 +03:00
db49ad16cc [#826] blobovniczatree: Do not create DB's on init
Blobovniczas will be created on write requests.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
8bbfb2df43 [#661] blobovniczatree: Pass object size limit from config
If actual small object size value lower than default
object size limit, then unnecessary buckets created.
If actual small object size value greated than default
object size limit, then error happens.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
f1c7905263 [#661] blobovniczatree: Make Rebuild concurrent
Different DBs can be rebuild concurrently.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
da4fee2d0b [#698] blobovniczatree: Init blobovniczas concurrently
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
7f6852bbd2 [#639] node: Refactor TTL cache
Migrate from internal to external TTL implementation

Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-11-30 12:54:51 +00:00
07390ad4e3 [#715] node: Unify config parameter names
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-22 17:13:50 +03:00
5cfb758e4e [#806] morph: Remove container list cache
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-17 10:40:08 +03:00
29fe8c41f3 [#655] storage: Drop ErrorHandler
The only one usage was for logging.
Now logging performed by storage anyway.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-16 17:27:38 +03:00
4d5be5ccb5 [#811] ape: Update policy-engine module version and rebase
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-16 11:31:37 +03:00
364f835b7e [#740] logs: Add Loki
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-11-14 19:01:05 +00:00
b62008daca [#506] node: Invalidate list cache after container add/removal
`update` already has problems mentioned in its doc-comment and the code
itself is not straightforward. Invalidating cache altogether seems like
a better option because we don't construct cache output ourselves (thus, no
"impossible" results).

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-11-14 19:00:11 +00:00
f13f5d3b0f [#506] node: Use DeletionInfo() method to get deleted container owner
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-11-14 19:00:11 +00:00
2393d13e4d [#787] morph: Return VUB for IR service calls
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-13 17:13:03 +03:00
8e11ef46b8 [#770] object: Introduce ape chain checker for object svc
* Introduce Request type converted from RequestInfo type
  to implement policy-engine's Request interface
* Implement basic ape checker to check if a request is
  permitted to be performed
* Make put handlers use APE checker instead EACL

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-08 13:34:03 +00:00
5ec73fe8a0 [#770] node: Introduce ape chain source
* Provide methods to access rule chains with access
  policy engine (APE) chain source
* Initialize apeChainSource within object service
  initialization
* Share apeChainSource with control service
* Implement dummy apeChainSource instance based on
  in-memory implementation

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-08 13:34:03 +00:00
c7a7229484 [#764] metrics: Fix epoch metric
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-01 10:57:31 +00:00
a26483fc30 [#749] morph: Fix panic when closing morph client
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-11-01 10:48:10 +00:00
c80b46fad3 [#754] blobstor: Estimate compressability
Now it is possible to enable compressability estimation.
If data is likely uncompressable, it should reduce CPU time and memory.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-01 11:24:32 +03:00
79088baa06 [#772] node: Apply gofumpt
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-31 17:03:03 +03:00
b0cf100427 [#49] node: React on SIGHUP only when node in READY state
Add more info in logs when node is going to shut down,
but initialization process still in progress.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-24 15:55:29 +00:00
d5c10612f4 [#735] policer: Register metrics
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-12 09:31:36 +03:00
3a997d1207 [#680] metrics: Initialize log metrics together with services
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-11 17:08:03 +03:00