Commit graph

145 commits

Author SHA1 Message Date
55b82e744b [#529] objectcore: Use common sender classifier
All checks were successful
DCO action / DCO (pull_request) Successful in 2m19s
Vulncheck / Vulncheck (pull_request) Successful in 3m5s
Build / Build Components (1.21) (pull_request) Successful in 4m8s
Build / Build Components (1.20) (pull_request) Successful in 4m24s
Tests and linters / Tests (1.20) (pull_request) Successful in 4m57s
Tests and linters / Staticcheck (pull_request) Successful in 4m43s
Tests and linters / Tests (1.21) (pull_request) Successful in 5m2s
Tests and linters / Lint (pull_request) Successful in 5m21s
Tests and linters / Tests with -race (pull_request) Successful in 6m17s
Use common sender classifier for ACL service and format validator.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-29 10:33:06 +03:00
ae81d6660a [#529] objectcore: Fix object content validation
There are old objects where the owner of the object
may not match the one who issued the token.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-29 10:33:06 +03:00
c9e3c9956e [#643] objsvc/put: Unify extraBroadcastEnabled usage
All checks were successful
DCO action / DCO (pull_request) Successful in 1m26s
Vulncheck / Vulncheck (pull_request) Successful in 2m44s
Build / Build Components (1.20) (pull_request) Successful in 3m26s
Tests and linters / Staticcheck (pull_request) Successful in 3m51s
Tests and linters / Tests (1.20) (pull_request) Successful in 5m4s
Tests and linters / Tests (1.21) (pull_request) Successful in 5m46s
Tests and linters / Tests with -race (pull_request) Successful in 6m23s
Build / Build Components (1.21) (pull_request) Successful in 13m1s
Tests and linters / Lint (pull_request) Successful in 19m46s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-24 11:03:17 +03:00
facd3b2c4b [#643] objsvc/put: Unify placement iterators
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-24 11:03:17 +03:00
3fcf56f2fb [#643] objsvc/put: Copy config to distributedTarget
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-24 11:03:17 +03:00
21800e9fcc [#162] core: Move literals to constants
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-08-11 15:48:42 +03:00
8d589314b5 [#560] node: Fix Put in multi REP with intersecting sets of nodes
Once the node was processed it skipped, at the step of forming
result in case when all nodes skipped, because processed for
previous REP, service mark the whole request as incomplete.

Example of policies which are unblocked:
- REP 1 REP 1 CBF 1
- REP 4 IN X REP 4 IN Y
  CBF 4
  SELECT 2 FROM FX AS X SELECT 2 FROM FY AS Y
  FILTER Country EQ Russia OR Country EQ Sweden OR Country EQ Finland AS FY
  FILTER Price GE 0 AS FX

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-08-08 10:22:53 +00:00
05ac9e3637 [#547] objectsvc: Work with traversal struct from a single thread
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-31 15:35:15 +00:00
ea32913430 [#543] putsvc: Fix PutSingle implementation
Add Lock and Delete handlers to local PutSingle.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-28 12:09:41 +00:00
5ff82ff04f [#6] services/object: Simplify local/remote targets
Some checks failed
ci/woodpecker/pr/pre-commit Pipeline was successful
Tests and linters / Tests (1.19) (pull_request) Successful in 14m18s
Vulncheck / Vulncheck (pull_request) Failing after 2m23s
Tests and linters / Staticcheck (pull_request) Successful in 4m8s
Build / Build Components (1.19) (pull_request) Successful in 3m26s
Build / Build Components (1.20) (pull_request) Successful in 2m56s
Tests and linters / Tests with -race (pull_request) Successful in 21m39s
Tests and linters / Tests (1.20) (pull_request) Successful in 22m36s
Tests and linters / Lint (pull_request) Successful in 1m39s
ci/woodpecker/push/pre-commit Pipeline was successful
We do not use the return result from Close() and we always execute both
methods in succession. It makes sense to unite them.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-21 18:39:12 +03:00
8966dd8e35 [#463] putsvc: Use PutSingle RPC for remote target
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-18 10:52:12 +00:00
ec9b738465 [#294] putsvcv2: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
800a685e84 [#294] putsvc: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
7da284f3e8 [#509] go.mod: Update sdk-go
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Tests and linters / Lint (pull_request) Successful in 2m15s
Tests and linters / Tests (1.19) (pull_request) Successful in 5m7s
Tests and linters / Tests with -race (pull_request) Successful in 4m50s
Tests and linters / Staticcheck (pull_request) Successful in 4m10s
Build / Build Components (1.19) (pull_request) Successful in 3m47s
Build / Build Components (1.20) (pull_request) Successful in 3m30s
Tests and linters / Tests (1.20) (pull_request) Successful in 20m39s
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-11 17:38:00 +03:00
80481c015c [#509] putsvc: Omit AccessIdentifiers from iteratePlacement()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-11 17:38:00 +03:00
a65e26878b [#486] put service: Fix error typo
Some checks failed
ci/woodpecker/pr/pre-commit Pipeline failed
Build / Build Components (1.19) (pull_request) Successful in 4m12s
Tests and linters / Tests (1.19) (pull_request) Successful in 2m52s
Tests and linters / Tests (1.20) (pull_request) Successful in 3m49s
Tests and linters / Staticcheck (pull_request) Successful in 3m13s
Tests and linters / Lint (pull_request) Successful in 10m50s
Build / Build Components (1.20) (pull_request) Successful in 11m28s
Tests and linters / Tests with -race (pull_request) Successful in 7m54s
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-10 15:49:21 +03:00
fcbf90d31b [#486] node: Add PutSingle implemetation
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-10 15:49:21 +03:00
7b76527759 [#486] node: Add PutSingle wrappers
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-10 15:49:21 +03:00
033eaf77e1 [#496] node: Fix linter importas
All checks were successful
Build / Build Components (1.20) (pull_request) Successful in 3m52s
Build / Build Components (1.19) (pull_request) Successful in 4m1s
ci/woodpecker/pr/pre-commit Pipeline was successful
Tests and linters / Tests with -race (pull_request) Successful in 5m36s
Tests and linters / Tests (1.20) (pull_request) Successful in 5m55s
Tests and linters / Lint (pull_request) Successful in 14m40s
Tests and linters / Tests (1.19) (pull_request) Successful in 15m29s
ci/woodpecker/push/pre-commit Pipeline was successful
Standardize the alias of the
import frostfs-sdk-go/object as objectSDK.

Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-07-06 15:36:41 +03:00
785d81a68a [#460] services/object: Reduce distibutedTarget memory footprint
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-21 17:07:50 +03:00
0400153b7d [#424] object: Do not store large slices in pool
Dynamically growing an unbounded buffers can cause a large
amount of memory to be pinned and never be freed.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 07:19:55 +00:00
74578052f9 [#412] node: Replace tracing package
Use observability module.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-01 13:23:11 +00:00
b2ffd7df53 [#291] object: Use PayloadSizeLimiter from SDK
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-05 19:07:06 +03:00
35ea207df6 [#291] object: Split validating target in two
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-05 19:07:06 +03:00
Pavel Karpy
ee58b390bb [#221] node: Allow using vector copies_number
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Also, take into account that value in general (it was not used before at
all).

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-05-05 16:07:13 +00:00
3d43b0f7f9 [#265] node: Fix after SDK & API-Go version up
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-18 12:09:19 +03:00
d62c6e4ce6 [#242] node: Add tracing spans
Add tracing spans for PUT requests.
Add tracing spans for DELETE requests.
Add tracing spans for SELECT requests.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-14 10:25:53 +00:00
200fc8b882 [#242] put: Pass context to relay function
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-14 10:25:53 +00:00
0e31c12e63 [#240] logs: Move log messages to constants
Drop duplicate entities.
Format entities.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-14 05:06:09 +00:00
2ed9fd3f94 [#223] objectsvc: Refactor request parameters
Resolve containedctx linter for commonPrm.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 14:16:34 +03:00
23575e1ac0 [#210] policier: Resolve contextcheck linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:55:52 +03:00
ece6c820e7 [#199] putsvc: Refactor streamer initialization
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Resolve funlen linter for initTarget method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-03 15:58:11 +00:00
27bdddc48f [#199] putsvc: Refactor put object
Resolve containedctx linter for streamer and remote target

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-03 15:58:11 +00:00
cecea8053a [#199] putsvc: Refactor streamer pool
Resolve staticcheck linter for putBytesPool

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-03 15:58:11 +00:00
14d894178e [#199] putsvc: Refactor placement iterator
Resolve funlen linter for iteratePlacement method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-03 15:58:11 +00:00
97c36ed3ec [#148] linter: Add funlen linter
Long functions are hard to understand and source of errors

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:54:41 +03:00
cc8ff015b4 [#148] linter: Add containedctx linter
Context has to be passed as an argument: https://pkg.go.dev/context

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:52:39 +03:00
Pavel Karpy
64bde68fb9 [#67] node: Accept expired locked objects
Allow replication of any (expired too) locked object. Information about
object locking is considered to be presented on the _container nodes_.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-16 16:22:19 +03:00
22be532cbd object/put: Persist session token till the end of a session
Previously a token could've expired in the middle of an object.PUT
stream, leading to upload being interrupted. This is bad, because user
doesn't always now what is the right values for the session token
lifetime. More than that, setting it to a very high value will
eventually blow up the session token database.

In this commit we read the session token once and reuse it for the whole
stream duration.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-16 06:45:50 +00:00
20de74a505 Rename package name
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-07 16:38:26 +03:00
73bb590cb1 [#64] node: Use pool_size_local and separate pool for local puts
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-02-22 13:43:19 +03:00
cb5468abb8 [#66] node: Replace interface{} with any
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-02-21 16:47:07 +03:00
Pavel Karpy
901d62567d [#57] node: Broadcast link objects
It boosts object assembling by an _average_ container node.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-17 11:58:27 +03:00
6f5edac730 [#2164] network/cache: Do not reconnect to failed clients immediately
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
04b5ec759b [#2139] object/put: Use sync.Pool for temporary payloads
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
923f84722a Move to frostfs-node
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-28 15:04:29 +03:00
Pavel Karpy
51963abce7 [#1972] node: Fix errors comments in the Put service
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 16:58:52 +03:00
Evgenii Stratonikov
2522d924b9 [#2037] services/object: Fix concurrent map writes in traverser
```
fatal error: concurrent map writes

goroutine 4337 [running]:
github.com/nspcc-dev/neofs-node/pkg/services/object/put.(*traversal).submitProcessed(...)
        github.com/nspcc-dev/neofs-node/pkg/services/object/put/distributed.go:78
github.com/nspcc-dev/neofs-node/pkg/services/object/put.(*distributedTarget).iteratePlacement.func1()
        github.com/nspcc-dev/neofs-node/pkg/services/object/put/distributed.go:198 +0x265
github.com/panjf2000/ants/v2.(*goWorker).run.func1()
        github.com/panjf2000/ants/v2@v2.4.0/worker.go:68 +0x97
created by github.com/panjf2000/ants/v2.(*goWorker).run
        github.com/panjf2000/ants/v2@v2.4.0/worker.go:48 +0x65
```

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
aab398f4f5 [#1972] node: Do not save objects if node not in a container
Do not use node's local storage if it is clear that an object will be
removed anyway as a redundant. It requires moving the changing local storage
logic from the validation step to the local target implementation.
It allows performing any relations checks (e.g. object locking) only if a
node is considered as a valid container member and is expected to store
(stored previously) all the helper objects (e.g. `LOCK`, `TOMBSTONE`, etc).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
d772e35aba [#1910] .golangci.yml: Add godot linker
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-18 15:08:26 +03:00