TrueCloudLab
/
frostfs-node
18
1
Fork 20
Code Issues 88 Pull Requests 8 Actions Releases Activity

Upd/neo-go subscriptions #73

Merged
carpawell merged 2 commits from carpawell/upd/neo-go-subs into master 2023-03-24 09:42:31 +00:00
Conversation 1 Commits 2 Files Changed 4
carpawell commented 2023-02-22 13:08:38 +00:00 (Migrated from github.com)

Only adopts new API. More adds that removes; could look strange a little. But suggest separating updates and refactors.

Only adopts new API. More adds that removes; could look strange a little. But suggest separating updates and refactors.
ale64bit (Migrated from github.com) reviewed 2023-02-22 13:08:38 +00:00
dstepanov-yadro (Migrated from github.com) reviewed 2023-02-22 13:08:38 +00:00
fyrchik (Migrated from github.com) reviewed 2023-02-22 16:37:58 +00:00
fyrchik (Migrated from github.com) left a comment

Correct me if I am wrong, in this PR our main goal is to remove some of the deprecated messages?

Correct me if I am wrong, in this PR our main goal is to remove some of the `deprecated` messages?
pkg/morph/client/multi.go Outdated
fyrchik (Migrated from github.com) commented 2023-02-22 16:29:43 +00:00

?

?
pkg/morph/client/multi.go Outdated
fyrchik (Migrated from github.com) commented 2023-02-22 16:29:56 +00:00

??

??
pkg/morph/client/notifications.go Outdated
fyrchik (Migrated from github.com) commented 2023-02-22 16:31:11 +00:00

If it is false, do can we take a mutex too?
It could look more readable without if.

If it is false, do can we take a mutex too? It could look more readable without `if`.
carpawell (Migrated from github.com) reviewed 2023-02-27 12:33:13 +00:00
carpawell (Migrated from github.com) left a comment

Correct me if I am wrong, in this PR our main goal is to remove some of the deprecated messages?

@fyrchik, yes, all the "deprecated" msgs about the notification processes are fixed (but that is not a refactor and that is not a PR that fixes all the linter messages)

> Correct me if I am wrong, in this PR our main goal is to remove some of the deprecated messages? @fyrchik, yes, all the "deprecated" msgs about the notification processes are fixed (but that is not a refactor and that is not a PR that fixes _all_ the linter messages)
carpawell (Migrated from github.com) reviewed 2023-02-27 12:33:45 +00:00
pkg/morph/client/notifications.go Outdated
carpawell (Migrated from github.com) commented 2023-02-27 12:33:45 +00:00

no, we would have a dead-lock then: background flag is false only if we have already lost connection and try to establish a new one, so switchLock has been already taken

no, we would have a dead-lock then: `background` flag is `false` only if we have already lost connection and try to establish a new one, so `switchLock` has been already taken
carpawell (Migrated from github.com) reviewed 2023-02-27 12:34:52 +00:00
pkg/morph/client/multi.go Outdated
carpawell (Migrated from github.com) commented 2023-02-27 12:34:52 +00:00

could not find unlocking line while was preparing that PR so decide to chose a better (IMO) spacing

could not find unlocking line while was preparing that PR so decide to chose a better (IMO) spacing
carpawell (Migrated from github.com) reviewed 2023-02-27 12:37:32 +00:00
pkg/morph/client/multi.go Outdated
carpawell (Migrated from github.com) commented 2023-02-27 12:37:32 +00:00

a separate commit/keep as it was?

a separate commit/keep as it was?
fyrchik (Migrated from github.com) approved these changes 2023-02-28 10:44:04 +00:00
pkg/morph/client/client.go Outdated
fyrchik (Migrated from github.com) commented 2023-02-28 10:43:15 +00:00

Can we embed subsInfo here? It allows us to get rid of updateSubs and possibly prevents us from forgetting something to copy when we add it.

Can we embed `subsInfo` here? It allows us to get rid of `updateSubs` and possibly prevents us from forgetting something to copy when we add it.
pkg/morph/client/multi.go Outdated
fyrchik (Migrated from github.com) commented 2023-02-28 10:41:17 +00:00

I would keep if it then linter doesn't complain.

I would keep if it then linter doesn't complain.
acid-ant (Migrated from github.com) approved these changes 2023-02-28 10:57:24 +00:00
carpawell added 72 commits 2023-03-23 21:25:44 +00:00
a1ab25b33e [#72] .github: Fix CODEOWNERS 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
c3a7039801 [TrueCloudLab/hrw#2] node: Optimize shard hash 
Compute shard hash only once

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
6925fb4c59 [TrueCloudLab/hrw#2] node: Use typed HRW methods 
Update HRW lib and use typed HRW methods to sort shards and nodes

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
88e3868f47 [#37] cli: Add `nns-name` and `nns-zone` for `container create` 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
e9f3c24229 [#65] Use `strings.Cut` instead of `strings.Split*` where possible 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
6d4f48f37a [TrueCloudLab#78] .github: Fix CODEOWNERS 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
79ba34714a [#79] cli: Fix panic when setting domain for container 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
20de74a505 Rename package name 
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
381e363a8b [#32] node: Always close general components after testing 
It will prevent test fails with `-race` flag on components that have
background processes and make some actions on test framework.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
f1f3c80dbf [#32] node: Init write-cache asynchronously 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
7486c02bbc [#88] adm: Fix method `nnsResolveKey` 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
9929dcf50b [#126] adm: Exclude `group.frostfs` key from output of the `dump-hashes` 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
f2250a316f [#129] tree: Do not remove tree if the netmap is empty 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
6226c3ba86 [#129] policer: Use safer defaults 
If `processNodes` exits earlier for some reason, `needLocalCopy` could
be false.
See https://github.com/nspcc-dev/neofs-node/issues/2267

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
24a540caa8 [#132] cli/util: Fix basic ACL rendering 
In previous implementation pretty-printer of basic ACL in NeoFS CLI had
mistakes:
 * F-bit was set to `Extendable()` property instead of its inversion
 * B-bits were set to `acl.RoleInnerRing` rights

Make `PrettyPrintTableBACL` to correctly render mentioned bits.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
861e9ab59a [#83] pre-commit: Add initial configuration 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
a4931ea4c7 [#83] .github: Remove CODEOWNERS and actions 
Issue templates are still supported by Gitea:
https://docs.gitea.io/en-us/issue-pull-request-templates/ .

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
0893689c6a [#83] pre-commit: Add golangci-lint hook 
Skip deprecated warning for now, adopting new neo-go API will be done in
another task.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
8226d49376 [#83] pre-commit: Add gitlint hook 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
05471d3827 [#83] util/autocomplete: Fix deprecated warning 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
bce5827f64 [#83] pre-commit: Add shellcheck hook 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
365adb4ebd [#133] .github: Restore logo.svg 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
5890cd4d7d [#50] ir: Fix config property name for prometheus 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
5ae4446280 [#50] ir: Add Health status 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
3e6fd4c611 [#82] pilorama: Allow to store last sync height 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
6cd806f998 [#82] services/tree: Save last synchronized height in a persistent storage 
Remember the last synchronized height and use it after service restart.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
4e244686cf [#83] Makefile fixes for pre-commit 
Add make targets to simplify pre-commit setup for individual developers.

Signed-off-by: Stanislav Bogatyrev <realloc@realloc.spb.ru>
b4582239bf [#130] adm: Fix adding of pub key for `group.frostfs` at init step 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
8b9e40a848 [#85] get-service: Add assembler 
Extract assemble logic to assembler

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2886b1581b [#85] get-service: Add unit tests 
Add unit tests to cover all assemble statements

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
07de839f18 [#85] get-service: Fix corrupted chain logic 
Should return an error in case of a broken LOB reference chain.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
b8e93d4c08 [#85] get-service: Use assembler to assemble LOB 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
ac0a278a05 [#85] get-service: Drop unused assemble flag 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
b1c165a93b [#83] Fix shellcheck pre-commit action 
Original shellcheck action requires Docker to run and it's not always
available, especially inside Docker containers. Replacing it with python
wrapper to simplify usage with Docker-based CI systems like Drone and
WoodpeckerCI.

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
724debfdcd [#81] node: Add basic read/write benchmarks for substorages 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
22be532cbd object/put: Persist session token till the end of a session 
Previously a token could've expired in the middle of an object.PUT
stream, leading to upload being interrupted. This is bad, because user
doesn't always now what is the right values for the session token
lifetime. More than that, setting it to a very high value will
eventually blow up the session token database.

In this commit we read the session token once and reuse it for the whole
stream duration.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
f006f3b342 [#67] node: Make engine's `IsLocked` public 
It will allow reusing that method in expiration checks.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
64bde68fb9 [#67] node: Accept expired locked objects 
Allow replication of any (expired too) locked object. Information about
object locking is considered to be presented on the _container nodes_.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
b70caa216b [#144] Don't copy cache inside Docker environment 
May make docker builds not so clean.

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
10c419adf0 [#67] node: Fix infinite recursion in SE's wrapper 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
abd21f8099 [#136] Revert "[#2260] services/object: Do not assemble object with TTL=1" 
This reverts commit 2567f8020e. It assumes
that assembling logic could break some failover scenarios if request
forwarding is done. However, it also breaks requesting big objects via a
non-container node with TTL=2. Failover has been rechecked without that
commit and no problems were found. Any (if found) other bugs related to
the forwarding and object assembling must be solved more carefully.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
392be818e5 [#149] Fix gitlint regex to match our policy 
In our policy we mark commits not having a PR/Issue yet with a `[#XX]`
reference to be replaced after PR creation.

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
d64fb887ff [#149] Reorder pre-commit hooks 
Minor changes to see what fails first

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
573d920821 [#149] Use custom image and kludges for node 
Until #139 is fixed, we can't use root inside Docker container running
CI, but Woodpecker CI can't run non-root containers until they fix
https://github.com/woodpecker-ci/woodpecker/issues/1077, hence we use
temporary kludges with custom image and manual permissions in pipelines.

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2dc86058c3 [#148] memstore: Drop space line 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
cc8ff015b4 [#148] linter: Add containedctx linter 
Context has to be passed as an argument: https://pkg.go.dev/context

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
97c36ed3ec [#148] linter: Add funlen linter 
Long functions are hard to understand and source of errors

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
481a1ca6f3 [#148] linter: Add gocognit linter 
Code with high cognitive complexity is hard intuitively to understand

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
ci/woodpecker/push/full-pre-commit Pipeline failed Details
ci/woodpecker/push/pre-commit Pipeline was successful Details
44b86bac5a [#148] linter: Add contextcheck linter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
ci/woodpecker/push/full-pre-commit Pipeline failed Details
ci/woodpecker/push/pre-commit Pipeline was successful Details
9cd8f7cea0 [#152] IR: Process empty basic incomes 
If network is not configured for basic income earnings, do not distribute
GAS by the Alphabet nodes.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
6c4a1699ef [#145] shard-gc: Expired locked unit test 
Added unit test that verifies that GC deletes expired
locked objects in one epoch.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
5059dcc19d [#145] shard-gc: Delete expired objects after locks 
GC deletes expired locks and objects sequentially. Expired locks and
objects are now being deleted concurrently in batches. Added a config
parameter that controls the number of concurrent workers and batch size.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
ci/woodpecker/pr/full-pre-commit Pipeline failed Details
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/full-pre-commit Pipeline failed Details
ci/woodpecker/push/pre-commit Pipeline was successful Details
7a31988a36 [#145] docs: Add expired object collector params 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
ci/woodpecker/push/full-pre-commit Pipeline failed Details
ci/woodpecker/push/pre-commit Pipeline was successful Details
9a4f40626c [#128] IR: Do not try to emit GAS to nobody 
Fix sending GAS to an empty extra wallets receivers list. Also, send GAS to
extra wallets even if netmap is empty.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
ci/woodpecker/push/pre-commit Pipeline was successful Details
9aeea0b974 [#153] ci: Minor pipeline fixes 
- We can skip full pre-commit run
- On a very slow agent golangci run may take up to 10 minutes

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
7eb9e88f8f Add bug label; Delte label addition 
Signed-off-by: Liza <e.chichindaeva@yadro.com>
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details
ec2c5d45b4 Add to Possible Solutions 
Signed-off-by: Liza <e.chichindaeva@yadro.com>
47e8c5bf23 [#156] pilorama: Remove CIDDescriptor from TreeApply() 
Initially it was there to check whether an update is being initiated by
a proper node. It is now obsolete for 2 reasons:
1. Background synchronization fetches all operations from a single node.
2. There are a lot more problems with trust in the tree service, it is
   only used in controlled environments.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
5368c4207a [#156] services/tree: Split syncLoop() in functions 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
3f6b962349 [#156] services/tree: Pass context to replicationWorker() 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
fb13902db9 [#156] shard: Make refillMetabase() pass linter checks 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
8014fdb21a [#156] metabase: Make freePotentialLocks() pass linter checks 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
484ac502ca [#156] frostfs-adm: Make dumpBalances() pass linter checks 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
ci/woodpecker/push/pre-commit Pipeline was successful Details
cbc2efb1d6 [#156] object/get: Make toHeadPrm() pass gocognit 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
1637a3edce [#155] search-service: Add search with ST test 
In case of session token (ST) with object IDs search should
return only objects allowed in static session

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
ci/woodpecker/push/pre-commit Pipeline was successful Details
49234b915e [#155] search-service: Fix search with ST 
Search should return only objects allowed in static session

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
ci/woodpecker/push/pre-commit Pipeline was successful Details
da8da1c63a [#98] fstree: Do not fail iteration over just removed files 
A directory is read and files are saved to a local variable. The iteration
over such files may lead to a non-existing files reading due to a normal SN
operation cycle and, therefore, may lead to a returning the OS error to a
caller. Skip just removed (or lost) files as the golang std library does in
similar situations:
5f1a0320b9/src/os/dir_unix.go (L128-L133).

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
ci/woodpecker/push/pre-commit Pipeline was successful Details
f111704ceb [#158] Fix gitlint run in pre-commit under CI 
Wrong stage was used.
It still may run gitlint against only latest commit in PR.

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details
342e571d89 [#159] Add handle __SYSTEM__ sys attributes 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
ci/woodpecker/push/pre-commit Pipeline failed Details
9808dec591 [#86] node: Move testing utils to one package 
Move testing utils from tests in local_object_storage package to
unified testutil package

Signed-off-by: Airat Arifullin <aarifullin@yadro.com>
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details
34329d67ff [#86] node: Fix unit test and linter errors 
Signed-off-by: Airat Arifullin <aarifullin@yadro.com>
carpawell commented 2023-03-23 21:26:21 +00:00
Collaborator

Can we embed subsInfo here? It allows us to get rid of updateSubs and possibly prevents us from forgetting something to copy when we add it.

@fyrchik, fixed

> Can we embed subsInfo here? It allows us to get rid of updateSubs and possibly prevents us from forgetting something to copy when we add it. @fyrchik, fixed
carpawell added the due date 2023-03-24 2023-03-23 23:14:14 +00:00
carpawell requested review from aarifullin 2023-03-23 23:15:04 +00:00
carpawell requested review from storage-core-developers 2023-03-23 23:15:31 +00:00
carpawell requested review from storage-core-committers 2023-03-23 23:15:43 +00:00
carpawell force-pushed carpawell/upd/neo-go-subs from 1189ac6874 to 7a7573bbe4 2023-03-24 00:28:52 +00:00 Compare
carpawell force-pushed carpawell/upd/neo-go-subs from 7a7573bbe4 to 8c092eebba 2023-03-24 00:30:56 +00:00 Compare
fyrchik approved these changes 2023-03-24 07:29:08 +00:00
dstepanov-yadro approved these changes 2023-03-24 08:49:29 +00:00
fyrchik merged commit f3ff9fd251 into master 2023-03-24 09:42:31 +00:00
fyrchik deleted branch carpawell/upd/neo-go-subs 2023-03-24 09:42:31 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
aarifullin
fyrchik
dstepanov-yadro
ale64bit
dstepanov-yadro
fyrchik
acid-ant
Labels
Clear labels   
P0

Highest   
P1

High   
P2

Medium   
P3

Low   
badger

Badger DB related issues   
frostfs-adm
   
frostfs-cli
   
frostfs-ir
   
frostfs-lens
   
frostfs-node
   
good first issue

Good for newcomers   
triage
  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff   
blocked

The issue or PR is blocked by something   
bug

Something is not working   
config

Configuration format update or breaking change   
discussion

Talking about something in order to reach a decision or to exchange ideas   
documentation

Documentation related   
duplicate

This issue or pull request already exists   
enhancement

New feature   
go

Language features adoption   
help wanted

Extra attention is needed   
internal
   
invalid

Something is wrong   
kludge

This is a kludge and we know it   
observability

Related to metrics, tracing and logs   
perfomance

Perfomance related   
question

More information is needed   
refactoring

Refactoring   
wontfix

This won't be fixed
No Label
P0
P1
P2
P3
badger
frostfs-adm
frostfs-cli
frostfs-ir
frostfs-lens
frostfs-node
good first issue
triage
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
4 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

2023-03-24

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-node#73
There is no content yet.
Delete Branch "carpawell/upd/neo-go-subs"

Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?