e03d906cb7
[ #334 ] engine: Make tests more predictable
...
There is a codecov issue because objects are not placed
in the engine the same way every unit test. Therefore
sometimes there are more coverage, sometimes there are
less. Seeded RNG should solve this issue for engine tests.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-21 11:00:23 +03:00
0745267411
[ #47 ] innerring: Increase minimal extra fee on contract invocations
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-20 16:36:10 +03:00
8865252daf
[ #325 ] innerring: Fix storage node re-bootstrap approval
...
Netmap snapshot table caches bootstrap `AddPeer` txs so inner ring
node does not produce redundant approval txs for bootstrapped nodes.
However if node updates states to `Offline`, then such node should
be flagged in snapshot table, so re-bootstrap will actually produce
approval tx.
`ev.PublicKey.String()` returns uncompressed representation of the
node's public key, while snapshot contains compressed values.
Therefore the node was not flagged and re-bootstrap tx was not
approved by inner ring nodes.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-18 18:54:07 +03:00
a9c1f88b8a
[ #323 ] storage/blobovnicza: Add Timeout option of BoltDB instances
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-18 18:53:39 +03:00
06085ba8be
[ #323 ] storage/shard: Fix message of the error returned by Open method
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-18 18:53:39 +03:00
1ab630eb24
[ #323 ] storage/shard: Fix message of the error returned by Init method
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-18 18:53:39 +03:00
b5a1c03fb9
[ #323 ] storage/engine: Fix message of the error returned by Open method
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-18 18:53:39 +03:00
619f8826e1
[ #315 ] control: Implement SetNetmapStatus on Server
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 17:46:12 +03:00
f39d08bda7
[ #315 ] control: Add SetNetmapStatus rpc
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 17:46:12 +03:00
e5dc4ede57
[ #312 ] control: Add newline at the end of proto file
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:52:08 +03:00
f327024ee7
[ #312 ] control/server: Extended HealthChecker interface
...
Add HealthStatus method to HealthChecker interface that should return
current health status of the node application.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:52:08 +03:00
6ae51e41f9
[ #312 ] control: Add health status field to health check response body
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:52:08 +03:00
0abb8acef3
[ #312 ] control: Define HealthStatus enum
...
Define enumeration of the statuses of storage node application.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:52:08 +03:00
83d4420a30
[ #313 ] control: Rename HealthStatus enum to NetmapStatus
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:52:08 +03:00
a89567a88d
[ #317 ] morph/client: Return complete eACL signature from contract
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-15 16:25:49 +03:00
c75a828adf
[ #311 ] control: Fix loss of parent attribute keys in netmap converter
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
6c9dab481c
[ #311 ] control: Cover StableMarshal methods with unit tests
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
5f22e7fac7
Add newline at end of Control service proto file
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
36d8e2d3b7
[ #311 ] control: Implement JSON marshaler on Netmap message
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
dde24bc9e5
[ #311 ] control: Rewrite methods for binary encoding and signatures
...
Rewrite marshalers and sizers using proto library from API repo.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
e9d4b2d25b
[ #311 ] Recompile protobuf files
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
3e882d963f
[ #311 ] services/control: Implement NetmapSnapshot method
...
Implement methods required for signatures. Receive network map from netmap
storage, convert it to Control service message and return in response.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
28777911fb
[ #311 ] services/control: Define NetmapSnapshot rpc
...
Add NetmapSnapshot rpc to ControlService protobuf definition. Recompile
proto files. Add required method to server structure.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
88023f3655
[ #311 ] services/control: Define Netmap structure
...
Define NodeInfo protobuf type. Define Netmap protobuf type.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
62de9f327e
[ #319 ] Do not return payload formatting error on regular objects
...
Regular objects by definition have valid payload, so there should
be no errors.
Related to #302
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-15 16:01:29 +03:00
f311be3ab6
[ #234 ] Fix ineffassign linter remark
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
0a0ee89665
[ #306 ] control: Rename WithAllowedKeys function to WithAuthorizedKeys
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
abd9952e46
[ #306 ] Rename Private service to Control service
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
44a0fb5a69
[ #306 ] private: Define and use HealthChecker interface to get the status
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
c1b8a4815f
[ #306 ] private: Implement server of gRPC private node service
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
e75ddb0549
[ #306 ] private: Implement setters on generated proto messages
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
85ec633938
[ #306 ] Define and compile proto files for private node service
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
842fb9248c
[ #304 ] cmd/neofs-node: Tidy default configuration and log output
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 19:20:27 +03:00
1e170c3812
[ #234 ] services/object: Support netmap epoch and lookup dead in read ops
...
Support processing of NetmapEpoch and NetmapLookupDepth X-headers when
processing object read operations. Placement for operations
Get/Head/GetRange/GetRangeHash/Search is built for the epoch specified in
NetmapEpoch X-header (by default latest). Also the specified operations are
processed until success is achieved for network maps from the past up to
NetmapLookupDepth value. Behavior for default values (zero or missing) left
unchanged.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-12 19:19:11 +03:00
2f4d90025f
[ #234 ] core/netmap: Extend Source interface
...
Add GetNetMapByEpoch method. Add Epoch method.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-12 19:19:11 +03:00
1d56e60589
[ #302 ] Remove unused FSBucket component
...
FSBucket became obsolete when storage object engine has
been implemented.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 18:47:02 +03:00
b62b19cb5a
[ #302 ] Fix goimports linter errors
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 18:47:02 +03:00
e88c1f750d
[ #302 ] Fix golint else/return linter errors
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 18:47:02 +03:00
74a68af5c2
[ #302 ] Fix golint underscore linter errors
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 18:47:02 +03:00
7e30b0b9c8
[ #302 ] Fix exhaustive linter errors
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 18:47:02 +03:00
04c0c1b8f5
[ #302 ] Fix errcheck linter errors
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 18:47:02 +03:00
5e231c515a
[ #302 ] Fix whitespace linter errors
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 18:47:02 +03:00
9ea75e51b2
[ #303 ] morph/netmap: Implement getting network map snapshot by epoch
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-12 15:48:01 +03:00
d58e28afee
[ #298 ] morph/netmap: Use netmap.NodeState argument in UpdatePeerState
...
Replace custom NodeState enum with the one from netmap lib. Remove no longer
used NodeState enum.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-11 18:42:09 +03:00
159351fd55
[ #298 ] morph/netmap: Fix invocation of UpdateState contract method
...
Fix incorrect method name. Fix the order of arguments (should be {state,
key}).
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-11 18:42:09 +03:00
d26716e792
[ #298 ] ir: Fix parser of UpdateState notification event of Netmap contract
...
The order of event stack items is {state, key}.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-11 18:42:09 +03:00
3a7f6701d8
[ #290 ] object/rangehash: Apply salt to data before hashing
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-11 18:40:59 +03:00
0e2c761481
[ #290 ] object/rangehash: Add salt to RangeHashPrm structure
...
Add binary salt field to RangeHashPrm struct. Implement field setter. Set
salt from the request in v2 service.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-11 18:40:59 +03:00
ba03f46316
[ #290 ] util: Implement salting writer
...
Implement wrapper over io.Writer that applies binary salt to written data.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-11 18:40:59 +03:00
3bbd0bc95c
[ #300 ] innerring: Fix status value in update peer state invocation
...
We can't use enum values from SDK library directly, they can be
different from API specification. Therefore we need to convert them
into protocol level format.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-11 16:40:25 +03:00
9a86fff7e0
[ #291 ] Remove some unused code from repository
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-30 10:27:53 +03:00
a51211eda7
[ #291 ] Remove unused code from pkg/services/object/head
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-30 10:27:53 +03:00
3260e9263e
[ #291 ] Remove unused pkg/services/object/rangehash packages
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-30 10:27:53 +03:00
d299d94049
[ #291 ] Remove unused pkg/services/object/search/query packages
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-30 10:27:53 +03:00
0ce124a6e3
[ #291 ] Remove empty pkg/network/peers dir
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-30 10:27:53 +03:00
786da5313e
[ #291 ] Remove unused pkg/network/muxer package
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-30 10:27:53 +03:00
293af7b78e
[ #291 ] Remove empty pkg/network/bootstrap dir
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-30 10:27:53 +03:00
1dd1762163
[ #285 ] object/put: Fix the loss of X-headers when forwarding objects
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-29 12:22:05 +03:00
2897e83fb2
[ #285 ] object/eacl: Validate X-headers from the requests, not the responses
...
In previous implementation of eACL service v2 the response X-headers were
validated at the stage of re-checking eACL. This provoked a mismatch of
records in the eACL table with requests. Fix this behavior by checking the
headers from the request, not the response.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-29 12:22:05 +03:00
c69f867af1
[ #285 ] services/object: Add X-Headers to client call options
...
Forward request X-headers to client calls during internal processing of
Object operations on the node.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-29 12:22:05 +03:00
fe2b1fdc4b
[ #283 ] morph/client: Implement all calls for audit contract
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
7174abcc7c
[ #281 ] service/audit: Fix typos
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
e5108cf135
[ #281 ] service/audit: Run each SG check in separate routing at PoR
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
77cf97c236
[ #281 ] innerring: Create audit context once for one audit round
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
0d7832f5e9
[ #281 ] service/audit: Add separate pool for SG checks in PoR
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
8e72abaab7
[ #281 ] service/audit: Make report structure threadsafe
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
44d9f95137
[ #276 ] service/audit: Count Head requests and retries in PoR
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
9273a89f61
[ #276 ] service/audit: Use random nodes for object.Head in PoR
...
Since PoR audit check uses object.Head with large TTL values to
make real proof of retrievability, we may use random remote
nodes.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
10b548275a
[ #276 ] innerring: Use Head with TTL in PoR
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
f9e81383ae
[ #259 ] innerring: Use blocking pools for audit
...
Audit task manager should not discard tasks if all workers are
busy, therefore pools should not be non-blocking.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
3ec342b2c3
[ #259 ] services/audit: Implement PDP check
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
310a4c3e4d
[ #275 ] innerring: Add storage group search timeout
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
6836975272
[ #275 ] innerring: Use crypto rand shuffle in audit
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
f593f0b2d6
[ #274 ] innerring: Implement GetRangeHash in audit communicator
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
6977adec66
[ #273 ] service/audit: Ignore coverage of very small objects
...
PDP audit check is not quite working with very small objects, so
we try to build coverage with bigger objects.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
afb83c610c
[ #273 ] service/audit: Add cache of head responses in PoR
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
9212864f42
[ #258 ] services/audit: Implement PoP check
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
a5320408a5
[ #271 ] service/audit: Implement PoR
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
5f65ec0265
[ #271 ] audit/report: Add PoR related setters
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
85773c419b
[ #271 ] service/object: Fill all fields of new short header structure
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
af7ae0ec4c
[ #271 ] acl: Add permission to get objects for inner ring
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
782bcadd92
[ #271 ] innerring: Implement `GetSG` and `Head` of audit communicator
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
dd5e67ff0c
[ #271 ] service/audit: Add netmap structure to audit task
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
64b022a5ed
[ #271 ] placement: Add FlattenNodes function
...
This function converts 2-dimension array of container nodes
into single dimension array. Useful when we need to iterate
over whole container.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
58033a3adc
[ #271 ] placement: Add BuildObjectPlacement function
...
This function allows to shuffle prepared container nodes
according to HRW weights from network map.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
a8c26378b1
[ #271 ] pkg/network: Add multiaddr to ipaddr converter
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
2ebcbe70fb
[ #255 ] ir/audit: Make task manager to return number of skipped tasks
...
Add numeric return from TaskManager.Reset method that shows the number of
canceled tasks. This values will be used for assessment of the progress of
the audit.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
03e3afb0e8
[ #255 ] services/audit: Define interface of container communicator
...
Define interface of the container communicator which methods are going to be
used in audit checks. Make innerring Server to implement this interface.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
754cd8b579
[ #255 ] ir: Replace Server.WriteReport method to another source file
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
54523d2949
[ #255 ] ir: Replace audit client creation to invoke package
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
76d4e53ea0
[ #255 ] services/audit: Skip all tasks from previous epoch in audit processor
...
Implement Reset method on audit task manager that cleans task queue.
Extended TaskManager interface with Reset method on IR side. Call Reset
method in audit processor before new audit start.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
33804e024d
[ #255 ] services/audit: Complete audit report after the last check
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
4dc09b19f3
[ #255 ] ir: Make audit processor to push tasks to audit task manager
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
580c9c974a
[ #255 ] services/audit: Implement task manager
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
0f0be2377b
[ #255 ] services/audit: Implement audit executor without checks
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
babfbc18f2
[ #255 ] services/audit: Implement Task structure
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
076f201807
[ #255 ] services/audit: Define Report structure and Reporter interface
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
919f4364f1
[ #269 ] morph/audit: Implement wrapper over Audit contract client
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
07da9d31f2
[ #269 ] morph/client: Implement Audit contract client
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
8dd7c689f2
[ #265 ] innerring: Select storage groups to audit
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
22cffbf529
[ #265 ] innerring: Add client cache
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
3d3d058b05
[ #265 ] innerring: Select containers to audit
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
87e1252065
[ #265 ] innerring: Generate new audit event
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
8c4e033db3
[ #80 ] core/object: Add SG-object format validation
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
43ee94a572
[ #80 ] storageroup: Implement function to collect structure from container
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
44466f4a29
[ #80 ] storagegroup: Implement search query generator
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
6245e58d1f
[ #80 ] services/object: Implement iterators over split-tree leaves
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
bb85ac24eb
[ #254 ] innerring: Increase extra fee for container registration
...
Some rough estimation until there will be p2p signature collection
inside the chain.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
feec559e47
[ #254 ] Update neo-go to preview4 compatible version
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
e53bf574b5
[ #279 ] container: Use new methods to work with container format
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-24 13:38:20 +03:00
bbe8f34ffd
[ #253 ] object/eacl: Process only CID/OID object headers in RANGEHASH/DELETE
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-18 18:31:20 +03:00
9e27263df3
[ #253 ] object/put: Fix loss of session token
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-18 18:31:20 +03:00
8e661d5c37
[ #264 ] object/get: Prevent receiving payloads through GETRANGE when GET
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-18 16:42:17 +03:00
66f9532857
[ #264 ] object/get: Check parent address in linking/last child
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-18 16:42:17 +03:00
ec21040542
[ #264 ] transformer: Finalize parent header once
...
In previous implementation parent object header finalized twice in size
limiter + formatter. On the one hand, this added redundant action, on the
other hand, it could provoke a difference in the headers of the linking and
the last part. Change formatter to finalize parent header if it does not
container the signature. Change size limiter to reuse parent header after
last child finalization in linking child.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-18 16:42:17 +03:00
edef9463d7
[ #253 ] metabase: Add prefix to Graveyard and ToMoveIt bucket names
...
In previous implementation DB.Containers method could return an error about
invalid container ID string format. This could happen if some of top-level
buckets had name w/o "_" substring.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-17 12:03:18 +03:00
49131f1bc7
[ #247 ] object/eacl: Use address from session token in request validation
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-15 13:21:13 +03:00
dba88c79b4
[ #247 ] object/range: Process only object_id and container_id headers
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-15 13:21:13 +03:00
8654458b19
[ #247 ] object/eacl: Use object ID from session token context
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-15 13:21:13 +03:00
168dcbdccd
[ #247 ] object/eacl: Process object address from request
...
In previous implementation eACL validator didn't take into account container
and object ID fields of request bodies.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-15 13:21:13 +03:00
bbe700fa37
[ #250 ] service/policer: Don't shrink node list at unknown error
...
Every unknown error must not decrease shortage counter and must not
exclude faulty node from the node list, because this list will be used
later for replication.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-14 21:49:50 +03:00
e6f2d84736
[ #246 ] blobovnivza: Fix deadlock on concurrent evict and open new blobovnicza
...
Deadlock occurs when `getActivate` function opens new blobovnicza and that
invokes evict in LRU cache of open blobovniczas. `getActivate` makes
`activeMtx.Lock()` and then cache evict makes `activeMtx.RLock()` and deadlock
happens.
Fix contains two steps:
- add separate mutex to open blobovniczas (1),
- split single Lock outside of `updateAndGet` (2).
As for the (1) `bbolt.Open()` locks when it tries to open the same file from
two threads. So separate mutex will prevent that.
As for the (2) `updateAndGet` function contains from two parts. At first it
checks if required blobovnicza is ready and it returns it. In this case we can
use the simple RLock. But then there is an option when we should open new
blobovnicza and update map of active blobovniczas.
In this case we call `openBlobovnicza` without activeMtx lock. Cache evict
happens there and it won't cause deadlock.
Then we lock activeMtx to update the map of active blobovniczas. Concurrency can
happen there. However `openBlobovnicza` will not open the same blobovnicza twice,
so we can make one more check if opened blobovnicza was activated while thread was
locked in activeMtx. If so, then return active blobovnicza, else finish activation.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-14 13:37:45 +03:00
91bea44a1a
[ #245 ] blobovnicza: Fix initializing dimensional buckets
...
In previous implementation Blobovnicza could incorrectly initialize
dimensional buckets: if SmallSizeLimit = 2 ^ X + Y && Y < 2 ^ X, then
largest dimensional bucket was [2 ^ (X - 1) : 2 ^ X]. This was caused by an
incorrect condition for stopping the iterator along the dimensional
boundaries.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-14 12:48:59 +03:00
fb50362dcc
[ #243 ] services/object: Share common parameters across services
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
a01262d8bd
[ #243 ] service/object: Fix object chain assembly for tombstone body
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
9265e31e65
[ #243 ] services/object: Fix lost tokens when generating requests
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
fe3906c295
[ #243 ] object/delete: Implement new service processing
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
510e9ff2ec
[ #243 ] core/object: Replace test content of tombstone with an API structure
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
4f5d5c7e45
Write system path to BoltDB on blobovnicza closing
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
627012e4fe
[ #241 ] object/search: Set container ID parameter of storage engine call
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
611a29f682
[ #241 ] object/search: Refactor service processing
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
3bfb18386b
object/get: Replace code with function call
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
63bc3aab82
[ #242 ] engine: Do not iterate over shards if cid is not set
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
9fb7190358
[ #242 ] service/object: Support ContainerID field in local search
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
511b5f782a
[ #242 ] engine: Support ContainerID argument in shard selects
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
f585772746
[ #242 ] shard: Support ContainerID argument in metabase selects
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
b97f818323
[ #242 ] metabase: Make ContainerID a mandatory parameter for Select
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
0e1f05ff45
[ #239 ] object/head: Implement new service processing
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
9dd83bdf0d
[ #235 ] blobstor: Return object.ErrRangeOutOfBounds from shallow dir
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
41b9fa5b45
[ #235 ] engine: Return OutOfBounds error
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
173d34a8a2
[ #235 ] object/get: Set common operation parameters
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
7e56427534
[ #235 ] object/getrangehash: Implement new service processing
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
0fc5ea674c
[ #235 ] object/getrange: Fix incorrect processing of range from last child
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
397d912e19
[ #235 ] object/head: Support raw flag in service
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
1d23483828
[ #235 ] services/object: Implement new GetRange algorithm
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
91d8e0a4de
[ #238 ] engine: Support raw flag in Head method
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
da42ee3a26
[ #238 ] engine: Stop shard iteration at split info error in Head
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
69dee6ec6d
[ #238 ] shard: Support raw flag in Head method
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
bf7a390304
[ #238 ] metabase: Support raw flag in Get method
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
c267a08886
[ #236 ] engine: Stop shard iteration at split info error in GetRange
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
590745204c
[ #237 ] metabase: Structure parameters and results of all operations
...
All parameters and resulting values of all metabase operations are
structured in new types. The most popular scenarios for using operations are
moved to auxiliary functions.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
a875d80491
Remove outdated code of metabase and localstore
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
869d9e571c
[ #233 ] object/head: Remove getting the right child header
...
With the update of the local storage engine, the headers of virtual objects
are directly given. In this regard, the step with obtaining the the right
child header is removed.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
f24daa10ff
[ #233 ] services/object: Implement new Get algorithm
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
26f03c6301
[ #232 ] Fix SplitInfo update in existed virtual object
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
e0350efe00
[ #231 ] services/policer: Use engine.List method
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
ee1e3931a5
[ #231 ] engine: Add List method
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
a7087faa1a
[ #231 ] shard: Add List method to return all available objects
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
a1cb48f800
[ #231 ] metabase: Add container lister
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
3a37eda410
[ #230 ] engine: Stop on first SplitInfoError in Get method
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
a488d202e1
[ #230 ] engine: Rename `WithAddress` to `WithAddresses` for delete operations
...
Delete works with multiple addresses at once so it is convenient to
show it in parameter setter function.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
7c97bd1360
[ #230 ] shard: Add test for getting SplitInfo
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
bf9e938a3b
[ #230 ] metabase: Return SplitInfoError in Exist method
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
e2de95e3f6
[ #230 ] metabase: Index split info structures
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
eaae5a5dd7
[ #229 ] blobovnicza: Store objects in a binary format
...
In previous implementation Blobovnicza's stored objects in protocol format
which did not allow working with externally compressed objects. To achieve
this goal, operations Get and Put no longer work with the structure of the
object, but only with abstract binary data. Operation GetRange has become
incorrect in its original purpose to receive the payload range. In this
regard, BlobStor receives the payload range of the object through Get
operation. In the future either Blobovnicza will learn to compress objects
by itself, or the GetRange operation will be eliminated.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
3d77fdb347
[ #222 ] Set write cache specific config on application level
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
af2719689d
[ #222 ] Do not compress data in blobovniczas
...
Blobovnicza returns object, so we can't put compressed
data there. Compressed data won't be deserialized correctly.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
6bec50963b
[ #222 ] Add tests for shards
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
ce92df3543
[ #222 ] Return nil in `metabase.IsSmall` if object is big
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
862f9a3f1a
[ #222 ] Put small objects in blobovnicza in blobStor
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
351e4b4592
[ #222 ] Support Inhume and Delete in object service
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
21708d5408
[ #222 ] Add Exists method in storage engine
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
f470022594
[ #222 ] Update Select method in storage engine
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
25486194b9
[ #222 ] Update Head method in storage engine
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
67d4b0db12
[ #222 ] Move ErrAlreadyRemoved to core package
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
6a3e3e5642
[ #222 ] Update Range method in storage engine
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
c9b1c6603e
[ #222 ] Update Get method in storage engine
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
f3e96f8c19
[ #222 ] Update Delete method in storage engine
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
aa8dd18b07
[ #222 ] Update Put method in storage engine
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
f45b7a048d
[ #222 ] Rename `WithTreeRoot..` to `WithRoot..` in blobstor opts
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
d4e48b28b1
[ #222 ] Add Inhume method to storage engine
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
92c95150cd
[ #222 ] Use write cache options in storage app
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
69ba295077
[ #222 ] Add ToMoveIt method in shard
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
ccd8760fbc
[ #222 ] Update Exists method in shard
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
97a2e26b71
[ #222 ] Add Head method in shard
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
244e244c0e
[ #222 ] Add GetRange method in shard
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
73505fe582
[ #222 ] Update Get method in shard
...
Now `Get`, `Head` and `GetRange` will be
separate methods on shard.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
31371771e8
[ #222 ] Update Delete method in shard
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
5fc54386b0
[ #222 ] Update Put method in shard
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
72f8ca1afd
[ #222 ] Add Inhume method for shard
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
1b76458684
[ #222 ] Add writeCache instance to shard
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
cbdcfb9159
[ #220 ] shard: Enable write cache through option
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
47d2239332
[ #220 ] localstorage: Replace basic errors to core library
...
Replace ErrNotFound and ErrRangeOutOfBounds to core/object package in order
to share them across the libraries.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
7512a5ba18
[ #220 ] blobstor: Implement "small" operations
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
7f20e6f9db
[ #220 ] blobstor: Implement GetRange on blobovnicza tree
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
ef095d4871
[ #220 ] blobovnicza: Implement GetRange operation
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
e758c246b7
[ #220 ] localstorage: Implement Open/Init/Close methods
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
2963473c08
[ #220 ] localstorage: Implement hrw.Hasher over Shard structure
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
88c1584e6a
[ #220 ] shard: Implement Open/Init/Close methods
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
d30454a572
[ #220 ] shard: Use new metabase implementation
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
d8d38d3476
[ #220 ] blobstor: Add blobovnicza tree structure to BlobStor
...
Add blobovnicza instance to BlobStor structure. Create blobovnicza tree in
BlobStor constructor. Implement Open/Init/Close methods.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
51ab6991d2
[ #220 ] blobstor: Implement method to close blobovnicza tree
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
34a259d5c0
[ #220 ] blobstor: Support zero shallow depth of blobovnicza tree
...
There is a need to support single blobovnicza in blobovnicza tree. This can
be achieved with a width of 1, and a depth of 0 or 1. With depth = 1 one
redundant directory is created, inside which there is a blobovnicza. If the
depth is zero, the blobobnivza will be in the root path. Fix negative
capacity in iterateDeepest method with zero depth.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
8bff073061
[ #220 ] blobstor: Fix hard-code in blobovnicza unit test
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
9d4fba923c
[ #199 ] Create metabase instance as storage engine component
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
8a293c76ed
[ #199 ] Ignore invalid search attributes
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
337e2f6a57
[ #199 ] Add SplitID index in metabase
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
74d44beb99
[ #199 ] Add CleanUp method to remove empty buckets from metabase
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
7fb87aac85
[ #199 ] Support Delete operation in metabase
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
e42fa69735
[ #199 ] Make option to get object from metabase without graveyard check
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
b45d523125
[ #199 ] Split IsSmall check in metabase
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
5bfae833fe
[ #199 ] Update blobovniczaID on existed objects in metabase
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
54cd91adff
[ #199 ] Simplify tests
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
a8d76f2ebb
[ #199 ] Implement ObjectID selection filter in metabase
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
700bd7de01
[ #199 ] Get virtual objects from metabase
...
Virtual objects are obtained differently with
relative lookup in parent bucket.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
4e7d49791b
[ #199 ] Index parent first in metabase
...
With exist check we should index parent first, because
as soon as child will be added to metabase, exist on
parent will return true even if it was not indexed yet.
Also this commit makes one db.Update instead of two for
parent and child.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
f6387a9b94
[ #199 ] Make exist check on all primary buckets in metabase
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
e478c0d024
[ #199 ] Refactor metabase internal structure
...
Accoring to MetaBase-Plan-B storage engine specification.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
1c81d507fd
[ #218 ] blobstor: Inherit the root path to Blobovnicza from BlobStor
...
Place the root of blobovnicza tree in a subdirectory of BlobStor with same
permissions. Abolish WithBlobovniczaRootPath and WithBlobovniczaPersmissions
options.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
4c3d231e82
[ #218 ] blobstor: Implement blobovnicza tree
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
3acbd3011b
[ #218 ] blobovnicza: Сreate at least one range bucket
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
1b0d2c67d1
[ #218 ] blobstor: Add blobovnicza ID value to DeleteSmallPrm
...
Blobovnicza ID parameter provides the ability to specify particular
blobovnicza to delete object from. In this case only specified blobovnicza
is processed.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
a97ab242eb
[ #218 ] blobstor: Add config values of blobovnicza tree
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
35ccda13a8
[ #216 ] blobovnicza: Change ID stringer
...
Replace hex encoding with slice to string conversion.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
07e998d846
[ #216 ] blobovnicza: Implement main methods and logic
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
52b0c1f19a
[ #216 ] blobovnicza: Add configuration values
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
bce16de72a
[ #216 ] blobstor: Classify objects by size according to binary format
...
In previous implementation objects were classified by size according to
payload size. From now they are classified by the size of their binary
representation.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
40b09f9266
[ #216 ] blobovnicza: Define Blobovnicza type
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
b9054e2ee0
[ #211 ] blobstor: Add a branch for processing objects by size in Put
...
Implement "big or small" property classifier (only the size of the payload
is temporarily considered). Save "big" objects in shallow dir. Save "small"
objects in shallow dir until the moment of implementation of blobovnicza.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
cfc770c3fe
[ #211 ] blobstor: Add SmallSizeLimit parameter
...
Add numeric parameter of maximum size of the "small" objects.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
3028718691
[ #211 ] blobovnicza: Refactor ID implementation
...
Replace UUID implementation of ID with byte slice.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
53b114cf8b
[ #211 ] blobstor: Remove global lock
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
59f7cf9873
[ #211 ] blobstor: Implement Exists method
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
1ba556f5e7
[ #211 ] blobstor: Define Iterate operation
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
3c0ef270f4
[ #211 ] blobstor: Define Exists operation
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
915243c69e
[ #211 ] blobstor: Define DeleteSmall operation
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
65da2025e5
[ #211 ] blobstor: Rename Delete to DeleteBig
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
b605ea3678
[ #211 ] blobstor: Refactor Put operation
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
7afcde659d
[ #211 ] blobstor: Define GetRangeSmall operation interface
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
0b130150ae
[ #211 ] blobstor: Rename GetRange to GetRangeBig
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
2ba4ebc8c9
[ #211 ] blobstor: Implement no-op GetSmall operation
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
2f61d8d63c
[ #211 ] localstorage: Define blobovnicza ID
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
6813f40665
[ #211 ] blobstor: Refactor GetBig parameters and result
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
fb6857a1cb
[ #211 ] blobstor: Rename Get operation to GetBig
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
4e8cc51e7b
[ #189 ] shard: Extend Info with WeightValues
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
4f16a10235
[ #189 ] localstorage: Implement DumpInfo method
...
Implement method to get the information about the local storage.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
8fbda2a588
[ #189 ] shard: Implement DumpInfo method
...
Implement method to get the information about the shard.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
24cf86e269
[ #189 ] metabase: Implement DumpInfo method
...
Implement method to get the information about the metabase.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
eaa7068e3c
[ #189 ] blobstor: Implement DumpInfo method
...
Implement method to get the information about the BLOB storage.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
dae8d3de5b
[ #189 ] shard: Fix missing return statement in SetMode method
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
Leonard Lyubich