a7ae88e1fb
[ #471 ] tree: Don't use sorted GetSubTree for nodes without FileName
...
/ DCO (pull_request) Successful in 1m15s
/ Vulncheck (pull_request) Successful in 1m17s
/ Builds (1.21) (pull_request) Successful in 1m38s
/ Builds (1.22) (pull_request) Successful in 1m36s
/ Lint (pull_request) Successful in 2m49s
/ Tests (1.21) (pull_request) Successful in 1m41s
/ Tests (1.22) (pull_request) Successful in 1m42s
Sorted GetSubTree doesn't return nodes without FileName attribute
if there are more than 1000 nodes in subtree
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-08-21 15:25:58 +03:00
2bae704d3e
[ #437 ] tree: Add cleanOldNodes method
...
/ Vulncheck (pull_request) Successful in 59s
/ DCO (pull_request) Successful in 56s
/ Builds (1.21) (pull_request) Successful in 1m26s
/ Builds (1.22) (pull_request) Successful in 1m18s
/ Lint (pull_request) Successful in 2m11s
/ Tests (1.21) (pull_request) Successful in 1m43s
/ Tests (1.22) (pull_request) Successful in 1m36s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-07-23 16:47:05 +03:00
689f7ee818
[ #437 ] tree: Support removing old split system nodes
...
/ DCO (pull_request) Successful in 1m47s
/ Builds (1.21) (pull_request) Successful in 2m20s
/ Builds (1.22) (pull_request) Successful in 2m22s
/ Vulncheck (pull_request) Successful in 2m6s
/ Lint (pull_request) Successful in 4m43s
/ Tests (1.21) (pull_request) Successful in 2m30s
/ Tests (1.22) (pull_request) Successful in 2m25s
It's need to fit user expectation on deleting CORs for example.
Previously after removing cors (that was uploaded in split manner)
we can still get some data (from other node)
because deletion worked only for latest node version.
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-07-22 10:42:11 +03:00
c0011ebb8d
[ #430 ] tree: Fix multipart having system name
...
Previously if multipart key has the same name as some system node
(e.g. bucket-settings, bucket-cors etc.) it shadows real system node
and bucket started to be unversioned again for example.
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-07-19 11:24:50 +03:00
456319d2f1
[ #430 ] Fix split tree
...
Update tree service to fix split tree problem.
Tree intermediate nodes can be duplicated, so we must handle this.
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-07-19 11:24:46 +03:00
9432782ce6
[ #401 ] Drop notifications
...
/ DCO (pull_request) Successful in 2m5s
/ Builds (1.20) (pull_request) Successful in 2m40s
/ Builds (1.21) (pull_request) Successful in 2m33s
/ Vulncheck (pull_request) Successful in 2m22s
/ Lint (pull_request) Successful in 4m24s
/ Tests (1.20) (pull_request) Successful in 2m48s
/ Tests (1.21) (pull_request) Successful in 2m45s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-06-25 15:49:37 +03:00
2b04fcb5ec
[ #406 ] Remove control api
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-06-21 06:36:56 +00:00
bb81afc14a
[ #398 ] Support retryer
...
Add two strategy for PutBucketSettings request retryer:
* exponential backoff (increasing up to `max_backoff` delays with jitter)
* constant backoff (always the same `max_backoff` delay between requests)
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-06-06 13:02:17 +00:00
9152b084ec
[ #387 ] Fix typo
...
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2024-05-22 15:06:02 +00:00
2f108c9951
[ #362 ] Expand control service
...
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2024-05-08 15:15:49 +03:00
d903de2457
[ #370 ] Fix fetching attributes from tree
...
/ DCO (pull_request) Successful in 2m6s
/ Vulncheck (pull_request) Successful in 2m48s
/ Builds (1.20) (pull_request) Successful in 3m19s
/ Builds (1.21) (pull_request) Successful in 3m15s
/ Lint (pull_request) Successful in 4m50s
/ Tests (1.20) (pull_request) Successful in 3m13s
/ Tests (1.21) (pull_request) Successful in 3m4s
Port #374
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-04-19 17:33:55 +03:00
cb83f7646f
[ #347 ] port: Explicitly specify sorting order of subtree for object listing
...
/ DCO (pull_request) Successful in 1m56s
/ Vulncheck (pull_request) Failing after 4m57s
/ Builds (1.20) (pull_request) Successful in 5m54s
/ Builds (1.21) (pull_request) Successful in 5m56s
/ Lint (pull_request) Successful in 13m10s
/ Tests (1.20) (pull_request) Successful in 5m34s
/ Tests (1.21) (pull_request) Successful in 3m22s
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2024-04-09 18:57:47 +03:00
6788306998
[ #328 ] Log invalid tree service KVs
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-03-04 15:35:23 +03:00
8050ca2d51
[ #306 ] Use session token for container read operations
...
/ DCO (pull_request) Successful in 1m54s
/ Vulncheck (pull_request) Successful in 1m55s
/ Builds (1.20) (pull_request) Successful in 2m49s
/ Builds (1.21) (pull_request) Successful in 1m56s
/ Lint (pull_request) Successful in 3m59s
/ Tests (1.20) (pull_request) Successful in 2m30s
/ Tests (1.21) (pull_request) Successful in 2m19s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-03-01 18:14:33 +03:00
ff15f9f28a
[ #306 ] Fix update settings for buckets without owner key in tree
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
3d0d2032c6
[ #306 ] acl: Handle put/get acl for APE buckets
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
1f2cf0ed67
[ #306 ] Use APE instead of eACL on bucket creation
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
c334adeb6d
[ #165 ] Sort nodes in ServiceClientMemory
...
/ DCO (pull_request) Successful in 1m22s
/ Builds (1.20) (pull_request) Successful in 1m52s
/ Builds (1.21) (pull_request) Successful in 1m17s
/ Vulncheck (pull_request) Successful in 1m51s
/ Lint (pull_request) Failing after 1m55s
/ Tests (1.20) (pull_request) Successful in 2m24s
/ Tests (1.21) (pull_request) Successful in 2m24s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:53:12 +03:00
4e15452853
[ #165 ] Fix lint errors
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
de32dfd7ce
[ #165 ] Rename SubTreeStreamImpl to SubTreeStreamMemoryImpl
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
8a30f18ff6
[ #165 ] Don't use recursion in list streaming
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
2d7973b3f1
[ #165 ] Refactor list versions
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
093de13f54
[ #165 ] Add stream listing tests
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
739a6ec9df
[ #165 ] Support latest only stream listing
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
c7ee628ab0
[ #165 ] Fix versions listing
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
4ad84b9b94
[ #165 ] listing: Use NodeVersion instead of ObjectInfo
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
b52552e8c2
[ #165 ] Add batching in streamin listing
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
6e8960b2ab
[ #165 ] Add list session cache
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
29ac91dfd5
[ #165 ] Support streaming listing
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
84af85ed67
[ #302 ] Update APE to support chain id as bytes
...
/ DCO (pull_request) Successful in 1m52s
/ Vulncheck (pull_request) Successful in 2m0s
/ Builds (1.20) (pull_request) Successful in 2m49s
/ Builds (1.21) (pull_request) Successful in 1m59s
/ Lint (pull_request) Successful in 2m44s
/ Tests (1.20) (pull_request) Successful in 2m30s
/ Tests (1.21) (pull_request) Successful in 2m24s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 12:53:45 +03:00
6f9ee3da76
[ #275 ] Change logic delete multipart upload
...
/ Vulncheck (pull_request) Successful in 1m13s
/ DCO (pull_request) Successful in 1m43s
/ Builds (1.20) (pull_request) Successful in 2m49s
/ Builds (1.21) (pull_request) Successful in 1m36s
/ Lint (pull_request) Successful in 3m33s
/ Tests (1.20) (pull_request) Successful in 2m34s
/ Tests (1.21) (pull_request) Successful in 2m29s
In order not to accidentally take outdated
information about downloaded parts from other
nodes, now when the multipart is abort or complete,
the root node of the multipart upload with the
finish flag remains in the tree.
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2023-12-27 13:06:45 +03:00
6509639540
[ #283 ] control: Changed type of chainID to bytes
...
/ DCO (pull_request) Successful in 1m38s
/ Builds (1.20) (pull_request) Successful in 2m11s
/ Builds (1.21) (pull_request) Successful in 2m7s
/ Vulncheck (pull_request) Successful in 1m54s
/ Lint (pull_request) Successful in 4m2s
/ Tests (1.20) (pull_request) Successful in 2m23s
/ Tests (1.21) (pull_request) Successful in 2m13s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-21 18:07:17 +03:00
6dbb07f0fa
[ #261 ] Update policy-engine dependency
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-14 17:52:13 +03:00
9272f4e108
[ #259 ] Support contract based policies
...
/ DCO (pull_request) Successful in 1m21s
/ Vulncheck (pull_request) Successful in 1m41s
/ Builds (1.20) (pull_request) Successful in 2m19s
/ Builds (1.21) (pull_request) Successful in 2m1s
/ Lint (pull_request) Successful in 3m20s
/ Tests (1.20) (pull_request) Successful in 2m14s
/ Tests (1.21) (pull_request) Successful in 2m10s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-11 10:01:46 +03:00
f58a0d04ff
[ #233 ] Clean tag node in the tree service instead of removal
...
With new retry policy of tree service pool, gateway should avoid
deletion of system nodes from tree. Absence of node in the tree
will trigger retry. Other storage in the network may return already
deleted node while tree is not completely synced, and client will
get unexpected result.
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-12-07 16:40:56 +03:00
473239bf36
[ #257 ] Add policy checker
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-06 17:47:51 +03:00
42862fd69e
[ #258 ] Support policy management in control svc
...
/ Vulncheck (pull_request) Successful in 3m20s
/ DCO (pull_request) Successful in 3m56s
/ Builds (1.20) (pull_request) Successful in 4m13s
/ Builds (1.21) (pull_request) Successful in 4m9s
/ Lint (pull_request) Successful in 3m9s
/ Tests (1.20) (pull_request) Successful in 4m14s
/ Tests (1.21) (pull_request) Successful in 3m59s
Add PutPolicies, RemovePolicies, GetPolicy, ListPolicies methods
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-01 15:47:12 +03:00
c7a65bd075
[ #258 ] Add control service
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-01 14:17:06 +03:00
25bb581fee
[ #205 ] Add md5 checksum in header
...
/ Vulncheck (pull_request) Successful in 1m50s
/ DCO (pull_request) Successful in 3m29s
/ Lint (pull_request) Successful in 3m37s
/ Tests (1.20) (pull_request) Successful in 2m9s
/ Tests (1.21) (pull_request) Successful in 2m5s
/ Builds (1.20) (pull_request) Successful in 5m47s
/ Builds (1.21) (pull_request) Successful in 1m29s
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-10-25 11:04:19 +03:00
e1ec61ddfc
[ #215 ] Fix get latest version node
...
When the object version is received,
the node of the secondary object may return.
Now we choose the right node ourselves.
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2023-10-06 09:21:41 +00:00
adec93af54
[ #185 ] tree: Fix getSubTreeMultipartUploads
...
Every tree node contains only FileName
but key in multipart info must contain FilePath
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-08-24 18:01:08 +03:00
8efcc957ea
[ #96 ] Move log messages to constants
...
/ DCO (pull_request) Successful in 1m35s
/ Builds (1.19) (pull_request) Successful in 2m14s
/ Builds (1.20) (pull_request) Successful in 2m9s
/ Vulncheck (pull_request) Successful in 5m39s
/ Lint (pull_request) Successful in 2m49s
/ Tests (1.19) (pull_request) Successful in 7m34s
/ Tests (1.20) (pull_request) Successful in 1m44s
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2023-08-23 18:32:31 +03:00
52931663e1
[ #176 ] multipart: Replace part on re-upload
...
/ Builds (1.19) (pull_request) Successful in 2m57s
/ Builds (1.20) (pull_request) Successful in 2m36s
/ DCO (pull_request) Successful in 1m15s
/ Vulncheck (pull_request) Successful in 5m15s
/ Lint (pull_request) Successful in 3m50s
/ Tests (1.19) (pull_request) Successful in 8m14s
/ Tests (1.20) (pull_request) Successful in 3m17s
We want to have exactly one object and tree node for each part number
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-21 16:20:04 +03:00
ad81b599dd
[ #63 ] Add fast multipart upload
...
Add new flag to object tree meta `isCombined` that means
the object payload is list of parts that forms real payload.
Set this attribute when complete multipart upload not to do unnecessary copying.
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-20 14:00:15 +03:00
fc90981c03
[ #149 ] Update inner imports after moving middlewares
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-11 17:25:09 +03:00
23593eee3d
[ #111 ] Use request scope logger
...
/ Lint (pull_request) Failing after 32s
/ Tests (1.19) (pull_request) Failing after 32s
/ Tests (1.20) (pull_request) Failing after 32s
/ Builds (1.19) (pull_request) Failing after 32s
/ Builds (1.20) (pull_request) Failing after 32s
/ Vulncheck (pull_request) Failing after 57s
/ DCO (pull_request) Failing after 1m0s
Store child zap logger with request scope fields into context.
Request scoped fields: request_id, api/method, bucket, object
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-19 13:54:51 +03:00
462589fc0c
[ #103 ] Return 504 http code on timeout errors
...
/ Vulncheck (pull_request) Successful in 1m42s
/ Lint (pull_request) Successful in 3m31s
/ Tests (1.19) (pull_request) Successful in 2m57s
/ Tests (1.20) (pull_request) Successful in 3m4s
/ Builds (1.19) (pull_request) Successful in 2m35s
/ Builds (1.20) (pull_request) Successful in 2m39s
/ DCO (pull_request) Successful in 1m7s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-14 09:16:21 +03:00
4a6e3a19ce
[ #125 ] Handle negative Content-Length on put
...
Add computing actual object size during calculating hash on put.
Use this actual value to save in tree and cache
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-02 14:16:19 +00:00
81e860481d
[ #122 ] Fix linter warnings
...
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-05-31 19:43:03 +03:00
ee8cce662b
[ #74 ] service/tree: Add logger
...
Log error instead of failing when multiple unversioned nodes are found
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-04-26 16:39:57 +03:00
