0644067496
[ #412 ] Store creation epoch in tree service
...
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-08-21 10:38:35 +03:00
481520705a
[ #42 ] Support expiration lifecycle
...
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-08-21 10:38:35 +03:00
20719bd85c
[ #456 ] PostObject: check object key for emptiness
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-08-13 09:27:26 +03:00
4f27e34974
[ #456 ] auth: Fix authentication for POST Object
...
During post object operation field AuthHeader in middleware.Box wasn't set
that led to panic
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-08-12 15:25:53 +03:00
3dc989d7fe
[ #451 ] Support Location in CompleteMultipart response
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-08-06 15:45:09 +03:00
69e77aecc9
[ #451 ] Don't skip [Next]PartNumberMarker in response
...
In ListParts response we should always keep PartNumberMarker (even if it's zero)
and NextPartNumberMarker (even if response isn't truncated)
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-08-06 15:44:56 +03:00
f5326b9f04
[ #437 ] tree: Support removing old split system nodes
...
It's need to fit user expectation on deleting CORS for example.
Previously after removing cors (that was uploaded in split manner)
we can still get some data (from other node)
because deletion worked only for latest node version.
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2024-07-26 06:54:10 +00:00
51c5c227c2
[ #31 ] Add force bucket delete flag
...
Signed-off-by: Pavel Pogodaev <p.pogodaev@yadro.com>
2024-07-25 14:04:54 +03:00
6cb0026007
[ #427 ] layer: Split FrostFS ReadObject to separate methods
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-07-23 16:53:59 +03:00
971006a28c
[ #422 ] Support separate container for CORS
...
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-07-23 12:33:29 +00:00
3213dd7236
[ #404 ] Add processing of encoding-type in versions listing
...
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-07-23 12:10:35 +00:00
977a20760b
[ #430 ] Delete all split version at once
...
Previously after split we can get two `null` versioned object with the same key
and deleting such key removes only one node/object.
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-07-19 11:26:51 +03:00
c0011ebb8d
[ #430 ] tree: Fix multipart having system name
...
Previously if multipart key has the same name as some system node
(e.g. bucket-settings, bucket-cors etc.) it shadows real system node
and bucket started to be unversioned again for example.
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-07-19 11:24:50 +03:00
456319d2f1
[ #430 ] Fix split tree
...
Update tree service to fix split tree problem.
Tree intermediate nodes can be duplicated, so we must handle this.
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-07-19 11:24:46 +03:00
f86b82351a
[ #398 ] Fix parameter parsing in bucket retryer
...
RetryStrategyExponential should use jitter backoff
instead of constant delay function
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2024-07-03 13:42:24 +03:00
465eaa816a
[ #372 ] Drop [e]ACL related code
...
Always consider buckets as APE compatible
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-07-01 16:58:44 +03:00
77f8bdac58
[ #372 ] Drop kludge.acl_enabled flag
...
Now only APE container can be created using s3-gw
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-07-01 16:26:19 +03:00
91541a432d
[ #411 ] Check uniqueness in DeleteMultipleObjects
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-06-26 16:39:06 +03:00
943b30d9f4
[ #411 ] Don't check object tags on deletion
...
By specification https://docs.aws.amazon.com/AmazonS3/latest/userguide/tagging-and-policies.html
we shouldn't check object tags on PUT and DELETE
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-06-26 16:38:56 +03:00
414f3943e2
[ #410 ] Drop layer.Client interface
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-06-25 15:57:55 +03:00
9432782ce6
[ #401 ] Drop notifications
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-06-25 15:49:37 +03:00
280d11c794
[ #407 ] Don't set full_control for bucket owner
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-06-19 10:55:24 +03:00
ed34b2cae4
[ #402 ] auth: Extend test coverage
...
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2024-06-14 10:06:00 +00:00
76f553d292
[ #403 ] Set resource tags into resource properties
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-06-13 11:12:40 +03:00
bb81afc14a
[ #398 ] Support retryer
...
Add two strategy for PutBucketSettings request retryer:
* exponential backoff (increasing up to `max_backoff` delays with jitter)
* constant backoff (always the same `max_backoff` delay between requests)
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-06-06 13:02:17 +00:00
e25dc90c20
[ #399 ] Add OPTIONS method for object operations
...
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-06-04 12:59:45 +00:00
b5fae316cf
[ #396 ] Add user to response
...
Signed-off-by: Pavel Pogodaev <p.pogodaev@yadro.com>
2024-06-04 09:37:55 +00:00
9152b084ec
[ #387 ] Fix typo
...
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2024-05-22 15:06:02 +00:00
21dbe3ea8e
[ #387 ] api: Add tests for middleware
...
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2024-05-22 15:06:02 +00:00
f4d174e740
[ #387 ] middleware: Extend test coverage
...
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2024-05-22 15:06:02 +00:00
8a758293b9
[ #387 ] middleware: Delete unused code
...
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2024-05-22 15:06:02 +00:00
fb521c7ac6
[ #367 ] policy: Set IAM-MFA property to false by default
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-05-22 12:05:42 +03:00
87b9e97a80
[ #354 ] Do not proceed on bucket remove error
...
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2024-05-17 20:38:39 +03:00
d62d8f3874
[ #385 ] Support the renaming of ObjectRequest and ObjectContainer
...
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2024-05-14 16:51:36 +03:00
6bf6a3b1a3
[ #362 ] Check user and groups during policy check
...
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2024-05-08 15:25:14 +03:00
c43ef040dc
[ #382 ] Fix request type determination
...
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-05-07 15:17:22 +03:00
2ab655b909
[ #380 ] Add test for credentials versioning
...
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-05-03 07:24:13 +00:00
db05021786
[ #379 ] Add Iana CharsetReader for Oracle integration
...
Signed-off-by: Pavel Pogodaev <p.pogodaev@yadro.com>
2024-04-25 17:44:38 +03:00
034396d554
[ #377 ] Add check of Source IP
...
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-04-22 15:29:18 +03:00
3c436d8de9
[ #365 ] Include iam user tags in query
...
Signed-off-by: Pavel Pogodaev <p.pogodaev@yadro.com>
2024-04-22 10:47:43 +03:00
45f77de8c8
[ #371 ] Add custom Source IP header configuration
...
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-04-22 07:42:45 +00:00
e22ff52165
[ #367 ] Add check of AccessBox attributes
...
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-04-19 06:25:26 +00:00
5315f7b733
[ #269 ] Create frostfsid wrapper with cache
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-04-18 09:32:30 +03:00
fec3b3f31e
[ #269 ] Add frostfsid cache configuration
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-04-17 12:11:23 +03:00
3ff027587c
[ #357 ] Add check of request and resource tags
...
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-04-17 07:06:58 +00:00
8307c73fef
[ #364 ] Fix removing combined object
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-04-12 14:56:38 +03:00
d8889fca56
[ #340 ] Fix encode object acl
...
In the process of encode the acl of an object,
we use a map. As a result, when traversing the
map, we can get a different sequence of permissions
each time. Therefore, a list is used instead of a map.
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2024-04-11 09:28:30 +00:00
61ff4702a2
[ #360 ] Reuse single target during policy check
...
Policy engine library is able to manage multiple
targets and resolve different status results.
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2024-04-10 17:56:47 +03:00
6da1acc554
[ #360 ] Use 'c' prefix for bucket policies instead of 'n'
...
With 'c' prefix, acl chains become shorter, thus gateway
receives shorter results and avoids sessions to neo-go.
There is still issue with many IAM rules.
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2024-04-10 17:56:47 +03:00
9c012d0a66
[ #355 ] Remove policies when delete bucket
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-04-09 15:49:46 +00:00
37d05dcefd
[ #353 ] Add check of listing parameters and versionID
...
Add properties in policy check:
* s3:delimiter
* s3:prefix
* s3:max-keys
* s3:VersionId
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-04-08 17:57:55 +03:00
8669bf6b50
[ #346 ] acl: Update APE and fix using
...
* Remove native policy when remove bucket policy
* Allow policies that contain only s3 compatible statements
(now deny rules cannot be converted to native rules)
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-04-02 12:43:04 +00:00
fbe7a784e8
[ #301 ] Support GetBucketPolicyStatus
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-03-28 09:13:25 +03:00
80c7b73eb9
[ #306 ] In APE buckets forbid canned acl except private
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-03-19 16:57:26 +03:00
62cc5a04a7
[ #328 ] Log error on failed response writing
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-03-15 11:02:26 +03:00
4ee3648183
[ #328 ] Log invalid lock enabled header
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-03-04 15:09:51 +03:00
ee48d1dc85
[ #325 ] Log error on failed request id generation
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-03-04 09:49:41 +00:00
f958eef2b3
[ #325 ] Use default empty data.LockInfo in get/head in case of error
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-03-04 09:49:41 +00:00
81b44ab3d3
[ #325 ] Fix mutex usage in controller
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-03-04 09:49:41 +00:00
8050ca2d51
[ #306 ] Use session token for container read operations
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-03-01 18:14:33 +03:00
c12e264697
[ #306 ] Simplify cid resolver for metrics
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-03-01 17:46:16 +03:00
e9f38a49e4
[ #306 ] Fix forming key for bucket cache
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-03-01 16:09:40 +03:00
fabb4134bc
[ #318 ] Use log msg from constants
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-29 17:30:28 +03:00
e1ee36b979
[ #318 ] Fix tests
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-29 17:30:28 +03:00
937367caaf
[ #318 ] Fix panic on invalid multipart form
...
Previously, simple 'curl -X POST http://localhost:8084/test ' leads to panic because of wrong handle matching
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-29 17:30:28 +03:00
7b86bac6ee
[ #318 ] Log unmatched requests
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-29 17:30:28 +03:00
529ec7e0b9
[ #318 ] Don't log empty bucket/name
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-29 17:30:28 +03:00
4741e74210
[ #318 ] Log successfully authenticated accessKeyIDs
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-29 17:30:28 +03:00
f1470bab4a
[ #318 ] auth: Add context for logged errors
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-29 17:30:28 +03:00
6e5bcaef97
[ #318 ] Log policy request checking
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-29 17:30:28 +03:00
1522db05c5
[ #318 ] Log namespace for requests
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-29 17:30:28 +03:00
31da31862a
[ #300 ] Update error logging in DeleteMultipleObjects
...
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-02-29 14:24:32 +00:00
7de1ffdbe9
[ #306 ] Fix billing tests
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 18:00:27 +03:00
3285a2e105
[ #306 ] policy: Change default access strategy
...
Use access strategy based on bucket type and/or config flags.
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:53:13 +03:00
56b50f2075
[ #306 ] Remove flag to disable policy contract
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
8f89f275bd
[ #306 ] Save bucket policy as native chain
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
ff15f9f28a
[ #306 ] Fix update settings for buckets without owner key in tree
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
c868af8a62
[ #306 ] Add flag to enable old ACL bucket creation
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
bac1b3fb2d
[ #306 ] Use zero basic acl to mark APE containers
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
c452d58ce2
[ #306 ] Reduce number of policy contract invocations
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
d9d12debc3
[ #306 ] Add tests
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
3d0d2032c6
[ #306 ] acl: Handle put/get acl for APE buckets
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
1f2cf0ed67
[ #306 ] Use APE instead of eACL on bucket creation
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
37be8851b3
[ #306 ] Simplify namespaces configuration
...
Resolve ns alias at the beginning of the request just once.
Keep in ns map only one default ns key.
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
c4c199defe
[ #306 ] Use s3 as chain id prefix to be consistent
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
2981a47e99
[ #321 ] Use correct owner id in billing metrics
...
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-02-28 14:52:44 +03:00
563c1d9bd7
[ #308 ] Fix linter issues
...
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2024-02-16 18:25:06 +03:00
924e87face
[ #305 ] Support checking if accessbox was removed
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-09 09:41:48 +03:00
c334adeb6d
[ #165 ] Sort nodes in ServiceClientMemory
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:53:12 +03:00
a74d498df2
[ #165 ] Return sort after HEAD in listing
...
We have to sort object after HEAD because we make request in different goroutines,
so the order is not deterministic.
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:15:08 +03:00
69297a4a38
[ #165 ] Delete object from tree in case of storage error
...
Extend storage node errors in case of which we continue deleting from tree
with 'object not found' error
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:15:08 +03:00
71d82d1cc8
[ #165 ] Fix lint issues
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:15:08 +03:00
fafe4af529
[ #165 ] Fix real object size in listing
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:15:03 +03:00
88f1acbdfc
[ #165 ] Cancel context in outdated list session
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
4e15452853
[ #165 ] Fix lint errors
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
da642a498a
[ #165 ] Listing fix data race
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
2d7973b3f1
[ #165 ] Refactor list versions
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
6d52f46012
[ #165 ] Fix v1 listing bookmark
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
0ae49eaab0
[ #165 ] Generalize allObjectListingParams
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
093de13f54
[ #165 ] Add stream listing tests
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
