dkirillov/frostfs-s3-gw
1
0
Fork 0
forked from TrueCloudLab/frostfs-s3-gw
Code Issues Pull requests Projects Releases Packages Wiki Activity
1151 commits 15 branches 31 tags 22 MiB
Commit graph

593 commits

Author SHA1 Message Date
Alex Vanin
3fbad97ae4 [#1] Rename files with mentions of previous project 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-05-19 07:30:04 +00:00
Artem Tataurov
e24bc3f2ce [#101] app: Refactor the default copies number setting 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-05-17 11:36:28 +03:00
Denis Kirillov
b366e75366 [#81] Use impersonate bearer token 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-05-03 17:22:52 +03:00
Artem Tataurov
e487ee5b7d [#70] Add arrays of copies numbers for location constraints 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-05-03 13:48:26 +03:00
Denis Kirillov
ee8cce662b [#74] service/tree: Add logger 
Log error instead of failing when multiple unversioned nodes are found

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-04-26 16:39:57 +03:00
Denis Kirillov
9e72fe1662 [#80] Refactor metrics, support dump descriptions 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-04-20 11:14:52 +03:00
Denis Kirillov
70ec5a0a5b [#83] Don't create extra delete marker 
We shouldn't create delete marker if:
1. object doesn't exist at all
2. last version is already a delete marker

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-04-19 17:56:11 +03:00
Artem Tataurov
a5c89b78bc [#79] Return X-Owner-Id in head-bucket response 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-04-17 10:12:25 +03:00
Artem Tataurov
fb99b26209 [#57] api: Clean up List and Name caches when object is missing in Tree service 2023-04-04 17:15:04 +03:00
Denis Kirillov
a025f2e9c5 [#59] tree: Make interface for tree service client 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-29 09:34:10 +03:00
Denis Kirillov
bd3164c57f [#68] Fix pre-commit issues 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-24 16:22:06 +03:00
Denis Kirillov
a0937126cb [#64] Support new system attributes 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-22 11:46:58 +03:00
Alex Vanin
ef556bd8ac [#60] Use session token to set eACL during Complete Multipart Upload 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-22 11:22:11 +03:00
Alex Vanin
5104683f68 [#60] Refactor start of periodic XML writer 
Reduce code duplication for error handling

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-22 11:02:16 +03:00
Alex Vanin
8151753eeb [#60] Use periodic white space XML writer in Complete Multipart Upload 
This mechanism is used by Amazon S3 to keep client's
connection alive while object is being constructed from
the upload parts.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-20 13:34:22 +03:00
Alex Vanin
2282c32822 [#60] Add *NoHeader functions 
Such functions should be used together with periodic white space
XML writer.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-20 12:07:36 +03:00
Alex Vanin
43685e03d9 [#60] Implement flusher on all http.ResponseWriters 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-20 12:04:34 +03:00
Alex Vanin
cf18158da4 [#60] Implement periodic white space XML writer 
Periodic white space XML writer sends XML header
and white spaces to the io.Writer.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-20 12:04:34 +03:00
Denis Kirillov
0af06c3bd9 [TrueCloudLab#40] Add param to configure xml decoder 
This parameter enables parsing xml body without
xmlns="http://s3.amazonaws.com/doc/2006-03-01/" attribute
for CompleteMultipartUpload requests

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-09 15:44:13 +03:00
Denis Kirillov
64e7356acc [TrueCloudLab#32] Add custom policy unmarshaler 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-09 11:19:01 +00:00
Denis Kirillov
32bf915502 [TrueCloudLab#37] Limit number of objects to delete 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-09 11:16:22 +00:00
Alex Vanin
813aa2f173 Rename package name 
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-07 17:38:08 +03:00
Denis Kirillov
740acadd37 [TrueCloudLab#36] Fix cors object payload 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-02-20 19:22:48 +03:00
Alex Vanin
aadefd98b6 [TrueCloudLab#25] Process allow and deny lists of zones in bucket head requests 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-02-17 13:47:48 +03:00
Alex Vanin
8ac630ee71 [TrueCloudLab#25] Add zone data to BucketInfo 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-02-17 13:47:48 +03:00
Denis Kirillov
9f823bd65a [TrueCloudLab#26] Add billing metrics to separate registry 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-02-10 10:55:38 +03:00
Denis Kirillov
9dcacc230e [TrueCloudLab#23] Return error on unknown LocationConstraint 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-02-08 15:04:12 +03:00
Artem Tataurov
5ee4bf80ae [#18] Return container name in head-bucket response 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-02-01 14:25:21 +03:00
Denis Kirillov
e278ab9362 [TrueCloudLab#5] Refactor middlewares 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-01-25 11:02:23 +03:00
Denis Kirillov
c5570e661d [TrueCloudLab#5] Add traffic metrics per user 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-01-25 11:02:23 +03:00
Denis Kirillov
fc5c09c084 [TrueCloudLab#5] Request metrics per user 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-01-25 11:02:23 +03:00
Artem Tataurov
8e14ccad9f [#754] Remove duplicating CID/OID logging calls 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
(cherry picked from commit 8c178fc3c7265fecc897a2526419bb748847c668)
 2022-12-28 17:06:32 +03:00
Artem Tataurov
af7fbd6f32 [#754] Add CID and OID to logs 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2022-12-28 17:06:32 +03:00
Denis Kirillov
000d9ed42a [#2] Update issue links 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2022-12-21 11:17:45 +03:00
Denis Kirillov
388482e230 [#2] Rename internals 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2022-12-21 11:17:45 +03:00
Denis Kirillov
765fd28a49 [#2] Update build 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2022-12-20 18:24:30 +03:00
Alex Vanin
96dff367db [#1] Build S3 Gateway with FrostFS dependencies 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2022-12-15 12:43:52 +03:00
Denis Kirillov
dd4f66712c [#742] Add multiple listeners 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-24 17:48:26 +03:00
Denis Kirillov
d2587b21af [#747] Reload policies on SIGHUP 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-18 13:14:25 +03:00
Angira Kekteeva
f5fe9a9b4b [#741] Fix data race 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-11-16 18:19:31 +03:00
Denis Kirillov
8f1bbb7dc1 [#749] Update SDK 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-15 10:16:41 +03:00
Denis Kirillov
094eb12578 [#726] Use client time on regular requests 
Use `X-Amz-Date` header as `now` when
* compute expiration epoch
* set Timestamp for object and container
* forming locks
* send notifications

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-14 14:53:55 +03:00
Denis Kirillov
d47840f137 [#568] Add configuration for region to policy map 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-11 13:12:49 +03:00
Denis Kirillov
2886ac161c [#740] Fix forming policy by ast 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-10 15:33:03 +03:00
Denis Kirillov
16ad0cadcb [#733] Fix container name 
User system name instead of regular.
Set name as CID if name attribute is missed

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-26 17:52:02 +03:00
Denis Kirillov
18a6aca4b4 [#617] api/handler: Simplify tests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-25 19:04:37 +03:00
Angira Kekteeva
712740b668 [#728] Remove redundant log message 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-10-25 19:04:11 +03:00
Angira Kekteeva
bbd7ba670a [#728] Add bucket and object to method logs 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-10-25 19:04:11 +03:00
Angira Kekteeva
da61fd8bcb [#728] Add error message to AbortMultipartUpload 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-10-25 19:04:11 +03:00
Denis Kirillov
0d0ecf5b10 [#650] Drop unused object cache methods 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-25 16:00:21 +03:00
Denis Kirillov
821df3d648 [#669] Optimize getNodeVersion for tags operation 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-20 17:56:09 +03:00
Denis Kirillov
90eb4f0188 [#713] Add check access control for system caches 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-11 15:26:42 +03:00
Denis Kirillov
cb55d36063 [#713] Update tests 
Add bearer token to test context

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-11 15:26:42 +03:00
Denis Kirillov
3d31c2ab4a [#713] Fix cache access 
User can access only keys that he puts into caches

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-11 15:26:42 +03:00
Denis Kirillov
c6098e53f6 [#719] Add tests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-11 11:13:20 +03:00
Denis Kirillov
b3d96e6460 [#719] Improve lock error handling 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-11 11:13:20 +03:00
Denis Kirillov
2a41929be3 [#702] Reload resolvers and TLS certs on SIGHUP 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-28 15:07:31 +03:00
Denis Kirillov
be4600988e [#709] Add expiration attribute for legal hold 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-26 16:51:53 +03:00
Denis Kirillov
3d08562843 [#505] Handle access denied from tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-13 18:05:16 +03:00
Denis Kirillov
80beedf13e [#504] Use bktInfo in tree service to check owner 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-13 18:05:16 +03:00
Denis Kirillov
4082cd6b54 [#606] Keep eacl records order on conflict 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-08 12:21:47 +03:00
Denis Kirillov
ffd259671a [#704] Fix nil pointer dereference on decReader 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-07 15:51:44 +03:00
Denis Kirillov
9cd4ef1ac4 [#657] Replace FileName with FilePath attribute 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-07 15:50:43 +03:00
Denis Kirillov
6108f8d929 [#624] Update tree service revision 
Remove extra request to tree service to know parentID

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-07 12:17:51 +03:00
Angira Kekteeva
3e4fc52428 [#674] Add AccessKeyID filter 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-09-06 13:47:20 +03:00
Alex Vanin
272c485706 [#669] Use object cache in PutLockInfo optimization 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2022-08-31 13:19:00 +03:00
Alex Vanin
5ffe28a42f [#669] Optimize PutLockInfo 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2022-08-31 13:19:00 +03:00
Denis Kirillov
96c7b79d1c [#683] Forbid copy to itself in unversioned bucket 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-30 13:31:48 +03:00
Denis Kirillov
44d5878a80 [#656] Read full body on upload error 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-30 09:39:20 +03:00
Alex Vanin
9590b39333 [#682] Return added node id in AddVersion 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2022-08-29 18:24:00 +03:00
Denis Kirillov
5c7a509cc9 [#684] Fix continuation token for directories 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-29 16:10:01 +03:00
Denis Kirillov
e38bdae07a [#676] Fix object acl 
Put object acl always add rules to specific version of object.
Get object acl consider READ rights as FULL_CONTROL
because WRITE cannot be applied to object

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-29 13:20:30 +03:00
Denis Kirillov
163038b37d [#672] Fix handling X-Amz-Copy-Source header 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-25 17:40:15 +03:00
Denis Kirillov
fdc926e70b [#661] Fix updating cache when request fails 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-25 12:55:49 +03:00
Denis Kirillov
1a580b6fa4 [#666] Support tagging directive for CopyObject 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-25 12:55:33 +03:00
Denis Kirillov
f013282290 [#660] Fix panic on invalid versioning status 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-25 11:50:44 +03:00
Denis Kirillov
24e1f7e1dd [#545] Drop object search 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-25 11:49:08 +03:00
Denis Kirillov
c3ad6d2faf [#637] Add header to override CopiesNumber 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-19 10:31:26 +03:00
Angira Kekteeva
3b343d1bd5 [#634] Fix tests 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-16 17:49:15 +03:00
Angira Kekteeva
5307211398 [#634] Add CopiesNumber in NeoFS requests 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-16 17:49:15 +03:00
Denis Kirillov
6aab3936d9 [#625] Limit listing generator with maxKeys 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-16 13:47:37 +03:00
Denis Kirillov
3824151699 [#616] Reduce number of requests during listing 
Check if object is directory before request to NeoFS

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-16 13:47:37 +03:00
Denis Kirillov
107d8a9033 [#595] Fix typos and rewording 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-13 10:26:00 +03:00
Denis Kirillov
7ab473a688 [#595] Simplify encryption.Params struct 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-13 10:26:00 +03:00
Denis Kirillov
94a6a55919 [#595] Move decrypter to separate package 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-13 10:26:00 +03:00
Denis Kirillov
d824db7f69 [#595] Allow SSE-C only with TLS 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-13 10:26:00 +03:00
Denis Kirillov
87c05eb514 [#595] Add SSE with customer key 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-13 10:26:00 +03:00
Angira Kekteeva
1987f271f9 [#577] Remove usages of deprecated io/ioutil 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva
f66e4a77cd [#577] Add IsDeleteMarker() to NodeVersion 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva
2a933f48be [#577] Remove redundant set of isLatest 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva
3188fd8eb7 [#577] Fix delete from tagging cache 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva
8031baddb6 [#577] Fix cache tests 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva
245e64900d [#577] Replace ObjInfo with ExtObjInfo in cache 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva
dc77ad4863 [#577] Fix tests 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva
93e0cae743 [#577] Fix version in GetObjectAttributes 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva
659e7bfc97 [#577] Fix version in GetObject and HeadObject 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva
dfd734b9ec [#577] Separate GetObjectInfo and GetExtendedObjectInfo 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva
37c46e926e [#577] Add Version() to ExtendedObjectInfo 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva
3ac3f1cc9d [#577] Rename objectInfo.Version() to VersionID() 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva
eaf76afe50 [#577] Remove ObjectVersionInfo 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00