TrueCloudLab/frostfs-s3-gw
15
3
Fork 19
Code Issues 30 Pull requests 7 Projects Releases 17 Packages Activity Actions
1200 commits 5 branches 41 tags 20 MiB
Commit graph

111 commits

Author SHA1 Message Date
Denis Kirillov
473239bf36 [#257] Add policy checker 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-12-06 17:47:51 +03:00
Marina Biryukova
93cf7c462b [#271] Add namespace label to billing metrics
All checks were successful
/ DCO (pull_request) Successful in 2m35s
Details
/ Vulncheck (pull_request) Successful in 3m3s
Details
/ Builds (1.20) (pull_request) Successful in 3m34s
Details
/ Builds (1.21) (pull_request) Successful in 2m20s
Details
/ Lint (pull_request) Successful in 5m27s
Details
/ Tests (1.20) (pull_request) Successful in 3m25s
Details
/ Tests (1.21) (pull_request) Successful in 3m12s
Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-12-04 14:54:40 +03:00
Denis Kirillov
42862fd69e [#258] Support policy management in control svc
All checks were successful
/ Vulncheck (pull_request) Successful in 3m20s
Details
/ DCO (pull_request) Successful in 3m56s
Details
/ Builds (1.20) (pull_request) Successful in 4m13s
Details
/ Builds (1.21) (pull_request) Successful in 4m9s
Details
/ Lint (pull_request) Successful in 3m9s
Details
/ Tests (1.20) (pull_request) Successful in 4m14s
Details
/ Tests (1.21) (pull_request) Successful in 3m59s
Details 
Add PutPolicies, RemovePolicies, GetPolicy, ListPolicies methods

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-12-01 15:47:12 +03:00
Denis Kirillov
c7a65bd075 [#258] Add control service 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-12-01 14:17:06 +03:00
Denis Kirillov
28c6bb4cb8 [#266] Support per namespace placement policies configuration
All checks were successful
/ DCO (pull_request) Successful in 1m34s
Details
/ Builds (1.20) (pull_request) Successful in 3m15s
Details
/ Builds (1.21) (pull_request) Successful in 2m55s
Details
/ Vulncheck (pull_request) Successful in 2m51s
Details
/ Lint (pull_request) Successful in 5m12s
Details
/ Tests (1.20) (pull_request) Successful in 2m57s
Details
/ Tests (1.21) (pull_request) Successful in 2m48s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-11-28 16:12:42 +03:00
Denis Kirillov
ff1ec56d24 [#260] Use namespace as domain when create bucket
All checks were successful
/ DCO (pull_request) Successful in 1m26s
Details
/ Vulncheck (pull_request) Successful in 2m6s
Details
/ Builds (1.20) (pull_request) Successful in 2m20s
Details
/ Builds (1.21) (pull_request) Successful in 2m21s
Details
/ Lint (pull_request) Successful in 3m15s
Details
/ Tests (1.20) (pull_request) Successful in 2m28s
Details
/ Tests (1.21) (pull_request) Successful in 2m17s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-11-27 15:30:12 +03:00
Denis Kirillov
055cc6a22a [#260] Use namespace as domain when resolve bucket 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-11-23 11:00:11 +03:00
Denis Kirillov
6304d7bfda [#260] Support frostfsid validation 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-11-23 11:00:11 +03:00
Denis Kirillov
cf7254f8cd [#260] Refactor api/auth/center.go 
Move the Center interface to middleware package where it's used

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-11-23 11:00:09 +03:00
Marina Biryukova
890a8ed237 [#227] Add versionID header after complete multipart 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-10-31 14:07:08 +00:00
Denis Kirillov
122af0b5a7 [#220] Support configuring web server timeout params 
Set IdleTimeout and ReadHeaderTimeout to `30s`.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-31 13:48:08 +00:00
Marina Biryukova
25bb581fee [#205] Add md5 checksum in header
All checks were successful
/ Vulncheck (pull_request) Successful in 1m50s
Details
/ DCO (pull_request) Successful in 3m29s
Details
/ Lint (pull_request) Successful in 3m37s
Details
/ Tests (1.20) (pull_request) Successful in 2m9s
Details
/ Tests (1.21) (pull_request) Successful in 2m5s
Details
/ Builds (1.20) (pull_request) Successful in 5m47s
Details
/ Builds (1.21) (pull_request) Successful in 1m29s
Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-10-25 11:04:19 +03:00
Marina Biryukova
298662df9d [#221] Expand xmlns field ignore
All checks were successful
/ Vulncheck (pull_request) Successful in 1m38s
Details
/ Lint (pull_request) Successful in 2m49s
Details
/ Tests (1.20) (pull_request) Successful in 1m56s
Details
/ Tests (1.21) (pull_request) Successful in 1m44s
Details
/ DCO (pull_request) Successful in 3m56s
Details
/ Builds (1.20) (pull_request) Successful in 6m25s
Details
/ Builds (1.21) (pull_request) Successful in 1m36s
Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-10-13 16:21:13 +03:00
Denis Kirillov
65412ce1d3 [#197] Configure buffer max size for PUT 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-11 12:32:48 +00:00
Marina Biryukova
b8c93ed391 [#172] Convert handler config to interface 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-10-04 11:01:27 +00:00
Marina Biryukova
69227b4845 [#199] Add metrics for HTTP endpoint status 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-09-05 13:30:27 +00:00
Roman Loginov
c66c09765d [#196] Support soft memory limit setting 
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-09-05 13:13:56 +00:00
Denis Kirillov
b5fce5c8d2 [#168] Skip only invalid policies and copies instead of ignoring all of them
All checks were successful
/ DCO (pull_request) Successful in 2m48s
Details
/ Vulncheck (pull_request) Successful in 3m30s
Details
/ Builds (1.19) (pull_request) Successful in 3m39s
Details
/ Builds (1.20) (pull_request) Successful in 3m28s
Details
/ Lint (pull_request) Successful in 4m14s
Details
/ Tests (1.19) (pull_request) Successful in 3m3s
Details
/ Tests (1.20) (pull_request) Successful in 2m45s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-25 12:05:30 +03:00
Roman Loginov
8efcc957ea [#96] Move log messages to constants
All checks were successful
/ DCO (pull_request) Successful in 1m35s
Details
/ Builds (1.19) (pull_request) Successful in 2m14s
Details
/ Builds (1.20) (pull_request) Successful in 2m9s
Details
/ Vulncheck (pull_request) Successful in 5m39s
Details
/ Lint (pull_request) Successful in 2m49s
Details
/ Tests (1.19) (pull_request) Successful in 7m34s
Details
/ Tests (1.20) (pull_request) Successful in 1m44s
Details 
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-08-23 18:32:31 +03:00
Denis Kirillov
6b1f365e65 [#192] Support client cut 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-23 06:26:55 +00:00
Roman Loginov
6a9d3261a7 [#117] Refactor fetch/parse config parameters functions
All checks were successful
/ Vulncheck (pull_request) Successful in 2m0s
Details
/ Lint (pull_request) Successful in 2m53s
Details
/ Tests (1.19) (pull_request) Successful in 2m14s
Details
/ Tests (1.20) (pull_request) Successful in 1m51s
Details
/ DCO (pull_request) Successful in 4m0s
Details
/ Builds (1.19) (pull_request) Successful in 6m30s
Details
/ Builds (1.20) (pull_request) Successful in 2m7s
Details 
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-08-22 08:05:26 +00:00
Denis Kirillov
18878b66d3 [#175] Use gate owner as object owner
Some checks failed
/ DCO (pull_request) Failing after 3s
Details
/ Vulncheck (pull_request) Failing after 2s
Details
/ Builds (1.19) (pull_request) Failing after 3s
Details
/ Builds (1.20) (pull_request) Successful in 7m47s
Details
/ Lint (pull_request) Failing after 4s
Details
/ Tests (1.19) (pull_request) Successful in 8m8s
Details
/ Tests (1.20) (pull_request) Failing after 3s
Details 
This is required because node check session token owner
TrueCloudLab/frostfs-node#528
For client cut TrueCloudLab/frostfs-sdk-go#114
such owner will be gate owner

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-08 12:22:33 +03:00
Denis Kirillov
b59aa06637 [#146] Add kludge.bypass_content_encoding_check_in_chunks flag
Some checks failed
/ Builds (1.19) (pull_request) Successful in 3m0s
Details
/ Builds (1.20) (pull_request) Successful in 2m50s
Details
/ DCO (pull_request) Failing after 1m9s
Details
/ Vulncheck (pull_request) Successful in 1m20s
Details
/ Lint (pull_request) Successful in 7m50s
Details
/ Tests (1.19) (pull_request) Successful in 3m1s
Details
/ Tests (1.20) (pull_request) Successful in 3m21s
Details 
Flag allows to skip checking `Content-Encoding` for `aws-chunked` value

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-18 14:49:52 +03:00
Denis Kirillov
fc90981c03 [#149] Update inner imports after moving middlewares 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-11 17:25:09 +03:00
Denis Kirillov
7be70243f7 [#166] Update sdk to support grpc schemes in tree pool 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-11 17:25:03 +03:00
Alex Vanin
751d66bde0 [#140] Use gRPC interceptor from observability package 
Previous SDK implementation had implicit gRPC interceptor
for tracing. Now pool constructors allow any dial options,
so gateway should manually pass tracing interceptors from
observability package.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-06-14 15:46:43 +00:00
Denis Kirillov
9dabaf6ecd [#133] Use tree pool from SDK 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-09 09:34:04 +03:00
Pavel Pogodaev
4e1fd9589b [#84] add tracing support
All checks were successful
Builds (1.20)
Details
Builds (1.19)
Details
DCO
Details
Tests (1.19)
Details
Tests (1.20)
Details
Lint
Details 
Signed-off-by: Pavel Pogodaev <p.pogodaev@yadro.com>
 2023-06-06 10:25:27 +03:00
Artem Tataurov
9f186d9aba [#104] app: Reload copies numbers on SIGHUP 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-05-23 13:19:58 +03:00
Artem Tataurov
e24bc3f2ce [#101] app: Refactor the default copies number setting 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-05-17 11:36:28 +03:00
Artem Tataurov
e487ee5b7d [#70] Add arrays of copies numbers for location constraints 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-05-03 13:48:26 +03:00
Denis Kirillov
ee8cce662b [#74] service/tree: Add logger 
Log error instead of failing when multiple unversioned nodes are found

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-04-26 16:39:57 +03:00
Denis Kirillov
ca8791a5fd [#74] Support multiple tree endpoints 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-04-26 16:39:57 +03:00
Denis Kirillov
9e72fe1662 [#80] Refactor metrics, support dump descriptions 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-04-20 11:14:52 +03:00
Denis Kirillov
4a8c382491 [#91] Update values for health metric 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-04-18 10:25:11 +03:00
Denis Kirillov
01afa1cae4 [#75] Make grpc tree client implementation internal 
Since we have pkg 'internal/frostfs/services/tree' that is downloading
during build we cannot export any package that is depended on it.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-04-03 14:04:35 +03:00
Denis Kirillov
a025f2e9c5 [#59] tree: Make interface for tree service client 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-29 09:34:10 +03:00
Alex Vanin
8151753eeb [#60] Use periodic white space XML writer in Complete Multipart Upload 
This mechanism is used by Amazon S3 to keep client's
connection alive while object is being constructed from
the upload parts.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-20 13:34:22 +03:00
Denis Kirillov
0af06c3bd9 [TrueCloudLab#40] Add param to configure xml decoder 
This parameter enables parsing xml body without
xmlns="http://s3.amazonaws.com/doc/2006-03-01/" attribute
for CompleteMultipartUpload requests

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-09 15:44:13 +03:00
Alex Vanin
813aa2f173 Rename package name 
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-07 17:38:08 +03:00
Alex Vanin
aadefd98b6 [TrueCloudLab#25] Process allow and deny lists of zones in bucket head requests 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-02-17 13:47:48 +03:00
Denis Kirillov
9f823bd65a [TrueCloudLab#26] Add billing metrics to separate registry 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-02-10 10:55:38 +03:00
Denis Kirillov
1ce8b8a30d [TrueCloudLab#21] Support multiple configs 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-02-07 12:26:40 +03:00
Denis Kirillov
533b12d8bb [TrueCloudLab#12] Require only one healthy server 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-01-24 17:56:18 +03:00
Denis Kirillov
cafe079072 [TrueCloudLab#13] Update frostfs SDK 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-01-23 13:59:32 +03:00
Denis Kirillov
388482e230 [#2] Rename internals 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2022-12-21 11:17:45 +03:00
Denis Kirillov
765fd28a49 [#2] Update build 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2022-12-20 18:24:30 +03:00
Alex Vanin
96dff367db [#1] Build S3 Gateway with FrostFS dependencies 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2022-12-15 12:43:52 +03:00
Denis Kirillov
dd4f66712c [#742] Add multiple listeners 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-24 17:48:26 +03:00
Denis Kirillov
d2587b21af [#747] Reload policies on SIGHUP 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-18 13:14:25 +03:00