ognev/frostfs-s3-gw
1
0
Fork 0
forked from TrueCloudLab/frostfs-s3-gw
Code Pull requests Activity
1361 commits 6 branches 13 tags 18 MiB
Commit graph

169 commits

Author SHA1 Message Date
Marina Biryukova
69227b4845 [#199] Add metrics for HTTP endpoint status 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-09-05 13:30:27 +00:00
Roman Loginov
c66c09765d [#196] Support soft memory limit setting 
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-09-05 13:13:56 +00:00
Marina Biryukova
2fc328a6d2 [#195] Add log constants linter 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-08-28 12:58:44 +03:00
Denis Kirillov
b5fce5c8d2 [#168] Skip only invalid policies and copies instead of ignoring all of them 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-25 12:05:30 +03:00
Roman Loginov
8efcc957ea [#96] Move log messages to constants 
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-08-23 18:32:31 +03:00
Denis Kirillov
6b1f365e65 [#192] Support client cut 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-23 06:26:55 +00:00
Roman Loginov
6a9d3261a7 [#117] Refactor fetch/parse config parameters functions 
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-08-22 08:05:26 +00:00
Denis Kirillov
18878b66d3 [#175] Use gate owner as object owner 
This is required because node check session token owner
TrueCloudLab/frostfs-node#528
For client cut TrueCloudLab/frostfs-sdk-go#114
such owner will be gate owner

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-08 12:22:33 +03:00
Denis Kirillov
b59aa06637 [#146] Add kludge.bypass_content_encoding_check_in_chunks flag 
Flag allows to skip checking `Content-Encoding` for `aws-chunked` value

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-18 14:49:52 +03:00
Denis Kirillov
fc90981c03 [#149] Update inner imports after moving middlewares 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-11 17:25:09 +03:00
Denis Kirillov
7be70243f7 [#166] Update sdk to support grpc schemes in tree pool 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-11 17:25:03 +03:00
Alex Vanin
751d66bde0 [#140] Use gRPC interceptor from observability package 
Previous SDK implementation had implicit gRPC interceptor
for tracing. Now pool constructors allow any dial options,
so gateway should manually pass tracing interceptors from
observability package.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-06-14 15:46:43 +00:00
Denis Kirillov
9dabaf6ecd [#133] Use tree pool from SDK 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-09 09:34:04 +03:00
Pavel Pogodaev
4e1fd9589b [#84] add tracing support 
Signed-off-by: Pavel Pogodaev <p.pogodaev@yadro.com>
 2023-06-06 10:25:27 +03:00
Alex Vanin
81e860481d [#122] Fix linter warnings 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-05-31 19:43:03 +03:00
Artem Tataurov
9f186d9aba [#104] app: Reload copies numbers on SIGHUP 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-05-23 13:19:58 +03:00
Artem Tataurov
e24bc3f2ce [#101] app: Refactor the default copies number setting 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-05-17 11:36:28 +03:00
Artem Tataurov
e487ee5b7d [#70] Add arrays of copies numbers for location constraints 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-05-03 13:48:26 +03:00
Denis Kirillov
ee8cce662b [#74] service/tree: Add logger 
Log error instead of failing when multiple unversioned nodes are found

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-04-26 16:39:57 +03:00
Denis Kirillov
ca8791a5fd [#74] Support multiple tree endpoints 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-04-26 16:39:57 +03:00
Denis Kirillov
9e72fe1662 [#80] Refactor metrics, support dump descriptions 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-04-20 11:14:52 +03:00
Denis Kirillov
4a8c382491 [#91] Update values for health metric 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-04-18 10:25:11 +03:00
Denis Kirillov
01afa1cae4 [#75] Make grpc tree client implementation internal 
Since we have pkg 'internal/frostfs/services/tree' that is downloading
during build we cannot export any package that is depended on it.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-04-03 14:04:35 +03:00
Denis Kirillov
a025f2e9c5 [#59] tree: Make interface for tree service client 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-29 09:34:10 +03:00
Alex Vanin
8151753eeb [#60] Use periodic white space XML writer in Complete Multipart Upload 
This mechanism is used by Amazon S3 to keep client's
connection alive while object is being constructed from
the upload parts.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-20 13:34:22 +03:00
Denis Kirillov
0af06c3bd9 [TrueCloudLab#40] Add param to configure xml decoder 
This parameter enables parsing xml body without
xmlns="http://s3.amazonaws.com/doc/2006-03-01/" attribute
for CompleteMultipartUpload requests

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-09 15:44:13 +03:00
Alex Vanin
813aa2f173 Rename package name 
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-07 17:38:08 +03:00
Alex Vanin
aadefd98b6 [TrueCloudLab#25] Process allow and deny lists of zones in bucket head requests 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-02-17 13:47:48 +03:00
Denis Kirillov
9f823bd65a [TrueCloudLab#26] Add billing metrics to separate registry 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-02-10 10:55:38 +03:00
Denis Kirillov
1ce8b8a30d [TrueCloudLab#21] Support multiple configs 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-02-07 12:26:40 +03:00
Denis Kirillov
533b12d8bb [TrueCloudLab#12] Require only one healthy server 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-01-24 17:56:18 +03:00
Denis Kirillov
cafe079072 [TrueCloudLab#13] Update frostfs SDK 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-01-23 13:59:32 +03:00
Denis Kirillov
388482e230 [#2] Rename internals 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2022-12-21 11:17:45 +03:00
Denis Kirillov
765fd28a49 [#2] Update build 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2022-12-20 18:24:30 +03:00
Alex Vanin
96dff367db [#1] Build S3 Gateway with FrostFS dependencies 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2022-12-15 12:43:52 +03:00
Denis Kirillov
5265afe213 [#742] Update docs 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-24 17:48:26 +03:00
Denis Kirillov
dd4f66712c [#742] Add multiple listeners 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-24 17:48:26 +03:00
Denis Kirillov
d2587b21af [#747] Reload policies on SIGHUP 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-18 13:14:25 +03:00
Denis Kirillov
d1825f97fb [#750] Update SDK to support timeout for stream 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-15 18:10:28 +03:00
Denis Kirillov
d47840f137 [#568] Add configuration for region to policy map 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-11 13:12:49 +03:00
Denis Kirillov
f59bf91a67 [#699] neofs/tree: Add healthcheck 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-15 13:59:54 +03:00
Denis Kirillov
3d31c2ab4a [#713] Fix cache access 
User can access only keys that he puts into caches

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-11 15:26:42 +03:00
Denis Kirillov
a18db163f2 [#716] Fix start TLS server 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-03 12:18:00 +03:00
Denis Kirillov
3a7ed8220e [#702] Refactor app initialization 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-28 15:07:31 +03:00
Denis Kirillov
1fa28dfacd [#702] Init app settings 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-28 15:07:31 +03:00
Denis Kirillov
2a41929be3 [#702] Reload resolvers and TLS certs on SIGHUP 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-28 15:07:31 +03:00
Denis Kirillov
42893ec046 [#702] Metrics reload on SIGHUP 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-28 15:07:31 +03:00
Denis Kirillov
63275f7876 [#702] Reload config level on SIGHUP 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-28 15:07:31 +03:00
Denis Kirillov
7668d5fb2f [#667] Add docs for listen_domains config param 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-06 16:32:04 +03:00
Angira Kekteeva
3e4fc52428 [#674] Add AccessKeyID filter 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-09-06 13:47:20 +03:00
Denis Kirillov
628c0f42fe [#680] Use gauge instead of counter for metrics 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-25 11:00:59 +03:00
Angira Kekteeva
5307211398 [#634] Add CopiesNumber in NeoFS requests 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-16 17:49:15 +03:00
Denis Kirillov
d824db7f69 [#595] Allow SSE-C only with TLS 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-13 10:26:00 +03:00
Angira Kekteeva
2d58b25ea0 [#643] Fix linter issues 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-05 15:32:46 +03:00
Angira Kekteeva
7bd824f3fa [#607] Improve wallet path param 
Made it configurable in yaml

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-03 22:25:59 +03:00
Angira Kekteeva
57ce0f25bf [#607] Don't print env vars with empty values 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-03 22:25:59 +03:00
Denis Kirillov
08898f4fb4 [#615] Expose pool metrics 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-02 16:04:31 +03:00
Denis Kirillov
87f6681281 [#633] Add config param for pool error threshold 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-29 11:25:11 +03:00
Denis Kirillov
e5c1acf1e5 [#591] Sync metrics and pprof configuration 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-29 09:32:08 +03:00
Denis Kirillov
0ae762ef06 [#592] Change pool defaults params 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 17:18:37 +03:00
Denis Kirillov
27e3086cbf [#600] Return pointer from newStateMetrics 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 17:18:04 +03:00
Denis Kirillov
89d240d831 [#600] Add health metric 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 17:18:04 +03:00
Denis Kirillov
1e86413667 [#413] Add signature for tree requests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
9ed081db3e [#407] Add service tree client prototype 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
f5441eb6cb [#578] Unify version output 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-15 12:30:40 +03:00
Leonard Lyubich
f0749fd23e [#537] Upgrade NeoFS SDK Go with changed netmap package 
`PlacementPolicy` type now provides methods to work with QL-encoded
policies. System network parameters can be read using dedicated method
without iterating. Applications can work with `PlacementPolicy`
variables directly so there is no need to use pointers.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-17 08:28:59 +03:00
Denis Kirillov
300d4359d8 [#465] Handle cache cast failure 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-07 17:20:26 +03:00
Alex Vanin
9d3e6f75be [#391] Remove unused notification controller from app state 
Now controller is a part of handler.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-05-06 11:03:20 +03:00
Denis Kirillov
94caa2247e [#391] Refactor notifications 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-05-06 10:59:31 +03:00
Denis Kirillov
e3c16a32dd [#409] Update SDK 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-04-26 12:51:52 +04:00
Elizaveta Chichindaeva
bf38007692 [#405] English Check 
Signed-off-by: Elizaveta Chichindaeva <elizaveta@nspcc.ru>
 2022-04-22 14:01:40 +03:00
Angira Kekteeva
9f017b2bba [#396] Rename rpc-endpoint and resolve-order 
To make cli params more consistent
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-18 11:58:54 +03:00
Angira Kekteeva
8d9e473804 [#396] Rename params 
session-token -> session-tokens
rebalance_timer -> rebalance interval
request_timeout -> healthcheck_timeout

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-18 11:58:54 +03:00
Angira Kekteeva
0f3ae334e5 [#396] Remove unused parameters 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-18 11:58:54 +03:00
Denis Kirillov
7710de39ec [#365] Update SDK 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-04-08 17:12:52 +03:00
Leonard Lyubich
83fb697f80 [#380] cmd/s3-gw: Write app name and version once on startup 
It's redundant to write app name and version in each log message, a
single entry at the start is enough.

Make `App.Wait` to write app name as `neofs-s3-gw` in `application
started` message. Do not add `app_name` and `app_version` fields to the
logger returned by `newLogger`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-04-06 17:31:01 +03:00
Leonard Lyubich
01c721ee53 [#380] cmd/s3-gw: Refactor logger 
Do not import `logger` package from NeoFS SDK Go. Discard unusable
`name` and `version` configuration values from `app` section. Discard
all unusable onfiguration values from `logger` section except `level`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-04-06 17:31:01 +03:00
Denis Kirillov
c399cfbdda [#382] Refactor neofs related types 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-31 11:47:24 +03:00
Angira Kekteeva
83bb6fc020 [#384] Add check of Notificator interface value 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-03-30 10:29:28 +03:00
Denis Kirillov
46e4b28489 [#195] Add logger to notification controller 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
4c3c4b6bee [#195] Add Lock tick listening 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
32e83db064 [#195] Set Lock type to system lock objects 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
13080d6d96 [#195] Refactor TestNeoFS 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Leonard Lyubich
0cbe09d211 [#346] s3-gw: Reuse single neofs.NeoFS in application 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-04 00:14:30 +03:00
Leonard Lyubich
cd64f41ce8 [#346] *: Refactor communication with NeoFS at the protocol level 
Make `tokens`, `authmate` and `layer` packages to depend from locally
defined `NeoFS` interface of the virtual connection to NeoFS network.
Create internal `neofs` package and implement these interfaces through
`pool.Pool` there. Implement mediators between `NeoFS` interfaces and
`neofs.NeoFS` implementation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-04 00:14:30 +03:00
Leonard Lyubich
34a221c5c9 [#346] Upgrade NeoFS SDK Go library 
Core changes:
  - `object.ID` moved to new package `oid`;
  - `object.Address` moved to new package `address`;
  - `pool.Object` interface changes.

Additionally:
  - Set container owner in `Agent.IssueSecret`.
  - Remove no longer needed fields from `GetObjectParams`
  - `Length` and `Offset` are never assigned. These values
  are set in `Range` field.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-04 00:14:30 +03:00
Angira Kekteeva
e0c6544567 [#340] Move notification controller to layer 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-02-18 16:59:42 +03:00
Angira Kekteeva
3277293bb3 [#341] Add notifications controller 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-02-04 14:41:22 +03:00
Angira Kekteeva
bffc09167a [#316] Remove minio mentions in commits 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-01-21 10:44:54 +03:00
Denis Kirillov
8872b6f196 [#285] Add resolving order 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-17 18:14:19 +03:00
Denis Kirillov
12c15504ef [#297] Update neofs-sdk-go and neo-go 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-12-30 13:00:23 +03:00
Denis Kirillov
c9f1cf3cb6 [#287] Add version 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-12-07 11:12:54 +03:00
Denis Kirillov
befe084900 [#219] Add container name resolving 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-11-24 13:24:39 +03:00
Denis Kirillov
ae87effb28 [#271] Add random key for no sign requests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-11-16 14:42:08 +03:00
Angira Kekteeva
7d0bc1e992 [#217] Add CORS support 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-10-12 09:03:09 +03:00
Denis Kirillov
19b917e3b5 [#253] Caches refactoring 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-09-14 10:46:18 +03:00
Denis Kirillov
951eb6fda8 [#253] Add access box cache 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-09-14 10:46:18 +03:00
Angira Kekteeva
1ece42b23f [#236] cache: Refactor ListObjectsCache 
Replaced map in ListObjectsCache by gcache.
Now ListObjectsCache keeps only objectIDs and
requests ObjectInfo from cache or NeoFS.
Refactored ListObjectsCache keys: removed delimiter and method fields.
Now ListObjectsCache keeps cache with all objects versions.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-09-07 11:08:48 +03:00
Angira Kekteeva
1bc2e51cbc [#236] api: Refactor caches: ObjectsList, Objects 
Move ObjectsList from layer to cache package
Rename object_cache.go to objects.go

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-09-07 11:08:48 +03:00
Angira Kekteeva
44a2f1b471 [#234] *: Update go.mod 
Remove neofs-node dep
Update versions neofs reps

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-08-30 17:01:41 +03:00