pogpp/frostfs-s3-gw
1
0
Fork 0
forked from TrueCloudLab/frostfs-s3-gw
Code Pull requests Activity
1199 commits 4 branches 21 tags 20 MiB
Commit graph

238 commits

Author SHA1 Message Date
Denis Kirillov
bd3164c57f [#68] Fix pre-commit issues 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-24 16:22:06 +03:00
Alex Vanin
8151753eeb [#60] Use periodic white space XML writer in Complete Multipart Upload 
This mechanism is used by Amazon S3 to keep client's
connection alive while object is being constructed from
the upload parts.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-20 13:34:22 +03:00
Denis Kirillov
0af06c3bd9 [TrueCloudLab#40] Add param to configure xml decoder 
This parameter enables parsing xml body without
xmlns="http://s3.amazonaws.com/doc/2006-03-01/" attribute
for CompleteMultipartUpload requests

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-09 15:44:13 +03:00
Alex Vanin
813aa2f173 Rename package name 
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-07 17:38:08 +03:00
Alex Vanin
aadefd98b6 [TrueCloudLab#25] Process allow and deny lists of zones in bucket head requests 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-02-17 13:47:48 +03:00
Denis Kirillov
9f823bd65a [TrueCloudLab#26] Add billing metrics to separate registry 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-02-10 10:55:38 +03:00
Denis Kirillov
1ce8b8a30d [TrueCloudLab#21] Support multiple configs 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-02-07 12:26:40 +03:00
Denis Kirillov
533b12d8bb [TrueCloudLab#12] Require only one healthy server 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-01-24 17:56:18 +03:00
Denis Kirillov
cafe079072 [TrueCloudLab#13] Update frostfs SDK 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-01-23 13:59:32 +03:00
Denis Kirillov
f1696c4725 [#760] authmate: Add flag for pool timeouts 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-12-28 17:06:32 +03:00
Denis Kirillov
388482e230 [#2] Rename internals 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2022-12-21 11:17:45 +03:00
Denis Kirillov
765fd28a49 [#2] Update build 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2022-12-20 18:24:30 +03:00
Alex Vanin
96dff367db [#1] Build S3 Gateway with FrostFS dependencies 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2022-12-15 12:43:52 +03:00
Denis Kirillov
5265afe213 [#742] Update docs 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-24 17:48:26 +03:00
Denis Kirillov
dd4f66712c [#742] Add multiple listeners 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-24 17:48:26 +03:00
Denis Kirillov
d2587b21af [#747] Reload policies on SIGHUP 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-18 13:14:25 +03:00
Denis Kirillov
d1825f97fb [#750] Update SDK to support timeout for stream 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-15 18:10:28 +03:00
Denis Kirillov
d47840f137 [#568] Add configuration for region to policy map 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-11 13:12:49 +03:00
Denis Kirillov
f59bf91a67 [#699] neofs/tree: Add healthcheck 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-15 13:59:54 +03:00
Denis Kirillov
3d31c2ab4a [#713] Fix cache access 
User can access only keys that he puts into caches

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-11 15:26:42 +03:00
Denis Kirillov
a18db163f2 [#716] Fix start TLS server 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-03 12:18:00 +03:00
Denis Kirillov
3a7ed8220e [#702] Refactor app initialization 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-28 15:07:31 +03:00
Denis Kirillov
1fa28dfacd [#702] Init app settings 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-28 15:07:31 +03:00
Denis Kirillov
2a41929be3 [#702] Reload resolvers and TLS certs on SIGHUP 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-28 15:07:31 +03:00
Denis Kirillov
42893ec046 [#702] Metrics reload on SIGHUP 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-28 15:07:31 +03:00
Denis Kirillov
63275f7876 [#702] Reload config level on SIGHUP 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-28 15:07:31 +03:00
Denis Kirillov
7668d5fb2f [#667] Add docs for listen_domains config param 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-06 16:32:04 +03:00
Angira Kekteeva
3e4fc52428 [#674] Add AccessKeyID filter 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-09-06 13:47:20 +03:00
Denis Kirillov
628c0f42fe [#680] Use gauge instead of counter for metrics 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-25 11:00:59 +03:00
Angira Kekteeva
5307211398 [#634] Add CopiesNumber in NeoFS requests 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-16 17:49:15 +03:00
Denis Kirillov
d824db7f69 [#595] Allow SSE-C only with TLS 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-13 10:26:00 +03:00
Angira Kekteeva
2d58b25ea0 [#643] Fix linter issues 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-05 15:32:46 +03:00
Angira Kekteeva
7bd824f3fa [#607] Improve wallet path param 
Made it configurable in yaml

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-03 22:25:59 +03:00
Angira Kekteeva
57ce0f25bf [#607] Don't print env vars with empty values 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-03 22:25:59 +03:00
Denis Kirillov
08898f4fb4 [#615] Expose pool metrics 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-02 16:04:31 +03:00
Denis Kirillov
87f6681281 [#633] Add config param for pool error threshold 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-29 11:25:11 +03:00
Denis Kirillov
e5c1acf1e5 [#591] Sync metrics and pprof configuration 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-29 09:32:08 +03:00
Denis Kirillov
0ae762ef06 [#592] Change pool defaults params 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 17:18:37 +03:00
Denis Kirillov
27e3086cbf [#600] Return pointer from newStateMetrics 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 17:18:04 +03:00
Denis Kirillov
89d240d831 [#600] Add health metric 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 17:18:04 +03:00
Denis Kirillov
85c203e157 [#488] Sync using cid.ID 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
1e86413667 [#413] Add signature for tree requests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
9ed081db3e [#407] Add service tree client prototype 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
f5441eb6cb [#578] Unify version output 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-15 12:30:40 +03:00
Angira Kekteeva
ceb00eefc5 [#583] authmate: Fix error messages and help 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-07 23:45:31 +04:00
Angira Kekteeva
5d52f056eb [#583] Fix error when policy file cannot be read 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-07 23:45:31 +04:00
Angira Kekteeva
d2a3ba0c06 [#562] Rename authmate directory 
To generate neofs-s3-authmate instead of neofs-authmate

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-04 11:08:50 +03:00
Denis Kirillov
7ca519cb32 [#539] Add context to errors 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-27 02:23:19 +04:00
Leonard Lyubich
f0749fd23e [#537] Upgrade NeoFS SDK Go with changed netmap package 
`PlacementPolicy` type now provides methods to work with QL-encoded
policies. System network parameters can be read using dedicated method
without iterating. Applications can work with `PlacementPolicy`
variables directly so there is no need to use pointers.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-17 08:28:59 +03:00
Denis Kirillov
d521af2065 [#529] Add presign URLs support 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-16 17:17:45 +03:00
Denis Kirillov
300d4359d8 [#465] Handle cache cast failure 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-07 17:20:26 +03:00
Alex Vanin
0e37242b65 [#463] Move DefaultLocationConstraint constant into api 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-05-30 14:41:32 +03:00
Alex Vanin
80d4d071d8 [#463] Restrict overriding default location constraint in authmate 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-05-30 14:41:32 +03:00
Alex Vanin
9d3e6f75be [#391] Remove unused notification controller from app state 
Now controller is a part of handler.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-05-06 11:03:20 +03:00
Denis Kirillov
94caa2247e [#391] Refactor notifications 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-05-06 10:59:31 +03:00
Alex Vanin
c9126fc0a9 [#409] authmate: Fix container ID parsing 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-04-29 00:04:47 +04:00
Denis Kirillov
e3c16a32dd [#409] Update SDK 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-04-26 12:51:52 +04:00
Elizaveta Chichindaeva
bf38007692 [#405] English Check 
Signed-off-by: Elizaveta Chichindaeva <elizaveta@nspcc.ru>
 2022-04-22 14:01:40 +03:00
Angira Kekteeva
9f017b2bba [#396] Rename rpc-endpoint and resolve-order 
To make cli params more consistent
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-18 11:58:54 +03:00
Angira Kekteeva
8d9e473804 [#396] Rename params 
session-token -> session-tokens
rebalance_timer -> rebalance interval
request_timeout -> healthcheck_timeout

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-18 11:58:54 +03:00
Angira Kekteeva
0f3ae334e5 [#396] Remove unused parameters 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-18 11:58:54 +03:00
Denis Kirillov
7710de39ec [#365] Update SDK 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-04-08 17:12:52 +03:00
Leonard Lyubich
83fb697f80 [#380] cmd/s3-gw: Write app name and version once on startup 
It's redundant to write app name and version in each log message, a
single entry at the start is enough.

Make `App.Wait` to write app name as `neofs-s3-gw` in `application
started` message. Do not add `app_name` and `app_version` fields to the
logger returned by `newLogger`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-04-06 17:31:01 +03:00
Leonard Lyubich
01c721ee53 [#380] cmd/s3-gw: Refactor logger 
Do not import `logger` package from NeoFS SDK Go. Discard unusable
`name` and `version` configuration values from `app` section. Discard
all unusable onfiguration values from `logger` section except `level`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-04-06 17:31:01 +03:00
Denis Kirillov
c399cfbdda [#382] Refactor neofs related types 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-31 11:47:24 +03:00
Denis Kirillov
4322da90d8 [#387] authmate: fix parsing session rules 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-31 11:44:26 +03:00
Angira Kekteeva
83bb6fc020 [#384] Add check of Notificator interface value 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-03-30 10:29:28 +03:00
Denis Kirillov
46e4b28489 [#195] Add logger to notification controller 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
4c3c4b6bee [#195] Add Lock tick listening 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
32e83db064 [#195] Set Lock type to system lock objects 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
13080d6d96 [#195] Refactor TestNeoFS 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
52f0af0ccc [#372] Check parameters before creating container 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-16 11:00:23 +03:00
Leonard Lyubich
ac0ffd6136 [#346] cmd/authmate: Fix NPE in createNeoFS 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-04 00:14:30 +03:00
Leonard Lyubich
0cbe09d211 [#346] s3-gw: Reuse single neofs.NeoFS in application 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-04 00:14:30 +03:00
Leonard Lyubich
cd64f41ce8 [#346] *: Refactor communication with NeoFS at the protocol level 
Make `tokens`, `authmate` and `layer` packages to depend from locally
defined `NeoFS` interface of the virtual connection to NeoFS network.
Create internal `neofs` package and implement these interfaces through
`pool.Pool` there. Implement mediators between `NeoFS` interfaces and
`neofs.NeoFS` implementation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-04 00:14:30 +03:00
Leonard Lyubich
34a221c5c9 [#346] Upgrade NeoFS SDK Go library 
Core changes:
  - `object.ID` moved to new package `oid`;
  - `object.Address` moved to new package `address`;
  - `pool.Object` interface changes.

Additionally:
  - Set container owner in `Agent.IssueSecret`.
  - Remove no longer needed fields from `GetObjectParams`
  - `Length` and `Offset` are never assigned. These values
  are set in `Range` field.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-04 00:14:30 +03:00
Denis Kirillov
6a4fba4d09 [#364] Support placement policy overriding 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-03 18:12:20 +03:00
Angira Kekteeva
e0c6544567 [#340] Move notification controller to layer 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-02-18 16:59:42 +03:00
Angira Kekteeva
3277293bb3 [#341] Add notifications controller 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-02-04 14:41:22 +03:00
Angira Kekteeva
204835ace3 [#338] Remove redundant flag for session tokens in authmate 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-02-02 18:11:19 +03:00
Denis Kirillov
f19c9315ea [#324] Drop default container name 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-21 15:39:05 +03:00
Angira Kekteeva
bffc09167a [#316] Remove minio mentions in commits 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-01-21 10:44:54 +03:00
Denis Kirillov
8872b6f196 [#285] Add resolving order 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-17 18:14:19 +03:00
Denis Kirillov
12c15504ef [#297] Update neofs-sdk-go and neo-go 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-12-30 13:00:23 +03:00
Angira Kekteeva
a23a97efd6 [#290] Add timeout to authmate 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-12-27 16:25:49 +03:00
Angira Kekteeva
adc500672f [#295] authmate: Fix replace in obtain-secret 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-12-27 13:00:31 +03:00
Denis Kirillov
c9f1cf3cb6 [#287] Add version 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-12-07 11:12:54 +03:00
Denis Kirillov
befe084900 [#219] Add container name resolving 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-11-24 13:24:39 +03:00
Denis Kirillov
1254f5dac0 [#282] Fix lifetime examples 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-11-22 15:09:49 +03:00
Denis Kirillov
d36dfe8c61 [#271] Update neo-sdk-go to the latest version 
Refactoring invoking pool methods for anonymous requests.

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-11-16 14:42:08 +03:00
Denis Kirillov
ae87effb28 [#271] Add random key for no sign requests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-11-16 14:42:08 +03:00
Denis Kirillov
71f578a883 [#258] Add explicit expiration epoch 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-11-08 12:41:22 +03:00
Denis Kirillov
214c55431b [#258] Treat lifetime as time.Duration 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-11-08 12:41:22 +03:00
Angira Kekteeva
7d0bc1e992 [#217] Add CORS support 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-10-12 09:03:09 +03:00
Denis Kirillov
19b917e3b5 [#253] Caches refactoring 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-09-14 10:46:18 +03:00
Denis Kirillov
951eb6fda8 [#253] Add access box cache 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-09-14 10:46:18 +03:00
Denis Kirillov
26f30e7ace [#255] Add expiration to access box 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-09-14 10:21:25 +03:00
Angira Kekteeva
1ece42b23f [#236] cache: Refactor ListObjectsCache 
Replaced map in ListObjectsCache by gcache.
Now ListObjectsCache keeps only objectIDs and
requests ObjectInfo from cache or NeoFS.
Refactored ListObjectsCache keys: removed delimiter and method fields.
Now ListObjectsCache keeps cache with all objects versions.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-09-07 11:08:48 +03:00
Angira Kekteeva
1bc2e51cbc [#236] api: Refactor caches: ObjectsList, Objects 
Move ObjectsList from layer to cache package
Rename object_cache.go to objects.go

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-09-07 11:08:48 +03:00
Denis Kirillov
345dafb29d [#241] Add aws-cli-credentials flag 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-09-03 13:59:40 +03:00