Commit graph

238 commits

Author SHA1 Message Date
bd3164c57f [#68] Fix pre-commit issues
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-03-24 16:22:06 +03:00
8151753eeb [#60] Use periodic white space XML writer in Complete Multipart Upload
This mechanism is used by Amazon S3 to keep client's
connection alive while object is being constructed from
the upload parts.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-20 13:34:22 +03:00
0af06c3bd9 [TrueCloudLab#40] Add param to configure xml decoder
This parameter enables parsing xml body without
xmlns="http://s3.amazonaws.com/doc/2006-03-01/" attribute
for CompleteMultipartUpload requests

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-03-09 15:44:13 +03:00
813aa2f173 Rename package name
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-07 17:38:08 +03:00
aadefd98b6 [TrueCloudLab#25] Process allow and deny lists of zones in bucket head requests
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-02-17 13:47:48 +03:00
9f823bd65a [TrueCloudLab#26] Add billing metrics to separate registry
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-02-10 10:55:38 +03:00
1ce8b8a30d [TrueCloudLab#21] Support multiple configs
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-02-07 12:26:40 +03:00
533b12d8bb [TrueCloudLab#12] Require only one healthy server
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-01-24 17:56:18 +03:00
cafe079072 [TrueCloudLab#13] Update frostfs SDK
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-01-23 13:59:32 +03:00
Denis Kirillov
f1696c4725 [#760] authmate: Add flag for pool timeouts
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-12-28 17:06:32 +03:00
388482e230 [#2] Rename internals
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2022-12-21 11:17:45 +03:00
765fd28a49 [#2] Update build
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2022-12-20 18:24:30 +03:00
96dff367db [#1] Build S3 Gateway with FrostFS dependencies
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-12-15 12:43:52 +03:00
Denis Kirillov
5265afe213 [#742] Update docs
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-11-24 17:48:26 +03:00
Denis Kirillov
dd4f66712c [#742] Add multiple listeners
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-11-24 17:48:26 +03:00
Denis Kirillov
d2587b21af [#747] Reload policies on SIGHUP
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-11-18 13:14:25 +03:00
Denis Kirillov
d1825f97fb [#750] Update SDK to support timeout for stream
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-11-15 18:10:28 +03:00
Denis Kirillov
d47840f137 [#568] Add configuration for region to policy map
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-11-11 13:12:49 +03:00
Denis Kirillov
f59bf91a67 [#699] neofs/tree: Add healthcheck
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-10-15 13:59:54 +03:00
Denis Kirillov
3d31c2ab4a [#713] Fix cache access
User can access only keys that he puts into caches

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-10-11 15:26:42 +03:00
Denis Kirillov
a18db163f2 [#716] Fix start TLS server
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-10-03 12:18:00 +03:00
Denis Kirillov
3a7ed8220e [#702] Refactor app initialization
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-09-28 15:07:31 +03:00
Denis Kirillov
1fa28dfacd [#702] Init app settings
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-09-28 15:07:31 +03:00
Denis Kirillov
2a41929be3 [#702] Reload resolvers and TLS certs on SIGHUP
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-09-28 15:07:31 +03:00
Denis Kirillov
42893ec046 [#702] Metrics reload on SIGHUP
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-09-28 15:07:31 +03:00
Denis Kirillov
63275f7876 [#702] Reload config level on SIGHUP
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-09-28 15:07:31 +03:00
Denis Kirillov
7668d5fb2f [#667] Add docs for listen_domains config param
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-09-06 16:32:04 +03:00
Angira Kekteeva
3e4fc52428 [#674] Add AccessKeyID filter
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-09-06 13:47:20 +03:00
Denis Kirillov
628c0f42fe [#680] Use gauge instead of counter for metrics
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-08-25 11:00:59 +03:00
Angira Kekteeva
5307211398 [#634] Add CopiesNumber in NeoFS requests
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-08-16 17:49:15 +03:00
Denis Kirillov
d824db7f69 [#595] Allow SSE-C only with TLS
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-08-13 10:26:00 +03:00
Angira Kekteeva
2d58b25ea0 [#643] Fix linter issues
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-08-05 15:32:46 +03:00
Angira Kekteeva
7bd824f3fa [#607] Improve wallet path param
Made it configurable in yaml

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-08-03 22:25:59 +03:00
Angira Kekteeva
57ce0f25bf [#607] Don't print env vars with empty values
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-08-03 22:25:59 +03:00
Denis Kirillov
08898f4fb4 [#615] Expose pool metrics
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-08-02 16:04:31 +03:00
Denis Kirillov
87f6681281 [#633] Add config param for pool error threshold
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-29 11:25:11 +03:00
Denis Kirillov
e5c1acf1e5 [#591] Sync metrics and pprof configuration
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-29 09:32:08 +03:00
Denis Kirillov
0ae762ef06 [#592] Change pool defaults params
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-22 17:18:37 +03:00
Denis Kirillov
27e3086cbf [#600] Return pointer from newStateMetrics
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-22 17:18:04 +03:00
Denis Kirillov
89d240d831 [#600] Add health metric
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-22 17:18:04 +03:00
Denis Kirillov
85c203e157 [#488] Sync using cid.ID
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-22 15:19:16 +03:00
Denis Kirillov
1e86413667 [#413] Add signature for tree requests
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-07-22 15:19:16 +03:00
Denis Kirillov
9ed081db3e [#407] Add service tree client prototype
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-22 15:19:16 +03:00
Denis Kirillov
f5441eb6cb [#578] Unify version output
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-15 12:30:40 +03:00
Angira Kekteeva
ceb00eefc5 [#583] authmate: Fix error messages and help
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-07-07 23:45:31 +04:00
Angira Kekteeva
5d52f056eb [#583] Fix error when policy file cannot be read
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-07-07 23:45:31 +04:00
Angira Kekteeva
d2a3ba0c06 [#562] Rename authmate directory
To generate neofs-s3-authmate instead of neofs-authmate

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-07-04 11:08:50 +03:00
Denis Kirillov
7ca519cb32 [#539] Add context to errors
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-06-27 02:23:19 +04:00
Leonard Lyubich
f0749fd23e [#537] Upgrade NeoFS SDK Go with changed netmap package
`PlacementPolicy` type now provides methods to work with QL-encoded
policies. System network parameters can be read using dedicated method
without iterating. Applications can work with `PlacementPolicy`
variables directly so there is no need to use pointers.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-17 08:28:59 +03:00
Denis Kirillov
d521af2065 [#529] Add presign URLs support
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-06-16 17:17:45 +03:00
Denis Kirillov
300d4359d8 [#465] Handle cache cast failure
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-06-07 17:20:26 +03:00
Alex Vanin
0e37242b65 [#463] Move DefaultLocationConstraint constant into api
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-05-30 14:41:32 +03:00
Alex Vanin
80d4d071d8 [#463] Restrict overriding default location constraint in authmate
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-05-30 14:41:32 +03:00
Alex Vanin
9d3e6f75be [#391] Remove unused notification controller from app state
Now controller is a part of handler.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-05-06 11:03:20 +03:00
Denis Kirillov
94caa2247e [#391] Refactor notifications
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-05-06 10:59:31 +03:00
Alex Vanin
c9126fc0a9 [#409] authmate: Fix container ID parsing
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-04-29 00:04:47 +04:00
Denis Kirillov
e3c16a32dd [#409] Update SDK
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-04-26 12:51:52 +04:00
Elizaveta Chichindaeva
bf38007692 [#405] English Check
Signed-off-by: Elizaveta Chichindaeva <elizaveta@nspcc.ru>
2022-04-22 14:01:40 +03:00
Angira Kekteeva
9f017b2bba [#396] Rename rpc-endpoint and resolve-order
To make cli params more consistent
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-04-18 11:58:54 +03:00
Angira Kekteeva
8d9e473804 [#396] Rename params
session-token -> session-tokens
rebalance_timer -> rebalance interval
request_timeout -> healthcheck_timeout

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-04-18 11:58:54 +03:00
Angira Kekteeva
0f3ae334e5 [#396] Remove unused parameters
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-04-18 11:58:54 +03:00
Denis Kirillov
7710de39ec [#365] Update SDK
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-04-08 17:12:52 +03:00
Leonard Lyubich
83fb697f80 [#380] cmd/s3-gw: Write app name and version once on startup
It's redundant to write app name and version in each log message, a
single entry at the start is enough.

Make `App.Wait` to write app name as `neofs-s3-gw` in `application
started` message. Do not add `app_name` and `app_version` fields to the
logger returned by `newLogger`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-04-06 17:31:01 +03:00
Leonard Lyubich
01c721ee53 [#380] cmd/s3-gw: Refactor logger
Do not import `logger` package from NeoFS SDK Go. Discard unusable
`name` and `version` configuration values from `app` section. Discard
all unusable onfiguration values from `logger` section except `level`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-04-06 17:31:01 +03:00
Denis Kirillov
c399cfbdda [#382] Refactor neofs related types
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-31 11:47:24 +03:00
Denis Kirillov
4322da90d8 [#387] authmate: fix parsing session rules
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-31 11:44:26 +03:00
Angira Kekteeva
83bb6fc020 [#384] Add check of Notificator interface value
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-03-30 10:29:28 +03:00
Denis Kirillov
46e4b28489 [#195] Add logger to notification controller
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-23 14:32:13 +04:00
Denis Kirillov
4c3c4b6bee [#195] Add Lock tick listening
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-23 14:32:13 +04:00
Denis Kirillov
32e83db064 [#195] Set Lock type to system lock objects
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-23 14:32:13 +04:00
Denis Kirillov
13080d6d96 [#195] Refactor TestNeoFS
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-23 14:32:13 +04:00
Denis Kirillov
52f0af0ccc [#372] Check parameters before creating container
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-16 11:00:23 +03:00
Leonard Lyubich
ac0ffd6136 [#346] cmd/authmate: Fix NPE in createNeoFS
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 00:14:30 +03:00
Leonard Lyubich
0cbe09d211 [#346] s3-gw: Reuse single neofs.NeoFS in application
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 00:14:30 +03:00
Leonard Lyubich
cd64f41ce8 [#346] *: Refactor communication with NeoFS at the protocol level
Make `tokens`, `authmate` and `layer` packages to depend from locally
defined `NeoFS` interface of the virtual connection to NeoFS network.
Create internal `neofs` package and implement these interfaces through
`pool.Pool` there. Implement mediators between `NeoFS` interfaces and
`neofs.NeoFS` implementation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 00:14:30 +03:00
Leonard Lyubich
34a221c5c9 [#346] Upgrade NeoFS SDK Go library
Core changes:
  - `object.ID` moved to new package `oid`;
  - `object.Address` moved to new package `address`;
  - `pool.Object` interface changes.

Additionally:
  - Set container owner in `Agent.IssueSecret`.
  - Remove no longer needed fields from `GetObjectParams`
  - `Length` and `Offset` are never assigned. These values
  are set in `Range` field.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 00:14:30 +03:00
Denis Kirillov
6a4fba4d09 [#364] Support placement policy overriding
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-03 18:12:20 +03:00
Angira Kekteeva
e0c6544567 [#340] Move notification controller to layer
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-02-18 16:59:42 +03:00
Angira Kekteeva
3277293bb3 [#341] Add notifications controller
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-02-04 14:41:22 +03:00
Angira Kekteeva
204835ace3 [#338] Remove redundant flag for session tokens in authmate
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-02-02 18:11:19 +03:00
Denis Kirillov
f19c9315ea [#324] Drop default container name
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-21 15:39:05 +03:00
Angira Kekteeva
bffc09167a [#316] Remove minio mentions in commits
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-01-21 10:44:54 +03:00
Denis Kirillov
8872b6f196 [#285] Add resolving order
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-17 18:14:19 +03:00
Denis Kirillov
12c15504ef [#297] Update neofs-sdk-go and neo-go
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-12-30 13:00:23 +03:00
Angira Kekteeva
a23a97efd6 [#290] Add timeout to authmate
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-12-27 16:25:49 +03:00
Angira Kekteeva
adc500672f [#295] authmate: Fix replace in obtain-secret
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-12-27 13:00:31 +03:00
Denis Kirillov
c9f1cf3cb6 [#287] Add version
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-12-07 11:12:54 +03:00
Denis Kirillov
befe084900 [#219] Add container name resolving
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-11-24 13:24:39 +03:00
Denis Kirillov
1254f5dac0 [#282] Fix lifetime examples
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-11-22 15:09:49 +03:00
Denis Kirillov
d36dfe8c61 [#271] Update neo-sdk-go to the latest version
Refactoring invoking pool methods for anonymous requests.

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-11-16 14:42:08 +03:00
Denis Kirillov
ae87effb28 [#271] Add random key for no sign requests
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-11-16 14:42:08 +03:00
Denis Kirillov
71f578a883 [#258] Add explicit expiration epoch
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-11-08 12:41:22 +03:00
Denis Kirillov
214c55431b [#258] Treat lifetime as time.Duration
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-11-08 12:41:22 +03:00
Angira Kekteeva
7d0bc1e992 [#217] Add CORS support
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-10-12 09:03:09 +03:00
Denis Kirillov
19b917e3b5 [#253] Caches refactoring
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-09-14 10:46:18 +03:00
Denis Kirillov
951eb6fda8 [#253] Add access box cache
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-09-14 10:46:18 +03:00
Denis Kirillov
26f30e7ace [#255] Add expiration to access box
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-09-14 10:21:25 +03:00
Angira Kekteeva
1ece42b23f [#236] cache: Refactor ListObjectsCache
Replaced map in ListObjectsCache by gcache.
Now ListObjectsCache keeps only objectIDs and
requests ObjectInfo from cache or NeoFS.
Refactored ListObjectsCache keys: removed delimiter and method fields.
Now ListObjectsCache keeps cache with all objects versions.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-09-07 11:08:48 +03:00
Angira Kekteeva
1bc2e51cbc [#236] api: Refactor caches: ObjectsList, Objects
Move ObjectsList from layer to cache package
Rename object_cache.go to objects.go

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-09-07 11:08:48 +03:00
Denis Kirillov
345dafb29d [#241] Add aws-cli-credentials flag
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-09-03 13:59:40 +03:00