Commit graph

148 commits

Author SHA1 Message Date
4a8c382491 [#91] Update values for health metric
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-04-18 10:25:11 +03:00
01afa1cae4 [#75] Make grpc tree client implementation internal
Since we have pkg 'internal/frostfs/services/tree' that is downloading
during build we cannot export any package that is depended on it.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-04-03 14:04:35 +03:00
a025f2e9c5 [#59] tree: Make interface for tree service client
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-03-29 09:34:10 +03:00
8151753eeb [#60] Use periodic white space XML writer in Complete Multipart Upload
This mechanism is used by Amazon S3 to keep client's
connection alive while object is being constructed from
the upload parts.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-20 13:34:22 +03:00
0af06c3bd9 [TrueCloudLab#40] Add param to configure xml decoder
This parameter enables parsing xml body without
xmlns="http://s3.amazonaws.com/doc/2006-03-01/" attribute
for CompleteMultipartUpload requests

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-03-09 15:44:13 +03:00
813aa2f173 Rename package name
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-07 17:38:08 +03:00
aadefd98b6 [TrueCloudLab#25] Process allow and deny lists of zones in bucket head requests
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-02-17 13:47:48 +03:00
9f823bd65a [TrueCloudLab#26] Add billing metrics to separate registry
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-02-10 10:55:38 +03:00
1ce8b8a30d [TrueCloudLab#21] Support multiple configs
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-02-07 12:26:40 +03:00
533b12d8bb [TrueCloudLab#12] Require only one healthy server
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-01-24 17:56:18 +03:00
cafe079072 [TrueCloudLab#13] Update frostfs SDK
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-01-23 13:59:32 +03:00
388482e230 [#2] Rename internals
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2022-12-21 11:17:45 +03:00
765fd28a49 [#2] Update build
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2022-12-20 18:24:30 +03:00
96dff367db [#1] Build S3 Gateway with FrostFS dependencies
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-12-15 12:43:52 +03:00
Denis Kirillov
5265afe213 [#742] Update docs
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-11-24 17:48:26 +03:00
Denis Kirillov
dd4f66712c [#742] Add multiple listeners
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-11-24 17:48:26 +03:00
Denis Kirillov
d2587b21af [#747] Reload policies on SIGHUP
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-11-18 13:14:25 +03:00
Denis Kirillov
d1825f97fb [#750] Update SDK to support timeout for stream
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-11-15 18:10:28 +03:00
Denis Kirillov
d47840f137 [#568] Add configuration for region to policy map
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-11-11 13:12:49 +03:00
Denis Kirillov
f59bf91a67 [#699] neofs/tree: Add healthcheck
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-10-15 13:59:54 +03:00
Denis Kirillov
3d31c2ab4a [#713] Fix cache access
User can access only keys that he puts into caches

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-10-11 15:26:42 +03:00
Denis Kirillov
a18db163f2 [#716] Fix start TLS server
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-10-03 12:18:00 +03:00
Denis Kirillov
3a7ed8220e [#702] Refactor app initialization
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-09-28 15:07:31 +03:00
Denis Kirillov
1fa28dfacd [#702] Init app settings
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-09-28 15:07:31 +03:00
Denis Kirillov
2a41929be3 [#702] Reload resolvers and TLS certs on SIGHUP
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-09-28 15:07:31 +03:00
Denis Kirillov
42893ec046 [#702] Metrics reload on SIGHUP
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-09-28 15:07:31 +03:00
Denis Kirillov
63275f7876 [#702] Reload config level on SIGHUP
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-09-28 15:07:31 +03:00
Denis Kirillov
7668d5fb2f [#667] Add docs for listen_domains config param
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-09-06 16:32:04 +03:00
Angira Kekteeva
3e4fc52428 [#674] Add AccessKeyID filter
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-09-06 13:47:20 +03:00
Denis Kirillov
628c0f42fe [#680] Use gauge instead of counter for metrics
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-08-25 11:00:59 +03:00
Angira Kekteeva
5307211398 [#634] Add CopiesNumber in NeoFS requests
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-08-16 17:49:15 +03:00
Denis Kirillov
d824db7f69 [#595] Allow SSE-C only with TLS
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-08-13 10:26:00 +03:00
Angira Kekteeva
2d58b25ea0 [#643] Fix linter issues
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-08-05 15:32:46 +03:00
Angira Kekteeva
7bd824f3fa [#607] Improve wallet path param
Made it configurable in yaml

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-08-03 22:25:59 +03:00
Angira Kekteeva
57ce0f25bf [#607] Don't print env vars with empty values
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-08-03 22:25:59 +03:00
Denis Kirillov
08898f4fb4 [#615] Expose pool metrics
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-08-02 16:04:31 +03:00
Denis Kirillov
87f6681281 [#633] Add config param for pool error threshold
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-29 11:25:11 +03:00
Denis Kirillov
e5c1acf1e5 [#591] Sync metrics and pprof configuration
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-29 09:32:08 +03:00
Denis Kirillov
0ae762ef06 [#592] Change pool defaults params
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-22 17:18:37 +03:00
Denis Kirillov
27e3086cbf [#600] Return pointer from newStateMetrics
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-22 17:18:04 +03:00
Denis Kirillov
89d240d831 [#600] Add health metric
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-22 17:18:04 +03:00
Denis Kirillov
1e86413667 [#413] Add signature for tree requests
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-07-22 15:19:16 +03:00
Denis Kirillov
9ed081db3e [#407] Add service tree client prototype
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-22 15:19:16 +03:00
Denis Kirillov
f5441eb6cb [#578] Unify version output
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-15 12:30:40 +03:00
Leonard Lyubich
f0749fd23e [#537] Upgrade NeoFS SDK Go with changed netmap package
`PlacementPolicy` type now provides methods to work with QL-encoded
policies. System network parameters can be read using dedicated method
without iterating. Applications can work with `PlacementPolicy`
variables directly so there is no need to use pointers.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-17 08:28:59 +03:00
Denis Kirillov
300d4359d8 [#465] Handle cache cast failure
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-06-07 17:20:26 +03:00
Alex Vanin
9d3e6f75be [#391] Remove unused notification controller from app state
Now controller is a part of handler.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-05-06 11:03:20 +03:00
Denis Kirillov
94caa2247e [#391] Refactor notifications
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-05-06 10:59:31 +03:00
Denis Kirillov
e3c16a32dd [#409] Update SDK
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-04-26 12:51:52 +04:00
Elizaveta Chichindaeva
bf38007692 [#405] English Check
Signed-off-by: Elizaveta Chichindaeva <elizaveta@nspcc.ru>
2022-04-22 14:01:40 +03:00
Angira Kekteeva
9f017b2bba [#396] Rename rpc-endpoint and resolve-order
To make cli params more consistent
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-04-18 11:58:54 +03:00
Angira Kekteeva
8d9e473804 [#396] Rename params
session-token -> session-tokens
rebalance_timer -> rebalance interval
request_timeout -> healthcheck_timeout

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-04-18 11:58:54 +03:00
Angira Kekteeva
0f3ae334e5 [#396] Remove unused parameters
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-04-18 11:58:54 +03:00
Denis Kirillov
7710de39ec [#365] Update SDK
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-04-08 17:12:52 +03:00
Leonard Lyubich
83fb697f80 [#380] cmd/s3-gw: Write app name and version once on startup
It's redundant to write app name and version in each log message, a
single entry at the start is enough.

Make `App.Wait` to write app name as `neofs-s3-gw` in `application
started` message. Do not add `app_name` and `app_version` fields to the
logger returned by `newLogger`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-04-06 17:31:01 +03:00
Leonard Lyubich
01c721ee53 [#380] cmd/s3-gw: Refactor logger
Do not import `logger` package from NeoFS SDK Go. Discard unusable
`name` and `version` configuration values from `app` section. Discard
all unusable onfiguration values from `logger` section except `level`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-04-06 17:31:01 +03:00
Denis Kirillov
c399cfbdda [#382] Refactor neofs related types
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-31 11:47:24 +03:00
Angira Kekteeva
83bb6fc020 [#384] Add check of Notificator interface value
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-03-30 10:29:28 +03:00
Denis Kirillov
46e4b28489 [#195] Add logger to notification controller
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-23 14:32:13 +04:00
Denis Kirillov
4c3c4b6bee [#195] Add Lock tick listening
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-23 14:32:13 +04:00
Denis Kirillov
32e83db064 [#195] Set Lock type to system lock objects
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-23 14:32:13 +04:00
Denis Kirillov
13080d6d96 [#195] Refactor TestNeoFS
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-23 14:32:13 +04:00
Leonard Lyubich
0cbe09d211 [#346] s3-gw: Reuse single neofs.NeoFS in application
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 00:14:30 +03:00
Leonard Lyubich
cd64f41ce8 [#346] *: Refactor communication with NeoFS at the protocol level
Make `tokens`, `authmate` and `layer` packages to depend from locally
defined `NeoFS` interface of the virtual connection to NeoFS network.
Create internal `neofs` package and implement these interfaces through
`pool.Pool` there. Implement mediators between `NeoFS` interfaces and
`neofs.NeoFS` implementation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 00:14:30 +03:00
Leonard Lyubich
34a221c5c9 [#346] Upgrade NeoFS SDK Go library
Core changes:
  - `object.ID` moved to new package `oid`;
  - `object.Address` moved to new package `address`;
  - `pool.Object` interface changes.

Additionally:
  - Set container owner in `Agent.IssueSecret`.
  - Remove no longer needed fields from `GetObjectParams`
  - `Length` and `Offset` are never assigned. These values
  are set in `Range` field.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 00:14:30 +03:00
Angira Kekteeva
e0c6544567 [#340] Move notification controller to layer
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-02-18 16:59:42 +03:00
Angira Kekteeva
3277293bb3 [#341] Add notifications controller
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-02-04 14:41:22 +03:00
Angira Kekteeva
bffc09167a [#316] Remove minio mentions in commits
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-01-21 10:44:54 +03:00
Denis Kirillov
8872b6f196 [#285] Add resolving order
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-17 18:14:19 +03:00
Denis Kirillov
12c15504ef [#297] Update neofs-sdk-go and neo-go
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-12-30 13:00:23 +03:00
Denis Kirillov
c9f1cf3cb6 [#287] Add version
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-12-07 11:12:54 +03:00
Denis Kirillov
befe084900 [#219] Add container name resolving
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-11-24 13:24:39 +03:00
Denis Kirillov
ae87effb28 [#271] Add random key for no sign requests
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-11-16 14:42:08 +03:00
Angira Kekteeva
7d0bc1e992 [#217] Add CORS support
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-10-12 09:03:09 +03:00
Denis Kirillov
19b917e3b5 [#253] Caches refactoring
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-09-14 10:46:18 +03:00
Denis Kirillov
951eb6fda8 [#253] Add access box cache
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-09-14 10:46:18 +03:00
Angira Kekteeva
1ece42b23f [#236] cache: Refactor ListObjectsCache
Replaced map in ListObjectsCache by gcache.
Now ListObjectsCache keeps only objectIDs and
requests ObjectInfo from cache or NeoFS.
Refactored ListObjectsCache keys: removed delimiter and method fields.
Now ListObjectsCache keeps cache with all objects versions.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-09-07 11:08:48 +03:00
Angira Kekteeva
1bc2e51cbc [#236] api: Refactor caches: ObjectsList, Objects
Move ObjectsList from layer to cache package
Rename object_cache.go to objects.go

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-09-07 11:08:48 +03:00
Angira Kekteeva
44a2f1b471 [#234] *: Update go.mod
Remove neofs-node dep
Update versions neofs reps

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-08-30 17:01:41 +03:00
Angira Kekteeva
2299db4e81 [#218] handler,s3-gw: Make policy configurable
Now default policy of placing containers can be set via config/env
variable.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-08-25 12:18:23 +03:00
Denis Kirillov
bf3d81f928 [#221] Unify source file naming
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-20 15:19:49 +03:00
Angira Kekteeva
b1c6629b10 [#206] Make caches' options configurable
Now caches' options can be configured in .yanl file.
Export caches' variables (listobjects and objects).

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-08-18 10:16:27 +03:00
Denis Kirillov
69b2004a2b [#101] Fix yaml config support
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-28 16:20:15 +03:00
Denis Kirillov
52c63d4c44 [#104] Support NEP-6 for authmate
Drop neofs-crypto.

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-25 12:16:24 +03:00
Denis Kirillov
fd8130a42f [#92] Remove keys generation
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-23 14:57:33 +03:00
Denis Kirillov
b5c8befa25 [#92] Support NEP-6 wallet
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-22 17:10:14 +03:00
Denis Kirillov
b86d20265d [#75] Using secp256r1 instead of curve25519
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-17 16:36:23 +03:00
Roman Khimov
df3746ad03 s3-gw: drop dead code
cmd/s3-gw/app-settings.go:31:2   deadcode  `defaultKeepaliveTime` is unused
cmd/s3-gw/app-settings.go:32:2   deadcode  `defaultKeepaliveTimeout` is unused

Signed-off-by: Roman Khimov <roman@nspcc.ru>
2021-06-15 18:47:50 +03:00
Denis Kirillov
2e96ce6dcd [#73] Dropped grpc keepalive options
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-11 09:49:32 +03:00
Roman Khimov
3cac112f5f s3-gw: expose TLS options in help
Signed-off-by: Roman Khimov <roman@nspcc.ru>
2021-06-10 15:27:13 +03:00
Roman Khimov
d15687f37c *: drop creds/neofs
Signed-off-by: Roman Khimov <roman@nspcc.ru>
2021-05-28 23:48:39 +03:00
Roman Khimov
35ffc1fbc1 *: use sdk-go instead of http-gw
Now that the packages have moved there we can do it.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
2021-05-28 14:05:46 +03:00
Roman Khimov
d19ce03072 *: drop old sdk dependecies, bump neofs-api-go version
I'm not sure it works, but it's enough code-wise for now. We're reusing some
http-gw components here that are to be moved into sdk-go in future.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
2021-05-26 21:01:46 +03:00
Roman Khimov
dbe65ae602 creds: move credential management into s3 gate
Mostly taken from old SDK (abe47687cd11266f946cad57f07572cc10c67226), but
error handling adapted to eliminate pkg/errors and internal packages.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
2021-05-25 23:00:19 +03:00
Roman Khimov
69e3e22dbc cmd: drop grace package dependency
Note that signal.NotifyContext is a Go 1.16 feature.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
2021-05-25 13:32:08 +03:00
Roman Khimov
92c8cce933 misc: move to internal/version, use for all binaries
No libmisc, please.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
2021-05-20 16:26:41 +03:00
Roman Khimov
ef8684c11d s3-gw: drop build time setting
It's not properly set anyway and it's a bad practice for build
reproducibility.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
2021-05-20 16:26:20 +03:00
Roman Khimov
e622f1987b Makefile: deduplicate binaries
Signed-off-by: Roman Khimov <roman@nspcc.ru>
2021-05-20 16:26:20 +03:00