TrueCloudLab/frostfs-s3-gw
15
3
Fork 18
Code Issues 28 Pull requests 4 Projects Releases 17 Packages Activity Actions
990 commits 5 branches 41 tags 20 MiB
Commit graph

179 commits

Author SHA1 Message Date
Angira Kekteeva
8d9e473804 [#396] Rename params 
session-token -> session-tokens
rebalance_timer -> rebalance interval
request_timeout -> healthcheck_timeout

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-18 11:58:54 +03:00
Angira Kekteeva
0f3ae334e5 [#396] Remove unused parameters 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-18 11:58:54 +03:00
Denis Kirillov
7710de39ec [#365] Update SDK 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-04-08 17:12:52 +03:00
Leonard Lyubich
83fb697f80 [#380] cmd/s3-gw: Write app name and version once on startup 
It's redundant to write app name and version in each log message, a
single entry at the start is enough.

Make `App.Wait` to write app name as `neofs-s3-gw` in `application
started` message. Do not add `app_name` and `app_version` fields to the
logger returned by `newLogger`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-04-06 17:31:01 +03:00
Leonard Lyubich
01c721ee53 [#380] cmd/s3-gw: Refactor logger 
Do not import `logger` package from NeoFS SDK Go. Discard unusable
`name` and `version` configuration values from `app` section. Discard
all unusable onfiguration values from `logger` section except `level`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-04-06 17:31:01 +03:00
Denis Kirillov
c399cfbdda [#382] Refactor neofs related types 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-31 11:47:24 +03:00
Denis Kirillov
4322da90d8 [#387] authmate: fix parsing session rules 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-31 11:44:26 +03:00
Angira Kekteeva
83bb6fc020 [#384] Add check of Notificator interface value 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-03-30 10:29:28 +03:00
Denis Kirillov
46e4b28489 [#195] Add logger to notification controller 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
4c3c4b6bee [#195] Add Lock tick listening 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
32e83db064 [#195] Set Lock type to system lock objects 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
13080d6d96 [#195] Refactor TestNeoFS 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
52f0af0ccc [#372] Check parameters before creating container 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-16 11:00:23 +03:00
Leonard Lyubich
ac0ffd6136 [#346] cmd/authmate: Fix NPE in createNeoFS 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-04 00:14:30 +03:00
Leonard Lyubich
0cbe09d211 [#346] s3-gw: Reuse single neofs.NeoFS in application 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-04 00:14:30 +03:00
Leonard Lyubich
cd64f41ce8 [#346] *: Refactor communication with NeoFS at the protocol level 
Make `tokens`, `authmate` and `layer` packages to depend from locally
defined `NeoFS` interface of the virtual connection to NeoFS network.
Create internal `neofs` package and implement these interfaces through
`pool.Pool` there. Implement mediators between `NeoFS` interfaces and
`neofs.NeoFS` implementation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-04 00:14:30 +03:00
Leonard Lyubich
34a221c5c9 [#346] Upgrade NeoFS SDK Go library 
Core changes:
  - `object.ID` moved to new package `oid`;
  - `object.Address` moved to new package `address`;
  - `pool.Object` interface changes.

Additionally:
  - Set container owner in `Agent.IssueSecret`.
  - Remove no longer needed fields from `GetObjectParams`
  - `Length` and `Offset` are never assigned. These values
  are set in `Range` field.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-04 00:14:30 +03:00
Denis Kirillov
6a4fba4d09 [#364] Support placement policy overriding 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-03 18:12:20 +03:00
Angira Kekteeva
e0c6544567 [#340] Move notification controller to layer 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-02-18 16:59:42 +03:00
Angira Kekteeva
3277293bb3 [#341] Add notifications controller 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-02-04 14:41:22 +03:00
Angira Kekteeva
204835ace3 [#338] Remove redundant flag for session tokens in authmate 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-02-02 18:11:19 +03:00
Denis Kirillov
f19c9315ea [#324] Drop default container name 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-21 15:39:05 +03:00
Angira Kekteeva
bffc09167a [#316] Remove minio mentions in commits 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-01-21 10:44:54 +03:00
Denis Kirillov
8872b6f196 [#285] Add resolving order 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-17 18:14:19 +03:00
Denis Kirillov
12c15504ef [#297] Update neofs-sdk-go and neo-go 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-12-30 13:00:23 +03:00
Angira Kekteeva
a23a97efd6 [#290] Add timeout to authmate 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-12-27 16:25:49 +03:00
Angira Kekteeva
adc500672f [#295] authmate: Fix replace in obtain-secret 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-12-27 13:00:31 +03:00
Denis Kirillov
c9f1cf3cb6 [#287] Add version 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-12-07 11:12:54 +03:00
Denis Kirillov
befe084900 [#219] Add container name resolving 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-11-24 13:24:39 +03:00
Denis Kirillov
1254f5dac0 [#282] Fix lifetime examples 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-11-22 15:09:49 +03:00
Denis Kirillov
d36dfe8c61 [#271] Update neo-sdk-go to the latest version 
Refactoring invoking pool methods for anonymous requests.

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-11-16 14:42:08 +03:00
Denis Kirillov
ae87effb28 [#271] Add random key for no sign requests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-11-16 14:42:08 +03:00
Denis Kirillov
71f578a883 [#258] Add explicit expiration epoch 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-11-08 12:41:22 +03:00
Denis Kirillov
214c55431b [#258] Treat lifetime as time.Duration 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-11-08 12:41:22 +03:00
Angira Kekteeva
7d0bc1e992 [#217] Add CORS support 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-10-12 09:03:09 +03:00
Denis Kirillov
19b917e3b5 [#253] Caches refactoring 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-09-14 10:46:18 +03:00
Denis Kirillov
951eb6fda8 [#253] Add access box cache 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-09-14 10:46:18 +03:00
Denis Kirillov
26f30e7ace [#255] Add expiration to access box 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-09-14 10:21:25 +03:00
Angira Kekteeva
1ece42b23f [#236] cache: Refactor ListObjectsCache 
Replaced map in ListObjectsCache by gcache.
Now ListObjectsCache keeps only objectIDs and
requests ObjectInfo from cache or NeoFS.
Refactored ListObjectsCache keys: removed delimiter and method fields.
Now ListObjectsCache keeps cache with all objects versions.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-09-07 11:08:48 +03:00
Angira Kekteeva
1bc2e51cbc [#236] api: Refactor caches: ObjectsList, Objects 
Move ObjectsList from layer to cache package
Rename object_cache.go to objects.go

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-09-07 11:08:48 +03:00
Denis Kirillov
345dafb29d [#241] Add aws-cli-credentials flag 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-09-03 13:59:40 +03:00
Angira Kekteeva
44a2f1b471 [#234] *: Update go.mod 
Remove neofs-node dep
Update versions neofs reps

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-08-30 17:01:41 +03:00
Angira Kekteeva
2299db4e81 [#218] handler,s3-gw: Make policy configurable 
Now default policy of placing containers can be set via config/env
variable.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-08-25 12:18:23 +03:00
Denis Kirillov
bf3d81f928 [#221] Unify source file naming 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-08-20 15:19:49 +03:00
Angira Kekteeva
b1c6629b10 [#206] Make caches' options configurable 
Now caches' options can be configured in .yanl file.
Export caches' variables (listobjects and objects).

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-08-18 10:16:27 +03:00
Denis Kirillov
1fbd192bd7 [#89] Add placement policy 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-16 16:01:35 +03:00
Denis Kirillov
1be8030dcd [#71] Supported json file rules in authmate 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-06-29 16:41:42 +03:00
Roman Khimov
c16b24239f Merge pull request #114 from masterSplinter01/feature/lifetime-authmate 
authmate: Add lifetime for tokens
 2021-06-28 17:58:43 +03:00
Angira Kekteeva
f97739898a [#108] authmate: Add lifetime for tokens 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-06-28 17:17:41 +03:00
Denis Kirillov
69b2004a2b [#101] Fix yaml config support 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-06-28 16:20:15 +03:00
Denis Kirillov
52c63d4c44 [#104] Support NEP-6 for authmate 
Drop neofs-crypto.

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-06-25 12:16:24 +03:00
Denis Kirillov
fd8130a42f [#92] Remove keys generation 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-06-23 14:57:33 +03:00
Denis Kirillov
b5c8befa25 [#92] Support NEP-6 wallet 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-06-22 17:10:14 +03:00
Denis Kirillov
b86d20265d [#75] Using secp256r1 instead of curve25519 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-06-17 16:36:23 +03:00
Angira Kekteeva
44da6cf70c [#72] authmate: Add session token 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-06-17 14:09:58 +03:00
Roman Khimov
df3746ad03 s3-gw: drop dead code 
cmd/s3-gw/app-settings.go:31:2   deadcode  `defaultKeepaliveTime` is unused
cmd/s3-gw/app-settings.go:32:2   deadcode  `defaultKeepaliveTimeout` is unused

Signed-off-by: Roman Khimov <roman@nspcc.ru>
 2021-06-15 18:47:50 +03:00
Denis Kirillov
2e96ce6dcd [#73] Dropped grpc keepalive options 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-06-11 09:49:32 +03:00
Roman Khimov
3cac112f5f s3-gw: expose TLS options in help 
Signed-off-by: Roman Khimov <roman@nspcc.ru>
 2021-06-10 15:27:13 +03:00
Angira Kekteeva
b3b190adc0 [#68] *: Replace deprecated 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-06-04 16:01:42 +03:00
Angira Kekteeva
1a818bac33 [#64] authmate, auth: Fix access key id 
Replaced forbidden symbol '/' in access key id by '_'
SecretKeyAddress and SecretKeyID are different things now
Renamed param of authmate from secretAddressFlag to accessKeyIDFlag,
that is more correct, imo.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-06-03 22:24:02 +03:00
Roman Khimov
d15687f37c *: drop creds/neofs 
Signed-off-by: Roman Khimov <roman@nspcc.ru>
 2021-05-28 23:48:39 +03:00
Roman Khimov
35ffc1fbc1 *: use sdk-go instead of http-gw 
Now that the packages have moved there we can do it.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
 2021-05-28 14:05:46 +03:00
Roman Khimov
d19ce03072 *: drop old sdk dependecies, bump neofs-api-go version 
I'm not sure it works, but it's enough code-wise for now. We're reusing some
http-gw components here that are to be moved into sdk-go in future.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
 2021-05-26 21:01:46 +03:00
Roman Khimov
dbe65ae602 creds: move credential management into s3 gate 
Mostly taken from old SDK (abe47687cd11266f946cad57f07572cc10c67226), but
error handling adapted to eliminate pkg/errors and internal packages.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
 2021-05-25 23:00:19 +03:00
Roman Khimov
69e3e22dbc cmd: drop grace package dependency 
Note that signal.NotifyContext is a Go 1.16 feature.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
 2021-05-25 13:32:08 +03:00
Roman Khimov
92c8cce933 misc: move to internal/version, use for all binaries 
No libmisc, please.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
 2021-05-20 16:26:41 +03:00
Roman Khimov
ef8684c11d s3-gw: drop build time setting 
It's not properly set anyway and it's a bad practice for build
reproducibility.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
 2021-05-20 16:26:20 +03:00
Roman Khimov
e622f1987b Makefile: deduplicate binaries 
Signed-off-by: Roman Khimov <roman@nspcc.ru>
 2021-05-20 16:26:20 +03:00
Angira Kekteeva
9d496d70a7 [#53] *: Remove external pkg/errors dependency 
Replaced functions from pkg/errors by functions from errors, fixed not fully correct comment

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-05-20 15:50:14 +03:00
Angira Kekteeva
1302c7ae78 [#46] *: Add comments, fix typos 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-05-19 19:27:02 +03:00
Angira Kekteeva
d46578ce0b [#46] authmate: Remove excessive definitions of capacities from slice creations 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-05-19 19:25:06 +03:00
Angira Kekteeva
3c61c69f44 [#46] authmate: Rename s3 package to authmate 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-05-19 12:35:48 +03:00
Angira Kekteeva
50e4eb3b43 [#46] authmate: Copy files from cdn-authmate repository to neofs-s3-gw 
Copy authmate main file and s3 agent

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-05-18 21:49:09 +03:00
Angira Kekteeva
a399590204 Replace s3-gate by s3-gw 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-05-18 14:12:24 +03:00
Roman Khimov
5eb863dc22 *: fix golint warnings about comments to exported things. 2021-05-13 23:25:31 +03:00
Roman Khimov
70a70bfa2c *: fix all godot errors 2021-05-13 22:25:32 +03:00
Roman Khimov
b851889934 *: fix whitespace errors 2021-05-13 22:13:09 +03:00
Roman Khimov
7ce5b3392f cmd/gate: fix gosimple suggestion 
cmd/gate/app-settings.go:108:36  gosimple    S1019: should use make(map[string]float64) instead
 2021-05-13 22:11:55 +03:00
Evgeniy Kulikov
50ef46f63c Fix README and settings 
- remove `generated`, not used anymore
- update `README` section about configuration

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-02 12:23:28 +03:00
Evgeniy Kulikov
2a93a216f8 Migrate to SDK 0.3.0 and fixes 
- fix displaying list objects
- simplify `ListObjects`
- simplify `GetObjectHandler`
- simplify `HeadObjectHandler`
- add method for `GetBucketVersioningHandler`
- add method for `ListMultipartUploadsHandler`
- improvements for `HeadObjectHandler`, to display folders meta
- update dependencies
  - github.com/aws/aws-sdk-go v1.36.26
  - github.com/google/uuid v1.1.4
  - github.com/gorilla/mux v1.8.0
  - github.com/nspcc-dev/cdn-sdk v0.3.0
  - github.com/nspcc-dev/neofs-api-go v1.22.0
  - github.com/prometheus/client_golang v1.9.0
  - github.com/stretchr/testify v1.7.0
  - google.golang.org/grpc v1.35.0

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-01-14 20:39:48 +03:00
Evgeniy Kulikov
793925497a Add listen-domains 
- add listen-domains config key
- add method to fetch domains list to listen

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-12-10 18:11:45 +03:00
Evgeniy Kulikov
709d9f37e8 Prepare to release 
- set output for flag set
- change env prefix for all environments
- migrate to CDN SDK release v0.1.0
- update README

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-12-02 12:46:51 +03:00
Evgeniy Kulikov
f5fb850c3f Change ENV prefix 
TODO should be replaced with S3_GW before release

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-12-01 10:02:26 +03:00
Evgeniy Kulikov
1cd636a24b Refactoring api/auth package 
Migrate to cred/bearer instead of CredentialsClient

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-11-24 19:31:57 +03:00
Evgeniy Kulikov
c1c8d56de5 Migrate cmd/gate to new components 
- remove unused methods
- refactoring app.Worker
- migrate to new pool, client, auth and credentials

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-11-24 10:09:58 +03:00
Evgeniy Kulikov
3bf6a847a2 Remove unused types 
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-11-24 10:07:14 +03:00
Evgeniy Kulikov
de77d7838e Add logger constructor and replace graceful context 
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-11-24 10:06:39 +03:00
Evgeniy Kulikov
8a3d2ba909 Remove unused components 
- remove graceful context
- remove logger component

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-11-24 10:05:53 +03:00
Evgeniy Kulikov
4d605d1113 [#25] Refactoring and make fixes 
closes #25
closes #32

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-10-23 03:21:14 +03:00
Evgeniy Kulikov
14517d682c [#25] Show default environments 
- Add possibility to display default environments
- Add Prefix constant
- Update README

closes #25

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-10-19 11:47:13 +03:00
Evgeniy Kulikov
424fb7a1d1 [#25] Migrate cmd/gate to NeoFS API v2 
Naive migration, without any tries to start application

closes #25

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-10-19 04:05:28 +03:00
Evgeniy Kulikov
86e07c8882 [#25] Fix linter warnings 
closes #25

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-10-13 12:43:24 +03:00
Evgeniy Kulikov
65b5d6e3d2 [#25] Fixes around cmd/gate 
closes #25

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-10-13 12:31:23 +03:00
Pavel Korotkov
8d4017f1d3 [#8] Use proper creds contructor to avoid unnatural convertions 
Signed-off-by: Pavel Korotkov <pkorotkov@gmail.com>
 2020-08-07 14:12:27 +03:00
Pavel Korotkov
c5489da3cc [#8] Check error after setting gate's auth keys 
Signed-off-by: Pavel Korotkov <pkorotkov@gmail.com>
 2020-08-07 14:12:27 +03:00
Pavel Korotkov
4f7a2f3a9a [#8] Rename key-related flags 
Signed-off-by: Pavel Korotkov <pkorotkov@gmail.com>
 2020-08-07 14:12:27 +03:00
Pavel Korotkov
c972682430 [#8] Switch outer code to using the new auth scheme 
* Removed CLI flag for RSA key
* Passed through peers to auth center to be able to independently interact with a NeoFS node
* Added flag and loader for curve25519 (private) key

Signed-off-by: Pavel Korotkov <pkorotkov@gmail.com>
 2020-08-07 14:12:27 +03:00
Evgeniy Kulikov
0161d2fbd3 [#13] Rename go module name according to NSPCC standards 
- refactoring s3 gate structure
- cleanup unused code
- rename go module to `github.com/nspcc-dev/neofs-s3-gate`

closes #13

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-08-06 15:02:13 +03:00
Evgeniy Kulikov
26dda9ef2b Fixes for handler initialize 2020-08-06 13:50:04 +03:00
Evgeniy Kulikov
f7dc2095b1 Fixes for new NeoFS layer client 
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-08-03 18:19:56 +03:00