FrostFS S3 Protocol Gateway
Find a file
Nikita Zinkevich 575ab4d294 [#369] Enhanced http requests logging
Signed-off-by: Nikita Zinkevich <n.zinkevich@yadro.com>
2024-09-11 15:25:09 +03:00
.docker [#369] Enhanced http requests logging 2024-09-11 15:25:09 +03:00
.forgejo/workflows [#470] Update Go version to 1.22 2024-08-23 12:18:42 +03:00
.github [#2] Keep issue templates in .github 2023-06-07 15:35:57 +00:00
api [#369] Enhanced http requests logging 2024-09-11 15:25:09 +03:00
authmate [#372] authmate: Don't create creds with eacl table 2024-07-01 16:26:21 +03:00
cmd [#369] Enhanced http requests logging 2024-09-11 15:25:09 +03:00
config [#369] Enhanced http requests logging 2024-09-11 15:25:09 +03:00
creds [#427] layer: Split FrostFS ReadObject to separate methods 2024-07-23 16:53:59 +03:00
debian [#68] Fix pre-commit issues 2023-03-24 16:22:06 +03:00
docs [#369] Enhanced http requests logging 2024-09-11 15:25:09 +03:00
internal [#369] Enhanced http requests logging 2024-09-11 15:25:09 +03:00
metrics [#447] Add tree pool request duration metric 2024-08-21 06:36:55 +00:00
pkg [#369] Enhanced http requests logging 2024-09-11 15:25:09 +03:00
.dockerignore [#471] Add docker/* target in Makefile 2022-06-16 11:12:42 +03:00
.gitignore [#133] Drop sync-tree 2023-06-09 09:34:36 +03:00
.gitlint [#65] Enable pre-commit 2023-03-24 07:28:04 +00:00
.golangci.yml [#474] lint: Update golangci-lint to v1.60 and fix issues 2024-08-22 19:13:52 +03:00
.pre-commit-config.yaml [#195] Add log constants linter 2023-08-28 12:58:44 +03:00
CHANGELOG.md [#466] Implement PATCH for multipart objects 2024-09-03 11:57:59 +00:00
CONTRIBUTING.md [#2] Update CONTRIBUTING 2023-06-07 15:35:57 +00:00
CREDITS.md [#68] Fix pre-commit issues 2023-03-24 16:22:06 +03:00
go.mod [#369] Enhanced http requests logging 2024-09-11 15:25:09 +03:00
go.sum [#369] Enhanced http requests logging 2024-09-11 15:25:09 +03:00
help.mk [#725] Fix help 2022-10-17 19:16:05 +03:00
LICENSE [#264] Change NeoFS S3 Gateway license to AGPLv3 2021-09-20 10:38:28 +03:00
Makefile [#369] Enhanced http requests logging 2024-09-11 15:25:09 +03:00
README.md [#480] Add fuzzing tests 2024-09-02 15:59:07 +03:00
SECURITY.md [#413] Add SECURITY.md 2024-09-03 11:45:05 +00:00
updateTestsResult.sh [#68] Fix pre-commit issues 2023-03-24 16:22:06 +03:00
VERSION Release v0.30.0 2024-07-19 17:09:58 +03:00

FrostFS logo

FrostFS is a decentralized distributed object storage integrated with the NEO Blockchain.


Report Release License

FrostFS S3 Gateway

FrostFS S3 gateway provides API compatible with Amazon S3 cloud storage service.

Installation

go get -u git.frostfs.info/TrueCloudLab/frostfs-s3-gw

Or you can call make to build it from the cloned repository (the binary will end up in bin/frostfs-s3-gw with authmate helper in bin/frostfs-s3-authmate). To build binaries in clean docker environment, call make docker/all.

Other notable make targets:

dep          Check and ensure dependencies
image        Build clean docker image
dirty-image  Build dirty docker image with host-built binaries
format       Run all code formatters
lint         Run linters
version      Show current version

Or you can also use a Docker image provided for released (and occasionally unreleased) versions of gateway (:latest points to the latest stable release).

Execution

Minimalistic S3 gateway setup needs:

  • FrostFS node(s) address (S3 gateway itself is not a FrostFS node) Passed via -p parameter or via S3_GW_PEERS_<N>_ADDRESS and S3_GW_PEERS_<N>_WEIGHT environment variables (gateway supports multiple FrostFS nodes with weighted load balancing).
  • a wallet used to fetch key and communicate with FrostFS nodes Passed via --wallet parameter or S3_GW_WALLET_PATH environment variable.

These two commands are functionally equivalent, they run the gate with one backend node, some keys and otherwise default settings:

$ frostfs-s3-gw -p 192.168.130.72:8080 --wallet wallet.json

$ S3_GW_PEERS_0_ADDRESS=192.168.130.72:8080 \
  S3_GW_WALLET=wallet.json \
  frostfs-s3-gw

It's also possible to specify uri scheme (grpc or grpcs) when using -p or environment variables:

$ frostfs-s3-gw -p grpc://192.168.130.72:8080 --wallet wallet.json

$ S3_GW_PEERS_0_ADDRESS=grpcs://192.168.130.72:8080 \
  S3_GW_WALLET=wallet.json \
  frostfs-s3-gw

Domains

By default, s3-gw enable only path-style access. To be able to use both: virtual-hosted-style and path-style access you must configure listen_domains:

$ frostfs-s3-gw -p 192.168.130.72:8080 --wallet wallet.json --listen_domains your.first.domain --listen_domains your.second.domain

So now you can use (e.g. HeadBucket. Make sure DNS is properly configured):

$ curl --head http://bucket-name.your.first.domain:8080
HTTP/1.1 200 OK
...

or

$ curl --head http://your.second.domain:8080/bucket-name
HTTP/1.1 200 OK
...

Also, you can configure domains using .env variables or yaml file.

Fuzzing

To run fuzzing tests use the following command:

$ make fuzz

This command will install dependencies for the fuzzing process and run existing fuzzing tests.

You can also use the following arguments:

FUZZ_TIMEOUT - time to run each fuzzing test (default 30) 
FUZZ_FUNCTIONS - fuzzing tests that will be started (default "all")
FUZZ_AUX - additional parameters for the fuzzer (for example, "-debug")
FUZZ_NGFUZZ_DIR - path to ngfuzz tool

Documentation

Credits

Please see CREDITS for details.