WIP: FrostFS HTTP Gateway
Find a file
Evgeniy Kulikov cbaf9e6142
Fixes after review
After discussion, we decided to simplify attribute translation for now. Full translation requires support for UTF-8 values encoded according to RFC 8187/7230, this can be clarified and implemented in further PRs.

1. Remove translation tables
2. Use simple translation rules only, for now ignoring UTF-8 keys and values

- `X-Attribute-NEOFS-` prefixed headers considered well-known system attributes
- System Attribute key is uppercased
- System Attribute key's `-` translates to `_`
- `X-Attribute-` prefixed headers considered regular object attributes
- Normal attribute key is a result of http header prefix trimming
- Value string should be left as is, for now

```
HTTP:
X-Attribute-NEOFS-Expiration-Epoch: 123
X-Attribute-FileName: cat.jpg

NeoFS:
__NEOFS__EXPIRATION_EPOCH: "123"
FileName: "cat.jpg"
```

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2021-02-12 15:21:53 +03:00
docs Rename how-to-check.mc to how-to-check.md 2020-12-03 19:12:00 +03:00
.gitignore ignore blast benchmark config 2020-02-28 19:52:33 +03:00
.test.env Fixes around get_by_attribute 2021-01-23 16:28:12 +03:00
app.go Fixes after review 2021-02-12 15:21:53 +03:00
Dockerfile Prepare to release 2020-12-02 12:25:18 +03:00
filter.go Fixes after review 2021-02-12 15:21:53 +03:00
go.mod Update dependencies 2021-02-12 12:19:43 +03:00
go.sum Update dependencies 2021-02-12 12:19:43 +03:00
go_dev.mod change module name 2020-08-21 02:50:14 +03:00
health.go Migrate to CDN SDK 2020-11-10 14:46:28 +03:00
help.mk Update Makefile and targets 2021-02-05 16:51:20 +03:00
LICENSE.md NSPCC-587 Add License and Readme 2019-11-07 20:29:29 +03:00
main.go Migrate to SDK 0.3.0 2021-01-14 13:31:40 +03:00
Makefile Update README 2021-02-05 17:13:37 +03:00
metrics.go metrics: fix doc-comments 2020-05-12 11:20:28 +03:00
misc.go Prepare to release 2020-12-02 12:25:18 +03:00
pprof.go pprof: fix route params 2020-05-12 11:20:28 +03:00
README.md Fixes after review 2021-02-12 15:21:53 +03:00
receive.go Fixes around get_by_attribute 2021-01-23 16:28:12 +03:00
settings.go Fixes after review 2021-02-12 15:21:53 +03:00
upload.go Fixes after review 2021-02-12 15:21:53 +03:00

NeoFS HTTP Gate

NeoFS HTTP Gate is example of tool that provides basic interactions with NeoFS.

  • you can download one file per request from NeoFS Network using NeoFS Gate
  • you can upload one file per request into NeoFS Network using NeoFS Gate

Notable make targets

  Usage:

    make <target>

  Targets:

    deps      Check and ensure dependencies
    dev       Build development docker images
    help      Show this help prompt
    image     Build docker image
    publish   Publish docker image
    version   Show current version

Install

go get -u github.com/nspcc-dev/neofs-http-gate

File uploading behaviors

  • you can upload on file per request
  • if FileName not provided by Header attributes, multipart/form filename will be used instead

Configuration

# Flags
      --pprof                      enable pprof
      --metrics                    enable prometheus
  -h, --help                       show help
  -v, --version                    show version
      --key string                 "generated" to generate key, path to private key file, hex string or wif (default "generated")
      --verbose                    debug gRPC connections
      --request_timeout duration   gRPC request timeout (default 5s)
      --connect_timeout duration   gRPC connect timeout (default 30s)
      --listen_address string      HTTP Gate listen address (default "0.0.0.0:8082")
  -p, --peers stringArray          NeoFS nodes

# Environments:

HTTP_GW_KEY=string                           - "generated" to generate key, path to private key file, hex string or wif (default "generated")
HTTP_GW_CONNECT_TIMEOUT=Duration             - timeout for connection
HTTP_GW_REQUEST_TIMEOUT=Duration             - timeout for request
HTTP_GW_REBALANCE_TIMER=Duration             - time between connections checks
HTTP_GW_LISTEN_ADDRESS=host:port             - address to listen connections
HTTP_GW_PEERS_<X>_ADDRESS=host:port          - address of NeoFS Node
HTTP_GW_PEERS_<X>_WEIGHT=float               - weight of NeoFS Node
HTTP_GW_PPROF=bool                           - enable/disable pprof (/debug/pprof)
HTTP_GW_METRICS=bool                         - enable/disable prometheus metrics endpoint (/metrics)
HTTP_GW_LOGGER_FORMAT=string                 - logger format
HTTP_GW_LOGGER_LEVEL=string                  - logger level
HTTP_GW_LOGGER_NO_CALLER=bool                - logger don't show caller
HTTP_GW_LOGGER_NO_DISCLAIMER=bool            - logger don't show application name/version
HTTP_GW_LOGGER_SAMPLING_INITIAL=int          - logger sampling initial
HTTP_GW_LOGGER_SAMPLING_THEREAFTER=int       - logger sampling thereafter
HTTP_GW_LOGGER_TRACE_LEVEL=string            - logger show trace on level
HTTP_GW_KEEPALIVE_TIME=Duration              - аfter a duration of this time if the client doesn't see any activity
it pings the server to see if the transport is still alive. 
HTTP_GW_KEEPALIVE_TIMEOUT=Duration           - after having pinged for keepalive check, the client waits for a duration
of Timeout and if no activity is seen even after that the connection is closed
HTTP_GW_KEEPALIVE_PERMIT_WITHOUT_STREAM=Bool - if true, client sends keepalive pings even with no active RPCs.
If false, when there are no active RPCs, Time and Timeout will be ignored and no keepalive pings will be sent.

HTTP_GW_UPLOAD_HEADER_USE_DEFAULT_TIMESTAMP=bool - enable/disable adding current timestamp attribute when object uploads

Peers preset:

HTTP_GW_PEERS_[N]_ADDRESS = string
HTTP_GW_PEERS_[N]_WEIGHT = 0..1 (float)