WIP: FrostFS HTTP Gateway
Find a file
Roman Khimov ed3c576b70 uploader: drain body buffer before ending request processing
Fixes

2021-04-28T00:03:36.514+0300    debug   uploader/upload.go:64   close temporary multipart/form file     {"cid": "Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ", "address": "Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ/2m8PtaoricLouCn5zE8hAFr3gZEBDCZFe9BEgVJTSocX", "filename": "vid.mp4"}
2021/04/28 00:03:36 error when serving connection "127.0.0.1:8082"<->"127.0.0.1:41390": error when reading request headers: cannot find http request method in "0\r\n\r\n". Buffer size=5, contents: "0\r\n\r\n"
2021-04-30 14:31:14 +03:00
connections connections: normalize weights, make config a bit more human-oriented 2021-04-15 09:45:43 +03:00
docs Rename how-to-check.mc to how-to-check.md 2020-12-03 19:12:00 +03:00
downloader Put artifacts into pool solely 2021-04-08 12:39:29 +03:00
global Add connection pool implementation (part 2) 2021-04-06 12:19:21 +03:00
logger [#19] Add a version with no cdn-sdk deps 2021-04-06 12:19:21 +03:00
neofs neofs: drop PrepareObjectOnsite option 2021-04-30 14:31:14 +03:00
tokens [#19] Extract uploading logic into a separate package 2021-04-06 12:19:21 +03:00
uploader uploader: drain body buffer before ending request processing 2021-04-30 14:31:14 +03:00
.gitignore Clean up Makefile and image build 2021-04-12 17:09:39 +03:00
.golangci.yml Clean up Makefile and image build 2021-04-12 17:09:39 +03:00
app.go app: drop FIXME comment 2021-04-30 14:31:14 +03:00
Dockerfile Clean up Makefile and image build 2021-04-12 17:09:39 +03:00
Dockerfile.dirty Clean up Makefile and image build 2021-04-12 17:09:39 +03:00
go.mod neofs: drop PrepareObjectOnsite option 2021-04-30 14:31:14 +03:00
go.sum neofs: drop PrepareObjectOnsite option 2021-04-30 14:31:14 +03:00
LICENSE.md NSPCC-587 Add License and Readme 2019-11-07 20:29:29 +03:00
main.go app: drop unused Worker/jobDone 2021-04-30 14:31:14 +03:00
Makefile Clean up Makefile and image build 2021-04-12 17:09:39 +03:00
metrics.go metrics: fix doc-comments 2020-05-12 11:20:28 +03:00
misc.go Clean up Makefile and image build 2021-04-12 17:09:39 +03:00
pprof.go pprof: fix route params 2020-05-12 11:20:28 +03:00
README.md app: add support for TLS key/cert options 2021-04-15 20:24:49 +03:00
settings.go app: add support for TLS key/cert options 2021-04-15 20:24:49 +03:00

NeoFS HTTP Protocol Gateway

NeoFS HTTP Protocol Gateway bridges NeoFS internal protocol and HTTP standard.

  • you can download one file per request from NeoFS Network
  • you can upload one file per request into the NeoFS Network

Notable make targets

dep          Check and ensure dependencies
image        Build clean docker image
dirty-image  Build diry docker image with host-built binaries
fmts         Run all code formatters
lint         Run linters
version      Show current version

Install

go get -u github.com/nspcc-dev/neofs-http-gate

File uploading behaviors

  • you can upload on file per request
  • if FileName not provided by Header attributes, multipart/form filename will be used instead

Configuration

# Flags:

      --pprof                      enable pprof
      --metrics                    enable prometheus
  -h, --help                       show help
  -v, --version                    show version
      --key string                 path to private key file, hex string or wif (the key will be autogenerated if not specified)
      --verbose                    debug gRPC connections
      --request_timeout duration   gRPC request timeout (default 5s)
      --connect_timeout duration   gRPC connect timeout (default 30s)
      --listen_address string      HTTP gate's listen address (default "0.0.0.0:8082")
      --tls_certificate string     TLS certificate path
      --tls_key string             TLS key path
  -p, --peers stringArray          NeoFS nodes

# Environments:

HTTP_GW_KEY=string                               - Path to private key file, hex string or wif string
HTTP_GW_CONNECT_TIMEOUT=duration                 - Timeout for connection
HTTP_GW_REQUEST_TIMEOUT=duration                 - Timeout for request
HTTP_GW_REBALANCE_TIMER=duration                 - Time between connections checks
HTTP_GW_LISTEN_ADDRESS=host:port                 - Address to listen connections
HTTP_GW_TLS_CERTIFICATE=path                     - File with TLS certificate
HTTP_GW_TLS_KEY=path                             - File with TLS private key
HTTP_GW_PEERS_<X>_ADDRESS=host:port              - Address of NeoFS Node
HTTP_GW_PEERS_<X>_WEIGHT=float                   - Weight of NeoFS Node (1 if not specified)
HTTP_GW_PPROF=bool                               - Enable/disable pprof (/debug/pprof)
HTTP_GW_METRICS=bool                             - Enable/disable prometheus metrics endpoint (/metrics)
HTTP_GW_LOGGER_FORMAT=string                     - Logger format
HTTP_GW_LOGGER_LEVEL=string                      - Logger level
HTTP_GW_LOGGER_NO_CALLER=bool                    - Logger don't show caller
HTTP_GW_LOGGER_NO_DISCLAIMER=bool                - Logger don't show application name/version
HTTP_GW_LOGGER_SAMPLING_INITIAL=int              - Logger sampling initial
HTTP_GW_LOGGER_SAMPLING_THEREAFTER=int           - Logger sampling thereafter
HTTP_GW_LOGGER_TRACE_LEVEL=string                - Logger show trace on level
HTTP_GW_KEEPALIVE_TIME=duration                  - After a duration of this time if the client sees no activity
                                                   it pings the server to see if the transport is still alive 
HTTP_GW_KEEPALIVE_TIMEOUT=duration               - After having pinged for keepalive check, the client waits for a duration
                                                   of Timeout and if no activity is seen even after that the connection
                                                   is closed
HTTP_GW_KEEPALIVE_PERMIT_WITHOUT_STREAM=bool     - If true, client sends keepalive pings even with no active RPCs.
                                                   If false, when there are no active RPCs, Time and Timeout will be
                                                   ignored and no keepalive pings will be sent
HTTP_GW_UPLOAD_HEADER_USE_DEFAULT_TIMESTAMP=bool - Enable/disable adding current timestamp attribute when object uploads

HTTP_GW_WEB_READ_BUFFER_SIZE=4096          - per-connection buffer size for requests' reading
HTTP_GW_WEB_READ_TIMEOUT=15s               - an amount of time allowed to read the full request including body
HTTP_GW_WEB_WRITE_BUFFER_SIZE=4096         - per-connection buffer size for responses' writing
HTTP_GW_WEB_WRITE_TIMEOUT=1m0s             - maximum duration before timing out writes of the response
HTTP_GW_WEB_STREAM_REQUEST_BODY=true       - enables request body streaming, and calls the handler sooner when given 
                                             body is larger then the current limit
HTTP_GW_WEB_MAX_REQUEST_BODY_SIZE=4194304  - maximum request body size, server rejects requests with bodies exceeding
                                             this limit

Peers preset:

HTTP_GW_PEERS_<N>_ADDRESS = string
HTTP_GW_PEERS_<N>_WEIGHT = float