mbiryukova/frostfs-http-gw
1
0
Fork 0
forked from TrueCloudLab/frostfs-http-gw
Code Pull requests Activity

[#26] Fix pre-commit issues

Browse source 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
This commit is contained in:
Denis Kirillov 2023-03-24 12:35:22 +03:00
parent 6f35d7198d
commit 7c16ffa250
13 changed files with 51 additions and 54 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
.gitlint
View file

@ -8,4 +8,4 @@ regex=^\[\#[0-9Xx]+\]\s
[ignore-by-title] [ignore-by-title]
regex=^Release(.*) regex=^Release(.*)
ignore=title-match-regex ignore=title-match-regex

2
.pre-commit-config.yaml
View file

@ -42,4 +42,4 @@ repos:
entry: make test entry: make test
pass_filenames: false pass_filenames: false
types: [go] types: [go]
language: system language: system

6
CHANGELOG.md
View file

@ -139,7 +139,7 @@ If you configure application using `.yaml` file change:
### Changed ### Changed
- Updated docs (#133, #140) - Updated docs (#133, #140)
- Increased default read/write timeouts (#154) - Increased default read/write timeouts (#154)
- Updated SDK (#137, #139) - Updated SDK (#137, #139)
- Updated go version to 1.17 (#143) - Updated go version to 1.17 (#143)
- Improved error messages (#144) - Improved error messages (#144)
@ -166,11 +166,11 @@ If you configure application using `.yaml` file change:
- System headers format (#111) - System headers format (#111)
### Added ### Added
- Different formats to set object's expiration: in epoch, duration, timestamp, - Different formats to set object's expiration: in epoch, duration, timestamp,
RFC3339 (#108) RFC3339 (#108)
- Support of nodes priority (#115) - Support of nodes priority (#115)
### Changed ### Changed
- Updated testcontainers dependency (#100) - Updated testcontainers dependency (#100)
## [0.17.0] - 2021-11-15 ## [0.17.0] - 2021-11-15

2
Makefile Normal file → Executable file
View file

@ -21,7 +21,7 @@ OS_RELEASE = $(shell lsb_release -cs)
PKG_VERSION ?= $(shell echo $(VERSION) | sed "s/^v//" | \ PKG_VERSION ?= $(shell echo $(VERSION) | sed "s/^v//" | \
sed -E "s/(.*)-(g[a-fA-F0-9]{6,8})(.*)/\1\3~\2/" | \ sed -E "s/(.*)-(g[a-fA-F0-9]{6,8})(.*)/\1\3~\2/" | \
sed "s/-/~/")-${OS_RELEASE} sed "s/-/~/")-${OS_RELEASE}
.PHONY: debpackage debclean .PHONY: debpackage debclean
# Make all binaries # Make all binaries
all: $(BINS) all: $(BINS)

64
README.md
View file

@ -48,8 +48,8 @@ can be done either via `-p` parameter or via `HTTP_GW_PEERS_<N>_ADDRESS` and
`HTTP_GW_PEERS_<N>_WEIGHT` environment variables (the gate supports multiple `HTTP_GW_PEERS_<N>_WEIGHT` environment variables (the gate supports multiple
FrostFS nodes with weighted load balancing). FrostFS nodes with weighted load balancing).
If you launch HTTP gateway in bundle with [frostfs-dev-env](https://git.frostfs.info/TrueCloudLab/frostfs-dev-env), If you launch HTTP gateway in bundle with [frostfs-dev-env](https://git.frostfs.info/TrueCloudLab/frostfs-dev-env),
you can get the IP address of the node in the output of `make hosts` command you can get the IP address of the node in the output of `make hosts` command
(with s0*.frostfs.devenv name). (with s0*.frostfs.devenv name).
These two commands are functionally equivalent, they run the gate with one These two commands are functionally equivalent, they run the gate with one
@ -86,12 +86,12 @@ $ HTTP_GW_PEERS_0_ADDRESS=192.168.130.71:8080 HTTP_GW_PEERS_0_WEIGHT=1 HTTP_GW_P
HTTP_GW_PEERS_2_ADDRESS=192.168.130.73:8080 HTTP_GW_PEERS_2_WEIGHT=1 HTTP_GW_PEERS_2_PRIORITY=2 \ HTTP_GW_PEERS_2_ADDRESS=192.168.130.73:8080 HTTP_GW_PEERS_2_WEIGHT=1 HTTP_GW_PEERS_2_PRIORITY=2 \
frostfs-http-gw frostfs-http-gw
``` ```
This command will make gateway use 192.168.130.71 while it is healthy. Otherwise, it will make the gateway use This command will make gateway use 192.168.130.71 while it is healthy. Otherwise, it will make the gateway use
192.168.130.72 for 90% of requests and 192.168.130.73 for remaining 10%. 192.168.130.72 for 90% of requests and 192.168.130.73 for remaining 10%.
### Keys ### Keys
You can provide a wallet via `--wallet` or `-w` flag. You can also specify the account address using `--address` You can provide a wallet via `--wallet` or `-w` flag. You can also specify the account address using `--address`
(if no address provided default one will be used). If wallet is used, you need to set `HTTP_GW_WALLET_PASSPHRASE` variable to decrypt the wallet. (if no address provided default one will be used). If wallet is used, you need to set `HTTP_GW_WALLET_PASSPHRASE` variable to decrypt the wallet.
If no wallet provided, the gateway autogenerates a key pair it will use for FrostFS requests. If no wallet provided, the gateway autogenerates a key pair it will use for FrostFS requests.
``` ```
$ frostfs-http-gw -p $FROSTFS_NODE -w $WALLET_PATH --address $ACCOUNT_ADDRESS $ frostfs-http-gw -p $FROSTFS_NODE -w $WALLET_PATH --address $ACCOUNT_ADDRESS
@ -162,7 +162,7 @@ All timing options accept values with suffixes, so "15s" is 15 seconds and
"2m" is 2 minutes. "2m" is 2 minutes.
### Zip streaming ### Zip streaming
The gateway supports downloading files by common prefix (like dir) in zip format. You can enable compression The gateway supports downloading files by common prefix (like dir) in zip format. You can enable compression
using config or `HTTP_GW_ZIP_COMPRESSION=true` environment variable. using config or `HTTP_GW_ZIP_COMPRESSION=true` environment variable.
### Logging ### Logging
@ -172,7 +172,7 @@ HTTP_GW_LOGGER_LEVEL=debug
``` ```
### Yaml file ### Yaml file
Configuration file is optional and can be used instead of environment variables/other parameters. Configuration file is optional and can be used instead of environment variables/other parameters.
It can be specified with `--config` parameter: It can be specified with `--config` parameter:
``` ```
$ frostfs-http-gw --config your-config.yaml $ frostfs-http-gw --config your-config.yaml
@ -207,7 +207,7 @@ supported.
### Preparation ### Preparation
Before uploading or downloading a file make sure you have a prepared container. Before uploading or downloading a file make sure you have a prepared container.
You can create it with instructions below. You can create it with instructions below.
Also, in case of downloading, you need to have a file inside a container. Also, in case of downloading, you need to have a file inside a container.
@ -226,13 +226,13 @@ resolve_order:
- nns - nns
``` ```
2. Make sure your container is registered in NNS contract. If you use [frostfs-dev-env](https://git.frostfs.info/TrueCloudLab/frostfs-dev-env) 2. Make sure your container is registered in NNS contract. If you use [frostfs-dev-env](https://git.frostfs.info/TrueCloudLab/frostfs-dev-env)
you can check if your container (e.g. with `container-name` name) is registered in NNS: you can check if your container (e.g. with `container-name` name) is registered in NNS:
```shell ```shell
$ curl -s --data '{"id":1,"jsonrpc":"2.0","method":"getcontractstate","params":[1]}' \ $ curl -s --data '{"id":1,"jsonrpc":"2.0","method":"getcontractstate","params":[1]}' \
http://morph-chain.frostfs.devenv:30333 | jq -r '.result.hash' http://morph-chain.frostfs.devenv:30333 | jq -r '.result.hash'
0x8e6c3cd4b976b28e84a3788f6ea9e2676c15d667 0x8e6c3cd4b976b28e84a3788f6ea9e2676c15d667
$ docker exec -it morph_chain neo-go \ $ docker exec -it morph_chain neo-go \
@ -241,7 +241,7 @@ $ docker exec -it morph_chain neo-go \
resolve string:container-name.container int:16 \ resolve string:container-name.container int:16 \
| jq -r '.stack[0].value | if type=="array" then .[0].value else . end' \ | jq -r '.stack[0].value | if type=="array" then .[0].value else . end' \
| base64 -d && echo | base64 -d && echo
7f3vvkw4iTiS5ZZbu5BQXEmJtETWbi3uUjLNaSs29xrL 7f3vvkw4iTiS5ZZbu5BQXEmJtETWbi3uUjLNaSs29xrL
``` ```
@ -257,7 +257,7 @@ You can create a container via [frostfs-cli](https://git.frostfs.info/TrueCloudL
``` ```
$ frostfs-cli -r $FROSTFS_NODE -w $WALLET container create --policy $POLICY --basic-acl $ACL $ frostfs-cli -r $FROSTFS_NODE -w $WALLET container create --policy $POLICY --basic-acl $ACL
``` ```
where `$WALLET` is a path to user wallet, where `$WALLET` is a path to user wallet,
`$ACL` -- hex encoded basic ACL value or keywords 'private, 'public-read', 'public-read-write' and `$ACL` -- hex encoded basic ACL value or keywords 'private, 'public-read', 'public-read-write' and
`$POLICY` -- QL-encoded or JSON-encoded placement policy or path to file with it `$POLICY` -- QL-encoded or JSON-encoded placement policy or path to file with it
@ -267,17 +267,17 @@ $ frostfs-cli -r 192.168.130.72:8080 -w ./wallet.json container create --policy
``` ```
If you have launched nodes via [frostfs-dev-env](https://git.frostfs.info/TrueCloudLab/frostfs-dev-env), If you have launched nodes via [frostfs-dev-env](https://git.frostfs.info/TrueCloudLab/frostfs-dev-env),
you can get the key value from `wallets/wallet.json` or write the path to you can get the key value from `wallets/wallet.json` or write the path to
the file `wallets/wallet.key`. the file `wallets/wallet.key`.
#### Prepare a file in a container #### Prepare a file in a container
To create a file via [frostfs-cli](https://git.frostfs.info/TrueCloudLab/frostfs-node/releases), run a command below: To create a file via [frostfs-cli](https://git.frostfs.info/TrueCloudLab/frostfs-node/releases), run a command below:
``` ```
$ frostfs-cli -r $FROSTFS_NODE -k $KEY object put --file $FILENAME --cid $CID $ frostfs-cli -r $FROSTFS_NODE -k $KEY object put --file $FILENAME --cid $CID
``` ```
where where
`$KEY` -- the key, please read the information [above](#create-a-container), `$KEY` -- the key, please read the information [above](#create-a-container),
`$CID` -- container ID. `$CID` -- container ID.
For example: For example:
@ -290,13 +290,13 @@ $ frostfs-cli -r 192.168.130.72:8080 -w ./wallet.json object put --file cat.png
#### Requests #### Requests
The following requests support GET/HEAD methods. The following requests support GET/HEAD methods.
##### By IDs ##### By IDs
Basic downloading involves container ID and object ID and is done via GET Basic downloading involves container ID and object ID and is done via GET
requests to `/get/$CID/$OID` path, where `$CID` is a container ID or its name if NNS is enabled, requests to `/get/$CID/$OID` path, where `$CID` is a container ID or its name if NNS is enabled,
`$OID` is an object's (i.e. your file's) ID. `$OID` is an object's (i.e. your file's) ID.
For example: For example:
@ -317,12 +317,12 @@ can be used as well. The generic syntax for it looks like this:
```/get_by_attribute/$CID/$ATTRIBUTE_NAME/$ATTRIBUTE_VALUE``` ```/get_by_attribute/$CID/$ATTRIBUTE_NAME/$ATTRIBUTE_VALUE```
where where
`$CID` is a container ID or its name if NNS is enabled, `$CID` is a container ID or its name if NNS is enabled,
`$ATTRIBUTE_NAME` is the name of the attribute we want to use, `$ATTRIBUTE_NAME` is the name of the attribute we want to use,
`$ATTRIBUTE_VALUE` is the value of this attribute that the target object should have. `$ATTRIBUTE_VALUE` is the value of this attribute that the target object should have.
**NB!** The attribute key and value should be url encoded, i.e., if you want to download an object with the attribute value **NB!** The attribute key and value should be url encoded, i.e., if you want to download an object with the attribute value
`a cat`, the value in the request must be `a+cat`. In the same way with the attribute key. If you don't escape such values `a cat`, the value in the request must be `a+cat`. In the same way with the attribute key. If you don't escape such values
everything can still work (for example you can use `d@ta` without encoding) but it's HIGHLY RECOMMENDED to encode all your attributes. everything can still work (for example you can use `d@ta` without encoding) but it's HIGHLY RECOMMENDED to encode all your attributes.
@ -346,7 +346,7 @@ Some other user-defined attributes:
$ wget http://localhost:8082/get_by_attribute/Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ/Ololo/100500 $ wget http://localhost:8082/get_by_attribute/Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ/Ololo/100500
``` ```
Or when the attribute includes special symbols: Or when the attribute includes special symbols:
``` ```
$ wget http://localhost:8082/get_by_attribute/Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ/Olo%2Blo/100500 # means Olo+lo $ wget http://localhost:8082/get_by_attribute/Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ/Olo%2Blo/100500 # means Olo+lo
``` ```
@ -365,7 +365,7 @@ You can download some dir (files with the same prefix) in zip (it will be compre
$ wget http://localhost:8082/zip/Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ/common/prefix $ wget http://localhost:8082/zip/Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ/common/prefix
``` ```
**Note:** the objects must have a valid `FilePath` attribute (it should not contain trailing `/`), **Note:** the objects must have a valid `FilePath` attribute (it should not contain trailing `/`),
otherwise they will not be in the zip archive. You can upload file with this attribute using `curl`: otherwise they will not be in the zip archive. You can upload file with this attribute using `curl`:
``` ```
@ -393,7 +393,7 @@ set of reply headers generated using the following rules:
##### Caching strategy ##### Caching strategy
HTTP Gateway doesn't control caching (doesn't anything with the `Cache-Control` header). Caching strategy strictly HTTP Gateway doesn't control caching (doesn't anything with the `Cache-Control` header). Caching strategy strictly
depends on application use case. So it should be carefully done by proxy server. depends on application use case. So it should be carefully done by proxy server.
### Uploading ### Uploading
@ -424,7 +424,7 @@ You can also add some attributes to your file using the following rules:
"X-Attribute-" prefix stripped, that is if you add "X-Attribute-Ololo: "X-Attribute-" prefix stripped, that is if you add "X-Attribute-Ololo:
100500" header to your request the resulting object will get "Ololo: 100500" header to your request the resulting object will get "Ololo:
100500" attribute 100500" attribute
* "X-Attribute-SYSTEM-*" headers are special * "X-Attribute-SYSTEM-*" headers are special
(`-SYSTEM-` part can also be `-system-` or`-System-` (and even legacy `-Neofs-` for some next releases)), they're used to set internal (`-SYSTEM-` part can also be `-system-` or`-System-` (and even legacy `-Neofs-` for some next releases)), they're used to set internal
FrostFS attributes starting with `__SYSTEM__` prefix, for these attributes all FrostFS attributes starting with `__SYSTEM__` prefix, for these attributes all
dashes get converted to underscores and all letters are capitalized. For dashes get converted to underscores and all letters are capitalized. For
@ -445,7 +445,7 @@ There are some reserved headers type of `X-Attribute-SYSTEM-*` (headers are arra
3. `X-Attribute-System-Expiration-Timestamp: 1637574797` 3. `X-Attribute-System-Expiration-Timestamp: 1637574797`
4. `X-Attribute-System-Expiration-RFC3339: 2021-11-22T09:55:49Z` 4. `X-Attribute-System-Expiration-RFC3339: 2021-11-22T09:55:49Z`
which transforms to `X-Attribute-System-Expiration-Epoch`. So you can provide expiration any convenient way. which transforms to `X-Attribute-System-Expiration-Epoch`. So you can provide expiration any convenient way.
--- ---
@ -484,7 +484,7 @@ the corresponding header to the upload request. Accessing the ACL protected data
works the same way. works the same way.
##### Example ##### Example
In order to generate a bearer token, you need to know the container owner key and In order to generate a bearer token, you need to know the container owner key and
the address of the sender who will do the request to FrostFS (in our case, it's a gateway wallet address). the address of the sender who will do the request to FrostFS (in our case, it's a gateway wallet address).
Suppose we have: Suppose we have:
@ -492,7 +492,7 @@ Suppose we have:
* **NhVtreTTCoqsMQV5Wp55fqnriiUCpEaKm3** (token owner address) * **NhVtreTTCoqsMQV5Wp55fqnriiUCpEaKm3** (token owner address)
* **BJeErH9MWmf52VsR1mLWKkgF3pRm3FkubYxM7TZkBP4K** (container id) * **BJeErH9MWmf52VsR1mLWKkgF3pRm3FkubYxM7TZkBP4K** (container id)
Firstly, we need to encode the container id and the sender address to base64 (now it's base58). Firstly, we need to encode the container id and the sender address to base64 (now it's base58).
So use **base58** and **base64** utils. So use **base58** and **base64** utils.
1. Encoding container id: 1. Encoding container id:
@ -540,7 +540,7 @@ $ frostfs-cli util sign bearer-token --from bearer.json --to signed.json -w ./wa
``` ```
Encoding to base64 to use via the header: Encoding to base64 to use via the header:
``` ```
$ base64 -w 0 signed.json $ base64 -w 0 signed.json
# output: Ck4KKgoECAIQBhIiCiCZGdlbN7DPGPMg9rsWqV+p2XdMzUqknRiexewSFp8kmBIbChk17MUri6OJ0X5ftsHzy7NERDNFB4C92PcaGgMIkE4SZgohAxpsb7vfAso1F0X6hrm6WpRS14WsT3/Ct1SMoqRsT89KEkEEGxKi8GjKSf52YqhppgaOTQHbUsL3jn7SHLqS3ndAQ7NtAATnmRHleZw2V2xRRSRBQdjDC05KK83LhdSax72Fsw== # output: Ck4KKgoECAIQBhIiCiCZGdlbN7DPGPMg9rsWqV+p2XdMzUqknRiexewSFp8kmBIbChk17MUri6OJ0X5ftsHzy7NERDNFB4C92PcaGgMIkE4SZgohAxpsb7vfAso1F0X6hrm6WpRS14WsT3/Ct1SMoqRsT89KEkEEGxKi8GjKSf52YqhppgaOTQHbUsL3jn7SHLqS3ndAQ7NtAATnmRHleZw2V2xRRSRBQdjDC05KK83LhdSax72Fsw==
``` ```
@ -599,8 +599,8 @@ File **eacl.json**:
### Metrics and Pprof ### Metrics and Pprof
If enabled, Prometheus metrics are available at `localhost:8084` endpoint If enabled, Prometheus metrics are available at `localhost:8084` endpoint
and Pprof at `localhost:8083/debug/pprof` by default. Host and port can be configured. and Pprof at `localhost:8083/debug/pprof` by default. Host and port can be configured.
See [configuration](./docs/gate-configuration.md). See [configuration](./docs/gate-configuration.md).
## Credits ## Credits

1
debian/control vendored
View file

@ -12,4 +12,3 @@ Package: frostfs-http-gw
Architecture: any Architecture: any
Depends: ${misc:Depends} Depends: ${misc:Depends}
Description: FrostFS HTTP Gateway bridges FrostFS internal protocol and HTTP standard. Description: FrostFS HTTP Gateway bridges FrostFS internal protocol and HTTP standard.

8
debian/copyright vendored
View file

@ -14,12 +14,12 @@ License: GPL-3
This program is free software: you can redistribute it and/or modify it This program is free software: you can redistribute it and/or modify it
under the terms of the GNU General Public License as published under the terms of the GNU General Public License as published
by the Free Software Foundation; version 3. by the Free Software Foundation; version 3.
This program is distributed in the hope that it will be useful, This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
General Public License for more details. General Public License for more details.
You should have received a copy of the GNU General Public License You should have received a copy of the GNU General Public License
along with this program or at /usr/share/common-licenses/GPL-3. along with this program or at /usr/share/common-licenses/GPL-3.
If not, see <http://www.gnu.org/licenses/>. If not, see <http://www.gnu.org/licenses/>.

4
debian/frostfs-http-gw.postinst vendored Normal file → Executable file
View file

@ -29,8 +29,8 @@ case "$1" in
chmod -f 0640 /etc/frostfs/$USERNAME/config.yaml || true chmod -f 0640 /etc/frostfs/$USERNAME/config.yaml || true
fi fi
USERDIR=$(getent passwd "frostfs-$USERNAME" | cut -d: -f6) USERDIR=$(getent passwd "frostfs-$USERNAME" | cut -d: -f6)
if ! dpkg-statoverride --list frostfs-$USERDIR >/dev/null; then if ! dpkg-statoverride --list frostfs-"$USERDIR" >/dev/null; then
chown -f frostfs-$USERNAME: $USERDIR chown -f frostfs-$USERNAME: "$USERDIR"
fi fi
;; ;;

0
debian/frostfs-http-gw.postrm vendored Normal file → Executable file
View file

0
debian/frostfs-http-gw.preinst vendored Normal file → Executable file
View file

0
debian/frostfs-http-gw.prerm vendored Normal file → Executable file
View file

6
debian/rules vendored
View file

@ -8,9 +8,7 @@ SERVICE = frostfs-http-gw
dh $@ dh $@
override_dh_installsystemd: override_dh_installsystemd:
dh_installsystemd --no-enable --no-start $(SERVICE).service dh_installsystemd --no-enable --no-start $(SERVICE).service
override_dh_installchangelogs: override_dh_installchangelogs:
dh_installchangelogs -k CHANGELOG.md dh_installchangelogs -k CHANGELOG.md

10
docs/gate-configuration.md
View file

@ -62,9 +62,9 @@ resolve_order:
- nns - nns
- dns - dns
connect_timeout: 5s connect_timeout: 5s
stream_timeout: 10s stream_timeout: 10s
request_timeout: 5s request_timeout: 5s
rebalance_timer: 30s rebalance_timer: 30s
pool_error_threshold: 100 pool_error_threshold: 100
``` ```
@ -83,8 +83,8 @@ pool_error_threshold: 100
```yaml ```yaml
wallet: wallet:
path: /path/to/wallet.json path: /path/to/wallet.json
address: NfgHwwTi3wHAS8aFAN243C5vGbkYDpqLHP address: NfgHwwTi3wHAS8aFAN243C5vGbkYDpqLHP
passphrase: pwd passphrase: pwd
``` ```
@ -201,7 +201,7 @@ upload_header:
```yaml ```yaml
zip: zip:
compression: false compression: false
``` ```
| Parameter | Type | SIGHUP reload | Default value | Description | | Parameter | Type | SIGHUP reload | Default value | Description |