+
-
+
FrostFS local Development and Testing environment @@ -137,66 +137,6 @@ Display addresses and host names for each running service, if available. Clean up `vendor` directory. -### s3cred - -Registers user wallet and issues s3 credentials. - -Usage and default parameter values: -```sh -make s3cred [password=""] [contract_password=s3] [wallet=""] [gate_public_key=0313b1ac3a8076e155a7e797b24f0b650cccad5941ea59d7cfd51a024a8b2a06bf] -``` - -As soon as the storage node is in the network map (see above) you can generate S3 -credentials: - -``` sh -$ make s3cred -{ - "access_key_id": "EXArWh8x1zeHG3851s1RtoCo7dowxF6rhLGA15nbMffT0AKRSjJ5fmcqf3Ht2VCAkfmPQUVARghRB77xHCA1BoN2p", - "secret_access_key": "d70c1dba83f0f90bb231f06f1ce0e0dfbcfb122f4b4345a3c18d3869c359b79f", - "owner_private_key": "140947599afd9ca89af4b358c3176eb046e554d942a0dc99a8e06f3e43c8f4ad", - "wallet_public_key": "0324e76288fcb900100d01802a14ef977cca45ad073561230446df14b344c858b6", - "container_id": "EXArWh8x1zeHG3851s1RtoCo7dowxF6rhLGA15nbMffT" -} -``` -Running without any parameters results in defaults which are based on the private key from -`/wallets/wallet.json` user wallet and `/wallet.json` contract wallet. -If `wallet` parameter is set, gate searches custom user wallet file in `/wallets` directory. - -Now let's configure an S3 client (AWS CLI will be used as example): - -``` sh -$ aws configure -AWS Access Key ID []: EXArWh8x1zeHG3851s1RtoCo7dowxF6rhLGA15nbMffT0AKRSjJ5fmcqf3Ht2VCAkfmPQUVARghRB77xHCA1BoN2p -AWS Secret Access Key []: d70c1dba83f0f90bb231f06f1ce0e0dfbcfb122f4b4345a3c18d3869c359b79f -Default region name []: us-east-1 -Default output format []: json -``` - -If you need to create credentials for different users, put user wallet to `wallets` dir and specify it via `wallet` parameter. -Pass wallet password in `password` parameter if it's not default. The same is for `contract_wallet` and `gate_public_key` params. - -```sh -$ make s3cred wallet=custom_wallet.json password=test -{ - "access_key_id": "jHhL5B33o16R4jQsb8wm9A3RRdS6KrTB5N4bja9Jys904W7xXFNKqem2ACvTRWRYJsZMCUikYFSokN7pPJziWyDi", - "secret_access_key": "21bb64fafa32c82417fd8b97ac56cc8a085998a3852632d52fe7042453daa440", - "owner_private_key": "10f6f9d7a47bb0bf68363ad8a99fe69f1493f8b6e1665b3e4e83feb2d5c7ee39", - "wallet_public_key": "03e38759973a6bb722baabc2dd84036a39f0b2f53d32fec45a4dacde8a50fe4b70", - "container_id": "jHhL5B33o16R4jQsb8wm9A3RRdS6KrTB5N4bja9Jys9" -} -``` - -To get credentials from custom wallet, place it in `wallets` dir before start. - -### cred - -Usage and default parameter values: -```sh -make cred [password=""] [contract_password=s3] [wallet=/user_wallet.json] -``` -The same as `s3cred`, but it doesn't issues s3 credentials. - ## Contributing Feel free to contribute to this project after reading the [contributing diff --git a/cli-cfg.yml b/cli-cfg.yml deleted file mode 100644 index 9a5c7eb..0000000 --- a/cli-cfg.yml +++ /dev/null @@ -1,3 +0,0 @@ -wallet: ./wallets/system-wallet.json -password: "" -rpc-endpoint: s01.frostfs.devenv:8080 diff --git a/docs/http_gate.md b/docs/http_gate.md index b6b2591..e2159ad 100644 --- a/docs/http_gate.md +++ b/docs/http_gate.md @@ -22,8 +22,8 @@ Image label prefix to use for containers. - Create a new container ``` $ frostfs-cli --rpc-endpoint s01.frostfs.devenv:8080 \ - --wallet wallets/wallet.key \ - container create --basic-acl private --await \ + --key wallets/wallet.key \ + container create --basic-acl readonly --await \ --policy "REP 1 SELECT 1 FROM *" container ID: 4LfREK1cetL4PUji5fqj9SgRTSmaC5jExEDK9HKCDjdP awaiting... @@ -33,7 +33,7 @@ container has been persisted on sidechain - Put an object into the newly created container ``` $ frostfs-cli --rpc-endpoint s01.frostfs.devenv:8080 \ - --wallet wallets/wallet.key \ + --key wallets/wallet.key \ object put --file /tmp/backup.jpeg \ --cid 4LfREK1cetL4PUji5fqj9SgRTSmaC5jExEDK9HKCDjdP [/tmp/backup.jpeg] Object successfully stored diff --git a/docs/morph.md b/docs/morph.md index d713bf9..56edfbc 100644 --- a/docs/morph.md +++ b/docs/morph.md @@ -4,16 +4,14 @@ A single-node N3 privnet deployment, running on Contracts deployed: - Alphabet (AZ) [contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/alphabet) +- Audit [contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/audit) - Balance [contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/balance) - Container [contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/container) -- FrostFS [contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/frostfs) -- FrostFSID [contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/frostfsid) -- NNS [contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/nns) - Netmap [contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/netmap) -- Policy [contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/policy) -- Processing [contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/processing) +- NeoFSID [contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/neofsid) - Proxy [contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/proxy) - +- Reputation [contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/reputation) + RPC available at `http://morph-chain.frostfs.devenv:30333`. ## .env settings diff --git a/services/basenet/docker-compose.yml b/services/basenet/docker-compose.yml index b2905a3..abac92a 100644 --- a/services/basenet/docker-compose.yml +++ b/services/basenet/docker-compose.yml @@ -1,5 +1,6 @@ --- +version: "2.4" services: basenet: diff --git a/services/grafana/docker-compose.yml b/services/grafana/docker-compose.yml index 3953400..90b67f2 100644 --- a/services/grafana/docker-compose.yml +++ b/services/grafana/docker-compose.yml @@ -1,3 +1,4 @@ +version: '2.4' services: grafana: image: ${GRAFANA_IMAGE}:${GRAFANA_VERSION} diff --git a/services/http_gate/cfg/config.yml b/services/http_gate/cfg/config.yml index 2f15e44..1684c58 100644 --- a/services/http_gate/cfg/config.yml +++ b/services/http_gate/cfg/config.yml @@ -22,6 +22,3 @@ server: wallet: path: /wallet.json # Path to wallet passphrase: one # Passphrase to decrypt wallet - -containers: - cors: cors.container diff --git a/services/http_gate/docker-compose.yml b/services/http_gate/docker-compose.yml index c2e68ae..7108ee5 100644 --- a/services/http_gate/docker-compose.yml +++ b/services/http_gate/docker-compose.yml @@ -1,5 +1,6 @@ --- +version: "2.4" services: http_gate: image: ${HTTP_GW_IMAGE}:${HTTP_GW_VERSION} diff --git a/services/ir/artifacts.mk b/services/ir/artifacts.mk index 1c2cf5d..95b2c22 100644 --- a/services/ir/artifacts.mk +++ b/services/ir/artifacts.mk @@ -25,14 +25,19 @@ endif # Download FrostFS CLI .ONESHELL: get.cli: FROSTFS_CLI_FILE=./vendor/frostfs-cli +get.cli: FROSTFS_CLI_ARCHIVE_FILE=${FROSTFS_CLI_FILE}.tar.gz get.cli: FROSTFS_CLI_PATH?= get.cli: @mkdir -p ./vendor ifeq (${FROSTFS_CLI_PATH},) - echo "⇒ Download FrostFS CLI binary from ${FROSTFS_CLI_IMAGE}:${FROSTFS_CLI_VERSION}" - $(shell docker cp `docker create --name tmp ${FROSTFS_CLI_IMAGE}:${FROSTFS_CLI_VERSION}`:/bin/frostfs-cli ${FROSTFS_CLI_FILE} && docker rm tmp >/dev/null) - chmod +x ${FROSTFS_CLI_FILE} + @echo "⇒ Download FrostFS CLI binary from ${FROSTFS_CLI_URL}" + @curl \ + -ksSL "${FROSTFS_CLI_URL}" \ + -o ${FROSTFS_CLI_ARCHIVE_FILE} + @tar -xvf ${FROSTFS_CLI_ARCHIVE_FILE} -C ./vendor | xargs -I {} \ + mv ./vendor/{} ${FROSTFS_CLI_FILE} + @rm ${FROSTFS_CLI_ARCHIVE_FILE} else @echo "⇒ Copy local binary from ${FROSTFS_CLI_PATH}" @cp ${FROSTFS_CLI_PATH} ${FROSTFS_CLI_FILE} diff --git a/services/ir/docker-compose.yml b/services/ir/docker-compose.yml index 4e671b2..77ec50d 100644 --- a/services/ir/docker-compose.yml +++ b/services/ir/docker-compose.yml @@ -1,5 +1,6 @@ --- +version: "2.4" services: ir01: @@ -12,19 +13,19 @@ services: ir_int: internet: ipv4_address: ${IPV4_PREFIX}.61 - stop_signal: SIGTERM - stop_grace_period: 15s + stop_signal: SIGKILL volumes: - ./az.json:/wallet.json - ./az.key:/wallet01.key - ./../../vendor/hosts:/etc/hosts - ./../../vendor/locode_db:/locode/db - ./../../vendor/frostfs-cli:/frostfs-cli + - ./healthcheck.sh:/healthcheck.sh - ./cfg:/etc/frostfs/ir env_file: [ ".env", ".ir.env", ".int_test.env" ] command: [ "frostfs-ir", "--config", "/etc/frostfs/ir/config.yml" ] healthcheck: - test: ["CMD-SHELL", "/frostfs-cli control ir healthcheck -q --wallet /wallet01.key --endpoint \"$$FROSTFS_IR_CONTROL_GRPC_ENDPOINT\""] + test: ["CMD-SHELL", "/healthcheck.sh"] interval: 2s timeout: 1s retries: 5 diff --git a/services/ir/healthcheck.sh b/services/ir/healthcheck.sh new file mode 100755 index 0000000..747bc8b --- /dev/null +++ b/services/ir/healthcheck.sh @@ -0,0 +1,6 @@ +#!/bin/sh + +/frostfs-cli control healthcheck \ + --endpoint "$FROSTFS_IR_CONTROL_GRPC_ENDPOINT" \ + --wallet /wallet01.key --ir | + grep "Health status: READY" diff --git a/services/jaeger/docker-compose.yml b/services/jaeger/docker-compose.yml index 1fb367b..a460548 100644 --- a/services/jaeger/docker-compose.yml +++ b/services/jaeger/docker-compose.yml @@ -1,3 +1,4 @@ +version: '2.4' services: jaeger: image: ${JAEGER_IMAGE}:${JAEGER_VERSION} diff --git a/services/morph_chain/artifacts.mk b/services/morph_chain/artifacts.mk index 4a4504a..3572e69 100644 --- a/services/morph_chain/artifacts.mk +++ b/services/morph_chain/artifacts.mk @@ -20,12 +20,15 @@ endif # Download FrostFS ADM tool get.adm: FROSTFS_ADM_DEST=./vendor/frostfs-adm +get.adm: FROSTFS_ADM_ARCHIVE=frostfs-adm.tar.gz get.adm: ifeq (${FROSTFS_ADM_PATH},) - @echo "⇒ Download FrostFS ADM binary from ${FROSTFS_ADM_IMAGE}:${FROSTFS_ADM_VERSION}" - $(shell docker cp `docker create --name tmp ${FROSTFS_ADM_IMAGE}:${FROSTFS_ADM_VERSION}`:/bin/frostfs-adm ${FROSTFS_ADM_DEST} && docker rm tmp >/dev/null) - @chmod +x ${FROSTFS_ADM_DEST} + @echo "⇒ Download FrostFS ADM binary from ${FROSTFS_ADM_URL}" + @curl -skSL ${FROSTFS_ADM_URL} -o ${FROSTFS_ADM_ARCHIVE} + @tar -xvf ${FROSTFS_ADM_ARCHIVE} -C ./vendor | xargs -I {} \ + mv ./vendor/{} ${FROSTFS_ADM_DEST} + @rm ${FROSTFS_ADM_ARCHIVE} else @echo "⇒ Copy frostfs-adm binary from ${FROSTFS_ADM_PATH}" @cp ${FROSTFS_ADM_PATH} ${FROSTFS_ADM_DEST} diff --git a/services/morph_chain/docker-compose.yml b/services/morph_chain/docker-compose.yml index c3a1481..4cba4b3 100644 --- a/services/morph_chain/docker-compose.yml +++ b/services/morph_chain/docker-compose.yml @@ -1,5 +1,6 @@ --- +version: "2.4" services: frostfs_morph_chain: image: ${NEOGO_IMAGE}:${NEOGO_VERSION} @@ -19,19 +20,9 @@ services: - ./config.yml:/wallets/config.yml - ./../../vendor/hosts:/etc/hosts - ./../../wallets/wallet.json:/wallets/wallet.json - - ./../../wallets/system-wallet.json:/wallets/system-wallet.json - - ./../s3_gate/wallet.json:/wallets/s3-wallet.json - - ./../storage/wallet01.json:/wallets/storage/wallet01.json - - ./../storage/wallet02.json:/wallets/storage/wallet02.json - - ./../storage/wallet03.json:/wallets/storage/wallet03.json - - ./../storage/wallet04.json:/wallets/storage/wallet04.json - - chains:/chains networks: chain_int: internet: external: true name: basenet_internet - -volumes: - chains: diff --git a/services/morph_chain/protocol.privnet.yml b/services/morph_chain/protocol.privnet.yml index 7cea631..560730a 100644 --- a/services/morph_chain/protocol.privnet.yml +++ b/services/morph_chain/protocol.privnet.yml @@ -11,14 +11,13 @@ ProtocolConfiguration: VerifyTransactions: true StateRootInHeader: true P2PSigExtensions: true - Hardforks: {} ApplicationConfiguration: SkipBlockVerification: false DBConfiguration: Type: "boltdb" BoltDBOptions: - FilePath: "/chains/morph.bolt" + FilePath: "./db/morph.bolt" P2P: Addresses: - ":20333" @@ -37,7 +36,7 @@ ApplicationConfiguration: Password: "one" RPC: Addresses: - - ":30333" + - "192.168.130.90:30333" Enabled: true SessionEnabled: true EnableCORSWorkaround: false diff --git a/services/s3_lifecycler/.env b/services/nats/.env similarity index 100% rename from services/s3_lifecycler/.env rename to services/nats/.env diff --git a/services/nats/.hosts b/services/nats/.hosts new file mode 100644 index 0000000..6ed5417 --- /dev/null +++ b/services/nats/.hosts @@ -0,0 +1 @@ +IPV4_PREFIX.101 nats.LOCAL_DOMAIN diff --git a/services/s3_lifecycler/.int_test.env b/services/nats/.int_test.env similarity index 100% rename from services/s3_lifecycler/.int_test.env rename to services/nats/.int_test.env diff --git a/services/nats/artifacts.mk b/services/nats/artifacts.mk new file mode 100644 index 0000000..b66cebf --- /dev/null +++ b/services/nats/artifacts.mk @@ -0,0 +1,7 @@ +# Create new TLS certs for NATS server and clients + +NATS_DIR=$(abspath services/nats) + +get.nats: + @echo "⇒ Creating certs for NATS server and clients" + ${NATS_DIR}/generate_cert.sh ${LOCAL_DOMAIN} > /dev/null diff --git a/services/nats/docker-compose.yml b/services/nats/docker-compose.yml new file mode 100644 index 0000000..d05dd04 --- /dev/null +++ b/services/nats/docker-compose.yml @@ -0,0 +1,31 @@ +--- + +version: "2.4" +services: + nats: + image: ${NATS_IMAGE}:${NATS_VERSION} + domainname: ${LOCAL_DOMAIN} + hostname: nats + container_name: nats + restart: on-failure + dns: + - ${IPV4_PREFIX}.101 + networks: + nats_int: + internet: + ipv4_address: ${IPV4_PREFIX}.101 + volumes: + - ./../../vendor/hosts:/etc/hosts + - ./nats.conf:/etc/nats/frostfs-nats-server.conf + - ./server-cert.pem:/certs/server-cert.pem + - ./server-key.pem:/certs/server-key.pem + - ./ca-cert.pem:/certs/ca-cert.pem + stop_signal: SIGKILL + env_file: [ ".env", ".int_test.env" ] + command: ["-c", "/etc/nats/frostfs-nats-server.conf"] + +networks: + nats_int: + internet: + external: true + name: basenet_internet diff --git a/services/nats/generate_cert.sh b/services/nats/generate_cert.sh new file mode 100755 index 0000000..4c4dd0b --- /dev/null +++ b/services/nats/generate_cert.sh @@ -0,0 +1,49 @@ +#!/bin/bash + +source bin/helper.sh + +WORKDIR=$(dirname "$0") +LOCAL_DOMAIN=$1 + +CA_KEY=$WORKDIR/ca-key.pem +CA_CRT=$WORKDIR/ca-cert.pem + +SRV_KEY=$WORKDIR/server-key.pem +SRV_REQ=$WORKDIR/server-req.csr +SRV_CRT=$WORKDIR/server-cert.pem + +CLI_KEY=$WORKDIR/client-key.pem +CLI_REQ=$WORKDIR/client-req.csr +CLI_CRT=$WORKDIR/client-cert.pem + +SUBJ="/O=TrueCloudLab" + +if [[ ! -f $CA_KEY || ! -f $CA_CRT ]]; then + openssl req -newkey rsa:4096 -x509 -days 365 -nodes -keyout $CA_KEY -out $CA_CRT -subj $SUBJ 2>&1 || + die "CA certificate was not created" +fi + +if [[ ! -f $SRV_KEY || ! -f $SRV_CRT ]]; then + openssl req -newkey rsa:4096 -nodes -keyout $SRV_KEY -out $SRV_REQ -subj $SUBJ 2>&1 || + die "Server certificate was not created" + + openssl x509 -req -days 365 -set_serial 01 -in $SRV_REQ -out $SRV_CRT -CA $CA_CRT -CAkey $CA_KEY \ + -extensions san -extfile <(printf "[san]\nsubjectAltName=DNS:nats.$LOCAL_DOMAIN") 2>&1 || { + rm $SRV_REQ + die "Server certificate was not signed by CA" + } + + rm $SRV_REQ +fi + +if [[ ! -f $CLI_KEY || ! -f $CLI_CRT ]]; then + openssl req -newkey rsa:4096 -nodes -keyout $CLI_KEY -out $CLI_REQ -subj $SUBJ 2>&1 || + die "Client certificate was not created" + + openssl x509 -req -days 365 -set_serial 01 -in $CLI_REQ -out $CLI_CRT -CA $CA_CRT -CAkey $CA_KEY 2>&1 || { + rm $CLI_REQ + die "Client certificate was not signed by CA" + } + + rm $CLI_REQ +fi diff --git a/services/nats/nats.conf b/services/nats/nats.conf new file mode 100644 index 0000000..0c7af0a --- /dev/null +++ b/services/nats/nats.conf @@ -0,0 +1,15 @@ +port: 4222 +monitor_port: 8222 + +jetstream { + store_dir=nats + max_memory_store: 1GB + max_file_store: 2GB +} + +tls { + cert_file: /certs/server-cert.pem + key_file: /certs/server-key.pem + ca_file: /certs/ca-cert.pem + verify: true +} diff --git a/services/prometheus/docker-compose.yml b/services/prometheus/docker-compose.yml index 31f5fbe..e9eb4f9 100644 --- a/services/prometheus/docker-compose.yml +++ b/services/prometheus/docker-compose.yml @@ -1,3 +1,4 @@ +version: '2.4' services: prometheus: image: ${PROMETHEUS_IMAGE}:${PROMETHEUS_VERSION} diff --git a/services/rest_gate/.env b/services/rest_gate/.env new file mode 120000 index 0000000..c7360fb --- /dev/null +++ b/services/rest_gate/.env @@ -0,0 +1 @@ +../../.env \ No newline at end of file diff --git a/services/rest_gate/.hosts b/services/rest_gate/.hosts new file mode 100644 index 0000000..ee7578e --- /dev/null +++ b/services/rest_gate/.hosts @@ -0,0 +1 @@ +IPV4_PREFIX.83 rest.LOCAL_DOMAIN diff --git a/services/rest_gate/.int_test.env b/services/rest_gate/.int_test.env new file mode 120000 index 0000000..582b6a2 --- /dev/null +++ b/services/rest_gate/.int_test.env @@ -0,0 +1 @@ +../../.int_test.env \ No newline at end of file diff --git a/services/rest_gate/cfg/config.yml b/services/rest_gate/cfg/config.yml new file mode 100644 index 0000000..0acdad3 --- /dev/null +++ b/services/rest_gate/cfg/config.yml @@ -0,0 +1,12 @@ +prometheus: + enabled: true + address: :9090 + +server: + # The IP and port to listen on. + listen-address: 0.0.0.0:8090 + +# Wallet settings +wallet: + path: /wallet.json # Path to wallet + passphrase: one # Password to decrypt wallet diff --git a/services/rest_gate/docker-compose.yml b/services/rest_gate/docker-compose.yml new file mode 100644 index 0000000..f5bac6b --- /dev/null +++ b/services/rest_gate/docker-compose.yml @@ -0,0 +1,32 @@ +--- + +version: "2.4" +services: + rest_gate: + image: ${REST_GW_IMAGE}:${REST_GW_VERSION} + domainname: ${LOCAL_DOMAIN} + hostname: rest + container_name: rest_gate + restart: on-failure + networks: + rest_gate_int: + internet: + ipv4_address: ${IPV4_PREFIX}.83 + volumes: + - ./wallet.json:/wallet.json + - ./../../vendor/hosts:/etc/hosts + - ./cfg:/etc/frostfs/rest + stop_signal: SIGKILL + env_file: [ ".env", ".int_test.env" ] + command: [ "frostfs-rest-gw", "--config", "/etc/frostfs/rest/config.yml" ] + environment: + - REST_GW_POOL_PEERS_0_ADDRESS=s01.${LOCAL_DOMAIN}:8080 + - REST_GW_POOL_PEERS_1_ADDRESS=s02.${LOCAL_DOMAIN}:8080 + - REST_GW_POOL_PEERS_2_ADDRESS=s03.${LOCAL_DOMAIN}:8080 + - REST_GW_POOL_PEERS_3_ADDRESS=s04.${LOCAL_DOMAIN}:8080 + +networks: + rest_gate_int: + internet: + external: true + name: basenet_internet diff --git a/services/s3_lifecycler/wallet.json b/services/rest_gate/wallet.json similarity index 57% rename from services/s3_lifecycler/wallet.json rename to services/rest_gate/wallet.json index ff8f34e..2b60501 100644 --- a/services/s3_lifecycler/wallet.json +++ b/services/rest_gate/wallet.json @@ -1,12 +1,12 @@ { - "version": "1.0", + "version": "3.0", "accounts": [ { - "address": "NTt1rxvmEDxEuuogLxs2xgxA71qhVaUcN7", - "key": "6PYR3XurAyTzVeDG5WV2Z8vnGdySw3mTLuKjr6Nwo7tae64SJ7XjZSMMPQ", - "label": "lifecycler", + "address": "NPFCqWHfi9ixCJRu7DABRbVfXRbkSEr9Vo", + "key": "6PYTAGjdaeicUDPqGv9mmgwb9kTwimWJJmmfNqJSDGH9qM79zSRcL9oHiB", + "label": "REST Gateway", "contract": { - "script": "DCED9z0M+WSGfXZGxYLj1yYwmgxJXE/kNA4+oWNi0q1uKCdBVuezJw==", + "script": "DCECcuPzZCZ2VyDsm2jKEOMnU6xEWO2bF1dvOvBWTDFYB1ZBVuezJw==", "parameters": [ { "name": "parameter0", diff --git a/services/s3_gate/cfg/config.yml b/services/s3_gate/cfg/config.yml index e5c0b53..03e84e0 100644 --- a/services/s3_gate/cfg/config.yml +++ b/services/s3_gate/cfg/config.yml @@ -47,8 +47,3 @@ frostfsid: policy: enabled: false - -containers: - cors: cors.container - mfa: mfa.container - website: website.container diff --git a/services/s3_gate/docker-compose.yml b/services/s3_gate/docker-compose.yml index 3c1f9f6..1445480 100644 --- a/services/s3_gate/docker-compose.yml +++ b/services/s3_gate/docker-compose.yml @@ -1,5 +1,6 @@ --- +version: "2.4" services: s3_gate: image: ${S3_GW_IMAGE}:${S3_GW_VERSION} @@ -12,17 +13,12 @@ services: internet: ipv4_address: ${IPV4_PREFIX}.82 volumes: - # Gate wallet - ./wallet.json:/wallet.json - # Folder for custom user wallets - - ./../../wallets/:/wallets/ - ./tls.key:/tls.key - ./tls.crt:/tls.crt - ./../../vendor/hosts:/etc/hosts - ./cfg:/etc/frostfs/s3 - - ./issue-creds.sh:/usr/bin/issue-creds.sh - stop_signal: SIGTERM - stop_grace_period: 15s + stop_signal: SIGKILL env_file: [ ".env", ".s3.env", ".int_test.env" ] command: [ "frostfs-s3-gw", "--config", "/etc/frostfs/s3/config.yml" ] environment: @@ -38,8 +34,6 @@ services: - S3_GW_PEERS_2_WEIGHT=0.2 - S3_GW_PEERS_3_ADDRESS=s04.${LOCAL_DOMAIN}:8080 - S3_GW_PEERS_3_WEIGHT=0.2 - - AUTHMATE_WALLET_PASSPHRASE= - - AUTHMATE_WALLET_CONTRACT_PASSPHRASE=s3 networks: s3_gate_int: diff --git a/services/s3_gate/issue-creds.sh b/services/s3_gate/issue-creds.sh deleted file mode 100755 index e2355f8..0000000 --- a/services/s3_gate/issue-creds.sh +++ /dev/null @@ -1,41 +0,0 @@ -#!/bin/bash - -initUser() { - /bin/frostfs-s3-authmate register-user \ - --wallet $WALLET_PATH \ - --rpc-endpoint http://morph-chain.frostfs.devenv:30333 \ - --username $USERNAME \ - --contract-wallet /wallet.json 1> /dev/null && touch $WALLET_CACHE/$USERNAME -} - -issueCreds() { - /bin/frostfs-s3-authmate issue-secret \ - --wallet $WALLET_PATH \ - --peer s01.frostfs.devenv:8080 \ - --gate-public-key $S3_GATE_PUBLIC_KEY \ - --container-placement-policy "REP 3" -} - -set -e - -WALLET_PATH=/wallets/$2 -if [[ -z "$2" ]]; then - WALLET_PATH=/wallets/wallet.json -fi - -S3_GATE_PUBLIC_KEY=$3 -if [[ -z "$3" ]]; then - S3_GATE_PUBLIC_KEY=0313b1ac3a8076e155a7e797b24f0b650cccad5941ea59d7cfd51a024a8b2a06bf -fi - -WALLET_CACHE=/data/wallets -mkdir -p $WALLET_CACHE - -USERNAME=$(echo $WALLET_PATH | md5sum | cut -d' ' -f1) -if [ ! -e $WALLET_CACHE/$USERNAME ]; then - initUser -fi - -if [ $1 == "s3" ]; then - issueCreds -fi diff --git a/services/s3_gate/prepare.mk b/services/s3_gate/prepare.mk deleted file mode 100644 index c97add8..0000000 --- a/services/s3_gate/prepare.mk +++ /dev/null @@ -1,14 +0,0 @@ -.PHONY: s3cred register - -password?= -contract_password?=s3 -gate_public_key?= -wallet?= - -# Register wallet & generate S3 credentials -s3cred: - @docker exec -e AUTHMATE_WALLET_PASSPHRASE="$(password)" -e AUTHMATE_WALLET_CONTRACT_PASSPHRASE="$(contract_password)" s3_gate /usr/bin/issue-creds.sh s3 "$(wallet)" "$(gate_public_key)" - -# Only registers user wallet -register: - @docker exec -e AUTHMATE_WALLET_PASSPHRASE="$(password)" -e AUTHMATE_WALLET_CONTRACT_PASSPHRASE="$(contract_password)" s3_gate /usr/bin/issue-creds.sh native "$(wallet)" diff --git a/services/s3_lifecycler/.hosts b/services/s3_lifecycler/.hosts deleted file mode 100644 index 8eb1796..0000000 --- a/services/s3_lifecycler/.hosts +++ /dev/null @@ -1 +0,0 @@ -IPV4_PREFIX.84 lifecycler.LOCAL_DOMAIN diff --git a/services/s3_lifecycler/cfg/config.yml b/services/s3_lifecycler/cfg/config.yml deleted file mode 100644 index 2555980..0000000 --- a/services/s3_lifecycler/cfg/config.yml +++ /dev/null @@ -1,42 +0,0 @@ -logger: - level: debug - -prometheus: - enabled: true - address: :9090 - -lifecycle: - job_fetcher_buffer: 1000 - executor_pool_size: 100 - -frostfs: - stream_timeout: 10s - connect_timeout: 10s - healthcheck_timeout: 15s - rebalance_interval: 60s - pool_error_threshold: 100 - tree_pool_max_attempts: 4 - -credential: - use: wallets - source: - wallets: - - path: /wallet.json - address: NTt1rxvmEDxEuuogLxs2xgxA71qhVaUcN7 - passphrase: "cycle" - - path: /user-wallet.json - address: NbUgTSFvPmsRxmGeWpuuGeJUoRoi6PErcM - passphrase: "" - -morph: - reconnect_clients_interval: 30s - dial_timeout: 5s - contract: - netmap: netmap.frostfs - frostfsid: frostfsid.frostfs - container: container.frostfs - -# Wallet configuration -wallet: - path: /wallet.json # Path to wallet - passphrase: "cycle" # Passphrase to decrypt wallet diff --git a/services/s3_lifecycler/docker-compose.yml b/services/s3_lifecycler/docker-compose.yml deleted file mode 100644 index 2fffa50..0000000 --- a/services/s3_lifecycler/docker-compose.yml +++ /dev/null @@ -1,37 +0,0 @@ ---- - -services: - s3_lifecycler: - image: ${S3_LIFECYCLER_IMAGE}:${S3_LIFECYCLER_VERSION} - domainname: ${LOCAL_DOMAIN} - hostname: s3_lifecycler - container_name: s3_lifecycler - restart: on-failure - networks: - s3_lifecycler_int: - internet: - ipv4_address: ${IPV4_PREFIX}.84 - volumes: - - ./wallet.json:/wallet.json - - ./../../vendor/hosts:/etc/hosts - - ./cfg:/etc/frostfs/s3-lifecycler - - ./../../wallets/wallet.json:/user-wallet.json - stop_signal: SIGKILL - env_file: [ ".env", ".int_test.env" ] - command: [ "frostfs-s3-lifecycler", "--config", "/etc/frostfs/s3-lifecycler/config.yml" ] - environment: - - S3_LIFECYCLER_MORPH_RPC_ENDPOINT_0_ADDRESS=ws://morph-chain:30333/ws - - S3_LIFECYCLER_FROSTFS_PEERS_0_ADDRESS=s01.${LOCAL_DOMAIN}:8080 - - S3_LIFECYCLER_FROSTFS_PEERS_0_WEIGHT=0.2 - - S3_LIFECYCLER_FROSTFS_PEERS_1_ADDRESS=s02.${LOCAL_DOMAIN}:8080 - - S3_LIFECYCLER_FROSTFS_PEERS_1_WEIGHT=0.2 - - S3_LIFECYCLER_FROSTFS_PEERS_2_ADDRESS=s03.${LOCAL_DOMAIN}:8080 - - S3_LIFECYCLER_FROSTFS_PEERS_2_WEIGHT=0.2 - - S3_LIFECYCLER_FROSTFS_PEERS_3_ADDRESS=s04.${LOCAL_DOMAIN}:8080 - - S3_LIFECYCLER_FROSTFS_PEERS_3_WEIGHT=0.2 - -networks: - s3_lifecycler_int: - internet: - external: true - name: basenet_internet diff --git a/services/storage/cfg/config.yml b/services/storage/cfg/config.yml index 0e2526d..6b3e7a8 100644 --- a/services/storage/cfg/config.yml +++ b/services/storage/cfg/config.yml @@ -32,6 +32,18 @@ morph: - address: ws://morph-chain:30333/ws priority: 1 +# Common storage node settings +node: + attribute_0: "User-Agent:FrostFS/0.34" + notification: + enabled: true # Turn on object notification service + endpoint: "tls://nats.frostfs.devenv:4222" # Notification server endpoint + timeout: "6s" # Timeout for object notification client connection + default_topic: "test" # Default topic for object notifications if not found in object's meta + certificate: "/etc/frostfs-node/nats.tls.cert" # Path to TLS certificate + key: "/etc/frostfs-node/nats.tls.key" # Path to TLS key + ca: "/etc/frostfs-node/nats.ca.crt" # Path to optional CA certificate + # Tree section tree: enabled: true diff --git a/services/storage/docker-compose.yml b/services/storage/docker-compose.yml index 785ce79..b32ba21 100644 --- a/services/storage/docker-compose.yml +++ b/services/storage/docker-compose.yml @@ -1,5 +1,6 @@ --- +version: "2.4" services: storage01: image: ${NODE_IMAGE}:${NODE_VERSION} @@ -17,10 +18,13 @@ services: - storage_s01:/storage - ./../../vendor/frostfs-cli:/frostfs-cli - ./cli-cfg.yml:/cli-cfg.yml + - ./healthcheck.sh:/healthcheck.sh - ./s04tls.crt:/etc/ssl/certs/s04tls.crt + - ../nats/client-cert.pem:/etc/frostfs-node/nats.tls.cert + - ../nats/client-key.pem:/etc/frostfs-node/nats.tls.key + - ../nats/ca-cert.pem:/etc/frostfs-node/nats.ca.crt - ./cfg:/etc/frostfs/storage - stop_signal: SIGTERM - stop_grace_period: 15s + stop_signal: SIGKILL env_file: [ ".env", ".storage.env", ".int_test.env" ] command: [ "frostfs-node", "--config", "/etc/frostfs/storage/config.yml" ] environment: @@ -29,11 +33,10 @@ services: - FROSTFS_NODE_ADDRESSES=s01.${LOCAL_DOMAIN}:8080 - FROSTFS_GRPC_0_ENDPOINT=s01.${LOCAL_DOMAIN}:8080 - FROSTFS_CONTROL_GRPC_ENDPOINT=s01.${LOCAL_DOMAIN}:8081 - - FROSTFS_NODE_ATTRIBUTE_0=User-Agent:FrostFS/${NODE_VERSION} - FROSTFS_NODE_ATTRIBUTE_1=UN-LOCODE:RU MOW - FROSTFS_NODE_ATTRIBUTE_2=Price:22 healthcheck: - test: ["CMD-SHELL", "/frostfs-cli control healthcheck -q -c /cli-cfg.yml --endpoint \"$$FROSTFS_CONTROL_GRPC_ENDPOINT\""] + test: ["CMD-SHELL", "/healthcheck.sh"] interval: 2s timeout: 1s retries: 5 @@ -55,10 +58,13 @@ services: - storage_s02:/storage - ./../../vendor/frostfs-cli:/frostfs-cli - ./cli-cfg.yml:/cli-cfg.yml + - ./healthcheck.sh:/healthcheck.sh - ./s04tls.crt:/etc/ssl/certs/s04tls.crt + - ../nats/client-cert.pem:/etc/frostfs-node/nats.tls.cert + - ../nats/client-key.pem:/etc/frostfs-node/nats.tls.key + - ../nats/ca-cert.pem:/etc/frostfs-node/nats.ca.crt - ./cfg:/etc/frostfs/storage - stop_signal: SIGTERM - stop_grace_period: 15s + stop_signal: SIGKILL env_file: [ ".env", ".storage.env", ".int_test.env" ] command: [ "frostfs-node", "--config", "/etc/frostfs/storage/config.yml" ] environment: @@ -67,11 +73,10 @@ services: - FROSTFS_NODE_ADDRESSES=s02.${LOCAL_DOMAIN}:8080 - FROSTFS_GRPC_0_ENDPOINT=s02.${LOCAL_DOMAIN}:8080 - FROSTFS_CONTROL_GRPC_ENDPOINT=s02.${LOCAL_DOMAIN}:8081 - - FROSTFS_NODE_ATTRIBUTE_0=User-Agent:FrostFS/${NODE_VERSION} - FROSTFS_NODE_ATTRIBUTE_1=UN-LOCODE:RU LED - FROSTFS_NODE_ATTRIBUTE_2=Price:33 healthcheck: - test: ["CMD-SHELL", "/frostfs-cli control healthcheck -q -c /cli-cfg.yml --endpoint \"$$FROSTFS_CONTROL_GRPC_ENDPOINT\""] + test: ["CMD-SHELL", "/healthcheck.sh"] interval: 2s timeout: 1s retries: 5 @@ -93,10 +98,13 @@ services: - storage_s03:/storage - ./../../vendor/frostfs-cli:/frostfs-cli - ./cli-cfg.yml:/cli-cfg.yml + - ./healthcheck.sh:/healthcheck.sh - ./s04tls.crt:/etc/ssl/certs/s04tls.crt + - ../nats/client-cert.pem:/etc/frostfs-node/nats.tls.cert + - ../nats/client-key.pem:/etc/frostfs-node/nats.tls.key + - ../nats/ca-cert.pem:/etc/frostfs-node/nats.ca.crt - ./cfg:/etc/frostfs/storage - stop_signal: SIGTERM - stop_grace_period: 15s + stop_signal: SIGKILL env_file: [ ".env", ".storage.env", ".int_test.env" ] command: [ "frostfs-node", "--config", "/etc/frostfs/storage/config.yml" ] environment: @@ -105,11 +113,10 @@ services: - FROSTFS_NODE_ADDRESSES=s03.${LOCAL_DOMAIN}:8080 - FROSTFS_GRPC_0_ENDPOINT=s03.${LOCAL_DOMAIN}:8080 - FROSTFS_CONTROL_GRPC_ENDPOINT=s03.${LOCAL_DOMAIN}:8081 - - FROSTFS_NODE_ATTRIBUTE_0=User-Agent:FrostFS/${NODE_VERSION} - FROSTFS_NODE_ATTRIBUTE_1=UN-LOCODE:SE STO - FROSTFS_NODE_ATTRIBUTE_2=Price:11 healthcheck: - test: ["CMD-SHELL", "/frostfs-cli control healthcheck -q -c /cli-cfg.yml --endpoint \"$$FROSTFS_CONTROL_GRPC_ENDPOINT\""] + test: ["CMD-SHELL", "/healthcheck.sh"] interval: 2s timeout: 1s retries: 5 @@ -131,11 +138,14 @@ services: - storage_s04:/storage - ./../../vendor/frostfs-cli:/frostfs-cli - ./cli-cfg.yml:/cli-cfg.yml + - ./healthcheck.sh:/healthcheck.sh - ./s04tls.crt:/tls.crt - ./s04tls.key:/tls.key + - ../nats/client-cert.pem:/etc/frostfs-node/nats.tls.cert + - ../nats/client-key.pem:/etc/frostfs-node/nats.tls.key + - ../nats/ca-cert.pem:/etc/frostfs-node/nats.ca.crt - ./cfg:/etc/frostfs/storage - stop_signal: SIGTERM - stop_grace_period: 15s + stop_signal: SIGKILL env_file: [ ".env", ".storage.env", ".int_test.env" ] command: [ "frostfs-node", "--config", "/etc/frostfs/storage/config.yml" ] environment: @@ -149,11 +159,10 @@ services: - FROSTFS_GRPC_1_TLS_ENABLED=true - FROSTFS_GRPC_1_TLS_CERTIFICATE=/tls.crt - FROSTFS_GRPC_1_TLS_KEY=/tls.key - - FROSTFS_NODE_ATTRIBUTE_0=User-Agent:FrostFS/${NODE_VERSION} - FROSTFS_NODE_ATTRIBUTE_1=UN-LOCODE:FI HEL - FROSTFS_NODE_ATTRIBUTE_2=Price:44 healthcheck: - test: ["CMD-SHELL", "/frostfs-cli control healthcheck -q -c /cli-cfg.yml --endpoint \"$$FROSTFS_CONTROL_GRPC_ENDPOINT\""] + test: ["CMD-SHELL", "/healthcheck.sh"] interval: 2s timeout: 1s retries: 5 diff --git a/services/storage/healthcheck.sh b/services/storage/healthcheck.sh new file mode 100755 index 0000000..792d2ad --- /dev/null +++ b/services/storage/healthcheck.sh @@ -0,0 +1,5 @@ +#!/bin/sh + +/frostfs-cli control healthcheck -c /cli-cfg.yml \ + --endpoint "$FROSTFS_CONTROL_GRPC_ENDPOINT" | + grep "Health status: READY" diff --git a/wallets/system-wallet.json b/wallets/system-wallet.json deleted file mode 100644 index 2f57db1..0000000 --- a/wallets/system-wallet.json +++ /dev/null @@ -1 +0,0 @@ -{"version":"1.0","accounts":[{"address":"NQijiVKHbL22PfF2AJQukv1CX75itxgzht","key":"6PYQKrpme57VqaucxuF7dDoSZRRA8d94oatHcScqhiFBauCXQvFDaYwEWa","label":"","contract":{"script":"DCEDRdLtpFIWeYyI7doTKRhIl4qYjaybGDveTyGpbqjsLZNBVuezJw==","parameters":[{"name":"parameter0","type":"Signature"}],"deployed":false},"lock":false,"isDefault":false}],"scrypt":{"n":16384,"r":8,"p":8},"extra":{"Tokens":null}} \ No newline at end of file