+
-
+
FrostFS local Development and Testing environment @@ -137,6 +137,66 @@ Display addresses and host names for each running service, if available. Clean up `vendor` directory. +### s3cred + +Registers user wallet and issues s3 credentials. + +Usage and default parameter values: +```sh +make s3cred [password=""] [contract_password=s3] [wallet=""] [gate_public_key=0313b1ac3a8076e155a7e797b24f0b650cccad5941ea59d7cfd51a024a8b2a06bf] +``` + +As soon as the storage node is in the network map (see above) you can generate S3 +credentials: + +``` sh +$ make s3cred +{ + "access_key_id": "EXArWh8x1zeHG3851s1RtoCo7dowxF6rhLGA15nbMffT0AKRSjJ5fmcqf3Ht2VCAkfmPQUVARghRB77xHCA1BoN2p", + "secret_access_key": "d70c1dba83f0f90bb231f06f1ce0e0dfbcfb122f4b4345a3c18d3869c359b79f", + "owner_private_key": "140947599afd9ca89af4b358c3176eb046e554d942a0dc99a8e06f3e43c8f4ad", + "wallet_public_key": "0324e76288fcb900100d01802a14ef977cca45ad073561230446df14b344c858b6", + "container_id": "EXArWh8x1zeHG3851s1RtoCo7dowxF6rhLGA15nbMffT" +} +``` +Running without any parameters results in defaults which are based on the private key from +`/wallets/wallet.json` user wallet and `/wallet.json` contract wallet. +If `wallet` parameter is set, gate searches custom user wallet file in `/wallets` directory. + +Now let's configure an S3 client (AWS CLI will be used as example): + +``` sh +$ aws configure +AWS Access Key ID []: EXArWh8x1zeHG3851s1RtoCo7dowxF6rhLGA15nbMffT0AKRSjJ5fmcqf3Ht2VCAkfmPQUVARghRB77xHCA1BoN2p +AWS Secret Access Key []: d70c1dba83f0f90bb231f06f1ce0e0dfbcfb122f4b4345a3c18d3869c359b79f +Default region name []: us-east-1 +Default output format []: json +``` + +If you need to create credentials for different users, put user wallet to `wallets` dir and specify it via `wallet` parameter. +Pass wallet password in `password` parameter if it's not default. The same is for `contract_wallet` and `gate_public_key` params. + +```sh +$ make s3cred wallet=custom_wallet.json password=test +{ + "access_key_id": "jHhL5B33o16R4jQsb8wm9A3RRdS6KrTB5N4bja9Jys904W7xXFNKqem2ACvTRWRYJsZMCUikYFSokN7pPJziWyDi", + "secret_access_key": "21bb64fafa32c82417fd8b97ac56cc8a085998a3852632d52fe7042453daa440", + "owner_private_key": "10f6f9d7a47bb0bf68363ad8a99fe69f1493f8b6e1665b3e4e83feb2d5c7ee39", + "wallet_public_key": "03e38759973a6bb722baabc2dd84036a39f0b2f53d32fec45a4dacde8a50fe4b70", + "container_id": "jHhL5B33o16R4jQsb8wm9A3RRdS6KrTB5N4bja9Jys9" +} +``` + +To get credentials from custom wallet, place it in `wallets` dir before start. + +### cred + +Usage and default parameter values: +```sh +make cred [password=""] [contract_password=s3] [wallet=/user_wallet.json] +``` +The same as `s3cred`, but it doesn't issues s3 credentials. + ## Contributing Feel free to contribute to this project after reading the [contributing diff --git a/cli-cfg.yml b/cli-cfg.yml new file mode 100644 index 0000000..9a5c7eb --- /dev/null +++ b/cli-cfg.yml @@ -0,0 +1,3 @@ +wallet: ./wallets/system-wallet.json +password: "" +rpc-endpoint: s01.frostfs.devenv:8080 diff --git a/docs/http_gate.md b/docs/http_gate.md index e2159ad..b6b2591 100644 --- a/docs/http_gate.md +++ b/docs/http_gate.md @@ -22,8 +22,8 @@ Image label prefix to use for containers. - Create a new container ``` $ frostfs-cli --rpc-endpoint s01.frostfs.devenv:8080 \ - --key wallets/wallet.key \ - container create --basic-acl readonly --await \ + --wallet wallets/wallet.key \ + container create --basic-acl private --await \ --policy "REP 1 SELECT 1 FROM *" container ID: 4LfREK1cetL4PUji5fqj9SgRTSmaC5jExEDK9HKCDjdP awaiting... @@ -33,7 +33,7 @@ container has been persisted on sidechain - Put an object into the newly created container ``` $ frostfs-cli --rpc-endpoint s01.frostfs.devenv:8080 \ - --key wallets/wallet.key \ + --wallet wallets/wallet.key \ object put --file /tmp/backup.jpeg \ --cid 4LfREK1cetL4PUji5fqj9SgRTSmaC5jExEDK9HKCDjdP [/tmp/backup.jpeg] Object successfully stored diff --git a/docs/morph.md b/docs/morph.md index 56edfbc..d713bf9 100644 --- a/docs/morph.md +++ b/docs/morph.md @@ -4,14 +4,16 @@ A single-node N3 privnet deployment, running on Contracts deployed: - Alphabet (AZ) [contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/alphabet) -- Audit [contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/audit) - Balance [contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/balance) - Container [contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/container) +- FrostFS [contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/frostfs) +- FrostFSID [contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/frostfsid) +- NNS [contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/nns) - Netmap [contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/netmap) -- NeoFSID [contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/neofsid) +- Policy [contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/policy) +- Processing [contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/processing) - Proxy [contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/proxy) -- Reputation [contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/src/branch/master/reputation) - + RPC available at `http://morph-chain.frostfs.devenv:30333`. ## .env settings diff --git a/services/basenet/docker-compose.yml b/services/basenet/docker-compose.yml index abac92a..b2905a3 100644 --- a/services/basenet/docker-compose.yml +++ b/services/basenet/docker-compose.yml @@ -1,6 +1,5 @@ --- -version: "2.4" services: basenet: diff --git a/services/grafana/docker-compose.yml b/services/grafana/docker-compose.yml index 90b67f2..3953400 100644 --- a/services/grafana/docker-compose.yml +++ b/services/grafana/docker-compose.yml @@ -1,4 +1,3 @@ -version: '2.4' services: grafana: image: ${GRAFANA_IMAGE}:${GRAFANA_VERSION} diff --git a/services/http_gate/cfg/config.yml b/services/http_gate/cfg/config.yml index 1684c58..2f15e44 100644 --- a/services/http_gate/cfg/config.yml +++ b/services/http_gate/cfg/config.yml @@ -22,3 +22,6 @@ server: wallet: path: /wallet.json # Path to wallet passphrase: one # Passphrase to decrypt wallet + +containers: + cors: cors.container diff --git a/services/http_gate/docker-compose.yml b/services/http_gate/docker-compose.yml index 7108ee5..c2e68ae 100644 --- a/services/http_gate/docker-compose.yml +++ b/services/http_gate/docker-compose.yml @@ -1,6 +1,5 @@ --- -version: "2.4" services: http_gate: image: ${HTTP_GW_IMAGE}:${HTTP_GW_VERSION} diff --git a/services/ir/artifacts.mk b/services/ir/artifacts.mk index 95b2c22..1c2cf5d 100644 --- a/services/ir/artifacts.mk +++ b/services/ir/artifacts.mk @@ -25,19 +25,14 @@ endif # Download FrostFS CLI .ONESHELL: get.cli: FROSTFS_CLI_FILE=./vendor/frostfs-cli -get.cli: FROSTFS_CLI_ARCHIVE_FILE=${FROSTFS_CLI_FILE}.tar.gz get.cli: FROSTFS_CLI_PATH?= get.cli: @mkdir -p ./vendor ifeq (${FROSTFS_CLI_PATH},) - @echo "⇒ Download FrostFS CLI binary from ${FROSTFS_CLI_URL}" - @curl \ - -ksSL "${FROSTFS_CLI_URL}" \ - -o ${FROSTFS_CLI_ARCHIVE_FILE} - @tar -xvf ${FROSTFS_CLI_ARCHIVE_FILE} -C ./vendor | xargs -I {} \ - mv ./vendor/{} ${FROSTFS_CLI_FILE} - @rm ${FROSTFS_CLI_ARCHIVE_FILE} + echo "⇒ Download FrostFS CLI binary from ${FROSTFS_CLI_IMAGE}:${FROSTFS_CLI_VERSION}" + $(shell docker cp `docker create --name tmp ${FROSTFS_CLI_IMAGE}:${FROSTFS_CLI_VERSION}`:/bin/frostfs-cli ${FROSTFS_CLI_FILE} && docker rm tmp >/dev/null) + chmod +x ${FROSTFS_CLI_FILE} else @echo "⇒ Copy local binary from ${FROSTFS_CLI_PATH}" @cp ${FROSTFS_CLI_PATH} ${FROSTFS_CLI_FILE} diff --git a/services/ir/docker-compose.yml b/services/ir/docker-compose.yml index 77ec50d..4e671b2 100644 --- a/services/ir/docker-compose.yml +++ b/services/ir/docker-compose.yml @@ -1,6 +1,5 @@ --- -version: "2.4" services: ir01: @@ -13,19 +12,19 @@ services: ir_int: internet: ipv4_address: ${IPV4_PREFIX}.61 - stop_signal: SIGKILL + stop_signal: SIGTERM + stop_grace_period: 15s volumes: - ./az.json:/wallet.json - ./az.key:/wallet01.key - ./../../vendor/hosts:/etc/hosts - ./../../vendor/locode_db:/locode/db - ./../../vendor/frostfs-cli:/frostfs-cli - - ./healthcheck.sh:/healthcheck.sh - ./cfg:/etc/frostfs/ir env_file: [ ".env", ".ir.env", ".int_test.env" ] command: [ "frostfs-ir", "--config", "/etc/frostfs/ir/config.yml" ] healthcheck: - test: ["CMD-SHELL", "/healthcheck.sh"] + test: ["CMD-SHELL", "/frostfs-cli control ir healthcheck -q --wallet /wallet01.key --endpoint \"$$FROSTFS_IR_CONTROL_GRPC_ENDPOINT\""] interval: 2s timeout: 1s retries: 5 diff --git a/services/ir/healthcheck.sh b/services/ir/healthcheck.sh deleted file mode 100755 index 747bc8b..0000000 --- a/services/ir/healthcheck.sh +++ /dev/null @@ -1,6 +0,0 @@ -#!/bin/sh - -/frostfs-cli control healthcheck \ - --endpoint "$FROSTFS_IR_CONTROL_GRPC_ENDPOINT" \ - --wallet /wallet01.key --ir | - grep "Health status: READY" diff --git a/services/jaeger/docker-compose.yml b/services/jaeger/docker-compose.yml index a460548..1fb367b 100644 --- a/services/jaeger/docker-compose.yml +++ b/services/jaeger/docker-compose.yml @@ -1,4 +1,3 @@ -version: '2.4' services: jaeger: image: ${JAEGER_IMAGE}:${JAEGER_VERSION} diff --git a/services/morph_chain/artifacts.mk b/services/morph_chain/artifacts.mk index 3572e69..4a4504a 100644 --- a/services/morph_chain/artifacts.mk +++ b/services/morph_chain/artifacts.mk @@ -20,15 +20,12 @@ endif # Download FrostFS ADM tool get.adm: FROSTFS_ADM_DEST=./vendor/frostfs-adm -get.adm: FROSTFS_ADM_ARCHIVE=frostfs-adm.tar.gz get.adm: ifeq (${FROSTFS_ADM_PATH},) - @echo "⇒ Download FrostFS ADM binary from ${FROSTFS_ADM_URL}" - @curl -skSL ${FROSTFS_ADM_URL} -o ${FROSTFS_ADM_ARCHIVE} - @tar -xvf ${FROSTFS_ADM_ARCHIVE} -C ./vendor | xargs -I {} \ - mv ./vendor/{} ${FROSTFS_ADM_DEST} - @rm ${FROSTFS_ADM_ARCHIVE} + @echo "⇒ Download FrostFS ADM binary from ${FROSTFS_ADM_IMAGE}:${FROSTFS_ADM_VERSION}" + $(shell docker cp `docker create --name tmp ${FROSTFS_ADM_IMAGE}:${FROSTFS_ADM_VERSION}`:/bin/frostfs-adm ${FROSTFS_ADM_DEST} && docker rm tmp >/dev/null) + @chmod +x ${FROSTFS_ADM_DEST} else @echo "⇒ Copy frostfs-adm binary from ${FROSTFS_ADM_PATH}" @cp ${FROSTFS_ADM_PATH} ${FROSTFS_ADM_DEST} diff --git a/services/morph_chain/docker-compose.yml b/services/morph_chain/docker-compose.yml index 4cba4b3..c3a1481 100644 --- a/services/morph_chain/docker-compose.yml +++ b/services/morph_chain/docker-compose.yml @@ -1,6 +1,5 @@ --- -version: "2.4" services: frostfs_morph_chain: image: ${NEOGO_IMAGE}:${NEOGO_VERSION} @@ -20,9 +19,19 @@ services: - ./config.yml:/wallets/config.yml - ./../../vendor/hosts:/etc/hosts - ./../../wallets/wallet.json:/wallets/wallet.json + - ./../../wallets/system-wallet.json:/wallets/system-wallet.json + - ./../s3_gate/wallet.json:/wallets/s3-wallet.json + - ./../storage/wallet01.json:/wallets/storage/wallet01.json + - ./../storage/wallet02.json:/wallets/storage/wallet02.json + - ./../storage/wallet03.json:/wallets/storage/wallet03.json + - ./../storage/wallet04.json:/wallets/storage/wallet04.json + - chains:/chains networks: chain_int: internet: external: true name: basenet_internet + +volumes: + chains: diff --git a/services/morph_chain/protocol.privnet.yml b/services/morph_chain/protocol.privnet.yml index 560730a..7cea631 100644 --- a/services/morph_chain/protocol.privnet.yml +++ b/services/morph_chain/protocol.privnet.yml @@ -11,13 +11,14 @@ ProtocolConfiguration: VerifyTransactions: true StateRootInHeader: true P2PSigExtensions: true + Hardforks: {} ApplicationConfiguration: SkipBlockVerification: false DBConfiguration: Type: "boltdb" BoltDBOptions: - FilePath: "./db/morph.bolt" + FilePath: "/chains/morph.bolt" P2P: Addresses: - ":20333" @@ -36,7 +37,7 @@ ApplicationConfiguration: Password: "one" RPC: Addresses: - - "192.168.130.90:30333" + - ":30333" Enabled: true SessionEnabled: true EnableCORSWorkaround: false diff --git a/services/nats/.hosts b/services/nats/.hosts deleted file mode 100644 index 6ed5417..0000000 --- a/services/nats/.hosts +++ /dev/null @@ -1 +0,0 @@ -IPV4_PREFIX.101 nats.LOCAL_DOMAIN diff --git a/services/nats/artifacts.mk b/services/nats/artifacts.mk deleted file mode 100644 index b66cebf..0000000 --- a/services/nats/artifacts.mk +++ /dev/null @@ -1,7 +0,0 @@ -# Create new TLS certs for NATS server and clients - -NATS_DIR=$(abspath services/nats) - -get.nats: - @echo "⇒ Creating certs for NATS server and clients" - ${NATS_DIR}/generate_cert.sh ${LOCAL_DOMAIN} > /dev/null diff --git a/services/nats/docker-compose.yml b/services/nats/docker-compose.yml deleted file mode 100644 index d05dd04..0000000 --- a/services/nats/docker-compose.yml +++ /dev/null @@ -1,31 +0,0 @@ ---- - -version: "2.4" -services: - nats: - image: ${NATS_IMAGE}:${NATS_VERSION} - domainname: ${LOCAL_DOMAIN} - hostname: nats - container_name: nats - restart: on-failure - dns: - - ${IPV4_PREFIX}.101 - networks: - nats_int: - internet: - ipv4_address: ${IPV4_PREFIX}.101 - volumes: - - ./../../vendor/hosts:/etc/hosts - - ./nats.conf:/etc/nats/frostfs-nats-server.conf - - ./server-cert.pem:/certs/server-cert.pem - - ./server-key.pem:/certs/server-key.pem - - ./ca-cert.pem:/certs/ca-cert.pem - stop_signal: SIGKILL - env_file: [ ".env", ".int_test.env" ] - command: ["-c", "/etc/nats/frostfs-nats-server.conf"] - -networks: - nats_int: - internet: - external: true - name: basenet_internet diff --git a/services/nats/generate_cert.sh b/services/nats/generate_cert.sh deleted file mode 100755 index 4c4dd0b..0000000 --- a/services/nats/generate_cert.sh +++ /dev/null @@ -1,49 +0,0 @@ -#!/bin/bash - -source bin/helper.sh - -WORKDIR=$(dirname "$0") -LOCAL_DOMAIN=$1 - -CA_KEY=$WORKDIR/ca-key.pem -CA_CRT=$WORKDIR/ca-cert.pem - -SRV_KEY=$WORKDIR/server-key.pem -SRV_REQ=$WORKDIR/server-req.csr -SRV_CRT=$WORKDIR/server-cert.pem - -CLI_KEY=$WORKDIR/client-key.pem -CLI_REQ=$WORKDIR/client-req.csr -CLI_CRT=$WORKDIR/client-cert.pem - -SUBJ="/O=TrueCloudLab" - -if [[ ! -f $CA_KEY || ! -f $CA_CRT ]]; then - openssl req -newkey rsa:4096 -x509 -days 365 -nodes -keyout $CA_KEY -out $CA_CRT -subj $SUBJ 2>&1 || - die "CA certificate was not created" -fi - -if [[ ! -f $SRV_KEY || ! -f $SRV_CRT ]]; then - openssl req -newkey rsa:4096 -nodes -keyout $SRV_KEY -out $SRV_REQ -subj $SUBJ 2>&1 || - die "Server certificate was not created" - - openssl x509 -req -days 365 -set_serial 01 -in $SRV_REQ -out $SRV_CRT -CA $CA_CRT -CAkey $CA_KEY \ - -extensions san -extfile <(printf "[san]\nsubjectAltName=DNS:nats.$LOCAL_DOMAIN") 2>&1 || { - rm $SRV_REQ - die "Server certificate was not signed by CA" - } - - rm $SRV_REQ -fi - -if [[ ! -f $CLI_KEY || ! -f $CLI_CRT ]]; then - openssl req -newkey rsa:4096 -nodes -keyout $CLI_KEY -out $CLI_REQ -subj $SUBJ 2>&1 || - die "Client certificate was not created" - - openssl x509 -req -days 365 -set_serial 01 -in $CLI_REQ -out $CLI_CRT -CA $CA_CRT -CAkey $CA_KEY 2>&1 || { - rm $CLI_REQ - die "Client certificate was not signed by CA" - } - - rm $CLI_REQ -fi diff --git a/services/nats/nats.conf b/services/nats/nats.conf deleted file mode 100644 index 0c7af0a..0000000 --- a/services/nats/nats.conf +++ /dev/null @@ -1,15 +0,0 @@ -port: 4222 -monitor_port: 8222 - -jetstream { - store_dir=nats - max_memory_store: 1GB - max_file_store: 2GB -} - -tls { - cert_file: /certs/server-cert.pem - key_file: /certs/server-key.pem - ca_file: /certs/ca-cert.pem - verify: true -} diff --git a/services/prometheus/docker-compose.yml b/services/prometheus/docker-compose.yml index e9eb4f9..31f5fbe 100644 --- a/services/prometheus/docker-compose.yml +++ b/services/prometheus/docker-compose.yml @@ -1,4 +1,3 @@ -version: '2.4' services: prometheus: image: ${PROMETHEUS_IMAGE}:${PROMETHEUS_VERSION} diff --git a/services/rest_gate/.env b/services/rest_gate/.env deleted file mode 120000 index c7360fb..0000000 --- a/services/rest_gate/.env +++ /dev/null @@ -1 +0,0 @@ -../../.env \ No newline at end of file diff --git a/services/rest_gate/.hosts b/services/rest_gate/.hosts deleted file mode 100644 index ee7578e..0000000 --- a/services/rest_gate/.hosts +++ /dev/null @@ -1 +0,0 @@ -IPV4_PREFIX.83 rest.LOCAL_DOMAIN diff --git a/services/rest_gate/.int_test.env b/services/rest_gate/.int_test.env deleted file mode 120000 index 582b6a2..0000000 --- a/services/rest_gate/.int_test.env +++ /dev/null @@ -1 +0,0 @@ -../../.int_test.env \ No newline at end of file diff --git a/services/rest_gate/cfg/config.yml b/services/rest_gate/cfg/config.yml deleted file mode 100644 index 0acdad3..0000000 --- a/services/rest_gate/cfg/config.yml +++ /dev/null @@ -1,12 +0,0 @@ -prometheus: - enabled: true - address: :9090 - -server: - # The IP and port to listen on. - listen-address: 0.0.0.0:8090 - -# Wallet settings -wallet: - path: /wallet.json # Path to wallet - passphrase: one # Password to decrypt wallet diff --git a/services/rest_gate/docker-compose.yml b/services/rest_gate/docker-compose.yml deleted file mode 100644 index f5bac6b..0000000 --- a/services/rest_gate/docker-compose.yml +++ /dev/null @@ -1,32 +0,0 @@ ---- - -version: "2.4" -services: - rest_gate: - image: ${REST_GW_IMAGE}:${REST_GW_VERSION} - domainname: ${LOCAL_DOMAIN} - hostname: rest - container_name: rest_gate - restart: on-failure - networks: - rest_gate_int: - internet: - ipv4_address: ${IPV4_PREFIX}.83 - volumes: - - ./wallet.json:/wallet.json - - ./../../vendor/hosts:/etc/hosts - - ./cfg:/etc/frostfs/rest - stop_signal: SIGKILL - env_file: [ ".env", ".int_test.env" ] - command: [ "frostfs-rest-gw", "--config", "/etc/frostfs/rest/config.yml" ] - environment: - - REST_GW_POOL_PEERS_0_ADDRESS=s01.${LOCAL_DOMAIN}:8080 - - REST_GW_POOL_PEERS_1_ADDRESS=s02.${LOCAL_DOMAIN}:8080 - - REST_GW_POOL_PEERS_2_ADDRESS=s03.${LOCAL_DOMAIN}:8080 - - REST_GW_POOL_PEERS_3_ADDRESS=s04.${LOCAL_DOMAIN}:8080 - -networks: - rest_gate_int: - internet: - external: true - name: basenet_internet diff --git a/services/s3_gate/cfg/config.yml b/services/s3_gate/cfg/config.yml index 03e84e0..e5c0b53 100644 --- a/services/s3_gate/cfg/config.yml +++ b/services/s3_gate/cfg/config.yml @@ -47,3 +47,8 @@ frostfsid: policy: enabled: false + +containers: + cors: cors.container + mfa: mfa.container + website: website.container diff --git a/services/s3_gate/docker-compose.yml b/services/s3_gate/docker-compose.yml index 1445480..3c1f9f6 100644 --- a/services/s3_gate/docker-compose.yml +++ b/services/s3_gate/docker-compose.yml @@ -1,6 +1,5 @@ --- -version: "2.4" services: s3_gate: image: ${S3_GW_IMAGE}:${S3_GW_VERSION} @@ -13,12 +12,17 @@ services: internet: ipv4_address: ${IPV4_PREFIX}.82 volumes: + # Gate wallet - ./wallet.json:/wallet.json + # Folder for custom user wallets + - ./../../wallets/:/wallets/ - ./tls.key:/tls.key - ./tls.crt:/tls.crt - ./../../vendor/hosts:/etc/hosts - ./cfg:/etc/frostfs/s3 - stop_signal: SIGKILL + - ./issue-creds.sh:/usr/bin/issue-creds.sh + stop_signal: SIGTERM + stop_grace_period: 15s env_file: [ ".env", ".s3.env", ".int_test.env" ] command: [ "frostfs-s3-gw", "--config", "/etc/frostfs/s3/config.yml" ] environment: @@ -34,6 +38,8 @@ services: - S3_GW_PEERS_2_WEIGHT=0.2 - S3_GW_PEERS_3_ADDRESS=s04.${LOCAL_DOMAIN}:8080 - S3_GW_PEERS_3_WEIGHT=0.2 + - AUTHMATE_WALLET_PASSPHRASE= + - AUTHMATE_WALLET_CONTRACT_PASSPHRASE=s3 networks: s3_gate_int: diff --git a/services/s3_gate/issue-creds.sh b/services/s3_gate/issue-creds.sh new file mode 100755 index 0000000..e2355f8 --- /dev/null +++ b/services/s3_gate/issue-creds.sh @@ -0,0 +1,41 @@ +#!/bin/bash + +initUser() { + /bin/frostfs-s3-authmate register-user \ + --wallet $WALLET_PATH \ + --rpc-endpoint http://morph-chain.frostfs.devenv:30333 \ + --username $USERNAME \ + --contract-wallet /wallet.json 1> /dev/null && touch $WALLET_CACHE/$USERNAME +} + +issueCreds() { + /bin/frostfs-s3-authmate issue-secret \ + --wallet $WALLET_PATH \ + --peer s01.frostfs.devenv:8080 \ + --gate-public-key $S3_GATE_PUBLIC_KEY \ + --container-placement-policy "REP 3" +} + +set -e + +WALLET_PATH=/wallets/$2 +if [[ -z "$2" ]]; then + WALLET_PATH=/wallets/wallet.json +fi + +S3_GATE_PUBLIC_KEY=$3 +if [[ -z "$3" ]]; then + S3_GATE_PUBLIC_KEY=0313b1ac3a8076e155a7e797b24f0b650cccad5941ea59d7cfd51a024a8b2a06bf +fi + +WALLET_CACHE=/data/wallets +mkdir -p $WALLET_CACHE + +USERNAME=$(echo $WALLET_PATH | md5sum | cut -d' ' -f1) +if [ ! -e $WALLET_CACHE/$USERNAME ]; then + initUser +fi + +if [ $1 == "s3" ]; then + issueCreds +fi diff --git a/services/s3_gate/prepare.mk b/services/s3_gate/prepare.mk new file mode 100644 index 0000000..c97add8 --- /dev/null +++ b/services/s3_gate/prepare.mk @@ -0,0 +1,14 @@ +.PHONY: s3cred register + +password?= +contract_password?=s3 +gate_public_key?= +wallet?= + +# Register wallet & generate S3 credentials +s3cred: + @docker exec -e AUTHMATE_WALLET_PASSPHRASE="$(password)" -e AUTHMATE_WALLET_CONTRACT_PASSPHRASE="$(contract_password)" s3_gate /usr/bin/issue-creds.sh s3 "$(wallet)" "$(gate_public_key)" + +# Only registers user wallet +register: + @docker exec -e AUTHMATE_WALLET_PASSPHRASE="$(password)" -e AUTHMATE_WALLET_CONTRACT_PASSPHRASE="$(contract_password)" s3_gate /usr/bin/issue-creds.sh native "$(wallet)" diff --git a/services/nats/.env b/services/s3_lifecycler/.env similarity index 100% rename from services/nats/.env rename to services/s3_lifecycler/.env diff --git a/services/s3_lifecycler/.hosts b/services/s3_lifecycler/.hosts new file mode 100644 index 0000000..8eb1796 --- /dev/null +++ b/services/s3_lifecycler/.hosts @@ -0,0 +1 @@ +IPV4_PREFIX.84 lifecycler.LOCAL_DOMAIN diff --git a/services/nats/.int_test.env b/services/s3_lifecycler/.int_test.env similarity index 100% rename from services/nats/.int_test.env rename to services/s3_lifecycler/.int_test.env diff --git a/services/s3_lifecycler/cfg/config.yml b/services/s3_lifecycler/cfg/config.yml new file mode 100644 index 0000000..2555980 --- /dev/null +++ b/services/s3_lifecycler/cfg/config.yml @@ -0,0 +1,42 @@ +logger: + level: debug + +prometheus: + enabled: true + address: :9090 + +lifecycle: + job_fetcher_buffer: 1000 + executor_pool_size: 100 + +frostfs: + stream_timeout: 10s + connect_timeout: 10s + healthcheck_timeout: 15s + rebalance_interval: 60s + pool_error_threshold: 100 + tree_pool_max_attempts: 4 + +credential: + use: wallets + source: + wallets: + - path: /wallet.json + address: NTt1rxvmEDxEuuogLxs2xgxA71qhVaUcN7 + passphrase: "cycle" + - path: /user-wallet.json + address: NbUgTSFvPmsRxmGeWpuuGeJUoRoi6PErcM + passphrase: "" + +morph: + reconnect_clients_interval: 30s + dial_timeout: 5s + contract: + netmap: netmap.frostfs + frostfsid: frostfsid.frostfs + container: container.frostfs + +# Wallet configuration +wallet: + path: /wallet.json # Path to wallet + passphrase: "cycle" # Passphrase to decrypt wallet diff --git a/services/s3_lifecycler/docker-compose.yml b/services/s3_lifecycler/docker-compose.yml new file mode 100644 index 0000000..2fffa50 --- /dev/null +++ b/services/s3_lifecycler/docker-compose.yml @@ -0,0 +1,37 @@ +--- + +services: + s3_lifecycler: + image: ${S3_LIFECYCLER_IMAGE}:${S3_LIFECYCLER_VERSION} + domainname: ${LOCAL_DOMAIN} + hostname: s3_lifecycler + container_name: s3_lifecycler + restart: on-failure + networks: + s3_lifecycler_int: + internet: + ipv4_address: ${IPV4_PREFIX}.84 + volumes: + - ./wallet.json:/wallet.json + - ./../../vendor/hosts:/etc/hosts + - ./cfg:/etc/frostfs/s3-lifecycler + - ./../../wallets/wallet.json:/user-wallet.json + stop_signal: SIGKILL + env_file: [ ".env", ".int_test.env" ] + command: [ "frostfs-s3-lifecycler", "--config", "/etc/frostfs/s3-lifecycler/config.yml" ] + environment: + - S3_LIFECYCLER_MORPH_RPC_ENDPOINT_0_ADDRESS=ws://morph-chain:30333/ws + - S3_LIFECYCLER_FROSTFS_PEERS_0_ADDRESS=s01.${LOCAL_DOMAIN}:8080 + - S3_LIFECYCLER_FROSTFS_PEERS_0_WEIGHT=0.2 + - S3_LIFECYCLER_FROSTFS_PEERS_1_ADDRESS=s02.${LOCAL_DOMAIN}:8080 + - S3_LIFECYCLER_FROSTFS_PEERS_1_WEIGHT=0.2 + - S3_LIFECYCLER_FROSTFS_PEERS_2_ADDRESS=s03.${LOCAL_DOMAIN}:8080 + - S3_LIFECYCLER_FROSTFS_PEERS_2_WEIGHT=0.2 + - S3_LIFECYCLER_FROSTFS_PEERS_3_ADDRESS=s04.${LOCAL_DOMAIN}:8080 + - S3_LIFECYCLER_FROSTFS_PEERS_3_WEIGHT=0.2 + +networks: + s3_lifecycler_int: + internet: + external: true + name: basenet_internet diff --git a/services/rest_gate/wallet.json b/services/s3_lifecycler/wallet.json similarity index 57% rename from services/rest_gate/wallet.json rename to services/s3_lifecycler/wallet.json index 2b60501..ff8f34e 100644 --- a/services/rest_gate/wallet.json +++ b/services/s3_lifecycler/wallet.json @@ -1,12 +1,12 @@ { - "version": "3.0", + "version": "1.0", "accounts": [ { - "address": "NPFCqWHfi9ixCJRu7DABRbVfXRbkSEr9Vo", - "key": "6PYTAGjdaeicUDPqGv9mmgwb9kTwimWJJmmfNqJSDGH9qM79zSRcL9oHiB", - "label": "REST Gateway", + "address": "NTt1rxvmEDxEuuogLxs2xgxA71qhVaUcN7", + "key": "6PYR3XurAyTzVeDG5WV2Z8vnGdySw3mTLuKjr6Nwo7tae64SJ7XjZSMMPQ", + "label": "lifecycler", "contract": { - "script": "DCECcuPzZCZ2VyDsm2jKEOMnU6xEWO2bF1dvOvBWTDFYB1ZBVuezJw==", + "script": "DCED9z0M+WSGfXZGxYLj1yYwmgxJXE/kNA4+oWNi0q1uKCdBVuezJw==", "parameters": [ { "name": "parameter0", diff --git a/services/storage/cfg/config.yml b/services/storage/cfg/config.yml index 6b3e7a8..0e2526d 100644 --- a/services/storage/cfg/config.yml +++ b/services/storage/cfg/config.yml @@ -32,18 +32,6 @@ morph: - address: ws://morph-chain:30333/ws priority: 1 -# Common storage node settings -node: - attribute_0: "User-Agent:FrostFS/0.34" - notification: - enabled: true # Turn on object notification service - endpoint: "tls://nats.frostfs.devenv:4222" # Notification server endpoint - timeout: "6s" # Timeout for object notification client connection - default_topic: "test" # Default topic for object notifications if not found in object's meta - certificate: "/etc/frostfs-node/nats.tls.cert" # Path to TLS certificate - key: "/etc/frostfs-node/nats.tls.key" # Path to TLS key - ca: "/etc/frostfs-node/nats.ca.crt" # Path to optional CA certificate - # Tree section tree: enabled: true diff --git a/services/storage/docker-compose.yml b/services/storage/docker-compose.yml index b32ba21..785ce79 100644 --- a/services/storage/docker-compose.yml +++ b/services/storage/docker-compose.yml @@ -1,6 +1,5 @@ --- -version: "2.4" services: storage01: image: ${NODE_IMAGE}:${NODE_VERSION} @@ -18,13 +17,10 @@ services: - storage_s01:/storage - ./../../vendor/frostfs-cli:/frostfs-cli - ./cli-cfg.yml:/cli-cfg.yml - - ./healthcheck.sh:/healthcheck.sh - ./s04tls.crt:/etc/ssl/certs/s04tls.crt - - ../nats/client-cert.pem:/etc/frostfs-node/nats.tls.cert - - ../nats/client-key.pem:/etc/frostfs-node/nats.tls.key - - ../nats/ca-cert.pem:/etc/frostfs-node/nats.ca.crt - ./cfg:/etc/frostfs/storage - stop_signal: SIGKILL + stop_signal: SIGTERM + stop_grace_period: 15s env_file: [ ".env", ".storage.env", ".int_test.env" ] command: [ "frostfs-node", "--config", "/etc/frostfs/storage/config.yml" ] environment: @@ -33,10 +29,11 @@ services: - FROSTFS_NODE_ADDRESSES=s01.${LOCAL_DOMAIN}:8080 - FROSTFS_GRPC_0_ENDPOINT=s01.${LOCAL_DOMAIN}:8080 - FROSTFS_CONTROL_GRPC_ENDPOINT=s01.${LOCAL_DOMAIN}:8081 + - FROSTFS_NODE_ATTRIBUTE_0=User-Agent:FrostFS/${NODE_VERSION} - FROSTFS_NODE_ATTRIBUTE_1=UN-LOCODE:RU MOW - FROSTFS_NODE_ATTRIBUTE_2=Price:22 healthcheck: - test: ["CMD-SHELL", "/healthcheck.sh"] + test: ["CMD-SHELL", "/frostfs-cli control healthcheck -q -c /cli-cfg.yml --endpoint \"$$FROSTFS_CONTROL_GRPC_ENDPOINT\""] interval: 2s timeout: 1s retries: 5 @@ -58,13 +55,10 @@ services: - storage_s02:/storage - ./../../vendor/frostfs-cli:/frostfs-cli - ./cli-cfg.yml:/cli-cfg.yml - - ./healthcheck.sh:/healthcheck.sh - ./s04tls.crt:/etc/ssl/certs/s04tls.crt - - ../nats/client-cert.pem:/etc/frostfs-node/nats.tls.cert - - ../nats/client-key.pem:/etc/frostfs-node/nats.tls.key - - ../nats/ca-cert.pem:/etc/frostfs-node/nats.ca.crt - ./cfg:/etc/frostfs/storage - stop_signal: SIGKILL + stop_signal: SIGTERM + stop_grace_period: 15s env_file: [ ".env", ".storage.env", ".int_test.env" ] command: [ "frostfs-node", "--config", "/etc/frostfs/storage/config.yml" ] environment: @@ -73,10 +67,11 @@ services: - FROSTFS_NODE_ADDRESSES=s02.${LOCAL_DOMAIN}:8080 - FROSTFS_GRPC_0_ENDPOINT=s02.${LOCAL_DOMAIN}:8080 - FROSTFS_CONTROL_GRPC_ENDPOINT=s02.${LOCAL_DOMAIN}:8081 + - FROSTFS_NODE_ATTRIBUTE_0=User-Agent:FrostFS/${NODE_VERSION} - FROSTFS_NODE_ATTRIBUTE_1=UN-LOCODE:RU LED - FROSTFS_NODE_ATTRIBUTE_2=Price:33 healthcheck: - test: ["CMD-SHELL", "/healthcheck.sh"] + test: ["CMD-SHELL", "/frostfs-cli control healthcheck -q -c /cli-cfg.yml --endpoint \"$$FROSTFS_CONTROL_GRPC_ENDPOINT\""] interval: 2s timeout: 1s retries: 5 @@ -98,13 +93,10 @@ services: - storage_s03:/storage - ./../../vendor/frostfs-cli:/frostfs-cli - ./cli-cfg.yml:/cli-cfg.yml - - ./healthcheck.sh:/healthcheck.sh - ./s04tls.crt:/etc/ssl/certs/s04tls.crt - - ../nats/client-cert.pem:/etc/frostfs-node/nats.tls.cert - - ../nats/client-key.pem:/etc/frostfs-node/nats.tls.key - - ../nats/ca-cert.pem:/etc/frostfs-node/nats.ca.crt - ./cfg:/etc/frostfs/storage - stop_signal: SIGKILL + stop_signal: SIGTERM + stop_grace_period: 15s env_file: [ ".env", ".storage.env", ".int_test.env" ] command: [ "frostfs-node", "--config", "/etc/frostfs/storage/config.yml" ] environment: @@ -113,10 +105,11 @@ services: - FROSTFS_NODE_ADDRESSES=s03.${LOCAL_DOMAIN}:8080 - FROSTFS_GRPC_0_ENDPOINT=s03.${LOCAL_DOMAIN}:8080 - FROSTFS_CONTROL_GRPC_ENDPOINT=s03.${LOCAL_DOMAIN}:8081 + - FROSTFS_NODE_ATTRIBUTE_0=User-Agent:FrostFS/${NODE_VERSION} - FROSTFS_NODE_ATTRIBUTE_1=UN-LOCODE:SE STO - FROSTFS_NODE_ATTRIBUTE_2=Price:11 healthcheck: - test: ["CMD-SHELL", "/healthcheck.sh"] + test: ["CMD-SHELL", "/frostfs-cli control healthcheck -q -c /cli-cfg.yml --endpoint \"$$FROSTFS_CONTROL_GRPC_ENDPOINT\""] interval: 2s timeout: 1s retries: 5 @@ -138,14 +131,11 @@ services: - storage_s04:/storage - ./../../vendor/frostfs-cli:/frostfs-cli - ./cli-cfg.yml:/cli-cfg.yml - - ./healthcheck.sh:/healthcheck.sh - ./s04tls.crt:/tls.crt - ./s04tls.key:/tls.key - - ../nats/client-cert.pem:/etc/frostfs-node/nats.tls.cert - - ../nats/client-key.pem:/etc/frostfs-node/nats.tls.key - - ../nats/ca-cert.pem:/etc/frostfs-node/nats.ca.crt - ./cfg:/etc/frostfs/storage - stop_signal: SIGKILL + stop_signal: SIGTERM + stop_grace_period: 15s env_file: [ ".env", ".storage.env", ".int_test.env" ] command: [ "frostfs-node", "--config", "/etc/frostfs/storage/config.yml" ] environment: @@ -159,10 +149,11 @@ services: - FROSTFS_GRPC_1_TLS_ENABLED=true - FROSTFS_GRPC_1_TLS_CERTIFICATE=/tls.crt - FROSTFS_GRPC_1_TLS_KEY=/tls.key + - FROSTFS_NODE_ATTRIBUTE_0=User-Agent:FrostFS/${NODE_VERSION} - FROSTFS_NODE_ATTRIBUTE_1=UN-LOCODE:FI HEL - FROSTFS_NODE_ATTRIBUTE_2=Price:44 healthcheck: - test: ["CMD-SHELL", "/healthcheck.sh"] + test: ["CMD-SHELL", "/frostfs-cli control healthcheck -q -c /cli-cfg.yml --endpoint \"$$FROSTFS_CONTROL_GRPC_ENDPOINT\""] interval: 2s timeout: 1s retries: 5 diff --git a/services/storage/healthcheck.sh b/services/storage/healthcheck.sh deleted file mode 100755 index 792d2ad..0000000 --- a/services/storage/healthcheck.sh +++ /dev/null @@ -1,5 +0,0 @@ -#!/bin/sh - -/frostfs-cli control healthcheck -c /cli-cfg.yml \ - --endpoint "$FROSTFS_CONTROL_GRPC_ENDPOINT" | - grep "Health status: READY" diff --git a/wallets/system-wallet.json b/wallets/system-wallet.json new file mode 100644 index 0000000..2f57db1 --- /dev/null +++ b/wallets/system-wallet.json @@ -0,0 +1 @@ +{"version":"1.0","accounts":[{"address":"NQijiVKHbL22PfF2AJQukv1CX75itxgzht","key":"6PYQKrpme57VqaucxuF7dDoSZRRA8d94oatHcScqhiFBauCXQvFDaYwEWa","label":"","contract":{"script":"DCEDRdLtpFIWeYyI7doTKRhIl4qYjaybGDveTyGpbqjsLZNBVuezJw==","parameters":[{"name":"parameter0","type":"Signature"}],"deployed":false},"lock":false,"isDefault":false}],"scrypt":{"n":16384,"r":8,"p":8},"extra":{"Tokens":null}} \ No newline at end of file