Make targets for issuing credentials #86
1
Makefile
|
@ -60,6 +60,7 @@ get: $(foreach SVC, $(GET_SVCS), get.$(SVC))
|
||||||
.PHONY: up
|
.PHONY: up
|
||||||
up: up/basic
|
up: up/basic
|
||||||
@$(foreach SVC, $(START_SVCS), $(shell docker-compose -f services/$(SVC)/docker-compose.yml up -d))
|
@$(foreach SVC, $(START_SVCS), $(shell docker-compose -f services/$(SVC)/docker-compose.yml up -d))
|
||||||
|
./vendor/frostfs-adm morph proxy-add-account --config frostfs-adm.yml --account=`docker container exec morph_chain neo-go wallet dump-keys -w /wallets/s3-wallet.json | head -1 | awk '{print $1}'` || die "Couldn't set s3-gw wallet as proxy wallet"
|
||||||
potyarkin marked this conversation as resolved
Outdated
aarifullin
commented
Could we, please, calculate
? Could we, please, calculate `--account` value using:
```bash
account=`docker container exec -it morph_chain neo-go wallet dump-keys -w services/s3_gate/wallet.json | head -1 | awk '{print $1}'`
```
?
|
|||||||
@echo "Full FrostFS Developer Environment is ready"
|
@echo "Full FrostFS Developer Environment is ready"
|
||||||
|
|
||||||
# Build up FrostFS
|
# Build up FrostFS
|
||||||
|
|
59
README.md
|
@ -137,6 +137,65 @@ Display addresses and host names for each running service, if available.
|
||||||
|
|
||||||
Clean up `vendor` directory.
|
Clean up `vendor` directory.
|
||||||
|
|
||||||
|
### s3cred
|
||||||
|
|
||||||
|
Registers user wallet and issues s3 credentials.
|
||||||
|
|
||||||
|
Usage and default parameter values:
|
||||||
|
```sh
|
||||||
|
make s3cred [password=""] [contract_password=s3] [wallet=/user_wallet.json] [gate_public_key=0313b1ac3a8076e155a7e797b24f0b650cccad5941ea59d7cfd51a024a8b2a06bf]
|
||||||
|
```
|
||||||
|
|
||||||
|
As soon as the storage node is in the network map (see above) you can generate S3
|
||||||
|
credentials:
|
||||||
|
|
||||||
|
``` sh
|
||||||
|
$ make s3cred
|
||||||
|
{
|
||||||
|
"access_key_id": "EXArWh8x1zeHG3851s1RtoCo7dowxF6rhLGA15nbMffT0AKRSjJ5fmcqf3Ht2VCAkfmPQUVARghRB77xHCA1BoN2p",
|
||||||
|
"secret_access_key": "d70c1dba83f0f90bb231f06f1ce0e0dfbcfb122f4b4345a3c18d3869c359b79f",
|
||||||
|
"owner_private_key": "140947599afd9ca89af4b358c3176eb046e554d942a0dc99a8e06f3e43c8f4ad",
|
||||||
|
"wallet_public_key": "0324e76288fcb900100d01802a14ef977cca45ad073561230446df14b344c858b6",
|
||||||
|
"container_id": "EXArWh8x1zeHG3851s1RtoCo7dowxF6rhLGA15nbMffT"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
Running without any parameters will result in defaults which are based on the private key from
|
||||||
|
`/user-wallet.json` file and `/wallet.json` contract wallet.
|
||||||
|
|
||||||
|
Now let's configure an S3 client (AWS CLI will be used as example):
|
||||||
|
|
||||||
|
``` sh
|
||||||
|
$ aws configure
|
||||||
|
AWS Access Key ID []: EXArWh8x1zeHG3851s1RtoCo7dowxF6rhLGA15nbMffT0AKRSjJ5fmcqf3Ht2VCAkfmPQUVARghRB77xHCA1BoN2p
|
||||||
|
AWS Secret Access Key []: d70c1dba83f0f90bb231f06f1ce0e0dfbcfb122f4b4345a3c18d3869c359b79f
|
||||||
|
Default region name []: us-east-1
|
||||||
|
Default output format []: json
|
||||||
|
```
|
||||||
|
|
||||||
|
If you need to create credentials for different users, put user wallets to `wallets` dir and specify them via `wallet` parameter.
|
||||||
|
Pass wallet password in `password` parameter if it's not default. The same is for `contract_wallet` and `gate_public_key` params.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
$ make s3cred wallet=custom_wallet.json password=test
|
||||||
|
{
|
||||||
|
"access_key_id": "jHhL5B33o16R4jQsb8wm9A3RRdS6KrTB5N4bja9Jys904W7xXFNKqem2ACvTRWRYJsZMCUikYFSokN7pPJziWyDi",
|
||||||
|
"secret_access_key": "21bb64fafa32c82417fd8b97ac56cc8a085998a3852632d52fe7042453daa440",
|
||||||
|
"owner_private_key": "10f6f9d7a47bb0bf68363ad8a99fe69f1493f8b6e1665b3e4e83feb2d5c7ee39",
|
||||||
|
"wallet_public_key": "03e38759973a6bb722baabc2dd84036a39f0b2f53d32fec45a4dacde8a50fe4b70",
|
||||||
|
"container_id": "jHhL5B33o16R4jQsb8wm9A3RRdS6KrTB5N4bja9Jys9"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
To get credentials from custom wallet, place it in `wallets` dir before start.
|
||||||
|
|
||||||
|
### cred
|
||||||
|
|
||||||
|
Usage and default parameter values:
|
||||||
|
```sh
|
||||||
|
make cred [password=""] [contract_password=s3] [wallet=/user_wallet.json]
|
||||||
|
```
|
||||||
|
The same as `s3cred`, but it doesn't issues s3 credentials.
|
||||||
|
|
||||||
## Contributing
|
## Contributing
|
||||||
|
|
||||||
Feel free to contribute to this project after reading the [contributing
|
Feel free to contribute to this project after reading the [contributing
|
||||||
|
|
|
@ -19,6 +19,7 @@ services:
|
||||||
- ./config.yml:/wallets/config.yml
|
- ./config.yml:/wallets/config.yml
|
||||||
- ./../../vendor/hosts:/etc/hosts
|
- ./../../vendor/hosts:/etc/hosts
|
||||||
- ./../../wallets/wallet.json:/wallets/wallet.json
|
- ./../../wallets/wallet.json:/wallets/wallet.json
|
||||||
|
- ./../s3_gate/wallet.json:/wallets/s3-wallet.json
|
||||||
- chains:/chains
|
- chains:/chains
|
||||||
|
|
||||||
networks:
|
networks:
|
||||||
|
|
|
@ -12,11 +12,17 @@ services:
|
||||||
internet:
|
internet:
|
||||||
ipv4_address: ${IPV4_PREFIX}.82
|
ipv4_address: ${IPV4_PREFIX}.82
|
||||||
volumes:
|
volumes:
|
||||||
|
# Gate wallet
|
||||||
fyrchik
commented
There are 3 different wallet volumes with the same name. There are 3 different wallet volumes with the same name.
Do we need them all?
From the name alone, it is not obvious what are the use-cases for each.
alexvanin
commented
Now these volumes have corresponding comments. One is a service wallet, another one is for user operation, and for more there are separate dir of custom wallets to use. Now these volumes have corresponding comments. One is a service wallet, another one is for user operation, and for more there are separate dir of custom wallets to use.
|
|||||||
- ./wallet.json:/wallet.json
|
- ./wallet.json:/wallet.json
|
||||||
dkirillov marked this conversation as resolved
Outdated
dkirillov
commented
Let's use Let's use `frostfs-dev-env/wallets/wallet.json` wallet by default
|
|||||||
|
# Custom user wallets
|
||||||
|
- ./wallets:/wallets
|
||||||
|
# Default user wallet
|
||||||
|
- ./../../wallets/wallet.json:/wallets/wallet.json
|
||||||
- ./tls.key:/tls.key
|
- ./tls.key:/tls.key
|
||||||
- ./tls.crt:/tls.crt
|
- ./tls.crt:/tls.crt
|
||||||
- ./../../vendor/hosts:/etc/hosts
|
- ./../../vendor/hosts:/etc/hosts
|
||||||
- ./cfg:/etc/frostfs/s3
|
- ./cfg:/etc/frostfs/s3
|
||||||
|
- ./issue-creds.sh:/usr/bin/issue-creds.sh
|
||||||
stop_signal: SIGTERM
|
stop_signal: SIGTERM
|
||||||
stop_grace_period: 15s
|
stop_grace_period: 15s
|
||||||
env_file: [ ".env", ".s3.env", ".int_test.env" ]
|
env_file: [ ".env", ".s3.env", ".int_test.env" ]
|
||||||
|
@ -34,6 +40,8 @@ services:
|
||||||
- S3_GW_PEERS_2_WEIGHT=0.2
|
- S3_GW_PEERS_2_WEIGHT=0.2
|
||||||
- S3_GW_PEERS_3_ADDRESS=s04.${LOCAL_DOMAIN}:8080
|
- S3_GW_PEERS_3_ADDRESS=s04.${LOCAL_DOMAIN}:8080
|
||||||
- S3_GW_PEERS_3_WEIGHT=0.2
|
- S3_GW_PEERS_3_WEIGHT=0.2
|
||||||
|
- AUTHMATE_WALLET_PASSPHRASE=
|
||||||
|
- AUTHMATE_WALLET_CONTRACT_PASSPHRASE=s3
|
||||||
|
|
||||||
networks:
|
networks:
|
||||||
s3_gate_int:
|
s3_gate_int:
|
||||||
|
|
41
services/s3_gate/issue-creds.sh
Executable file
|
@ -0,0 +1,41 @@
|
||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
initUser() {
|
||||||
|
/bin/frostfs-s3-authmate register-user \
|
||||||
|
--wallet $WALLET_PATH \
|
||||||
|
--rpc-endpoint http://morph-chain.frostfs.devenv:30333 \
|
||||||
|
--username $USERNAME \
|
||||||
|
--contract-wallet /wallet.json 1> /dev/null && touch $WALLET_CACHE/$USERNAME
|
||||||
|
}
|
||||||
|
|
||||||
|
issueCreds() {
|
||||||
pogpp
commented
issueCreds issueCreds
|
|||||||
|
/bin/frostfs-s3-authmate issue-secret \
|
||||||
|
--wallet $WALLET_PATH \
|
||||||
|
--peer s01.frostfs.devenv:8080 \
|
||||||
|
--gate-public-key $S3_GATE_PUBLIC_KEY \
|
||||||
|
--container-placement-policy "REP 3"
|
||||||
dkirillov marked this conversation as resolved
Outdated
dkirillov
commented
Probably we should use different policy. Probably we should use different policy. `REP 3` for example
dstepanov-yadro marked this conversation as resolved
Outdated
dstepanov-yadro
commented
What's happen if my container has other policy? What's happen if my container has other policy?
alexvanin
commented
This policy is used to create container to store 'accessbox' object with tokens for s3-gateway to use for authorization. So any valid policy can be used here. For example, I often use This policy is used to create container to store 'accessbox' object with tokens for s3-gateway to use for authorization. So any valid policy can be used here.
For example, I often use `REP 4` policy when I test some split-brain issues in dev-env. This way all nodes contain accessbox and authorization does not affect the case I want to check.
|
|||||||
|
}
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
WALLET_PATH=/wallets/$2
|
||||||
|
if [[ -z "$2" ]]; then
|
||||||
|
WALLET_PATH=/wallets/wallet.json
|
||||||
|
fi
|
||||||
|
|
||||||
|
S3_GATE_PUBLIC_KEY=$3
|
||||||
|
if [[ -z "$3" ]]; then
|
||||||
|
S3_GATE_PUBLIC_KEY=0313b1ac3a8076e155a7e797b24f0b650cccad5941ea59d7cfd51a024a8b2a06bf
|
||||||
|
fi
|
||||||
|
|
||||||
|
WALLET_CACHE=/data/wallets
|
||||||
|
mkdir -p $WALLET_CACHE
|
||||||
|
|
||||||
|
USERNAME=$(echo $WALLET_PATH | md5sum | cut -d' ' -f1)
|
||||||
|
if [ ! -e $WALLET_CACHE/$USERNAME ]; then
|
||||||
|
initUser
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ $1 == "s3" ]; then
|
||||||
|
issueCreds
|
||||||
|
fi
|
14
services/s3_gate/prepare.mk
Normal file
|
@ -0,0 +1,14 @@
|
||||||
|
.PHONY: s3cred register
|
||||||
|
|
||||||
|
password?=
|
||||||
potyarkin marked this conversation as resolved
Outdated
potyarkin
commented
Empty strings in Makefile are just empty. This should be Quotes work for now, but that's more of an accident:
A few Empty strings in Makefile are just empty. This should be `password?=`
Quotes work for now, but that's more of an accident:
- You build the command line below: `AUTHMATE_WALLET_PASSPHRASE="$(password)"`
- Make substitutes that for `AUTHMATE_WALLET_PASSPHRASE=""""`
- It works because shell treats it as a concatenation of two empty strings: first+second quotes produce an empty string, third+fourth quotes produce an empty string
A few ~~accidents~~edits later it might cost someone a long debug session.
|
|||||||
|
contract_password?=s3
|
||||||
|
gate_public_key?=
|
||||||
potyarkin marked this conversation as resolved
Outdated
potyarkin
commented
This recipe asks for This recipe asks for `/user_wallet.json` password on my machine, and `s3` is not accepted as valid. Maybe we can avoid interactive prompt here?
nzinkevich
commented
Fixed, now if wallet has non-empty password, you can pass it via Fixed, now if wallet has non-empty password, you can pass it via `password` parameter. Now it also has optional params `contract_password` and `gate_public_key` if contract wallet is not default. Example given in README.md
|
|||||||
|
wallet?=
|
||||||
|
|
||||||
|
# Register wallet & generate S3 credentials
|
||||||
|
s3cred:
|
||||||
potyarkin marked this conversation as resolved
Outdated
potyarkin
commented
Did you test this? I'm not sure that Did you test this? I'm not sure that `$(wallet)` is visible from inside the `s3_gate` container.
nzinkevich
commented
Added Added `wallets` directory for storing custom wallets
|
|||||||
|
@docker exec -e AUTHMATE_WALLET_PASSPHRASE="$(password)" -e AUTHMATE_WALLET_CONTRACT_PASSPHRASE="$(contract_password)" s3_gate /usr/bin/issue-creds.sh s3 "$(wallet)" "$(gate_public_key)"
|
||||||
potyarkin marked this conversation as resolved
Outdated
potyarkin
commented
This will not work if password contains whitespace This will not work if password contains whitespace
|
|||||||
|
|
||||||
|
# Only registers user wallet
|
||||||
dkirillov marked this conversation as resolved
Outdated
dkirillov
commented
Incorrect comment Incorrect comment
|
|||||||
|
register:
|
||||||
dkirillov marked this conversation as resolved
Outdated
dkirillov
commented
Should we rename this target? To be more clear that this command just register user and don't create any credentials Should we rename this target? To be more clear that this command just register user and don't create any credentials
|
|||||||
|
@docker exec -e AUTHMATE_WALLET_PASSPHRASE="$(password)" -e AUTHMATE_WALLET_CONTRACT_PASSPHRASE="$(contract_password)" s3_gate /usr/bin/issue-creds.sh native "$(wallet)"
|
What wallet is this hardcoded account from? A quick grep through my copy of the repo did not come up with any results
Fixed. It's a contract wallet address