forked from TrueCloudLab/frostfs-testcases
Update README on running allure from docker
Also cleanup README from description specific to Robot framework. Signed-off-by: Vladimir Domnich <v.domnich@yadro.com>
This commit is contained in:
parent
7e31610462
commit
e2ab4d3774
3 changed files with 44 additions and 294 deletions
124
README.md
124
README.md
|
@ -1,3 +1,15 @@
|
||||||
|
## Testcases structure
|
||||||
|
|
||||||
|
Tests written with PyTest Framework are located under `pytest_tests/testsuites` directory.
|
||||||
|
|
||||||
|
These tests rely on resources and utility modules that have been originally developed for Robot Framework:
|
||||||
|
|
||||||
|
`robot/resources/files` - static files that are used in tests' commands.
|
||||||
|
|
||||||
|
`robot/resources/lib/` - common Python libraries that provide utility functions used as building blocks in tests.
|
||||||
|
|
||||||
|
`robot/variables/` - constants and configuration variables for tests.
|
||||||
|
|
||||||
## Testcases execution
|
## Testcases execution
|
||||||
|
|
||||||
### Initial preparation
|
### Initial preparation
|
||||||
|
@ -40,84 +52,6 @@ libssl-dev
|
||||||
As we use neofs-dev-env, you'll also need to install
|
As we use neofs-dev-env, you'll also need to install
|
||||||
[prerequisites](https://github.com/nspcc-dev/neofs-dev-env#prerequisites) of this repository.
|
[prerequisites](https://github.com/nspcc-dev/neofs-dev-env#prerequisites) of this repository.
|
||||||
|
|
||||||
## Robot Framework
|
|
||||||
|
|
||||||
### Run
|
|
||||||
|
|
||||||
1. Prepare virtualenv
|
|
||||||
|
|
||||||
```
|
|
||||||
$ make venv.localtest
|
|
||||||
$ . venv.localtest/bin/activate
|
|
||||||
```
|
|
||||||
|
|
||||||
2. Run tests
|
|
||||||
|
|
||||||
In the activated virtualenv, execute the following command(s) to run a singular testsuite or all the suites in the directory
|
|
||||||
```
|
|
||||||
$ robot --outputdir artifacts/ robot/testsuites/integration/<UserScenario>
|
|
||||||
$ robot --outputdir artifacts/ robot/testsuites/integration/<UserScenario>/<testcase>.robot
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
### Generation of documentation
|
|
||||||
|
|
||||||
To generate Keywords documentation:
|
|
||||||
```
|
|
||||||
python3 -m robot.libdoc robot/resources/lib/neofs.py docs/NeoFS_Library.html
|
|
||||||
python3 -m robot.libdoc robot/resources/lib/payment_neogo.py docs/Payment_Library.html
|
|
||||||
```
|
|
||||||
|
|
||||||
To generate testcases documentation:
|
|
||||||
```
|
|
||||||
python3 -m robot.testdoc robot/testsuites/integration/ docs/testcases.html
|
|
||||||
```
|
|
||||||
|
|
||||||
### Source code overview
|
|
||||||
|
|
||||||
`robot/` - Files related/depended on Robot Framework.
|
|
||||||
|
|
||||||
`robot/resources/` - All resources (Robot Framework Keywords, Python Libraries, etc) which could be used for creating test suites.
|
|
||||||
|
|
||||||
`robot/resources/lib/` - Common Python Libraries depended on Robot Framework (with Keywords). For example neofs.py, payment.py.
|
|
||||||
|
|
||||||
`robot/variables/` - All variables for tests. It is possible to add the auto-loading logic of parameters from the smart-contract in the future. Contain python files.
|
|
||||||
|
|
||||||
`robot/testsuites/` - Robot TestSuites and TestCases.
|
|
||||||
|
|
||||||
`robot/testsuites/integration/` - Integration test suites and testcases
|
|
||||||
|
|
||||||
### Code style
|
|
||||||
|
|
||||||
Robot Framework keyword should use space as a separator between particular words
|
|
||||||
|
|
||||||
The name of the library function in Robot Framework keyword usage and the name of the same function in the Python library must be identical.
|
|
||||||
|
|
||||||
The name of GLOBAL VARIABLE must be in UPPER CASE, the underscore ('_')' symbol must be used as a separator between words.
|
|
||||||
|
|
||||||
The name of local variable must be in lower case, the underscore symbol must be used as a separator between words.
|
|
||||||
|
|
||||||
The names of Python variables, functions and classes must comply with accepted rules, in particular:
|
|
||||||
Name of variable/function must be in lower case with underscore symbol between words
|
|
||||||
Name of class must start with a capital letter. It is not allowed to use underscore symbol in name, use capital for each particular word.
|
|
||||||
For example: NeoFSConf
|
|
||||||
|
|
||||||
Name of other variables should not be ended with underscore symbol
|
|
||||||
|
|
||||||
On keywords definition, one should specify variable type, e.g. path: str
|
|
||||||
|
|
||||||
### Robot style
|
|
||||||
|
|
||||||
You should always complete the [Tags] and [Documentation] sections for Testcases and Documentation for Test Suites.
|
|
||||||
|
|
||||||
### Robot-framework User Guide
|
|
||||||
|
|
||||||
http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html
|
|
||||||
|
|
||||||
## PyTest
|
|
||||||
|
|
||||||
Tests written with PyTest framework are located under `pytest_tests/testsuites` directory.
|
|
||||||
|
|
||||||
### Run and get report
|
### Run and get report
|
||||||
|
|
||||||
1. Prepare virtualenv
|
1. Prepare virtualenv
|
||||||
|
@ -129,7 +63,10 @@ $ . venv.local-pytest/bin/activate
|
||||||
|
|
||||||
2. Install Allure CLI
|
2. Install Allure CLI
|
||||||
|
|
||||||
Allure CLI installation is not an easy task. You may select one of the following ways. If none of the options would help you please complete the instruction with your approach:
|
Allure CLI installation is not an easy task, so a better option might be to run allure from
|
||||||
|
docker container (please, refer to p.4 of this section for instructions).
|
||||||
|
|
||||||
|
To install Allure CLI you may select one of the following ways:
|
||||||
|
|
||||||
- Follow the [instruction](https://docs.qameta.io/allure/#_linux) from the official website
|
- Follow the [instruction](https://docs.qameta.io/allure/#_linux) from the official website
|
||||||
- Consult [the thread](https://github.com/allure-framework/allure2/issues/989)
|
- Consult [the thread](https://github.com/allure-framework/allure2/issues/989)
|
||||||
|
@ -140,6 +77,8 @@ $ sudo apt install ./allure_2.18.1-1_all.deb
|
||||||
```
|
```
|
||||||
You also need the `default-jre` package installed.
|
You also need the `default-jre` package installed.
|
||||||
|
|
||||||
|
If none of the options worked for you, please complete the instruction with your approach.
|
||||||
|
|
||||||
3. Run tests
|
3. Run tests
|
||||||
|
|
||||||
In the activated virtualenv, execute the following command(s) to run a singular testsuite or all the suites in the directory
|
In the activated virtualenv, execute the following command(s) to run a singular testsuite or all the suites in the directory
|
||||||
|
@ -150,7 +89,7 @@ $ pytest --alluredir my-allure-123 pytest_tests/testsuites/
|
||||||
|
|
||||||
4. Generate report
|
4. Generate report
|
||||||
|
|
||||||
To generate a report, execute the command `allure generate`. The report will be under the `allure-report` directory.
|
If you opted to install Allure CLI, you can generate a report using the command `allure generate`. The web representation of the report will be under `allure-report` directory:
|
||||||
```
|
```
|
||||||
$ allure generate my-allure-123
|
$ allure generate my-allure-123
|
||||||
$ ls allure-report/
|
$ ls allure-report/
|
||||||
|
@ -161,3 +100,28 @@ To inspect the report in a browser, run
|
||||||
```
|
```
|
||||||
$ allure serve my-allure-123
|
$ allure serve my-allure-123
|
||||||
```
|
```
|
||||||
|
|
||||||
|
If you prefer to run allure from Docker, you can use the following command:
|
||||||
|
```
|
||||||
|
mkdir -p $PWD/allure-reports
|
||||||
|
docker run -p 5050:5050 -e CHECK_RESULTS_EVERY_SECONDS=30 -e KEEP_HISTORY=1 \
|
||||||
|
-v $PWD/my-allure-123:/app/allure-results \
|
||||||
|
-v $PWD/allure-reports:/app/default-reports \
|
||||||
|
frankescobar/allure-docker-service
|
||||||
|
```
|
||||||
|
|
||||||
|
Then, you can check the allure report in your browser [by this link](http://localhost:5050/allure-docker-service/projects/default/reports/latest/index.html?redirect=false)
|
||||||
|
|
||||||
|
NOTE: feel free to select a different location for `allure-reports` directory, there is no requirement to have it inside `neofs-testcases`. For example, you can place it under `/tmp` path.
|
||||||
|
|
||||||
|
## Code style
|
||||||
|
|
||||||
|
The names of Python variables, functions and classes must comply with [PEP8](https://peps.python.org/pep-0008) rules, in particular:
|
||||||
|
* Name of a variable/function must be in snake_case (lowercase, with words separated by underscores as necessary to improve readability).
|
||||||
|
* Name of a global variable must be in UPPER_SNAKE_CASE, the underscore (`_`) symbol must be used as a separator between words.
|
||||||
|
* Name of a class must be in PascalCase (the first letter of each compound word in a variable name is capitalized).
|
||||||
|
* Names of other variables should not be ended with the underscore symbol.
|
||||||
|
|
||||||
|
Line length limit is set as 100 characters.
|
||||||
|
|
||||||
|
Imports should be ordered in accordance with [isort default rules](https://pycqa.github.io/isort/).
|
||||||
|
|
|
@ -1,185 +0,0 @@
|
||||||
#!/usr/bin/python3.8
|
|
||||||
|
|
||||||
###################################
|
|
||||||
# eACL tables generation functions
|
|
||||||
###################################
|
|
||||||
|
|
||||||
import json
|
|
||||||
|
|
||||||
VERBS = [
|
|
||||||
'GET',
|
|
||||||
'HEAD',
|
|
||||||
'PUT',
|
|
||||||
'DELETE',
|
|
||||||
'SEARCH',
|
|
||||||
'GETRANGE',
|
|
||||||
'GETRANGEHASH'
|
|
||||||
]
|
|
||||||
|
|
||||||
ROLES = [
|
|
||||||
'OTHERS',
|
|
||||||
'USER',
|
|
||||||
'SYSTEM'
|
|
||||||
]
|
|
||||||
|
|
||||||
ACCESS = [
|
|
||||||
'DENY',
|
|
||||||
'ALLOW'
|
|
||||||
]
|
|
||||||
|
|
||||||
TABLES_DIR = '../files/eacl_tables/'
|
|
||||||
|
|
||||||
|
|
||||||
def deny_allow_tables_per_role():
|
|
||||||
for a in ACCESS:
|
|
||||||
for r in ROLES:
|
|
||||||
table_dict = {
|
|
||||||
"records": []
|
|
||||||
}
|
|
||||||
for v in VERBS:
|
|
||||||
table_record = {
|
|
||||||
"operation": v,
|
|
||||||
"action": a,
|
|
||||||
"filters": [],
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"role": r
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
table_dict['records'].append(table_record)
|
|
||||||
with open(f"{TABLES_DIR}/gen_eacl_{a.lower()}_all_{r}", "w+") as f:
|
|
||||||
json.dump(table_dict, f, indent=4)
|
|
||||||
|
|
||||||
def allow_pubkey_deny_others():
|
|
||||||
table_dict = {
|
|
||||||
"records": []
|
|
||||||
}
|
|
||||||
for v in VERBS:
|
|
||||||
table_record = {
|
|
||||||
"operation": v,
|
|
||||||
"action": "ALLOW",
|
|
||||||
"filters": [],
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
# TODO: where do we take this value from?
|
|
||||||
"keys": [ 'A9tDy6Ye+UimXCCzJrlAmRE0FDZHjf3XRyya9rELtgAA' ]
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
table_dict['records'].append(table_record)
|
|
||||||
for v in VERBS:
|
|
||||||
table_record = {
|
|
||||||
"operation": v,
|
|
||||||
"action": "DENY",
|
|
||||||
"filters": [],
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"role": 'OTHERS'
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
table_dict['records'].append(table_record)
|
|
||||||
with open(f"{TABLES_DIR}/gen_eacl_allow_pubkey_deny_OTHERS", "w+") as f:
|
|
||||||
json.dump(table_dict, f, indent=4)
|
|
||||||
|
|
||||||
def compound_tables():
|
|
||||||
compounds = {
|
|
||||||
'get': {
|
|
||||||
'GET': 'ALLOW',
|
|
||||||
'GETRANGE': 'ALLOW',
|
|
||||||
'GETRANGEHASH': 'ALLOW',
|
|
||||||
'HEAD': 'DENY'
|
|
||||||
},
|
|
||||||
'del': {
|
|
||||||
'DELETE': 'ALLOW',
|
|
||||||
'PUT': 'DENY',
|
|
||||||
'HEAD': 'DENY'
|
|
||||||
},
|
|
||||||
'get_hash': {
|
|
||||||
'GETRANGEHASH': 'ALLOW',
|
|
||||||
'GETRANGE': 'DENY',
|
|
||||||
'GET': 'DENY'
|
|
||||||
}
|
|
||||||
}
|
|
||||||
for op, compound in compounds.items():
|
|
||||||
for r in ROLES:
|
|
||||||
table_dict = {
|
|
||||||
"records": []
|
|
||||||
}
|
|
||||||
for verb, access in compound.items():
|
|
||||||
table_record = {
|
|
||||||
"operation": verb,
|
|
||||||
"action": access,
|
|
||||||
"filters": [],
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"role": r
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
table_dict['records'].append(table_record)
|
|
||||||
|
|
||||||
with open(f"{TABLES_DIR}/gen_eacl_compound_{op}_{r}", "w+") as f:
|
|
||||||
json.dump(table_dict, f, indent=4)
|
|
||||||
|
|
||||||
def xheader_tables():
|
|
||||||
filters = {
|
|
||||||
'headerType': 'REQUEST',
|
|
||||||
'matchType': 'STRING_EQUAL',
|
|
||||||
'key': 'a',
|
|
||||||
'value': '2'
|
|
||||||
}
|
|
||||||
table_dict = {
|
|
||||||
"records": []
|
|
||||||
}
|
|
||||||
for verb in VERBS:
|
|
||||||
table_record = {
|
|
||||||
"operation": verb,
|
|
||||||
"action": "DENY",
|
|
||||||
"filters": [filters],
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"role": "OTHERS"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
table_dict['records'].append(table_record)
|
|
||||||
with open(f"{TABLES_DIR}/gen_eacl_xheader_deny_all", "w+") as f:
|
|
||||||
json.dump(table_dict, f, indent=4)
|
|
||||||
|
|
||||||
table_dict = {
|
|
||||||
"records": []
|
|
||||||
}
|
|
||||||
for verb in VERBS:
|
|
||||||
table_record = {
|
|
||||||
"operation": verb,
|
|
||||||
"action": "ALLOW",
|
|
||||||
"filters": [filters],
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"role": "OTHERS"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
table_dict['records'].append(table_record)
|
|
||||||
|
|
||||||
table_record = {
|
|
||||||
"operation": verb,
|
|
||||||
"action": "DENY",
|
|
||||||
"filters": [],
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"role": "OTHERS"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
table_dict['records'].append(table_record)
|
|
||||||
with open(f"{TABLES_DIR}/gen_eacl_xheader_allow_all", "w+") as f:
|
|
||||||
json.dump(table_dict, f, indent=4)
|
|
||||||
|
|
||||||
|
|
||||||
deny_allow_tables_per_role()
|
|
||||||
allow_pubkey_deny_others()
|
|
||||||
compound_tables()
|
|
||||||
xheader_tables()
|
|
|
@ -1,29 +0,0 @@
|
||||||
*** Variables ***
|
|
||||||
|
|
||||||
${ACL_TEST_FILES} = robot/resources/files/eacl_tables
|
|
||||||
|
|
||||||
${EACL_DENY_ALL_OTHERS} = ${ACL_TEST_FILES}/gen_eacl_deny_all_OTHERS
|
|
||||||
${EACL_ALLOW_ALL_OTHERS} = ${ACL_TEST_FILES}/gen_eacl_allow_all_OTHERS
|
|
||||||
|
|
||||||
${EACL_DENY_ALL_USER} = ${ACL_TEST_FILES}/gen_eacl_deny_all_USER
|
|
||||||
${EACL_ALLOW_ALL_USER} = ${ACL_TEST_FILES}/gen_eacl_allow_all_USER
|
|
||||||
|
|
||||||
${EACL_DENY_ALL_SYSTEM} = ${ACL_TEST_FILES}/gen_eacl_deny_all_SYSTEM
|
|
||||||
${EACL_ALLOW_ALL_SYSTEM} = ${ACL_TEST_FILES}/gen_eacl_allow_all_SYSTEM
|
|
||||||
|
|
||||||
${EACL_ALLOW_ALL_Pubkey} = ${ACL_TEST_FILES}/gen_eacl_allow_pubkey_deny_OTHERS
|
|
||||||
|
|
||||||
${EACL_COMPOUND_GET_OTHERS} = ${ACL_TEST_FILES}/gen_eacl_compound_get_OTHERS
|
|
||||||
${EACL_COMPOUND_GET_USER} = ${ACL_TEST_FILES}/gen_eacl_compound_get_USER
|
|
||||||
${EACL_COMPOUND_GET_SYSTEM} = ${ACL_TEST_FILES}/gen_eacl_compound_get_SYSTEM
|
|
||||||
|
|
||||||
${EACL_COMPOUND_DELETE_OTHERS} = ${ACL_TEST_FILES}/gen_eacl_compound_del_OTHERS
|
|
||||||
${EACL_COMPOUND_DELETE_USER} = ${ACL_TEST_FILES}/gen_eacl_compound_del_USER
|
|
||||||
${EACL_COMPOUND_DELETE_SYSTEM} = ${ACL_TEST_FILES}/gen_eacl_compound_del_SYSTEM
|
|
||||||
|
|
||||||
${EACL_COMPOUND_GET_HASH_OTHERS} = ${ACL_TEST_FILES}/gen_eacl_compound_get_hash_OTHERS
|
|
||||||
${EACL_COMPOUND_GET_HASH_USER} = ${ACL_TEST_FILES}/gen_eacl_compound_get_hash_USER
|
|
||||||
${EACL_COMPOUND_GET_HASH_SYSTEM} = ${ACL_TEST_FILES}/gen_eacl_compound_get_hash_SYSTEM
|
|
||||||
|
|
||||||
${EACL_XHEADER_DENY_ALL} = ${ACL_TEST_FILES}/gen_eacl_xheader_deny_all
|
|
||||||
${EACL_XHEADER_ALLOW_ALL} = ${ACL_TEST_FILES}/gen_eacl_xheader_allow_all
|
|
Loading…
Reference in a new issue