Kirill Sosnovskikh d050a94272 Fix determination of fixture parameters

- Added ability to pass any parameters to `params` keyword in `pytest.fixture`, except:
  1) `params=<callable object>`, but `params=[<callable object>, ...]` is allowed
  2) `params=[*<any collection>]`

- Analysis of `pytest_generate_tests` is almost equivalent to regular fixtures, except:
  1) `metafunc.parametrize(*<any collection>, **<any dict>)`
  2) `metafunc.parametrize("any", <callable object>)`
  3) `metafunc.parametrize("any1, any2", [(<callable object>, [*<any collection>])]`

Signed-off-by: Kirill Sosnovskikh <k.sosnovskikh@yadro.com>

2024-11-15 16:08:24 +03:00

3.4 KiB

Raw Permalink Blame History

allure-validator

Linter for checking uniqueness and correctness of allure.title in the entire test database.

Installation

pip install git+https://git.frostfs.info/TrueCloudLab/allure-validator.git@master

Usage

To start, simply specify the folder with the test database that needs to be checked:

allure-validator <folder> [OPTIONS]

Additionally, allure-validator can be added to pre-commit hooks:

# .pre-commit-config.yaml
repos:
  - repo: https://git.frostfs.info/TrueCloudLab/allure-validator
    rev: latest
    hooks:
      - id: allure-validator
        args: ["pytest_tests/"]
        pass_filenames: false

Ignore mechanism

The linter can be made to ignore functions/classes. To do this, just write a comment # noqa: allure-validator to the right of the function/class name. For example:

@pytest.mark.parametrize("test_param", [1, 2])
def test_ignored_function(test_param):  # noqa: allure-validator
    ...


@pytest.mark.parametrize("class_param", [1, 2])
class TestIngoredClass:  # noqa: allure-validator
    def test_1(self, class_param):
        ...

    def test_2(self, class_param):
        ...

Note: this may also miss fixtures, so use with caution.

Options

The linter supports the following flags:

--plugins - regex patterns that plugins must match (paths are compiled according to the template: plugin_name / src / plugin_name).
--files - paths to files with fixtures.

Configuration options can also be provided using special command line arguments, for example:

allure-validator <tests folder> --plugins <plugin regex>, ... --files <path to file with fixtures>, ...

Work example

Let's run the linter for some sample tests:

NOT UNIQUE TITLE: Not unique test
In the following tests:
        tests/test_validator.py:20:0: title is not unique by `test_not_unique_2`
        tests/test_validator.py:119:4: title is not unique by `test_not_unique_3`
        tests/test_validator.py:15:0: title is not unique by `test_not_unique_1`

NOT UNIQUE TITLE: Test title
In the following tests:
        tests/test_validator.py:30:0: title is not unique by `test_title_not_unique_2`
        tests/test_validator.py:25:0: title is not unique by `test_title_not_unique_1`

tests/test_validator.py:5:0: EMPTY TITLE: Missing title in `test_empty_title`
tests/test_validator.py:10:0: EMPTY TITLE: Missing title in `test_empty_title_2`
tests/test_validator.py:20:0: MISSING PARAMS: Parameters are missing from title: object_size
tests/test_validator.py:62:0: MISSING PARAMS: Parameters are missing from title: param_two OR param_three
tests/test_validator.py:115:4: MISSING PARAMS: Parameters are missing from title: root_fixture
tests/test_validator.py:119:4: MISSING PARAMS: Parameters are missing from title: param_class_1

Undetectable cases

Since the linter works with AST python code, any dynamic titles will not be found. For example:

import allure


def test_example():
    data = 10
    allure.dynamic.title(f"Dynamic title {data}")  # Undetected title
    ...

Dynamic fixtures are only supported if they are declared in pytest_generate_tests:

def pytest_generate_tests(metafunc):
    ...

    metafunc.parametrize("fixture1, fixture2", [(1, 2), (3, 4)])  # OK
    
    ...
    
    if some_condition:
        metafunc.parametrize("fixture3", *some_value_generator)  # OK

3.4 KiB Raw Permalink Blame History