Commit graph

1301 commits

Author SHA1 Message Date
ff4f31b6f3 [#103] protogen: Handle files in all packages
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-09 11:18:17 +03:00
a0a9b765f3 [#101] Fix make test
All checks were successful
DCO action / DCO (pull_request) Successful in 43s
Tests and linters / Tests (1.19) (pull_request) Successful in 34s
Tests and linters / Tests (1.20) (pull_request) Successful in 38s
Tests and linters / Lint (pull_request) Successful in 57s
Tests and linters / Tests with -race (pull_request) Successful in 1m5s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-08-09 09:35:25 +03:00
280d052cef [#101] Remove usage of folder vendor
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-08-08 17:38:49 +03:00
35e7397d48 [#87] netmap: Extend enum Operation
All checks were successful
DCO action / DCO (pull_request) Successful in 39s
Tests and linters / Tests (1.19) (pull_request) Successful in 44s
Tests and linters / Tests (1.20) (pull_request) Successful in 51s
Tests and linters / Tests with -race (pull_request) Successful in 1m17s
Tests and linters / Lint (pull_request) Successful in 1m59s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-08-07 16:21:50 +03:00
174773454e [#87] netmap: Regenerate to add LIKE operation for filter
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-08-06 17:20:29 +03:00
b72aa14bab [#87] proto: Process files with protoc version 27.2
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-08-06 17:20:29 +03:00
42e50c9633 [#87] Makefile: Add target protoc-install
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-08-06 17:20:21 +03:00
611355510c [#87] go.mod: Update google.golang.org/grpc to v1.63.2
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-08-06 16:57:57 +03:00
ebaf78c8fa [#100] session: Introduce ObjectPatch verb
All checks were successful
DCO action / DCO (pull_request) Successful in 39s
Tests and linters / Tests (1.19) (pull_request) Successful in 51s
Tests and linters / Tests (1.20) (pull_request) Successful in 52s
Tests and linters / Lint (pull_request) Successful in 1m3s
Tests and linters / Tests with -race (pull_request) Successful in 1m7s
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-06 12:31:11 +03:00
1473fa588f [#98] rpc: Accept interface in place of ClientConn
All checks were successful
Tests and linters / Tests (1.19) (pull_request) Successful in 48s
DCO action / DCO (pull_request) Successful in 44s
Tests and linters / Tests (1.20) (pull_request) Successful in 1m2s
Tests and linters / Tests with -race (pull_request) Successful in 1m9s
Tests and linters / Lint (pull_request) Successful in 1m29s
gRPC client load-balancing API is ugly as f:
1. It is configured by pre-registering a balancer and the providing JSON
   configuration.
2. It doesn't allow different credentials for different endpoints
   (consider using "insecure" localhost and external endpoint).
3. To support frostfs usecase we also need to implement a resolver,
   which has its own difficulties.
4. https://github.com/grpc/grpc-go/issues/239#issuecomment-264548415

Using interface in place of grpc.ClientConn allows us to provide custom
implentation for it (load-balancing, circuit breaker etc.).

Refs TrueCloudLab/frostfs-node#1268

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-01 08:50:24 +03:00
c27b978770 [#97] signature: Add Patch messages to serviceMessageBody
All checks were successful
DCO action / DCO (pull_request) Successful in 54s
Tests and linters / Tests (1.20) (pull_request) Successful in 50s
Tests and linters / Tests (1.19) (pull_request) Successful in 1m0s
Tests and linters / Lint (pull_request) Successful in 1m3s
Tests and linters / Tests with -race (pull_request) Successful in 1m5s
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-30 17:52:54 +03:00
8609f29a60 [#97] object: Refactor Patch related structs
* Add getters and setters for related types;
* Fix unit-tests.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-30 17:52:49 +03:00
8ce8cd6ec2 [#96] .golangci.yml: Fix deprecated config options
All checks were successful
DCO action / DCO (pull_request) Successful in 55s
Tests and linters / Tests (1.19) (pull_request) Successful in 52s
Tests and linters / Tests (1.20) (pull_request) Successful in 54s
Tests and linters / Tests with -race (pull_request) Successful in 1m18s
Tests and linters / Lint (pull_request) Successful in 2m35s
```
WARN [config_reader] The configuration option `run.skip-files` is deprecated, please use `issues.exclude-files`.
WARN [config_reader] The configuration option `output.format` is deprecated, please use `output.formats`
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-29 15:45:01 +03:00
8580b49c8d [#94] rpc: Introduce ObjectService.Patch method
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-29 08:55:14 +00:00
9b90d139c5 [#94] object: Generate protobufs for Patch method
* Generate protobufs for patch method;
* Create marshalers, unmarshalers, converters for gererated types;
* Add unit-tests.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-29 08:55:14 +00:00
3dfa2f4fd6 [#95] *: Regenerate proto files
All checks were successful
DCO action / DCO (pull_request) Successful in 1m0s
Tests and linters / Tests (1.19) (pull_request) Successful in 1m22s
Tests and linters / Tests (1.20) (pull_request) Successful in 1m25s
Tests and linters / Lint (pull_request) Successful in 1m32s
Tests and linters / Tests with -race (pull_request) Successful in 2m20s
Remove SetExtendedEACL and AnnounceUsedSpace methods from the container
package.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-26 10:24:25 +03:00
f517e39491 [#91] protogen: Support unpacked repeated uint64 fields
All checks were successful
DCO action / DCO (pull_request) Successful in 1m18s
Tests and linters / Lint (pull_request) Successful in 1m31s
Tests and linters / Tests (1.20) (pull_request) Successful in 1m39s
Tests and linters / Tests (1.19) (pull_request) Successful in 1m45s
Tests and linters / Tests with -race (pull_request) Successful in 2m21s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-16 14:39:20 +03:00
3639563d80 [#90] apemanager: Run gofumpt
All checks were successful
DCO action / DCO (pull_request) Successful in 1m43s
Tests and linters / Lint (pull_request) Successful in 3m2s
Tests and linters / Tests (1.19) (pull_request) Successful in 2m57s
Tests and linters / Tests (1.20) (pull_request) Successful in 3m0s
Tests and linters / Tests with -race (pull_request) Successful in 3m41s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-15 16:40:21 +03:00
610c450a65 [#90] proto/test: Fix go vet warnings
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-15 16:40:21 +03:00
3f92d7bfb0 [#90] proto/test: Fix proto file formatting
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-15 16:40:21 +03:00
dafc9e5476 [#86] status: Regenerate common status
All checks were successful
DCO action / DCO (pull_request) Successful in 1m35s
Tests and linters / Tests (1.20) (pull_request) Successful in 1m38s
Tests and linters / Tests (1.19) (pull_request) Successful in 1m54s
Tests and linters / Tests with -race (pull_request) Successful in 2m16s
Tests and linters / Lint (pull_request) Successful in 6m8s
* INVALID_ARGUMENT is a new common status constant.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-06-24 11:55:48 +03:00
2f6d3209e1 [#84] object: Regenerate EC-header type
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-30 15:28:26 +00:00
9e825239ac [#85] acl: Regenerate protobufs for Bearer token
All checks were successful
DCO action / DCO (pull_request) Successful in 57s
Tests and linters / Tests (1.20) (pull_request) Successful in 1m31s
Tests and linters / Lint (pull_request) Successful in 1m42s
Tests and linters / Tests (1.19) (pull_request) Successful in 1m41s
Tests and linters / Tests with -race (pull_request) Successful in 2m17s
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-29 19:45:44 +03:00
9789b79b1d [#85] apemanager: Generate protobufs for ape package
* Regenerate protobufs as APE specific type are defined
  as separate package;
* Move `ape` package related utils methods from `apemanager`.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-29 14:15:49 +03:00
0803bc6ded [#83] object: Regenerate protobufs for ECHeader
All checks were successful
Tests and linters / Tests (1.19) (pull_request) Successful in 11m59s
Tests and linters / Tests (1.20) (pull_request) Successful in 12m5s
DCO action / DCO (pull_request) Successful in 12m27s
Tests and linters / Tests with -race (pull_request) Successful in 12m25s
Tests and linters / Lint (pull_request) Successful in 12m52s
* Fix marshalers and converters;
* Fix unit-tests.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-16 16:31:03 +03:00
4fe42ac4ad [#82] object: Add FilterHeaderECParent filter for v2
All checks were successful
DCO action / DCO (pull_request) Successful in 1m0s
Tests and linters / Tests (1.20) (pull_request) Successful in 1m20s
Tests and linters / Tests (1.19) (pull_request) Successful in 1m30s
Tests and linters / Lint (pull_request) Successful in 1m53s
Tests and linters / Tests with -race (pull_request) Successful in 1m49s
* `FilterHeaderECParent` is used by `Object.Search`

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-13 17:03:55 +03:00
063ce11c24 [#82] object: Erase field Signature from ECHeader v2 type
All checks were successful
DCO action / DCO (pull_request) Successful in 1m17s
Tests and linters / Tests (1.20) (pull_request) Successful in 1m45s
Tests and linters / Tests (1.19) (pull_request) Successful in 1m58s
Tests and linters / Lint (pull_request) Successful in 2m6s
Tests and linters / Tests with -race (pull_request) Successful in 2m9s
* The field `Signature` is not used and should be factored out.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-08 15:09:16 +03:00
bc7b49eed2 [#82] object: Introduce new fields for EC header
* Regenerate protobufs as frostfs-api introduced `parent_split_id` and
  `parent_split_index` fields.
* Fix marshaller and converter for EC header.
* Extend v2 type `ECHeader` with `ParentSplitID` and `ParentSplitIndex` fields.
* Fix message_test for `Object`. Also generate EC headers and check it.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-08 15:09:12 +03:00
b171364079 [#81] rpc: Fix apemanager rpc service name
All checks were successful
DCO action / DCO (pull_request) Successful in 1m4s
Tests and linters / Tests (1.19) (pull_request) Successful in 1m10s
Tests and linters / Tests (1.20) (pull_request) Successful in 1m27s
Tests and linters / Lint (pull_request) Successful in 1m44s
Tests and linters / Tests with -race (pull_request) Successful in 1m48s
* Introduce `frostfsServiceNamePrefix` for rpc interface
  since `apemanager` uses `frostfs` root instead `neo.fs`.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-06 14:46:54 +03:00
0fe6b9adbb [#76] Make setter names more consistent
All checks were successful
DCO action / DCO (pull_request) Successful in 1m2s
Tests and linters / Tests (1.20) (pull_request) Successful in 1m33s
Tests and linters / Tests (1.19) (pull_request) Successful in 1m38s
Tests and linters / Lint (pull_request) Successful in 1m46s
Tests and linters / Tests with -race (pull_request) Successful in 1m44s
This will make the transition to a new codegen more seamless.
1. Setter should be named as the field itself.
2. Setter should accept 1 parameter.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-05-03 08:52:04 +03:00
fa8f92d662 [#75] audit: Remove deprecated service
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-05-02 11:54:35 +00:00
387b850e5e [#74] apemanager: Generate protobufs for apemanager service
* Generate protobufs.
* Make marshallers, unmarshallers, json-encoders etc.
* Create message encoding/decoding unit-tests.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-02 11:53:26 +00:00
67c6f305b2 [#79] object: Implement Range\RangeHash requests for EC object
All checks were successful
DCO action / DCO (pull_request) Successful in 1m16s
Tests and linters / Tests (1.20) (pull_request) Successful in 1m37s
Tests and linters / Lint (pull_request) Successful in 1m56s
Tests and linters / Tests (1.19) (pull_request) Successful in 2m9s
Tests and linters / Tests with -race (pull_request) Successful in 2m2s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-04-27 23:04:46 +03:00
df9b65324a [#73] object: Implement Get\Head requests for EC object
All checks were successful
DCO action / DCO (pull_request) Successful in 45s
Tests and linters / Tests (1.20) (pull_request) Successful in 1m13s
Tests and linters / Tests (1.19) (pull_request) Successful in 1m27s
Tests and linters / Lint (pull_request) Successful in 1m35s
Tests and linters / Tests with -race (pull_request) Successful in 1m58s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-04-22 18:14:50 +03:00
4a330a5706 [#72] session: Unify unmarshaling code
All checks were successful
DCO action / DCO (pull_request) Successful in 1m1s
Tests and linters / Tests (1.19) (pull_request) Successful in 1m24s
Tests and linters / Tests (1.20) (pull_request) Successful in 1m32s
Tests and linters / Lint (pull_request) Successful in 1m43s
Tests and linters / Tests with -race (pull_request) Successful in 2m1s
gopatch:
```
@@
var err, msg, data, t, protojson identifier
var obj expression
@@
+import "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc/message"
-import protojson "google.golang.org/protobuf/encoding/protojson"

-msg := obj
-if err := protojson.Unmarshal(data, msg); err != nil {
-return err
-}
-return t.FromGRPCMessage(msg)
+return message.UnmarshalJSON(t, data, obj)

@@
var err, msg, data, t, goproto identifier
var obj expression
@@
+import "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc/message"
-import goproto "google.golang.org/protobuf/proto"

-msg := obj
-if err := goproto.Unmarshal(data, msg); err != nil {
-return err
-}
-return t.FromGRPCMessage(msg)
+return message.Unmarshal(t, data, obj)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-20 14:19:19 +03:00
491a47e7fe [#70] netmap: Regenerate protobuf for Replica
All checks were successful
DCO action / DCO (pull_request) Successful in 1m3s
Tests and linters / Tests (1.19) (pull_request) Successful in 1m32s
Tests and linters / Tests (1.20) (pull_request) Successful in 1m28s
Tests and linters / Lint (pull_request) Successful in 1m38s
Tests and linters / Tests with -race (pull_request) Successful in 2m0s
* Since EC field tags are renumerated, it is required
  to regenerate netmap grpc.
* Add a generator with EC replica to check it with
  unit-test.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-03-27 12:56:03 +03:00
6e9d385f3c [#71] container: Introduce CONTAINER_ACCESS_DENIED status
All checks were successful
DCO action / DCO (pull_request) Successful in 1m19s
Tests and linters / Tests (1.20) (pull_request) Successful in 1m56s
Tests and linters / Tests (1.19) (pull_request) Successful in 2m2s
Tests and linters / Lint (pull_request) Successful in 2m12s
Tests and linters / Tests with -race (pull_request) Successful in 2m28s
* Regenerate protobufs for object and container service to
  use a new frostfs-api status CONTAINER_ACCESS_DENIED.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-03-26 18:55:05 +03:00
1772b92182 [#69] object: Initial EC implementation
All checks were successful
DCO action / DCO (pull_request) Successful in 1m7s
Tests and linters / Tests (1.19) (pull_request) Successful in 1m37s
Tests and linters / Tests (1.20) (pull_request) Successful in 1m33s
Tests and linters / Lint (pull_request) Successful in 1m46s
Tests and linters / Tests with -race (pull_request) Successful in 1m45s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-03-19 15:23:01 +03:00
a85146250b [#68] Makefile: Use gofumpt for formatting
All checks were successful
DCO action / DCO (pull_request) Successful in 1m17s
Tests and linters / Tests (1.20) (pull_request) Successful in 2m22s
Tests and linters / Lint (pull_request) Successful in 2m51s
Tests and linters / Tests (1.19) (pull_request) Successful in 2m48s
Tests and linters / Tests with -race (pull_request) Successful in 2m40s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-03-14 10:31:58 +03:00
ea6e390c7b [#68] pre-commit: Add unit test hook
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-03-14 10:31:58 +03:00
4d35da7c77 [#68] Makefile: Allow to override testflags
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-03-14 10:31:58 +03:00
a7bcad6aa9 [#68] Makefile: Drop GO111MODULE usages
It is ignored since go1.17.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-03-14 10:31:58 +03:00
6fb9cae479 [#68] pre-commit: Update golangci-lint to 1.56.2
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-03-14 10:31:58 +03:00
f69ad7ade0 [#68] pre-commit: Remove gitlint
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-03-14 10:31:58 +03:00
c1c7b344b9 [#67] go.mod: Bump protobuf version
All checks were successful
DCO action / DCO (pull_request) Successful in 6m8s
Tests and linters / Tests (1.19) (pull_request) Successful in 6m16s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m27s
Tests and linters / Tests with -race (pull_request) Successful in 6m40s
Tests and linters / Lint (pull_request) Successful in 7m0s
Found by vulncheck:
Vulnerability #1: GO-2024-2611
    Infinite loop in JSON unmarshaling in google.golang.org/protobuf
  More info: https://pkg.go.dev/vuln/GO-2024-2611
  Module: google.golang.org/protobuf
    Found in: google.golang.org/protobuf@v1.32.0
    Fixed in: google.golang.org/protobuf@v1.33.0

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-06 13:18:14 +03:00
634e24aba7 [#65] object: Fix SetMarshalData for PutSingle request
All checks were successful
DCO action / DCO (pull_request) Successful in 1m11s
Tests and linters / Tests (1.20) (pull_request) Successful in 1m25s
Tests and linters / Tests (1.19) (pull_request) Successful in 1m33s
Tests and linters / Tests with -race (pull_request) Successful in 1m56s
Tests and linters / Lint (pull_request) Successful in 1m14s
Allow to reset marshal data

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-15 15:44:01 +03:00
2a124b95bc [#64] object: Allow to reset marshal data
All checks were successful
DCO action / DCO (pull_request) Successful in 58s
Tests and linters / Tests (1.19) (pull_request) Successful in 1m24s
Tests and linters / Tests (1.20) (pull_request) Successful in 1m32s
Tests and linters / Lint (pull_request) Successful in 1m43s
Tests and linters / Tests with -race (pull_request) Successful in 2m5s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-15 14:47:28 +03:00
d60ce83e42 [#63] .forgejo: Update dco-go to v3
All checks were successful
DCO action / DCO (pull_request) Successful in 5m9s
Tests and linters / Tests (1.20) (pull_request) Successful in 7m17s
Tests and linters / Tests with -race (pull_request) Successful in 8m39s
Tests and linters / Tests (1.19) (pull_request) Successful in 10m2s
Tests and linters / Lint (pull_request) Successful in 12m37s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-01-26 12:19:53 +03:00
72885aae83 [#62] signature: Refactor BufferPool
All checks were successful
Tests and linters / Tests (1.19) (pull_request) Successful in 7m46s
DCO action / DCO (pull_request) Successful in 8m16s
Tests and linters / Tests (1.20) (pull_request) Successful in 9m24s
Tests and linters / Tests with -race (pull_request) Successful in 9m25s
Tests and linters / Lint (pull_request) Successful in 9m49s
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-01-12 18:09:28 +03:00
b46e8cfbda [#59] util: Rename stableMarshaler
All checks were successful
DCO action / DCO (pull_request) Successful in 1m23s
Tests and linters / Tests (1.20) (pull_request) Successful in 1m46s
Tests and linters / Lint (pull_request) Successful in 1m59s
Tests and linters / Tests (1.19) (pull_request) Successful in 2m7s
Tests and linters / Tests with -race (pull_request) Successful in 2m22s
It should be with a single `l`, see `json.Marshaler`.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-11-21 15:28:03 +03:00