JSON compatibility for new message versions #11

New issue

Open

opened 2023-03-13 08:40:34 +00:00 by alexvanin · 0 comments

alexvanin commented

2023-03-13 08:40:34 +00:00

Owner

We use protobuf definition to keep compatibility when new version of message may contain some extra fields. Some structures are use both in binary format and JSON format, e.g. placement.Policy structure.

UnmarshalJSON functions seem to use default protojson.Unmarshal implementation. Default implementation produces error when it sees unknown field (similar to default JSON library implementation).

const policy = `
{
  "replicas": [
    {
      "count": 1,
      "selector": "X"
    }
  ],
  "containerBackupFactor": 1,
  "selectors": [
    {
      "name": "X",
      "count": 2,
      "clause": "CLAUSE_UNSPECIFIED",
      "filter": "*"
    }
  ],
  "filters": [],
  "newField": "newValue"
}
`
var p netmap.PlacementPolicy
err := p.UnmarshalJSON([]byte(policy)) // proto: (line 19:3): unknown field "newField"

Developers may use custom unmarshaler with DiscardUnknown flag.

um := protojson.UnmarshalOptions{
	DiscardUnknown: true,
}

var pv2 = new(netmapGRPC.PlacementPolicy)
err = um.Unmarshal([]byte(policy), pv2) // nil

Consider using custom unmarshaler for JSON format.

/cc @TrueCloudLab/storage-core-committers

We use protobuf definition to keep compatibility when new version of message may contain some extra fields. Some structures are use both in binary format and JSON format, e.g. `placement.Policy` structure. UnmarshalJSON functions seem to use default `protojson.Unmarshal` implementation. Default implementation produces error when it sees unknown field (similar to default JSON library implementation). ```go const policy = ` { "replicas": [ { "count": 1, "selector": "X" } ], "containerBackupFactor": 1, "selectors": [ { "name": "X", "count": 2, "clause": "CLAUSE_UNSPECIFIED", "filter": "*" } ], "filters": [], "newField": "newValue" } ` var p netmap.PlacementPolicy err := p.UnmarshalJSON([]byte(policy)) // proto: (line 19:3): unknown field "newField" ``` Developers may use custom unmarshaler with [DiscardUnknown](https://pkg.go.dev/google.golang.org/protobuf/encoding/protojson#UnmarshalOptions) flag. ```go um := protojson.UnmarshalOptions{ DiscardUnknown: true, } var pv2 = new(netmapGRPC.PlacementPolicy) err = um.Unmarshal([]byte(policy), pv2) // nil ``` Consider using custom unmarshaler for JSON format. /cc @TrueCloudLab/storage-core-committers