stackitem: add limit to serialized items

Standard binary serialization/deserialization is mostly used in VM to put/get
elements into/from storage, so they never should exceed MaxSize (otherwise one
won't be able to deserialize these items).

This patch leaves EncodeBinaryStackItem unprotected, but that's a streaming
interface, so it's up to the user of it to ensure its appropriate use (and our
uses are mostly for native contract's data, so they're fine).

pkg/vm/stackitem/serialization_test.go

@@ -0,0 +1,21 @@
+package stackitem
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/require"
+)
+
+func TestSerializationMaxErr(t *testing.T) {
+	base := make([]byte, MaxSize/2+1)
+	item := Make(base)
+
+	arr := []Item{item, item.Dup()}
+	aitem := Make(arr)
+
+	_, err := SerializeItem(item)
+	require.NoError(t, err)
+
+	_, err = SerializeItem(aitem)
+	require.Error(t, err)
+}