forked from TrueCloudLab/frostfs-api-go
initial
This commit is contained in:
Evgeniy Kulikov
commit
1cf33e5ffd
87 changed files with 29835 additions and 0 deletions
143
object/doc.go
Normal file
143
object/doc.go
Normal file
|
|
@ -0,0 +1,143 @@
|
|||
/*
|
||||
Package object manages main storage structure in the system. All storage
|
||||
operations are performed with the objects. During lifetime object might be
|
||||
transformed into another object by cutting its payload or adding meta
|
||||
information. All transformation may be reversed, therefore source object
|
||||
will be able to restore.
|
||||
|
||||
Object structure
|
||||
|
||||
Object consists of Payload and Header. Payload is unlimited but storage nodes
|
||||
may have a policy to store objects with a limited payload. In this case object
|
||||
with large payload will be transformed into the chain of objects with small
|
||||
payload.
|
||||
|
||||
Headers are simple key-value fields that divided into two groups: system
|
||||
headers and extended headers. System headers contain information about
|
||||
protocol version, object id, payload length in bytes, owner id, container id
|
||||
and object creation timestamp (both in epochs and unix time). All these fields
|
||||
must be set up in the correct object.
|
||||
|
||||
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
|
||||
| System Headers |
|
||||
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
|
||||
| Version : 1 |
|
||||
| Payload Length : 21673465 |
|
||||
| Object ID : 465208e2-ba4f-4f99-ad47-82a59f4192d4 |
|
||||
| Owner ID : AShvoCbSZ7VfRiPkVb1tEcBLiJrcbts1tt |
|
||||
| Container ID : FGobtRZA6sBZv2i9k4L7TiTtnuP6E788qa278xfj3Fxj |
|
||||
| Created At : Epoch#10, 1573033162 |
|
||||
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
|
||||
| Extended Headers |
|
||||
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
|
||||
| User Header : <user-defined-key>, <user-defined-value> |
|
||||
| Verification Header : <session public key>, <owner's signature> |
|
||||
| Homomorphic Hash : 0x23d35a56ae... |
|
||||
| Payload Checksum : 0x1bd34abs75... |
|
||||
| Integrity Header : <header checksum>, <session signature> |
|
||||
| Transformation : Payload Split |
|
||||
| Link-parent : cae08935-b4ba-499a-bf6c-98276c1e6c0b |
|
||||
| Link-next : c3b40fbf-3798-4b61-a189-2992b5fb5070 |
|
||||
| Payload Checksum : 0x1f387a5c36... |
|
||||
| Integrity Header : <header checksum>, <session signature> |
|
||||
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
|
||||
| Payload |
|
||||
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
|
||||
| 0xd1581963a342d231... |
|
||||
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
|
||||
|
||||
There are different kinds of extended headers. A correct object must contain
|
||||
verification header, homomorphic hash header, payload checksum and
|
||||
integrity header. The order of headers is matter. Let's look through all
|
||||
these headers.
|
||||
|
||||
Link header points to the connected objects. During object transformation, large
|
||||
object might be transformed into the chain of smaller objects. One of these
|
||||
objects drops payload and has several "Child" links. We call this object as
|
||||
zero-object. Others will have "Parent" link to the zero-object, "Previous"
|
||||
and "Next" links in the payload chain.
|
||||
|
||||
[ Object ID:1 ] = > transformed
|
||||
`- [ Zero-Object ID:1 ]
|
||||
`- Link-child ID:2
|
||||
`- Link-child ID:3
|
||||
`- Link-child ID:4
|
||||
`- Payload [null]
|
||||
`- [ Object ID:2 ]
|
||||
`- Link-parent ID:1
|
||||
`- Link-next ID:3
|
||||
`- Payload [ 0x13ba... ]
|
||||
`- [ Object ID:3 ]
|
||||
`- Link-parent ID:1
|
||||
`- Link-previous ID:2
|
||||
`- Link-next ID:4
|
||||
`- Payload [ 0xcd34... ]
|
||||
`- [ Object ID:4 ]
|
||||
`- Link-parent ID:1
|
||||
`- Link-previous ID:3
|
||||
`- Payload [ 0xef86... ]
|
||||
|
||||
Storage groups are also objects. They have "Storage Group" links to all
|
||||
objects in the group. Links are set by nodes during transformations and,
|
||||
in general, they should not be set by user manually.
|
||||
|
||||
Redirect headers are not used yet, they will be implemented and described
|
||||
later.
|
||||
|
||||
User header is a key-value pair of string that can be defined by user. User
|
||||
can use these headers as search attribute. You can store any meta information
|
||||
about object there, e.g. object's nicename.
|
||||
|
||||
Transformation header notifies that object was transformed by some pre-defined
|
||||
way. This header sets up before object is transformed and all headers after
|
||||
transformation must be located after transformation header. During reverse
|
||||
transformation, all headers under transformation header will be cut out.
|
||||
|
||||
+-+-+-+-+-+-+-+-+-+- +-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+
|
||||
| Payload checksum | | Payload checksum | | Payload checksum |
|
||||
| Integrity header | => | Integrity header | + | Integrity header |
|
||||
+-+-+-+-+-+-+-+-+-+- | Transformation | | Transformation |
|
||||
| Large payload | | New Checksum | | New Checksum |
|
||||
+-+-+-+-+-+-+-+-+-+- | New Integrity | | New Integrity |
|
||||
+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+
|
||||
| Small payload | | Small payload |
|
||||
+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+
|
||||
|
||||
For now, we use only one type of transformation: payload split transformation.
|
||||
This header set up by node automatically.
|
||||
|
||||
Tombstone header notifies that this object was deleted by user. Objects with
|
||||
tombstone header do not have payload, but they still contain meta information
|
||||
in the headers. This way we implement two-phase commit for object removal.
|
||||
Storage nodes will eventually delete all tombstone objects. If you want to
|
||||
delete object, you must create new object with the same object id, with
|
||||
tombstone header, correct signatures and without payload.
|
||||
|
||||
Verification header contains session information. To put the object in
|
||||
the system user must create session. It is required because objects might
|
||||
be transformed and therefore must be re-signed. To do that node creates
|
||||
a pair of session public and private keys. Object owner delegates permission to
|
||||
re-sign objects by signing session public key. This header contains session
|
||||
public key and owner's signature of this key. You must specify this header
|
||||
manually.
|
||||
|
||||
Homomorphic hash header contains homomorphic hash of the source object.
|
||||
Transformations do not affect this header. This header used by data audit and
|
||||
set by node automatically.
|
||||
|
||||
Payload checksum contains checksum of the actual object payload. All payload
|
||||
transformation must set new payload checksum headers. This header set by node
|
||||
automatically.
|
||||
|
||||
Integrity header contains checksum of the header and signature of the
|
||||
session key. This header must be last in the list of extended headers.
|
||||
Checksum is calculated by marshaling all above headers, including system
|
||||
headers. This header set by node automatically.
|
||||
|
||||
Storage group header is presented in storage group objects. It contains
|
||||
information for data audit: size of validated data, homomorphic has of this
|
||||
data, storage group expiration time in epochs or unix time.
|
||||
|
||||
|
||||
*/
|
||||
package object
|
||||
Loading…
Add table
Add a link
Reference in a new issue