acid-ant/tzhash
1
0
Fork 0
forked from TrueCloudLab/tzhash
Code Pull requests Activity
tzhash/tz/digest_avx2_amd64.s

57 lines
1.3 KiB
ArmAsm
Raw Permalink Normal View History

Inline asm function in loop for AVX2 implementation Right now AVX2 implementation looses to C binding in speed. This is probably, because of 2 things: 1. Go does not inline `mulBitRightx2` in loop iteration. 2. `minmax` is loaded every time from memory. In this PR: 1. Unroll `mulBitRightx2` manually and use `mulByteRightx2` instead. 2. Generate `minmax` in place without `LOAD/LEA` instructions.
2019-07-19 13:11:01 +00:00
#include "textflag.h"
Interleave carry registers for successive bits 8 instructions less per byte. Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-30 08:29:59 +00:00
#define mulBit(bit, in_1, in_2, out_1, out_2) \
tz: optimize AVX2 implementation 1. Perform masking with 2 instructions instead of 3 (use arithmetic shift). 2. Broadcast data byte in one instruction at the start of byte-processing 3. Reorder instructions to reduce the amount of data hazards and resources contention. ``` name old time/op new time/op delta Sum/AVX2_digest-8 1.39ms ± 0% 1.22ms ± 0% -12.18% (p=0.000 n=9+7) name old speed new speed delta Sum/AVX2_digest-8 71.7MB/s ± 0% 81.7MB/s ± 0% +13.87% (p=0.000 n=9+7) ``` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-10 19:44:16 +00:00
VPSLLW bit, Y10, Y11 \
*: format assembly code with `asmfmt` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-09 11:38:53 +00:00
VPSLLQ $1, in_1, Y1 \
tz: optimize AVX2 implementation 1. Perform masking with 2 instructions instead of 3 (use arithmetic shift). 2. Broadcast data byte in one instruction at the start of byte-processing 3. Reorder instructions to reduce the amount of data hazards and resources contention. ``` name old time/op new time/op delta Sum/AVX2_digest-8 1.39ms ± 0% 1.22ms ± 0% -12.18% (p=0.000 n=9+7) name old speed new speed delta Sum/AVX2_digest-8 71.7MB/s ± 0% 81.7MB/s ± 0% +13.87% (p=0.000 n=9+7) ``` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-10 19:44:16 +00:00
VPSRAW $15, Y11, Y12 \
*: format assembly code with `asmfmt` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-09 11:38:53 +00:00
VPALIGNR $8, Y1, in_1, Y2 \
VPAND Y1, Y14, Y3 \
tz: optimize AVX2 implementation 1. Perform masking with 2 instructions instead of 3 (use arithmetic shift). 2. Broadcast data byte in one instruction at the start of byte-processing 3. Reorder instructions to reduce the amount of data hazards and resources contention. ``` name old time/op new time/op delta Sum/AVX2_digest-8 1.39ms ± 0% 1.22ms ± 0% -12.18% (p=0.000 n=9+7) name old speed new speed delta Sum/AVX2_digest-8 71.7MB/s ± 0% 81.7MB/s ± 0% +13.87% (p=0.000 n=9+7) ``` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-10 19:44:16 +00:00
VPSRLQ $63, Y2, Y2 \
*: format assembly code with `asmfmt` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-09 11:38:53 +00:00
VPUNPCKHQDQ Y3, Y3, Y3 \
tz: optimize AVX2 implementation 1. Perform masking with 2 instructions instead of 3 (use arithmetic shift). 2. Broadcast data byte in one instruction at the start of byte-processing 3. Reorder instructions to reduce the amount of data hazards and resources contention. ``` name old time/op new time/op delta Sum/AVX2_digest-8 1.39ms ± 0% 1.22ms ± 0% -12.18% (p=0.000 n=9+7) name old speed new speed delta Sum/AVX2_digest-8 71.7MB/s ± 0% 81.7MB/s ± 0% +13.87% (p=0.000 n=9+7) ``` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-10 19:44:16 +00:00
VPXOR Y1, Y2, Y7 \
*: format assembly code with `asmfmt` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-09 11:38:53 +00:00
VPXOR Y3, in_2, out_1 \
tz: optimize AVX2 implementation 1. Perform masking with 2 instructions instead of 3 (use arithmetic shift). 2. Broadcast data byte in one instruction at the start of byte-processing 3. Reorder instructions to reduce the amount of data hazards and resources contention. ``` name old time/op new time/op delta Sum/AVX2_digest-8 1.39ms ± 0% 1.22ms ± 0% -12.18% (p=0.000 n=9+7) name old speed new speed delta Sum/AVX2_digest-8 71.7MB/s ± 0% 81.7MB/s ± 0% +13.87% (p=0.000 n=9+7) ``` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-10 19:44:16 +00:00
VPXOR Y7, out_1, out_1 \
VPAND out_1, Y12, Y4 \
*: format assembly code with `asmfmt` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-09 11:38:53 +00:00
VPXOR Y4, in_1, out_2 \
Use macros in asm code
2019-10-09 15:11:53 +00:00
Rewrite AVX2 loop in assembly Helps to get rid of MOV and generating constants for each iteration. ``` name old time/op new time/op delta Sum/AVX2Inline_digest-8 1.57ms ± 2% 1.41ms ± 0% -10.52% (p=0.000 n=9+9) name old speed new speed delta Sum/AVX2Inline_digest-8 63.6MB/s ± 1% 71.1MB/s ± 0% +11.76% (p=0.000 n=9+9) ``` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-10 15:01:15 +00:00
// func mulByteSliceRightx2(c00c10, c01c11 *[4]uint64, n int, data *byte)
*: format assembly code with `asmfmt` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-09 11:38:53 +00:00
TEXT ·mulByteSliceRightx2(SB), NOSPLIT, $0
tz: optimize AVX2 implementation 1. Perform masking with 2 instructions instead of 3 (use arithmetic shift). 2. Broadcast data byte in one instruction at the start of byte-processing 3. Reorder instructions to reduce the amount of data hazards and resources contention. ``` name old time/op new time/op delta Sum/AVX2_digest-8 1.39ms ± 0% 1.22ms ± 0% -12.18% (p=0.000 n=9+7) name old speed new speed delta Sum/AVX2_digest-8 71.7MB/s ± 0% 81.7MB/s ± 0% +13.87% (p=0.000 n=9+7) ``` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-10 19:44:16 +00:00
MOVQ c00c10+0(FP), AX
MOVQ c01c11+8(FP), BX
Optimize AVX2 implementation We use 6 instructions only to calculate mask based on single bit value. Use only 3 now and calculate multiple masks in parallel. Also `VPSUB*` is faster than VPBROADCAST*, see https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html . ``` name old time/op new time/op delta Sum/AVX2Inline_digest-8 1.83ms ± 0% 1.62ms ± 1% -11.23% (p=0.000 n=46+42) name old speed new speed delta Sum/AVX2Inline_digest-8 54.7MB/s ± 0% 61.6MB/s ± 1% +12.65% (p=0.000 n=46+42) ``` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-28 17:24:59 +00:00
*: format assembly code with `asmfmt` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-09 11:38:53 +00:00
VPXOR Y13, Y13, Y13 // Y13 = 0x0000...
VPCMPEQB Y14, Y14, Y14 // Y14 = 0xFFFF...
VPSUBQ Y14, Y13, Y10
VPSLLQ $63, Y10, Y14 // Y14 = 0x10000000... (packed quad-words with HSB set)
Optimize AVX2 implementation We use 6 instructions only to calculate mask based on single bit value. Use only 3 now and calculate multiple masks in parallel. Also `VPSUB*` is faster than VPBROADCAST*, see https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html . ``` name old time/op new time/op delta Sum/AVX2Inline_digest-8 1.83ms ± 0% 1.62ms ± 1% -11.23% (p=0.000 n=46+42) name old speed new speed delta Sum/AVX2Inline_digest-8 54.7MB/s ± 0% 61.6MB/s ± 1% +12.65% (p=0.000 n=46+42) ``` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-28 17:24:59 +00:00
*: format assembly code with `asmfmt` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-09 11:38:53 +00:00
MOVQ n+16(FP), CX
MOVQ data+24(FP), DX
Rewrite AVX2 loop in assembly Helps to get rid of MOV and generating constants for each iteration. ``` name old time/op new time/op delta Sum/AVX2Inline_digest-8 1.57ms ± 2% 1.41ms ± 0% -10.52% (p=0.000 n=9+9) name old speed new speed delta Sum/AVX2Inline_digest-8 63.6MB/s ± 1% 71.1MB/s ± 0% +11.76% (p=0.000 n=9+9) ``` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-10 15:01:15 +00:00
tz: optimize AVX2 implementation 1. Perform masking with 2 instructions instead of 3 (use arithmetic shift). 2. Broadcast data byte in one instruction at the start of byte-processing 3. Reorder instructions to reduce the amount of data hazards and resources contention. ``` name old time/op new time/op delta Sum/AVX2_digest-8 1.39ms ± 0% 1.22ms ± 0% -12.18% (p=0.000 n=9+7) name old speed new speed delta Sum/AVX2_digest-8 71.7MB/s ± 0% 81.7MB/s ± 0% +13.87% (p=0.000 n=9+7) ``` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-10 19:44:16 +00:00
VMOVDQU (AX), Y0
VMOVDQU (BX), Y8
Rewrite AVX2 loop in assembly Helps to get rid of MOV and generating constants for each iteration. ``` name old time/op new time/op delta Sum/AVX2Inline_digest-8 1.57ms ± 2% 1.41ms ± 0% -10.52% (p=0.000 n=9+9) name old speed new speed delta Sum/AVX2Inline_digest-8 63.6MB/s ± 1% 71.1MB/s ± 0% +11.76% (p=0.000 n=9+9) ``` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-10 15:01:15 +00:00
loop:
*: format assembly code with `asmfmt` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-09 11:38:53 +00:00
CMPQ CX, $0
JEQ finish
Rewrite AVX2 loop in assembly Helps to get rid of MOV and generating constants for each iteration. ``` name old time/op new time/op delta Sum/AVX2Inline_digest-8 1.57ms ± 2% 1.41ms ± 0% -10.52% (p=0.000 n=9+9) name old speed new speed delta Sum/AVX2Inline_digest-8 63.6MB/s ± 1% 71.1MB/s ± 0% +11.76% (p=0.000 n=9+9) ``` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-10 15:01:15 +00:00
tz: optimize AVX2 implementation 1. Perform masking with 2 instructions instead of 3 (use arithmetic shift). 2. Broadcast data byte in one instruction at the start of byte-processing 3. Reorder instructions to reduce the amount of data hazards and resources contention. ``` name old time/op new time/op delta Sum/AVX2_digest-8 1.39ms ± 0% 1.22ms ± 0% -12.18% (p=0.000 n=9+7) name old speed new speed delta Sum/AVX2_digest-8 71.7MB/s ± 0% 81.7MB/s ± 0% +13.87% (p=0.000 n=9+7) ``` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-10 19:44:16 +00:00
VPBROADCASTB (DX), Y10
*: format assembly code with `asmfmt` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-09 11:38:53 +00:00
ADDQ $1, DX
tz: optimize AVX2 implementation 1. Perform masking with 2 instructions instead of 3 (use arithmetic shift). 2. Broadcast data byte in one instruction at the start of byte-processing 3. Reorder instructions to reduce the amount of data hazards and resources contention. ``` name old time/op new time/op delta Sum/AVX2_digest-8 1.39ms ± 0% 1.22ms ± 0% -12.18% (p=0.000 n=9+7) name old speed new speed delta Sum/AVX2_digest-8 71.7MB/s ± 0% 81.7MB/s ± 0% +13.87% (p=0.000 n=9+7) ``` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-10 19:44:16 +00:00
SUBQ $1, CX
Inline asm function in loop for AVX2 implementation Right now AVX2 implementation looses to C binding in speed. This is probably, because of 2 things: 1. Go does not inline `mulBitRightx2` in loop iteration. 2. `minmax` is loaded every time from memory. In this PR: 1. Unroll `mulBitRightx2` manually and use `mulByteRightx2` instead. 2. Generate `minmax` in place without `LOAD/LEA` instructions.
2019-07-19 13:11:01 +00:00
tz: optimize AVX2 implementation 1. Perform masking with 2 instructions instead of 3 (use arithmetic shift). 2. Broadcast data byte in one instruction at the start of byte-processing 3. Reorder instructions to reduce the amount of data hazards and resources contention. ``` name old time/op new time/op delta Sum/AVX2_digest-8 1.39ms ± 0% 1.22ms ± 0% -12.18% (p=0.000 n=9+7) name old speed new speed delta Sum/AVX2_digest-8 71.7MB/s ± 0% 81.7MB/s ± 0% +13.87% (p=0.000 n=9+7) ``` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-10 19:44:16 +00:00
mulBit($8, Y0, Y8, Y5, Y6)
mulBit($9, Y5, Y6, Y0, Y8)
mulBit($10, Y0, Y8, Y5, Y6)
mulBit($11, Y5, Y6, Y0, Y8)
mulBit($12, Y0, Y8, Y5, Y6)
mulBit($13, Y5, Y6, Y0, Y8)
mulBit($14, Y0, Y8, Y5, Y6)
mulBit($15, Y5, Y6, Y0, Y8)
Inline asm function in loop for AVX2 implementation Right now AVX2 implementation looses to C binding in speed. This is probably, because of 2 things: 1. Go does not inline `mulBitRightx2` in loop iteration. 2. `minmax` is loaded every time from memory. In this PR: 1. Unroll `mulBitRightx2` manually and use `mulByteRightx2` instead. 2. Generate `minmax` in place without `LOAD/LEA` instructions.
2019-07-19 13:11:01 +00:00
*: format assembly code with `asmfmt` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-09 11:38:53 +00:00
JMP loop
Rewrite AVX2 loop in assembly Helps to get rid of MOV and generating constants for each iteration. ``` name old time/op new time/op delta Sum/AVX2Inline_digest-8 1.57ms ± 2% 1.41ms ± 0% -10.52% (p=0.000 n=9+9) name old speed new speed delta Sum/AVX2Inline_digest-8 63.6MB/s ± 1% 71.1MB/s ± 0% +11.76% (p=0.000 n=9+9) ``` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-10 15:01:15 +00:00
finish:
*: format assembly code with `asmfmt` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-09 11:38:53 +00:00
VMOVDQU Y0, (AX)
tz: optimize AVX2 implementation 1. Perform masking with 2 instructions instead of 3 (use arithmetic shift). 2. Broadcast data byte in one instruction at the start of byte-processing 3. Reorder instructions to reduce the amount of data hazards and resources contention. ``` name old time/op new time/op delta Sum/AVX2_digest-8 1.39ms ± 0% 1.22ms ± 0% -12.18% (p=0.000 n=9+7) name old speed new speed delta Sum/AVX2_digest-8 71.7MB/s ± 0% 81.7MB/s ± 0% +13.87% (p=0.000 n=9+7) ``` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-10 19:44:16 +00:00
VMOVDQU Y8, (BX)
Use macros in asm code
2019-10-09 15:11:53 +00:00
*: format assembly code with `asmfmt` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-09 11:38:53 +00:00
RET
Copy permalink