acid-ant/tzhash
1
0
Fork 0
forked from TrueCloudLab/tzhash
Code Pull requests Activity
tzhash/tz/avx2_amd64.s

88 lines
2.1 KiB
ArmAsm
Raw Normal View History

Inline asm function in loop for AVX2 implementation Right now AVX2 implementation looses to C binding in speed. This is probably, because of 2 things: 1. Go does not inline `mulBitRightx2` in loop iteration. 2. `minmax` is loaded every time from memory. In this PR: 1. Unroll `mulBitRightx2` manually and use `mulByteRightx2` instead. 2. Generate `minmax` in place without `LOAD/LEA` instructions.
2019-07-19 13:11:01 +00:00
#include "textflag.h"
Optimize AVX2 implementation We use 6 instructions only to calculate mask based on single bit value. Use only 3 now and calculate multiple masks in parallel. Also `VPSUB*` is faster than VPBROADCAST*, see https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html . ``` name old time/op new time/op delta Sum/AVX2Inline_digest-8 1.83ms ± 0% 1.62ms ± 1% -11.23% (p=0.000 n=46+42) name old speed new speed delta Sum/AVX2Inline_digest-8 54.7MB/s ± 0% 61.6MB/s ± 1% +12.65% (p=0.000 n=46+42) ``` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-28 17:24:59 +00:00
#define mask(bit, tmp, to) \
VPSRLW bit, Y10, tmp \
Fix comments Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-30 08:40:35 +00:00
VPAND Y12, tmp, to \ // to = 0x000<bit>000<bit>...
VPSUBW to, Y13, to // to = 0xFFFF.. or 0x0000 depending on bit
Use macros in asm code
2019-10-09 15:11:53 +00:00
Interleave carry registers for successive bits 8 instructions less per byte. Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-30 08:29:59 +00:00
#define mulBit(bit, in_1, in_2, out_1, out_2) \
VPSLLQ $1, in_1, Y1 \
VPALIGNR $8, Y1, in_1, Y2 \
Use macros in asm code
2019-10-09 15:11:53 +00:00
VPSRLQ $63, Y2, Y2 \
VPXOR Y1, Y2, Y2 \
Replace two shifts with a single AND We need to isolate HSB in every quad-word, this can be done with a simple mask. Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-29 11:49:20 +00:00
VPAND Y1, Y14, Y3 \
Use macros in asm code
2019-10-09 15:11:53 +00:00
VPUNPCKHQDQ Y3, Y3, Y3 \
VPXOR Y2, Y3, Y3 \
Optimize AVX2 implementation We use 6 instructions only to calculate mask based on single bit value. Use only 3 now and calculate multiple masks in parallel. Also `VPSUB*` is faster than VPBROADCAST*, see https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html . ``` name old time/op new time/op delta Sum/AVX2Inline_digest-8 1.83ms ± 0% 1.62ms ± 1% -11.23% (p=0.000 n=46+42) name old speed new speed delta Sum/AVX2Inline_digest-8 54.7MB/s ± 0% 61.6MB/s ± 1% +12.65% (p=0.000 n=46+42) ``` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-28 17:24:59 +00:00
mask(bit, Y11, Y2) \
Interleave carry registers for successive bits 8 instructions less per byte. Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-30 08:29:59 +00:00
VPXOR Y3, in_2, out_1 \
VPAND out_1, Y2, Y4 \
VPXOR Y4, in_1, out_2 \
Use macros in asm code
2019-10-09 15:11:53 +00:00
Rewrite AVX2 loop in assembly Helps to get rid of MOV and generating constants for each iteration. ``` name old time/op new time/op delta Sum/AVX2Inline_digest-8 1.57ms ± 2% 1.41ms ± 0% -10.52% (p=0.000 n=9+9) name old speed new speed delta Sum/AVX2Inline_digest-8 63.6MB/s ± 1% 71.1MB/s ± 0% +11.76% (p=0.000 n=9+9) ``` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-10 15:01:15 +00:00
// func mulByteSliceRightx2(c00c10, c01c11 *[4]uint64, n int, data *byte)
TEXT ·mulByteSliceRightx2(SB),NOSPLIT,$0
Inline asm function in loop for AVX2 implementation Right now AVX2 implementation looses to C binding in speed. This is probably, because of 2 things: 1. Go does not inline `mulBitRightx2` in loop iteration. 2. `minmax` is loaded every time from memory. In this PR: 1. Unroll `mulBitRightx2` manually and use `mulByteRightx2` instead. 2. Generate `minmax` in place without `LOAD/LEA` instructions.
2019-07-19 13:11:01 +00:00
MOVQ c00c10+0(FP), AX
Use unaligned move in AVX2 implementation Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-29 06:48:28 +00:00
VMOVDQU (AX), Y0
Inline asm function in loop for AVX2 implementation Right now AVX2 implementation looses to C binding in speed. This is probably, because of 2 things: 1. Go does not inline `mulBitRightx2` in loop iteration. 2. `minmax` is loaded every time from memory. In this PR: 1. Unroll `mulBitRightx2` manually and use `mulByteRightx2` instead. 2. Generate `minmax` in place without `LOAD/LEA` instructions.
2019-07-19 13:11:01 +00:00
MOVQ c01c11+8(FP), BX
Use unaligned move in AVX2 implementation Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-29 06:48:28 +00:00
VMOVDQU (BX), Y8
Optimize AVX2 implementation We use 6 instructions only to calculate mask based on single bit value. Use only 3 now and calculate multiple masks in parallel. Also `VPSUB*` is faster than VPBROADCAST*, see https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html . ``` name old time/op new time/op delta Sum/AVX2Inline_digest-8 1.83ms ± 0% 1.62ms ± 1% -11.23% (p=0.000 n=46+42) name old speed new speed delta Sum/AVX2Inline_digest-8 54.7MB/s ± 0% 61.6MB/s ± 1% +12.65% (p=0.000 n=46+42) ``` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-28 17:24:59 +00:00
VPXOR Y13, Y13, Y13 // Y13 = 0x0000...
Replace two shifts with a single AND We need to isolate HSB in every quad-word, this can be done with a simple mask. Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-29 11:49:20 +00:00
VPCMPEQB Y14, Y14, Y14 // Y14 = 0xFFFF...
VPSUBQ Y14, Y13, Y10
VPSUBW Y14, Y13, Y12 // Y12 = 0x00010001... (packed words of 1)
VPSLLQ $63, Y10, Y14 // Y14 = 0x10000000... (packed quad-words with HSB set)
Optimize AVX2 implementation We use 6 instructions only to calculate mask based on single bit value. Use only 3 now and calculate multiple masks in parallel. Also `VPSUB*` is faster than VPBROADCAST*, see https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html . ``` name old time/op new time/op delta Sum/AVX2Inline_digest-8 1.83ms ± 0% 1.62ms ± 1% -11.23% (p=0.000 n=46+42) name old speed new speed delta Sum/AVX2Inline_digest-8 54.7MB/s ± 0% 61.6MB/s ± 1% +12.65% (p=0.000 n=46+42) ``` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-28 17:24:59 +00:00
Rewrite AVX2 loop in assembly Helps to get rid of MOV and generating constants for each iteration. ``` name old time/op new time/op delta Sum/AVX2Inline_digest-8 1.57ms ± 2% 1.41ms ± 0% -10.52% (p=0.000 n=9+9) name old speed new speed delta Sum/AVX2Inline_digest-8 63.6MB/s ± 1% 71.1MB/s ± 0% +11.76% (p=0.000 n=9+9) ``` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-10 15:01:15 +00:00
MOVQ n+16(FP), CX
MOVQ data+24(FP), DX
loop:
CMPQ CX, $0
JEQ finish
SUBQ $1, CX
VPBROADCASTB (DX), X10 // X10 = packed bytes of b.
Optimize AVX2 implementation We use 6 instructions only to calculate mask based on single bit value. Use only 3 now and calculate multiple masks in parallel. Also `VPSUB*` is faster than VPBROADCAST*, see https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html . ``` name old time/op new time/op delta Sum/AVX2Inline_digest-8 1.83ms ± 0% 1.62ms ± 1% -11.23% (p=0.000 n=46+42) name old speed new speed delta Sum/AVX2Inline_digest-8 54.7MB/s ± 0% 61.6MB/s ± 1% +12.65% (p=0.000 n=46+42) ``` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-28 17:24:59 +00:00
VPMOVZXBW X10, Y10 // Extend with zeroes to packed words.
Rewrite AVX2 loop in assembly Helps to get rid of MOV and generating constants for each iteration. ``` name old time/op new time/op delta Sum/AVX2Inline_digest-8 1.57ms ± 2% 1.41ms ± 0% -10.52% (p=0.000 n=9+9) name old speed new speed delta Sum/AVX2Inline_digest-8 63.6MB/s ± 1% 71.1MB/s ± 0% +11.76% (p=0.000 n=9+9) ``` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-10 15:01:15 +00:00
ADDQ $1, DX
Inline asm function in loop for AVX2 implementation Right now AVX2 implementation looses to C binding in speed. This is probably, because of 2 things: 1. Go does not inline `mulBitRightx2` in loop iteration. 2. `minmax` is loaded every time from memory. In this PR: 1. Unroll `mulBitRightx2` manually and use `mulByteRightx2` instead. 2. Generate `minmax` in place without `LOAD/LEA` instructions.
2019-07-19 13:11:01 +00:00
Interleave carry registers for successive bits 8 instructions less per byte. Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-30 08:29:59 +00:00
mulBit($7, Y0, Y8, Y5, Y6)
mulBit($6, Y5, Y6, Y0, Y8)
mulBit($5, Y0, Y8, Y5, Y6)
mulBit($4, Y5, Y6, Y0, Y8)
mulBit($3, Y0, Y8, Y5, Y6)
mulBit($2, Y5, Y6, Y0, Y8)
mulBit($1, Y0, Y8, Y5, Y6)
mulBit($0, Y5, Y6, Y0, Y8)
Inline asm function in loop for AVX2 implementation Right now AVX2 implementation looses to C binding in speed. This is probably, because of 2 things: 1. Go does not inline `mulBitRightx2` in loop iteration. 2. `minmax` is loaded every time from memory. In this PR: 1. Unroll `mulBitRightx2` manually and use `mulByteRightx2` instead. 2. Generate `minmax` in place without `LOAD/LEA` instructions.
2019-07-19 13:11:01 +00:00
Rewrite AVX2 loop in assembly Helps to get rid of MOV and generating constants for each iteration. ``` name old time/op new time/op delta Sum/AVX2Inline_digest-8 1.57ms ± 2% 1.41ms ± 0% -10.52% (p=0.000 n=9+9) name old speed new speed delta Sum/AVX2Inline_digest-8 63.6MB/s ± 1% 71.1MB/s ± 0% +11.76% (p=0.000 n=9+9) ``` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-10 15:01:15 +00:00
JMP loop
finish:
Use unaligned move in AVX2 implementation Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-29 06:48:28 +00:00
VMOVDQU Y8, (BX)
VMOVDQU Y0, (AX)
Use macros in asm code
2019-10-09 15:11:53 +00:00
Inline asm function in loop for AVX2 implementation Right now AVX2 implementation looses to C binding in speed. This is probably, because of 2 things: 1. Go does not inline `mulBitRightx2` in loop iteration. 2. `minmax` is loaded every time from memory. In this PR: 1. Unroll `mulBitRightx2` manually and use `mulByteRightx2` instead. 2. Generate `minmax` in place without `LOAD/LEA` instructions.
2019-07-19 13:11:01 +00:00
RET
Move mulBitRightx2 to avx2 assembly file
2019-10-16 11:39:35 +00:00
// func mulBitRightx2(c00c10, c01c11 *[4]uint64, e *[2]uint64)
TEXT ·mulBitRightx2(SB),NOSPLIT,$0
MOVQ c00c10+0(FP), AX
Use unaligned move in AVX2 implementation Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-29 06:48:28 +00:00
VMOVDQU (AX), Y0
Move mulBitRightx2 to avx2 assembly file
2019-10-16 11:39:35 +00:00
MOVQ c01c11+8(FP), BX
Use unaligned move in AVX2 implementation Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-29 06:48:28 +00:00
VMOVDQU (BX), Y8
Move mulBitRightx2 to avx2 assembly file
2019-10-16 11:39:35 +00:00
VPSLLQ $1, Y0, Y1
VPALIGNR $8, Y1, Y0, Y2
VPSRLQ $63, Y2, Y2
VPXOR Y1, Y2, Y2
VPSRLQ $63, Y1, Y3
VPSLLQ $63, Y3, Y3
VPUNPCKHQDQ Y3, Y3, Y3
VPXOR Y2, Y3, Y3
MOVQ e+16(FP), CX
VBROADCASTI128 (CX), Y2
VPXOR Y3, Y8, Y3
VPAND Y3, Y2, Y4
VPXOR Y4, Y0, Y8
Use unaligned move in AVX2 implementation Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-29 06:48:28 +00:00
VMOVDQU Y8, (BX)
VMOVDQU Y3, (AX)
Move mulBitRightx2 to avx2 assembly file
2019-10-16 11:39:35 +00:00
RET
Copy permalink