bisync: optimize --resync performance -- partially addresses #5681
Before this change, --resync was handled in three steps, and needed to do a lot of unnecessary work to implement its own --ignore-existing logic, which also caused problems with unicode normalization, in addition to being pretty slow. After this change, it is refactored to produce the same result much more efficiently, by reducing the three steps to two and letting ci.IgnoreExisting do the work instead of reinventing the wheel. The behavior and sync order remain unchanged for now -- just faster (but see the ongoing lively discussions about potential future changes in #5681!)
This commit is contained in:
nielash
parent
f7f4651828
commit
9c96c13a35
65 changed files with 261 additions and 373 deletions
|
|
@ -1,5 +1,5 @@
|
|||
# bisync listing v1 from test
|
||||
- 19 - - 2001-01-05T00:00:00.000000000+0000 "file1.txt"
|
||||
- 19 - - 2001-01-05T00:00:00.000000000+0000 "folder/HeLlO,wOrLd!.txt"
|
||||
- 19 - - 2001-01-05T00:00:00.000000000+0000 "folder/éééö.txt"
|
||||
- 19 - - 2001-01-05T00:00:00.000000000+0000 "測試_Русский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éééö/測試_Русский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éééö.txt"
|
||||
- 19 md5:7fe98ed88552b828777d8630900346b8 - 2001-01-05T00:00:00.000000000+0000 "file1.txt"
|
||||
- 19 md5:7fe98ed88552b828777d8630900346b8 - 2001-01-05T00:00:00.000000000+0000 "folder/HeLlO,wOrLd!.txt"
|
||||
- 19 md5:7fe98ed88552b828777d8630900346b8 - 2001-01-05T00:00:00.000000000+0000 "folder/éééö.txt"
|
||||
- 19 md5:7fe98ed88552b828777d8630900346b8 - 2001-01-05T00:00:00.000000000+0000 "測試_Русский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éééö/測試_Русский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éééö.txt"
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
# bisync listing v1 from test
|
||||
- 19 - - 2001-01-03T00:00:00.000000000+0000 "file1.txt"
|
||||
- 19 - - 2001-01-03T00:00:00.000000000+0000 "folder/HeLlO,wOrLd!.txt"
|
||||
- 19 - - 2001-01-03T00:00:00.000000000+0000 "folder/éééö.txt"
|
||||
- 19 - - 2001-01-02T00:00:00.000000000+0000 "測試_Русский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éééö/測試_Русский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éééö.txt"
|
||||
- 19 md5:7fe98ed88552b828777d8630900346b8 - 2001-01-03T00:00:00.000000000+0000 "file1.txt"
|
||||
- 19 md5:7fe98ed88552b828777d8630900346b8 - 2001-01-03T00:00:00.000000000+0000 "folder/HeLlO,wOrLd!.txt"
|
||||
- 19 md5:7fe98ed88552b828777d8630900346b8 - 2001-01-03T00:00:00.000000000+0000 "folder/éééö.txt"
|
||||
- 19 md5:7fe98ed88552b828777d8630900346b8 - 2001-01-02T00:00:00.000000000+0000 "測試_Русский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éééö/測試_Русский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éééö.txt"
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
# bisync listing v1 from test
|
||||
- 19 - - 2001-01-05T00:00:00.000000000+0000 "file1.txt"
|
||||
- 19 - - 2001-01-05T00:00:00.000000000+0000 "folder/hello,WORLD!.txt"
|
||||
- 19 - - 2001-01-05T00:00:00.000000000+0000 "folder/éééö.txt"
|
||||
- 19 - - 2001-01-05T00:00:00.000000000+0000 "測試_Русский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éééö/測試_Русский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éééö.txt"
|
||||
- 19 md5:7fe98ed88552b828777d8630900346b8 - 2001-01-05T00:00:00.000000000+0000 "file1.txt"
|
||||
- 19 md5:7fe98ed88552b828777d8630900346b8 - 2001-01-05T00:00:00.000000000+0000 "folder/hello,WORLD!.txt"
|
||||
- 19 md5:7fe98ed88552b828777d8630900346b8 - 2001-01-05T00:00:00.000000000+0000 "folder/éééö.txt"
|
||||
- 19 md5:7fe98ed88552b828777d8630900346b8 - 2001-01-05T00:00:00.000000000+0000 "測試_Русский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éééö/測試_Русский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éééö.txt"
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
# bisync listing v1 from test
|
||||
- 19 - - 2001-01-03T00:00:00.000000000+0000 "file1.txt"
|
||||
- 19 - - 2001-01-03T00:00:00.000000000+0000 "folder/hello,WORLD!.txt"
|
||||
- 19 - - 2001-01-03T00:00:00.000000000+0000 "folder/éééö.txt"
|
||||
- 19 - - 2001-01-02T00:00:00.000000000+0000 "測試_Русский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éééö/測試_Русский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éééö.txt"
|
||||
- 19 md5:7fe98ed88552b828777d8630900346b8 - 2001-01-03T00:00:00.000000000+0000 "file1.txt"
|
||||
- 19 md5:7fe98ed88552b828777d8630900346b8 - 2001-01-03T00:00:00.000000000+0000 "folder/hello,WORLD!.txt"
|
||||
- 19 md5:7fe98ed88552b828777d8630900346b8 - 2001-01-03T00:00:00.000000000+0000 "folder/éééö.txt"
|
||||
- 19 md5:7fe98ed88552b828777d8630900346b8 - 2001-01-02T00:00:00.000000000+0000 "測試_Русский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éééö/測試_Русский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éééö.txt"
|
||||
|
|
|
|||
|
|
@ -1,2 +0,0 @@
|
|||
"folder/hello,WORLD!.txt"
|
||||
"folder/éééö.txt"
|
||||
|
|
@ -6,7 +6,8 @@
|
|||
[36m(04) :[0m [34mbisync resync[0m
|
||||
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
|
||||
INFO : Copying unique Path2 files to Path1
|
||||
INFO : Resynching Path1 to Path2
|
||||
INFO : - [34mPath2[0m [35mResync is copying UNIQUE files to[0m - [36mPath1[0m
|
||||
INFO : - [34mPath1[0m [35mResync is copying UNIQUE OR DIFFERING files to[0m - [36mPath2[0m
|
||||
INFO : Resync updating listings
|
||||
INFO : [32mBisync successful[0m
|
||||
|
||||
|
|
@ -56,9 +57,8 @@ INFO : [32mBisync successful[0m
|
|||
[36m(16) :[0m [34mbisync resync[0m
|
||||
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
|
||||
INFO : Copying unique Path2 files to Path1
|
||||
INFO : - [34mPath2[0m [35mResync will copy to Path1[0m - [36mfile1.txt[0m
|
||||
INFO : - [34mPath2[0m [35mResync is doing queued copies to[0m - [36mPath1[0m
|
||||
INFO : Resynching Path1 to Path2
|
||||
INFO : - [34mPath2[0m [35mResync is copying UNIQUE files to[0m - [36mPath1[0m
|
||||
INFO : - [34mPath1[0m [35mResync is copying UNIQUE OR DIFFERING files to[0m - [36mPath2[0m
|
||||
INFO : Resync updating listings
|
||||
INFO : [32mBisync successful[0m
|
||||
|
||||
|
|
@ -103,10 +103,8 @@ INFO : [32mBisync successful[0m
|
|||
[36m(26) :[0m [34mbisync resync norm[0m
|
||||
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
|
||||
INFO : Copying unique Path2 files to Path1
|
||||
INFO : - [34mPath2[0m [35mResync will copy to Path1[0m - [36mfolder/éééö.txt[0m
|
||||
INFO : - [34mPath2[0m [35mResync will copy to Path1[0m - [36mfolder/hello,WORLD!.txt[0m
|
||||
INFO : - [34mPath2[0m [35mResync is doing queued copies to[0m - [36mPath1[0m
|
||||
INFO : Resynching Path1 to Path2
|
||||
INFO : - [34mPath2[0m [35mResync is copying UNIQUE files to[0m - [36mPath1[0m
|
||||
INFO : - [34mPath1[0m [35mResync is copying UNIQUE OR DIFFERING files to[0m - [36mPath2[0m
|
||||
INFO : Resync updating listings
|
||||
INFO : [32mBisync successful[0m
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue