Speed up CRC32 calculation

mirror of https://github.com/loot/loot-condition-interpreter.git synced 2026-07-27 14:16:09 -07:00

It turns out that passing in one byte at a time is really inefficient
(which really should have been obvious...). For small files the
difference is insignificant, but for Skyrim SE's Update.esm (18 MB)
feeding in the buffered file content slice made the calculation twice
as fast, and this seems to have non-linear effect, as the benchmark
against Dragonborn.esm (63 MB) would have taken ~ 1137 seconds to run
when passing one byte at a time (the benchmark using a buffer took ~ 5
seconds).

On top of that, switching to the crc32fast crate has an insignificant
effect on benchmarking with Blank.esm, but Update.esm's benchmark
was twice as fast again, and when benchmarking with Dragonborn.esm,
using crc32fast was ~ 170x faster than using the crc crate!

This commit is contained in:

Oliver Hamlet

2020-06-13 23:17:32 +01:00

parent 3b1e54a4ed

commit d0bde327c2

2 changed files with 14 additions and 10 deletions

									
										Cargo.toml
									
		+1
		-1
	
												View File
												
				@@ -6,7 +6,7 @@ license = "MIT"

				edition = "2018"

				[dependencies]

				crc = "1.0.0"

				crc32fast = "1.2.0"

				esplugin = "3.0.0"

				nom = "5.0.0"

				pelite = "0.8.0"