This makes the code less readable, but gets us a massive improvement
to performance. Comparing ~36M completely different files now takes
~40% of the time. Compared to GNU cmp, we now run the same comparison
in ~26% of the time.
This also improves comparing binary files. A comparison of chromium
and libxul now takes ~60% of the time. We also beat GNU cmpi by about
the same margin.
Before:
> hyperfine --warmup 1 -i --output=pipe \
'../target/release/diffutils cmp -l huge huge.3'
Benchmark 1: ../target/release/diffutils cmp -l huge huge.3
Time (mean ± σ): 2.000 s ± 0.016 s [User: 1.603 s, System: 0.392 s]
Range (min … max): 1.989 s … 2.043 s 10 runs
Warning: Ignoring non-zero exit code.
> hyperfine --warmup 1 -i --output=pipe \
'../target/release/diffutils cmp -l -b \
/usr/lib64/chromium-browser/chromium-browser \
/usr/lib64/firefox/libxul.so'
Benchmark 1: ../target/release/diffutils cmp -l -b /usr/lib64/chromium-browser/chromium-browser /usr/lib64/firefox/libxul.so
Time (mean ± σ): 24.704 s ± 0.162 s [User: 21.948 s, System: 2.700 s]
Range (min … max): 24.359 s … 24.889 s 10 runs
Warning: Ignoring non-zero exit code.
After:
> hyperfine --warmup 1 -i --output=pipe \
'../target/release/diffutils cmp -l huge huge.3'
Benchmark 1: ../target/release/diffutils cmp -l huge huge.3
Time (mean ± σ): 849.5 ms ± 6.2 ms [User: 538.3 ms, System: 306.8 ms]
Range (min … max): 839.4 ms … 857.7 ms 10 runs
Warning: Ignoring non-zero exit code.
> hyperfine --warmup 1 -i --output=pipe \
'../target/release/diffutils cmp -l -b \
/usr/lib64/chromium-browser/chromium-browser \
/usr/lib64/firefox/libxul.so'
Benchmark 1: ../target/release/diffutils cmp -l -b /usr/lib64/chromium-browser/chromium-browser /usr/lib64/firefox/libxul.so
Time (mean ± σ): 14.646 s ± 0.040 s [User: 12.328 s, System: 2.286 s]
Range (min … max): 14.585 s … 14.702 s 10 runs
Warning: Ignoring non-zero exit code.
Octal conversion and simple integer to string both show up in profiling.
This change improves comparing ~36M completely different files wth both
-l and -b by ~11-13%.
The utility should support all the arguments supported by GNU cmp and
perform slightly better.
On a "bad" scenario, ~36M files which are completely different, our
version runs in ~72% of the time of the original on my M1 Max:
> hyperfine --warmup 1 -i --output=pipe \
'cmp -l huge huge.3'
Benchmark 1: cmp -l huge huge.3
Time (mean ± σ): 3.237 s ± 0.014 s [User: 2.891 s, System: 0.341 s]
Range (min … max): 3.221 s … 3.271 s 10 runs
Warning: Ignoring non-zero exit code.
> hyperfine --warmup 1 -i --output=pipe \
'../target/release/diffutils cmp -l huge huge.3'
Benchmark 1: ../target/release/diffutils cmp -l huge huge.3
Time (mean ± σ): 2.392 s ± 0.009 s [User: 1.978 s, System: 0.406 s]
Range (min … max): 2.378 s … 2.406 s 10 runs
Warning: Ignoring non-zero exit code.
Our cmp runs in ~116% of the time when comparing libxul.so to the
chromium-browser binary with -l and -b. In a best case scenario of
comparing 2 files which are the same except for the last byte, our
tool is slightly faster.
This is in preparation for adding the other diffutils commands, cmp,
diff3, sdiff.
We use a similar strategy to uutils/coreutils, with the single binary
acting as one of the supported tools if called through a symlink with
the appropriate name. When using the multi-tool binary directly, the
utility needds to be the first parameter.
Sylvestre Ledru
Olivier Tilloy
Gustavo Noronha Silva
Daniel Hofstetter
renovate[bot]