21 Commits

Author SHA1 Message Date
Ernesto A. Fernández 09a7616c2f Support case sensitivity
Add support for case sensitivity to unicode.c, and take advantage of
this to simplify the testing procedure.
2018-11-13 00:39:35 -03:00
Ernesto A. Fernández 4755d6f68a Rework unicode functions so they cannot fail
The ->d_compare() dentry operation is not allowed to fail. Before
the APFS module can implement it, apfs_init_unicursor() and
apfs_normalize_next() must be reworked accordingly.
2018-07-26 18:38:04 -03:00
Ernesto A. Fernández 82973b2e0f Fix documentation for apfs_try_decompose_hangul()
This function will not actually normalize all Hangul, just precomposed
characters.
2018-07-19 13:14:54 -03:00
Ernesto A. Fernández c560883dce Fix compilation warnings
Modify the makefile so that requesting warnings from the compiler is
easier; fix those that matter.
2018-06-20 21:20:30 -03:00
Ernesto A. Fernández 0f387d50a7 Further simplify ascii normalization
Convert ascii characters to UTF-32 by simple casting, without calling
utf8_to_utf32().
2018-06-10 16:49:59 -03:00
Ernesto A. Fernández 6e11fb3883 Optimize normalization for the ascii case
Handle ascii characters with the isascii() and tolower() kernel macros,
and skip the costly trie searches.
2018-06-09 22:38:05 -03:00
Ernesto A. Fernández 600e2f55d0 Give proper credit to Unicode and mkutf8data 2018-06-08 18:48:58 -03:00
Ernesto A. Fernández 26f4956d1a Add README file 2018-06-08 17:11:03 -03:00
Ernesto A. Fernández 5a0f3e712b Run tests during build
Move the head of unicode.c and unicode.h to separate files, and write
alternative versions for running tests in user space. Since there are
no official unicode tests for case folding, the mktrie script must also
produce another version of the case folding trie, with no values.

Setup the Makefile so that the tests are run with every recompilation,
printing the results to build/test.out.
2018-06-07 19:38:53 -03:00
Ernesto A. Fernández d160ba3d5e Add official data for normalization tests
It is better to add the data before the actual test code, because this
is a large file and it would make the next patch hard to review.
2018-06-07 19:37:48 -03:00
Ernesto A. Fernández 0159907aef Have git ignore .swp files 2018-06-07 19:37:48 -03:00
Ernesto A. Fernández 68bdb08d9e Fix style issues in output code
Correct a few problems with whitespace that mktrie was introducing in
unicode.c. They were found by the checkpatch.pl script of the Linux
kernel. Also replace kmalloc() with kmalloc_array() for the cursor
buffer.
2018-06-07 19:37:06 -03:00
Ernesto A. Fernández c185595fa7 Add the C unicode functions to the repository
Soon this script will be putting the C code through the normalization
tests provided by Unicode. The most practical way to do this is to keep
the C functions inside the same repository, and later provide two build
options: one for use by the kernel module, and another for testing in
user space.
2018-06-07 17:57:45 -03:00
Ernesto A. Fernández 3b5daea6d3 Parse the canonical composition codes
Extend the script to also produce a trie for the canonical composition
codes. The possible codes are very few and all fit in 8 bits, so simply
use an array of bytes to represent the trie, with no need for a separate
value array.
2018-05-30 15:29:03 -03:00
Ernesto A. Fernández f7b4008002 Define the tries as static
They will be part of the unicode.c file in the apfs module, and they
will only be needed there.
2018-05-29 18:21:04 -03:00
Ernesto A. Fernández 556e54ccca Build to a separate directory 2018-05-29 17:35:12 -03:00
Ernesto A. Fernández 548eb3050d Add unicode data files
Provide the unicode data files for version 9.0.0. Before this patch the
user needed to provide their own data for the build. Of course, this can
still be done by replacing the files in the ucd directory with different
versions.
2018-05-29 16:44:25 -03:00
Ernesto A. Fernández bd1912d01f Store size of leaf node values in the tries
Use the last bits of the value position to store its length, as the
hfsplus module does. We use three bits here, while hfsplus seems to
be using only two, which might be a bug.
2018-05-29 16:39:31 -03:00
Ernesto A. Fernández 3466890992 Iterate the NFD decomposition
Until now the generated trie was only providing the first stage in the
decomposition process. The full decomposition requires iterating several
times. In the hfsplus module the final result is given directly by the
trie, so do the same here.
2018-05-29 14:33:53 -03:00
Ernesto A. Fernández afe1607227 Parse case folding data
The script is only parsing the decomposition data; extend it so that it
can also be used for full case folding.

Also remove a couple of printf() calls for debugging that are no longer
necessary.
2018-05-28 00:32:16 -03:00
Ernesto A. Fernández 64fee37487 Begin work on unicode parsing script
Write a simple script that parses unicode decomposition data into tries
represented as C arrays. A big portion of the code was taken from a
version of the mkutf8data script by Olaf Weber [1], but adapted to work
with NFD, and to produce data structures more in line with those used
by hfsplus.

[1] https://marc.info/?l=linux-fsdevel&m=152584896931191&w=2
2018-05-27 14:17:54 -03:00