Convert every submodule into subrepo (#170)

* remove ZAPD submodule

* git subrepo clone https://github.com/zeldaret/ZAPD.git tools/ZAPD

subrepo:
  subdir:   "tools/ZAPD"
  merged:   "ca229f19"
upstream:
  origin:   "https://github.com/zeldaret/ZAPD.git"
  branch:   "master"
  commit:   "ca229f19"
git-subrepo:
  version:  "0.4.3"
  origin:   "???"
  commit:   "???"

* git subrepo clone https://github.com/simonlindholm/decomp-permuter.git tools/decomp-permuter

subrepo:
  subdir:   "tools/decomp-permuter"
  merged:   "1e4b85a7"
upstream:
  origin:   "https://github.com/simonlindholm/decomp-permuter.git"
  branch:   "main"
  commit:   "1e4b85a7"
git-subrepo:
  version:  "0.4.3"
  origin:   "???"
  commit:   "???"

* Remove asm-differ

* git subrepo clone https://github.com/simonlindholm/asm-differ.git tools/asm-differ

subrepo:
  subdir:   "tools/asm-differ"
  merged:   "eaf72269"
upstream:
  origin:   "https://github.com/simonlindholm/asm-differ.git"
  branch:   "master"
  commit:   "eaf72269"
git-subrepo:
  version:  "0.4.3"
  origin:   "???"
  commit:   "???"

* remove asm-processor

* git subrepo clone https://github.com/simonlindholm/asm-processor.git tools/asm-processor

subrepo:
  subdir:   "tools/asm-processor"
  merged:   "85288fcd"
upstream:
  origin:   "https://github.com/simonlindholm/asm-processor.git"
  branch:   "master"
  commit:   "85288fcd"
git-subrepo:
  version:  "0.4.3"
  origin:   "???"
  commit:   "???"

* remove .gitmodules file

* Update REAMDE

* Update warnings
This commit is contained in:
Anghelo Carvajal
2021-06-07 18:31:56 -04:00
committed by GitHub
parent 2bf68a5263
commit b6904aa2cc
418 changed files with 113047 additions and 62 deletions
+1
View File
@@ -0,0 +1 @@
*.o
+12
View File
@@ -0,0 +1,12 @@
; DO NOT EDIT (unless you know what you are doing)
;
; This subdirectory is a git "subrepo", and this file is maintained by the
; git-subrepo command. See https://github.com/git-commands/git-subrepo#readme
;
[subrepo]
remote = https://github.com/simonlindholm/asm-processor.git
branch = master
commit = 85288fcdfefd5d2644d9bac1230c8b4aa7c92024
parent = 492b92579d6bc30f635cbcc3852de1b17ed1bc43
method = merge
cmdver = 0.4.3
+24
View File
@@ -0,0 +1,24 @@
This is free and unencumbered software released into the public domain.
Anyone is free to copy, modify, publish, use, compile, sell, or
distribute this software, either in source code form or as a compiled
binary, for any purpose, commercial or non-commercial, and by any
means.
In jurisdictions that recognize copyright laws, the author or authors
of this software dedicate any and all copyright interest in the
software to the public domain. We make this dedication for the benefit
of the public at large and to the detriment of our heirs and
successors. We intend this dedication to be an overt act of
relinquishment in perpetuity of all present and future rights to this
software under copyright law.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
OTHER DEALINGS IN THE SOFTWARE.
For more information, please refer to <http://unlicense.org>
+93
View File
@@ -0,0 +1,93 @@
# asm-processor
Pre-process .c files and post-process .o files to enable embedding MIPS assembly into IDO-compiled C.
## Usage
Let's say you have a file compiled with `-g` on the IDO compiler, that looks like this:
```c
float func4(void) {
"func4";
return 0.2f;
}
```
This script enables replacing it by:
```asm
GLOBAL_ASM(
.rdata
.word 0x66756e63 # func
.word 0x34000000 # 4\0\0\0
.late_rodata
glabel rv
.word 0x3e4ccccd # 0.2f
.text
glabel func4
lui $at, %hi(rv)
jr $ra
lwc1 $f0, %lo(rv)($at)
jr $ra
nop
jr $ra
nop
)
```
To compile the file, run `./compile.sh file.c`, or invoke the `asm_processor.py` script in a similar manner. (`compile.sh` is mostly just intended to describe example usage.)
Reading assembly from file is also supported, e.g. `GLOBAL_ASM("file.s")`.
### What is supported?
`.text`, `.data`, `.bss` and `.rodata` sections, `.word`/`.incbin`, `.ascii`/`.asciz`, and `-g`, `-g3`, `-O1`, `-O2` and `-framepointer` flags to the IDO compiler.
### What is not supported?
* complicated assembly (.ifdef, macro declarations/calls other than `glabel`, pseudo-instructions that expand to several real instructions)
* non-IDO compilers
* `-mips1` (`-mips3` may also not work fully)
C `#ifdef`s only work outside of `GLOBAL_ASM` calls, but is otherwise able to replace `.ifdef`.
### What's up with "late rodata"?
The IDO compiler emits rodata in two passes: first array/string contents, then large literals/switch jump tables.
Data declared within `.rdata`/`.section .rodata` will end up in the first half, and `.late_rodata`/`.section .late_rodata` in the second half.
### How does it work?
It's a bit of a hack!
The basic idea is replace `GLOBAL_ASM` blocks with dummy C functions/global vars of the same sections sizes as the assembly.
Then the C file gets compiled, and the dummy contents overwritten with the injected assembly.
To accomplish this, asm-processor has logic for guessing the size of assembly contents
(which assumes the assembly isn't too complicated, e.g. no macros),
and for emitting C code of exact sizes for a bunch of different IDO compiler flags.
The assembler code is padded with nops to line it up with its correct position in the C;
this allows C and asm ELF files to be merged easily without having to fix up e.g. symbol addresses.
The most difficulty part is `late_rodata`, which is hard to create programmatically.
asm-processor does that by emitting code that uses dummy float literals/double literals/jump tables,
assembles the late_rodata at another location of the .rodata section, then overwrites the dummy rodata.
This does require some movement of symbols and relocations, and quite a bit of care in what code to
emit and how to preserve .double alignment.
It's worth noting some alternative ways in which asm-processor would have been implemented:
- One idea to get rid of the C/asm size estimations is to emit arbitrary code, and then move code,
symbols and relocations to the correct place after the sizes are known.
Given the machinery for `late_rodata` this wouldn't have been too difficult, and it would have the upside of improved portability.
There is a big downside, however: using dummy code of incorrect size throws off alignment and can introduce unintended padding.
Fixing this would require running multiple passes of asm-processor, with one compile per `ASM_GLOBAL`.
- Another idea is to run the compiler with -S to emit assembly, modify the emitted assembly, then run the assembler
(which in IDO's case may perform additional instruction reordering etc.).
This option has not been investigated in much detail, and would perhaps be superior to the current implementation.
It does have a few unknowns to it, e.g. instruction encoding differences between GNU `as` and IDO's assembler,
how to avoid reordering the injected assembly, and how .rodata/.late_rodata are implemented.
### Testing
There are a few tests to ensure you don't break anything when hacking on asm-processor: `./run-tests.sh` should exit without output if they pass, or else output a diff from previous to new version.
+4
View File
@@ -0,0 +1,4 @@
#!/usr/bin/env bash
for A in "$@"; do
./compile.sh "$A" && mips-linux-gnu-objdump -s "${A%.c}.o" > "${A%.c}.objdump"
done
File diff suppressed because it is too large Load Diff
+23
View File
@@ -0,0 +1,23 @@
#!/bin/bash
set -e
set -o pipefail
INPUT="$1"
OUTPUT="${INPUT%.c}.o"
CC="$QEMU_IRIX -silent -L $IRIX_ROOT $IRIX_ROOT/usr/bin/cc"
CFLAGS="-Wab,-r4300_mul -non_shared -G 0 -Xcpluscomm -fullwarn -wlint -woff 819,820,852,821 -signed -DVERSION_JP=1" # -I include
AS="mips-linux-gnu-as"
ASFLAGS="-march=vr4300 -mabi=32 --defsym VERSION_JP=1" # -I include
set +e
OPTFLAGS=$(grep '^// COMPILE-FLAGS: ' $INPUT | sed 's#^// COMPILE-FLAGS: ##')
ISET=$(grep '^// COMPILE-ISET: ' $INPUT | sed 's#^// COMPILE-ISET: ##')
set -e
if [[ -z "$OPTFLAGS" ]]; then
OPTFLAGS="-g"
fi
if [[ -z "$ISET" ]]; then
CFLAGS="$CFLAGS -mips2"
fi
python3 asm_processor.py $OPTFLAGS $ISET "$INPUT" | $CC -c $CFLAGS include-stdin.c -o "$OUTPUT" $OPTFLAGS
python3 asm_processor.py $OPTFLAGS $ISET "$INPUT" --post-process "$OUTPUT" --assembler "$AS $ASFLAGS" --asm-prelude prelude.s
+2
View File
@@ -0,0 +1,2 @@
// (this is used for piping input to the IRIX compiler without needing to make a temporary .c file)
#include "/dev/stdin"
+8
View File
@@ -0,0 +1,8 @@
.set noat
.set noreorder
.set gp=64
.macro glabel label
.global \label
\label:
.endm
+8
View File
@@ -0,0 +1,8 @@
#!/usr/bin/env bash
for A in tests/*.c; do
OBJDUMPFLAGS=$(grep '^// OBJDUMP-FLAGS: ' "$A" | sed 's#^// OBJDUMP-FLAGS: ##')
if [[ -z "$OBJDUMPFLAGS" ]]; then
OBJDUMPFLAGS="-s"
fi
./compile.sh "$A" && mips-linux-gnu-objdump $OBJDUMPFLAGS "${A%.c}.o" | diff - "${A%.c}.objdump" || echo FAIL "$A"
done
+19
View File
@@ -0,0 +1,19 @@
void foo(void) { "abcdef"; }
GLOBAL_ASM(
.rdata
.ascii "AB"
.ascii "CD", "EF"
.ascii "GH\n\n\n\0\11\222\3333\44444\x1234567\n\nIJK"
)
void bar(void) { "hello"; }
GLOBAL_ASM(
.rdata
.asciiz "12"
.asciiz "34", "56"
.asciiz "78\n\n\n\0\11\222\3333\44444\x1234567\n\n9A"
)
void baz(void) { "ghijkl"; }
+19
View File
@@ -0,0 +1,19 @@
tests/ascii.o: file format elf32-tradbigmips
Contents of section .text:
0000 03e00008 00000000 03e00008 00000000 ................
0010 03e00008 00000000 03e00008 00000000 ................
0020 03e00008 00000000 03e00008 00000000 ................
Contents of section .rodata:
0000 61626364 65660000 41424344 45464748 abcdef..ABCDEFGH
0010 0a0a0a00 0992db33 24343467 0a0a494a .......3$44g..IJ
0020 4b000000 68656c6c 6f000000 31320033 K...hello...12.3
0030 34003536 0037380a 0a0a0009 92db3324 4.56.78.......3$
0040 3434670a 0a394100 6768696a 6b6c0000 44g..9A.ghijkl..
Contents of section .options:
0000 01200000 00000000 80000000 00000000 . ..............
0010 00000000 00000000 00000000 00007ff0 ................
Contents of section .reginfo:
0000 80000000 00000000 00000000 00000000 ................
0010 00000000 00007ff0 ........
+6
View File
@@ -0,0 +1,6 @@
const char before[] = "^";
GLOBAL_ASM(
.rdata
.asciz "aaaa /* bbbb */ # cccc" /**//**//**//**/ /*/ "xxxx" /*/ /* dddd " eeee */ "# ffff" # gggg "hhhh" /* iiii */
)
const char after[] = "$";
@@ -0,0 +1,13 @@
tests/comments.o: file format elf32-tradbigmips
Contents of section .rodata:
0000 5e000000 61616161 202f2a20 62626262 ^...aaaa /* bbbb
0010 202a2f20 23206363 63630023 20666666 */ # cccc.# fff
0020 66000000 24000000 00000000 00000000 f...$...........
Contents of section .options:
0000 01200000 00000000 00000000 00000000 . ..............
0010 00000000 00000000 00000000 00007ff0 ................
Contents of section .reginfo:
0000 00000000 00000000 00000000 00000000 ................
0010 00000000 00007ff0 ........
@@ -0,0 +1,7 @@
GLOBAL_ASM(
.rdata
.word 0x12345678
glabel blah
.word blah2
/*a*/ blah2: /*b*/ .word blah /*c*/
)
@@ -0,0 +1,11 @@
tests/label-sameline.o: file format elf32-tradbigmips
Contents of section .rodata:
0000 12345678 00000008 00000000 00000000 .4Vx............
Contents of section .options:
0000 01200000 00000000 00000000 00000000 . ..............
0010 00000000 00000000 00000000 00007ff0 ................
Contents of section .reginfo:
0000 00000000 00000000 00000000 00000000 ................
0010 00000000 00007ff0 ........
+164
View File
@@ -0,0 +1,164 @@
GLOBAL_ASM(
glabel test
addiu $sp, $sp, -24
sw $zero, 4($sp)
lw $t6, 4($sp)
addu $t7, $a0, $t6
sb $zero, ($t7)
lw $t8, 4($sp)
addiu $t9, $t8, 1
slt $at, $t9, $a1
sw $t9, 4($sp)
nop
jr $ra
addiu $sp, $sp, 24
addiu $sp, $sp, -24
sw $zero, 4($sp)
lw $t6, 4($sp)
addu $t7, $a0, $t6
sb $zero, ($t7)
lw $t8, 4($sp)
addiu $t9, $t8, 1
slt $at, $t9, $a1
sw $t9, 4($sp)
nop
jr $ra
addiu $sp, $sp, 24
addiu $sp, $sp, -24
sw $zero, 4($sp)
lw $t6, 4($sp)
addu $t7, $a0, $t6
sb $zero, ($t7)
lw $t8, 4($sp)
addiu $t9, $t8, 1
slt $at, $t9, $a1
sw $t9, 4($sp)
nop
jr $ra
addiu $sp, $sp, 24
addiu $sp, $sp, -24
sw $zero, 4($sp)
lw $t6, 4($sp)
addu $t7, $a0, $t6
sb $zero, ($t7)
lw $t8, 4($sp)
addiu $t9, $t8, 1
slt $at, $t9, $a1
sw $t9, 4($sp)
nop
jr $ra
addiu $sp, $sp, 24
addiu $sp, $sp, -24
sw $zero, 4($sp)
lw $t6, 4($sp)
addu $t7, $a0, $t6
sb $zero, ($t7)
lw $t8, 4($sp)
addiu $t9, $t8, 1
slt $at, $t9, $a1
sw $t9, 4($sp)
nop
jr $ra
addiu $sp, $sp, 24
addiu $sp, $sp, -24
sw $zero, 4($sp)
lw $t6, 4($sp)
addu $t7, $a0, $t6
sb $zero, ($t7)
lw $t8, 4($sp)
addiu $t9, $t8, 1
slt $at, $t9, $a1
sw $t9, 4($sp)
nop
jr $ra
addiu $sp, $sp, 24
addiu $sp, $sp, -24
sw $zero, 4($sp)
lw $t6, 4($sp)
addu $t7, $a0, $t6
sb $zero, ($t7)
lw $t8, 4($sp)
addiu $t9, $t8, 1
slt $at, $t9, $a1
sw $t9, 4($sp)
nop
jr $ra
addiu $sp, $sp, 24
addiu $sp, $sp, -24
sw $zero, 4($sp)
lw $t6, 4($sp)
addu $t7, $a0, $t6
sb $zero, ($t7)
lw $t8, 4($sp)
addiu $t9, $t8, 1
slt $at, $t9, $a1
sw $t9, 4($sp)
nop
jr $ra
addiu $sp, $sp, 24
addiu $sp, $sp, -24
sw $zero, 4($sp)
lw $t6, 4($sp)
addu $t7, $a0, $t6
sb $zero, ($t7)
lw $t8, 4($sp)
addiu $t9, $t8, 1
slt $at, $t9, $a1
sw $t9, 4($sp)
nop
jr $ra
addiu $sp, $sp, 24
addiu $sp, $sp, -24
sw $zero, 4($sp)
lw $t6, 4($sp)
addu $t7, $a0, $t6
sb $zero, ($t7)
lw $t8, 4($sp)
addiu $t9, $t8, 1
slt $at, $t9, $a1
sw $t9, 4($sp)
nop
jr $ra
addiu $sp, $sp, 24
addiu $sp, $sp, -24
sw $zero, 4($sp)
lw $t6, 4($sp)
addu $t7, $a0, $t6
sb $zero, ($t7)
lw $t8, 4($sp)
addiu $t9, $t8, 1
slt $at, $t9, $a1
sw $t9, 4($sp)
nop
jr $ra
addiu $sp, $sp, 24
addiu $sp, $sp, -24
sw $zero, 4($sp)
lw $t6, 4($sp)
addu $t7, $a0, $t6
sb $zero, ($t7)
lw $t8, 4($sp)
addiu $t9, $t8, 1
slt $at, $t9, $a1
sw $t9, 4($sp)
nop
jr $ra
addiu $sp, $sp, 24
addiu $sp, $sp, -24
sw $zero, 4($sp)
lw $t6, 4($sp)
addu $t7, $a0, $t6
sb $zero, ($t7)
lw $t8, 4($sp)
addiu $t9, $t8, 1
slt $at, $t9, $a1
sw $t9, 4($sp)
nop
jr $ra
addiu $sp, $sp, 24
)
void foo(void) {}
+50
View File
@@ -0,0 +1,50 @@
tests/large.o: file format elf32-tradbigmips
Contents of section .text:
0000 27bdffe8 afa00004 8fae0004 008e7821 '.............x!
0010 a1e00000 8fb80004 27190001 0325082a ........'....%.*
0020 afb90004 00000000 03e00008 27bd0018 ............'...
0030 27bdffe8 afa00004 8fae0004 008e7821 '.............x!
0040 a1e00000 8fb80004 27190001 0325082a ........'....%.*
0050 afb90004 00000000 03e00008 27bd0018 ............'...
0060 27bdffe8 afa00004 8fae0004 008e7821 '.............x!
0070 a1e00000 8fb80004 27190001 0325082a ........'....%.*
0080 afb90004 00000000 03e00008 27bd0018 ............'...
0090 27bdffe8 afa00004 8fae0004 008e7821 '.............x!
00a0 a1e00000 8fb80004 27190001 0325082a ........'....%.*
00b0 afb90004 00000000 03e00008 27bd0018 ............'...
00c0 27bdffe8 afa00004 8fae0004 008e7821 '.............x!
00d0 a1e00000 8fb80004 27190001 0325082a ........'....%.*
00e0 afb90004 00000000 03e00008 27bd0018 ............'...
00f0 27bdffe8 afa00004 8fae0004 008e7821 '.............x!
0100 a1e00000 8fb80004 27190001 0325082a ........'....%.*
0110 afb90004 00000000 03e00008 27bd0018 ............'...
0120 27bdffe8 afa00004 8fae0004 008e7821 '.............x!
0130 a1e00000 8fb80004 27190001 0325082a ........'....%.*
0140 afb90004 00000000 03e00008 27bd0018 ............'...
0150 27bdffe8 afa00004 8fae0004 008e7821 '.............x!
0160 a1e00000 8fb80004 27190001 0325082a ........'....%.*
0170 afb90004 00000000 03e00008 27bd0018 ............'...
0180 27bdffe8 afa00004 8fae0004 008e7821 '.............x!
0190 a1e00000 8fb80004 27190001 0325082a ........'....%.*
01a0 afb90004 00000000 03e00008 27bd0018 ............'...
01b0 27bdffe8 afa00004 8fae0004 008e7821 '.............x!
01c0 a1e00000 8fb80004 27190001 0325082a ........'....%.*
01d0 afb90004 00000000 03e00008 27bd0018 ............'...
01e0 27bdffe8 afa00004 8fae0004 008e7821 '.............x!
01f0 a1e00000 8fb80004 27190001 0325082a ........'....%.*
0200 afb90004 00000000 03e00008 27bd0018 ............'...
0210 27bdffe8 afa00004 8fae0004 008e7821 '.............x!
0220 a1e00000 8fb80004 27190001 0325082a ........'....%.*
0230 afb90004 00000000 03e00008 27bd0018 ............'...
0240 27bdffe8 afa00004 8fae0004 008e7821 '.............x!
0250 a1e00000 8fb80004 27190001 0325082a ........'....%.*
0260 afb90004 00000000 03e00008 27bd0018 ............'...
0270 03e00008 00000000 03e00008 00000000 ................
Contents of section .options:
0000 01200000 00000000 80000000 00000000 . ..............
0010 00000000 00000000 00000000 00007ff0 ................
Contents of section .reginfo:
0000 a300c032 00000000 00000000 00000000 ...2............
0010 00000000 00007ff0 ........
@@ -0,0 +1,80 @@
GLOBAL_ASM(
.late_rodata
.float 4.1
.float 4.2
.float 4.3
.float 4.4
.text
glabel a
nop
nop
nop
nop
nop
nop
nop
nop
nop
nop
nop
nop
nop
nop
nop
nop
)
float foo(void) { "foo"; return 1.1f; }
GLOBAL_ASM(
.late_rodata
.late_rodata_alignment 4
.float 5.1
.float 5.2
.float 5.3
.float 5.4
.text
glabel b
nop
nop
nop
nop
nop
nop
nop
nop
nop
nop
nop
nop
nop
)
float bar(void) { "bar"; return 1.2f; }
GLOBAL_ASM(
.late_rodata
.late_rodata_alignment 8
.float 6.1
.float 6.2
.float 6.3
.float 6.4
.float 6.5
.text
glabel c
nop
nop
nop
nop
nop
nop
nop
nop
nop
nop
nop
nop
nop
nop
)

Some files were not shown because too many files have changed in this diff Show More