Overhaul the build system (#234)

* wip

* fix

* add disassembler

* Disasm builds OK

* Variable addends

* More wip

* Rodata migration implemented

* Cleanup old tools

* Try fix submodule -> subrepo merge

* git subrepo pull --force --remote=https://github.com/zeldaret/ZAPD.git tools/ZAPD

subrepo:
  subdir:   "tools/ZAPD"
  merged:   "602e609"
upstream:
  origin:   "https://github.com/zeldaret/ZAPD.git"
  branch:   "master"
  commit:   "602e609"
git-subrepo:
  version:  "0.4.3"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "2f68596"

* Builds again but assets are totally broken

* git subrepo pull --force tools/asm-processor

subrepo:
  subdir:   "tools/asm-processor"
  merged:   "1ffdb08a"
upstream:
  origin:   "https://github.com/simonlindholm/asm-processor.git"
  branch:   "master"
  commit:   "1ffdb08a"
git-subrepo:
  version:  "0.4.3"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "2f68596"

* More cleanup, move functions.txt and variables.txt to tools/disasm and rm tables

* rm z64compress in preparation for subrepo

* git subrepo clone (merge) https://github.com/z64me/z64compress.git tools/z64compress

subrepo:
  subdir:   "tools/z64compress"
  merged:   "eb11085c"
upstream:
  origin:   "https://github.com/z64me/z64compress.git"
  branch:   "main"
  commit:   "eb11085c"
git-subrepo:
  version:  "0.4.3"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "2f68596"

* Fix asset extraction

* Fix diff-init make rule

* Split code bss

* Split assumed linker bug padding from assembly files

* add filelists for mm.us.rev1

* Maybe working, but I'm not sure

* add overlays to spec

* Add rodata to actos

* Everything compiles

* Make a lot of C files for code

* Add almost every file in code to spec

* whoops

* 3 code files left

* add scenes to spec

* More progress on progress.py

* Fix skelanime in spec

* audio files!

* Fix merge issues

* Fix some C files in code

* Fix remaining code files

* Use existing O1 C files in spec

* reorder boot order in spec

* update spec

* fault.c

* Convert relocs on completed actors, fixbaserom uses current rom name

* more boot files

* Add VT macros and script

* finish already existing boot files

* most of  libultra

* fix 64bits libultra files

* Use C files for libultra, wrap some functions in NON_MATCHING

* Remove duplicate of OS_CLOCK_RATE from fault.c

* C files for fbdemos

* delete dumb files

* bootstrap C files, still need to add them to the spec

* update fixbaserom

* boot OK?

* I forgot to commit the spec

* C for gamestates

* C for kaleido

* Change all includes to ""

* copy actor sizes script from oot

* I forgot to delete those files

* Basic C files for effects

* Add effects initvars names

* Remove mislabelled boot functions from header/txt

* Begin porting bootstrap_fx, some sizes

* Fix <>

* Fix enum

* Fix diff.py

* fix libultra stuff

* update regconvert

* update setup warnings

* add some missing ;

* Fix some makefile stuff and other fixes on some non_matching functions

* add executable flag in extract_baserom and fixbaserom

* fix relative path

* copy assist from oot

* fix map path

* another assist path fix

* Delete C files for handwritten files

* add code_801A51F0 to spec

* add gfxbuffers to spec

* Move rodata to top of each file when possible

* UNK_TYPEs for func_801A51F0

* Remove kaleido rodata from spec

* Update spec and undefined_syms for recent merge

* GCC warnings and fix errors in nonmatchings,

* round percentage numbers

* progress script: format changes

* progress: error on non-existing files

* fix warning in z_scene_table

* Match 2 nonmatchings in z_actor

* Warnings in lightswitch and invadepoh

* Fix warning in z_actor_dlftbls

* I though I fixed this one

* whoops

* Comment out CC_CHECK

* Removed redundant ultra64.h includes

* Update asm_processor, sorted boot_O1 into other folders, completed the fbdemo bootstrap, cleaned up undefined_syms

* Completed gamestates bootstrap

* Split kaleido_scope

* Remove section.h and segment.h, move keep object externs to a common location in variables.h

* Completed effects bootstrap

* Segmented address externs for effects, fbdemos, gamestates and kaleido

* Move actor data externs out of the if 0

* Segmented address externs for actors

* Prepare actionfunc detection

* fix script, how did it even work before

* Fix actionfunc script again, re-introduce some more intermediate prints to the disassembler

* Automated actionFunc detection in actors

* Segmented addresses from player .text

* rm old segment addrs script and fix build

* Move sizes folder to tools

* Make build.py executable

* New Jenkinsfile Prayge

* Remove numpy dependencies

* Add warnings_disasm_current.txt

* my bad

* Update spec and undefined_syms

* Add z_eff_ss_hahen to pametfrog

* git subrepo pull (merge) --force tools/z64compress

subrepo:
  subdir:   "tools/z64compress"
  merged:   "163ca2af"
upstream:
  origin:   "https://github.com/z64me/z64compress.git"
  branch:   "main"
  commit:   "163ca2af"
git-subrepo:
  version:  "0.4.3"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "2f68596"

* Make z64compress print to stdout

* sneeky commit to update warnings tooling

* test

* Another test

* Mark fixing overlay reloc generating as a TODO

* Update warnings stuff

* Communicate the return code from running z64compress back to the Makefile through the wrapper

* Run formatter, remove extra commented copy of function

* Re-fix some includes

* Convert atan to hex to conform to decided style

* Some tidying up, remove c for fp and the other two handwritten code files

* BSS in z_collision_check & z_scene_proc

* add static back in

* Fix timerintr bss, add file to spec, some cleanup

* Remove externs

* Newline

* Readd enums

* Typo

* Colours

* Comments for hitmark enum values

Co-authored-by: EllipticEllipsis <73679967+EllipticEllipsis@users.noreply.github.com>

* Improvements and suggestions

* Organize and remove unused imports and use env for python3 scripts, delete unused overlay.py

Co-authored-by: angie <angheloalf95@gmail.com>
Co-authored-by: Elliptic Ellipsis <elliptic.ellipsis@gmail.com>
Co-authored-by: engineer124 <engineer124engineer124@gmail.com>
Co-authored-by: EllipticEllipsis <73679967+EllipticEllipsis@users.noreply.github.com>

src/boot_O2_g3/CIC6105.c

@@ -1,5 +1,4 @@
-#include <ultra64.h>
-#include <global.h>
+#include "global.h"
 
 void CIC6105_Nop80081820(void) {
 }

src/boot_O2_g3/__osMemcmp.c

@@ -1,10 +1,9 @@
-#include <ultra64.h>
-#include <global.h>
+#include "global.h"
 
 /*
  * Compare strings (strcmp).
  */
-s32 func_800968B0(const char* str1, const char* str2) {
+s32 __osMemcmp(const char* str1, const char* str2) {
     char currentCharStr1;
     char currentCharStr2;
     do {

src/boot_O2_g3/boot_0x80081920.c

@@ -1,11 +0,0 @@
-#include <ultra64.h>
-#include <global.h>
-
-// This file contains print functions that are used by fault.c, so they are placed in the fault module
-
-void Fault_Nop80081920(UNK_TYPE4 param_1, ...) {
-}
-void Fault_Log(char* str, ...) {
-}
-void Fault_Nop80081958(UNK_TYPE4 param_1, ...) {
-}

src/boot_O2_g3/boot_main.c

@@ -1,5 +1,4 @@
-#include <ultra64.h>
-#include <global.h>
+#include "global.h"
 
 void bootproc(void) {
     StackCheck_Init(&sBootThreadInfo, sBootThreadStack, sBootThreadStack + sizeof(sBootThreadStack), 0, -1, "boot");

src/boot_O2_g3/fault_drawer.c

@@ -0,0 +1,180 @@
+#include "global.h"
+#include "vt.h"
+
+FaultDrawer* sFaultDrawContext = &sFaultDrawerStruct;
+FaultDrawer sFaultDrawerDefault = {
+    (u16*)0x803DA800,                   // fb - TODO map out buffers in this region and avoid hard-coded pointer
+    SCREEN_WIDTH,                       // w
+    SCREEN_HEIGHT,                      // h
+    16,                                 // yStart
+    223,                                // yEnd
+    22,                                 // xStart
+    297,                                // xEnd
+    GPACK_RGBA5551(255, 255, 255, 255), // foreColor
+    GPACK_RGBA5551(0, 0, 0, 0),         // backColor
+    22,                                 // cursorX
+    16,                                 // cursorY
+    (u32*)&sFaultDrawerFont,            // font
+    8,                                  // charW
+    8,                                  // charH
+    0,                                  // charWPad
+    0,                                  //  charHPad
+    {
+        // printColors
+        GPACK_RGBA5551(0, 0, 0, 1),
+        GPACK_RGBA5551(255, 0, 0, 1),
+        GPACK_RGBA5551(0, 255, 0, 1),
+        GPACK_RGBA5551(255, 255, 0, 1),
+        GPACK_RGBA5551(0, 0, 255, 1),
+        GPACK_RGBA5551(255, 0, 255, 1),
+        GPACK_RGBA5551(0, 255, 255, 1),
+        GPACK_RGBA5551(255, 255, 255, 1),
+        GPACK_RGBA5551(120, 120, 120, 1),
+        GPACK_RGBA5551(176, 176, 176, 1),
+    },
+    0,    // escCode
+    0,    // osSyncPrintfEnabled
+    NULL, // inputCallback
+};
+
+#pragma GLOBAL_ASM("asm/non_matchings/boot/fault_drawer/sFaultDrawerFont.s")
+
+void FaultDrawer_SetOsSyncPrintfEnabled(u32 enabled) {
+    sFaultDrawContext->osSyncPrintfEnabled = enabled;
+}
+
+void FaultDrawer_DrawRecImpl(s32 xStart, s32 yStart, s32 xEnd, s32 yEnd, u16 color) {
+    u16* fb;
+    s32 x, y;
+    s32 xDiff = sFaultDrawContext->w - xStart;
+    s32 yDiff = sFaultDrawContext->h - yStart;
+    s32 xSize = xEnd - xStart + 1;
+    s32 ySize = yEnd - yStart + 1;
+
+    if (xDiff > 0 && yDiff > 0) {
+        if (xDiff < xSize) {
+            xSize = xDiff;
+        }
+
+        if (yDiff < ySize) {
+            ySize = yDiff;
+        }
+
+        fb = sFaultDrawContext->fb + sFaultDrawContext->w * yStart + xStart;
+        for (y = 0; y < ySize; y++) {
+            for (x = 0; x < xSize; x++) {
+                *fb++ = color;
+            }
+            fb += sFaultDrawContext->w - xSize;
+        }
+
+        osWritebackDCacheAll();
+    }
+}
+
+#pragma GLOBAL_ASM("asm/non_matchings/boot/fault_drawer/FaultDrawer_DrawChar.s")
+
+s32 FaultDrawer_ColorToPrintColor(u16 color) {
+    s32 i;
+
+    for (i = 0; i < 10; i++) {
+        if (color == sFaultDrawContext->printColors[i]) {
+            return i;
+        }
+    }
+    return -1;
+}
+
+void FaultDrawer_UpdatePrintColor() {
+    s32 idx;
+
+    if (sFaultDrawContext->osSyncPrintfEnabled) {
+        osSyncPrintf(VT_RST);
+        idx = FaultDrawer_ColorToPrintColor(sFaultDrawContext->foreColor);
+        if (idx >= 0 && idx < 8) {
+            osSyncPrintf(VT_SGR("3%d"), idx);
+        }
+        idx = FaultDrawer_ColorToPrintColor(sFaultDrawContext->backColor);
+        if (idx >= 0 && idx < 8) {
+            osSyncPrintf(VT_SGR("4%d"), idx);
+        }
+    }
+}
+
+void FaultDrawer_SetForeColor(u16 color) {
+    sFaultDrawContext->foreColor = color;
+    FaultDrawer_UpdatePrintColor();
+}
+
+void FaultDrawer_SetBackColor(u16 color) {
+    sFaultDrawContext->backColor = color;
+    FaultDrawer_UpdatePrintColor();
+}
+
+void FaultDrawer_SetFontColor(u16 color) {
+    FaultDrawer_SetForeColor(color | 1); // force alpha to be set
+}
+
+void FaultDrawer_SetCharPad(s8 padW, s8 padH) {
+    sFaultDrawContext->charWPad = padW;
+    sFaultDrawContext->charHPad = padH;
+}
+
+void FaultDrawer_SetCursor(s32 x, s32 y) {
+    if (sFaultDrawContext->osSyncPrintfEnabled) {
+        osSyncPrintf(VT_CUP("%d", "%d"),
+                     (y - sFaultDrawContext->yStart) / (sFaultDrawContext->charH + sFaultDrawContext->charHPad),
+                     (x - sFaultDrawContext->xStart) / (sFaultDrawContext->charW + sFaultDrawContext->charWPad));
+    }
+    sFaultDrawContext->cursorX = x;
+    sFaultDrawContext->cursorY = y;
+}
+
+void FaultDrawer_FillScreen() {
+    if (sFaultDrawContext->osSyncPrintfEnabled) {
+        osSyncPrintf(VT_CLS);
+    }
+
+    FaultDrawer_DrawRecImpl(sFaultDrawContext->xStart, sFaultDrawContext->yStart, sFaultDrawContext->xEnd,
+                            sFaultDrawContext->yEnd, sFaultDrawContext->backColor | 1);
+    FaultDrawer_SetCursor(sFaultDrawContext->xStart, sFaultDrawContext->yStart);
+}
+
+#pragma GLOBAL_ASM("asm/non_matchings/boot/fault_drawer/FaultDrawer_FormatStringFunc.s")
+
+#pragma GLOBAL_ASM("asm/non_matchings/boot/fault_drawer/D_80099080.s")
+
+void FaultDrawer_VPrintf(const char* str, char* args) { // va_list
+    _Printf((printf_func)FaultDrawer_FormatStringFunc, sFaultDrawContext, str, args);
+}
+
+void FaultDrawer_Printf(const char* fmt, ...) {
+    va_list args;
+    va_start(args, fmt);
+
+    FaultDrawer_VPrintf(fmt, args);
+}
+
+void FaultDrawer_DrawText(s32 x, s32 y, const char* fmt, ...) {
+    va_list args;
+    va_start(args, fmt);
+
+    FaultDrawer_SetCursor(x, y);
+    FaultDrawer_VPrintf(fmt, args);
+}
+
+void FaultDrawer_SetDrawerFB(void* fb, u16 w, u16 h) {
+    sFaultDrawContext->fb = (u16*)fb;
+    sFaultDrawContext->w = w;
+    sFaultDrawContext->h = h;
+}
+
+void FaultDrawer_SetInputCallback(void (*callback)()) {
+    sFaultDrawContext->inputCallback = callback;
+}
+
+void FaultDrawer_Init() {
+    sFaultDrawContext = &sFaultDrawerStruct;
+    bcopy(&sFaultDrawerDefault, sFaultDrawContext, sizeof(FaultDrawer));
+    sFaultDrawContext->fb = (u16*)((osMemSize | 0x80000000) - 0x25800);
+}

src/boot_O2_g3/idle.c

@@ -1,5 +1,4 @@
-#include <ultra64.h>
-#include <global.h>
+#include "global.h"
 
 u8 D_80096B20 = 1;
 vu8 gViConfigUseDefault = 1;
@@ -35,8 +34,8 @@ void Idle_InitMemory(void) {
     void* memEnd = (void*)(0x80000000 + osMemSize);
 
     Idle_ClearMemory((void*)0x80000400, &gFramebuffer1);
-    Idle_ClearMemory(&D_80025D00, (int*)&boot_text_start);
-    Idle_ClearMemory(&code_bss_end, memEnd);
+    Idle_ClearMemory(&D_80025D00, (int*)&bootproc);
+    Idle_ClearMemory(&gGfxSPTaskYieldBuffer, memEnd);
 }
 
 #ifdef NON_MATCHING
@@ -52,7 +51,7 @@ void Idle_InitCodeAndMemory(void) {
     oldSize = sDmaMgrDmaBuffSize;
     sDmaMgrDmaBuffSize = 0;
 
-    DmaMgr_SendRequestImpl(&dmaReq, (u32)&code_text_start, (u32)_codeSegmentRomStart,
+    DmaMgr_SendRequestImpl(&dmaReq, (u32)_codeSegmentStart, (u32)_codeSegmentRomStart,
                            (u32)_codeSegmentRomEnd - (u32)_codeSegmentRomStart, 0, &queue, 0);
     Idle_InitScreen();
     Idle_InitMemory();
@@ -60,10 +59,10 @@ void Idle_InitCodeAndMemory(void) {
 
     sDmaMgrDmaBuffSize = oldSize;
 
-    Idle_ClearMemory(&code_bss_start, &code_bss_end);
+    Idle_ClearMemory(_codeSegmentBssStart, _codeSegmentBssEnd);
 }
 #else
-#pragma GLOBAL_ASM("./asm/non_matchings/boot/idle/Idle_InitCodeAndMemory.asm")
+#pragma GLOBAL_ASM("asm/non_matchings/boot/idle/Idle_InitCodeAndMemory.s")
 #endif
 
 void Main_ThreadEntry(void* arg) {

src/boot_O2_g3/irqmgr.c

@@ -1,5 +1,4 @@
-#include <ultra64.h>
-#include <global.h>
+#include "global.h"
 
 vs32 gIrqMgrResetStatus = 0;
 volatile OSTime sIrqMgrResetTime = 0;

src/boot_O2_g3/syncprintf.c

@@ -0,0 +1,10 @@
+#include "global.h"
+
+// This file contains print functions that are used by fault.c, so they are placed in the fault module
+
+void __osSyncVPrintf(UNK_TYPE4 param_1, ...) {
+}
+void osSyncPrintf(char* str, ...) {
+}
+void rmonPrintf(UNK_TYPE4 param_1, ...) {
+}

src/boot_O2_g3/viconfig.c

@@ -1,5 +1,4 @@
-#include <ultra64.h>
-#include <global.h>
+#include "global.h"
 
 void ViConfig_UpdateVi(u32 mode) {
     if (mode != 0) {
@@ -27,7 +26,7 @@ void ViConfig_UpdateVi(u32 mode) {
         osViSetMode(&gViConfigMode);
 
         if (gViConfigAdditionalScanLines != 0) {
-            func_80087E00(gViConfigAdditionalScanLines);
+            osViExtendVStart(gViConfigAdditionalScanLines);
         }
 
         if (gViConfigFeatures != 0) {

src/boot_O2_g3/yaz0.c

@@ -1,5 +1,4 @@
-#include <ultra64.h>
-#include <global.h>
+#include "global.h"
 
 u8 sYaz0DataBuffer[0x400];
 u8* sYaz0CurDataEnd;
@@ -50,7 +49,7 @@ void* Yaz0_NextDMA(void* curSrcPos) {
     } else {
         oldPri = osGetThreadPri(NULL);
         osSetThreadPri(NULL, 0x7F);
-        Fault_Log("圧縮展開異常\n");
+        osSyncPrintf("圧縮展開異常\n");
         osSetThreadPri(NULL, oldPri);
     }
 
@@ -131,7 +130,7 @@ void Yaz0_Decompress(u32 romStart, void* dst, u32 size) {
 
     if (sYaz0CurDataEnd != NULL) {
         while (sYaz0CurDataEnd != NULL) {
-            func_80087A1C(10);
+            Sleep_Usec(10);
         }
     }
 

src/boot_O2_g3/z_std_dma.c

@@ -1,5 +1,4 @@
-#include <ultra64.h>
-#include <global.h>
+#include "global.h"
 
 u32 sDmaMgrDmaBuffSize = 0x2000;
 
@@ -203,8 +202,6 @@ s32 DmaMgr_SendRequest0(void* vramStart, u32 vromStart, u32 size) {
     return 0;
 }
 
-const char dmamgrThreadName[] = "dmamgr";
-
 #ifdef NON_MATCHING
 // TODO missing a useless move initializing v0, and some reorderings
 void DmaMgr_Start() {
@@ -226,7 +223,7 @@ void DmaMgr_Start() {
     osStartThread(&sDmaMgrThread);
 }
 #else
-#pragma GLOBAL_ASM("./asm/non_matchings/boot/z_std_dma/DmaMgr_Start.asm")
+#pragma GLOBAL_ASM("asm/non_matchings/boot/z_std_dma/DmaMgr_Start.s")
 #endif
 
 void DmaMgr_Stop() {