mirror of
https://github.com/izzy2lost/2ship2harkinian-Android.git
synced 2026-06-19 01:20:08 -07:00
Overhaul the build system (#234)
* wip * fix * add disassembler * Disasm builds OK * Variable addends * More wip * Rodata migration implemented * Cleanup old tools * Try fix submodule -> subrepo merge * git subrepo pull --force --remote=https://github.com/zeldaret/ZAPD.git tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "602e609" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "602e609" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo" commit: "2f68596" * Builds again but assets are totally broken * git subrepo pull --force tools/asm-processor subrepo: subdir: "tools/asm-processor" merged: "1ffdb08a" upstream: origin: "https://github.com/simonlindholm/asm-processor.git" branch: "master" commit: "1ffdb08a" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo" commit: "2f68596" * More cleanup, move functions.txt and variables.txt to tools/disasm and rm tables * rm z64compress in preparation for subrepo * git subrepo clone (merge) https://github.com/z64me/z64compress.git tools/z64compress subrepo: subdir: "tools/z64compress" merged: "eb11085c" upstream: origin: "https://github.com/z64me/z64compress.git" branch: "main" commit: "eb11085c" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo" commit: "2f68596" * Fix asset extraction * Fix diff-init make rule * Split code bss * Split assumed linker bug padding from assembly files * add filelists for mm.us.rev1 * Maybe working, but I'm not sure * add overlays to spec * Add rodata to actos * Everything compiles * Make a lot of C files for code * Add almost every file in code to spec * whoops * 3 code files left * add scenes to spec * More progress on progress.py * Fix skelanime in spec * audio files! * Fix merge issues * Fix some C files in code * Fix remaining code files * Use existing O1 C files in spec * reorder boot order in spec * update spec * fault.c * Convert relocs on completed actors, fixbaserom uses current rom name * more boot files * Add VT macros and script * finish already existing boot files * most of libultra * fix 64bits libultra files * Use C files for libultra, wrap some functions in NON_MATCHING * Remove duplicate of OS_CLOCK_RATE from fault.c * C files for fbdemos * delete dumb files * bootstrap C files, still need to add them to the spec * update fixbaserom * boot OK? * I forgot to commit the spec * C for gamestates * C for kaleido * Change all includes to "" * copy actor sizes script from oot * I forgot to delete those files * Basic C files for effects * Add effects initvars names * Remove mislabelled boot functions from header/txt * Begin porting bootstrap_fx, some sizes * Fix <> * Fix enum * Fix diff.py * fix libultra stuff * update regconvert * update setup warnings * add some missing ; * Fix some makefile stuff and other fixes on some non_matching functions * add executable flag in extract_baserom and fixbaserom * fix relative path * copy assist from oot * fix map path * another assist path fix * Delete C files for handwritten files * add code_801A51F0 to spec * add gfxbuffers to spec * Move rodata to top of each file when possible * UNK_TYPEs for func_801A51F0 * Remove kaleido rodata from spec * Update spec and undefined_syms for recent merge * GCC warnings and fix errors in nonmatchings, * round percentage numbers * progress script: format changes * progress: error on non-existing files * fix warning in z_scene_table * Match 2 nonmatchings in z_actor * Warnings in lightswitch and invadepoh * Fix warning in z_actor_dlftbls * I though I fixed this one * whoops * Comment out CC_CHECK * Removed redundant ultra64.h includes * Update asm_processor, sorted boot_O1 into other folders, completed the fbdemo bootstrap, cleaned up undefined_syms * Completed gamestates bootstrap * Split kaleido_scope * Remove section.h and segment.h, move keep object externs to a common location in variables.h * Completed effects bootstrap * Segmented address externs for effects, fbdemos, gamestates and kaleido * Move actor data externs out of the if 0 * Segmented address externs for actors * Prepare actionfunc detection * fix script, how did it even work before * Fix actionfunc script again, re-introduce some more intermediate prints to the disassembler * Automated actionFunc detection in actors * Segmented addresses from player .text * rm old segment addrs script and fix build * Move sizes folder to tools * Make build.py executable * New Jenkinsfile Prayge * Remove numpy dependencies * Add warnings_disasm_current.txt * my bad * Update spec and undefined_syms * Add z_eff_ss_hahen to pametfrog * git subrepo pull (merge) --force tools/z64compress subrepo: subdir: "tools/z64compress" merged: "163ca2af" upstream: origin: "https://github.com/z64me/z64compress.git" branch: "main" commit: "163ca2af" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo" commit: "2f68596" * Make z64compress print to stdout * sneeky commit to update warnings tooling * test * Another test * Mark fixing overlay reloc generating as a TODO * Update warnings stuff * Communicate the return code from running z64compress back to the Makefile through the wrapper * Run formatter, remove extra commented copy of function * Re-fix some includes * Convert atan to hex to conform to decided style * Some tidying up, remove c for fp and the other two handwritten code files * BSS in z_collision_check & z_scene_proc * add static back in * Fix timerintr bss, add file to spec, some cleanup * Remove externs * Newline * Readd enums * Typo * Colours * Comments for hitmark enum values Co-authored-by: EllipticEllipsis <73679967+EllipticEllipsis@users.noreply.github.com> * Improvements and suggestions * Organize and remove unused imports and use env for python3 scripts, delete unused overlay.py Co-authored-by: angie <angheloalf95@gmail.com> Co-authored-by: Elliptic Ellipsis <elliptic.ellipsis@gmail.com> Co-authored-by: engineer124 <engineer124engineer124@gmail.com> Co-authored-by: EllipticEllipsis <73679967+EllipticEllipsis@users.noreply.github.com>
This commit is contained in:
Tharo
co-authored by
angie
Elliptic Ellipsis
engineer124
EllipticEllipsis
angie
Elliptic Ellipsis
engineer124
EllipticEllipsis
parent
60ff207982
commit
7743e5a2c4
+73
-43
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
import os, argparse, json, re, csv, git
|
||||
import argparse, csv, git, json, os, re
|
||||
|
||||
parser = argparse.ArgumentParser()
|
||||
|
||||
@@ -9,8 +9,11 @@ parser.add_argument("-m", "--matching", dest='matching', action='store_true',
|
||||
help="Output matching progress instead of decompilation progress")
|
||||
args = parser.parse_args()
|
||||
|
||||
NON_MATCHING_PATTERN = r"#ifdef\s+NON_MATCHING.*?#pragma\s+GLOBAL_ASM\s*\(\s*\"(.*?)\"\s*\).*?#endif"
|
||||
NOT_ATTEMPTED_PATTERN = r"#pragma\s+GLOBAL_ASM\s*\(\s*\"(.*?)\"\s*\)"
|
||||
NON_MATCHING_PATTERN = r'#ifdef\s+NON_MATCHING.*?#pragma\s+GLOBAL_ASM\s*\(\s*"(.*?)"\s*\).*?#endif'
|
||||
NOT_ATTEMPTED_PATTERN = r'#pragma\s+GLOBAL_ASM\s*\(\s*"(.*?)"\s*\)'
|
||||
|
||||
# TODO: consider making this a parameter of this script
|
||||
GAME_VERSION = "mm.us.rev1"
|
||||
|
||||
def GetFunctionsByPattern(pattern, files):
|
||||
functions = []
|
||||
@@ -38,18 +41,30 @@ def GetFiles(path, ext):
|
||||
|
||||
return files
|
||||
|
||||
def GetRemovableSize(functions_to_count, path):
|
||||
def GetCsvFilelist(version, filelist):
|
||||
path = os.path.join("tools", "filelists", version, filelist)
|
||||
with open(path, newline='') as f:
|
||||
return list(csv.reader(f, delimiter=','))
|
||||
|
||||
def GetRemovableSize(functions_to_count):
|
||||
size = 0
|
||||
|
||||
asm_files = GetFiles(path, ".asm")
|
||||
|
||||
for asm_file_path in asm_files:
|
||||
for asm_file_path in functions_to_count:
|
||||
if "//" in asm_file_path:
|
||||
raise RuntimeError(f"Invalid file path: {asm_file_path}")
|
||||
file_size = 0
|
||||
if asm_file_path in functions_to_count:
|
||||
asm_lines = ReadAllLines(asm_file_path)
|
||||
asm_lines = ReadAllLines(asm_file_path)
|
||||
shouldCount = True
|
||||
|
||||
for asm_line in asm_lines:
|
||||
if (asm_line[0:2] == "/*" and asm_line[30:32] == "*/"):
|
||||
for asm_line in asm_lines:
|
||||
if asm_line[0] == ".":
|
||||
if asm_line.startswith(".text") or asm_line.startswith(".section .text"):
|
||||
shouldCount = True
|
||||
elif ".rdata" in asm_line or ".late_rodata" in asm_line:
|
||||
shouldCount = False
|
||||
|
||||
if shouldCount:
|
||||
if (asm_line[0:2] == "/*" and asm_line[28:30] == "*/"):
|
||||
file_size += 4
|
||||
|
||||
size += file_size
|
||||
@@ -72,11 +87,11 @@ if not args.matching:
|
||||
non_matching_functions = []
|
||||
|
||||
# Get asset files
|
||||
audio_files = GetFiles("baserom/assets/audio", "")
|
||||
misc_files = GetFiles("baserom/assets/misc", "")
|
||||
object_files = GetFiles("baserom/assets/objects", "")
|
||||
scene_files = GetFiles("baserom/assets/scenes", "")
|
||||
texture_files = GetFiles("baserom/assets/textures", "")
|
||||
audio_files = GetCsvFilelist(GAME_VERSION, "audio.csv")
|
||||
misc_files = GetCsvFilelist(GAME_VERSION, "misc.csv")
|
||||
object_files = GetCsvFilelist(GAME_VERSION, "object.csv")
|
||||
scene_files = GetCsvFilelist(GAME_VERSION, "scene.csv")
|
||||
texture_files = GetCsvFilelist(GAME_VERSION, "texture.csv")
|
||||
|
||||
# Initialize all the code values
|
||||
src = 0
|
||||
@@ -140,14 +155,22 @@ src_boot += src_libultra
|
||||
asm_boot += asm_libultra
|
||||
|
||||
# Calculate Non-Matching
|
||||
non_matching_asm_ovl = GetRemovableSize(non_matching_functions, "./asm/non_matchings/overlays")
|
||||
non_matching_asm_code = GetRemovableSize(non_matching_functions, "./asm/non_matchings/code")
|
||||
non_matching_asm_boot = GetRemovableSize(non_matching_functions, "./asm/non_matchings/boot")
|
||||
non_matching_functions_ovl = list(filter(lambda x: "/overlays/" in x, non_matching_functions))
|
||||
non_matching_functions_code = list(filter(lambda x: "/code/" in x, non_matching_functions))
|
||||
non_matching_functions_boot = list(filter(lambda x: "/boot/" in x, non_matching_functions))
|
||||
|
||||
non_matching_asm_ovl = GetRemovableSize(non_matching_functions_ovl)
|
||||
non_matching_asm_code = GetRemovableSize(non_matching_functions_code)
|
||||
non_matching_asm_boot = GetRemovableSize(non_matching_functions_boot)
|
||||
|
||||
# Calculate Not Attempted
|
||||
not_attempted_asm_ovl = GetRemovableSize(not_attempted_functions, "./asm/non_matchings/overlays")
|
||||
not_attempted_asm_code = GetRemovableSize(not_attempted_functions, "./asm/non_matchings/code")
|
||||
not_attempted_asm_boot = GetRemovableSize(not_attempted_functions, "./asm/non_matchings/boot")
|
||||
not_attempted_functions_ovl = list(filter(lambda x: "/overlays/" in x, not_attempted_functions))
|
||||
not_attempted_functions_code = list(filter(lambda x: "/code/" in x, not_attempted_functions))
|
||||
not_attempted_functions_boot = list(filter(lambda x: "/boot/" in x, not_attempted_functions))
|
||||
|
||||
not_attempted_asm_ovl = GetRemovableSize(not_attempted_functions_ovl)
|
||||
not_attempted_asm_code = GetRemovableSize(not_attempted_functions_code)
|
||||
not_attempted_asm_boot = GetRemovableSize(not_attempted_functions_boot)
|
||||
|
||||
# All the non matching asm is the sum of non-matching code
|
||||
non_matching_asm = non_matching_asm_ovl + non_matching_asm_code + non_matching_asm_boot
|
||||
@@ -172,16 +195,16 @@ misc_size = 0
|
||||
object_size = 0
|
||||
scene_size = 0
|
||||
texture_size = 0
|
||||
for f in audio_files:
|
||||
audio_size += os.stat(f).st_size
|
||||
for f in misc_files:
|
||||
misc_size += os.stat(f).st_size
|
||||
for f in object_files:
|
||||
object_size += os.stat(f).st_size
|
||||
for f in scene_files:
|
||||
scene_size += os.stat(f).st_size
|
||||
for f in texture_files:
|
||||
texture_size += os.stat(f).st_size
|
||||
for index, f in audio_files:
|
||||
audio_size += os.stat(os.path.join("baserom", f)).st_size
|
||||
for index, f in misc_files:
|
||||
misc_size += os.stat(os.path.join("baserom", f)).st_size
|
||||
for index, f in object_files:
|
||||
object_size += os.stat(os.path.join("baserom", f)).st_size
|
||||
for index, f in scene_files:
|
||||
scene_size += os.stat(os.path.join("baserom", f)).st_size
|
||||
for index, f in texture_files:
|
||||
texture_size += os.stat(os.path.join("baserom", f)).st_size
|
||||
|
||||
# Calculate asm and src totals
|
||||
src = src_code + src_boot + src_ovl
|
||||
@@ -194,12 +217,17 @@ asm += non_matching_asm + not_attempted_asm
|
||||
# Calculate the total amount of decompilable code
|
||||
total = src + asm
|
||||
|
||||
# Calculate assets totals
|
||||
assets = audio + misc + object_ + scene + texture
|
||||
assets_total = audio_size + misc_size + object_size + scene_size + texture_size
|
||||
|
||||
# Convert vaules to percentages
|
||||
src_percent = 100 * src / total
|
||||
asm_percent = 100 * asm / total
|
||||
code_percent = 100 * code / code_size
|
||||
boot_percent = 100 * boot / boot_size
|
||||
ovl_percent = 100 * ovl / ovl_size
|
||||
assets_percent = 100 * assets / assets_total
|
||||
audio_percent = 100 * audio / audio_size
|
||||
misc_percent = 100 * misc / misc_size
|
||||
object_percent = 100 * object_ / object_size
|
||||
@@ -266,16 +294,18 @@ elif args.format == 'shield-json':
|
||||
elif args.format == 'text':
|
||||
adjective = "decompiled" if not args.matching else "matched"
|
||||
|
||||
print(str(total) + " total bytes of decompilable code\n")
|
||||
print(str(src) + " bytes " + adjective + " in src " + str(src_percent) + "%\n")
|
||||
print(str(boot) + "/" + str(boot_size) + " bytes " + adjective + " in boot " + str(boot_percent) + "%\n")
|
||||
print(str(code) + "/" + str(code_size) + " bytes " + adjective + " in code " + str(code_percent) + "%\n")
|
||||
print(str(ovl) + "/" + str(ovl_size) + " bytes " + adjective + " in overlays " + str(ovl_percent) + "%\n")
|
||||
print(str(audio) + "/" + str(audio_size) + " bytes reconstructed in audio " + str(audio_percent) + "%\n")
|
||||
print(str(misc) + "/" + str(misc_size) + " bytes reconstructed in misc " + str(misc_percent) + "%\n")
|
||||
print(str(object_) + "/" + str(object_size) + " bytes reconstructed in objects " + str(object_percent) + "%\n")
|
||||
print(str(scene) + "/" + str(scene_size) + " bytes reconstructed in scenes " + str(scene_percent) + "%\n")
|
||||
print(str(texture) + "/" + str(texture_size) + " bytes reconstructed in textures " + str(texture_percent) + "%\n")
|
||||
print("src: {:>9} / {:>8} total bytes {:<13} {:>9.4f}%".format(src, total, adjective, round(src_percent, 4)))
|
||||
print(" boot: {:>9} / {:>8} bytes {:<13} {:>9.4f}%".format(boot, boot_size, adjective, round(boot_percent, 4)))
|
||||
print(" code: {:>9} / {:>8} bytes {:<13} {:>9.4f}%".format(code, code_size, adjective, round(code_percent, 4)))
|
||||
print(" overlays: {:>9} / {:>8} bytes {:<13} {:>9.4f}%".format(ovl, ovl_size, adjective, round(ovl_percent, 4)))
|
||||
print()
|
||||
print("assets: {:>9} / {:>8} bytes reconstructed {:>9.4f}%".format(assets, assets_total, round(assets_percent, 4)))
|
||||
print(" audio: {:>9} / {:>8} bytes reconstructed {:>9.4f}%".format(audio, audio_size, round(audio_percent, 4)))
|
||||
print(" misc: {:>9} / {:>8} bytes reconstructed {:>9.4f}%".format(misc, misc_size, round(misc_percent, 4)))
|
||||
print(" objects: {:>9} / {:>8} bytes reconstructed {:>9.4f}%".format(object_, object_size, round(object_percent, 4)))
|
||||
print(" scenes: {:>9} / {:>8} bytes reconstructed {:>9.4f}%".format(scene, scene_size, round(scene_percent, 4)))
|
||||
print(" textures: {:>9} / {:>8} bytes reconstructed {:>9.4f}%".format(texture, texture_size, round(texture_percent, 4)))
|
||||
print()
|
||||
print("------------------------------------\n")
|
||||
|
||||
if (rupees > 0):
|
||||
@@ -283,4 +313,4 @@ elif args.format == 'text':
|
||||
else:
|
||||
print('You have {}/{} masks.\n'.format(masks, num_masks))
|
||||
else:
|
||||
print("Unknown format argument: " + args.format)
|
||||
print("Unknown format argument: " + args.format)
|
||||
|
||||
Reference in New Issue
Block a user