Initial commit

This commit is contained in:
Luke Street
2023-10-04 23:30:39 -04:00
commit 5b8fbc6cd5
29 changed files with 2575 additions and 0 deletions
+13
View File
@@ -0,0 +1,13 @@
# Auto detect text files and perform LF normalization
* text=auto
# Explicitly declare text files
*.py text
# Enforce platform-specific encodings
*.bat text eol=crlf
*.sh text eol=lf
*.sha1 text eol=lf
# DTK keeps these files with LF
config/**/*.txt text eol=lf
+11
View File
@@ -0,0 +1,11 @@
__pycache__
.idea
.vscode
.ninja_*
*.exe
build
build.ninja
objdiff.json
orig/*/*
!orig/*/.gitkeep
/*.txt
+22
View File
@@ -0,0 +1,22 @@
{
"configurations": [
{
"name": "Linux",
"includePath": [
"${workspaceFolder}/include/**"
],
"cStandard": "c99",
"cppStandard": "c++98",
"intelliSenseMode": "linux-clang-x86",
"compilerPath": "",
"configurationProvider": "ms-vscode.makefile-tools",
"browse": {
"path": [
"${workspaceFolder}/include"
],
"limitSymbolsToIncludedHeaders": true
}
}
],
"version": 4
}
+26
View File
@@ -0,0 +1,26 @@
{
"[c]": {
"files.encoding": "utf8",
"editor.defaultFormatter": "xaver.clang-format"
},
"[cpp]": {
"files.encoding": "utf8",
"editor.defaultFormatter": "xaver.clang-format"
},
"[python]": {
"editor.defaultFormatter": "ms-python.black-formatter"
},
"files.insertFinalNewline": true,
"files.trimFinalNewlines": true,
"files.associations": {
"*.inc": "cpp"
},
"search.useIgnoreFiles": false,
"search.exclude": {
"build/*/config.json": true,
"build/**/*.MAP": true,
"build.ninja": true,
".ninja_*": true,
"objdiff.json": true
}
}
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2023 Luke Street
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+114
View File
@@ -0,0 +1,114 @@
Some Game
[![Build Status]][actions] ![Progress] ![DOL Progress] ![RELs Progress] [![Discord Badge]][discord]
=============
<!--
Replace with your repository's URL.
-->
[Build Status]: https://github.com/zeldaret/tww/actions/workflows/build.yml/badge.svg
[actions]: https://github.com/zeldaret/tww/actions/workflows/build.yml
<!---
Code progress URL:
https://progress.decomp.club/data/[project]/[version]/all/?mode=shield&measure=code
URL encoded then appended to: https://img.shields.io/endpoint?label=Code&url=
-->
[Progress]: https://img.shields.io/endpoint?label=Code&url=https%3A%2F%2Fprogress.decomp.club%2Fdata%2Ftww%2FGZLE01%2Fall%2F%3Fmode%3Dshield%26measure%3Dcode
<!---
DOL progress URL:
https://progress.decomp.club/data/[project]/[version]/dol/?mode=shield&measure=code
URL encoded then appended to: https://img.shields.io/endpoint?label=DOL&url=
-->
[DOL Progress]: https://img.shields.io/endpoint?label=DOL&url=https%3A%2F%2Fprogress.decomp.club%2Fdata%2Ftww%2FGZLE01%2Fdol%2F%3Fmode%3Dshield%26measure%3Dcode
<!--
REL progress URL:
https://progress.decomp.club/data/[project]/[version]/modules/?mode=shield&measure=code
URL encoded then appended to: https://img.shields.io/endpoint?label=RELs&url=
-->
[RELs Progress]: https://img.shields.io/endpoint?label=RELs&url=https%3A%2F%2Fprogress.decomp.club%2Fdata%2Ftww%2FGZLE01%2Fmodules%2F%3Fmode%3Dshield%26measure%3Dcode
<!--
Replace with your Discord server's ID and invite URL.
-->
[Discord Badge]: https://img.shields.io/discord/727908905392275526?color=%237289DA&logo=discord&logoColor=%23FFFFFF
[discord]: https://discord.gg/hKx3FJJgrV
A work-in-progress decompilation of Some Game.
This repository does **not** contain any game assets or assembly whatsoever. An existing copy of the game is required.
Supported versions:
- `GAMEID`: Rev 0 (USA)
Dependencies
============
Windows:
--------
On Windows, it's **highly recommended** to use native tooling. WSL or msys2 are **not** required.
When running under WSL, [objdiff](#diffing) is unable to get filesystem notifications for automatic rebuilds.
- Install [Python](https://www.python.org/downloads/) and add it to `%PATH%`.
- Also available from the [Windows Store](https://apps.microsoft.com/store/detail/python-311/9NRWMJP3717K).
- Download [ninja](https://github.com/ninja-build/ninja/releases) and add it to `%PATH%`.
- Quick install via pip: `pip install ninja`
macOS:
------
- Install [ninja](https://github.com/ninja-build/ninja/wiki/Pre-built-Ninja-packages):
```
brew install ninja
```
- Install [wine-crossover](https://github.com/Gcenx/homebrew-wine):
```
brew install --cask --no-quarantine gcenx/wine/wine-crossover
```
After OS upgrades, if macOS complains about `Wine Crossover.app` being unverified, you can unquarantine it using:
```sh
sudo xattr -rd com.apple.quarantine '/Applications/Wine Crossover.app'
```
Linux:
------
- Install [ninja](https://github.com/ninja-build/ninja/wiki/Pre-built-Ninja-packages).
- For non-x86(_64) platforms: Install wine from your package manager.
- For x86(_64), [WiBo](https://github.com/decompals/WiBo), a minimal 32-bit Windows binary wrapper, will be automatically downloaded and used.
Building
========
- Clone the repository:
```
git clone https://github.com/my/repo.git
```
- Using [Dolphin Emulator](https://dolphin-emu.org/), extract your game to `orig/GAMEID`.
![](assets/dolphin-extract.png)
- To save space, the only necessary files are the following. Any others can be deleted.
- `sys/main.dol`
- `files/rels/*.rel`
- Configure:
```
python configure.py
```
To use a version other than `GAMEID` (USA), specify it with `--version`.
- Build:
```
ninja
```
Visual Studio Code
==================
If desired, use the recommended Visual Studio Code settings by renaming the `.vscode.example` directory to `.vscode`.
Diffing
=======
Once the initial build succeeds, an `objdiff.json` should exist in the project root.
Download the latest release from [encounter/objdiff](https://github.com/encounter/objdiff). Under project settings, set `Project directory`. The configuration should be loaded automatically.
Select an object from the left sidebar to begin diffing. Changes to the project will rebuild automatically: changes to source files, headers, `configure.py`, `splits.txt` or `symbols.txt`.
![](assets/objdiff.png)
+65
View File
@@ -0,0 +1,65 @@
decomp-toolkit Project Template
===============================
If starting a new GameCube / Wii decompilation project, this repository can be used as a scaffold.
See [decomp-toolkit](https://github.com/encounter/decomp-toolkit) for background on the concept and more information on the tooling used.
Documentation
-------------
- [Dependencies](docs/dependencies.md)
- [Getting Started](docs/getting_started.md)
- [`symbols.txt`](docs/symbols.md)
- [`splits.txt`](docs/splits.md)
General:
- [Common BSS](docs/common_bss.md)
- [`.comment` section](docs/comment_section.md)
References
--------
- [Discord: GC/Wii Decompilation](https://discord.gg/hKx3FJJgrV) (Come to `#dtk` for help!)
- [objdiff](https://github.com/encounter/objdiff) (Local diffing tool)
- [decomp.me](https://decomp.me) (Collaborate on matches)
- [frogress](https://github.com/decompals/frogress) (Decompilation progress API)
- [wibo](https://github.com/decompals/wibo) (Minimal Win32 wrapper for Linux)
- [sjiswrap](https://github.com/encounter/sjiswrap) (UTF-8 to Shift JIS wrapper)
Projects using this structure:
- [zeldaret/tww](https://github.com/zeldaret/tww)
- [PrimeDecomp/echoes](https://github.com/PrimeDecomp/echoes)
- [DarkRTA/rb3](https://github.com/DarkRTA/rb3)
- [InputEvelution/wp](https://github.com/InputEvelution/wp)
- [Rainchus/ttyd_dtk](https://github.com/Rainchus/ttyd_dtk)
Features
--------
- Few external dependencies: Just `python` for the generator and `ninja` for the build system. See [Dependencies](docs/dependencies.md).
- Simple configuration: Everything lives in `config.yml`, `symbols.txt`, and `splits.txt`.
- Multi-version support: Separate configurations for each game version, and a `configure.py --version` flag to switch between them.
- Feature-rich analyzer: Many time-consuming tasks are automated, allowing you to focus on the decompilation itself. See [Analyzer features](https://github.com/encounter/decomp-toolkit#analyzer-features).
- REL support: RELs each have their own `symbols.txt` and `splits.txt`, and will automatically be built and linked against the main binary.
- No manual assembly: decomp-toolkit handles splitting the DOL into relocatable objects based on the configuration. No game assets are committed to the repository.
- Progress calculation and upload script for [frogress](https://github.com/decompals/frogress).
- Integration with [objdiff](https://github.com/encounter/objdiff) for a diffing workflow.
- (TODO) CI workflow template for GitHub Actions.
Project structure
-----------------
- `configure.py` - Project configuration and generator script.
- `config/[GAMEID]` - Configuration files for each game version.
- `config/[GAMEID]/build.sha1` - SHA-1 hashes for each built artifact, for final verification.
- `build/` - Build artifacts generated by the the build process. Ignored by `.gitignore`.
- `orig/[GAMEID]` - Original game files, extracted from the disc. Ignored by `.gitignore`.
- `orig/[GAMEID]/.gitkeep` - Empty checked-in file to ensure the directory is created on clone.
- `src/` - C/C++ source files.
- `include/` - C/C++ header files.
- `tools/` - Scripts shared between projects.
Temporary, delete when done:
- `config/GAMEID/config.example.yml` - Example configuration file and documentation.
- `docs/` - Documentation for decomp-toolkit configuration.
- `README.md` - This file, replace with your own. For a template, see [`README.example.md`](README.example.md).
Binary file not shown.

After

Width:  |  Height:  |  Size: 12 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 65 KiB

+2
View File
@@ -0,0 +1,2 @@
0123456789abcdef0123456789abcdef01234567 build/GAMEID/main.dol
0123456789abcdef0123456789abcdef01234567 build/GAMEID/module/module.rel
+100
View File
@@ -0,0 +1,100 @@
# Path to the main.dol file.
object: orig/GAMEID/sys/main.dol
# (optional) SHA-1 hash of the main.dol file for verification.
hash: 0123456789abcdef0123456789abcdef01234567
# (optional) Name override. Defaults to "main".
name: main
# (optional) Path to the symbols.txt file.
# This file will be created if it does not exist.
# See docs/symbols.md for more information.
symbols: config/GAMEID/symbols.txt
# (optional) Path to the splits.txt file.
# This file will be created if it does not exist.
# See docs/splits.md for more information.
splits: config/GAMEID/splits.txt
# (optional) Path to the DOL's .map file.
# This should only used for initial analysis, and generating the symbols and splits files.
# Once those files are generated, remove this to avoid conflicts.
map: orig/GAMEID/files/main.MAP
# (optional) Start address of common BSS symbols, if any.
# Useful along with `map`, but not required otherwise, since common BSS
# is marked in the splits file.
common_start: 0x80001234
# (optional) Version used to generate `.comment` sections in the split objects.
# If not specified, no `.comment` sections will be generated.
# See docs/comment_section.md for more information.
mw_comment_version: 8
# (optional) Path to `selfile.sel` for Wii games with RSO files.
selfile: orig/GAMEID/files/selfile.sel
# (optional) SHA-1 hash of the `selfile.sel` file for verification.
selfile_hash: 0123456789abcdef0123456789abcdef01234567
# (optional) When enabled, function boundary analysis will be skipped.
# Only valid _after_ initial analysis has been performed and
# the symbols and splits files have been generated.
quick_analysis: false
# (optional) When enabled, the analyzer will attempt to detect sizes
# and data types of objects based on code usage and alignment.
detect_objects: true
# (optional) When enabled, the analyzer will attempt to detect strings,
# wide strings, and string tables.
detect_strings: true
# (optional) Whether to write disassembly to the split output directory.
# While not used in the build process, the disassembly is useful
# for reading and usage with other tools, like decomp.me.
write_asm: true
# (optional) If symbols are _fully_ known (e.g. from a complete map file),
# this can be set to true to skip most analysis steps, and ensure new
# symbols are not created by the analyzer.
# If you're not sure, leave this false.
symbols_known: false
# (optional) Whether to create `gap_` symbols to prevent the linker from
# adjusting the alignment / address of symbols.
# When alignments are fully known (e.g. from a complete map file),
# this can be set to false.
fill_gaps: true
# (optional) Custom template for `ldscript.lcf`. Avoid unless necessary.
# See https://github.com/encounter/decomp-toolkit/blob/main/assets/ldscript.lcf
ldscript_template: config/GAMEID/module/ldscript.tpl
# (optional) Configuration for modules.
modules:
- # Path to the module.
object: orig/GAMEID/files/module.rel
# (optional) SHA-1 hash of the module for verification.
hash: 0123456789abcdef0123456789abcdef01234567
# (optional) Name of the module. Defaults to the module's filename.
name: module
# (optional) Path to the module's symbols.txt file.
# This file will be created if it does not exist.
# See docs/symbols.md for more information.
symbols: config/GAMEID/module/symbols.txt
# (optional) Path to the module's splits.txt file.
# This file will be created if it does not exist.
# See docs/splits.md for more information.
splits: config/GAMEID/module/splits.txt
# (optional) Path to the module's .map file.
# See `map` above for more information.
map: orig/GAMEID/files/module.MAP
# (optional) Mark symbols as "force active" / "exported".
force_active: []
# (optional) Custom template for `ldscript.lcf`, if needed.
# See https://github.com/encounter/decomp-toolkit/blob/main/assets/ldscript_partial.lcf
ldscript_template: config/GAMEID/module/ldscript.tpl
+12
View File
@@ -0,0 +1,12 @@
# See config.example.yml for documentation.
object: orig/GAMEID/sys/main.dol
hash: 0123456789abcdef0123456789abcdef01234567
symbols: config/GAMEID/symbols.txt
splits: config/GAMEID/splits.txt
mw_comment_version: 8
modules:
- object: orig/GAMEID/files/module.rel
hash: 0123456789abcdef0123456789abcdef01234567
symbols: config/GAMEID/module/symbols.txt
splits: config/GAMEID/module/splits.txt
+1
View File
@@ -0,0 +1 @@
// Intentionally empty. Initial analysis will generate this automatically.
+1
View File
@@ -0,0 +1 @@
// Intentionally empty. Initial analysis will generate this automatically.
+233
View File
@@ -0,0 +1,233 @@
#!/usr/bin/env python3
###
# Generates build files for the project.
# This file also includes the project configuration,
# such as compiler flags and the object matching status.
#
# Usage:
# python3 configure.py
# ninja
#
# Append --help to see available options.
###
import sys
import argparse
from pathlib import Path
from tools.project import (
Object,
ProjectConfig,
calculate_progress,
generate_build,
is_windows,
)
# Game versions
DEFAULT_VERSION = 0
VERSIONS = [
"GAMEID", # 0
]
if len(VERSIONS) > 1:
versions_str = ", ".join(VERSIONS[:-1]) + f" or {VERSIONS[-1]}"
else:
versions_str = VERSIONS[0]
parser = argparse.ArgumentParser()
parser.add_argument(
"mode",
default="configure",
help="configure or progress (default: configure)",
nargs="?",
)
parser.add_argument(
"--version",
dest="version",
default=VERSIONS[DEFAULT_VERSION],
help=f"version to build ({versions_str})",
)
parser.add_argument(
"--build-dir",
dest="build_dir",
type=Path,
default=Path("build"),
help="base build directory (default: build)",
)
parser.add_argument(
"--compilers",
dest="compilers",
type=Path,
help="path to compilers (optional)",
)
parser.add_argument(
"--map",
dest="map",
action="store_true",
help="generate map file(s)",
)
parser.add_argument(
"--debug",
dest="debug",
action="store_true",
help="build with debug info (non-matching)",
)
if not is_windows():
parser.add_argument(
"--wrapper",
dest="wrapper",
type=Path,
help="path to wibo or wine (optional)",
)
parser.add_argument(
"--build-dtk",
dest="build_dtk",
type=Path,
help="path to decomp-toolkit source (optional)",
)
parser.add_argument(
"--sjiswrap",
dest="sjiswrap",
type=Path,
help="path to sjiswrap.exe (optional)",
)
parser.add_argument(
"--verbose",
dest="verbose",
action="store_true",
help="print verbose output",
)
args = parser.parse_args()
config = ProjectConfig()
config.version = args.version.upper()
if config.version not in VERSIONS:
sys.exit(f"Invalid version '{config.version}', expected {versions_str}")
version_num = VERSIONS.index(config.version)
# Apply arguments
config.build_dir = args.build_dir
config.build_dtk_path = args.build_dtk
config.compilers_path = args.compilers
config.debug = args.debug
config.generate_map = args.map
config.sjiswrap_path = args.sjiswrap
if not is_windows():
config.wrapper = args.wrapper
# Tool versions
config.compilers_tag = "1"
config.dtk_tag = "v0.5.5"
config.sjiswrap_tag = "v1.1.1"
config.wibo_tag = "0.6.3"
# Project
config.config_path = Path("config") / config.version / "config.yml"
config.check_sha_path = Path("config") / config.version / "build.sha1"
config.ldflags = [
"-fp hardware",
"-nodefaults",
"-listclosure",
]
# Base flags, common to most GC/Wii games.
# Generally leave untouched, with overrides added below.
cflags_base = [
"-nodefaults",
"-proc gekko",
"-align powerpc",
"-enum int",
"-fp hardware",
"-Cpp_exceptions off",
# "-W all",
"-O4,p",
"-inline auto",
'-pragma "cats off"',
'-pragma "warn_notinlined off"',
"-maxerrors 1",
"-nosyspath",
"-RTTI off",
"-fp_contract on",
"-str reuse",
"-i include",
"-i libc",
"-enc SJIS",
f"-DVERSION={version_num}",
]
# Debug flags
if config.debug:
cflags_base.extend(["-sym on", "-DDEBUG=1"])
else:
cflags_base.append("-DNDEBUG=1")
# Metrowerks library flags
cflags_runtime = [
*cflags_base,
"-use_lmw_stmw on",
"-str reuse,pool,readonly",
"-gccinc",
"-common off",
"-inline auto",
]
# REL flags
cflags_rel = [
*cflags_base,
"-sdata 0",
"-sdata2 0",
]
config.linker_version = "Wii/1.3"
# Helper function for Dolphin libraries
def DolphinLib(lib_name, objects):
return {
"lib": lib_name,
"mw_version": "Wii/1.1",
"cflags": cflags_base,
"host": False,
"objects": objects,
}
# Helper function for REL script objects
def Rel(lib_name, objects):
return {
"lib": lib_name,
"mw_version": "Wii/1.3",
"cflags": cflags_rel,
"host": True,
"objects": objects,
}
Matching = True
NonMatching = False
config.warn_missing_config = True
config.warn_missing_source = False
config.libs = [
{
"lib": "Runtime.PPCEABI.H",
"mw_version": config.linker_version,
"cflags": cflags_runtime,
"host": False,
"objects": [
Object(NonMatching, "Runtime.PPCEABI.H/global_destructor_chain.c"),
Object(NonMatching, "Runtime.PPCEABI.H/__init_cpp_exceptions.cpp"),
],
},
]
if args.mode == "configure":
# Write build.ninja and objdiff.json
generate_build(config)
elif args.mode == "progress":
# Print progress and write progress.json
config.progress_each_module = args.verbose
calculate_progress(config)
else:
sys.exit("Unknown mode: " + args.mode)
+106
View File
@@ -0,0 +1,106 @@
# CodeWarrior `.comment` section
Files built with `mwcc` contain a `.comment` section:
```
$ powerpc-eabi-readelf -We object.o
Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000000 000000 00 0 0 0
[ 1] .text PROGBITS 00000000 000034 000708 00 AX 0 0 4
...
[16] .comment PROGBITS 00000000 00153b 0001b4 01 0 0 1
```
The `.comment` section contains information that `mwld` uses during linking, primarily symbol alignment and a "force active" / export flag.
If missing, `mwld` will **not** adjust the alignment of symbols or remove any unused symbols.
This behavior is quite useful in some cases. When we split our program into objects, we're working from the final post-aligned, post-stripped result, and don't want the linker to make any changes. Most decompilation projects rely on this behavior unintentionally, since their generated objects don't contain a `.comment` section. (For example, objects built with `powerpc-eabi-as`.)
However, we need the `.comment` section for some purposes:
- Reproducing the [common BSS inflation bug](common_bss.md#inflation-bug) requires the `.comment` section present, due to the above. The linker inflates the size of the first common BSS symbol in a TU, but won't actually move any data around unless the `.comment` section is present.
- In newer versions of the linker, using common BSS at all _without_ a valid `.comment` section will cause an internal linker error.
When the `.comment` section is generated, decomp-toolkit will mark all global symbols as "exported" to prevent any deadstripping, since the presence of the `.comment` section itself enables deadstripping.
Generating the `.comment` section and setting the "export" flag is also useful to prevent the linker from removing entire objects. A missing `.comment` section will prevent the removal of unused symbols _inside_ of an object, but the linker will still remove the entire object itself if it thinks it's unused.
## Contents
The contents of this section follow a very simple format:
### Header
`[0x0 size: 0xB]` Magic: `43 6F 64 65 57 61 72 72 69 6F 72` ("CodeWarrior")
`[0xB size: 0x1]` Version(?): `XX`
It's not known whether this field actually affects `mwld` in any way, but it's configurable for completeness sake. (See `mw_comment_version` in [`config.example.yml`](/config/GAMEID/config.example.yml).)
Known values:
- `08` - CodeWarrior for GameCube 1.0+
- `0A` - CodeWarrior for GameCube 1.3.2+
- `0B`, `0C` - CodeWarrior for GameCube 2.7+ (difference unknown)
- `0E`, `0F` - CodeWarrior for GameCube 3.0a3+ (difference unknown)
`[0xC size: 0x4]` Compiler version: `XX XX XX XX`
First 3 bytes are major, minor, and patch version numbers.
4th byte is unknown, but is always `01`.
Example: `Version 2.3.3 build 144` -> `02 03 00 01`
Often the `.exe`'s properties (which `--help` reads from) and the internal version number (here) will differ.
`[0x10 size: 1]` Pool data: `XX`
- `00` - Data pooling disabled
- `01` - Data pooling enabled
`[0x11 size: 1]` Float type: `XX`
- `00` - Floating point disabled
- `01` - Software floating point
- `02` - Hardware floating point
`[0x12 size: 2]` Processor type: `00 16` (Gekko)
`[0x14 size: 1]` Unknown, always `2C`. Possibly the start of symbol entries.
`[0x15 size: 1]` "Quirk" flags: `XX`
Bitfield of miscellaneous flags. Known flags:
- `01` - "Incompatible return small structs"
- `02` - "Incompatible SFPE double params"
- `04` - "Unsafe global reg vars"
`[0x16 size: 22]` Padding until `0x2C`
### Symbol entry
At `0x2C` is the first symbol entry. There is one 8 byte entry per ELF symbol.
This includes the "null" ELF symbol, so the first entry will be all 0's.
`[0x0 size: 4]` Alignment: `XX XX XX XX`
`[0x4 size: 1]` Visibility flags(?): `XX`
Known values:
- `00` - Default
- `0D` - Weak
- `0E` - Unknown, also weak?
`[0x5 size: 1]` Active flags(?): `XX`
Known values:
- `00` - Default
- `08` - Force active / export. Prevents the symbol from being deadstripped.
When applied on a section symbol, the entire section is kept as-is. This is used
by `mwcc` when data pooling is triggered (indicated by a symbol like `...data.0`), likely to prevent the hard-coded section-relative offsets from breaking.
Can also be set using `#pragma force_active on` or `__declspec(export)`.
- `10` - Unknown
- `20` - Unknown
`[0x6 size: 2]` Padding(?): `00 00`
+69
View File
@@ -0,0 +1,69 @@
# Common BSS
When passed the `-common on` flag, `mwcc` will generate global BSS symbols as **common**. The linker deduplicates common symbols with the same name, and allocates an area at the **end** of `.bss` for them.
This is a legacy feature, allowing uninitialized global variables to be defined in headers without linker errors:
```c
// foo.h
int foo;
```
With `-common on`, any TU that includes `foo.h` will define `foo` as a **common** symbol. The linker will deduplicate `foo` across TUs, similar to weak symbols. Common symbols are then generated at the **end** of `.bss`, after all other `.bss` symbols.
With `-common off`, `foo` would be defined as a **global** symbol, and the linker would error out with a duplicate symbol error if `foo.h` was included in multiple TUs.
In `splits.txt`, common BSS can be defined with the `common` attribute:
```
foo.cpp:
.text start:0x80047E5C end:0x8004875C
.ctors start:0x803A54C4 end:0x803A54C8
.data start:0x803B1B40 end:0x803B1B60
.bss start:0x803DF828 end:0x803DFA8C
.bss start:0x8040D4AC end:0x8040D4D8 common
```
As shown above, a file can contain both regular `.bss` and common `.bss`. Marking common `.bss` appropriately is important for determining the final link order.
## Detection
Example from Pikmin 2:
```
00016e60 00000c 805069c0 1 .bss utilityU.a PSMainSide_CreaturePrm.cpp
00016e60 00000c 805069c0 4 @3464 utilityU.a PSMainSide_CreaturePrm.cpp
00016e6c 000048 805069cc 4 saoVVOutput_direction___Q214JStudio_JStage14TAdaptor_light JSystem.a object-light.cpp
00016eb4 0000d0 80506a14 4 saoVVOutput___Q214JStudio_JStage14TAdaptor_actor JSystem.a object-actor.cpp
```
In this example, we see a symbol from `utilityU.a PSMainSide_CreaturePrm.cpp`. We know that this file is very close to the _end_ of the link order. Afterwards, there's a symbol from `JSystem.a object-light.cpp`, which is very close to the _beginning_ of the link order.
A file can't be both at the beginning and end of the link order, so it's a strong indication that `saoVVOutput_direction___Q214JStudio_JStage14TAdaptor_light` marks the beginning of the common BSS section.
One other indication from this example is the lack of a `.bss` section symbol from `JSystem.a object-actor.cpp` and any following files in the link order. Section symbols aren't generated for common BSS.
Without a map, it's harder to tell if there's a common BSS section, but guesses can be made. When looking at XREFs in Ghidra, if a symbol is close to the _end_ of `.bss`, but has XREFs from various addresses close to the _beginning_ of `.text`, it could be an indication of common BSS.
For games built with older versions of the linker, the inflation bug (described below) can also be used to detect common BSS.
## Inflation bug
In older versions of the linker (<= GC 2.6?), when calculating the size of common symbols, the linker will accidentally set the size of the first common symbol in a TU to the size of the _entire_ common section in that TU.
Example from Pikmin 2:
```
# Section Addr | Size | Addr | Alignment | Name | File
00017260 000188 80506dc0 4 mPadList__10JUTGamePad JSystem.a JUTGamePad.cpp
000173e8 000030 80506f48 4 mPadStatus__10JUTGamePad JSystem.a JUTGamePad.cpp
00017418 0000c0 80506f78 4 mPadButton__10JUTGamePad JSystem.a JUTGamePad.cpp
000174d8 000040 80507038 4 mPadMStick__10JUTGamePad JSystem.a JUTGamePad.cpp
00017518 000040 80507078 4 mPadSStick__10JUTGamePad JSystem.a JUTGamePad.cpp
00017558 00000c 805070b8 4 sPatternList__19JUTGamePadLongPress JSystem.a JUTGamePad.cpp
```
In this example, `mPadList__10JUTGamePad` is the first common symbol in the TU, and was inflated to include the size of all other common symbols in the TU. In reality, it's only supposed to be `0xC` bytes, given `0x188 - 0x30 - 0xC0 - 0x40 - 0x40 - 0xC`.
This can be useful to determine if symbols are in the same TU without a map: if a `.bss` symbol is much larger than expected, it could be the first common symbol in a TU. One can subtract the sizes of following symbols to find the true size of the symbol, along with the end of the TU's common symbols.
To reproduce this behavior, the `.comment` section must be present in the object. See [`.comment` section](comment_section.md) for more details.
+33
View File
@@ -0,0 +1,33 @@
# Dependencies
## Windows:
On Windows, it's **highly recommended** to use native tooling. WSL or msys2 are **not** required.
When running under WSL, [objdiff](#diffing) is unable to get filesystem notifications for automatic rebuilds.
- Install [Python](https://www.python.org/downloads/) and add it to `%PATH%`.
- Also available from the [Windows Store](https://apps.microsoft.com/store/detail/python-311/9NRWMJP3717K).
- Download [ninja](https://github.com/ninja-build/ninja/releases) and add it to `%PATH%`.
- Quick install via pip: `pip install ninja`
## macOS:
- Install [ninja](https://github.com/ninja-build/ninja/wiki/Pre-built-Ninja-packages):
```
brew install ninja
```
- Install [wine-crossover](https://github.com/Gcenx/homebrew-wine):
```
brew install --cask --no-quarantine gcenx/wine/wine-crossover
```
After OS upgrades, if macOS complains about `Wine Crossover.app` being unverified, you can unquarantine it using:
```sh
sudo xattr -rd com.apple.quarantine '/Applications/Wine Crossover.app'
```
## Linux:
- Install [ninja](https://github.com/ninja-build/ninja/wiki/Pre-built-Ninja-packages).
- For non-x86(_64) platforms: Install wine from your package manager.
- For x86(_64), [WiBo](https://github.com/decompals/WiBo), a minimal 32-bit Windows binary wrapper, will be automatically downloaded and used.
+124
View File
@@ -0,0 +1,124 @@
# Getting Started
See [Dependencies](dependencies.md) first.
Clone this template repository.
Rename `orig/GAMEID` to the game's ID. (For example, `GLZE01` for _The Legend of Zelda: The Wind Waker_.)
Extract your game to `orig/[GAMEID]`. In Dolphin, use "Extract Entire Disc" for GameCube games, or use "Data Partition" -> "Extract Entire Partition" for Wii games.
Rename `config/GAMEID` to the game's ID and modify `config/[GAMEID]/config.yml` appropriately, using [`config.example.yml`](/config/GAMEID/config.example.yml) as a reference. If the game doesn't use RELs, the `modules` list in `config.yml` can be removed.
Generate a `config/[GAMEID]/build.sha1` file for verification. This file is a list of SHA-1 hashes for each build artifact. One possible way:
```shell
$ dtk shasum orig/[GAMEID]/sys/main.dol orig/[GAMEID]/files/*.rel > config/[GAMEID]/build.sha1
```
Then, modify the paths in `config/[GAMEID]/build.sha1` to point to the `build` directory instead of `orig`. The DOL will be built at `build/[GAMEID]/main.dol`, and modules will be built at `build/[GAMEID]/[module_name]/[module_name].rel`.
Update `VERSIONS` in [`configure.py`](/configure.py) with the game ID.
Run `python configure.py` to generate the initial `build.ninja`.
Run `ninja` to perform initial analysis.
If all goes well, the initial `symbols.txt` and `splits.txt` should be automatically generated. Though it's likely it won't build yet. See [Post-analysis](#post-analysis) for next steps.
## Using a `.map`
If the game has `.map` files matching the DOL (and RELs, if applicable), they can be used to fill out `symbols.txt` and `splits.txt` automatically during the initial analysis.
Add the `map` key to `config.yml`, pointing to the `.map` file from the game disc. (For example, `orig/[GAMEID]/files/main.map`.) For RELs, add a `map` key to each module in `config.yml`.
If the game uses [common BSS](common_bss.md), be sure to set `common_start` as well. (See [`config.example.yml`](/config/GAMEID/config.example.yml).) Otherwise, the final link order may fail to be determined.
Once the initial analysis is completed, `symbols.txt` and `splits.txt` will be generated from the map information. **Remove** the `map` fields from `config.yml` to avoid conflicts.
## Post-analysis
After the initial analysis, `symbols.txt` and `splits.txt` will be generated. These files can be modified to adjust symbols and split points.
If the game uses C++ exceptions, it's required to set up a split for the `__init_cpp_exceptions.cpp` file. This differs between linker versions.
Often indicated by the following error:
```
# runtime sources 'global_destructor_chain.c' and
# '__init_cpp_exceptions.cpp' both need to be updated to latest version.
```
### GC 1.0 - 2.6 linkers:
```yaml
# splits.txt
Runtime.PPCEABI.H/__init_cpp_exceptions.cpp:
.text start:0x803294EC end:0x80329568
.ctors start:0x80338680 end:0x80338684
.dtors start:0x80338820 end:0x80338828
.sdata start:0x803F67F0 end:0x803F67F8
```
`.text`:
Find the following symbols in `symbols.txt`:
```
GetR2__Fv = .text:0x803294EC; // type:function size:0x8 scope:local align:4
__fini_cpp_exceptions = .text:0x803294F4; // type:function size:0x34 scope:global align:4
__init_cpp_exceptions = .text:0x80329528; // type:function size:0x40 scope:global align:4
```
The split end is the address of `__init_cpp_exceptions` + size.
`.ctors`:
Find the address of `__init_cpp_exception_reference` or `_ctors` in symbols.txt.
Always size 4.
`.dtors`:
Look for the address of `__destroy_global_chain_reference` or `_dtors` in symbols.txt.
If `__fini_cpp_exceptions_reference` is present, it's size 8, otherwise size 4
`.sdata`:
Find the following symbol in `symbols.txt`:
```
fragmentID = .sdata:0x803F67F0; // type:object size:0x4 scope:local align:4 data:4byte
```
The split end includes any inter-TU padding, so it's usually size 8.
### GC 2.7+ and Wii linkers:
```yaml
# splits.txt
Runtime.PPCEABI.H/__init_cpp_exceptions.cpp:
.text start:0x80345C34 end:0x80345CA4
.ctors start:0x803A54A0 end:0x803A54A4 rename:.ctors$10
.dtors start:0x803A56A0 end:0x803A56A4 rename:.dtors$10
.dtors start:0x803A56A4 end:0x803A56A8 rename:.dtors$15
.sdata start:0x80418CA8 end:0x80418CB0
```
`.text`:
Find the following symbols in `symbols.txt`:
```
__fini_cpp_exceptions = .text:0x80345C34; // type:function size:0x34 scope:global
__init_cpp_exceptions = .text:0x80345C68; // type:function size:0x3C scope:global
```
The split end is the address of `__init_cpp_exceptions` + size.
`.ctors$10`:
Find the address of `__init_cpp_exception_reference` or `_ctors` in symbols.txt.
Always size 4.
`.dtors$10`:
Look for the address of `__destroy_global_chain_reference` or `_dtors` in symbols.txt.
Always size 4.
`.dtors$15`:
Look for the address of `__fini_cpp_exceptions_reference` in symbols.txt.
Always size 4.
`.sdata`:
Find the following symbol in `symbols.txt`:
```
fragmentID = .sdata:0x80418CA8; // type:object size:0x4 scope:local data:4byte
```
The split end includes any inter-TU padding, so it's usually size 8.
+42
View File
@@ -0,0 +1,42 @@
# `splits.txt`
This file contains file splits for a module.
Example:
```
path/to/file.cpp:
.text start:0x80047E5C end:0x8004875C
.ctors start:0x803A54C4 end:0x803A54C8
.data start:0x803B1B40 end:0x803B1B60
.bss start:0x803DF828 end:0x803DFA8C
.bss start:0x8040D4AC end:0x8040D4D8 common
```
## Format
```
path/to/file.cpp: [file attributes]
section [section attributes]
...
```
- `path/to/file.cpp` The name of the source file, usually relative to `src`. The file does **not** need to exist to start.
This corresponds to an entry in `configure.py` for specifying compiler flags and other options.
### File attributes
- `comment:` Overrides the `mw_comment_version` setting in [`config.yml`](/config/GAMEID/config.example.yml) for this file. See [Comment section](comment_section.md).
`comment:0` is used to disable `.comment` section generation for a file that wasn't compiled with `mwcc`.
Example: `TRK_MINNOW_DOLPHIN/ppc/Export/targsupp.s: comment:0`
This file was assembled and only contains label symbols. Generating a `.comment` section for it will crash `mwld`.
### Section attributes
- `start:` The start address of the section within the file. For DOLs, this is the absolute address (e.g. `0x80001234`). For RELs, this is the section-relative address (e.g. `0x1234`).
- `end:` The end address of the section within the file.
- `align:` Specifies the alignment of the section. If not specified, the default alignment for the section is used.
- `rename:` Writes this section under a different name when generating the split object. Used for `.ctors$10`, etc.
- `common` Only valid for `.bss`. See [Common BSS](common_bss.md).
- `skip` Skips this data when writing the object file. Used for ignoring data that's linker-generated.

Some files were not shown because too many files have changed in this diff Show More