Merge pull request #149 from yenatch/new-install

reformat install + fix source preprocessing

INSTALL.md

@@ -1,21 +1,112 @@
 # Getting Started
 
-Compiling requires a certain Pokemon Crystal ROM:
+Compiling **pokecrystal.gbc** requires a certain **Pokemon Crystal** rom:
 
 ```
 Pokemon - Crystal Version (UE) (V1.0) [C][!].gbc
 md5: 9f2922b235a5eeb78d65594e82ef5dde
 ```
 
-Save it as `baserom.gbc` in the repository.
+Save it as **baserom.gbc** in the repository.
 
 
-Feel free to ask us on nucleus.kafuka.org #skeetendo if something goes wrong (remember to tell where)!
+Feel free to ask us on **[nucleus.kafuka.org #skeetendo](https://kiwiirc.com/client/irc.nolimitzone.com/?#skeetendo)** if something goes wrong!
 
-Don't know how to use IRC? Try [mibbit](http://chat.mibbit.com/?server=nucleus.kafuka.org&channel=#skeetendo).
+# Windows
+
+If you are on Windows and can't install Linux, **Cygwin** is a great alternative.
+
+## Installing Cygwin
+
+Cygwin provides a virtual Linux environment on Windows systems. Just get **setup.exe**.
+
+**http://cygwin.com/install.html**
+
+During the install:
+
+* Keep the defaults.
+
+* Most mirrors are molasses. Use **http://mirrors.kernel.org**.
 
 
-## Linux
+## Using Cygwin
+
+Launch the **Cygwin terminal**. Maybe you know your way around the Linux terminal, **bash**. If not, a crash course:
+```bash
+# list files in current directory
+ls
+
+# show current directory
+pwd
+
+# change directory
+cd /away/we/go
+```
+
+
+## Getting up and running
+
+We need a couple more things to be able to compile.
+
+If you're feeling lazy, just paste these commands into your terminal.
+
+```bash
+apt-cyg install make git wget python python-setuptools
+easy_install pip
+```
+
+**rgbds** will let you compile Game Boy roms.
+
+```bash
+cd ~
+
+# download rgbds binaries
+wget http://diyhpl.us/~bryan/irc/pokered/rgbds/rgbds.zip
+unzip rgbds.zip
+rm rgbds.zip
+
+# make rgbds accessible for all time
+export PATH=$PATH:`pwd`/rgbds
+echo "export PATH=$PATH" >> ~/.bashrc
+```
+
+Set up the **pokecrystal** repository:
+
+```bash
+git clone https://github.com/kanzure/pokecrystal
+cd pokecrystal
+
+# install python requirements
+pip install -r requirements.txt
+```
+
+## Don't forget baserom.gbc!!
+
+Make sure you downloaded a base rom. Name it **baserom.gbc**.
+
+Now you should be able to build **pokecrystal.gbc** for the first time.
+
+This compiles a new rom from the source code, with any patches filled in from the base rom.
+```bash
+make
+```
+
+This ought to take **between 3 and 15 seconds**, depending on your computer.
+
+If you see `cmp baserom.gbc pokecrystal.gbc` as the last line, the build was successful!
+
+Your first compile processes every source file at once. After that, **only modified source files have to be reprocessed**, so compiling again should be a few seconds faster.
+
+Other **make targets** that may come in handy:
+
+`make clean` deletes any preprocessed source files (.tx), rgbds object files and pokecrystal.gbc, in case something goes wrong.
+
+`make pngs` decompresses any **lz** files in gfx/ and then exports any graphics files to **png**.
+
+`make lzs` does the reverse. This is already part of the build process, so **modified pngs will automatically be converted to 2bpp and lz-compressed** without any additional work.
+
+
+# Linux
 
 ```bash
 sudo apt-get install make gcc bison git python python-setuptools 
@@ -46,36 +137,21 @@ pip install -r requirements.txt
 git config diff.hex.textconv hexdump
 ```
 
-To compile the ROM from ASM source:
+To compile the rom from asm source:
-```
+```bash
-make clean && make
+make
 ```
 
-That will take between 3 and 15 seconds, depending on your computer. If you see
+That will take between 3 and 15 seconds, depending on your computer. If you see `cmp baserom.gbc pokecrystal.gbc` as the last line, the build was successful! Rejoice!
-`cmp baserom.gbc pokecrystal.gbc` as the last line, the build was successful! Rejoice!
 
 
-## Windows
+# Now what?
 
-Set up [GitHub for Windows](http://windows.github.com/) and clone this repository.
+**main.asm** is a good starting point. The structure of the source is laid out here.
 
-If you haven't already, get [Python 2.7](http://www.python.org/ftp/python/2.7.3/python-2.7.3.msi) ([64-bit](http://www.python.org/ftp/python/2.7.3/python-2.7.3.amd64.msi)).
+* **Can't find something?** Anyone can add to the source. There's lots to be uncovered.
 
-Extract the following files from the [RGBDS](https://github.com/downloads/bentley/rgbds/rgbds-0.0.1.zip) package into the repository:
+* **Do your own thing!** The asm source is hack-friendly, and the supplementary scripts in extras/ can be used for other projects.
-`rgbasm.exe`
-`rgbds.exe`
-`rgbfix.exe`
-`rgblink.exe`
 
-Install [make](http://gnuwin32.sourceforge.net/downlinks/make.php) for Windows.
+* We'll be happy to answer any **questions** on **[nucleus.kafuka.org #skeetendo](https://kiwiirc.com/client/irc.nolimitzone.com/?#skeetendo)**.
-
-To compile the ROM from ASM source, run `pokecrystal.bat`.
-
-That will take between 3 and 15 seconds, depending on your computer. If you see
-`FC: no differences encountered`, the build was successful! Rejoice!
-
-Now you may try messing around with `main.asm`, or just do whatever you wanted to.
-
-
-# Contributions are welcome!
 

Makefile

@@ -1,29 +1,24 @@
 .SUFFIXES: .asm .tx .o .gbc .png .2bpp .lz
 
-TEXTFILES = \
+TEXTFILES := $(shell find ./ -type f -name '*.asm' | grep -v pokecrystal.asm | grep -v constants.asm | grep -v gbhw.asm | grep -v hram.asm | grep -v constants | grep -v wram.asm)
-		text/sweethoney.tx \
+TEXTQUEUE :=
-		text/phone/bill.tx \
-		text/phone/elm.tx \
-		text/phone/mom.tx \
-		text/phone/trainers1.tx \
-		text/common.tx \
-		text/common_2.tx \
-		text/common_3.tx \
-		main.tx
 
-PNG_GFX    = $(shell find gfx/ -type f -name '*.png')
+PNG_GFX    := $(shell find gfx/ -type f -name '*.png')
-LZ_GFX     = $(shell find gfx/ -type f -name '*.lz')
+LZ_GFX     := $(shell find gfx/ -type f -name '*.lz')
-TWOBPP_GFX = $(shell find gfx/ -type f -name '*.2bpp')
+TWOBPP_GFX := $(shell find gfx/ -type f -name '*.2bpp')
 
 all: pokecrystal.gbc
 	cmp baserom.gbc $<
 clean:
-	rm -f main.tx pokecrystal.o pokecrystal.gbc ${TEXTFILES}
+	rm -f pokecrystal.o pokecrystal.gbc
-pokecrystal.o: pokecrystal.asm constants.asm wram.asm ${TEXTFILES} lzs
+	@echo 'rm -f $(TEXTFILES:.asm=.tx)'
+	@rm -f $(TEXTFILES:.asm=.tx)
+pokecrystal.o: $(TEXTFILES:.asm=.tx) $(LZ_GFX) $(TWOBPP_GFX)
+	python prequeue.py $(TEXTQUEUE)
 	rgbasm -o pokecrystal.o pokecrystal.asm
-
 .asm.tx:
-	python preprocessor.py < $< > $@
+	$(eval TEXTQUEUE := $(TEXTQUEUE) $<)
+	@rm $@
 
 pokecrystal.gbc: pokecrystal.o
 	rgblink -o $@ $<
@@ -33,6 +28,7 @@ pngs:
 	cd extras && python gfx.py mass-decompress && python gfx.py dump-pngs
 
 lzs: $(LZ_GFX) $(TWOBPP_GFX)
+	@:
 
 gfx/pics/%/front.lz: gfx/pics/%/front.png gfx/pics/%/tiles.2bpp
 	python extras/gfx.py png-to-lz --front $^

README.md

@@ -7,7 +7,7 @@ The source code in this project successfully converts back into a ROM image. All
 
 ## Base ROM
 
-The following ROM is required for compiling:
+The following rom is required for compiling:
 
 ```
 Pokemon - Crystal Version (UE) (V1.0) [C][!].gbc
@@ -17,16 +17,18 @@ md5: 9f2922b235a5eeb78d65594e82ef5dde
 Eventually this will not be necessary.
 
 
-## See also
+## What can I do?
-
-* disassembly of [Pokémon Red](http://bitbucket.org/iimarckus/pokered).
-
-
-## Contributing
-
-* Hang out with us on IRC: `nucleus.kafuka.org #skeetendo` (for example, by
-using [mibbit](http://chat.mibbit.com/)).
 
 * Are we missing something? Make a pull request! Contributions are welcome.
 
-* Tackle some [issues](https://github.com/kanzure/pokecrystal/issues)!
+* Take a look at some of the disasm tools in **extras/**. Most of the scripts are generalized enough to take apart other Game Boy games.
+
+* Tackle some **[issues](https://github.com/kanzure/pokecrystal/issues)**!
+
+
+## See also
+
+* Hang out with us on irc: **[nucleus.kafuka.org #skeetendo](https://kiwiirc.com/client/irc.nolimitzone.com/?#skeetendo)**
+
+* Disassembly of **[Pokémon Red](http://bitbucket.org/iimarckus/pokered)**.
+

preprocessor.py

@@ -310,24 +310,14 @@ def separate_comment(l):
     """
     Separates asm and comments on a single line.
     """
-
-    asm        = ""
-    comment    = None
     in_quotes  = False
-
+    for i in xrange(len(l)):
-    # token either belongs to the line or to the comment
-    for token in l:
-        if comment:
-            comment += token
-        else:
         if not in_quotes:
-                if token == ";":
+            if l[i] == ";":
-                    comment = ";"
+                break
-                    continue
+        if l[i] == "\"":
-            if token == "\"":
             in_quotes = not in_quotes
-            asm += token
+    return l[:i], l[i:] or None
-    return asm, comment
 
 def quote_translator(asm):
     """
@@ -335,63 +325,50 @@ def quote_translator(asm):
     """
 
     # split by quotes
-    asms = asm.split("\"")
+    asms = asm.split('"')
 
     # skip asm that actually does use ASCII in quotes
-    lowasm = asms[0].lower()
+    if "SECTION" in asms[0]\
-
+    or "INCBIN"  in asms[0]\
-    if "section" in lowasm \
+    or "INCLUDE" in asms[0]:
-    or "incbin" in lowasm:
+        return asm
-        sys.stdout.write(asm)
-        return
 
     print_macro = False
     if asms[0].strip() == 'print':
         asms[0] = asms[0].replace('print','db 0,')
         print_macro = True
 
-    output = ""
+    output = ''
     even = False
-    i = 0
     for token in asms:
-        i = i + 1
-
         if even:
             characters = []
             # token is a string to convert to byte values
             while len(token):
                 # read a single UTF-8 codepoint
                 char = token[0]
-                if ord(char) >= 0xFC:
+                if ord(char) < 0xc0:
-                    char = char + token[1:6]
+                    token = token[1:]
-                    token = token[6:]
+                    # certain apostrophe-letter pairs are considered a single character
-                elif ord(char) >= 0xF8:
+                    if char == "'" and token:
-                    char = char + token[1:5]
+                        if token[0] in 'dlmrstv':
-                    token = token[5:]
+                            char += token[0]
-                elif ord(char) >= 0xF0:
+                            token = token[1:]
-                    char = char + token[1:4]
+                elif ord(char) < 0xe0:
-                    token = token[4:]
-                elif ord(char) >= 0xE0:
-                    char = char + token[1:3]
-                    token = token[3:]
-                elif ord(char) >= 0xC0:
                     char = char + token[1:2]
                     token = token[2:]
+                elif ord(char) < 0xf0:
+                    char = char + token[1:3]
+                    token = token[3:]
+                elif ord(char) < 0xf8:
+                    char = char + token[1:4]
+                    token = token[4:]
+                elif ord(char) < 0xfc:
+                    char = char + token[1:5]
+                    token = token[5:]
                 else:
-                    token = token[1:]
+                    char = char + token[1:6]
-
+                    token = token[6:]
-                    # certain apostrophe-letter pairs are only a single byte
-                    if char == "'" and len(token) > 0 and \
-                        (token[0] == "d" or \
-                         token[0] == "l" or \
-                         token[0] == "m" or \
-                         token[0] == "r" or \
-                         token[0] == "s" or \
-                         token[0] == "t" or \
-                         token[0] == "v"):
-                        char = char + token[0]
-                        token = token[1:]
-
                 characters += [char]
 
             if print_macro:
@@ -421,32 +398,26 @@ def quote_translator(asm):
 
             output += ", ".join(["${0:02X}".format(chars[char]) for char in characters])
 
-        # if not even
         else:
-            output += (token)
+            output += token
 
         even = not even
 
-    sys.stdout.write(output)
+    return output
-
-    return
 
 def extract_token(asm):
-    token = asm.split(" ")[0].replace("\t", "").replace("\n", "")
+    return asm.split(" ")[0].strip()
-    return token
 
 def make_macro_table():
-    return dict([(macro.macro_name, macro) for macro in macros])
+    return dict(((macro.macro_name, macro) for macro in macros))
 macro_table = make_macro_table()
 
 def macro_test(asm):
     """
     Returns a matching macro, or None/False.
     """
-
     # macros are determined by the first symbol on the line
     token = extract_token(asm)
-
     # check against all names
     if token in macro_table:
         return (macro_table[token], token)
@@ -600,64 +571,46 @@ def macro_translator(macro, token, line):
 
     sys.stdout.write(output)
 
-def include_file(asm):
-    """This is more reliable than rgbasm/rgbds including files on its own."""
-
-    prefix = asm.split("INCLUDE \"")[0] + '\n'
-    filename = asm.split("\"")[1]
-    suffix = asm.split("\"")[2]
-
-    read_line(prefix)
-
-    lines = open(filename, "r").readlines()
-
-    for line in lines:
-        read_line(line)
-
-    read_line(suffix)
-
 def read_line(l):
     """Preprocesses a given line of asm."""
 
-    # strip and store any comment on this line
+    # strip comments from asm
-    if ";" in l:
     asm, comment = separate_comment(l)
-    else:
-        asm     = l
-        comment = None
 
-    # handle INCLUDE as a special case
+    # export all labels
-    if "INCLUDE \"" in l:
+    if ':' in asm[:asm.find('"')]:
-        include_file(asm)
+    	sys.stdout.write('GLOBAL ' + asm.split(':')[0] + '\n')
+
+    # expect preprocessed .asm files
+    if "INCLUDE" in asm:
+        asm = asm.replace('.asm','.tx')
+        sys.stdout.write(asm)
 
     # ascii string macro preserves the bytes as ascii (skip the translator)
-    elif len(asm) > 6 and "\tascii " in [asm[:7], "\t" + asm[:6]]:
+    elif len(asm) > 6 and "ascii " == asm[:6] or "\tascii " == asm[:7]:
         asm = asm.replace("ascii", "db", 1)
         sys.stdout.write(asm)
 
     # convert text to bytes when a quote appears (not in a comment)
     elif "\"" in asm:
-        quote_translator(asm)
+        sys.stdout.write(quote_translator(asm))
 
     # check against other preprocessor features
     else:
         macro, token = macro_test(asm)
-
         if macro:
             macro_translator(macro, token, asm)
         else:
             sys.stdout.write(asm)
 
-    # show line comment
+    if comment: sys.stdout.write(comment)
-    if comment != None:
-        sys.stdout.write(comment)
 
 def preprocess(lines=None):
     """Main entry point for the preprocessor."""
 
     if not lines:
         # read each line from stdin
-        lines = sys.stdin
+        lines = (sys.stdin.readlines())
     elif not isinstance(lines, list):
         # split up the input into individual lines
         lines = lines.split("\n")

prequeue.py

@@ -0,0 +1,17 @@
+# coding: utf-8
+
+# Starting a new python process to preprocess each source file
+# creates too much overhead. Instead, a list of files to preprocess
+# is fed into a script run from a single process.
+
+import os
+import sys
+import preprocessor
+
+if __name__ == '__main__':
+	for source in sys.argv[1:]:
+		dest = os.path.splitext(source)[0] + '.tx'
+		sys.stdin  = open(source, 'r')
+		sys.stdout = open(dest, 'w')
+		preprocessor.preprocess()
+