Another file format design update.

File Format

@@ -186,10 +186,7 @@ hashes and CRCs in an index, with offsets to each file's data.
 A 4 GB file can store of order 10^8 index entries, or ~10^7 plugin entries
 including the index. It might be prudent to allow for some leeway by specifying
 offsets using 64 bit integers, just in case my upper bound is too low, but we're
-talking about a database file over 4 GB big. There's no way anyone is
-downloading that. If the upper bound is surpassed, we'll just have multiple
-files. That's something worth considering earlier too, because we want to avoid
-people having to download things they'll never use.
+talking about a database file over 4 GB big. That's just crazy.
 
 Storing the index at the beginning of the file would require the entire file to
 be rewritten whenever it is updated, so instead it will be stored at the end
@@ -215,3 +212,32 @@ A plugin data entry:
 											of how they appeared in the plugin.
 	FormIDs				uint32_t[len]		len = (Size - Masters length) / 4
 
+
+However, it turns out that storing the data in the above manner is inefficient.
+Let's say the database file holds the info for 20,000 plugins, which is a very 
+conservative estimate. That means that the file will have to be at least 500 kB
+even without any FormID info at all (ie. a totally useless database). Assuming
+each plugin changes 10 records, and has 2 masters with filenames 10 characters
+long, the size is ~1.6 MB.
+
+If instead each plugin's data is stored in a separate file, with different
+entries for different CRCs of that plugin, each index entry can shrink by 8
+bytes, as the hash is no longer required. If a plugin has 5 versions, then
+using the same assumptions as above, the size of each file will be 390 bytes.
+
+However, because we're dealing with lots of small files, HTTP request/response
+and connection time has to be accounted for. Google says that the best average
+for the upload/download speed ratio of most Internet connections is 1:4, and
+that headers are roughly 300 bytes long without cookies. Each file's effective
+size is therefore 1890 bytes. However, this still doesn't take into account
+the connection time, but the increase in this due to accessing several files
+over just one can be mitigated by using a persistent connection. In any case,
+it's unlikely that the connection time for accessing 200 files will be ~1,000
+times longer than for accessing one file. It is therefore more efficient to use
+separate files for the database.
+
+Going with that, the format of the index entries changes from above to:
+	File data CRC32		uint32_t
+	Data offset			uint32_t			From beginning of file.
+	
+The rest of the file structure remains unchanged.