Table Regression Tests

Bernd Mielke

About This Document

This document describes my guerilla approach to successfully running the table layout regression tests. It should encourage more people to run those tests.

Subject Overview

The table element and its children are one of the major layout components that can create some structure on the screen. They are widely (ab)used by page authors. Every small change in the table code can generate a bunch of bugzilla entries and a lot of seemingly unrelated dupes. With the high frequency of checkins in to the source tree it is difficult and time consuming to track the regression down to a single checkin. The risk of becoming very infamous can be reduced by running the layout regression tests.

Prerequisites

In order to run these tests, you will need to have:

  1. a debug build with a working viewer, an optimized build will not work!
  2. a mozilla tree with the testfiles at %MOZ_SRC%/layout/html/tests,
  3. DISABLE_TESTS should not be defined,
  4. patience and time.

Test Runs

Be prepared to cycle.

How the layout regression tests work

First the URL's to test are red from file_list.txt.

snippet from file_list.txt
file:///s|/mozilla/layout/html/tests/table/dom/appendCells1.html
file:///s|/mozilla/layout/html/tests/table/dom/appendCellsRebuild1.html
file:///s|/mozilla/layout/html/tests/table/dom/appendCol1.html
file:///s|/mozilla/layout/html/tests/table/dom/appendCol2.html
file:///s|/mozilla/layout/html/tests/table/dom/appendColGroup1.html
file:///s|/mozilla/layout/html/tests/table/dom/appendRowsExpand1.html

The regression tests compare the frame tree dumps recorded at two different times. The URL's to test are red from file_list.txt For every file in file_list.txtviewer takes the root frame and dumps the whole tree via nsFrame::DumpRegression.

The frame tree dumps are invoked by the -o switch to the viewer (see commandline arguments ). When -o is specified together with a output directory for instance (-o s:\mozilla\layout\html\tests\table\dom\), the frame trees are dumped to separate files in that directory for all URL's in file_list.txt (see: nsWebCrawler::DumpRegressionData) .

Theire name is generated by replacing the old extension .html or .xml with .rgd. For file:///s|/mozilla/layout/html/tests/table/dom/appendCells1.html a file appendCells1.rgd will be created in the s:\mozilla\layout\html\tests\table\dom\ directory.

There is a special type of frame dumps - the printing regression tests, they are invoked by the -Prt command line argument to the viewer and include first a display then a frame dump taking into account the printer pages. A typical beginning of a dump (*.rgd file) looks like:

<frame va="15022440" type="Viewport(-1)" state="270340" parent="0">
  <view va="47171904">
  </view>
  <stylecontext va="15022232">
    <font serif 240 240 0 />
    <color data="-16777216"/>
    <background data="0 2 3 -1 0 0 "/>
    <spacing data="left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  left: 1[0x1]enum top: 1[0x1]enum right: 1[0x1]enum bottom: 1[0x1]enum  left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  1[0x1]enum 0" />
    <list data="100 100 " />
    <position data="left: Auto top: Auto right: Auto bottom: Auto  Auto  0[0x0]tw  Null  Auto  0[0x0]tw  Null  0 Auto  " />
    <text data="0 0 0 Normal  Normal  0[0x0]tw  Normal  " />
    <textreset data="0 10[0xa]enum  " />
    <display data="0 1 0 0 0 0 0 0 0 0 0 0 " />
    <visibility data="0 1 1.000000" />
    <table data="0 0 4 -1 1 " />
    <tableborder data="1 Null  Null  0 2 " />
    <content data="0 0 0 Null  " />
    <quotes data="0 " />
    <ui data="3 0 0 1 " />
    <uireset data="7 0 4" />
    <xul data="0 0 0 0 0 1    <svg data="0 1.000000 1.000000 0 1.000000" />
  </stylecontext>

The first run of the regression tests is started with the argument baseline and it creates the reference frame dumps.

The baseline log will look like:

Type Manifest File: C:\MOZ_SOUR\MOZILLA\MOZILLA\DIST\WIN32_D.OBJ\BIN\components\xpti.dat
nsNativeComponentLoader: autoregistering begins.
nsNativeComponentLoader: autoregistering succeeded
Going to create the event queue
WEBSHELL+ = 1
Note: verifyreflow is disabled
Note: styleverifytree is disabled
Note: frameverifytree is disabled
+++ file:///s|/mozilla/layout/html/tests/table/dom/appendCells1.html: done loading (11810 msec)
appendCells1.rgd - being written
+++ file:///s|/mozilla/layout/html/tests/table/dom/appendCellsRebuild1.html: done loading (1260 msec)
appendCellsRebuild1.rgd - being written
+++ file:///s|/mozilla/layout/html/tests/table/dom/appendCol1.html: done loading (270 msec)
appendCol1.rgd - being written
+++ file:///s|/mozilla/layout/html/tests/table/dom/appendCol2.html: done loading (220 msec)
appendCol2.rgd - being written
+++ file:///s|/mozilla/layout/html/tests/table/dom/appendColGroup1.html: done loading (390 msec)
appendColGroup1.rgd - being written
+++ file:///s|/mozilla/layout/html/tests/table/dom/appendRowsExpand1.html: done loading (220 msec)
appendRowsExpand1.rgd - being written

The second run with the argument verify creates first the frame dumps in a subdirectory (verify) and compares each frame dump with the corresponding reference frame dump. If they differ in critical points the test fails.

A typical part of the verify log would look like:

Type Manifest File: C:\MOZ_SOUR\MOZILLA\MOZILLA\DIST\WIN32_D.OBJ\BIN\components\xpti.dat
nsNativeComponentLoader: autoregistering begins.
nsNativeComponentLoader: autoregistering succeeded
Going to create the event queue
WEBSHELL+ = 1
Note: verifyreflow is disabled
Note: styleverifytree is disabled
Note: frameverifytree is disabled
+++ file:///s|/mozilla/layout/html/tests/table/dom/appendCells1.html: done loading (2200 msec)
frame bbox mismatch: 0,285,930,435 vs. 0,285,1305,435
Node 1:
  TableOuter(table)(1) 0x10004 0,285,930,435, |null attr|-16777216|left: 0[0x0]tw top: 0[0x0]tw right: 0[0x0]tw bottom: 0[0x0]tw  left: 0[0x0]tw top: 0[0x0]tw right: 0[0x0]tw bottom: 0[0x0]tw  left: 1[0x1]enum top: 1[0x1]enum right: 1[0x1]enum bottom: 1[0x1]enum  left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  1[0x1]enum 0|100 100 |left: Inherit top: Inherit right: Inherit bottom: Inherit  0[0x0]tw  0[0x0]tw  Null  0[0x0]tw  0[0x0]tw  Null  0 Auto  |0 0 0 Normal  Normal  0[0x0]tw  Normal  |0 8 0 0 0 0 0 1 0 0 0 0 |0 0 4 -1 1 |0 0 0 Null
Node 2:
  TableOuter(table)(1) 0x10004 0,285,1305,435, |null attr|-16777216|left: 0[0x0]tw top: 0[0x0]tw right: 0[0x0]tw bottom: 0[0x0]tw  left: 0[0x0]tw top: 0[0x0]tw right: 0[0x0]tw bottom: 0[0x0]tw  left: 1[0x1]enum top: 1[0x1]enum right: 1[0x1]enum bottom: 1[0x1]enum  left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  1[0x1]enum 0|100 100 |left: Inherit top: Inherit right: Inherit bottom: Inherit  0[0x0]tw  0[0x0]tw  Null  0[0x0]tw  0[0x0]tw  Null  0 Auto  |0 0 0 Normal  Normal  0[0x0]tw  Normal  |0 8 0 0 0 0 0 1 0 0 0 0 |0 0 4 -1 1 |0 0 0 Null  
frame bbox mismatch: 0,0,930,435 vs. 0,0,1305,435
Node 1:
  Table(table)(1) 0x10004 0,0,930,435, |null attr|-16777216|left: Null top: 0[0x0]tw right: Null bottom: 0[0x0]tw  left: Null top: Null right: Null bottom: Null  left: 15[0xf]tw top: 15[0xf]tw right: 15[0xf]tw bottom: 15[0xf]tw  left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  1[0x1]enum 0|100 100 |left: Auto top: Auto right: Auto bottom: Auto  Auto  0[0x0]tw  Null  Auto  0[0x0]tw  Null  2 Auto  |0 0 0 Normal  Normal  0[0x0]tw  Normal  |0 8 0 0 0 0 0 0 0 0 0 0 |0 8 4 -1 1 |0 0 0 Null 
Node 2:
  Table(table)(1) 0x10004 0,0,1305,435, |null attr|-16777216|left: Null top: 0[0x0]tw right: Null bottom: 0[0x0]tw  left: Null top: Null right: Null bottom: Null  left: 15[0xf]tw top: 15[0xf]tw right: 15[0xf]tw bottom: 15[0xf]tw  left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  1[0x1]enum 0|100 100 |left: Auto top: Auto right: Auto bottom: Auto  Auto  0[0x0]tw  Null  Auto  0[0x0]tw  Null  2 Auto  |0 0 0 Normal  Normal  0[0x0]tw  Normal  |0 8 0 0 0 0 0 0 0 0 0 0 |0 8 4 -1 1 |0 0 0 Null 
frame bbox mismatch: 45,15,840,405 vs. 45,15,1215,405
Node 1:
  TableColGroup(table)(1) 0x80010004 45,15,840,405, |null attr|-16777216|left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  left: 1[0x1]enum top: 1[0x1]enum right: 1[0x1]enum bottom: 1[0x1]enum  left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  1[0x1]enum 0|100 100 |left: Auto top: Auto right: Auto bottom: Auto  Auto  0[0x0]tw  Null  Auto  0[0x0]tw  Null  0 Auto  |0 0 0 Normal  Normal  0[0x0]tw  Normal  |0 12 0 0 0 0 0 0 0 0 0 0 |0 0 4 -1 1 |0 0 0 Null 
Node 2:
  TableColGroup(table)(1) 0x80010004 45,15,1215,405, |null attr|-16777216|left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  left: 1[0x1]enum top: 1[0x1]enum right: 1[0x1]enum bottom: 1[0x1]enum  left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  1[0x1]enum 0|100 100 |left: Auto top: Auto right: Auto bottom: Auto  Auto  0[0x0]tw  Null  Auto  0[0x0]tw  Null  0 Auto  |0 0 0 Normal  Normal  0[0x0]tw  Normal  |0 12 0 0 0 0 0 0 0 0 0 0 |0 0 4 -1 1 |0 0 0 Null 
frame state mismatch: 0x30010406 vs. 0x30010004
Node 1:
  TableCol(table)(1) 0x30010406 0,0,0,0, |null attr|-16777216|left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  left: 1[0x1]enum top: 1[0x1]enum right: 1[0x1]enum bottom: 1[0x1]enum  left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  1[0x1]enum 0|100 100 |left: Auto top: Auto right: Auto bottom: Auto  Auto  0[0x0]tw  Null  Auto  0[0x0]tw  Null  0 Auto  |0 0 0 Normal  Normal  0[0x0]tw  Normal  |0 11 0 0 0 0 0 0 0 0 0 0 |0 0 4 -1 1 |0 0 0 Null 
Node 2:
  TableCol(table)(1) 0x30010004 870,0,345,405, |null attr|-16777216|left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  left: 1[0x1]enum top: 1[0x1]enum right: 1[0x1]enum bottom: 1[0x1]enum  left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  1[0x1]enum 0|100 100 |left: Auto top: Auto right: Auto bottom: Auto  Auto  0[0x0]tw  Null  Auto  0[0x0]tw  Null  0 Auto  |0 0 0 Normal  Normal  0[0x0]tw  Normal  |0 11 0 0 0 0 0 0 0 0 0 0 |0 0 4 -1 1 |0 0 0 Null  
frame bbox mismatch: 0,0,0,0 vs. 870,0,345,405
Node 1:
  TableCol(table)(1) 0x30010406 0,0,0,0, |null attr|-16777216|left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  left: 1[0x1]enum top: 1[0x1]enum right: 1[0x1]enum bottom: 1[0x1]enum  left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  1[0x1]enum 0|100 100 |left: Auto top: Auto right: Auto bottom: Auto  Auto  0[0x0]tw  Null  Auto  0[0x0]tw  Null  0 Auto  |0 0 0 Normal  Normal  0[0x0]tw  Normal  |0 11 0 0 0 0 0 0 0 0 0 0 |0 0 4 -1 1 |0 0 0 Null 
Node 2:
  TableCol(table)(1) 0x30010004 870,0,345,405, |null attr|-16777216|left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null left: 1[0x1]enum top: 1[0x1]enum right: 1[0x1]enum bottom: 1[0x1]enum  left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  1[0x1]enum 0|100 100 |left: Auto top: Auto right: Auto bottom: Auto  Auto  0[0x0]tw  Null  Auto  0[0x0]tw  Null  0 Auto  |0 0 0 Normal  Normal  0[0x0]tw  Normal  |0 11 0 0 0 0 0 0 0 0 0 0 |0 0 4 -1 1 |0 0 0 Null  
regression test s:\mozilla\layout\html\tests\table\dom\verify\appendCells1.rgd failed
+++ file:///s|/mozilla/layout/html/tests/table/dom/appendCellsRebuild1.html: done loading (390 msec)
044regression test s:\mozilla\layout\html\tests\table\dom\verify\appendCellsRebuild1.rgd passed
+++ file:///s|/mozilla/layout/html/tests/table/dom/appendCol1.html: done loading (280 msec)
regression test s:\mozilla\layout\html\tests\table\dom\verify\appendCol1.rgd passed
+++ file:///s|/mozilla/layout/html/tests/table/dom/appendCol2.html: done loading (220 msec)
regression test s:\mozilla\layout\html\tests\table\dom\verify\appendCol2.rgd passed
+++ file:///s|/mozilla/layout/html/tests/table/dom/appendColGroup1.html: done loading (220 msec)
regression test s:\mozilla\layout\html\tests\table\dom\verify\appendColGroup1.rgd passed

The first regression test has failed and it is marked so.

Adding new regression tests

Once you have checked in the code:

I would like to thank Chris Karnaze for his guidance and fantasai for his language support.

<regression.html>  <Last updated: 2002-06-16 Bernd Mielke