Containerless scrolling means that the pan zoom controller applies it's transforms (to compensate for differences between the state of layout the last time we painted and the current state as composited to the screen) to the layers that are scrolled instead of the container layer that contains the layers that scroll.
When running test_animations_omta.html there is a zoom of 1.306122 applied, and the page is scrolled down to 67 screen pixels (before the test starts, not sure why exactly). Gecko scrolls as close to 67 screen pixels as it can: 67/1.306122 = 51.29689 css pixels, which is 3077.813 appunits. Gecko scrolls to 3078 app units. When AsyncCompositionManager::TransformScrollableLayer runs we calculate the scroll position of gecko and the current scroll position that the pan zoom controller is using. Since there are no async pan or zoom operations taking place these should match. However when the gecko scroll position is calculated we get 3078/60*1.306122 = 67.0040586. So it applies a transform of 0.0040586 to the container layer for the transform that test_animations_omta.html is animating off main thread. When test_animations_omta.html reads the transform of this layer it fails because it's expecting 0 and 0.0040586 is outside of it's epsilon for considering it to be close enough.