Get help with testing, discuss unit testing strategies etc.


Post by nickolay »

Edit: Oh well, looks like adding/removing the title will reset the checked options in the GUI. That makes sense. So the thrown exception mystery is solved for me.
Ah, I see now. Probably the title is used as a cookie name.

Post by paulb »

Now that this misunderstanding is out of the way. What about the 'Subtest 3' being launched after the main test is exited?

Post by nickolay »

For me its not launched - please check that you use the top-level test instance for the `exit` call?

Post by paulb »

You sure you have the right test case? 'Subtest 3' is not launched in the first one but in the second (it slightly differs because it is using chain and waitFor.
StartTest(function (t) {
	t.describe('Test with multiple steps', function(st) {
		st.it('Subtest 1', function(sst) {
			sst.is(1, 1);
		});

		st.it('Subtest 2', function(sst) {
			sst.chain({
				waitFor: 10
			}, function(next) {
				sst.is(2, 'mustFail');
				t.exit('failed');
				sst.is(3, 3);
				next();
			}, {
				waitFor: 10
			});
		});

		st.it('Subtest 3', function(sst) {
			sst.chain(function(next) {
				sst.is(4, 4);
				sst.is(5, 5);
				next();
			}, {
				waitFor: 10
			}, function() {
				sst.is(6, 6);
			});
		});
	});
});

Post by nickolay »

Ah, indeed it is launched, checking now.

Post by nickolay »

Pushed the fix to the sources, please verify in the tomorrow nightly build.

Post by paulb »

Thanks it is working now as expected.

Now back to my goal of a config to stop a test in a project once it reports a failure. My test example is the following:
StartTest(function (t) {
	t.describe('Test with multiple steps', function(st) {
		st.it('Subtest 1', function(sst) {
			sst.is(1, 1);
		});

		st.it('Subtest 2', function(sst) {
			sst.chain(function(next) {
				sst.waitForElementVisible('foo', next);
				sst.is(2, 2);
			}, function(next) {
				sst.waitForElementVisible('foo', next);
				sst.is(3, 3);
			});
		});

		st.it('Subtest 3', function(sst) {
			sst.chain(function(next) {
				sst.is(4, 4);
				next();
			}, {
				waitFor: 10
			}, function() {
				sst.is(5, 5);
			});
		});
	});
});
What I tried is to listen to the testupdate event on the project and exit the main test when a test reports a failure:
project.on('testupdate', function(e, test) {
	if (!test.isFailed()) {
		return;
	}
	
	// get the main test
	while (test.parent) {
		test = test.parent;
	}
	
	// make sure exit is only called once else we get an infinite loop
	if (!test.isExiting) {
		test.isExiting = true;
		test.exit('failed');
	}
});
When I execute the code above I get the following output:
Image

I am facing two problems here: Firstly an uncaught __SIESTA_TEST_EXIT_EXCEPTION__ (this time with Transparent exceptions being unchecked, yes I double checked this time :)) and secondly the waitFor failure message is being obscured with 'Waiting aborted'.

Is there some other event/code I could use to achieve my goal?

Post by nickolay »

The exception appears because the `t.exit()` method is called in the context of project page, which does not have that special exceptions handler. Perhaps it would be better to override the `onFailedAssertion` method of the test class (guide). It is private but should be ok. This is where the `breakOnFail` option of the project is handled.

Actually it seems should be very easy to add this config on test level, let me try.

Post by nickolay »

It went deeper than expected as usually, but anyway pushed the `breakTestOnFail` and `breakSubTestOnFail` config options. The 1st one breaks the top-test, the 2nd - only the current.

Also removed the exception throwing, if the test has already completed the synchronous part and is doing some asynchronous actions. This is probably better usability, but may require some attention - calling `t.exit()` will not exit the currently running function.

Please verify in the tomorrow nightly build.

Post by paulb »

Still not working. :-(

I adjusted the 'Subtest 2' to the following:
st.it('Subtest 2', function(sst) {
	sst.chain(function(next) {
		sst.is(100, 101);
		sst.is(2, 2);
		sst.waitForElementVisible('foo', next);
	}, function(next) {
		sst.waitForElementVisible('foo', next);
		sst.is(3, 3);
	});
});
And this is what I get when running the test with breakSubTestOnFail (using breakTestOnFail produces the same output without 'Subtest 3' being launched):
Image

Post Reply