[ANSWERED] Exit test on fail in subtest

Get help with testing, discuss unit testing strategies etc.


User avatar
nickolay
Core Developer
Core Developer
Posts: 3466
Joined: Mon May 16, 2011 10:48 am

Re: Exit test on fail in subtest

Post by nickolay »

Edit: Oh well, looks like adding/removing the title will reset the checked options in the GUI. That makes sense. So the thrown exception mystery is solved for me.
Ah, I see now. Probably the title is used as a cookie name.
We offer training in both Ext JS and our products, read more here.
Read the API documentation

User avatar
paulb
Posts: 24
Joined: Thu Apr 11, 2019 9:47 am

Re: Exit test on fail in subtest

Post by paulb »

Now that this misunderstanding is out of the way. What about the 'Subtest 3' being launched after the main test is exited?

User avatar
nickolay
Core Developer
Core Developer
Posts: 3466
Joined: Mon May 16, 2011 10:48 am

Re: Exit test on fail in subtest

Post by nickolay »

For me its not launched - please check that you use the top-level test instance for the `exit` call?
We offer training in both Ext JS and our products, read more here.
Read the API documentation

User avatar
paulb
Posts: 24
Joined: Thu Apr 11, 2019 9:47 am

Re: Exit test on fail in subtest

Post by paulb »

You sure you have the right test case? 'Subtest 3' is not launched in the first one but in the second (it slightly differs because it is using chain and waitFor.

Code: Select all

StartTest(function (t) {
	t.describe('Test with multiple steps', function(st) {
		st.it('Subtest 1', function(sst) {
			sst.is(1, 1);
		});

		st.it('Subtest 2', function(sst) {
			sst.chain({
				waitFor: 10
			}, function(next) {
				sst.is(2, 'mustFail');
				t.exit('failed');
				sst.is(3, 3);
				next();
			}, {
				waitFor: 10
			});
		});

		st.it('Subtest 3', function(sst) {
			sst.chain(function(next) {
				sst.is(4, 4);
				sst.is(5, 5);
				next();
			}, {
				waitFor: 10
			}, function() {
				sst.is(6, 6);
			});
		});
	});
});

User avatar
nickolay
Core Developer
Core Developer
Posts: 3466
Joined: Mon May 16, 2011 10:48 am

Re: Exit test on fail in subtest

Post by nickolay »

Ah, indeed it is launched, checking now.
We offer training in both Ext JS and our products, read more here.
Read the API documentation

User avatar
nickolay
Core Developer
Core Developer
Posts: 3466
Joined: Mon May 16, 2011 10:48 am

Re: Exit test on fail in subtest

Post by nickolay »

Pushed the fix to the sources, please verify in the tomorrow nightly build.
We offer training in both Ext JS and our products, read more here.
Read the API documentation

User avatar
paulb
Posts: 24
Joined: Thu Apr 11, 2019 9:47 am

Re: Exit test on fail in subtest

Post by paulb »

Thanks it is working now as expected.

Now back to my goal of a config to stop a test in a project once it reports a failure. My test example is the following:

Code: Select all

StartTest(function (t) {
	t.describe('Test with multiple steps', function(st) {
		st.it('Subtest 1', function(sst) {
			sst.is(1, 1);
		});

		st.it('Subtest 2', function(sst) {
			sst.chain(function(next) {
				sst.waitForElementVisible('foo', next);
				sst.is(2, 2);
			}, function(next) {
				sst.waitForElementVisible('foo', next);
				sst.is(3, 3);
			});
		});

		st.it('Subtest 3', function(sst) {
			sst.chain(function(next) {
				sst.is(4, 4);
				next();
			}, {
				waitFor: 10
			}, function() {
				sst.is(5, 5);
			});
		});
	});
});
What I tried is to listen to the testupdate event on the project and exit the main test when a test reports a failure:

Code: Select all

project.on('testupdate', function(e, test) {
	if (!test.isFailed()) {
		return;
	}
	
	// get the main test
	while (test.parent) {
		test = test.parent;
	}
	
	// make sure exit is only called once else we get an infinite loop
	if (!test.isExiting) {
		test.isExiting = true;
		test.exit('failed');
	}
});
When I execute the code above I get the following output:
Image

I am facing two problems here: Firstly an uncaught __SIESTA_TEST_EXIT_EXCEPTION__ (this time with Transparent exceptions being unchecked, yes I double checked this time :)) and secondly the waitFor failure message is being obscured with 'Waiting aborted'.

Is there some other event/code I could use to achieve my goal?

User avatar
nickolay
Core Developer
Core Developer
Posts: 3466
Joined: Mon May 16, 2011 10:48 am

Re: Exit test on fail in subtest

Post by nickolay »

The exception appears because the `t.exit()` method is called in the context of project page, which does not have that special exceptions handler. Perhaps it would be better to override the `onFailedAssertion` method of the test class (guide). It is private but should be ok. This is where the `breakOnFail` option of the project is handled.

Actually it seems should be very easy to add this config on test level, let me try.
We offer training in both Ext JS and our products, read more here.
Read the API documentation

User avatar
nickolay
Core Developer
Core Developer
Posts: 3466
Joined: Mon May 16, 2011 10:48 am

Re: Exit test on fail in subtest

Post by nickolay »

It went deeper than expected as usually, but anyway pushed the `breakTestOnFail` and `breakSubTestOnFail` config options. The 1st one breaks the top-test, the 2nd - only the current.

Also removed the exception throwing, if the test has already completed the synchronous part and is doing some asynchronous actions. This is probably better usability, but may require some attention - calling `t.exit()` will not exit the currently running function.

Please verify in the tomorrow nightly build.
We offer training in both Ext JS and our products, read more here.
Read the API documentation

User avatar
paulb
Posts: 24
Joined: Thu Apr 11, 2019 9:47 am

Re: Exit test on fail in subtest

Post by paulb »

Still not working. :-(

I adjusted the 'Subtest 2' to the following:

Code: Select all

st.it('Subtest 2', function(sst) {
	sst.chain(function(next) {
		sst.is(100, 101);
		sst.is(2, 2);
		sst.waitForElementVisible('foo', next);
	}, function(next) {
		sst.waitForElementVisible('foo', next);
		sst.is(3, 3);
	});
});
And this is what I get when running the test with breakSubTestOnFail (using breakTestOnFail produces the same output without 'Subtest 3' being launched):
Image

Post Reply