[ANSWERED] Exit test on fail in subtest

Get help with testing, discuss unit testing strategies etc.


User avatar
paulb
Posts: 24
Joined: Thu Apr 11, 2019 9:47 am

[ANSWERED] Exit test on fail in subtest

Post by paulb »

Hi,
I want to immediately exit a test when any failure in any spawned subtest (describe, it, chain, ...) occurs. I know about the breakOnFail config but this will exit the whole project (i.e. when having multiple test classes the next one is not run).
Is there any config I missed or another option to enforce this behaviour?


User avatar
nickolay
Core Developer
Core Developer
Posts: 3466
Joined: Mon May 16, 2011 10:48 am

Re: Exit test on fail in subtest

Post by nickolay »

We offer training in both Ext JS and our products, read more here.
Read the API documentation

User avatar
nickolay
Core Developer
Core Developer
Posts: 3466
Joined: Mon May 16, 2011 10:48 am

Re: Exit test on fail in subtest

Post by nickolay »

I don't think we have "breakOnFail" for individual tests, but probably it can be implemented manually, by listening to test update events.
We offer training in both Ext JS and our products, read more here.
Read the API documentation

User avatar
paulb
Posts: 24
Joined: Thu Apr 11, 2019 9:47 am

Re: Exit test on fail in subtest

Post by paulb »

So sadly no missed config. Ok, time to dig deeper. I played around with the exit function but always found a constellation that did not meet my requirements. For example this test:

Code: Select all

me.test.describe('Test with multiple steps', function(st) {
    st.it('Subtest 1', function(sst) {
        sst.is(1, 1);
    });

    st.it('Subtest 2', function(sst) {
        sst.is(2, 'mustFail');
        me.test.exit('failed');
        sst.is(3, 3);
    });

    st.it('Subtest 3', function(sst) {
        sst.is(4, 4);
        sst.is(5, 5);
    });
});
I expected the test to exit and be done but unfortunately the exit method throws an unhandled __SIESTA_TEST_EXIT_EXCEPTION__ error. Wrapping the exit call in try/catch will produce the desired output but I do not see the point of a framework function that you manually have to try/catch.

The next example is this:

Code: Select all

me.test.describe('Test with multiple steps', function(st) {
    st.it('Subtest 1', function(sst) {
        sst.is(1, 1);
    });

    st.it('Subtest 2', function(sst) {
        sst.chain({
            waitFor: 10
        }, function(next) {
            sst.is(2, 'mustFail');
            try { me.test.exit('failed'); } catch (e) {}
            sst.is(3, 3);
            next();
        }, {
            waitFor: 10
        });
    });

    st.it('Subtest 3', function(sst) {
        sst.chain(function(next) {
            sst.is(4, 4);
            sst.is(5, 5);
            next();
        }, {
            waitFor: 10
        }, function() {
            sst.is(6, 6);
        });
    });
});
Again, expected behaviour would be that the test simply stops when calling exit. This time however 'Subtest 3' is started before exiting (without test 6 being evaluated) and a javascript error is thrown: 'Uncaught TypeError: data.next is not a function'. And of course the warning 'Adding assertions after the test has finished' is shown. Note that 'Subtest 3' is not started if you remove the first waitFor in the chain.

Changing 'Subtest 2' gives yet another behaviour:

Code: Select all

    st.it('Subtest 2', function(sst) {
        sst.chain({
            waitFor: 10
        }, function(next) {
            sst.waitForElementVisible('foo', next);
            sst.is(2, 'mustFail');
            try { me.test.exit('failed'); } catch (e) {}
            sst.is(3, 3);
        }, {
            waitFor: 10
        });
    });
Now the javascript error 'Uncaught TypeError: data.next is not a function' is not thrown but all the other problems mentioned above still occur.

User avatar
nickolay
Core Developer
Core Developer
Posts: 3466
Joined: Mon May 16, 2011 10:48 am

Re: Exit test on fail in subtest

Post by nickolay »

Well, the only way to stop the execution of the function is to throw an exception (or `return` a value). The exception that is thrown is handled in a special way, that it does not even appear in the console. For example, for this test:

Code: Select all

StartTest(function (t) {
    t.describe('Test with multiple steps', function (st) {
        st.it('Subtest 1', function (sst) {
            sst.is(1, 1);
        });

        st.it('Subtest 2', function (sst) {
            sst.is(2, 'mustFail');
            sst.exit('failed');
            sst.is(3, 3);
        });

        st.it('Subtest 3', function (sst) {
            sst.is(4, 4);
            sst.is(5, 5);
        });
    });
})
I have this result:
Siesta generic browser examples - Google Chrome_007.png
Siesta generic browser examples - Google Chrome_007.png (320.04 KiB) Viewed 1105 times
And for this test, with exit using top-level `t` instance, the 3rd `it` is not launched:

Code: Select all

StartTest(function (t) {

    t.describe('Test with multiple steps', function (st) {
        st.it('Subtest 1', function (sst) {
            sst.is(1, 1);
        });

        st.it('Subtest 2', function (sst) {
            sst.is(2, 'mustFail');
            t.exit('failed');
            sst.is(3, 3);
        });

        st.it('Subtest 3', function (sst) {
            sst.is(4, 4);
            sst.is(5, 5);
        });
    });
})
Siesta generic browser examples - Google Chrome_008.png
Siesta generic browser examples - Google Chrome_008.png (318.46 KiB) Viewed 1105 times
No exception in console in both cases.

It seems your setup is different from "plain" Siesta, can you describe how exactly and why do you observe the exception for test exit?
We offer training in both Ext JS and our products, read more here.
Read the API documentation

User avatar
paulb
Posts: 24
Joined: Thu Apr 11, 2019 9:47 am

Re: Exit test on fail in subtest

Post by paulb »

The 'Subtest 3' is only launched for my second example using chain and waitFor. Could you elaborate on that?

I will look into why the exception is thrown in my case.

User avatar
paulb
Posts: 24
Joined: Thu Apr 11, 2019 9:47 am

Re: Exit test on fail in subtest

Post by paulb »

Found out why the exception is thrown. You will not believe me until you tried it yourself. Try removing the title config in the configure call of the project...

User avatar
nickolay
Core Developer
Core Developer
Posts: 3466
Joined: Mon May 16, 2011 10:48 am

Re: Exit test on fail in subtest

Post by nickolay »

Hm.. Just did, looks the same for me? (using the `siesta/examples/browser` suite)
We offer training in both Ext JS and our products, read more here.
Read the API documentation

User avatar
paulb
Posts: 24
Joined: Thu Apr 11, 2019 9:47 am

Re: Exit test on fail in subtest

Post by paulb »

Did you check the transparentExceptions options in the GUI?

Edit: Oh well, looks like adding/removing the title will reset the checked options in the GUI. That makes sense. So the thrown exception mystery is solved for me.

User avatar
nickolay
Core Developer
Core Developer
Posts: 3466
Joined: Mon May 16, 2011 10:48 am

Re: Exit test on fail in subtest

Post by nickolay »

It makes the difference, yes. However, this is the intention of the "transparent exceptions" mode - to not catch any exceptions, to ease the debugging (otherwise one would have to use "stop on caught exceptions" debugger mode). These are a bit contradicting requirements, I think the ease of debugging is important, so this behavior can't be changed easily.
We offer training in both Ext JS and our products, read more here.
Read the API documentation

Post Reply