Page 1 of 3

[ANSWERED] Exit test on fail in subtest

Posted: Thu May 07, 2020 1:25 pm
by paulb

Hi,
I want to immediately exit a test when any failure in any spawned subtest (describe, it, chain, ...) occurs. I know about the breakOnFail config but this will exit the whole project (i.e. when having multiple test classes the next one is not run).
Is there any config I missed or another option to enforce this behaviour?


Re: Exit test on fail in subtest

Posted: Thu May 07, 2020 5:54 pm
by nickolay

Re: Exit test on fail in subtest

Posted: Thu May 07, 2020 5:56 pm
by nickolay
I don't think we have "breakOnFail" for individual tests, but probably it can be implemented manually, by listening to test update events.

Re: Exit test on fail in subtest

Posted: Mon May 11, 2020 8:08 am
by paulb
So sadly no missed config. Ok, time to dig deeper. I played around with the exit function but always found a constellation that did not meet my requirements. For example this test:
me.test.describe('Test with multiple steps', function(st) {
    st.it('Subtest 1', function(sst) {
        sst.is(1, 1);
    });

    st.it('Subtest 2', function(sst) {
        sst.is(2, 'mustFail');
        me.test.exit('failed');
        sst.is(3, 3);
    });

    st.it('Subtest 3', function(sst) {
        sst.is(4, 4);
        sst.is(5, 5);
    });
});
I expected the test to exit and be done but unfortunately the exit method throws an unhandled __SIESTA_TEST_EXIT_EXCEPTION__ error. Wrapping the exit call in try/catch will produce the desired output but I do not see the point of a framework function that you manually have to try/catch.

The next example is this:
me.test.describe('Test with multiple steps', function(st) {
    st.it('Subtest 1', function(sst) {
        sst.is(1, 1);
    });

    st.it('Subtest 2', function(sst) {
        sst.chain({
            waitFor: 10
        }, function(next) {
            sst.is(2, 'mustFail');
            try { me.test.exit('failed'); } catch (e) {}
            sst.is(3, 3);
            next();
        }, {
            waitFor: 10
        });
    });

    st.it('Subtest 3', function(sst) {
        sst.chain(function(next) {
            sst.is(4, 4);
            sst.is(5, 5);
            next();
        }, {
            waitFor: 10
        }, function() {
            sst.is(6, 6);
        });
    });
});
Again, expected behaviour would be that the test simply stops when calling exit. This time however 'Subtest 3' is started before exiting (without test 6 being evaluated) and a javascript error is thrown: 'Uncaught TypeError: data.next is not a function'. And of course the warning 'Adding assertions after the test has finished' is shown. Note that 'Subtest 3' is not started if you remove the first waitFor in the chain.

Changing 'Subtest 2' gives yet another behaviour:
    st.it('Subtest 2', function(sst) {
        sst.chain({
            waitFor: 10
        }, function(next) {
            sst.waitForElementVisible('foo', next);
            sst.is(2, 'mustFail');
            try { me.test.exit('failed'); } catch (e) {}
            sst.is(3, 3);
        }, {
            waitFor: 10
        });
    });
Now the javascript error 'Uncaught TypeError: data.next is not a function' is not thrown but all the other problems mentioned above still occur.

Re: Exit test on fail in subtest

Posted: Mon May 11, 2020 10:26 am
by nickolay
Well, the only way to stop the execution of the function is to throw an exception (or `return` a value). The exception that is thrown is handled in a special way, that it does not even appear in the console. For example, for this test:
StartTest(function (t) {
    t.describe('Test with multiple steps', function (st) {
        st.it('Subtest 1', function (sst) {
            sst.is(1, 1);
        });

        st.it('Subtest 2', function (sst) {
            sst.is(2, 'mustFail');
            sst.exit('failed');
            sst.is(3, 3);
        });

        st.it('Subtest 3', function (sst) {
            sst.is(4, 4);
            sst.is(5, 5);
        });
    });
})
I have this result:
Siesta generic browser examples - Google Chrome_007.png
Siesta generic browser examples - Google Chrome_007.png (319.07 KiB) Viewed 3731 times
And for this test, with exit using top-level `t` instance, the 3rd `it` is not launched:
StartTest(function (t) {

    t.describe('Test with multiple steps', function (st) {
        st.it('Subtest 1', function (sst) {
            sst.is(1, 1);
        });

        st.it('Subtest 2', function (sst) {
            sst.is(2, 'mustFail');
            t.exit('failed');
            sst.is(3, 3);
        });

        st.it('Subtest 3', function (sst) {
            sst.is(4, 4);
            sst.is(5, 5);
        });
    });
})
Siesta generic browser examples - Google Chrome_008.png
Siesta generic browser examples - Google Chrome_008.png (317.4 KiB) Viewed 3731 times
No exception in console in both cases.

It seems your setup is different from "plain" Siesta, can you describe how exactly and why do you observe the exception for test exit?

Re: Exit test on fail in subtest

Posted: Mon May 11, 2020 11:39 am
by paulb
The 'Subtest 3' is only launched for my second example using chain and waitFor. Could you elaborate on that?

I will look into why the exception is thrown in my case.

Re: Exit test on fail in subtest

Posted: Mon May 11, 2020 1:54 pm
by paulb
Found out why the exception is thrown. You will not believe me until you tried it yourself. Try removing the title config in the configure call of the project...

Re: Exit test on fail in subtest

Posted: Mon May 11, 2020 3:15 pm
by nickolay
Hm.. Just did, looks the same for me? (using the `siesta/examples/browser` suite)

Re: Exit test on fail in subtest

Posted: Mon May 11, 2020 4:26 pm
by paulb
Did you check the transparentExceptions options in the GUI?

Edit: Oh well, looks like adding/removing the title will reset the checked options in the GUI. That makes sense. So the thrown exception mystery is solved for me.

Re: Exit test on fail in subtest

Posted: Mon May 11, 2020 4:46 pm
by nickolay
It makes the difference, yes. However, this is the intention of the "transparent exceptions" mode - to not catch any exceptions, to ease the debugging (otherwise one would have to use "stop on caught exceptions" debugger mode). These are a bit contradicting requirements, I think the ease of debugging is important, so this behavior can't be changed easily.