Tests fail randomly during execution

May 2, 2012 at 12:30 AM

Thanks to White, now we have a big suite of around 900 tests running everyday.

Just thought of putting the problem we are facing out here so someone can help us.

During the test execution, we notice that 12 to 15 tests failing randomly every time. When we re-run the tests it works fine. Considering the no of tests we are running and the amount of time it takes to re-run the failed tests, project can live with this.

The reasons for failure are mainly around "object reference not set to an instance of the object", but we use WaitGet method to get all the controls.

It would be great if anyone else could share their thoughts and experience related to this.

May 2, 2012 at 6:24 AM

I have the same problem. Currently we have just 35 tests, but typically one or two tests fail. Each time, different tests fail.

May 2, 2012 at 7:20 AM

we now this issue too, running on VM's the count of failing tests is much higher.

 

We have 2000 Tests running.

 

In some cases there are problems with members in test classes, which takes content of other tests.

 

Throndorin

May 3, 2012 at 1:17 AM

Thank you for your reply.

If it helps you all, We were able to reduce the no of failures by extending the UIItemContainer class, adding Retry On Null and other methods to wait for a certain period like 200 ms to get the elements.

This hasn't impacted the execution time. Now the problem at hand is, if we integrate the auto tests to the CI process and let it determine the build Light (Red/Green) based on the test results, it is going to be red always :-)

 

 

May 3, 2012 at 6:58 AM

Doesn't White retry with the current implementation?

It would be nice if the developer of White would comment on this. This issue is currently preventing me from using White. As written above, when the tests are not deterministic is is hard to trust them.

May 4, 2012 at 1:54 AM

we are not using the current version of the code. We took the source code around mid 2010 and have customised it as per our needs.

May 5, 2012 at 12:01 PM

I am currently investigating this problem now, and it looks like it is related to that I use Remote Desktop to log into the test server.

During a test run, I will typically minimize the remote desktop, and Windows will therefore stop updating the screen on that host (!!). This is a known issue not really related to White (search for "UIAutomation" and "remote desktop" on Google). Some of the tests will therefore fail because the tests were not able to find a GUI object within the timeout, and some will be able pass.

It would be useful for me, and perhaps for the rest of you, if each of you could do a few experiments here:

* Are you able to reproduce the failing build if you do not minimize (just make sure it is in full focus for the whole session)?

The next question is then how to "fix it". I guess running the tests from the console of the VM without using remote desktop for the console should fix it? Anyone?