September 19, 2008

GUI Test Automation Considered Harmful

I have been saying this for years, and Alan Page finally posted about it:

GUI Schmooey
"The point is that I think testers, in general, spend too much time trying to automate GUIs. I’m not against automation – just write automation starting at a level below the GUI and move down from there."

The GUI tends to be the most fragile layer of an application. It also tends to be the layer where a lot of automated test infrastructure is built. I don't like this style of "black box automation". It is tempting to automate a regular user's UI experience; and indeed this can be useful. However, in most cases, you are not being as productive as you could be by taking a manual/unscripted approach to your GUI functional testing.

Manual and Exploratory testing seems like a good fit for GUI testing, while lower layers can benefit most from test automation.

12 comments:

Sai said...

Good post. I wrote a post about the same topic of why GUI tests fail a lot at http://developer-in-test.blogspot.com/2008/09/problems-with-gui-automation-testing.html

Regards,
Sai

Joe said...

It's a good point, and I do agree that too much test automaton is solely at the UI level.

Butlet me provide a counter-argument.

For users, the user interface IS the application. If the UI fails, then the application doesn't work.

So at least some testing at the UI level is very reasonable. And UI automation provides a nice way to ensure that the element most important to the user is still functioning.

Corey Goldberg said...

@Joe:

> So at least some testing at the UI
> level is very reasonable.

completely. in fact.. a lot of manual testing should be focused there. my point is in regards to where you spend your time on automation.

Shrini Kulkarni said...

What is your experience in IT automation area? My experience of last 4-5 years in IT outsourced automation, any automation below the skin is simply not feasible due to following reasons

1. Automation folks do not have programming skills - what all they know is tool knowledge say in QTP or Rational Functional Tester or TestPartener

2. Automation teams do not have access to or are not in good colloborative approach with developers. Since testing, development and automation all outsourced (and to different vendors mostly). Automation happens nearly in silo.

What do you do under such circumstances. IT automation scenarion can not simply embrace Non GUI automation

Corey Goldberg said...

@srini,

why would you have tools developers with no programming experience?

whether testing or development is outsourced is irrelevant. The point of the post was dealing with automation and which layers of a system or program you should access for test automation.

-Corey

Shrini Kulkarni said...

>>>why would you have tools developers with no programming experience?

Do I have a choice? If I were an IT manager with some budget and an outsourcing mandate - I just can not ask for more developers and programmers and have the automation going for me. Just talk to your friends who are in Banks, Insurance companies and other large IT organizations. Ask them how they do the automation? Especially when it is outsourced.
Unless you are prepared to ignore this HUGE community (huge in terms of spend and number of people doing it), you need to be sensitive about the practice that is too BIG to ignore. Why IT companies approach IT like that -- well, lot it has to do with overall testing practices in IT - understaffed test team, always pressure to work with shrinking budgets. This leads to IT managers to look for cheaper (not necessarily the best) ways of doing testing and hence automation.

When the work goes offshore, things get further complicated.

Large part of problem is about "How IT percives testing and automation".

>>> whether testing or development is outsourced is irrelevant.

I beg to differ with you. It is not. If development is outsourced to a company X and Company Y is doing automation - how do you expect the whole communication to work. More the playes, different org and cultural structures, business imperatives and expectations - Automation really becomes the scapegoat. So people end up in doing what is quick and dirty, make their money and exit.

>>>The point of the post was dealing with automation and which layers of a system or program you should access for test automation.

I agree with the point. But I am debating on the contexts where this is not applicable purely from a non-technical standpoint. What do you suggest here? In IT world, your assertion that GUI automation is harmful - will sound like "offensive" as a huge population of IT folks, IT services companies, tool vendors, IT sales and ,marketing folks, Software Training companies, Many consultants" - live on GUI automation. What do you have for them?

Corey Goldberg said...

@Shrini:

If your tools and automation engineers don't have programming skills, they are in the wrong line of work. Do the developers know how to program? If so, what's the difference? Automation *is* software development.

> live on GUI automation.
> What do you have for them?

All I have is a statement for them: you are using your testing time inefficiently.

Alan said...

Thanks for the callout Corey - I think we're of one mind on this topic.

Michael said...

I entirely concur that automating tests at the GUI is of highly limited usefulness. One could argue that a decent macro or scripting language for certain kinds of complex actions or for setup might be worthwhile, if the goal is to deliver the human tester to a certain point in the application, such that he could begin operating and observing the product from there. But I agree with the Go Below The GUI heuristic.

A couple of questions, though.

Corey: It is tempting to automate a regular user's UI experience...

Why? Does anyone think he knows how to do this?

Regards,

---Michael B.

Corey Goldberg said...

@Michael

>> Corey: It is tempting to
>> automate a regular user's UI
>> experience...

> Why? Does anyone think he
> knows how to do this?

I think most automation developers "think" this is possible... and to some degree it is. You can automate a hypothetical single user's happy path through an application and possibly some cursory error handling.

However, you can't realistically automate all user scenarios, or even a decent percentage of them.

Rads said...

@Corey

80% of the user goes for the hypothetical happy path

only the other 20 % of the user goes for the other 80% of all the user scenarios

So its is worth to automate the happy path as well as a few very common negative paths which is hard to regress/test in the first place like invalid character at some field

But we should have team which does a lot of exploration on the UI and thus can find some scenarios which are worth to be automated

Outsourced Software Testing said...

I agree with you that "The GUI tends to be the most fragile layer of an application". Thanks for sharing the article.