I wanted to start looking at alternatives to our current set of cucumber feature tests. At the moment on the web team we’re using using FireWatir and Capybara. So I though I’d take at look at what was available in Node.js. Many people think it’s strange that a .Net shop would use a something written for testing Ruby or even consider something that isn’t from the .Net community. Personally I think it’s a benefit to truly look at something form the outside in. Should it matter what you’re using to drive your end product or what language your using to test it? Not really. So what are the motivations for moving away from Ruby, Capybara and FireWatir? In a word ‘flaky’, we’ve had heaps of issues getting our feature tests, AATs and smoke tests reliable. When it comes to testing, consistency should be king. They should be as solid as your unit tests. If they fail you want to know that for definite you’ve broken something, rather than thinking it’s a problem with the webdriver.
It is with this aim in mind that I started looking at the following.
Cucumber.js is definitely in it’s infancy, there’s lots of stuff missing but there’s enough there to get going.
Zombie.js is a headless browser, it claims to be insanely fast, no complaints here.
First up we got something working with the current implementation of cucumber-js . The progress formatter works fine and the usual “you can implement step definitions for undefined steps” are a real help. Interestingly rather than requiring zombie.js in our step definitions we ended up going down the route of implementing our own DSL inside world.js. We could have used another DSL like capybara to protect us from changing the browser/driver we use. This is currently done with our Ruby implementation, the problem is that we’ve ending up implementing our own hacks to get round the limitations/flakiness of selnium/webdriver and to date we have never ‘just swapped out the driver’ to see what happens when they run against chrome/ie. That said should you be using cucumber tests to test the browser? I don’t think you should. With that in mind we ended up implementing directly against zombie.js from our own DSL.
There are a lot things yet to be implemented in cucmber.js one that gives me great satisfaction is the pretty formatter. Look everything is green! It’s no where near ready for production but you do get a nice pretty formatter.