spec by example ruby framework
Post date: Feb 16, 2012 12:2:1 PM
THIS IS A DRAFT;)
This is my first blog post in 2 years, so bare with me. Not sure about google sites as a blogging platform though ...
I am a ruby developer and a part time Agile Coach. I have been evangelizing about User Acceptance Testing aka "Specification by Example" with cucumber since 2008. My biggest project involving cucumber happened inside a betterplace.org btw 2009..2010. We used it to communicate with the Product Owners and run acceptance tests (a 30 minutes build without parallel_tests). I aslo wrote courgette, a Rails engine to browse features. If you want to know more, I recommend you follow @gojkoadzic and buy his book "Specifications by example" or even better see him live at a conference: he is a real performer;)
cucumber features have an high maintenance cost.
I have felt the pain in 3 different projects - I am about to tear into parts my t-shirt with a giant cuke icon - it's time to move on!
It seems that I am not the only one;)
"Where exactly should I put my step definitions? What if they're corelated - can I abstract them in a simple way? Can I reuse them across projects? Could I even test them?
With cucumber, you usually run into this kind of situations and there's no easy way to get over it. You should use better step file namings perhaps? Create some methods that live next to each other in the cucumber World? Nah, it just doesn't feel good."
Josep Jaume Rey, 21st of October 2011
"In the beginning, it was fantastic, the overhead of Cucumber was gone, we were insanely productive. But over time, cracks appeared. As the projects grew larger, the tests became more and more difficult to maintain."
Jonas Nicklas, 17th of November 2011
My 2 main concerns:
readability. In a complex project, you constantly have to switch btw 2-5 screens to be able to read all the step definitions of a given scenario. And I agree with both Josep and Jonas, Regular expressions are a pain and make it even harder to read.
slowness. In scotlandonrails.com 2010, I attended a talk by @josephwilk about scaling cucumber tests. At some point, he suggested running the cukes in parallel on EC2, for a very expensive cost. Cucumber code base is huge, it takes some time to load and run. Parsing plain text features has a cost, even with a C based parser. As a TDD & BDD practitioner, I get depressed if I have to wait for too long when running a test, I loose focus and ... ultimately I have to admit my productivity is affected.
rspec-set or before :all to faster unit tests. There is no equivalent in cuke & co.
Note: I haven't used these libraries in real projects.
Both Spinach and Turnip only run on ruby 1.9.
Both use plain text features a la cucumber, parsing it with Gherkin for Turnip and a gherkin-ruby for Spinach, which unfortunately only supports english at the moment. This is an issue, as an Agile coach living in non-english speaking country (France!), I want to write features in my native tongue.
Turnip has global steps but it is possible to scope them using tags. In Spinach, each feature has its own steps (so no more global steps). That solves the problem of steps madness.
Both have dropped regular expressions! Bravo:) In Turnip, you can however use placeholders in your step definitions, like this:
step "there is a monster called :name" do |name| @monster = Monster.new(name)end
Spinach only supports plain text step but to be honest, that's not a big deal. Scenarios should be written at the right level of abstraction for your customers, in a short and declarative style. Placeholders can encourage your to write scenarios in an imperative style.
I disagree with the design choice of plain text features which need to be parsed in order to be executed as tests. One one hand, as I wrote earlier, it is SLOW. On the the other hand, I can't help thinking it would be so much practical to have it all in 1 place. Having to read multiple step definitions which are not sorted because they are "re-usable" and match them with the scenario is a mental effort which slows me down.
Feature "Define a feature" do
Scenario "Demonstrate how coulda works" do
Given "a pending prereq" do
# Precondition code could go here
It can render features as plain text in Gherkin format which is cool to communicate with customers. It's easy to DRY step definitions are the DSL is pure ruby. Example:
module LoginSteps def given_I_am_logged_in Given "I am logged in as a Guest" do log_as_guest end end
Something feels weird: it is built on top of Test::Unit which is an Unit Testing framework and was never designed to run acceptance tests. The theory of unit tests is that all test_ methods should be independent and can be run in any order (there is more of course). By contrast, an acceptance scenario is a sequence of steps which must be run in order.
Compare to Spinach top notch logo, website and documentation, coulda feels old school. It is just an impression, its codebase is small and easy to grasp but it does not seem to be an active project. Spinach runs scenarios with its own runner, which makes a lot of sense to me. It is supported by codegram amazing team and the code is lovely ;) Disclaimer: I have met only once with these guys, in euruko 2011 but I don't own them anything!
Turnip is an extension for RSpec. It also makes me feel awkward. On one hand, RSpec is not the fastest test framework, it has too much magic. Check this blog post, http://blog.rawonrails.com/2012/01/very-cursory-test-of-rspec-28-speed.html to see what I mean.
Don't get me wrong, I have been using RSpec since the beginning of 2007 and I am not sure still if I'll try minitest/spec.
On the other hand, look at the code base, it is not all tighted to RSpec and could be perfectly independent of RSpec and used with Test::Unit or minitest. A final note, Jonas and elabs are very active in the testing ruby community (Thank you for capybara;) No doubt turnip will be an active project.
Let's brainstorm and work together towards an elegant solution;)
This will take time, probably most of 2012, but I would love if the ruby cuke community works together on this.
I am trying to do my bit with this blog post. I have done more: I have written some code!
As far as I am concerned, the big win is the time saved when writing feature text: you don't have to synch the ./features/my_cool.feature & steps/my_cool.rb files. It's very easy to generate automatically the plain text feature using a Spinach reporter.
A final note, solving Spinach mono linguist short sighting will be easy too. No Gherkin parser is needing, we can tweak the Spinach DSL to add language and write a simple Spinach reporter based on https://github.com/cucumber/gherkin/blob/master/lib/gherkin/i18n.yml