Cucumber vs, RSpec

Please forgive the x-post.

I just got back from the Great Lakes Ruby Bash. They had several good
presentations, two specific to BDD and Cucumber. I also talked to
several CEOs and devs afterwards, and the overall takeaway I gathered
was a shift to less RSpec and more Cucumber. Some people even claimed
a 90/10 split (cukes/specs) on current projects.

This was surorising to me and not at all how I worked up to this
point. I was more 20/80. I usually cuked a feature, then spec’ed the
code to make the cuke work. Each release had some new features and
specs for all the underlying code. Apparently, the feeling is that you
should do all your main thrusts with Cucumber and use RSpec for edge
cases. The theory is that you can change out all the underlying code
and the cukes still pass.

What is the communities consensus on this?

Cheers,
Ed

Ed Howland

http://twitter.com/ed_howland

On Tue, Apr 20, 2010 at 2:33 PM, Ed Howland [email protected]
wrote:

code to make the cuke work. Each release had some new features and
specs for all the underlying code. Apparently, the feeling is that you
should do all your main thrusts with Cucumber and use RSpec for edge
cases. The theory is that you can change out all the underlying code
and the cukes still pass.

What is the communities consensus on this?

Hi Ed,

I was also at the GLRB, and was a bit aghast at the claim that you
should have a 90/10 split between cukes and rspec. In my experience,
favoring Cucumber so heavily invites developing code that behaves
correctly, but is messy and difficult to change. I would go so far as
to claim there is a positive correlation between over-reliance on
Cucumber features and rampant violations of the SOLID principles.
Cucumber simply doesn’t excel at enforcing simple, testable contracts
between the objects in your code base the way RSpec does. The result
is that your code is hard to refactor and change, which from the
developer’s point of view is practically the whole reason to maintain
a good test suite in the first place. This isn’t the whole of the
story by any means, but I think it’s close to the place to start.

$0.02
Mike

P.S. Hello RSpec Group! What’s the etiquette here for cross-posting?

On Apr 20, 2010, at 1:57 PM, Mike S. wrote:

point. I was more 20/80. I usually cuked a feature, then spec’ed the

a good test suite in the first place. This isn’t the whole of the
story by any means, but I think it’s close to the place to start.

$0.02
Mike

P.S. Hello RSpec Group! What’s the etiquette here for cross-posting?

Etiquette, schmetiquette :slight_smile:

I’d say, in the interest of keeping the thread in one place, post to the
rspec list w/ a link to this thread in the cuke group and invite folks
to join the convo.

Cheers,
David

With Cucumber, you can’t possibly specify tests for all the corner
cases. That’s what your rspecs (unit tests) are for.

On Tue, Apr 20, 2010 at 2:05 PM, David C. [email protected]
wrote:

a 90/10 split (cukes/specs) on current projects.

between the objects in your code base the way RSpec does. The result
Etiquette, schmetiquette :slight_smile:

I’d say, in the interest of keeping the thread in one place, post to the rspec list w/ a link to this thread in the cuke group and invite folks to join the convo.

I answered this before I realized you were already x-posting :slight_smile:

I think this conversation is relevant to both communities, so let’s
keep it going in both.

On Tue, Apr 20, 2010 at 8:05 PM, David C. [email protected]
wrote:

a 90/10 split (cukes/specs) on current projects.

between the objects in your code base the way RSpec does. The result
is that your code is hard to refactor and change, which from the
developer’s point of view is practically the whole reason to maintain
a good test suite in the first place. This isn’t the whole of the
story by any means, but I think it’s close to the place to start.

First up Rspec and Cucumber are just tools, they can be used in many
ways. So this answer belies my personal usage of these tools.

A big issue for me is scaling tests. Cucumber tests tend to be
end-to-end tests so they cut through the whole application stack. This
is great in terms of freeing you up to refactor the heck out of your
code without having to rewrite lots of tests. But end-to-end tests are
slow, now this can be ok if you working on a small project. In smaller
projects I’ve worked on I’ve only used Cukes. In others I’ve only
tended to drop down to Rspec (which I very much use as a specing/unit
testing tool) when there is complexity, I feel the feedback loop is
not fast enough or I need to explore the design more.

If however you are working on an application thats long lived or lives
in a domain where you’re dealing with asynchronous issues (such as
javascript or evented systems) I’ve seen people very quickly hit 1
hour + test build time. Primarily because they have such a heavy focus
on Cucumber or end-to-end tests. So one direction to help avoid this
is to exploring a few good and bad paths with cucumber but having more
detailed spec coverage. This would help you manage better test build
times.

The other option is to not worry about heavy Cukes usage and throw
lots of hardware at the scaling problem. Ok for some, but it does
end-up costing lots.

One other point is that Cucumber for me is part of a process about
facilitating conversations with non techs. So as a developer its not
always a question of how many Cukes do I think I should have. Its a
question of how much does the stakeholders who I’m writing the
software want. Do they want to edit and write the cukes with us? Will
they go back and reference the cukes in the future?

So in conclusion my split on Cucumbers/Rspecs really depends on the
context of the project. An important factor to think about is scaling
when you only use end-to-end tests.

Joseph W.

+44 (0)7812 816431

Watch Ben M.'s slides and talk at Ruby Conf on outside in development
with Cucumber. It positions rspec and cucumber properly

On Tue, Apr 20, 2010 at 9:12 PM, John G. [email protected]
wrote:

Watch Ben M.'s slides and talk at Ruby Conf on outside in development
with Cucumber. It positions rspec and cucumber properly

I used an analogy at Scotland on Rails that might be helpful. I was
thinking about Rspec and Cucumber in terms of planning a driving route
between London and Edinburgh.

Cucumber would test the output, the value:

“Then I am in Edinburgh”

So while driving, knowing I’m not in Edinburgh yet is not really not
that much help for me. Its my end value but the feedback loop is too
big. I could drive to Edinburgh via the USA and still achieve the
value. I like to think of Rspec as a route planner which is giving me
continuous feedback, helping me improve the design of my route. The
feedback loop is fast and it helps me get to my value.

You checkout the presentation:


Joseph W.

+44 (0)7812 816431

Here’s an idea relevant to this discussion that came up last week when
I did a talk on Cucumber to ScotRUG.

Paul Wilson was describing how he used to use Fit, back in the day,
for testing at different levels in the stack rather than just for end-
to-end tests. The idea was to surface the tests at the same customer-
facing level, but steer them at the appropriate level in the stack,
depending on what was being tested.

So for example in the canonical problem of roles with different access
rights to a system, you could write the nice readable tabulated test
in Cucumber, but instead of running that cuke against the whole stack,
stepping through the login form etc, you could just run it against
whatever class is responsible for managing roles.

In order for this to work, you’d need to write the Cukes in a
declarative style, avoiding too much detail about exactly how you’d
carry out the step. This allows the same step to be run against the
whole stack or a focussed piece of the system instead. I tend to think
now that this is just good practice anyway.

This isn’t something I’ve tried myself, but I imagine it means you’d
still get the benefit of customer-readable tests that validate the
behaviour of the system, but running potentially much faster than if
they were end-to-end tests. I can see a disadvantage that it could
make the test code more complex and confusing. You’d obviously still
need a few end-to-end tests in order to make sure the whole thing
still wired up together.

On a team with good customer-developer trust, I’d probably just use
RSpec for these kind of tests, but it’s interesting to try thinking
about cucumber as The Place where business rules are surfaced.

Anyone tried doing anything like this?

On 20 Apr 2010, at 19:33, Ed Howland wrote:

code to make the cuke work. Each release had some new features and

For more options, visit this group at http://groups.google.com/group/cukes?hl=en
.

cheers,
Matt

+447974 430184

Cucumber features are the best tool I know of for capturing requirements
from my customer. RSpec specs are the best tool I know of for
communicating intent and gauging code quality among the developer team.

I’m not sure how exactly you’re quantifying a 90/10 or 80/20 split. I
would expect that there would be a lot of overlap in coverage. That is,
any given line of code is likely to have some cucumber and some rspec
coverage. Personally I shoot for 100% RSpec coverage, and Cucumber is
based entirely on what my customer wants. If we discuss a new feature
or bug fix and they feel I know exactly what they’re talking about and
don’t need a cucumber test for it, I don’t write the cucumber test.
Cukes are for teasing out requirements, which is most important when
there’s domain complexity that I don’t understand, because I’m a
programmer and not a domain expert. Every line of code I write gets
RSpec coverage. That’s how I personally feel confident that my code
does what I think it does, and also helps me keep my dependencies under
control.

It’s true that you can change out all the underlying code and cucumber
tests still pass. But you should be able to change out a lot of code
and have your specs still pass, as well. If you’re changing the API
then some specs might no longer be valid, or need to be moved, or
whatever. That’s just a part of refactoring. Although to be honest I
think focused specs help me refactor more than other people, because I
take really small steps when I refactor. When most people “refactor”
they tear out a bunch of shit and see if it still works.

Cucumber tests tend to take longer because they’re written at a much
higher level. That requires more setup, and more steps through the
codebase. For that reason, Cucumber isn’t particularly good at giving
me rapid feedback. I want feedback in 10 seconds rather than 10
minutes.

The best mantra I have for using Cucumber & RSpec in harmony is, “RSpec
lets me know my code works right, Cucumber lets me know my code is doing
the right work.”

Pat

For me, cukes acts more as an acceptance test that tells me when to stop
the
development and release a feature as rspec goes down to the internals of
the
app in order to make sure everything works ok.
In this context, I think there is no magic recipe that tells you how
much
cucumber code you should have on your project in contrast with rspec
code.
It all depends on the context you’re working. Complex enterprise
applications require extensive unit testing and there i wouldn’t wanna
miss
rspec.
Im my personal experience, i use cucumber-only for CRUDS and other
down-to-the-basic stuff and whenever i need to develop a challenging
chunk
of code i firstly spec it with rspec and then make sure it works with
the
rest by running the cucumber feature. Sometimes it’s like 500 lines of
rspec
and 30 lines of cucumber. It has been working very well!

Lucas Prim
[email protected]
+55 (48) 9921-9303
+55 (48) 3334-1979

On Wed, Apr 21, 2010 at 7:05 PM, Pat M. [email protected]
wrote:

Cucumber features are the best tool I know of for capturing requirements from my customer. Â RSpec specs are the best tool I know of for communicating intent and gauging code quality among the developer team.

I’m not sure how exactly you’re quantifying a 90/10 or 80/20 split. Â I would expect that there would be a lot of overlap in coverage. Â That is, any given line of code is likely to have some cucumber and some rspec coverage. Â Personally I shoot for 100% RSpec coverage, and Cucumber is based entirely on what my customer wants. Â If we discuss a new feature or bug fix and they feel I know exactly what they’re talking about and don’t need a cucumber test for it, I don’t write the cucumber test. Â Cukes are for teasing out requirements, which is most important when there’s domain complexity that I don’t understand, because I’m a programmer and not a domain expert. Â Every line of code I write gets RSpec coverage. Â That’s how I personally feel confident that my code does what I think it does, and also helps me keep my dependencies under control.

It’s true that you can change out all the underlying code and cucumber tests still pass. Â But you should be able to change out a lot of code and have your specs still pass, as well. Â If you’re changing the API then some specs might no longer be valid, or need to be moved, or whatever. Â That’s just a part of refactoring. Â Although to be honest I think focused specs help me refactor more than other people, because I take really small steps when I refactor. Â When most people “refactor” they tear out a bunch of shit and see if it still works.

Cucumber tests tend to take longer because they’re written at a much higher level. Â That requires more setup, and more steps through the codebase. Â For that reason, Cucumber isn’t particularly good at giving me rapid feedback. Â I want feedback in 10 seconds rather than 10 minutes.

The best mantra I have for using Cucumber & RSpec in harmony is, “RSpec lets me know my code works right, Cucumber lets me know my code is doing the right work.”

I don’t think the percentage splits were meant quantitatively, but
rather as a way of saying that integration/full-stack tests with
Cucumber drastically reduce the need for RSpec or unit tests. I’m
skeptical of such claims myself, and not only because I think Cucumber
and RSpec are tools that target very different needs. I’m sure my own
limitations as a developer have a lot to do with feeling this way, but
leaning hard on integration tests reminds me of the Brian Kernighan
quote about debugging being twice as hard as writing the program in
the first place. Full-stack black box tests make it too easy for me to
write overly-complex code and still have a system that satisfies the
business rules. The idea of writing two sets of steps, one for
integration and one for the domain is very intriguing, and one that I
might try in the future, but I think that sidesteps the issue that
prompted Ed’s post in the first place.

I think your mantra hits the nail on the head, but then I’m a sucker
for a good chiasmus. :slight_smile:

Mike

On Wed, Apr 21, 2010 at 7:05 PM, Pat M.
[email protected]wrote:

test for it, I don’t write the cucumber test. Cukes are for teasing out
just a part of refactoring. Although to be honest I think focused specs
lets me know my code works right, Cucumber lets me know my code is doing the
right work."

+1

+1

BTW, here is the URL for his talk:

http://www.benmabey.com/2009/03/14/slides-from-outside-in-development-with-cucumber/

It is back in the webrat days (pre-capybara). And the World() method
is simpler nowadays.

Cheers,
Ed

Ed Howland

http://twitter.com/ed_howland

I just got back from the Great Lakes Ruby Bash. They had several good
presentations, two specific to BDD and Cucumber. I also talked to
several CEOs and devs afterwards, and the overall takeaway I gathered
was a shift to less RSpec and more Cucumber. Some people even claimed
a 90/10 split (cukes/specs) on current projects.

I spoke at the GLRB, specifically reflecting on my experience with BDD
and
outside-in development over the past few years, but I didn’t attend the
first half of the conference, so I’m not familiar with the other talks
or
comments made in those talks about a 90/10 split, or some of the other
things Ed mentioned in original post.

Ed, if you got any of this impression from me, please contact me off
list.

Below are my thoughts in general…

This was surorising to me and not at all how I worked up to this
point. I was more 20/80. I usually cuked a feature, then spec’ed the
code to make the cuke work. Each release had some new features and
specs for all the underlying code. Apparently, the feeling is that you
should do all your main thrusts with Cucumber and use RSpec for edge
cases.

For me, Cucumber and RSpec fill different roles in my approach to
software
development. It’s not really an either/or proposition. I do want a tool
that
supports high level plain text customer readable language which
describes
the application. I also want a tool to drive behaviour, interaction, and
design of underlying components. These tools have different audiences
and
provide different types of value.

There are times when I will use a Cucumber feature to drive the
behaviour of
the application and not drop down to RSpec. And there are times when I
add
change behaviour at a low-level where I do not expose a new Cucumber
feature. This all depends on the system, what’s currently in-place,
what’s
being changed or added, etc.

In my experience there are several parts of webapps which are extremely
shallow and simplistic. There’s not rich behaviour, interesting
interactions, or really any complexities. It’s just monotonous work
(like
CRUD, writing a basic RSS feed, etc.)

In many cases, after an application has evolved to a certain point a lot
of
new features are simply re-using existing functionality you’ve already
built. It’s just being packaged differently to present information in
another way or to provide downloadable reports, etc. This is another
area
where I may not drop down to RSpec, and I let Cucumber drive this
behaviour.
But I don’t do this for everything. I do this when it makes sense.

I’m not in the business of writing examples just for the sake of writing
examples. They need to provide value. Example-driving code usually help
drive behaviour because they support emergent design, provide
regression,
readability/maintainability. If these things are already handled, then I
need to ask myself why I am writing the example.

This is the case for a lot of vanilla Rails stuff. Rails has made
decisions
for you on the design (ie: how associations are defined), so you don’t
receive that value by example driving an association at a low level. If
you
are driving the feature from the outside-in with Cucumber then Cucumber
will
provide regression to ensure that association exists in the future. And
since a declarative statement like “belongs_to” is not complex, does not
drive design, and adding an example doesn’t offer additional
readability,
why are people still writing specs around the association? I don’t care
how
easy shoulda makes it to write, it doesn’t mean it should be written.

Knowing when to not use a tool is just as important as knowing how to
use
the tool.

One thing I mentioned in my talk @ GLRB was that Cucumber provides
implementation flexibility. This was in relation to my observation that
example-driving code well (with tools like RSpec/etc) requires
experience,
courage, self-reflection, and willingness to change. A lot of folks
don’t
have that experience. They want to do things well, but they’re kind of
guessing as they go. While I love apprenticeship, mentoring, and truly
engaged/collaborative/thought-provoking teams, I realize not everyone is
in
these environments. I want to help folks to get some experience before
making a commitment to a decision, I encourage folks to utilize spikes.
Cucumber happens to act as a wonderful umbrella for folks to spike on
lower
level details so they can get feedback on if what they’re thinking is
moving
them in the right direction.

Spikes provide learning, they help folks obtain some more experience,
and
folks are ultimately able to make better decisions since spikes are
focused
on problems being solved now. It may not be the best decision in the
universe, but making a better decision is usually much much better than
making simply a uninformed decision which can easily lead to commitment.
Spikes aren’t only tied to implementation either, they can be an
exploration
for how you would go about writing a good example to describe and drive
some
behaviour.

My intent for the statement about “Cucumber provides flexibility” wasn’t
to
encourage folks to drop unit-level examples because they have supreme
flexibility with Cucumber. It was to encourage folks to take a step back
and
explore with a spike what they’re attempting to accomplish, so they can
come
back and make a better decision moving forward.

I would love to rid the world of unnecessary examples, brittle examples,
impossible to read/maintain examples. Spikes put folks in a better
position
to make bette decisions in that regard so I feel like my encouragement
and
observations are still accurate and relevant.

And as Joseph said it’s all project, client, and context specific.

My 2 cents. :slight_smile:

Personally SRP violations are making my work hell right now on a 2 year
old
rails system. My first concern when re-factoring is putting test in
place -
there isn’t much coverage when i started.

I’ve been doing that for the past 2 weeks and I’ve found that cucumber
is
doing well for getting higher level coverage of ‘it works’ or ‘it needs
to
work like this’ - but even so - the real guts of the problem is far and
away
from Cucumber and so i’ve been using Cucumber to assist my design by
covering basic behaviors and then writing tests for the places where new
things must live and then moving them in while keeping the cukes
functioning
an extracting responsibilities as the rspec tests require.

It’s quite a job - now testing the views is another thing entirely -
cuke
sorta works for this - but not so much - since i’m in a codebase that
has
degenerated to rhtml bog in many ways - it’s a bit of a trick and i have
no
really good way to ensure coverage or confidence.

Great thread.

My other contribution to this thread is that if your code is tested
primarily via Cucumber, you probably have SRP violations all over the
place. That’s what I’ve noticed in my own code, anyway.

The tricky thing is that SRP violations are not necessarily crippling,
particularly in Rails apps (where SRP violations are pretty much
built-in). They just make your test suite a bit slower, limit reuse,
etc… things that are valuable to us but typically not as valuable as
just shipping.

Pat

Single responsibility Principle

On Tue, Apr 27, 2010 at 11:38 AM, Curtis S. <