I’m trying to make sure that my cucumber selenium suite fails if my
Rails app returns any 500 response codes. I’m working on what the best
way to identify this state, but the question I’d like to put to the
list is this: If I want to check for this error condition in an After
or AfterStep, how can I force a failure in the scenario?
i.e. something like the following non-functional snippet:
AfterStep do |scenario|
if foo_error_state_detected?
scenario.fail(“A foo failure has been detected.”)
end
end
I’m trying to make sure that my cucumber selenium suite fails if my Rails
app returns any 500 response codes. I’m working on what the best way to
identify this state, but the question I’d like to put to the list is this:
If I want to check for this error condition in an After or AfterStep, how
can I force a failure in the scenario?
To be honest, I don’t remember what happened to AfterStep. I think we
had it at some point, but can’t see it in the codebase. Did I remove
it? Or was that back in the pre-Cucumber days? Help me remember
here…
i.e. something like the following non-functional snippet:
AfterStep do |scenario|
if foo_error_state_detected?
scenario.fail(“A foo failure has been detected.”)
end
end
Ideas?
I’m not so sure I like the idea of AfterStep - smells like a
workaround for something that belongs elsewhere. Can’t your have
selenium-rc (or a helper method you create around it) raise those
exceptions?
Given /bla/ do
don’t remember the API, but you get the idea
@browser.goto_strict(’/bla’) # raises on 500 errors
end
On May 15, 2009, at 4:25 AM, aslak hellesoy wrote:
To be honest, I don’t remember what happened to AfterStep. I think we
had it at some point, but can’t see it in the codebase. Did I remove
it? Or was that back in the pre-Cucumber days? Help me remember
here…
I’m not sure. I saw a reference for it on the wiki:
[http://wiki.github.com/aslakhellesoy/cucumber/hooks
] and it seemed to fit this problem, but it’s not in the version of
cucumber we’re running. I thought perhaps it was in trunk but it
sounds like that’s not the case.
I’m not so sure I like the idea of AfterStep - smells like a
workaround for something that belongs elsewhere. Can’t your have
selenium-rc (or a helper method you create around it) raise those
exceptions?
Given /bla/ do
don’t remember the API, but you get the idea
@browser.goto_strict(‘/bla’) # raises on 500 errors
end
Yes, I’d prefer to be able to configure selenium-rc to fail on 500s.
The problem I ran into is that selenium client doesn’t appear to be
able to access the HTTP response code. In addition, ajax-y HTTP
requests can get kicked off in our app by lots of different things (a
click here, a mouseover there, etc).
In our suite today, of step matchers will usually fail if a 500 is
raised, due to the user-facing impact (e.g. a div is not updated or a
form does not appear), but it usually takes us a few minutes to
realize that there was a 500 error raised (vs. incorrect javascript,
etc.). I am hoping to short-circuit that investigation time and also
catch the cases where a 500 is triggered and we don’t have a step
matcher that fails.
The approach is a little smelly to me because it adds a little
translucency to what is mostly black-box-testing, but I think it would
be a net positive, at least for our app.
On May 15, 2009, at 10:16 AM, aslak hellesoy wrote:
Can you show me the code you would want to put in AfterStep that would
detect a 500 error?
I’ve written a simple piece of rack middleware (included below) that
will create a file in the event a 500 is raised. I would clear the
file before each scenario and check for it’s existence in the
AfterStep. For example:
Before do
File.delete(‘/tmp/rails.error’) if File.exist?(‘/tmp/rails.error’)
end
AfterStep do |scenario|
if File.exist?(‘/tmp/rails.error’)
scenario.fail(“Failed because app raised a 50x error.”)
end
end
Here’s the rack piece, for the curious:
class RackErrorTouch
def initialize(app, options = {}) @app = app @path = options[:path]
end
def call(env)
rack_response = @app.call(env)
if rack_response.first >= 500 touch #{@path}
end
rack_response
end
end
On May 15, 2009, at 10:16 AM, aslak hellesoy wrote:
Can you show me the code you would want to put in AfterStep that would
detect a 500 error?
I’ve written a simple piece of rack middleware (included below) that will
create a file in the event a 500 is raised. I would clear the file before
each scenario and check for it’s existence in the AfterStep. For example:
Thanks for doing that. I have one more favour to ask: Can you show me
an example of a StepDefinition that would cause the file to be
created?
Then I might be able to suggest an alternative solution with the same
result.
On May 15, 2009, at 3:36 PM, aslak hellesoy wrote:
Thanks for doing that. I have one more favour to ask: Can you show me
an example of a StepDefinition that would cause the file to be
created?
That is a tougher question, because any browser interaction that
causes an HTTP request to the app under test could result in a server
error.
Ajax requests can be triggered by javascript, and in various parts of
our app, they are triggered by clicking on links, submitting forms,
dragging and dropping divs, hovering over a div, waiting for a page
load to complete, as a callback from interacting with a 3rd party web
service, or waiting for a setTimeout to execute.
So one way I could tackle it is to add a step to the end of every
scenario that says “Then the application should not have issued any
responses with 50x status codes”, but that would be a) repetitive, and
b) not fail until all steps had run, making it harder to track down
the step in the test where the failure occurred.
One way to look at may be that I’m trying to enforce an invariant.
Under no circumstances in my test suite is it appropriate for my app
to raise a 50x error. In my Rails integration suite, webrat enforces
this particular invariant for me, but can’t find a good hook in the
selenium test stack to do this, which is why I’m looking at this route.
Any further thoughts on this, Aslak? I’d prefer to go with a solution in
line with the future direction of Cucumber if possible.
Sorry, forgot to get back to you Luke.
I’m convinced. We’ll add an AfterStep hook. Do you want to give a stab
at a patch? Start by writing a feature - see
features/after_block_exceptions.feature for a good example of the
style we like…
On May 19, 2009, at 7:04 PM, aslak hellesoy wrote:
I’m convinced. We’ll add an AfterStep hook. Do you want to give a stab
at a patch? Start by writing a feature - see
features/after_block_exceptions.feature for a good example of the
style we like…
Sounds good. I’ll give it a whirl tonight.
Cheers,
Luke
Thanks for doing that. I have one more favour to ask: Can you
app, they are triggered by clicking on links, submitting forms,
responses with
50x error. In my Rails integration suite, webrat enforces this [email protected] http://www.lukemelia.com/
Not crazy about the way it tracks the current_scenario or the
exception_fails_scenario parameter on Hook#execute_in. Unlike Before and
After hooks, I think it’s preferable for AfterStep to fail the step than to
call scenario.fail!
I agree - I’m not crazy about those small hacks you did, but
completely understand why you did it. I’d probably have done the same
myself. Some refactorings are needed in order to make this cleaner.
They are small enough hacks that I’ll pull it in
On May 19, 2009, at 7:04 PM, aslak hellesoy wrote:
I’m convinced. We’ll add an AfterStep hook. Do you want to give a
stab
at a patch? Start by writing a feature - see
features/after_block_exceptions.feature for a good example of the
style we like…
Sounds good. I’ll give it a whirl tonight.
Here’s my first pass:
Not crazy about the way it tracks the current_scenario or the
exception_fails_scenario parameter on Hook#execute_in. Unlike Before
and After hooks, I think it’s preferable for AfterStep to fail the
step than to call scenario.fail!
I’m convinced. We’ll add an AfterStep hook. Do you want to give a
That would be confusing, since After already takes arguments (tags).
So why not make the API be like
Having less methods makes the protocol more extensible, IMO.
What you suggest would break backwards compatibility.
I actually think that one method makes the protocol less extensible,
since there will always be compatibility issues.
Besides, I actually think Before(*tags), After(*tags) and
AfterStep(*tags) is more readable.
In any case, you don’t have to write this a lot, only a couple of
times per project.
So we’ll stick with the current API
Aslak
This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.
