Assertions for asynchronous behaviour

dubstep · September 13, 2011, 1:59pm

Hi all,

In GOOS[1] they use an assertion called assertEventually which samples
the system for a success state until a certain timeout has elapsed. This
allows you to synchronise the tests with asynchronous code.

Do we have an equivalent of that in the Ruby / RSpec world already? I
know capybara has wait_until { } but that’s fairly rudimentary - the
failure message isn’t very helpful. Is there anything else already out
there?

[1] http://www.growing-object-oriented-software.com/

cheers,
Matt

–
Freelance programmer & coach
Author, Search (with Aslak
Hellesy)
Founder, http://relishapp.com
+44(0)7974430184 | http://twitter.com/mattwynne

mwalsh · September 13, 2011, 7:11pm

the_object.should eventually_call(:foo).within(2).seconds
TDDing multithreaded apps. Good times.

Best,
Sidu.

mwalsh · September 13, 2011, 5:33pm

On Tue, Sep 13, 2011 at 6:56 AM, Matt W. [email protected] wrote:

[1] http://www.growing-object-oriented-software.com/

rspec-users mailing list
[email protected]
http://rubyforge.org/mailman/listinfo/rspec-users

the_object.should eventually_call(:foo).within(2).seconds

Sounds like Matt is going to make us this matcher!

mwalsh · September 13, 2011, 8:01pm

Sure. “wait_for” is a method Brian T. and I originally wrote for
use in Selenium tests, then IIRC it made it into the Selenium gem and
now lots of libraries use it (or their own version – I make no patent
claim on polling :-)). The wait_for I remember allowed you to
customize the failure message. Let me go see if it’s on GitHub or
anything…

Ah, here’s one:
https://github.com/pivotal/selenium/blob/master/lib/selenium/wait_for.rb

(Maybe I should put it in Wrong.)

A

On Tue, Sep 13, 2011 at 3:56 AM, Matt W. [email protected] wrote:

–

–
Alex C. - [email protected]
http://alexch.github.com
http://twitter.com/alexch

mwalsh · September 20, 2011, 2:01pm

Thanks for all the ideas. I just rolled my own which expects a block
with an assertion in it:

gist.github.com

https://gist.github.com/mattwynne/1228927

gistfile1.txt

# usage:
# it "should return a result of 5" do
#   eventually { long_running_thing.result.should eq(5) }
# end
module AsyncHelper
  def eventually(:options = {})
    timeout = options[:timeout] || 2
    interval = options[:interval] || 0.1
    time_limit = Time.now + timeout
    loop do

This file has been truncated. show original

Could we put this into RSpec somewhere? I’d rather not dump the source
into The Cucumber Book - it’s too low level. I could put it into it’s
own little gem but that seems like creating clutter in the gemsphere.

WDYT?

On 13 Sep 2011, at 17:41, Alex C. wrote:

(Maybe I should put it in Wrong.)

message isn’t very helpful. Is there anything else already out there?

http://twitter.com/alexch

rspec-users mailing list
[email protected]
http://rubyforge.org/mailman/listinfo/rspec-users

cheers,
Matt

–
Freelance programmer & coach
Author, Search (with Aslak
Hellesy)
Founder, http://relishapp.com
+44(0)7974430184 | http://twitter.com/mattwynne

mwalsh · September 21, 2011, 12:50am

On Tue, Sep 20, 2011 at 4:55 AM, Matt W. [email protected] wrote:

Thanks for all the ideas. I just rolled my own which expects a block with an
assertion in it:

I love the language!

eventually { white.should be_black }

Could we put this into RSpec somewhere?

It’s not actually RSpec-specific. I’ll put it (or probably a hybrid
between your new code and my old code) into Wrong and you can use it
via the wrong rspec adapter.

def two
rand(3)
end

require “rspec”
require “wrong/adapters/rspec”
describe “two” do
it “should eventually be half of four” do
eventually { (two + two).should == 4 }
end
end

should work soonish…

–
Alex C. - [email protected]
http://alexchaffee.com
http://twitter.com/alexch

mwalsh · September 21, 2011, 1:09pm

On 20 Sep 2011, at 23:38, Alex C. wrote:

It’s not actually RSpec-specific. I’ll put it (or probably a hybrid
between your new code and my old code) into Wrong and you can use it
via the wrong rspec adapter.

Thanks!

I know it’s not RSpec specific. I suppose my motivation is I want to
keep the number of tools / gems we have to mention in the book to a
minimum. Since we’re already using RSpec in the book already it made
sense to me if it became part of the RSpec assertion toolkit. I’d have
thought it would be useful to other RSpec users too.

I’ve never used Wrong, only read about it–and I like the idea very
much, I must say. Is there any danger of adverse effects (other than an
extra line the Gemfile) if we have to use the Wrong RSpec adapter in the
book alongside the existing vanilla RSpec assertions?

end
end

should work soonish…

–
Alex C. - [email protected]
http://alexchaffee.com
http://twitter.com/alexch

cheers,
Matt

–
Freelance programmer & coach
Author, Search (with Aslak
Hellesy)
Founder, http://relishapp.com
+44(0)7974430184 | http://twitter.com/mattwynne

mwalsh · September 21, 2011, 11:08pm

On 21 Sep 2011, at 17:46, Alex C. wrote:

or this

eventually { rand(10).should == 0 }

and if, after 2 sec or whatever, it keeps either being false or
raising an ExpectationNotMetError, then the waiter will itself raise
an ExpectationNotMetError.

IOW, should the following pass or should it fail?

eventually { false == true }

To be honest, I just ignored the true / false version in my
implementation because I want a helpful error message when the test
fails, rather than a crude TimeoutError or whatever.

I’m ambivalent about this: I would worry that allowing falsiness to
cause an assertion to be raised is not idiomatic RSpec, but perhaps it
is idiomatic Wrong, since you can do all that magic to infer an error
message anyway, right?

I think as long as I can use my own assertions too I don’t have any
objections to it doing both.

P.S. (Lurkers please feel free to chime in too.

[1] https://github.com/pivotal/selenium/blob/master/lib/selenium/wait_for.rb
[2] eventually helper method for making assertions against asynchronous systems · GitHub

cheers,
Matt

–
Freelance programmer & coach
Author, Search (with Aslak
Hellesy)
Founder, http://relishapp.com
+44(0)7974430184 | http://twitter.com/mattwynne

mwalsh · September 21, 2011, 7:11pm

On Wed, Sep 21, 2011 at 3:42 AM, Matt W. [email protected] wrote:

I’ve never used Wrong, only read about it–and I like the idea very much, I must
say. Is there any danger of adverse effects (other than an extra line the Gemfile)
if we have to use the Wrong RSpec adapter in the book alongside the existing
vanilla RSpec assertions?

There shouldn’t be, but I’d love for more people to verify that. The
adapter source is at

github.com

sconover/wrong/blob/master/lib/wrong/adapters/rspec.rb

require "wrong"
require "wrong/terminal"

if Object.const_defined? :RSpec
  # RSpec 2

  if RSpec.const_defined? :Rails
    require 'rails/version'
    if Rails::VERSION::MAJOR == 3
      # RSpec 2 plus Rails 3
      module RSpec::Rails::TestUnitAssertionAdapter
        included do
          define_assertion_delegators
          class_eval do
            remove_method :assert
          end
        end
      end
    end
  end

This file has been truncated. show original

(It doesn’t actually do much, just includes Wrong inside RSpec’s
ExampleGroup. The bulk of that code is in case anyone uses the Wrong
feature “alias_assert” which allows users to define their own DSLy
name for assert. I like “expect” but RSpec already has its own
“expect”; David kindly provided a way to cleanly remove RSpec’s
“expect” before defining my own, but I couldn’t quite get it to work
so I’m still using the brute force way which just chops it out using
Ruby’s “Module#remove_method” method. In any case, none of that is
relevant unless users go out of their way and call
“Wrong.config.alias_assert :expect, :override=>true” which most won’t
do.)

There’s a semantic issue in your eventually method that I’d like to
discuss. My wait_for[1] and friends take a predicate (in the form of
a block) and wait for it to return true(ish). Your eventually[2]
ignores the return condition, and merely waits for it to not raise an
error. I think using a predicate is more useful, and strictly no less
powerful since the waiter code will also wait for it to not raise an
error, so you could do either this

eventually { rand(10) == 0 }

or this

eventually { rand(10).should == 0 }

and if, after 2 sec or whatever, it keeps either being false or
raising an ExpectationNotMetError, then the waiter will itself raise
an ExpectationNotMetError.

IOW, should the following pass or should it fail?

eventually { false == true }

A

P.S. (Lurkers please feel free to chime in too.

[1]
https://github.com/pivotal/selenium/blob/master/lib/selenium/wait_for.rb
[2] eventually helper method for making assertions against asynchronous systems · GitHub

–
Alex C. - [email protected]
http://alexchaffee.com
http://twitter.com/alexch

mwalsh · September 22, 2011, 3:06am

On Wed, Sep 21, 2011 at 1:01 PM, Matt W. [email protected] wrote:

I want a helpful error message when the test fails, rather than a crude
TimeoutError or whatever.

I hear you.

I’m ambivalent about this: I would worry that allowing falsiness to cause an
assertion to be raised is not idiomatic RSpec, but perhaps it is idiomatic Wrong,
since you can do all that magic to infer an error message anyway, right?

I hope so… I mean, uh, yes! Definitely!

–
Alex C. - [email protected]
http://alexchaffee.com
http://twitter.com/alexch

mwalsh · September 28, 2011, 2:34am

After a week of stealing minutes, I eventually wrote eventually!
Please check this out and give me feedback. I can ship it in a new
Wrong gem as soon as you all tell me it’s ready.

docs:

test (spec):

github.com

alexch/wrong/blob/master/test/eventually_test.rb

# based on
# * https://gist.github.com/1228927
# * https://github.com/pivotal/selenium/blob/master/lib/selenium/wait_for.rb
# see
# http://rubyforge.org/pipermail/rspec-users/2011-September/020575.html

here = File.expand_path(File.dirname(__FILE__))

require "#{here}/test_helper"
require "wrong/assert"
require "wrong/helpers"
require "wrong/d"

require "wrong/eventually"

# todo: move this to a monkey patch file
class Object
  unless Object.method_defined? :singleton_class
    def singleton_class
       class << self

This file has been truncated. show original

code:

github.com

alexch/wrong/blob/master/lib/wrong/eventually.rb

module Wrong
  module Eventually
    NO_BLOCK_PASSED = "you must pass a block to the eventually method"
    
    def eventually options = {}, &block
      raise NO_BLOCK_PASSED if block.nil?
      timeout = options[:timeout] || 5
      delay = options[:delay] || 0.25
      last_error = nil
      begin_time = Time.now      
      while (Time.now - begin_time) < timeout
        begin
          aver(:assert, &block)
          return
        rescue Exception => e
          last_error = e
          sleep delay
        end
      end
      raise last_error

This file has been truncated. show original

The only major feature I haven’t done is editing the error message
from inside the block, since Wrong seems to do a good job of this on
its own. If the block contains a “should” or a Wrong “assert” then it
ends up looking like it just got called and failed the final time.
It’d be nice if I could sneak into an exception and append “(after 5
sec)” to e.message but I don’t know if I want to go there…

–
Alex C. - [email protected]
http://alexchaffee.com
http://twitter.com/alexch

mwalsh · October 13, 2011, 10:15pm

FYI, I’ve just released Wrong 0.6.0 with eventually “as is” – i.e. no
extra exception message fiddling.

I also added a message param to Wrong’s “d” method, e.g.

d("math is hard") { 2 + 2 }

prints

math is hard: (2 + 2) is 4

to the console. Useful for debugging (which is what “d” stands for)
when you don’t want the test flow to stop with an assert or should.

A

–
Alex C. - [email protected]
http://alexchaffee.com
http://twitter.com/alexch

mwalsh · October 3, 2011, 12:03am

On 28 Sep 2011, at 01:09, Alex C. wrote:

code:
wrong/lib/wrong/eventually.rb at master · alexch/wrong · GitHub

The only major feature I haven’t done is editing the error message
from inside the block, since Wrong seems to do a good job of this on
its own. If the block contains a “should” or a Wrong “assert” then it
ends up looking like it just got called and failed the final time.
It’d be nice if I could sneak into an exception and append “(after 5
sec)” to e.message but I don’t know if I want to go there…

I haven’t used it, but this looks good to me.

–
Alex C. - [email protected]
http://alexchaffee.com
http://twitter.com/alexch

rspec-users mailing list
[email protected]
http://rubyforge.org/mailman/listinfo/rspec-users

cheers,
Matt

–
Freelance programmer & coach
Author, Search (with Aslak
Hellesy)
Founder, http://relishapp.com
+44(0)7974430184 | http://twitter.com/mattwynne