Ruby On Rails Blog

Rails: Black Box Testing Complex Models

July 4th, 2009

In this article I will show you how you can perform complete end-to-end testing of very complex models using a method called Black Box Testing. I will demonstrate a solution that scales to hundreds of tests and upwards, without having to write any additional code or fixtures for each one.

The method I am going to show you can in fact be used for testing any model where you have potentially hundreds of different inputs, perform some math, and output an answer. You may have done simple Black Box’s in Math at school, where the teacher asked you to work out what the black box was doing, for example:

“Susie holds a black box in her hand. She puts in some numbers, and writes down the numbers that come out the other end. What is Susie’s black box doing?”

Input 10, something happens, out comes 15.
Input 22, something happens, out comes 27.
Input 31, something happens, out comes 36.

What is Susie’s “something happens” black box doing?

Answer: adding 5.

The example I am going to demonstrate with is a complex shopping cart. Stay with me, I will return to the black-box analogy in a moment.

First, let us look at a basic shopping cart. You have a cart, the products in the cart, sum up the sub total, add shipping and VAT (sales-tax) to get your final sales figure (grand total), and you are done. You would have different functions inside the cart model to calculate different aspects of the cart, maybe something like this:

VAT_RATE=1.15 # 15%
FLAT_RATE_SHIPPING_CHARGE=200

Cart < ActiveRecord::Base
  has_many :products
  belongs_to :customer

  def sub_total
    products.map(&:price).sum
  end

  def vat
    (sub_total * VAT_RATE) - sub_total
  end

  def shipping
    FLAT_RATE_SHIPPING_CHARGE
  end

  def grand_total
    sub_total + vat + shipping
  end
end

We can easily write some basic unit tests that check this is working OK:

class CartTest < ActiveSupport::TestCase

  # Load a valid cart from the fixtures
  def setup
    @valid_cart = carts(:valid)
  end

  def test_sub_total_addition
    assert_equal 2640, @valid_cart.sub_total, "Two products in fixtures should total 2640 pence"
  end

  def test_vat
    assert_equal 396, @valid_cart.vat, "VAT on valid cart in fixtures should be 396 pence"
  end

  def test_shipping
    assert_equal 200, @valid_cart.shipping, "Shipping on valid cart in fixtures should be 200 pence"
  end

  def test_grand_total
    assert_equal 3236, @valid_cart.grand_total, "grand total on valid cart in fixtures should be 32.36 pence (sub_total(2640) + vat(396) + shipping(200))"
  end
end

There we have it. A simple cart and a simple set of tests that prove a cart stored in the fixtures works. (TDD: you would have wrote your tests first, right?)

For a toy-cart, that is reasonably adequate, but what happens when this gets more complex?

Any good E-commerce company will be continually experimenting with different price points, discounts, shipping options, special offers and promotions, in order to find the combination that yield the most profit for the company. When you move beyond the toy-cart and into the real-world cart stage, you will need a set of more in-depth tests.

Here is a larger list of some things that might need to be checked in a real-world cart scenario:

• Customer Personal Discount
• Customer Segmentation Discount (Some types of customers receive different rates/offers)
• Voucher Codes (percentage or fixed amount)
• Flat product discount (5% off this product today!
• Bulk Discount (buy 10 of single/range of product(s) for additional discount
• Some discounts apply to single products, others to the entire order
• Discount order: Some discounts can only be used when others are not / or used in place of others
• Different VAT (sales-tax) rates
• VAT Depends on where you are delivering to, and where the order is being placed
• Some products are VAT-able, others are not
• Shipping Costs: Typically based on customers delivery country and Package weight
• Multiple available shipping options within a country (customer can choose between fast & expensive vs slower and cheap, depending on urgency)
• Free delivery available in some circumstances
• VAT applicable on shipping or not
• Multi-Currency support (the ability to display & charge customers in currencies outside of your own currency)
• Math: When discount percentages, VAT or Currency conversion take your prices into fractions of pennies you need to ensure accuracy

If we are to extend our toy-cart a bit further, you will soon see that the simple tests we wrote before start to become un-maintainable, as we would have to store many many carts in the fixtures (with all associated products, prices, discounts, customers, countries, VAT rates, shipping options, etc).

As you might very well be able imagine, individual unit testing (White box testing) will only take you so far. After you have added a few of those tests to your test suite, you are going to end up with a very big and very complex set of tests, complex and time-consuming fixtures. While some solutions have been developed to help manage fixture complexity (eg, Factory-Girl), the reality is for hundreds of test cases with so many input variables you just exchange a set of unmaintainable fixtures with a set of unmaintainable Factory-Girl syntax.

This is where we come back to our black-box analogy. This time, our black box (the Cart model) does not just have 1 input and one output, we have many inputs and many outputs, that are all interconnected. While we can (and do) test each of our functions in part, we also need to devise a test system that will allow us to test everything together, as well as the parts.

The Solution:

When testing like this, we do know and can control for two things: the numbers we put in, and the answer we expect to get out. This is called “Black Box Testing“.

Using this, we will create a “cart test engine” that we can feed information about the products, their prices, discounts, VAT rate, etc, and also the numbers we expect to come out the other end. Our test engine can then setup each test for us, run it, and check the answers on the other end are correct.

First of all, let us create an Integration test called cart_engine:

$ ./script/generate integration_test cart_engine
      create  test/integration/
      create  test/integration/cart_engine_test.rb
$

We are using an integration test because we are testing not just one individual model (a unit), but a series of inter-connected models within our complex cart.

The next thing we are going to do, is make a new directory called test/cart_engine_tests, this will hold each of our cart engine tests:

$ mkdir ./test/cart_engine_tests
$

Now let us create a test engine for our simple cart in test/integration/cart_engine_test.rb:

require File.dirname(__FILE__) + '/../test_helper'

class CartEngineTest < ActionController::IntegrationTest
  test_files = Dir.glob(File.join(File.dirname(__FILE__), "..", "cart_engine_tests", "*.yml"))

  test_files.each do |test_file|
    test_details = YAML::load(File.open(test_file))

    define_method("test_#{File.split(test_file)[1]}") do
      ensure_test_file_valid(test_details)

      # Setup the products defined in the test
      test_details["products"].each do |product|
        new_product = Product.new(:part_no => product["part_no"], :price => product["price"])
        new_product.save!
      end

      # Perform the test
      cart = Cart.new

      # Add products to cart
      test_details["products"].each do |product|
        cart.products << Product.find_by_part_no(product["part_no"])
      end

      # Test Expected Totals
      assert_equal test_details["expected"]["sub_total"], cart.total
      assert_equal test_details["expected"]["vat"], cart.vat
      assert_equal test_details["expected"]["shipping"], cart.shipping
      assert_equal test_details["expected"]["grand_total"], cart.grand_total
    end
  end #test_files.each

  private
    # Pass the contents of a a test file to this function to ensure that it has
    # all of the required key/values contained within and we can parse it ok.
    def ensure_test_file_valid(test_details)
      assert test_details["products"]
      assert test_details["products"].is_a?(Array)
      test_details["products"].each do |product|
        assert product["part_no"]
        assert product["price"]
        assert product["price"].is_a?(Integer)
      end

      assert test_details["expected"]
      assert test_details["expected"].is_a?(Hash)
      assert_equal 4, test_details["expected"].size

      expected_expected_fields = %w(sub_total vat shipping grand_total)
      expected_expected_fields.each do |expected_field|
        assert test_details["expected"][expected_field]
      end
    end
end

This will read each of the .yml files in our new cart_engine_tests directory, setup the required products (thus; replaces the fixtures), then runs the test, by creating a new cart, adding those products into the cart, and checking the expected totals.

We need some tests to run. Remember to basic cart from earlier? This is what my “test/cart_engine_tests/basic_cart.yml” test file looks like:

# Test a basic cart
---
products:
  -
    part_no: "123-123"
    price: 640
  -
    part_no: "333-322"
    price: 2000
expected:
  sub_total: 2640
  vat: 396
  shipping: 200
  grand_total: 3236

It performs exactly the same testing as before, but within a scalable framework.

This file allows us to control the input figures (e.g., product prices), then ensure that for those input figures we get the expected results.

We can now create other yml test files in our test/cart_engine_tests/ directory when we add additional functionality to the cart, and just make a small amendment to our cart_engine_test.rb file to incorporate the testing of that new functionality.

Final Notes:

1. Don’t forget to document what each of your yml files specifically test (I use # comments at the top of each file)
2. Does this scale? Yes. I can assure you it works much better than the alternatives that i’ve tried!
3. The only downside? When you add a new method to the cart, and you want to check the expected result, you sometimes have to go back across all your yml files to add the new key/value in. This may seem like a pain (it is, sometimes), but be-aware that for that small amount of pain, you end up with some FANTASTIC test coverage. There is of course a way around that (check for existance of a key before attempting to use it), but I prefer the full-coverage approach. (PS: If you wrote full end-to-end integration tests using any other method, you would likely come across the same issue anyway if you were testing everything fully).
4. Need to test interface functionality? You can use an integration test exactly like this, add the products to the db, then use webrat or the standard xUnit controller manipulation to simulate the user adding the products to the cart, going through checkout and testing all the figures on-screen along the way.

Categories: Rails, Testing

Tags: activerecord, black box, interesting ideas, polymorphic, Rails, scalable, tdd, Testing, white box Leave a comment