Mock the Web Service

By Phlip Plumlee
May 3, 2010 | Comments: 2

This post shows how to write a web service using Test-Driven Development. Our source code example is the exemplary active_merchant contribution to Ruby on Rails. It reveals how developer tests can correctly attack remote web services. Programmers writing clients (or servers) for any kind of web service should use these techniques. My next post will extend this one into the Abstract Test Pattern.

A Big Problem

The great thing about capitalism is we consumers can choose from many different financial institutions, all competing with each other to provide excellent services.

<a beat>

Okay, I'm glad I got that out of my system (and into yours). I'll start again.

The great thing about standards is we can chose from so many. When a program locks itself into one standard, switching to another can be inordinately expensive. The solution is abstraction, abstraction, abstraction. (But not too much abstraction!)

active_merchant is a valuable example of the Abstract Template Pattern in action. For all the diversity of payment gateways out there in e-commerce land, they all perform a common subset of transactions. They all authorize, capture, purchase, and refund online credit cards. And they all support multiple subsets of extra features, such as automated clearing houses, point-of-sale capacities, express & scenic routes, safe account number storage, recurring subscription charges, and whatnot. All these features make life very interesting for all you developers out there, who are just trying to "add a danged shopping cart, already!" to your clients' websites. All that diversity does not necessarily work to your advantage.

To constrain and tame diversity, my best weapon, as usual, is automated test cases.

The Problem with Tests & Webservices

When a developer converts a web service's example code into a test case, one common mistake is allowing that test to "hit the wire" - to actually call that web service over The Cloud. The main problem with this anti-pattern is it works. The web service will instantly return testable results, each time the test case runs. And who doesn't have constant internet connectivity??

A test suite that hits the wire is an anti-pattern for reasons too numerous & obvious to mention. The most self-interested reason is the simplest. That test suite cannot change that service's response, and all its test cases should cover many paths.

When tests can't change a response, they need a mock object.

Collect a Sample

To fix this dastardly anti-pattern, first commit the anti-pattern! Write a test case that carelessly and abusively whacks that web service. (Many web services come with alternate URIs for remote testing, and sample code, so start there. active-merchant supports the test? flag for this reason.)

Once the test works, find the code which posts your message to the service and fetches the response back. Add a trace statement which prints out that response, formatted for easy comprehension:

      def commit(request_body, request_type = nil)
          request = build_request(request_body, request_type)
          headers = build_headers(request.size)
        
          xml = ssl_post(test? ? TEST_URL : LIVE_URL, request, headers)

          print IO.popen('tidy -i -xml -wrap 130', 'w').write(xml) # TODO croak this line!
          
          response = parse(xml)

             build_response(response[:result] == "0", response[:message], response,
               :test => test?,
               :authorization => response[:pn_ref] || response[:rp_ref],
               :cvv_result => CVV_CODE[response[:cv_result]],
               :avs_result => { :code => response[:avs_result] }
           )
        end

That calls tidy, to ensure the output won't be painful to look at & work with. The argument -i -xml indents the response message as XML, and the argument -wrap 130 makes the wild assumption you are not programming on an 80-column glass teletype.

The output looks a little bit like this:

<ResponseData>
    <AuthCode>094016</AuthCode>
    <AvsResult>Y</AvsResult>
    <CvResult>Match</CvResult>
    <HostCode>000</HostCode>
    <IavsResult>N</IavsResult>
    <Message>Approved</Message>
    <Partner>verisign</Partner>
    <PnRef>VUJN1A6E11D9</PnRef>
    <ResponseText>AP</ResponseText>
    <Result>0</Result>
    <StreetMatch>Match</StreetMatch>
    <Vendor>ActiveMerchant</Vendor>
    <ZipMatch>Match</ZipMatch>
</ResponseData>

Send the Sample thru a Mock

That string is long and ugly. We will presently beautify it, but for now just drop it into a method with a good name:

  def successful_authorization_response
    <<-XML
<ResponseData>
...
</ResponseData>
    XML
  end

Now this active_merchant test case can use the nifty Mocha mock system to "stubs" our method, and leave that poor abused wire alone!

  def test_successful_authorization
    @gateway.stubs(:ssl_post).returns(successful_authorization_response)
    
    assert response = @gateway.authorize(@amount, @credit_card, @options)
    assert_equal "Approved", response.message
    assert_success response
    assert response.test?
    assert_equal "VUJN1A6E11D9", response.authorization
  end

The test suite is now free to write cases which vary that response — and even insert errors & exceptions — without relying on the real web service's behavior. And our last step makes our code easier to refactor.

Use a Builder

That long XML statement is ugly, and hard to work with. Each time you think to change it, you must perform string surgery on it, which is another anti-pattern.

LAMP languages usually allow you to build XML content using your native language's notation. This, in turn, allows your language syntax to automatically validate details such as encodings, escapings, and tag nesting. You can nest real math expressions, and real conditional blocks, inside such constructs. You can break run-on XML up into modular functions and insert their returned strings. If your editor supports automated refactors, then they will see no difference between XML-generating code and normal program code.

This example shows Builder::XmlMarkup in action:

      def build_request(body, request_type = nil)
        xml = Builder::XmlMarkup.new
        xml.instruct!
        xml.tag! 'XMLPayRequest', 'Timeout' => 30, 'version' => "2.1", "xmlns" => XMLNS do
          xml.tag! 'RequestData' do
            xml.tag! 'Vendor', @options[:login]
            xml.tag! 'Partner', @options[:partner]
            if request_type == :recurring
              xml << body
            else
...
            end
          end
...
        end
        xml.target!
      end

Note that many builders, including Builder::XmlMarkup, permit a method_missing shortcut. xml.tag! 'RequestData' can be xml.RequestData. Only identifier-ready tags may apply; "Request Data" would require the explicit xml.tag!.

Also note that your production code should only generate XML using such builders.

From there, a useful test-side utility is a snip of code that walks an XML DOM and outputs each element as correctly nested Builder notation. Here's an example - inexplicably written in Python instead of Ruby. The recursive pattern is, naturally, the same in any language:

    def convert_xml_to_element_maker(self, thang):
        'script that coverts XML to its ElementMaker notation'

        from lxml import etree
        doc = etree.XML(thang)
        return self._convert_child_nodes(doc)

    def _convert_child_nodes(self, node, depth=0):
        code = '\n' + ' ' * depth * 2 + 'XML.' + node.tag + '('
        children = node.xpath('*')

        if node.text and not re.search('^\s*$', node.text):
            code += repr(node.text)
            if node.attrib or children:  code += ', '

        if children:
            child_nodes = [ self._convert_child_nodes(n, depth + 1) for n in children ]
            code += (',' ).join(child_nodes)
            if node.attrib:  code += ', '

        if node.attrib:
            attribs = [ '%s=%r' % (kv) for kv in node.attrib.items() ]
            code += ', '.join(attribs)

        code += ')'
        return code

That writes code that then responds to an XML = lxml.builder.ElementMaker() object.

Conclusion

Any wire protocol (including ones using a stack layer below TCP/IP) should get tested like this. And note these tests neglect the "last inch", from the actual ssl_post() method to the socket. That runs test-free. Developer tests are not "unit" test, and we are not here to invent a network stack.

Developer tests permit aggressive design refactors, and provide assurance that code features, over many changes, do not drift into bug-land. And that means we are now ready for the next post, where we refactor these tests into the Abstract Test Pattern.


You might also be interested in:

2 Comments

I enjoyed seeing this approach to testing web services. Recently, I've been using FakeWeb to help stub out these responses. I really like the ability to say

FakeWeb.allow_net_connect = false

and it will raise an exception any time it opens an external connection that is not stubbed.

+1 for FakeWeb, have used it before, didn't use it here because sometimes mocks are better. And I wanted to remain platform-agnostic.

News Topics

Recommended for You

Got a Question?