Merge import/ in other directories (develop/ and use/).

develop/advanced_plugin_testing.rst

@@ -0,0 +1,303 @@
+***********************
+Advanced Plugin Testing
+***********************
+  The complete guide to writing tests for your plugins.
+
+Why Write Tests?
+================
+  Why should I write tests for my plugin? Here's why.
+
+For those of you asking "Why should I write tests for my plugin? I tried it
+out, and it works!", read on. For those of you who already realize that
+Testing is Good (TM), skip to the next section.
+
+Here are a few quick reasons why to test your Supybot plugins.
+
+    * When/if we rewrite or change certain features in Supybot, tests make
+      sure your plugin will work with these changes. It's much easier to run
+      supybot-test MyPlugin after upgrading the code and before even reloading
+      the bot with the new code than it is to load the bot with new code and
+      then load the plugin only to realize certain things don't work. You may
+      even ultimately decide you want to stick with an older version for a while
+      as you patch your custom plugin. This way you don't have to rush a patch
+      while restless users complain since you're now using a newer version that
+      doesn't have the plugin they really like.
+
+    * Running the automated tests takes a few seconds, testing plugins in IRC
+      on a live bot generally takes quite a bit longer. We make it so that
+      writing tests generally doesn't take much time, so a small initial
+      investment adds up to lots of long-term gains.
+
+    * If you want your plugin to be included in any of our releases (the core
+      Supybot if you think it's worthy, or our supybot-plugins package), it has
+      to have tests. Period.
+
+For a bigger list of why to write unit tests, check out this article:
+
+  http://www.onjava.com/pub/a/onjava/2003/04/02/javaxpckbk.html
+
+and also check out what the Extreme Programming folks have to say about unit
+tests:
+
+  http://www.extremeprogramming.org/rules/unittests.html
+
+Plugin Tests
+============
+  How to write tests for commands in your plugins.
+
+Introduction
+------------
+
+This tutorial assumes you've read through the plugin author tutorial, and that
+you used supybot-plugin-create to create your plugin (as everyone should). So,
+you should already have all the necessary imports and all that boilerplate
+stuff in test.py already, and you have already seen what a basic plugin test
+looks like from the plugin author tutorial. Now we'll go into more depth about
+what plugin tests are available to Supybot plugin authors.
+
+Plugin Test Case Classes
+------------------------
+
+Supybot comes with two plugin test case classes, PluginTestCase and
+ChannelPluginTestCase. The former is used when it doesn't matter whether or
+not the commands are issued in a channel, and the latter is used for when it
+does. For the most part their API is the same, so unless there's a distinction
+between the two we'll treat them as one and the same when discussing their
+functionality.
+
+The Most Basic Plugin Test Case
+-------------------------------
+
+At the most basic level, a plugin test case requires three things:
+
+* the class declaration (subclassing PluginTestCase or
+  ChannelPluginTestCase)
+* a list of plugins that need to be loaded for these tests (does not
+  include Owner, Misc, or Config, those are always automatically loaded) -
+  often this is just the name of the plugin that you are writing tests for
+* some test methods
+
+Here's what the most basic plugin test case class looks like (for a plugin
+named MyPlugin)::
+
+    class MyPluginTestCase(PluginTestCase):
+        plugins = ('MyPlugin',)
+
+        def testSomething(self):
+            # assertions and such go here
+
+Your plugin test case should be named TestCase as you see above, though it
+doesn't necessarily have to be named that way (supybot-plugin-create puts that
+in place for you anyway). As you can see we elected to subclass PluginTestCase
+because this hypothetical plugin apparently doesn't do anything
+channel-specific.
+
+As you probably noticed, the plugins attribute of the class is where the list
+of necessary plugins goes, and in this case just contains the plugin that we
+are testing. This will be the case for probably the majority of plugins. A lot
+of the time test writers will use a bot function that performs some function
+that they don't want to write code for and they will just use command nesting
+to feed the bot what they need by using that plugin's functionality. If you
+choose to do this, only do so with core bot plugins as this makes distribution
+of your plugin simpler. After all, we want people to be able to run your
+plugin tests without having to have all of your plugins!
+
+One last thing to note before moving along is that each of the test methods
+should describe what they are testing. If you want to test that your plugin
+only responds to registered users, don't be afraid to name your test method
+testOnlyRespondingToRegisteredUsers or testNotRespondingToUnregisteredUsers.
+You may have noticed some rather long and seemingly unwieldy test method names
+in our code, but that's okay because they help us know exactly what's failing
+when we run our tests. With an ambiguously named test method we may have to
+crack open test.py after running the tests just to see what it is that failed.
+For this reason you should also test only one thing per test method. Don't
+write a test method named testFoobarAndBaz. Just write two test methods,
+testFoobar and testBaz. Also, it is important to note that test methods must
+begin with test and that any method within the class that does begin with test
+will be run as a test by the supybot-test program. If you want to write
+utility functions in your test class that's fine, but don't name them
+something that begins with test or they will be executed as tests.
+
+Including Extra Setup
+---------------------
+
+Some tests you write may require a little bit of setup. For the most part it's
+okay just to include that in the individual test method itself, but if you're
+duplicating a lot of setup code across all or most of your test methods it's
+best to use the setUp method to perform whatever needs to be done prior to
+each test method.
+
+The setUp method is inherited from the whichever plugin test case class you
+chose for your tests, and you can add whatever functionality you want to it.
+Note the important distinction, however: you should be adding to it and not
+overriding it. Just define setUp in your own plugin test case class and it
+will be run before all the test methods are invoked.
+
+Let's do a quick example of one. Let's write a setUp method which registers a
+test user for our test bot::
+
+    def setUp(self):
+        ChannelPluginTestCase.setUp(self)  # important!!
+        # Create a valid user to use
+        self.prefix = 'foo!bar@baz'
+        self.feedMsg('register tester moo', to=self.nick, frm=self.prefix))
+        m = self.getMsg()  # Response to registration.
+
+Now notice how the first line calls the parent class's setUp method first?
+This must be done first. Otherwise several problems are likely to arise. For
+one, you wouldn't have an irc object at self.irc that we use later on nor
+would self.nick be set.
+
+As for the rest of the method, you'll notice a few things that are available
+to the plugin test author. self.prefix refers to the hostmask of the
+hypothetical test user which will be "talking" to the bot, issuing commands.
+We set it to some generically fake hostmask, and then we use feedMsg to send
+a private message (using the bot's nick, accessible via self.nick) to the bot
+registering the username "tester" with the password "moo". We have to do it
+this way (rather than what you'll find out is the standard way of issuing
+commands to the bot in test cases a little later) because registration must be
+done in private. And lastly, since feedMsg doesn't dequeue any messages from
+the bot after being fed a message, we perform a getMsg to get the response.
+You're not expected to know all this yet, but do take note of it since using
+these methods in test-writing is not uncommon. These utility methods as well as
+all of the available assertions are covered in the next section.
+
+So, now in any of the test methods we write, we'll be able to count on the
+fact that there will be a registered user "tester" with a password of "moo",
+and since we changed our prefix by altering self.prefix and registered after
+doing so, we are now identified as this user for all messages we send unless
+we specify that they are coming from some other prefix.
+
+The Opposite of Setting-up: Tearing Down
+----------------------------------------
+
+If you did some things in your setUp that you want to clean up after, then
+this code belongs in the tearDown method of your test case class. It's
+essentially the same as setUp except that you probably want to wait to invoke
+the parent class's tearDown until after you've done all of your tearing down.
+But do note that you do still have to invoke the parent class's tearDown
+method if you decide to add in your own tear-down stuff.
+
+Setting Config Variables for Testing
+------------------------------------
+
+Before we delve into all of the fun assertions we can use in our test methods
+it's worth noting that each plugin test case can set custom values for any
+Supybot config variable they want rather easily. Much like how we can simply
+list the plugins we want loaded for our tests in the plugins attribute of our
+test case class, we can set config variables by creating a mapping of
+variables to values with the config attribute.
+
+So if, for example, we wanted to disable nested commands within our plugin
+testing for some reason, we could just do this::
+
+    class MyPluginTestCase(PluginTestCase):
+        config = {'supybot.commands.nested': False}
+
+        def testThisThing(self):
+            # stuff
+
+And now you can be assured that supybot.commands.nested is going to be off for
+all of your test methods in this test case class.
+
+Plugin Test Methods
+===================
+  The full list of test methods and how to use them.
+
+Introduction
+------------
+
+You know how to make plugin test case classes and you know how to do just
+about everything with them except to actually test stuff. Well, listed below
+are all of the assertions used in tests. If you're unfamiliar with what an
+assertion is in code testing, it is basically a requirement of something that
+must be true in order for that test to pass. It's a necessary condition. If
+any assertion within a test method fails the entire test method fails and it
+goes on to the next one.
+
+Assertions
+----------
+
+All of these are methods of the plugin test classes themselves and hence are
+accessed by using self.assertWhatever in your test methods. These are sorted
+in order of relative usefulness.
+
+assertResponse(query, expectedResponse)
+    Feeds query to the bot as a
+    message and checks to make sure the response is expectedResponse. The
+    test fails if they do not match (note that prefixed nicks in the
+    response do not need to be included in the expectedResponse).
+
+assertError(query)
+    Feeds query to the bot and expects an error in
+    return. Fails if the bot doesn't return an error.
+
+assertNotError(query)
+    The opposite of assertError. It doesn't matter
+    what the response to query is, as long as it isn't an error. If it is
+    not an error, this test passes, otherwise it fails.
+
+assertRegexp(query, regexp, flags=re.I)
+    Feeds query to the bot and
+    expects something matching the regexp (no m// required) in regexp with
+    the supplied flags. Fails if the regexp does not match the bot's
+    response.
+
+assertNotRegexp(query, regexp, flags=re.I)
+    The opposite of
+    assertRegexp. Fails if the bot's output matches regexp with the
+    supplied flags.
+
+assertHelp(query)
+    Expects query to return the help for that command.
+    Fails if the command help is not triggered.
+
+assertAction(query, expectedResponse=None)
+    Feeds query to the bot and
+    expects an action in response, specifically expectedResponse if it is
+    supplied. Otherwise, the test passes for any action response.
+
+assertActionRegexp(query, regexp, flags=re.I)
+    Basically like
+    assertRegexp but carries the extra requirement that the response must
+    be an action or the test will fail.
+
+Utilities
+---------
+
+feedMsg(query, to=None, frm=None) 
+    Simply feeds query to whoever is
+    specified in to or to the bot itself if no one is specified. Can also
+    optionally specify the hostmask of the sender with the frm keyword.
+    Does not actually perform any assertions.
+
+getMsg(query)
+    Feeds query to the bot and gets the response.
+
+Other Tests
+===========
+  If you had to write helper code for a plugin and want to test it, here's
+  how.
+
+Previously we've only discussed how to test stuff in the plugin that is
+intended for IRC. Well, we realize that some Supybot plugins will require
+utility code that doesn't necessarily require all of the overhead of setting
+up IRC stuff, and so we provide a more lightweight test case class,
+SupyTestCase, which is a very very light wrapper around unittest.TestCase
+(from the standard unittest module) that basically just provides a little
+extra logging. This test case class is what you should use for writing those
+test cases which test things that are independent of IRC.
+
+For example, in the MoobotFactoids plugin there is a large chunk of utility
+code dedicating to parsing out random choices within a factoid using a class
+called OptionList. So, we wrote the OptionListTestCase as a SupyTestCase for
+the MoobotFactoids plugin. The setup for test methods is basically the same as
+before, only you don't have to define plugins since this is independent of
+IRC.
+
+You still have the choice of using setUp and tearDown if you wish, since those
+are inherited from unittest.TestCase. But, the same rules about calling the
+setUp or tearDown method from the parent class still apply.
+
+With all this in hand, now you can write great tests for your Supybot plugins!
+