by Richard Taylor : 2020-02-12
We knocked ideas around to get our basic API. Now's the time to solidify what we have by writing some more tests.
If the structure of the API or the underlying components isn't quite right then those issues should surface as we add unit tests. Then we can add some higher level tests on top of that stronger foundation.
There is not a great deal to test in the translator.py module. But one thing did come up. Sorting the JSON keys makes the output more consistent and thus easier to test.
json.dumps(values, sort_keys=True)
And in case you didn't know; there is an assertRaises method in unittest:
self.assertRaises(TranslationError,
self.translate.to_item, '{blah}')
Which makes testing for exceptions easier. Note that the method under test to_item and its parameter '{blah}' are two separate parameters to assertRaises not one. The two lines above are equivalent to:
try:
self.translate.to_item('{blah}')
self.fail("expected a TranslationError to be raised")
except TranslationError:
pass
but a lot shorter.
There is also not a lot to test in the storage.py module. But I quickly noticed that I had been lazy and returned None from the find_item method when there was no item found. This is poor design so I changed it to raise a new ItemNotFound exception instead.
I expect this change to cause a failure elsewhere in code that assumes None means not found. So lets try and provoke that with a test.
Sure enough the first few tests for the api.py module fail when we try to call get_item for a non-existent identifier.
And it seems clear that we need to move the definitions of the API exceptions out of the lower-level modules into a error.py module so that api.py does not have to catch the exceptions from storage.py just to rethrow them as a more generic type.
For example, in storage.py we have:
def find_item(self, id):
try:
return self.items[id]
except KeyError:
raise ItemNotFound()
Where ItemNotFound is an exception from error.py which is also the expected result from API.get_item when the item does not exist. So all that method needs is:
return storage.find_item(id)
which will raise the ItemNotFound when expected.
That gets us to this version of the project. We have unit tests for all the functionality, but no formal black-box tests for the whole API.
To test the whole API we need a server running, so the tests have to be separate from the unit tests. Also it is a good idea to not use the code that we are testing in black-box tests.
I created a separate runner for these tests:
bin/api_test_thelca.sh
which runs all tests fitting the pattern api_tests_*.py
Note that the first new test api_tests_item.py does not import any modules from thelca but instead uses the popular python 'requests' module that you will probably need to install.
import requests
import unittest
class TestBlackBoxItemAPI(unittest.TestCase):
def setUp(self):
self.url = 'http://localhost:2207/v1/items'
def test_get_fails_for_non_item(self):
response = requests.get(self.url + '/no-such-id')
self.assertEqual(404, response.status_code)
As you can see this makes it very easy to write black-box tests for the API using requests and the unittest framework to make assertions.
Once you step back from the code and write black-box tests then you often find bugs that you completely overlooked when you were close to the code. For example, I discovered that the API was returning SERVICE_UNAVAILABLE when I passed in certain 'bad' JSON strings, instead of BAD_REQUEST. That's because when I wrote the code I thought it could only fail if there was an internal problem... but then put some checking into that method which meant it could fail if the request was bad.
Another thing I discovered was that passing in additional fields just propagated them through instead of stripping them out. This test:
def test_post_ignores_extra_fields(self):
response = requests.post(self.url, data = '''
{
"properties": { "type": "BUG", "status": "OPEN" },
"secret": "you aint seen me, right?"
}
''')
self.assertEqual(200, response.status_code)
json = response.json()
self.assertNotIn('secret', json)
failed until I fixed the code and 'secret' vanished.
Having written the tests above I knew I needed to implement the PUT method next, so that Items can be updated once created. Since I now have some nice black-box tests for GET and POST it seemed natural to write the tests for PUT before writing the code - Test Driven Development. So this is what I did:
And after that we have 27 unit tests and 17 API tests and the project is in this state. Note that I added some notes to the README on how to run the API tests.
Now we have implemented the main API methods for Items and written tests to show that the implementation is correct. It is tempting to crack on and implement Links as well. But before I do that I want to tackle the strong logging that I promised.
So the next step is to log all the changes made by the API in such a way that every change can be audited; and the state of all the stored Items can be restored by replaying the events recorded in the log.