{"id":148,"date":"2017-01-08T00:44:06","date_gmt":"2017-01-08T00:44:06","guid":{"rendered":"http:\/\/old.arxcruz.net\/?p=148"},"modified":"2017-01-08T00:44:06","modified_gmt":"2017-01-08T00:44:06","slug":"testing-http-requests-with-mock-or-httmock","status":"publish","type":"post","link":"https:\/\/arxcruz.net\/index.php\/2017\/01\/08\/testing-http-requests-with-mock-or-httmock\/","title":{"rendered":"Testing http requests with mock or HTTMock?"},"content":{"rendered":"<p>So, these days I was working in a project that uses requests to make some http\/https calls:<\/p>\n<pre class=\"brush: python; title: ; notranslate\" title=\"\">\nimport requests\n\n\nURL = 'https:\/\/us.api.battle.net\/wow\/auction\/data\/{}?locale=en_US'\nAPI = 'myapikey'\n\ndef request_with_api(url):\n    apikey = {'apikey': API}\n    return requests.get(url, params=apikey)\n\n\ndef get_auctions_url(realm):\n    url = URL.format(realm)\n    r = request_with_api(url)\n    if r.status_code == 200:\n        data = r.json()\n        data_url = data['files'][0]['url']\n        return data_url\n    return None\n\n\ndef get_auctions(realm):\n    url = get_auctions_url(realm)\n    if url:\n        r = request_with_api(url)\n        if r.status_code == 200:\n            return r.json()\n        else:\n            raise Exception('Error in return code: %s' % r.status_code)\n    else:\n        raise Exception('Auction data not found')\n\n\nif __name__ == '__main__':\n    auctions = get_auctions('medivh')\n    print auctions\n\n<\/pre>\n<p>And as usual, I wanted to have my unit tests, and requests is very tricky, because, you don&#8217;t want to keep making requests to the real server, every time you run your unit tests, and here is a couple of reasons:<\/p>\n<ol>\n<li>I&#8217;ll depend of internet to access the server all the time<\/li>\n<li>The server has a limit of requests that I don&#8217;t want to waste testing<\/li>\n<li>Some servers may block your account if you do too many requests in a small amount of time.<\/li>\n<\/ol>\n<h2>Mocking<\/h2>\n<p>So, the easiest way to test would be using <a href=\"https:\/\/github.com\/testing-cabal\/mock\">mock<\/a>, and then do something like this:<\/p>\n<pre class=\"brush: python; title: ; notranslate\" title=\"\">\nimport unittest\nimport mock\n\nimport wow\n\nfrom wow import request_with_api\n\nclass TestWow(unittest.TestCase):\n\n    @mock.patch('wow.request_with_api')\n    def test_get_auctions_url(self, wow_patch):\n        wow_patch.return_value.status_code = 200\n        wow_patch.return_value.json.return_value = {'files': [\n            {'url': ('http:\/\/auction-api-us.worldofwarcraft.com\/'\n                     'auction-data')}\n        ]}\n        data = wow.get_auctions_url('medivh')\n        wow_patch.assert_called_once()\n        self.assertEquals(data,\n                'http:\/\/auction-api-us.worldofwarcraft.com\/auction-data')\n\n        wow_patch.reset_mock()\n        wow_patch.return_value.status_code = 403\n        data = wow.get_auctions_url('medivh')\n        wow_patch.assert_called_once()\n        self.assertIsNone(data)\n\n\nif __name__ == '__main__':\n    unittest.main()\n\n<\/pre>\n<p>This looks good, all I have to do is set the mock return values, reset the mock, set the values of the status_code, the return of the r.json() value, and all of this inside our tests. Of course, I could create the setUp method, and configure the values there:<\/p>\n<pre class=\"brush: python; title: ; notranslate\" title=\"\">\nimport unittest\nimport mock\n\nimport wow\n\nfrom wow import request_with_api\n\nclass TestWow(unittest.TestCase):\n    def setUp(self):\n        self.ok_return = ('http:\/\/auction-api-us.worldofwarcraft.com\/'\n                          'auction-data')\n        self.json_ok_return = {\n            'files': [\n                {\n                    'url': self.ok_return\n                }\n            ]\n        }\n\n    @mock.patch('wow.request_with_api')\n    def test_get_auctions_url(self, wow_patch):\n        wow_patch.return_value.status_code = 200\n        wow_patch.return_value.json.return_value = self.json_ok_return\n        data = wow.get_auctions_url('medivh')\n        wow_patch.assert_called_once()\n        self.assertEquals(data, self.ok_return)\n\n        wow_patch.reset_mock()\n\n        wow_patch.return_value.status_code = 403\n        data = wow.get_auctions_url('medivh')\n        wow_patch.assert_called_once()\n        self.assertIsNone(data)\n\n\nif __name__ == '__main__':\n    unittest.main()\n\n<\/pre>\n<p>Now, so far, we are only testing the get_auctions_url method, let&#8217;s add the test to get_auctions method, who calls the get_auctions_url method as well:<\/p>\n<pre class=\"brush: python; title: ; notranslate\" title=\"\">\nimport unittest\nimport mock\n\nimport wow\n\n\nclass TestWow(unittest.TestCase):\n    def setUp(self):\n        self.ok_return = ('http:\/\/auction-api-us.worldofwarcraft.com\/'\n                          'auction-data')\n        self.json_ok_return = {\n            'files': [\n                {\n                    'url': self.ok_return\n                }\n            ]\n        }\n\n        self.json_auction_data = {\n            'content': {\n                &quot;realms&quot;: [\n                    {&quot;name&quot;: &quot;Medivh&quot;, &quot;slug&quot;: &quot;medivh&quot;},\n                    {&quot;name&quot;: &quot;Exodar&quot;, &quot;slug&quot;: &quot;exodar&quot;}\n                ],\n                &quot;auctions&quot;: [\n                    {\n                        &quot;auc&quot;: 1283349179, &quot;item&quot;: 129158,\n                        &quot;owner&quot;: &quot;Toiletcow&quot;,\n                        &quot;ownerRealm&quot;: &quot;Medivh&quot;, &quot;bid&quot;: 8360, &quot;buyout&quot;: 8800,\n                        &quot;quantity&quot;: 1, &quot;timeLeft&quot;: &quot;LONG&quot;, &quot;rand&quot;: 0,\n                        &quot;seed&quot;: 0, &quot;context&quot;: 0\n                    }\n                ]\n            }\n        }\n\n    @mock.patch('wow.request_with_api')\n    def test_get_auctions_url(self, wow_patch):\n        wow_patch.return_value.status_code = 200\n        wow_patch.return_value.json.return_value = self.json_ok_return\n        data = wow.get_auctions_url('medivh')\n        wow_patch.assert_called_once()\n        self.assertEquals(data, self.ok_return)\n\n        wow_patch.reset_mock()\n\n        wow_patch.return_value.status_code = 403\n        data = wow.get_auctions_url('medivh')\n        wow_patch.assert_called_once()\n        self.assertIsNone(data)\n\n    @mock.patch('wow.get_auctions_url')\n    @mock.patch('wow.request_with_api')\n    def test_get_auctions(self, request_with_api_mock,\n                          get_auctions_url_mock):\n        get_auctions_url_mock.return_value = None\n        self.assertRaises(Exception, wow.get_auctions, 'medivh')\n        get_auctions_url_mock.assert_called_once()\n\n        get_auctions_url_mock.reset_mock()\n\n        request_with_api_mock.return_value.status_code = 200\n        request_with_api_mock.return_value.json.return_value = (\n                self.json_auction_data)\n        get_auctions_url_mock.return_value = self.ok_return\n        get_auctions_url_mock.json.return_value = self.json_auction_data\n        data = wow.get_auctions('medivh')\n        self.assertDictEqual(data, self.json_auction_data)\n\n\nif __name__ == '__main__':\n    unittest.main()\n\n<\/pre>\n<p>So, now things are starting to get a little bit messy, here we need to mock both wow.get_auctions_url, and wow.request_with_api, and reset the mocks and assert that the mock is being called once. It works, but still, it&#8217;as a mess for the next guy who will check this code.<\/p>\n<h2>HTTMock<\/h2>\n<p>The coolest part of the <a href=\"https:\/\/github.com\/patrys\/httmock\">HTTMock<\/a> is that it returns a requests object for you, which means you can call other requests method if you need it without the need of mock these methods. In our example, we only mocked the json() and status_code, but imagine if we had to mock the headers, responses, etc. HTTMock do that for you.<\/p>\n<p>In my opinion, HTTMock keeps your test methods more clean:<\/p>\n<pre class=\"brush: python; title: ; notranslate\" title=\"\">\nimport unittest\nfrom httmock import urlmatch, HTTMock\n\nimport wow\n\n\n@urlmatch(netloc=r'(.*\\.)?us\\.api\\.battle\\.net.*$')\ndef auctions_url(url, request):\n    not_found = ('http:\/\/auction-api-us.worldofwarcraft.com\/'\n                 'auction-data\/notfound')\n    found = 'http:\/\/auction-api-us.worldofwarcraft.com\/auction-data'\n\n    if 'fail' in url.path:\n        return {'status_code': 403}\n\n    return {\n           'status_code': 200,\n           'content':\n           {\n               'files': [\n                   {'url': not_found if 'notfound' in url.path else found}\n               ]\n           }\n       }\n\n\nclass TestAuctions(unittest.TestCase):\n\n    def test_get_auctions_url(self):\n        with HTTMock(auctions_url):\n            data = wow.get_auctions_url('medivh')\n            self.assertEquals(data, ('http:\/\/auction-api-us.worldofwarcraft.'\n                                     'com\/auction-data'))\n\n        with HTTMock(auctions_url):\n            data = wow.get_auctions_url('fail')\n            self.assertIsNone(data)\n\nif __name__ == '__main__':\n    unittest.main()\n<\/pre>\n<p>As you can see, we can set a method with the @urlmatch decorator and all requests that match that particular regular expression will be handle by this method.<br \/>\nInside that method we also can set some conditionals as I did to test when we have &#8216;not found&#8217;.<br \/>\nOf course, there are other decorators. You can use, for example the @all_requests decorator<\/p>\n<pre class=\"brush: python; title: ; notranslate\" title=\"\">\nimport unittest\nfrom httmock import all_requests, HTTMock\n\nimport wow\n\n\n@all_requests\ndef auctions_url_req(url, request):\n    not_found = ('http:\/\/auction-api-us.worldofwarcraft.com\/'\n                 'auction-data\/notfound')\n    found = 'http:\/\/auction-api-us.worldofwarcraft.com\/auction-data'\n    if 'fail' in url.path:\n        return {'status_code': 403}\n\n    return {\n           'status_code': 200,\n           'content':\n           {\n               'files': [\n                   {'url': not_found if 'notfound' in url.path else found}\n               ]\n           }\n       }\n\n\nclass TestAuctions(unittest.TestCase):\n\n    def test_get_auctions_url(self):\n        with HTTMock(auctions_url_req):\n            data = wow.get_auctions_url('medivh')\n            self.assertEquals(data, ('http:\/\/auction-api-us.worldofwarcraft.'\n                                     'com\/auction-data'))\n\n        with HTTMock(auctions_url_req):\n            data = wow.get_auctions_url('fail')\n            self.assertIsNone(data)\n\nif __name__ == '__main__':\n    unittest.main()\n<\/pre>\n<p>In this case, all the requests you do no matter the URL, will be handled by the auctions_url_req:<\/p>\n<pre class=\"brush: python; title: ; notranslate\" title=\"\">\nimport requests\n\nwith HTTMock(auctions_url_req):\n    data = requests.get('https:\/\/www.google.com')\n    print(data.status_code)\n    print(data.json())\n<\/pre>\n<p>And you get this output:<\/p>\n<pre class=\"brush: python; title: ; notranslate\" title=\"\">\n200\n{u'files': [{u'url': u'http:\/\/auction-api-us.worldofwarcraft.com\/auction-data'}]}\n<\/pre>\n<h2>Bottom line<\/h2>\n<p>Use or not to use HTTMock is up to you, I believe this can make your tests cleaner, it&#8217;s much more easy look to a with HTTMock and say: oh, hey, so, this will return whatever is in the with method, cool!<br \/>\nWhile using mock only, I have to be like, okay&#8230; this object is a mock, and the returned value of this method here is 200, and oh, it&#8217;s resetting the mock here, so let&#8217;s forget about all of this, and start over with the&#8230; well, you get the idea.<br \/>\nAnd of course, you can use more than one method to handle the requests, as you can see on line 65:<\/p>\n<pre class=\"brush: python; title: ; notranslate\" title=\"\">\nimport unittest\nfrom httmock import urlmatch, HTTMock, all_requests\n\nimport wow\n\nAUCTION_CONTENT = {\n    &quot;realms&quot;: [\n        {&quot;name&quot;: &quot;Medivh&quot;, &quot;slug&quot;: &quot;medivh&quot;},\n        {&quot;name&quot;: &quot;Exodar&quot;, &quot;slug&quot;: &quot;exodar&quot;}\n    ],\n    &quot;auctions&quot;: [\n        {\n            &quot;auc&quot;: 1283349179, &quot;item&quot;: 129158,\n            &quot;owner&quot;: &quot;Toiletcow&quot;,\n            &quot;ownerRealm&quot;: &quot;Medivh&quot;, &quot;bid&quot;: 8360, &quot;buyout&quot;: 8800,\n            &quot;quantity&quot;: 1, &quot;timeLeft&quot;: &quot;LONG&quot;, &quot;rand&quot;: 0,\n            &quot;seed&quot;: 0, &quot;context&quot;: 0\n         }\n    ]\n}\n\n\n@urlmatch(netloc=r'(.*\\.)?us\\.api\\.battle\\.net.*$')\ndef auctions_url(url, request):\n    not_found = ('http:\/\/auction-api-us.worldofwarcraft.com\/'\n                 'auction-data\/notfound')\n    found = 'http:\/\/auction-api-us.worldofwarcraft.com\/auction-data'\n\n    if 'fail' in url.path:\n        return {'status_code': 403}\n\n    return {\n           'status_code': 200,\n           'content':\n           {\n               'files': [\n                   {'url': not_found if 'notfound' in url.path else found}\n               ]\n           }\n       }\n\n\n@urlmatch(path=r'^\\\/auction\\-data.*$')\ndef auctions(url, request):\n    if 'notfound' in url.path:\n        return {'status_code': 404}\n\n    return {'status_code': 200,\n            'content': AUCTION_CONTENT}\n\n\nclass TestAuctions(unittest.TestCase):\n\n    def test_get_auctions_url(self):\n        with HTTMock(auctions_url_req):\n            data = wow.get_auctions_url('medivh')\n            self.assertEquals(data, ('http:\/\/auction-api-us.worldofwarcraft.'\n                                     'com\/auction-data'))\n\n        with HTTMock(auctions_url_req):\n            data = wow.get_auctions_url('fail')\n            self.assertIsNone(data)\n\n    def test_get_auctions(self):\n        with HTTMock(auctions, auctions_url):\n            data = wow.get_auctions('medivh')\n            self.assertDictEqual(data, AUCTION_CONTENT)\n\n        with HTTMock(auctions, auctions_url):\n            self.assertRaises(Exception, wow.get_auctions, 'notfound')\n\nif __name__ == '__main__':\n    unittest.main()\n    import requests\n\n<\/pre>\n<p>And that&#8217;s it! Now you have two choices to test your python code that uses requests.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>So, these days I was working in a project that uses requests to make some http\/https calls: And as usual, I wanted to have my unit tests, and requests is very tricky, because, you don&#8217;t want to keep making requests to the real server, every time you run your unit tests, and here is a &hellip; <a href=\"https:\/\/arxcruz.net\/index.php\/2017\/01\/08\/testing-http-requests-with-mock-or-httmock\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Testing http requests with mock or HTTMock?&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","jetpack_publicize_message":"","jetpack_is_tweetstorm":false},"categories":[6,1],"tags":[13,14,7,15],"jetpack_featured_media_url":"","jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p8bIYB-2o","jetpack-related-posts":[{"id":127,"url":"https:\/\/arxcruz.net\/index.php\/2015\/06\/13\/writing-tests-in-python-for-beginners-part-1-doctest\/","url_meta":{"origin":148,"position":0},"title":"Writing tests in python for beginners - Part 1 - doctest","date":"June 13, 2015","format":false,"excerpt":"So, I just started an interest on python tests and I would like to share what I learned so far. Let\u2019s skip all that theory about how tests are important for the code, bla bla bla, if you want to learn test, probably you already have your own reason, so,\u2026","rel":"","context":"In &quot;Development&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":153,"url":"https:\/\/arxcruz.net\/index.php\/2017\/09\/21\/debugging-tempest\/","url_meta":{"origin":148,"position":1},"title":"Debugging tempest","date":"September 21, 2017","format":false,"excerpt":"Introduction One of the common tasks when you're working on TripleO is to debug tempest when you found some failures. This is very important because once you found an error, you need to provide all the information needed when you're gonna fill a bug, or you can even actually, find\u2026","rel":"","context":"In &quot;OpenStack&quot;","img":{"alt_text":"","src":"https:\/\/i2.wp.com\/arxcruz.net\/wp-content\/uploads\/2017\/09\/Selection_005-300x207.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":119,"url":"https:\/\/arxcruz.net\/index.php\/2015\/02\/05\/new-small-project-stackquery-dashboard\/","url_meta":{"origin":148,"position":2},"title":"New small project: stackquery-dashboard","date":"February 5, 2015","format":false,"excerpt":"I\u2019ve started a new project called stackquery-dashboard. Basically, it\u2019s a web interface to show reports about user contribution in OpenStack, using the\u00a0stackalytics.com\u00a0API and part of the code from launchpadstats\u00a0from\u00a0\u00a0Martina Kollarova. The first reason I start this project is because stackalytics.com only shows reports for each user, and sometimes managers in\u2026","rel":"","context":"In &quot;OpenStack&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/arxcruz.net\/index.php\/wp-json\/wp\/v2\/posts\/148"}],"collection":[{"href":"https:\/\/arxcruz.net\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/arxcruz.net\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/arxcruz.net\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/arxcruz.net\/index.php\/wp-json\/wp\/v2\/comments?post=148"}],"version-history":[{"count":0,"href":"https:\/\/arxcruz.net\/index.php\/wp-json\/wp\/v2\/posts\/148\/revisions"}],"wp:attachment":[{"href":"https:\/\/arxcruz.net\/index.php\/wp-json\/wp\/v2\/media?parent=148"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/arxcruz.net\/index.php\/wp-json\/wp\/v2\/categories?post=148"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/arxcruz.net\/index.php\/wp-json\/wp\/v2\/tags?post=148"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}