Published: March 28, 2022
Testing (especially unit testing) is a great way to build determinism into your software. But, when external API calls are involved, it can become challenging to build and test quickly and with idempotence. AND, if the code making these calls is part of a third-party package or library, it can be even more challenging.
Luckily, most languages and testing platforms account for this complexity using some semblance of mocking or patching. Here, I want to discuss how we use “monkey patching” in Python to unit test usage of the Azure Python SDK in our security software.
Azure provides us nearlly all of the underlying infrastructure we require for our incredible security operations. From alerting in Sentinel, to automation with Logic Apps, to custom analysis with Azure Fuctions. All we need:
Using Python (which is deeply integrated into the Microsoft ecosystem), Github, and solid SecDevOps practices allows us to accomplish this in a neatly integrated way. All part of our SOARganic framework, our Security Operations can take advantage of tried and true software development practices to build:
I love unit tests. They:
As a byproduct, when you automate unit tests, they also:
The idea is that you can execute the tests frequently, as needed to ensure your changes are working. Frequently running these tests will cause issues if the tests:
So, we all agree that unit test rocks. Why not just do it?
We use the Azure SDK for Python in several of our security platforms. It does an amazing job of interfacing with the Azure ecosystem and honestly, makes development easier (versus generating manual API calls.) But, there is a bit of magic baked into the package which can make unit tests complex.
Often, an API call like this could be “mocked” using something like the requests-mock library. Consider the following contrived example:
import requests
def make_request():
response = requests.request("GET", "https://meetascent.com")
response.raise_for_status()
return response
The above code will make an HTTP GET request to the specified URL.
This is an easy call to mock because we specify all the logic and actually create request.
In order to unit test the above, we would want to run tests with some various HTTP Status codes in the response, some errors in the response, some successful messages, etc.
And we would like to do this without requesting the site every time.
What if the site is ACTUALLY down, our tests would break even though the code is fine.
import pytest
from requests.exceptions import HTTPError
from requests_mock import Mocker
from examples import make_request
def test_make_request():
with Mocker() as m:
m.register_uri("GET", "https://meetascent.com", text="Got it!", status_code=200)
response = make_request()
assert response
def test_make_request_teapot():
with Mocker() as m:
m.register_uri("GET", "https://meetascent.com", text="Am teapot", status_code=418)
with pytest.raises(HTTPError) as e:
_ = make_request()
assert e
In the above tests, test_make_request uses the requests-mock library to create a Mocker.
Then, we register the URI (which we know) and provide a known, pre-determined result (Got it with a status code of 200), which we can test against.
Alternatively, we can also test the case when the response fails using the pytest.raises context manager.
We define a failed response (Am teapot, 418) and validate that the raise_for_exception method is working in our code.
Now, consider the method (forgive the long namespacing):
azure.mgmt.storage.StorageManagementClient.storage_accounts.check_name_availability which queries the Azure Storage Account API to see if a specified account name is available.
The backed magic of this method:
Attempting mock all of these requests, provide expected responses to the library, and handle the next steps just became exponentially more challenging. Instead we can use a concept in Python called “Monkey Patching.” The idea here is to “replace” the called method, with one of our own definition.
In the below contrived example, the defined method return_7 will always return the value 7:
class Foo:
def return_7(self, a):
return 7
But, we can monkey patch (or temporarily replace) the functionality of the method. Pytest even has built in functionality to accomplish this using a fixture.
from examples import Foo
def test_foo(monkeypatch):
def patch_return_7(_):
return 8
monkeypatch.setattr(Foo, Foo.return_7.__name__, patch_return_7)
result = Foo().return_7()
assert result
assert result != 7
monkeypatch.setattr replaces the original method Foo.return_7 with the “patched” method of our choice.
In this case test_foo.patch_return_7 which will return the value 8 and the test will pass.
So, we can copy the same construct as above, but use it replace the magic filled Azure SDK method as such:
from azure.identity import DefaultAzureCredential
from azure.mgmt.storage import StorageManagementClient
class AzureSDK:
def check_name_available(self, account_name):
client = StorageManagementClient(DefaultAzureCredential, "the-subscription-id")
response = client.storage_accounts.check_name_availability(account_name)
return response
In the above code, the AzureSDK.check_name_available method will query the Azure Storage Account API and check if the account name specified is available.
If we inspect the code, we can see that storage_accounts is of type azure.mgmt.storage.[version].StorageAccountsOperations.
So, we will need to patch the method azure.mgmt.storage.[version].StorageAccountsOperations.check_name_availability.
Inspecting that code, we see it returns a response of type azure.mgmt.storage.[version].models.CheckNameAvailabilityResult and we will need to mock this type.
from azure.mgmt.storage.v2021_08_01.operations import StorageAccountsOperations
from examples import AzureSDK
class MockAvailableResult:
name_available = True
class MockStorageAccountOperations:
def check_name_availability(instance, name):
assert instance
assert name
def test_check_name_available(monkeypatch):
def mock_check_name_availability(*args):
MockStorageAccountOperations.check_name_availability(*args)
result = MockAvailableResult
result.name_available = True
return result
# Replace the call to check_name_availability with a method of our own
monkeypatch.setattr(
StorageAccountsOperations,
StorageAccountsOperations.check_name_availability.__name__,
mock_check_name_availability,
)
result = AzureSDK().check_name_available("mock-name")
assert result
def test_check_name_error(monkeypatch):
def mock_check_name_availability(*args):
raise HttpResponseError("Mock error checking name availability")
monkeypatch.setattr(
StorageAccountsOperations,
StorageAccountsOperations.check_name_availability.__name__,
mock_check_name_availability,
)
with pytest.raises(HttpResponseError) as e:
_ = AzureSDK().check_name_available("mock-name")
assert e
Now, when we run the test, the test will run the “patched” method in place of the specified external requirement. Hooray! Our unit tests won’t reach out to the network, touch real APIs, or change data in any way. AND we can build testable outputs for all of the methods!
When replacing/mocking/patching an existing method, we do run the risk of replacing real bugs with our “good” code. But, there are a several options we can use to reduce our potential risk.
Between the two of the (and our unit tests), we have drastically increased the confidence that our code works, and increased that confidence quickly.
Mocking and Monkey Patching are excellent patterns to simplify unit testing when relient on external, “uncontrolled” packages. Especially if these packages make API calls, touch data, or take a long time to execute. Python itself, and many testing frameworks, natively support these concepts and we should also consider using them when appropriate!
While these concepts are originally specific to software development, the positive impact they can have on security operations is tangible: