Michele Martone

About Resume Contact Me

I recently walked through my approach for testing a new service with someone new to testing. Substitute service for library, application, or any other distinct piece of code. A lot goes into how I think about and draw opinions on the quality of the service.

Assumptions:

• I have access to the code
• I know nothing about the service

This is not always true, and in fact while I always expect access to the code, I usually also always expect to know something about the service. Being involved early is important, but that’s another blog post. For this post I’m going to assume no knowledge.

Code structure and README:

I first look at the code structure and any documentation. I check out what language the service is built in. The quality of the documentation tells me some information about the quality of the service. Is the service’s purpose explained? Are there instructions on how to run the service and what dependencies are needed? How to run tests? If there is an api, is that documented with some kind of spec file? Is the code organized well?

Run some things:

Next I’ll use the documentation to run the service. Is the documentation accurate and running simple? After that I’ll run any tests. Do they pass and run quickly?

Some times, I’ll get stopped right here because the service isn’t easy to run, or the tests don’t run, or do not pass locally. That’s usually not a good sign.

Deeper dive into the tests:

I inspect the tests at the code level next. What language and libraries do they use? What is the structure of the tests and do they follow good coding standards? What types of tests exist? For a service with an api, I’d expect api tests and unit tests. If there is a UI, I’d expect some, but not too many UI (end to end) tests. If there is a database I’d expect to see it mocked for the most part, but some unmocked, and the same with other dependencies like a queuing system or event bus. Are the tests well described? To me tests provide documentation of the system as much as other text documentation, so I want to see tests that are described with detail.

I would also love to see some type of performance or load test performed with included statistics, especially if the service is going under heavy usage. In my experience this is rare to see though.

I may bring out Postman or Insomnia and make some api calls myself to see what happens with the running application. Most of the time this also helps educate myself on the service under test. But I especially may do this to find edge cases that may be un-tested.

If there is a UI I’ll also manually run through that to see how it works from a user perspective. I will also look for edge cases.

I may have some opinion on technical choices of libraries used to create tests. This is one reason I like testers to be involved early. If I think there is a better way to accomplish things I’ll mention that to the development team.

Static analysis:

Is it being performed locally or ci?

Code coverage:

I check to see if code coverage is measured. If not it’s usually simple to add it and I’d do that. From the coverage report I’d check to see if the majority of the service is tested within unit tests. I’d also prefer to see the service tests set to fail if coverage dips below a certain level. That’s usually a setting you can add to whatever test runner is used.

Logs/Observability:

I check that there are logs and they explain what is happening, especially in the case of errors. If the company uses a logging standard, do the logs comply with that standard?

CI and deployment:

After that I turn to how the application is built and tested in ci. I would expect all tests that I found to be running in ci as well. In addition, I would also expect some kind of smoke test to verify minimal functionality after the application is deployed to a real environment. Hopefully it goes without saying but the application should not deploy if any tests fail.

Metrics, monitoring and SLOs:

If this application makes it to production, I’d like to see that there is proper alerting for things that may go wrong. To accomplish this, the service should be emitting metrics of it’s core functions. Alerts can then be built off those metrics and inform the right people. I also look for any dashboards that the service is running properly. Finally I look to see if there are uptime and other important SLOs defined, captured and alerted on if not met.

Closing:

By the time I’m done with this inspection, I know a lot about the service. I will have a very good idea about the quality of the service and where it needs work, if at all. I can then give kudos to the developers, or work with them to improve.

If I was involved in the beginning, these are the things that I would be asking and contributing as the service is developed.

It occurs to me that this type of service inspection is highly correlated to what I used to do as an auditor in my accounting career. I used to be this young punk going into this big bank and telling executives how to run their business. I would look at systems and processes with a critical eye and make sure everything was above board and explained.