Use Test Plans to become a more effective Software Developer

Shipping high-quality software is the responsibility of each software developer. Not only are the days when handing untested code to the QA team for validation a common practice long gone, but many companies have also moved away from QA-based testing, making developers fully own the quality of the product. This creates a problem: how can you ensure developers do their due diligence and validate their changes? At Meta (a.k.a. Facebook), we use “test plans.”

What are test plans?

Test plans describe how authors tested their changes. They are an integral part of Meta’s code review process: the code review tool does not allow submitting code for review if the test plan is empty.

While it is common for test plans to say: “unit tests,” many are much more interesting and often include:

  • screenshots showing the UI before and after the change
  • videos showing the change in action
  • API requests and corresponding responses (e.g., JSON payloads)
  • dashboard snapshots from canary runs
  • terminal printouts
  • Funny memes – especially if the testing was not comprehensive or applicable (e.g., auto-generated code, changes to unit tests only, etc.)

Benefits of requiring Test Plans in commits

The most obvious benefit of requiring test plans is forcing developers to think about validating their changes. While not all test plans are comprehensive, most are decent.

Occasionally, test plans reveal that the code change does not work as the author intended. I once proudly included a graph hovering a little below 100% as my test plan, only for a reviewer to point out that my graph represented not the success rate, as I claimed, but the error rate.

However, one often overlooked benefit of requiring test plans is that they can be lifesavers when working with unfamiliar code.

Imagine you are tasked with building a new feature in a mobile app. While working on the feature, you discover that the API powering your app doesn’t return all the necessary information. You can ask the API team to add it, but they may not be able to accommodate your request on short notice. Perhaps it would be faster if you implemented this change yourself. The only problem is that you are not familiar with the backend code. You don’t know how to test it to ensure you didn’t break anything. In these situations, test plans can come in extremely handy. You can check the commit history of the code you want to change and see how developers who regularly contribute to it test it. In addition, checking past test plans may help you discover edge cases you need to consider. I successfully used this strategy multiple times to change an unfamiliar codebase I would otherwise be afraid to touch.

“My unit test coverage is 100%”

Test plans should not be considered a replacement for unit tests. As great as unit testing is, it is not always sufficient. Additional end-to-end validation helps confirm that the changes worked as intended outside of the isolation provided by unit tests and that they didn’t introduce unwanted behavior. I have seen (and caused) situations where my application wouldn’t start even though all unit tests were passing.

Call To Action

Including test plans in pull requests is not a common practice, let alone a requirement, in most companies or teams. Despite that, I encourage you to follow this practice. Your team members will notice it and may start doing the same if they find it useful. And even if they don’t, you will still benefit from this habit. Going the extra mile can help you find issues before they impact users. With time, it will get easier because you can reuse past test plans to validate some of your current changes. You may need to tweak them a little, but you won’t have to start from scratch each time.

Why Should You Care About Minimal Reproducible Examples (and how to create one)

You’ve spent hours debugging a tricky bug. You can reproduce it but can’t quite figure out the root cause. You’re starting to believe that the bug might not be in your code, but in the library, you are using. Given how much time you’ve already spent on this investigation, you are getting desperate and want to ask for help. What’s the best way to do it? Create a minimal repro!

What’s a minimal repro?

Minimal repro, sometimes called Minimal Reproducible Example, is a code snippet reproducing a bug and providing context with as little code as possible. In the ideal case, another person should be able to copy the code and run it on their machine to reproduce the bug successfully. This is often impossible to achieve, but the closer to this ideal, the better.

How to create a minimal repro?

Creating a minimal repro requires some thought. It’s not about code-golfing. The minimal repro should be as little as possible, but it should also retain all necessary context. Here are a few tips:

  • Remove any code that is not needed to reproduce the issue, but make sure that your example still compiles
  • Avoid changes that make code shorter at the expense of understandability – e.g., don’t shorten the names of variables if it makes code harder to comprehend
  • Keep only important data – if your array has 1 million items but you need only two items to reproduce the issue, only include these two.
  • Remove any artifacts, like configuration files, that are not needed to reproduce the issue. If possible, set all mandatory options all inputs directly in the code.
  • Reduce the additional steps needed to reproduce the issue to the absolute minimum.

Pro tip: Occasionally, instead of removing unneeded code to isolate the issue, it is better to start a new project and try to write code replicating a bug from scratch.

Why create a minimal repro?

Surprisingly, the main benefit of creating a minimal repro is not making a code snippet that you could use to ask for help. Rather, ruthlessly eliminating noise helps build a deeper understanding of the problem and frequently leads to finding the root cause of the issue and a proper fix.

If creating a minimal repro didn’t help you figure out the cause of the bug and a fix, you have something you can use to ask for assistance. You can share your example with your teammates, post it on StackOverflow, or include it when opening a GitHub issue. This is where minimal repros shine – they are critical in getting help quickly.

During my time at Microsoft, I was one of the maintainers of EntityFramework, Asp.NET Core, and SignalR repos. As part of my job, I investigated hundreds of issues reported by users. Clean, concise repros were one of the main factors deciding whether or not an issue was resolved quickly.

For most reported issues containing a concise, clean repro, engineers needed a glance to determine it was indeed a bug. If it was, they could often find the culprit in the code in a few minutes. Finally, they frequently used the example included in the bug report to create a unit test for the fix.

Reports with convoluted or incomplete examples dragged on for weeks. Building a repro often required multiple follow-ups with the author. Due to excruciatingly slow progress, these bug reports had a higher abandon rate.

The bottom line is that you will get help faster if you make it easy to give it. Minimal repro is an effective way to do this.

A Powerful Git Trick No One Knows About

Here is a Git trick I learned a long time ago that I can’t live without (and when I say no one knows about it, I mean it – it is not well documented, and no developers I have worked with knew about it):

Some git commands take - (dash) as the reference to the previous branch.

git checkout and git merge are two commands I use it with all the time.

git checkout - switches to the previous branch. This makes toggling between the two most recently used branches quick and super easy:

Using git checkout -

git merge - merges the previous branch to the current branch. It is especially powerful when combined with git checkout -. You can switch to the target branch and then merge from the previous branch like this:

Using git merge -

One command I wish supported - is git branch -d. It would make cleaning branches after merging effortless. Presumably, this option is not available to prevent accidentally deleting the wrong branches.

Bonus trick

While we are at it – did you know that the cd (change directory) shell command also supports -? You can use cd - to toggle between the two most recent directories.

Don’t let “later” derail your software engineering career

One thing I learned during my career as a software engineer is that leaving unfinished work to complete it “later” never works.

Resuming paused work is simply so hard that it hardly ever happens without external motivation.

Does it matter, though?

If the work was left unfinished not because it was deprioritized but because it was boring, tedious, or difficult, then more often than not, it does matter. The remaining tasks are usually in the “important but not urgent” category.

What happens if this work is never finished?

Sometimes, there are no consequences. Occasionally, things explode. In most cases, it’s a toll the team is silently paying every day.

From my observations, the most common software engineering activities left “for later” are these:

  • adding tests
  • fixing temporary hacks
  • writing documentation

Adding test “later”

Insufficient test coverage slows teams down. Verifying each change manually and thoroughly takes time, so it is often skipped. This results in a high number of bugs the team needs to focus on instead of building new features. What’s worse, many bugs are re-occurring as there is no easy way to prevent them.

I don’t think there is ever a good reason to leave writing tests for “later.” The best developers I know treat test code like they treat product code. They wouldn’t ship a half-baked feature, and they won’t ship code without tests – no exceptions.

Fixing temporary hacks “later”

Software developers introduce temporary hacks to their code for many reasons. The problem is that there is nothing more permanent than temporary solutions. “The show must go on,” so new code gets added on top of the existing hacks. This new code often includes additional hacks required to work around the previous hacks. With time, adding new features or fixing bugs becomes extremely difficult, and removing the “temporary” hack is impossible without a major rewrite.

In an ideal world, software developers would never need to resort to hacks. The reality is more complex than that. Most hacks are added for good reasons, like working around an issue in someone else’s code, fixing an urgent and critical bug, or shipping a product on time. However, the decision to introduce a hack should include a commitment to the proper, long-term solution. Otherwise, the tech debt will grow quickly and impact everyone working with that codebase.

Writing documentation “later”

Internal documentation for software projects is almost always an afterthought. Yet, it is another area that, if neglected, will cause team pain. Anyone who has been on call knows how difficult it is to troubleshoot and mitigate an issue quickly without a decent runbook.

In addition, documentation also saves a lot of time when working with other teams or onboarding new team members. It is always faster to send a link to a wiki describing the architecture of your system than to explain it again and again.

One way to ensure that documentation won’t be forgotten is to include writing documentation as a project milestone. To make it easier for the team, this milestone could be scheduled for after coding has been completed or even after the product has shipped. If the entire team participates, the most important topics can be covered in just a few days.

How does “later” impact YOU?

Leaving unfinished work for “later” impacts you in two significant ways. First, it strains your mental capacity. The brain tends to constantly remind us about unfinished tasks, which leads to stress and anxiety (Zeigarnik effect). Second, being routinely “done, except for” can create an impression of unreliability. This perception may hurt your career, as it could result in fewer opportunities to work on critical projects.

Top 5 Jokes Software Developers Tell Themselves and their Managers

Software developers are boring. Not only do they keep repeating the same jokes all the time, but they also take them very seriously!

Here are the most common ones

I will add unit tests later

“I will add unit tests later” is one of the most common jokes software developers tell. In most cases, they believe it. But then reality catches up, and tests are never added. If you were too busy to add unit tests when it was the easiest, you won’t have time to make up for this later, when it becomes more difficult, and you might even be questioned about the priority and value of this work.

I am 99% done

I am terrified to hear that someone “is 99% done,” as if it were positive. I have seen too many times how the last 1% took more time and effort than the first 99%. Even worse, I participated in projects that were 99% completed but never shipped.

My code has no bugs

Claiming that someone’s code has no bugs is a very bold statement. I did it several times at the beginning of my career, only to be humbled by spectacular crashes or non-working core functionality. I found that saying: “So far, I haven’t found bugs in my code. I tested the following scenarios: …” is a much better option. Listing what I did for validation can be especially useful as it invites inquiring about scenarios I might have missed.

This joke is especially funny when paired with “I will add unit tests later.”

No risk – it’s just one line of code

While a one-line change can feel less risky than bigger changes, that doesn’t mean there is no risk. Many serious outages have been caused by one-line configuration changes, and millions of dollars are lost every year due to last-minute “one-liners.”

Estimates

Most of the estimates provided by software development are jokes. The reason for this is simple: estimating how much time even a software development task is more art than science. Trivial mistakes, like misplaced semicolons, can completely derail even simple development tasks. Bigger, multi-month, multi-person projects have so many unknowns that providing accurate estimates is practically impossible. From my experience, this is how this game works:

  • software developers provide estimates, often adding some buffer
  • their managers feel that these estimates are too low, so they add more buffer
  • project managers promise to deliver the project by some date that has nothing to do with estimates they got from engineering
  • the project takes longer than the most pessimistic estimates
  • everyone keeps a straight face

Bonus

“My on-call was pretty good! I was woken up only three times this week.”