Using mental models to think about software

No person can function properly without building mental models to understand the world around them. The reality is simply too complex to deal with as is, so we need to abstract it.

Software is no different. The complexity of software systems has been only growing. As a result, the ability to quickly build mental models and use them to reason about these systems is an important software engineering skill.

Building good mental models is important because they allow us to better communicate with other software engineers, explain our ideas, and understand the impact of our changes on our software.

Building mental models

Reasoning about software systems requires building mental models that represent them. The granularity of these models depends on their purpose. Often, one level of granularity is not sufficient. The most effective software engineers can quickly build a few models and switch between them depending on the situation.

For example, if your team owns a few services, you can draw a lines-and-boxes diagram where boxes represent services and lines represent dependencies. You can then zoom in and build a model for each service. These service-level models could illustrate interactions between the libraries a service consists of. If you want more details, you can create a class diagram. You can continue zooming in and focus on methods, code blocks, statements, etc.

Each of these models describes your system at a different level of granularity and has a unique set of applications. The high-level model could be useful to troubleshoot larger outages (or when talking with your director) but is unlikely to help you fix a small bug. The more detailed models are best suited for solving gnarly issues but won’t be helpful when explaining your infrastructure to other teams.

Building models covering different aspects of the same system is also common. If you want to analyze your system from the security perspective, your model will include different details than when focusing on performance.

Caveats

Mental models are so natural to people that we often forget about their flaws.

All models are wrong

Models, by definition, ignore details. As they only capture certain aspects of reality, they are inaccurate. Furthermore, some relevant information is often omitted because it doesn’t fit the model. Edge cases are an excellent example.

For instance, when software developers explain how their code works, they rarely mention error cases. They focus on their ifs and fors and the program flow but omit exceptions because exceptions make their model murkier. The problem is that error scenarios are important. Incorrect or missing error handling is a common cause of system outages.

Models get more wrong with time

The world, including our software systems, is in constant flux. But mental models don’t automatically keep up with changing reality. Outdated mental models lead to misunderstandings and bad decisions.

I experienced this very problem recently. I started working on a feature that depended on a system I had never seen changing. Everything was going swimmingly, and I only needed to tie up a few loose ends related to some new data requirements. But when doing this, I discovered, to my dismay, that the system I depended on had recently changed. It had been updated to accommodate the same data requirements I struggled with. Making my feature work now required additional information I didn’t have. Plumbing this data meant revisiting my implementation. Working off of an outdated mental model cost me implementing my feature twice.

No two models are identical

Building mental models requires deciding which details are important depending on the purpose of the model. However, even if the purpose of the model is well understood, different people will consider different information relevant.

Also, mental models built at different times will naturally differ because they capture different system versions.

The interesting fact about this phenomenon is that the overlap between mental models built by different people is usually significant. The differences are often discovered unexpectedly, e.g., when discussing small but important details.

How to quickly ramp up on new codebases

Joining a new team is intimidating. There is always much to learn – new people, new processes, and a new code base.

Ramping up on your new team’s code base is difficult, but knowing how to work effectively in it is crucial to your success. So, how do you do it quickly?

My mid- and early senior developer years were intense. Due to a mix of reorgs and personal interests, I found myself on a new team every year or so. As a result, I had to learn new codebases in quick succession. They included .NET System.Xml, OData, Entity Framework, Entity Framework Designer, ASP.Net SignalR, ASP.Net Core, and the Alexa mobile app, and most of them were over one hundred thousand lines.

The first couple of transitions were slow and overwhelming. But they helped me develop a strategy I used later to quickly get up to speed on new codebases.

Get your hands dirty ASAP

The sooner you start working actively with the code base, the sooner you will get productive. Reading documentation can be helpful, but nothing can replace the hands-on experience.

When I joined Amazon, I asked my manager to assign me a simple bug to fix on my first day. Even though the fix amounted to a single line of code, it took me a few days to send it for review. It might seem long, but this quest was not about fixing the bug. Rather, it was about forcing myself to set up my development environment, learn how to build and run the code, write unit tests, and debug our product.

Review code

Each team has its way of letting members know there is a new PR (Pull Request) for review. It could be email, chat, or review tool notifications. Whatever it is, subscribe to this channel and start reviewing PRs. You won’t understand much initially, but you may still catch some bugs (e.g., off-by-one errors). More importantly, these reviews will allow you to ask questions and gather a broader context of what the team is working on.

Identify code that matters

The 80/20 rule does apply to codebases. This is especially noticeable in the bigger ones, where most code changes developers make are concentrated in one area. Knowing which code it is allows you to focus your ramp-up on the area you are likely to work on soon.

There are a couple of easy ways to tell which code is in the top 20%:

paying attention to code reviews
checking the commit history

Take notes, draw diagrams

I discovered that taking notes and drawing diagrams is a very effective way to grok the most complex parts of the code. Call graphs, class diagrams, and dependency graphs all help organize the information and are great reference material in case you need to refresh your memory.

After I joined Amazon, I drew diagrams of a few areas of code I couldn’t understand. They became an instant hit. Even people who had been on the team for much longer than me wanted a copy.

Debug code

Reading a new codebase is like reading a book in a language you barely know. You can do it, but it is excruciating.

For me, one of the best ways to overcome this is to step through the code with the debugger. Initially, I have no idea what I am looking at. But if I follow the same code path a few times, I begin to recognize code I have already seen, and soon, everything starts to fall into place.

To get the most out of my debugging sessions, I do two things:

I continuously inspect the state – the stack trace, parameters, and local and class variables
I take notes and draw diagrams

Using the debugger to learn the code base is slower and narrower than reading the code. But it is also much deeper. In fact, when debugging, I often realize that the understanding I gained from reading the code was incomplete or sometimes even incorrect.

Read documentation

Reading documentation can help accelerate your onboarding. It could be especially useful when it comes to the high-level architecture and concepts that are hard to deduce from code.

However, you should take documentation with a grain of salt. It is often sparse and outdated. But this could be good news for you. Updating the documentation as part of your ramp-up could be a great contribution to your new team.

Join on-call rotation

Joining the on-call rotation to accelerate team onboarding might sound extreme but I did it on my first team at Facebook. I did it because I was concerned that my ramp-up was slow, so I decided to push myself. I learned more about our services this week than in weeks before. After my shift ended, I knew what services our team owned, their dependencies and where to find their code. Alerts immediately pointed me to the hot code paths, and troubleshooting issues forced me to dig into the code.

The Must-Have Skill Every Senior Developer Needs

Writing code is a fundamental skill every junior software developer needs to master. However, coding skills are no longer the biggest differentiator at senior and above levels. Every senior engineer is expected to have solid coding skills, and growing to higher levels based on coding alone is rare.

If not coding, then what?

If coding is not the skill to grow beyond senior levels, what skill is it?

This question has no correct answer, as no single skill can elevate you to the staff+ levels.

However, software development is a team sport, and successful senior engineers must focus on many areas besides coding. They are often responsible for projects spanning one or more teams. They drive the design, collaborate with partner teams, communicate progress, etc. Doing all this work effectively requires good communication skills, especially writing (which I consider one of the Universal Skills.)

Why writing?

Writing clarifies thinking and promotes the exploration of ideas. I don’t know how many times I thought I understood something, only to struggle to summarize it in writing. But once I succeeded, I had a much deeper grasp of the concept and noticed new insights I hadn’t considered before.

Thanks to its durability and asynchronous nature, writing is also a great way to scale. You can write something once and refer to it later. Your readers can benefit from it even if you are not around. Here are a few examples from the software engineering field:

Project execution plans are useful for aligning all interested parties: the team that will execute the project, partner teams, your manager, etc., without having to talk to them individually.
Documentation helps avoid explaining the same concepts again and again. It protects the team from scrambling when a key team member leaves the project or the team (see also: bus factor)
Design documents allow for gathering feedback without holding a meeting for all interested parties. They are also an invaluable resource to understand why certain design choices were made and what alternatives were considered.

Writing is difficult

Writing is not natural for most people. Making the content clear, concise, and well-organized is grueling work.

I often see software developers dismay when I ask them to write a rollout plan or a design doc. Some tell me they were relieved to graduate from college because it meant they would never have to write again, and I am shuttering their world.

There are also other reasons why writing is difficult. Many developers have to write in a non-native language. But even native speakers often struggle because the way of writing they learned at school does not serve them at work.

Opportunities to practice writing.

Even though writing becomes important gradually, it doesn’t mean you should wait to improve it. On the contrary, the sooner you start, the better. Fortunately, every developer has plenty of opportunities to practice writing on the job.

Emails

Emails are everyone’s bread and butter these days. However, many emails are hard to read and understand and, as a result, fail to achieve their goal.

In my first job, our manager asked us to send a weekly email summarizing what we worked on and accomplished in the past week. I was proud of my reports: they were very detailed and explained everything. Despite these emails, my manager kept asking me what I had been working on. When I saw one of these emails years later, I understood. He never read them. I couldn’t blame him – it was an unbearable wall of text.

Memos / Announcements

Posts, memos, and announcements meant for a wide audience need to be tailored to that audience. Otherwise, readers won’t understand them and will give up reading them.

I recently read a post from my co-worker reporting on the status of our project. The audience of this post was broad (more than 150 people) and included managers, directors, and partner teams. The technical details in this post left me lost despite my heavy involvement in this project. I can only guess what others took away from this post.

Design documents

Good design documents explain complex topics using simple language. This combination makes them hard to write, but the payoff is worth the effort. Confusing design documents lead to lengthy discussions, feedback on unimportant matters, and frustration.

I once asked a junior engineer to write a design document explaining how he plans to implement a feature we promised to deliver. What I got was an untitled Google Doc with no text and two pictures – a diagram and “The Starry Night” by van Gogh. While I have nothing against “The Starry Night”, the document didn’t give me the faintest idea about the design of the feature, assumptions, and considered alternatives.

Code review feedback

The main purpose of sending code for review is to gather feedback. But giving short, clear, and actionable feedback professionally is an art. The conclusion: if you want to improve your writing, you should review a lot of code (and provide feedback).

Code comments

I am not a huge fan of writing code comments, but in some situations, they are warranted. Unfortunately, many code comments are so poorly written that it is sometimes hard to tell if they are there to help you or make you more confused.

The main challenge with code comments is that they need to be short to not overshadow the code but must clearly explain intricate ideas that the code cannot express. These requirements make writing code comments good practice.

Documentation

Writing documentation is one of the least favorite tasks software developers want to do. Yet, it often is one of the most impactful they can do. Good documentation helps put out on-call fires faster, makes onboarding new team members easier, and reduces randomization caused by repeatedly answering the same questions. By writing documentation, you help your team achieve more and polish your writing skills.

Bug reports

If you want someone to do something for you, you need to make it as easy as possible for them to do it. If you don’t, what you are asking for will take a long time or will never get done.

This rule applies perfectly to bug reports. If you encounter a bug that blocks your work, writing a clear bug report dramatically increases the chances of getting the issue fixed. Despite this, many reported bugs are incomprehensible.

At Microsoft, I worked on a few high-profile open-source projects like Entity Framework or ASP.Net Core. As thousands of developers used our products, we received a decent number of bug reports. Unfortunately, we often couldn’t understand what issue was being reported, how to reproduce it, and the expected behavior. Following up on these issues was painful. The back-and-forth took weeks. The “bugs” slipped from release to release while we were waiting for the details we requested. Eventually, we closed most of these bugs without resolution as it was hard to prioritize them over other issues we could immediately investigate and fix.

The 3 categories of skills every software developer needs to know

100% of software engineers who don’t keep their skills sharp become obsolete[*]. Many who do keep their skills sharp also become obsolete. It all boils down to what skills they focus on.

I divide skills into three main categories: company-specific, job-specific, and universal. This categorization makes it easy for me to decide where to invest my time when it comes to skill development.

Company-specific skills

Each company has internal infrastructure, processes, and tools. Knowing them before joining the company is impossible, but everyone must learn them after joining.

Being strategic about what to focus on can save you a lot of time. Company-specific knowledge and skills are, by definition, not transferable. You need to learn them to do your job, but they become useless the moment you move on.

My strategy is to have a solid understanding of company-specific tools and processes but not dive too deeply unless I have to. Being in the dark will slow me down, but drilling into all the details is not much better. Most of these things constantly change, and I will quickly forget what I don’t use. Instead, my time is better spent developing other, more general skills.

When I worked at Microsoft, I had a colleague named Ben. Ben was the most knowledgeable person I knew when it came to the .NET Framework build system. Although he was not on the build team, he knew all the scripts, hacks, and environment variables used in this build system. Acquiring and keeping this knowledge fresh ate much of Ben’s time. Having Ben around was great for the team. We could (and constantly did) ping him for help with build-related issues. Eventually, Ben moved to a different team outside our organization. His expertise evaporated instantly, even though he didn’t go far.

Job-specific skills

Job-specific skills are usually the skills that get us hired. Companies look for people who can hit the ground running, and having skills in demand increases the chances of getting hired substantially.

Job-specific skills are transferable. You learned them before getting hired and may use them in your next job. There is a caveat, though. It is only true if you keep your skills up-to-date. Knowing React and knowing React-as-of-four-years-ago is not the same thing.

I learned that trying to keep skills sharp just by using them at work doesn’t always work. Companies rarely move as fast as technology. For instance, the product you work on might have been on the bleeding edge a few years ago. But it got stuck there because there was never a good enough business reason to migrate it to the newest framework version. You still need to maintain it but can’t use the latest features.

Another thing to pay attention to is the signs that the technology you specialize in is becoming obsolete. Some technologies are more durable, but some can fade fast. You don’t want to wake up one day only to realize that everyone except you has moved on.

CoffeeScript was one of the most followed projects on GitHub in the early to mid-2010s. It was incredibly popular, and there was a lot of hype around it. Today, hardly anyone remembers it.

Even if you got hired for your job-specific skills, it doesn’t mean you can’t learn new skills on the job. If you want to develop a new skill, you may consider joining a new project or moving to a different team. It isn’t always easy because you don’t have the skills they need, but it is doable – especially if you are known as someone who learns fast.

And the biggest secret: not only do you develop a new skill that may land you your next job, but you are also paid to do this.

Universal skills

Finally, there are the universal skills. These are skills that never become obsolete. You can use them at your current job, at your next job, or for non-work-related purposes. You can also apply them instantly.

These are skills like writing, effective time management, delivering great presentations, etc.

The biggest problem with universal skills is that developing them never seems urgent. As a software developer, you will not lose your job because you write lengthy emails. But eventually, you may hit a glass ceiling and realize that what inhibits you is not coding but the lack of these universal skills.

These 5 habits will make you a great code reviewer

High-quality code reviews are hard.

They are time-consuming and require significant mental effort.

Code reviewing is also not taught at school, and figuring it out on your own is hard work. In this post, I share five crucial habits all great code reviewers I know have in common.

Make the code review about the code

Code reviews are about the code, not the person who wrote the code. If you think the proposed change is incorrect or have suggestions to improve it, then, by all means, leave your comments (just remember to make them professional and high-quality). However, leave out comments that don’t relate to the code under review.

Understand the code and ask questions when in doubt

Understanding the code under review is the foundation of a solid code review.

It is also the most difficult part.

Whenever you have doubts about proposed changes, you should ask the author for clarification. You might not be aware of an assumption the author is making or need help understanding how their changes fit in. However, the difficulty in grasping the changes is also often a sign of a mistake.

Asking a question will prompt the author to explain their thought process. As a result, they will either answer the question and clear up your doubts or realize that something is indeed wrong and needs to be fixed.

Be clear about your expectations

Code reviews can generate a wide variety of comments. Some are nits that you would like to see fixed but are not real issues. Some, however, identify serious problems that must be corrected before merging. If you leave a comment, make sure that the author understands which category it falls into. Doing this will save you and the PR author time and frustration.

Sometimes, you may take on a PR that is outside of your area of expertise. You may realize it only after you already left some comments. You are now in a weird situation: the author expects you to finish the review and approve the PR, but you don’t feel confident you can. If this happens, instead of accepting the change you don’t understand, it is better to leave a comment recommending the author get a review from someone more familiar with the code they are changing.

Side note: reviewing code that is outside of your area of expertise is a good thing. Even if you don’t understand the change enough to approve it, you can still provide useful feedback, identify bugs, and learn something. Just make sure the author does not expect to get your approval.

Cross-check with the existing code

Code reviews show only code that changed. Most of the time, it is sufficient. But sometimes, to understand the change fully, you need to check how the new code works with the code not included in the review because it hasn’t changed.

This idea may be obvious to most developers, but I was surprised to meet some who had never considered checking the existing code when reviewing PRs. In my experience, the biggest surprises are caused by not what’s included in the PR but by what’s missing.

Occasionally, examining the existing code may not give you all the answers. The ultimate weapon for these situations is to use a debugger. You can check out the PR branch and step through the code. It should resolve all your doubts. I resort to a debugger very rarely. It’s almost always easier and faster to ask the author.

Resist the “stamp” pressure

The pressure to merge changes quickly for projects on tight timelines is high. And it only grows as the deadline nears.

During the end game, engineers enter a Pull Request frenzy. Eventually, due to the number of PRs, code reviews often become a bottleneck. So, the engineers try to unblock themselves by asking to “stamp the diff” (i.e., approve changes without looking).

On the one hand it is understandable – no one wants to miss the deadline. On the other hand, these are the times when code changes need even more scrutiny than usual. Due to the time pressure, most PRs are coded very hastily. The changes may not be validated thoroughly (or at all) and the stress only increases the likelihood of mistakes.

While a proper code review takes time, merging code with issues a review could have caught is more costly. At best, fixing the problem will require sending a new PR (which, by the way, will trigger a review). At worst, an embarrassing bug ships to customers.

I remember when one of my teams worked extremely hard to finish a project on time. We were very close, and then one of the team members dropped a 1000-line PR a few hours before the deadline. The manager was trying hard to find someone willing to approve the PR. It wasn’t easy because the PR had a bunch of red flags, like spotty test coverage or many TODO comments, but he eventually succeeded. As soon as our product shipped, we started getting reports from angry customers complaining that important scenarios stopped working. We found that the hastily merged PR was the culprit. The team scrambled to fix the issues, but the damage was done. This one PR cost us the reputation we’d been building for a long time.