Just a reminder that I have worked in the IT field for 25+ years (ok everyone say it. you are old.) and in that time there are things that come up often. This time it was a production outage of one of our internal build software products. The developer’s immediate reaction was “Nothing changed on our side” When something stops working the immediate reaction is ‘it worked yesterday.’ It could be your car, the fridge, your cell phone, anything. In technology the notion that if a piece of software is in production that means it is bug free. One side note, in the 90s I heard a story from a large tech company that bug free was designated on software if something had less than 50,000 known bugs.
On this day we had a production issue with something that had been running for a long time. The first thing most managers ask (including myself) is “has anything changed?” And in a chorus what do you think the answer is? “Nothing on our side…” The first thing about that statement is pushing blame on someone else. It immediately is saying that you had nothing to do with something ‘you’ did.
The next question is, has anything changed on outside systems etc. This could be the operating system, user desktop, upstream feeds etc. Of course, in many cases there usually are a few changes. There are always security fixes to server and desktops, so changes happen. Upstream systems can do updates (hopefully they test with you but sometimes they do not) but that is always a possibility.
Next step in debugging, can we reproduce it in a test environment? Try to figure out how to setup a test environment exactly like production and see if you can replicate the error. Sometimes it works, sometimes it does not. But hopefully it will help the team figure out what is wrong or point in the direction of debugging. If you are lucky and have unit tests you can run them, and if you are lucky (I am not) you have UI automated tests and can run those. All to see if you can find the issue.
Now if you are not bored already as technology people are like “tell me something I do not know” and non-technology people are thinking “what does this have to do with me.” Read on, and it will.
In the beginning I mention that every day people see this when their fridge breaks, the car stops working or the cell phone dies. What I did not say is about yourself. When you are not feeling great, injured, sick, struggling or just not working the way you should what are the steps? Our first reaction is often blaming some outside force. We can blame work, our partner, our kids, something we ate etc. Our mind instantly defends ourselves and looks for the outside force that is to blame.
I mentioned previously being wrong is like being punched, and as soon as you feel a certain way your immediate reaction is to figure out what punched you. I also stated previously when I take my dog to the veterinarian when she is not well, they often ask ‘What are you feeding your dog?’ but a doctor only asks what is wrong. Where am I taking you?
As this blog is introspective, I am asking this about myself. I found myself not in the shape I wanted to be, and I stated the list of excuses: I was sick, my daughter was asking me to go for ice cream a lot, I was going out to eat a lot with friends/family etc. Where I had to back away from my first thought and dig deeper. I was the one picking my meals (sick or healthy) I was the one who chose to get ice cream and they are all choices I made. Comparing myself to the production piece of software, my body was the same thing as it was long ago. The change that was going on was the choices I was making.
I realized something simple is that I should not judge myself on my first thought. That I need to be more responsible for my second thoughts. My second thought was how I did this before and started to make those changes again. And in making those changes I found myself heading back in the direction I wanted. I am not there yet, but I am chugging away at it.
Now back to the technology issue, there was code written a long time ago and as would be our luck had no unit tests as well as difficult to replicate. The problem was with our code. The team’s first thought was to blame others (nothing changed comment) but our next actions were to dig and find if it is. I know people often get frustrated, but getting the team away from the defensive mode into the debug mode is what a good manager does.
The next time you hear the following with something goes wrong “Nothing has changed on our side.” Do not make yourself responsible for your first thought, but hold yourself responsible for your second, third etc. thoughts. And it is those thoughts that need to drive you to solving problems.
This opinion is mine, and mine only, my current or former employers have nothing to do with it. I do not write for any financial gain, I do not take advertising and any product company listed was not done for payment. But if you do like what I write you can donate to the charity I support (with my wife who passed away in 2017) Morgan Stanley’s Children’s Hospital or donate to your favorite charity. I pay to host my site out of my own pocket, my intention is to keep it free. I do read all feedback, I mostly wont post any of them.
This Blog is a labor of love, and was originally going to be a book. With the advent of being able to publish yourself on the web I chose this path. I will write many of these and not worry too much about grammar or spelling (I will try to come back later and fix it) but focus on content. I apologize in advance for my ADD as often topics may flip. I hope one day to turn this into a book and or a podcast, but for now it will remain a blog. AI is not used in this writing other than using the web to find information.Images without notes are created using and AI tool that allows me to reuse them.