Getting fired for a debug “print” statement?
Did I ever tell you about that one time I took down the customer portal for a seven figure SaaS business with an accidental debug print statement?
It was easily one of my most uncomfortable moments in my developer career.
I had just joined this team maintaining and building a big Python web app that was the core part of this SaaS company I worked for.
In my first few days on the team I’d worked through some smaller tickets to familiarize myself with the code base. Pretty straightforward web dev stuff: extend a REST API here, sort these query results differently there.
Things were going fine and I felt right on track for demonstrating to my colleagues that I wasn’t a bozo…
Until I pushed this small and unsuspecting deploy to production…
Things looked fine for a moment. Then,
…this spike of “500 Internal Server Error” messages shows up in our monitoring dashboard.
Turns out my “small and unsuspecting” deploy to production had still contained a debug “print” statement that I forgot to remove.
Everything had worked fine while testing locally and on the staging system.
At the time the company I worked for had partners in Japan. The system used a unicode text encoding to represent text fields like first and last names in the database.
This was all fine and dandy, but when my newly added “print” statement started printing these Japanese names with unicode characters to the logging console on stdout, the Python interpreter freaked out:
The I/O encoding had been set up to use ASCII…and ASCII can’t represent Kanji characters like 大 or 翔.
Because it was a fairly “hot” request handler that contained the print statement, the whole app came crashing down in a hail of “UnicodeDecodeError” errors.
The load balancer decided to take the faulty app instance out of rotation, but of course the exact same thing happened to its twin brother.
A few seconds later the whole system was practically DOWN as core functionality became unreachable for our customers.
After we realized what was going on we quickly used Git to track down the commit where the print statement was added. We removed the offending line and brought the system back up.
I wasn’t fired for that screw-up but boy did I feel embarrassed in front of my team.
And I swore to myself I’d never deploy (or even commit) an accidental “print” statement again.
Some time later I found this great little Sublime Text plugin that can help you avoid this kind of (potentially career-ending) mistake: it’s called GitGutter.
GitGutter shows you which lines of a file were modified compared to the latest commit in Git. Right inside your Sublime Text editor window.
A heads-up display for Sublime Text
The greatest benefit of GitGutter is that it makes you more aware of how you change your code. It adds helpful but unobtrusive Git diff icons to the gutter area (that little margin on the left of the text area).
To give you an example, in the screenshot above you can easily see that lines 96-98 were added (green plus marks). Some lines around line 103 were removed (red arrows), and lines 105-108 were modified (bluish squares).
This allows you to quickly see the state of your file as you are editing it—the extra awareness helps prevent accidentally committing an unwanted change, like a debug “print” statement or an accidental debugger breakpoint.
Selectively roll-back file modifications
GitGutter has another cool feature that lets you selectively roll-back file modifications right inside Sublime Text. This means you don’t have to switch to another tool, like a Git client, to make changes like that.
Of course you can use the full power of Sublime’s undo and redo functionality with GitGutter to undo and re-apply modifications as you see fit. In the other Git tools I tested, working with file modifications never felt as fluid as it did with GitGutter.
GitGutter is one of the packages that I have installed and activated in my Sublime Text at all times. It’s fast, pretty, and stable. I also like the philosophy behind it—GitGutter has a small footprint. It does one thing only and it does it well. I’m not a fan of ginormous plugins that add a ton of features I never use.
I highly recommend GitGutter if you use Git in your development workflow. You can install GitGutter via Package Control. If you’re not using Git for source control you might want to check out VCS Gutter as an alternative. It’s a fork of GitGutter that supports Mercurial and Subversion.