By Dan Bader / December 8, 2016

Getting fired for a debug “print” statement?

Did I ever tell you about that one time I took down the customer portal for a seven figure SaaS business with an accidental debug print statement?

It was easily one of my most uncomfortable moments in my developer career.

I had just joined this team maintaining and building a big Python web app that was the core part of this SaaS company I worked for.

In my first few days on the team I’d worked through some smaller tickets to familiarize myself with the code base. Pretty straightforward web dev stuff: extend a REST API here, sort these query results differently there.

Things were going fine and I felt right on track for demonstrating to my colleagues that I wasn’t a bozo…

Until I pushed this small and unsuspecting deploy to production…

Things looked fine for a moment. Then,

KABLAMO! –

…this spike of “500 Internal Server Error” messages shows up in our monitoring dashboard.

Uh oh.

Turns out my “small and unsuspecting” deploy to production had still contained a debug “print” statement that I forgot to remove.

Everything had worked fine while testing locally and on the staging system.

At the time the company I worked for had partners in Japan. The system used a unicode text encoding to represent text fields like first and last names in the database.

This was all fine and dandy, but when my newly added “print” statement started printing these Japanese names with unicode characters to the logging console on stdout, the Python interpreter freaked out:

The I/O encoding had been set up to use ASCII…and ASCII can’t represent Kanji characters like 大 or 翔.

Because it was a fairly “hot” request handler that contained the print statement, the whole app came crashing down in a hail of “UnicodeDecodeError” errors.

The load balancer decided to take the faulty app instance out of rotation, but of course the exact same thing happened to its twin brother.

A few seconds later the whole system was practically DOWN as core functionality became unreachable for our customers.

After we realized what was going on we quickly used Git to track down the commit where the print statement was added. We removed the offending line and brought the system back up.

I wasn’t fired for that screw-up but boy did I feel embarrassed in front of my team.

And I swore to myself I’d never deploy (or even commit) an accidental “print” statement again.

Some time later I found this great little Sublime Text plugin that can help you avoid this kind of (potentially career-ending) mistake: it’s called GitGutter.

GitGutter shows you which lines of a file were modified compared to the latest commit in Git. Right inside your Sublime Text editor window.

A heads-up display for Sublime Text

The greatest benefit of GitGutter is that it makes you more aware of how you change your code. It adds helpful but unobtrusive Git diff icons to the gutter area (that little margin on the left of the text area).

 gitgutter-example

To give you an example, in the screenshot above you can easily see that lines 96-98 were added (green plus marks). Some lines around line 103 were removed (red arrows), and lines 105-108 were modified (bluish squares).

This allows you to quickly see the state of your file as you are editing it—the extra awareness helps prevent accidentally committing an unwanted change, like a debug “print” statement or an accidental debugger breakpoint.

Selectively roll-back file modifications

gitgutter-rollback

GitGutter has another cool feature that lets you selectively roll-back file modifications right inside Sublime Text. This means you don’t have to switch to another tool, like a Git client, to make changes like that.

Of course you can use the full power of Sublime’s undo and redo functionality with GitGutter to undo and re-apply modifications as you see fit. In the other Git tools I tested, working with file modifications never felt as fluid as it did with GitGutter.

GitGutter is one of the packages that I have installed and activated in my Sublime Text at all times. It’s fast, pretty, and stable. I also like the philosophy behind it—GitGutter has a small footprint. It does one thing only and it does it well. I’m not a fan of ginormous plugins that add a ton of features I never use.

I highly recommend GitGutter if you use Git in your development workflow. You can install GitGutter via Package Control. If you’re not using Git for source control you might want to check out VCS Gutter as an alternative. It’s a fork of GitGutter that supports Mercurial and Subversion.

About the author

Dan Bader

Hey, I’m Dan and I love helping developers take their coding skills and productivity to the next level. I’m an independent software engineer, productivity nut, author, and speaker. Check out my Python tutorials and videos.

10 comments
Alexis - December 8, 2016

What about unit and regression tests?, you guys have a test team or are pretty much up to the developer to check if is working ok? integration team?… dude, Jenkins?, what about the architects or designers to take in count about supporting different charset? my point is, that’s a team fault, developer are humans. BTW GitGutter, awesome.

Reply
    Dan Bader - December 9, 2016

    Hey Alexis, you’re spot on – one of the next developments in that story was to add a bunch of regression tests for this particular issue 🙂 Glad you like GitGutter!

    Reply
Aleph Melo - December 8, 2016

Nice one, bro.

Reply
    Dan Bader - December 9, 2016

    🙂 Thank you, Aleph!

    Reply
brad - December 8, 2016

Sorry but this is not your fault, the company should have had testers at least that would of seen this’s. Not only that you can have static code test that check for these sorts of mistakes that must be run before something goes live.

Reply
    Dan Bader - December 9, 2016

    Hey Brad, thanks for your support 🙂 I agree there are so many ways this situation could’ve been prevented. Like a staging environment with real (but anonymized) data, for example. Or just a bunch of test cases that exercise the unicode support… If I remember correctly what made this particular issue so challenging was that otherwise the unicode support in the app worked fine, it was just the print() call that blew up because the stdout encoding wasn’t set correctly.

    Reply
Jamie - December 8, 2016

GitGutter is one of my absolute favorite addons. I had no idea that it had the popup functionality… I thought it was simply a red/green indicator.

I’m also in love with GitSavvy.

Reply
    Dan Bader - December 9, 2016

    Hey Jamie – yeah, it took me a while to discover the popup, too 🙂 But it’s awesome. Saves so much time not having to switch to a Git client to roll-back changes. Definitely also one of my favorite ST plugins.

    I’ve got to check out GitSavvy, thanks for the recommendation!

    Reply
Michael - December 9, 2016

Nice to read an I can fully understand your embarrassment – though it could have been prevented easily as you now 😉
It’s cool this corp does learn from mistakes and is not firing anyone for such a little mistake with so much unexpected effect 🙂
GitGutter looks like a really nice one! Sublime is awesome anyway – lets increase it to overawesome!

Reply
Vinicius Coelho - December 10, 2016

Hello, I’m struggling to find a package for Sublime that can do auto imports (or at least suggest what to import) when working wih Python and Django. Any recomendations would be appreaciated. Thanks!

Reply
Click here to add a comment

Leave a comment: