The simplest ALM practices become impossible if you don’t have your culture right.
Remember the build light?
Back in the day, when continuous integration was a new thing, teams had a lot of fun setting up red-light-green-light on their automated builds. A developer checks in, the server compiles and perhaps runs tests, if good the light goes green, if bad the light flashes red and everybody knows the build is broken. Some teams had fun punitive measures for the developer who last broke the build. Name and shame, but keep it lighthearted: you get the rubber chicken at your desk until you can give it away.
CI builds are such second nature to teams now that, in my experience anyway, they’ve become ordinary and boring. We leaned into the pain of broken builds years ago, and our teams and our industry adopted practices and tools to keep them from breaking. (Gated check-in is an obvious one, but slightly different and I’m leaving it aside for now.) Something as simple as get latest before check-in (reverse-merge before merge), resolve, recompile, rerun unit tests, is a solid build hygiene practice and it works. My team at UW literally went about a year with no gated check-in and no broken builds. We didn’t notice it until we hired a new dev and forgot to tell him to do what had become second nature to us. (And then we couldn’t find the rubber chicken!)
If you’re reading this and saying “duh” so far, good for you.
Recently, I visited a customer that doesn’t have automated builds set up. At all. I know – it’s bad – that’s why they need an ALM consultant, though, innit?
This seems like an easy enough fix, right? I can demo setting up a CI build on a garden-variety .NET solution in TFS in about 15 seconds. Zzzz. My customer has the skills in-house, in spades, to solve this problem. Why on earth have they not done it already?
And then I spent time getting to know the people.
I talked with the lead of a true tiger team within the organization – a small band of rockstars who achieved something incredibly innovative within the company, a goal no one had ever dreamed of meeting before. Delivered exciting new functionality to their customers, modeled an exciting new process for their peers, and made it work. The Friday before I arrived onsite, the team lead’s boss’ boss had strolled by the tiger team, who had just pulled off a tricky production deployment, pointed out that there was a small bug, and left.
Again and again, as I talked throughout the organization, I heard it: big wins overlooked for small faults. Forest for the trees. In a word, perfectionism. And it was palpable. These teams were not safe to fail. They were so not safe to fail that we couldn’t even use the phrase “safe to fail” in their presence, because it has the word “fail” in it.
And so, when we got together with a cross-section of team leads to discuss problems in their (heinous) branching strategy and how to fix it, and their fear of other teams breaking them hung heavy in the room, we broached the need for CI builds on their proposed branches in order to keep more volatile branches clean. Someone suggested that if anyone breaks the build, we should definitely call them out. Breaking the build is BAD and needs to be STOPPED. Heck, Northwest Cadence ourselves had shown them pictures of Big Visible Displays from our other customers with photos (and a picture of Tom Selleck) calling out who last broke the build in those other organizations. What a great idea, they said!
I brought up what I’d observed about their culture. Not just a tendency to criticize very small faults, but person after person who took the slightest critique deeply to heart. Teams who saw wins as failures.
There was a pause. All team leads’ eyes on me. And their TFS Admin, who hadn’t been with the company all that long, looked around, first at them, then at me. He nodded. “I tried it here before. I programmed my Build Bunny to say something funny each time. But that’s probably why it didn’t go over.” Sure enough: just outside the room where we were meeting, the Build Bunny sat on someone’s desk, unused, unloved.
Imagine if a simple stupid build light were corrosive to your teams’ morale. What other obvious, proven, utterly mainstream practices become hard to do when you don’t have your culture right?
Is your team Safe for Build Bunny?