Continous Integration Vs Pre Submit Tests

In a company that I have been consulting for, I recently found myself in a discussion to improve code quality before it reached staging or testing servers.

This inevitably led to the discussion of integrating automated smoke tests before anyone could submit their branch for merging or using a continuous integration pipeline where the current head is synced and tested automatically.

I was of the impression that continuous integration as a means of maintaining pre-staging quality was a done and dusted topic. The advantages were clear, such as

• Parallel, asynchronous and automated process which did not add any overhead on dev team.
• Overall health of a build was easily reflected in the integration history
• Repeat offenders could be easily reminded via a slack message/email to the entire group of the offending CL (cough public shaming cough).
• Could be nicely integrated with Slack or any dev team platform, even an orb.

Our team at Google used to keep an orb which would turn red if the build was breaking. As submitters and reviewers, we were reminded to "Keep the orb green".

We worked with the Orb for some time, inspired perhaps by this blog by Mike Swanson. I would say that worked well because this process was soon upgraded into the mainstream via wall mounted LED screens.

Anyway, so the merits of continuous integration were pretty obvious to me. However, what was not obvious to people and that leads me to guess it must be a debate for other teams too who are making a transition from a agility first to stability first.

I can understand the urge to include your unit tests, automation tests in a pre-submit process. Everyone needs to run tests before they submit code they say. Perhaps your tests take a second to run and maybe its okay for now, however, do consider the implications of the process when you are about to invest in that direction.

Disturbing the Flow

You know what a developer wants to do when they are in the flow, keep developing. You know what everyone else should do when they are in the flow, don't disturb the flow.

Compulsorily, at this point, I have to bring up this seminal piece by Paul Graham, which tells you a lot about time chunks that creator's need to build things. Running tests, compiling code, repeated testing steps all break the flow. Don't do it, at least not while they are in the flow.

Process overhead is like distraction. It breaks the flow. XKCD certainly got this one right.

Smoke Tests Bloat

Do you know one of the biggest issues continuous integrations face? It is testing bloat. If you have too many or too lengthy tests running in every cycle, between any two cycles, the branch has moved by miles.

The same issue crops up in Pre Submit Smoke tests, where smoke tests become, "Have written this amazing module, and those who break it shall not pass" tests, and then to "Well, if you think this should pass then so should this" tests and soon there is a Gandalf in front of every team member, well, at least for the time it takes to break their flow.

Less Maintenance

This I think is the most specific and changes from team to team. However, usually its easier to maintain a test database and a production like environment on a single continuous integration machine then to reproduce that environment on every dev machine. Arguably, test results on integration machines are more trustworthy, however, this may not apply to every team.

Regardless, I would say if you are just starting testing discipline and you just want to raise the bar, sure go ahead with pre-submit tests but if you find your developers commenting out the hook or the build still fails mysteriously in staging, don't take it out on the team, consider reviewing your process.

Remember, don't break the flow.