Cracking the Nut - Open Sourcing from a Single Source Tree
Why a single source tree ?At a previous employer the code started out with the main code base in a single checkout with only infrastructure components of the code in their own repositories. As the code base grew the build time gradually become painful. To resolve this developers began to pull out components into their own repositories. This initially solved the problem of slow build times and improved productivity for the developers working on a single component. However, it soon became apparent that making changes to components that had many internal dependencies was particularly cumbersome. It also made it difficult to make changes across multiple repositories.
This was solved by moving everything back into a single source tree. What about the slow build times? The short answer is Buck - a build tool from Facebook which is inspired by Google's internal build tool called Blaze, now open sourced as Bazel. In summary these build tools allow you to easily modularize your repository and only build and test modules that have changed and any of their dependencies. This allows small code changes to be built very quickly while still making changes across the entire organisation easy and atomic.
Managing Open Source componentsOur own open source components were treated as external dependencies and therefore their source code was kept in their own repositories. This meant changes to an open source component required a commit to the external repository, an artifact to be built and then downloaded and committed to the internal repository. This was not as simple as changes to any other code in our single source tree.
We didn't want this problem at TransFICC as we want to open source as many components as possible without any associated overhead. A change to an open source component should be no different from a change to any internal component.
To achieve this we view our internal single source tree as the source of truth, which is a Git repository that contains all our open source modules. We maintain our open source modules within the single source tree and make use of Git Subrepo to clone the modules subdirectory to a public Git repository. We then have a Job in our Continuous Delivery pipeline that automatically pushes changes that pass the build to the public source repository. See image below.