The highly scalable characteristic of the ApartCI system is the result of its unique architecture, combining multiple different techniques and making them work together in such manner as to compensate for each other's limitations and to enhance each other's advantages.
ApartCI's non-blocking
design keeps the project free from breakages and regressions thus completely eliminating the time and resources
wasted on recovery, regardless of the number of developers working on the project and the incoming rate of
changes.
The unattended, entirely automated operation of the ApartCI operation pipeline keeps the integration branch open
for business 24x7x365. Peaks in incoming changeset rates, either occasional or associated with the developer
working schedules are absorbed and, if necessary, automatically distributed across periods of lower activity.
The biggest scalability challenge for a non-blocking CI system is the mandatory serialization of the pre-commit QA verifications. In order to address it ApartCI uses centralized coordination and execution of these verifications, which allows multiple individual changesets to be bundled and verified together like a single super-changeset. A successful bundle verification automatically validates all the changesets in the bundle, in the same time and using the same resources as a single regular changeset verification. The bigger the bundle - the higher the efficiency of the system.
But changeset bundling comes with its own disadvantage - the impact of the failures in the bundle verification.
Typically the bigger the bundle - the higher the chances of a verification failure to occur and the higher
the number of bundle splits required for culprit isolation. The impact is
particularly damaging on the critical path - the mandatory serialized verifications. To minimize the impact
ApartCI utilizes two separate changeset pools in the pre-commit
verification stage.
Multiple, parallel verifications are performed on regular bundles,
created from the first, higher risk changeset pool. These verifications will be used to detect, isolate and
reject all faulty changesets but only some of the conflicting changesets reaching the
pre-commit stage.
All successfully verified regular risk changesets are promoted to the second, low-risk changeset pool.
Serialized verifications will be performed only on primary bundles,
created from this low-risk changeset pool. These verifications will lead to the capture and rejection of all
remaining conflicting changesets. Due to the lower risk of failures the primary bundles can be much larger than
the regular bundles, thus their verifications are much more efficient than those of regular bundles,
compensating for their serialized execution.
In the rare event of a primary bundle verification failure ApartCI limits the impact of the higher number of
subsequent bundle splits by moving the culprit identification activity off the critical path and starting a
new primary bundle instead when such action yields a higher verification efficiency.
ApartCI also supports an optional chain of one or more preliminary verification stages to be placed in front of the pre-commit stage. The bundle verifications in the preliminary stages function just like the regular bundle verifications in the pre-commit stage, but with just a subset of the pre-commit stage verification criteria. The preliminary stages are typically used for faster rejection of trouble changesets, which helps increase the bundle sizes and reduce the rate of bundle splits and verification costs in subsequent verification stages.
With this architecture in place the scalability of an ApartCI system is virtually unlimited. Whatever throughput or sustained incoming changeset rates the system has at some point, it can always be increased by:
- adding QA verification resources to the pipeline stage(s) in need
- adding preliminary verification stage(s)
- speeding up QA verifications, for example by splitting long QA tests into multiple shorter tests that can be executed in parallel