Parity Testing With Feature Flags

Originally posted January 2023 on the Split Blog (with minor edits Jan 2024 by Joy specifically for this blog)

Feature Parity Testing, sometimes called TAP compare testing, ensures a new system behaves the same way as an old one. It is used when replacing part or all of an old system. At a high level, you mirror your traffic to both systems and compare the results, logging any that differ. Unlike many other tests, parity testing relies on actual traffic rather than test scenarios written by an engineer. This makes it a great way to identify forgotten features or find broken edge cases not caught by other forms of testing.

How do we set up feature parity testing? First, you’ll need an identical interface on the old and new systems. You can create an interface on your new system that matches your existing one. Or add a thin façade on top of the old system that is the interface you want on the new one. Every scenario is different, but if you want to change your interface, I recommend adding the façade.

Now that your two systems have the same interface, it’s possible to add a feature flag that controls traffic to each system. This will seamlessly move the traffic between them.

How to Add Feature Flags to Your System For Parity Testing

If you’re using a feature-flagging product like Split, you can create more than two treatments. As a result, you can set this up as four options: NEW, OLD, MIRROR, and PARITY. NEW routes all of the traffic to the new system. OLD sends all of the traffic only to the old system. MIRROR sends the traffic to the old system and also asynchronously sends it to the new system. PARITY does the same thing as MIRROR but also sends the response from the old system to compare and log when the responses don’t match. If you’re using a product that only allows two treatments, you can do something similar but with more overlapping flags.