Evolution Without Rewrites: Escaping Retail's Data Rigidity Trap

A landmark 2012 research study conducted by McKinsey and the University of Oxford across more than 5,400 IT projects found that large technology initiatives run 45 percent over budget on average while delivering 56 percent less value than predicted, and that 17 percent go so badly they threaten the survival of the company that funded them. These findings have held up in the years since,across industries. However, they carry particular weight in retail, e-commerce, and marketplace businesses, where the pace of change in the operating model routinely outruns the pace of change the underlying systems can absorb.

This gap produces a familiar strategic dilemma, where engineering leaders at retail companies tend to inherit one of two postures. The first accepts the constraints of the existing system and manages around them, absorbing slower delivery as the cost of stability. The second commits to a full re-architecture, accepting years of parallel investment and the failure rates described above in exchange for a system that fits the business again. Both postures treat the same assumption as fixed, namely that the way a system stores its history determines how expensive it is to change, and that escaping one history means writing another from scratch.

That assumption deserves scrutiny, because it is an artifact of a particular architectural choice rather than a law of software. There is a third posture available, one in which systems evolve continuously instead of being periodically replaced, and reaching it requires understanding why retail systems calcify in the first place.

Anatomy of Data Rigidity

Data rigidity is the condition in which a company's accumulated data structure, rather than its code, becomes the primary constraint on business change. Code can be refactored incrementally. Data structures resist incremental change because every table, schema, and integration encodes assumptions about how the business worked when it was designed.

The condition follows a recognizable progression in retail and commerce environments. Systems begin as applications built around a central relational database, and the database grows with the business until it effectively becomes the business: the single structure through which orders, inventory, fulfillment, and partner integrations all flow. Performance pressure arrives first. Teams respond with tuning, indexing, caching, and increasingly creative workarounds. Each buys time while deepening the coupling between the data structure and everything that depends on it.

Organizational symptoms follow the technical ones. Knowledge of how the system actually behaves concentrates in a small number of senior engineers because only long exposure teaches them which changes are safe. Those engineers become review bottlenecks, and their eventual departure becomes an enterprise risk. Estimation inflates because every change request must be priced against the unknown blast radius of touching shared structures. The business hears longer and longer timelines for requests that sound simple. Leadership concludes that engineering has slowed down. In reality, the system has hardened.

Decomposition efforts stall for the same underlying reason. Approaches that gradually carve apart a monolith depend on being able to trace what depends on what. A state-based system offers no such trace. The database records where every value ended up, but not how or why it got there. A system that stores only current state has, in a literal sense, no story to tell. Teams cannot safely decompose what they cannot trace.

The result is a compounding tax on every future initiative. Engineers spend more time understanding existing behavior, assessing blast radius, and protecting against unintended consequences, and less time building what the business actually needs. The longer the system operates this way, the more expensive change becomes.

Why Do Retail Modernization Projects Fail?

Retail modernization projects fail most often because they treat historical data as an obstacle to be migrated rather than an asset to be reinterpreted. A rewrite obliges the organization to translate its entire accumulated history into a new structure before the new system can fully operate, which converts a technology project into a high-stakes translation of the past.

That translation is where the risk concentrates. Migration requires freezing or carefully synchronizing a business that cannot afford to pause, since retail operations run continuously across time zones, channels, and partner networks. It demands resolving every inconsistency and undocumented assumption buried in years of accumulated data, on a deadline, and it culminates in a cutover, which is where the black swans described above tend to live.

There is a second, quieter reason these projects disappoint even when they technically succeed. A re-architected system moves only as fast as the systems around it, because upstream sources still deliver data shaped by old assumptions and downstream consumers still expect the old contracts. Organizations that concentrate years of investment into replacing one system often discover that the constraint has simply relocated to the seams, which is why the all-at-once approach does not retire risk so much as concentrate it into a single, non-repeatable bet.

The pattern suggests that the problem is not execution quality but the premise itself, the belief that change requires rewriting history.

The Third Option: Systems That Evolve

There is an architectural pattern that removes that premise. In event sourcing, a system records every business fact as an immutable event in an append-only log, rather than storing only current state and discarding how the state arose. Martin Fowler describes the pattern as ensuring that all changes to application state are stored as a sequence of events, which can then be queried and used to reconstruct past states or reinterpret history for new purposes.

The consequences for rigidity are structural rather than cosmetic. When history is preserved as events, the current data model stops being the system of record and becomes one view over the record. An order stops being a row that gets overwritten and becomes the accumulated sequence of what actually happened, received, allocated, picked, shipped, delivered, and every exception along the way.

Changing the system then means something different, because instead of migrating data into a new shape, teams derive new views, new domain boundaries, and even new business models from the same underlying events. Old and new coexist during a transition, which turns the cutover cliff into a gradual slope, and domain boundaries are free to evolve as understanding of the business evolves rather than needing to be fixed correctly on day one. In practice, the boundaries of a domain behave less like a blueprint and more like a living document, revised as decomposition reveals what the organization actually owns.

The theoretical difference becomes concrete in a thought experiment. Consider a retailer shifting from a product-centric operating model to a customer-centric one, a transition many commerce businesses now face. Under a state-based architecture, that shift means restructuring core data models, migrating history into the new shape, and coordinating a cutover across every dependent system, with all of the risk described in the previous section. Under an event-sourced architecture, both models are readings of the same history, so the product-centric view and the customer-centric view can be derived from one shared sequence of events and can operate side by side while the organization transitions between them. The change in business model becomes a change in interpretation rather than a bet-the-company migration, and the pace of the transition is set by the surrounding systems rather than by the terror of the cutover.

Traceability arrives as a side effect rather than a project. For supply chain, fulfillment, and marketplace operations, where the perennial business questions are where things are, why they moved, and what happened, an architecture whose native format is "what happened" answers those questions by construction. The system finally has a story to tell, and audit, operations, analytics, and increasingly AI initiatives all read from the same one.

What is event sourcing?

Event sourcing is an architectural pattern in which every change to a system is captured as an immutable event in an append-only log, rather than overwriting the current state. The full sequence of events becomes the source of truth, allowing teams to reconstruct any past state and derive new data models from existing history.

Axoniq and Your Legacy System

The architecture described above is what Axoniq builds. Axon Framework is open source and lives in the Java and Spring ecosystem most retail engineering organizations already run, so the learning investment goes into event sourcing and domain-driven design rather than into an unfamiliar stack.

Axon Server is the purpose-built event store that makes evolution operational, storing history once and indexing it for reinterpretation, with dynamic consistency boundaries enforcing business rules across entities atomically and retagging applying new meaning to historical events in place, without moving data or taking systems down. Just as importantly, the platform removes the second trap that capable teams walk into, spending engineering quarters building and maintaining evolution infrastructure themselves when the entire point was to move faster on the business.

Legacy System Modernization

Retail legacy system modernization doesn't have to mean a rip-and-replace decision, which would simply reintroduce the trap under a new name. Event-sourced systems are routinely adopted alongside existing estates, one domain at a time, with the legacy system continuing to operate while new capabilities grow beside it.

For a deeper treatment of the architectural and business case, The Event-Driven Advantage examines what changes when systems are built around events, and When to Event Source, and When Not To offers an honest framework for deciding whether the pattern fits a given problem, because it does not fit every one. For a business-level introduction to dynamic consistency boundaries, the Axoniq Academy overview course is free to start.

Rigidity is a choice that systems make on behalf of companies, one schema at a time. Evolution is a capability you build once, and the organizations that thrive through the next decade of retail change will be the ones whose systems were designed to change with them.

Frequently Asked Questions

What is data rigidity?

Data rigidity is the condition in which a company's accumulated data structure, rather than its code, becomes the primary constraint on business change. It develops as schemas, integrations, and workarounds encode old assumptions, making every new business requirement expensive to implement and risky to deploy.

What is the best approach to legacy system modernization in retail?

Incremental evolution outperforms big-bang replacement. Rather than migrating everything at once, retailers capture business events alongside the legacy system and grow new capabilities domain by domain, letting old and new operate in parallel. This avoids the budget overruns and cutover risk that cause most large modernization projects to fail.

Can you modernize a retail system without a full rewrite?

Yes. Event-sourced architectures preserve business history as an immutable sequence of events, so new data models, domain boundaries, and business capabilities are derived from existing history rather than built through migration. Old and new models coexist during the transition, removing the cutover risk that sinks conventional rewrites.

How does event sourcing differ from a database migration?

A database migration relocates and reshapes data to fit a new model, which requires downtime windows, translation of historical records, and a high-risk cutover. Event sourcing reinterprets history instead, deriving new views from the same immutable events, so the business model changes without any data moving.

How long does a business model transition take with event sourcing?

Transitions that would require months or quarters of migration work in a conventional architecture can complete far faster when history is reusable, because the work shifts from moving data to reinterpreting it. Actual timelines depend on the surrounding systems, which continue to set the pace of end-to-end change.