Migration is the part of a software project that gets a line in the plan and a third of the effort.
It is also where projects fail in the least visible way. A system goes live, everyone is pleased, and six weeks later someone tries to answer a question about last year and discovers the history is incomplete, or duplicated, or subtly wrong in a way that has been quietly informing decisions since launch.
The failure is rarely technical. Moving data between systems is a solved engineering problem. The difficulty is that years of accumulated records contain decisions, exceptions and inconsistencies that nobody documented, and migration is the first time anyone has to look at all of it at once.
What migration actually surfaces
The first honest run of any migration produces a list of problems that existed all along and were invisible because nothing forced them into the open.
The same entity recorded several ways. One customer as "Kumar Traders", "Kumar Traders." and "kumar traders" — three records, three balances, one business. Every operation with a few years of history has this, and nobody knows how many.
Fields used for something other than their name. A notes field holding delivery instructions. A reference field holding a second phone number. These conventions grew because the system lacked a place for something, and they are invisible until you try to map the field.
Records nobody can explain. Transactions with no counterpart, adjustments with no reason, entries from a person who left. Somebody has to decide what happens to them.
Balances that do not reconcile. Ledger totals that differ from the sum of transactions. Almost always present at some scale, and migration is when it becomes undeniable.
None of this is a reason to postpone. It is a reason to schedule properly, because every one of these needs a decision from someone in the business, not from whoever is writing the migration.
Decide what actually needs to move
The instinct is to move everything. It is usually wrong, and it is expensive.
Three categories, and they deserve different treatment:
Operational data — open orders, current stock, live customer balances, active records. This must move, must be complete, and must be correct on day one. The system cannot run without it.
Reference data — customers, suppliers, products, price lists. Must move, and is the category most needing cleaning first, because errors here propagate into everything.
Historical data — closed transactions from previous years. Must be accessible, not necessarily migrated. Frequently the better answer is to move a defined recent period and keep older records queryable in an export or the old system in read-only form.
That last distinction saves a great deal of money. Businesses ask for ten years of history and, in practice, look at eighteen months. Ask what questions people actually ask of old data before deciding to migrate all of it.
Clean before, not during
The temptation is to clean data as part of the migration. This produces a bad outcome, because the migration then has two jobs and its correctness becomes hard to verify.
The better sequence:
- Extract the current data as it is
- Profile it — duplicates, missing fields, inconsistent formats, orphans
- Decide the rules, with the business, not the developer
- Clean in the source or in a staging copy, with those decisions applied
- Migrate the cleaned data
- Reconcile against the original
Step three is the one that gets skipped, and it is the only one that requires the business. Whether two similar customer records are the same customer is not a technical question.
Reconciliation is the whole test
A migration is not finished when data has moved. It is finished when you have proved it moved correctly.
At minimum:
Counts. Number of customers, products, open orders, transactions in a period. They should match exactly, and where they do not, the difference should be explained rather than accepted.
Totals. Sum of outstanding receivables, stock valuation, ledger balances. These are the figures that will be noticed if wrong, and noticed late.
Spot checks on the awkward cases. Not random records — the difficult ones. The customer with a credit note and a partial payment. The product sold in two units. The order that was partly delivered. If those are right, the ordinary cases are almost certainly right.
Run reconciliation before switching off the old system, not after. This is the step that pressure removes from a timeline and the one that makes the difference.
Run both in parallel
For a short period, keep the old system updated alongside the new one and compare.
It duplicates effort for a few weeks and it is the cheapest insurance available. A gap discovered while both systems are live is a correction. The same gap discovered after the old system is decommissioned is a reconstruction from bank statements and memory.
This is the same principle as deploying location by location rather than switching everything at once, described in how Truffaire builds software.
What we have seen
Migration is consistently the largest source of timeline risk across the ten systems we have delivered — more than any technical requirement.
The specific pattern: a business estimates its data is broadly clean, the first extract shows several thousand near-duplicate entities, and resolving them requires decisions only the business can make, from people who are also doing their normal jobs.
The related finding is that undocumented spreadsheet logic is the hardest thing to migrate, because the rules live in formulas and in one person's judgement rather than in data. The real cost of running on spreadsheets covers why that logic is effectively the specification.
Where a business has clean, consistent records, migration is fast. That is rarer than most expect, and knowing which case you are in is worth establishing early.
Frequently asked questions
How long does migration take?
The extract and load are fast. Profiling, deciding cleaning rules and reconciling take the time, and they depend on your data rather than on the software. A first extract in week one tells you far more about the timeline than any estimate.
Can we migrate everything from day one?
You can, and it is often unnecessary. Operational and reference data must be complete. Historical data can frequently be accessible rather than migrated, which is faster and cheaper.
What if the old system will not export cleanly?
Ask this before buying anything, not at migration. Export capability is a purchasing criterion — a system whose data cannot leave in standard formats has a rising switching cost that becomes a negotiating position against you.
Who decides how to resolve duplicates?
The business, not the developer. Whether two similar records are one customer is a commercial judgement. The most useful preparation is naming who will make those calls before the work starts.
Should we clean the data first or move it first?
Clean first, in a staging copy, with rules the business agreed. Cleaning during migration makes it impossible to tell whether a discrepancy came from the data or the process.
Where to start
Do one thing before committing to any project: take a full export of your current data and look at it.
Count the customers. Sort them alphabetically and look for near-duplicates. Check whether every product has a consistent unit. Total the outstanding balances and compare against your ledger.
That afternoon of work will tell you more about the real timeline than any vendor estimate, and it moves the unpleasant discoveries to before the decision rather than after.
If you want help profiling what you have, get in touch.