Reconstructing a Legacy Enterprise Data Platform

Reconstructing a Legacy Enterprise Data Platform: when migration means reconstruction

This one comes with an asterisk. It's an NDA engagement, so no client name, no source platform name, no dates, no repo link, nothing that would let you fingerprint the client. What I can talk about is the architecture, because the problem underneath it is universal: move a massive, proprietary data platform onto a modern one, at a scale where "just rewrite it" stops being a joke and starts being a two-year plan.

Challenge

The platform being replaced didn't just store code. It stored meaning: a transformation wasn't only its logic, it was that logic plus a build spec, a schedule, a health contract, and a spot in a lineage graph, all held by the platform instead of the artifact. The target platform modeled almost none of that the same way. So the real goals were:

1. Reconstruct platform-held meaning as explicit, target-side declarations, not just port syntax
2. Handle a dependency graph that spans project boundaries the source platform never exposes cleanly
3. Prove that a converted pipeline behaves like the original, not just that it runs
4. Keep humans out of the repetitive 90% of the work without handing them a black box for the other 10%
5. Leave behind something the client's own engineers could read and extend, not a system only the delivery team understood

Solution

The approach that actually held up under scale was deterministic-first, with AI treated like a scalpel instead of a hammer:

• An intermediate representation as the seam. Extraction never talks target-platform language, and conversion never talks source-platform language. Everything crosses through a stable, versioned middle layer.
• A rules engine that owns structure. Most conversion decisions are mechanical once you've seen the pattern once. Those get codified as explicit, reviewable mappings, not regenerated from scratch every time.
• AI as bounded fallback, never as the architect. When a rule doesn't exist yet, a sandboxed model proposes a fix for that one gap. It doesn't get to make structural calls about how the whole system fits together.
• Sequencing driven by dependency closure, not by folder order. A migrated unit can be perfectly translated and still produce garbage if what it depends on hasn't landed yet. The plan has to respect that or it's fiction.

Implementation

How the pieces fit together:

1. Platform-agnostic extraction: Pull everything the source platform knows about an asset into a structured, file-based representation that has zero opinions about where it's going next.
2. Deterministic conversion layer: Route each asset type to its own dedicated mapping logic; keep the units small enough that no single piece has to reason about the whole system at once.
3. Bounded generative fallback: For the gaps the rules don't cover yet, a scoped model fills in the specific missing piece, then that pattern gets promoted into the deterministic layer for next time.
4. Dependency graph derivation: Reconstruct the cross-project ordering from the extracted metadata itself, since the source platform won't hand you that graph directly.
5. Parity verification: Compare output against the original system for the same inputs. Running successfully is necessary; it's not proof of anything on its own.
6. Regenerate, never patch: Fixes go into the mapping logic, then get regenerated. Nobody hand-edits the output tree, because that's how you end up with drift nobody can explain a year later.

Key features

Deterministic-first conversion

• Explicit, inspectable mappings own every structural decision
• A reviewer can trace an output back to the exact rule that produced it, no guessing

Platform-agnostic intermediate layer

• Extraction and conversion never share assumptions about each other's platform
• Swapping the target platform later doesn't mean re-extracting everything from scratch

Dependency-aware sequencing

• Migration order comes from a derived graph, not from wishful project planning
• Unresolved references get surfaced explicitly instead of silently assumed away

Bounded AI fallback

• The model only fires where the deterministic layer has a known gap
• It fixes the one gap in front of it; it doesn't get to redesign the system around itself

Results

The program covered 60 PB across more than 450 use cases and represented $11.9M in program value.

At this scale, the win isn't any single converted pipeline, it's that the system behaves the same way on pipeline four thousand as it did on pipeline four. That gets you:

• A conversion process that's explainable, not a black box the client has to trust blindly
• A clear audit trail from source behavior to target behavior, which matters a lot more than it sounds like it should when compliance gets involved
• An engineering approach the client's own team can keep running after the delivery team leaves, which was the actual point the whole time

Try it

There's no public repo for this one; it's a client engagement under NDA, source and all. What's shareable is the pattern: separate extraction from conversion with a stable intermediate layer, let deterministic rules own anything structural, and keep generative tools on a short leash doing exactly the narrow job they're good at. If you're staring down a "we have to rebuild this whole platform" problem, that shape scales a lot better than "point an LLM at it and hope."