When Your Real-World Data Says 'Significant' but the Registry Shows Noise: A Cross-Source Riddle

You run the analysi on electronic health records. The p-value is 0.02. Nice, you think. Then you pull the registry data for the same question. P-value: 0.34. What gives?

This isn't a glitch. It's a feature of real-world data. Different source capture different slices of reality. And when they disagree, the riddle isn't which one is 'sound' — it's understanding why they differ. That's what this article is about.

Why This Riddle Matters Now

The rise of multi-source RWD studies — and why seams matter

Real-world data studies used to run on a lone source. One claim database. One hospital setup. One registry. You knew the quirks — the coding gaps, the miss lab values, the silent shifts in how doctors billed. That was manageable. Now the FDA, EMA, and health-tech payers pull multi-source evidence. Combine electronic health records with a disease registry and a pharmacy claim feed. Sounds smarter. The catch is — each source carries its own version of the truth. I have seen units spend six months harmonizing variables, only to find the treatment effect flips sign when they swap the primary source. That's the riddle: your data says 'significant,' the registry next door says noise. Same patient. Same calendar years. Different reality.

Regulatory decisions depend on source consistency — brittle is dangerous

‘We validated the algorithm in our EHR. The registry said the same cohort showed no benefit. Both were correct — but only for the patient each source kept.’

— A quality assurance specialist, medical device compliance

False positives and false negatives carry real costs

False positives burn millions. You push a drug into a Phase IV commitment based on a noisy signal — only to waste resources chasing an effect that was a coding artifact. False negatives are worse: a real treatment effect gets buried because the registry omitted sicker patient who stopped attending follow-up. flawed sequence. Either way, a patient misses or receives a therapy that hurts. The tricky bit is — most units skip the cross-source sanity check. They merge dataset, adjust for confounding, run the regression, and publish. That works when source agree. When they diverge, the paper becomes a liability. Practical pain: one pharma client ran a 50-patient pilot, found a 30 % improvement in a claim-only analysi, then saw the same metric flatline in the physician-submitted registry. The difference was a mission death flag — the registry coded deaths only if the family reported them. The claim setup captured every funeral. That's not noise. That's a structural seam. And seams rip if you don't map them early.

The Core Idea in Plain Language

Different source, different stories

Here’s the uncomfortable truth: two dataset can look at the same patient, the same drug, the same slot window—and still scream opposite conclusions. I have debugged this exact mess. The clinical registry says the treatment works; the real-world claim database says it does nothing. Both are arguably correct, given what each source sees. The core trick is that a dataset is never a neutral window onto reality. It’s a pipe that filters what gets recorded—and what gets left out. A hospital registry captures patient who stay sick enough to follow up. An insurance claim database captures people well enough to submit a bill. Those two populations are not the same crowd, even if they share a diagnosis code.

Confounding structures differ

Most groups skip this: confounding doesn't travel neatly between source. A confounder is a hidden third factor—say, age or income—that influences both whether someone gets the drug and whether they get better. In a registry, that confounder might be documented (a nurse asks about smoking, they record the answer). In claim data, smoking is invisible—nobody bills for “cigarettes today.” So the statistical correction you apply in the registry works. Apply the same model to claim and the hidden confounder twists the result. The odd part is—both dataset are technically clean. The seam blows out because the structure of mission data is different.

Mentor hours, peer critique, revision sprints, portfolio cuts, and rejection logs teach pacing better than viral tips.

Nebari jin moss needs patience.

“You're not comparing apples to oranges. You're comparing apples drawn from a tree with oranges scooped from a barrel.”

— paraphrased from a statistician who lost a week to this riddle

The catch is that most analysts assume “big data” means robust signal. That hurts. A 100,000-row claim set can produce a p-value of 0.001 for a treatment that actually kills people—if the unmeasured confounders are strong enough. The registry, with only 4,000 rows and missed half the variables, might show the true null effect. Bigger doesn't mean less biased; it means precisely estimated bias. off group.

Selection and measurement shape the signal

Selection is the quiet killer. A registry enrolls patient through clinics—so everyone in the dataset had to show up for a visit. That alone selects for people who are adherent, or desperate, or both. claim data selects for people who have insurance and submit reimbursement forms. Someone who pays cash for the same drug and never files a claim? Invisible. Someone who gets the drug free through a patient-assistance program? Also invisible. Those invisible patient might be healthier (they have the energy to navigate bureaucracy) or sicker (they can't afford insurance). You never know. The result is that the same statistical model run on registry vs. claim can flip the sign of the treatment effect—positive becomes negative.

Measurement compounds the issue. registrie often use a clinician-verified outcome: “Did blood pressure drop below 130?”. claim data uses a proxy: “Was a prescription filled?”. Those are not the same. A patient fills the script but never takes the pill. Claims says “treated”; registry (with a blood pressure reading) says “no effect.” Both are sound, within their measurement frame. The fix? Stop asking which dataset is “correct.” begin asking which selection filter and which measurement definition your question demands. Otherwise you chase noise dressed as insight.

Flag this for medical: shortcuts expense a day.

Flag this for medical: shortcuts cost a day.

Spec sheets, torque tolerances, pneumatic feeds, laminate rollers, and ultrasonic welders each pull separate maintenance cadences.

Vendors, contractors, couriers, inspectors, dyers, embroiderers, and patternmakers hand off partial truth unless logs stay current.

Lens flares, color grades, audio beds, storyboards, and render farms each invent their own silent failure modes overnight.

Rosin mute reed knives chatter.

Nebari jin moss needs patience.

Ledger reconciliations, accrual quirks, invoice aging, cash forecasts, and variance notes expose wander before board decks do.

Nebari jin moss needs patience.

Nebari jin moss needs patience.

Most practical next shift: draw a one-page diagram of who is missed from each source before you run a lone regression. If you can't list three specific groups lost to selection, you're not ready to compare results. That diagram will show you why the registry says “significant” and the claims say “noise”—without either being a liar.

Vendor reps rarely volunteer the maintenance interval; however boring it sounds, the calibration log is what keeps your spec tolerance from drifting into customer returns during the first seasonal push.

How It Works Under the Hood

Data generation mechanisms

EHR data is a byproduct of care, not a designed experiment. A clinician enters a HbA1c value because a patient happened to visit, got a lab group, and the lab returned results — three separate failure points before the data even lands in your extraction query. registrie, by contrast, are built with a specific research purpose: every bench has a protocol, a defined collection window, a trained abstractor. That sounds fine until you realize the registry’s structure itself introduces selection — only certain centers participate, only consented patient appear, only scheduled follow-ups get recorded. The EHR captures the frantic 3 A.M. admission; the registry captures the tidy 10 A.M. research visit. Different machines, different biases.

The odd part is — units often treat these source as interchangeable. They merge them. They average them. They declare significance when the p-value drops below 0.05. What usually breaks primary is window alignment: the EHR’s timestamp reflects when the data was entered, not when the event occurred. A registry’s date is retrospective, verified, often months old. Mix them without a careful clock, and you’re correlating a clinical noise spike with a research silence — and calling it a signal.

missed data patterns and bias

EHR missingness is rarely random — it’s driven by workflow, urgency, and reimbursement. Sick patient get more tests; healthy ones vanish from the record. That means the miss indicator itself encodes severity. registrie, however, actively chase miss values: queries go back to sites, forms are re-reviewed, missingness is reduced and then coded as a deliberate exclusion. The catch: a registry’s complete case is often the least interesting patient — easy to follow, compliant, no messy comorbidities. What does that mean for your cross-source analysi? The EHR’s missingness skews toward the sick; the registry’s completeness skews toward the healthy. Merge them naively and your treatment effect flips direction.

Trail markers, water caches, weather windows, blister kits, and bailout routes matter more than brand-new gear lists.

Koji miso brine smells alive.

'We thought the drug lowered HbA1c by 0.5% — turned out the EHR mission data was hiding all the non-responders who never came back.'

— actual feedback from a health-data engineer, 2023

I have seen this repeat three times in the past year. Each staff spent weeks debugging, only to find the seam between source was the actual glitch — not the drug, not the model, but the unspoken assumption that 'miss' means the same thing in both worlds.

Statistical properties of each source

EHR values are noisy, autocorrelated, and often rounded to clinically convenient thresholds (e.g., 140/90 exactly). Registry values are cleaned, standardized, but potentially outdated — the abstractor’s transcription error replaces the clinician’s rounding error. Different error structures produce different variance; different variance inflates or deflates your standard errors. That's how you get a significant p-value from the EHR but a null result from the registry: the EHR’s noise looks like real variation, while the registry’s precision suppresses any real effect. One source shouts, the other whispers.

Most groups skip this step: they never plot the distribution of the same variable from both source on the same axis. Do it once. You will see spikes at whole numbers in the EHR data, and smooth curves in the registry data. That gap is not a bug — it's a fingerprint of how the data was born. And ignoring it's how a 'significant' finding turns into a replication failure.

Worked Example: diabete Treatment Effect

EHR cohort construction

We pulled 2,847 adult type 2 diabete patient from an academic hospital’s electronic health record—anyone with a opening HbA1c ≥ 7.5% between January 2018 and June 2019. plain inclusion: age 30–75, no prior insulin, at least two HbA1c readings in the next 18 months. I built the cohort in an afternoon. Clean, fast, satisfying. The treatment group got metformin monotherapy (n=1,203); the control group received only lifestyle counseling documented in the notes (n=1,644). We matched on age, baseline HbA1c, and BMI using nearest-neighbor propensity scores. Balance looked good—standardized mean differences all under 0.1. Too good, maybe.

Registry cohort construction

Same clinical question, different source. We turned to a national diabete registry—mandatory quarterly submissions from 47 clinics. Inclusion criteria identical on paper: age 30–75, initial HbA1c ≥ 7.5%, same date window. But the registry forced us to drop anyone with mission BMI (n=412) or incomplete pharmacy data (n=289). Final registry cohort: 1,381 patient in the metformin group, 1,007 controls. The odd part is—the registry required a specific diagnosis code for type 2 diabete, while the EHR let us use any mention of “diabetes mellitus” in a snag list. That solo mismatch changed the cohort. The registry patient skewed older (mean 61 vs. 57 years) and had more comorbidities recorded. We matched again. Covariate balance held, but I felt uneasy—these were different populations wearing the same label.

analysi and results comparison

Outcome: mean HbA1c reduction at 12 months. In the EHR analysi, metformin patient dropped 1.4% (SD 1.1) versus 0.7% (SD 1.3) for controls. p-value: 0.008. The registry painted a hazier picture: 1.2% reduction for metformin, 0.9% for controls. p-value: 0.21. Same drug. Same rough effect size. One p-value screams “publish me”; the other whispers “random noise.”

“The registry data didn’t contradict the EHR—it just had less signal because the control group partially received metformin from non-registry source.”

— data analyst’s field note, after the third re-run

Mycelium jars, still-air boxes, agar plates, grain masters, and fruiting chambers collapse when sterile theater replaces sterile habit.

Serac crevasse bridges rewrite courage.

Not every medical checklist earns its ink.

Not every medical checklist earns its ink.

Preproduction, top-of-production, inline, midline, final, and pre-shipment audits catch different classes of wander.

Rosin mute reed knives chatter.

Watershed buffers, riparian corridors, sediment traps, canopy gaps, and nesting cavities respond to disturbance on mismatched clocks.

Sourdough hydration, autolyse rests, coil folds, batard shaping, and dutch-oven preheats fail when timers replace feel.

Nebari jin moss needs patience.

Rosin mute reed knives chatter.

Stone-ground flour, millstone dress, bolter screens, bran streams, and ash tests hold bakers honest about wheat.

Rosin mute reed knives chatter.

Loom heddles, shuttle races, warp tension, weft floats, and selvedge drift expose shortcuts at the primary wash.

Bolter bran streams keep bakers honest.

The catch is hidden in documentation habits. In the registry, controls were patient whose clinics never recorded a metformin prescription. But many of those same patient may have gotten metformin from a hospital pharmacy outside the registry network—the registry captured only prescriptions written within reporting clinics. That leakage diluted the treatment effect. The EHR, by contrast, pulled medication orders from the entire hospital stack—fewer blind spots. So which p-value is flawed? Neither. They answer different questions. The EHR says: “What happens when we prescribe metformin within this integrated stack?” The registry says: “What happens when we prescribe metformin documented in a national reporting framework?” Those are not the same thing. Most groups skip this—they pick one source, run the trial, and shift on. That hurts. A false positive from the EHR would waste research dollars; a false negative from the registry would bury a real effect.

Edge Cases and Exceptions

Crossover Treatments — When a Cure Becomes Confounding

The cleanest dataset hide a dirty secret: patient switch arms. In a diabetes registry, you might see Drug A assigned to 500 patient and Drug B to 500 others. The registry shows Drug A reduces HbA1c by 0.8% — significant, happy, done. But dig into the real-world pharmacy claims, and suddenly 40% of the Drug A group also picked up Drug B refills six months in. The registry never flagged it because the patient never told the specialist. I have seen this blow a model apart. The supposed 'treatment effect' you measure is actually a mix of two drugs, or worse — no treatment at all for the patient who stopped taking A but the registry still counts them. The fix is nasty: you must cross-reference prescription fill dates against patient visit logs, and even then, you guess.

What kills the analysi is the assumption of purity. registrie treat assignment as permanent. Real life treats it as a suggestion. When you see cross-source disagreement that suddenly reverses a positive effect into neutral territory, check for crossover primary. Nine times out of ten, the 'noise' is actually a patient who switched treatments behind the setup's back.

Immortal slot Bias — The Gap That Lies

Here is a concrete anecdote from my own effort. A registry showed that patient who received a new diabetes injection had 30% lower mortality. Impressive. Except the real-world appointment data revealed a five-month lag between diagnosis and the primary injection — and patient who died during that lag were never recorded as treated. The registry only saw the survivors, the ones who lived long enough to get the shot. The 'immortal window' between enrollment and treatment was invisible in one source but obvious in the other. The effect? The cross-source discrepancy wasn't noise at all. It was a systematic exclusion of dead people.

Most units skip this: they merge by patient ID and assume phase alignment. faulty queue. The registry timestamp may say 'treatment started at month 0', but the pharmacy claims say 'opening fill at month 5'. That five-month gap is immortal slot. If you don't subtract it from the denominator, your hazard ratio will look miraculous. The trade-off: aligning timestamps across source often halves your sample size because records don't match cleanly. That hurts — but a smaller honest dataset beats a large liar.

Subgroup Effects That Reverse — Simpson's Paradox, Now With Real Data

The odd part is that both source can be correct and still lie to you together. A registry might show Drug A works better for women than men. Your wearable data, though, shows the opposite — women using Drug A have higher average glucose spikes. How? The registry grouped by sex. The wearable data grouped by actual activity level. Women in the registry happened to exercise more, so their glucose looked controlled. The wearable data caught the sedentary women. The cross-source disagreement here is not error; it's granularity mismatch. One source sees broad categories; the other sees behavior. Neither is fake — they just measure different slices of the same human.

'You're not reconciling truth with noise. You're reconciling one truth with another truth that uses a different ruler.'

— paraphrased from a biostatistician who ran into this at a pharma R&D meeting

The catch is: you can't just average the two. That would bury the real subgroup effect. Instead, you must re-stratify both dataset by the same third variable — activity, age band, window since diagnosis — and then check if the disagreement persists. If it does, you have found a genuine edge case where one source's inclusion criteria filter out the very patient who drive the other source's result. That's when you stop debugging and launch designing a new study.

Oboe reeds, clarinet ligatures, trombone slides, tuba spit valves, and timpani pedals each invent unique maintenance rituals.

Koji miso brine smells alive.

Limits of the Approach

Residual confounding remains

The cleanest cross-source match still carries ghosts. You match patient by age, sex, and HbA1c—but the registry might capture people who visit clinics more often, or who own smartphones for self-monitoring, or who live in postal codes with better pharmacy access. I have seen a VA record set where the treatment arm looked healthier, simply because veterans with stable housing were easier to follow. The effect size looked real. It was real—but only for housed patient. That's not the same as a universal truth. The gap between what you measured and what you wanted to measure never fully closes.

You can adjust, stratify, use propensity scores—still, unobserved confounders leak through. The registry doesn't tell you if someone lost their job halfway through the trial. The real-world dataset doesn't record the divorce. Those events adjustment adherence, adjustment stress hormones, change outcomes. The seam between sources hides exactly what matters most.

External validity constraints

A cross-source comparison that works in one population often fails in another. The diabetes example that held up in a German statutory-insurance cohort? It collapsed when re-run on a rural Indian clinic network. Different prescribing habits. Different food environments. Different definitions of 'control' in blood glucose. The riddle is not solved once—it must be re-solved for each context.

Worse: the very act of linking two sources creates a filtered subset. You only analyze patient present in both dataset. People who fall out of one system—say, they switch insurers or leave the country—vanish from your view. Those dropouts are not random. They're the mobile, the sick, the disenfranchised. The study silently generalizes to the people who stay put and stay documented, not to the population you set out to study. That shift is quiet. And it destroys generalizability without a warning label.

Reality check: name the research owner or stop.

Woven, knit, jersey, denim, twill, satin, mesh, and interfacing behave differently when needles heat up mid-batch.

Koji miso brine smells alive.

Cutters, graders, pressers, finishers, trimmers, handlers, inkers, and packers rarely share identical checklist verbs.

Hemming, fusing, bartacking, coverstitching, overlocking, and flatlocking introduce distinct failure signatures under rush orders.

Chronograph bare-shaft tuning exposes ego.

Koji miso brine smells alive.

Reality check: name the research owner or stop.

Zinc rivets, quinoa starch, glyph markers, ember trays, and nexus clamps rarely share the same reorder cadence.

Koji miso brine smells alive.

“Agreement between two noisy dataset doesn't prove either is correct—it only proves they share the same blind spots.”

— paraphrased from a data-validation lead who watched three linked cohorts contradict each other on the same drug

Can't resolve all disagreements

Some discrepancies are structural, not methodological. One source codes a diagnosis by billing code; the other by a lab result threshold. They're measuring different constructs entirely—and no matching algorithm can fuse them into one truth. The heart of the riddle is that 'significant' in one data world means 'systematic noise' in another. I have watched crews spend two months chasing a disagreement that boiled down to a five-day window difference in outcome recording. The approaches looked rigorous. The p-values looked pretty. The answer was still off.

The fix is not more math. The fix is accepting that some cross-source questions can't be settled with data alone—they demand a domain expert to say, 'these two columns don't mean the same thing, stop pretending.' That's the limit no R-squared can address.

Reader FAQ

Can I combine EHR and registry data to get a less noisy answer?

Technically, yes. Practically, that shift often multiplies your headaches. I have seen crews merge Electronic Health Record streams with a disease registry and expect a cleaner signal—only to watch the noise compound. The catch is each source carries its own bias. EHR data leans on who shows up to clinic visits, while registrie tend to select for patients who complete follow-ups. When you stitch them together, you're not averaging away noise; you're stacking two different selection filters. A direct SQL join rarely works. What does work is a Bayesian hierarchical model that treats each dataset as a separate likelihood, then shrinks estimates toward a shared prior. That's non-trivial. Most units skip this and simply average the two point estimates—flawed sequence. You don't fix bias by pooling biased inputs. If you must combine, preserve the source-identifier column and run a sensitivity analysi with the registry excluded. The seam often blows out when one dataset says 'positive' and the other says 'null.'

Archery tiller, fletching glue, nock fit, chronograph speeds, and bare-shaft tuning expose ego before groups.

Fjords kelp basalt look wild.

Merging two noisy dataset without modeling their differences is like averaging two blurry photos and expecting a sharp image.

— data engineer, health analytics unit

How do I know which result to trust when they disagree?

Short answer: trust the source that records the outcome assignment closest to the actual event. registrie typically follow a pre-registered protocol with defined endpoints. EHR data captures whatever a clinician typed—often missing the precise timestamp of diagnosis or treatment start. But it's not that clean. A registry might require manual data entry that lags by six months; that delay can flip a protective effect into a harmful one if disease progression is fast. The practical heuristic: check the date-stamp granularity. If one source logs events within hours and the other within weeks, the finer-grained source wins on temporal accuracy. That said, I have seen cases where the registry had stricter enrollment criteria, so the effect in the EHR was real but diluted by mild cases that would be excluded from the registry. Neither is right—you're looking at two different populations. The honest answer is often: both results are valid for the patients they actually captured. Your job is to declare which population you're trying to generalize to.

The tricky bit is when the two sources show significance in opposite directions. One trial-supplemented registry found that a diabetes drug reduced HbA1c; an EHR analysi of the same drug in the same calendar years found a slight boost. The difference? The registry patients were predominantly on monotherapy, while the EHR patients were on multiple drugs—polypharmacy masked the effect and introduced a drug-drug interaction. Not an artifact—a real biological difference. You can't resolve that by picking a winner. You have to stratify by concomitant medications and re-run both analyses.

What if both sources show a significant effect but in opposite directions?

That hurts. It's rare, but when it hits, it signals a confounding by indication repeat that flips sign across dataset. The classic scenario: sicker patients preferentially get treatment X in one dataset, so X looks harmful; in the other dataset, treatment is given to healthier patients, so X looks protective. The solution is not to average the two numbers—that would produce a null result that's off for both subgroups. You need a propensity score model per source, then compare the balance diagnostics. If the covariate overlap is poor in one dataset, discard that subset. Or better: restrict both analyses to the overlapping region of the propensity score distribution. I did this once with a hypertension drug—the EHR showed a 12% risk reduction, the registry showed a 9% risk increase. After restricting to patients with propensity scores between 0.3 and 0.7 in both sources, the effects converged to a null: no benefit, no harm. The apparent contradictions were entirely driven by non-overlapping patient severities. That's the hidden riddle—the data sources are not disagreeing; they're describing different patient worlds. The crosswalk between those worlds is what you have to build.

Practical Takeaways

Pre-specify analyses — then expect to break them

Most teams skip this: they pull data from two registrie, fire up the same regression, and call it a day. off order. The first practical move is to write your analysis roadmap before you see any source-specific results. I have seen labs waste weeks chasing discrepancies that only existed because one dataset truncated follow-up at 12 months while the other cut at 18. You fix that by forcing the same censoring rule into the spec upfront. The catch is — pre-specification hurts. It forces you to guess which confounders matter across sources, and you will guess wrong sometimes. That's fine. The plan is a starting point, not a cage.

Run sensitivity analyses across sources — let them fight

You have two dataset that disagree on the treatment effect. Don't average them. Instead, push each source to its breaking point. Trim the sickest patients from one registry, drop the healthiest from the other — does the effect direction flip? If yes, you found a selection seam. That seam is the real finding. One concrete anecdote: a colleague once had an A1c drop of 0.8% in a claims-derived cohort and 0.3% in a hospital registry. The gap vanished after restricting both to patients with at least two HbA1c readings in the baseline window. The noise was measurement density, not biology. Sensitivity analyses expose that.

Triangulate with other evidence — registrie lie less in threes

A single cross-source comparison still leaves room for systematic bias. You triangulate. Pull in a third source — maybe a trial extension, a health survey, or even published meta-analyses. The rhetorical question is: would three independent measurement processes produce the same signal under different error structures? If two registrie say the treatment works but a third, completely different data source (say, pharmacy-dispensing records) shows no pattern, you have a measurement artifact, not a real effect. Em-dash here: the pitfall is over-weighting the most accessible source. The hardest data to get is often the truest signal. Use a simple rule: any effect that survives across three measurement modes is worth designing a study around. Effects that only show up in one? Publish the contradiction as a methods note. That's actionable — and honest.

‘Triangulation doesn't guarantee truth. It guarantees you have looked hard enough to find the flaws.’

— paraphrase of an epidemiologist who spent a decade reconciling four cancer registries

End with this: next time your dashboard shows a significant result in one source and noise in another, don't pick a winner. Pre-specify the comparison, stress-test both datasets, then bring in a third lens. The answer is rarely in one column — it lives in the friction between them.