One of the most expensive failures in R&D is also one of the least visible.
The experiment succeeds.
That is the failure.
A team runs a sequence of experiments. Most are incremental. Then one result moves materially. A metric improves. The result looks important.
The first question should be straightforward:
What changed?
Too often, the organization cannot answer with confidence.
The model artifact exists. The metric exists. There may even be an experiment record. But the exact configuration, code revision, dataset state, preprocessing, evaluator, environment, seed, or protocol is missing—or scattered across enough systems that reconstruction becomes uncertain.
The organization has produced a valuable result without preserving the knowledge required to explain, reproduce, or safely generalize it.
I have seen versions of this problem repeatedly in applied AI and research environments. The more I have worked on systems that connect research to production, the less I see this as an experiment-tracking problem.
It is an evidence architecture problem.
And for AI organizations, that makes it a leadership problem as much as a tooling problem.
The result is not the evidence
Modern R&D organizations are very good at generating observations.
Training runs produce metrics. Evaluation pipelines produce scores. Dashboards show movement. Experiments create artifacts. Researchers write conclusions.
But a statement such as:
WER improved by 8%.
or:
Accuracy increased from 84.2% to 87.1%.
does not establish very much on its own.
Its meaning depends on the conditions under which both measurements were produced.
- Were the datasets identical?
- Was the evaluator unchanged?
- Did preprocessing change?
- Was the test population the same?
- Did multiple experimental variables move simultaneously?
- Was the baseline recomputed?
- Was the code identical apart from the intended intervention?
A metric is an observation.
For it to become useful evidence, its provenance and comparability conditions have to survive with it.
That distinction matters more as organizations scale. In a small team, context is often carried by the person who ran the experiment. In a larger organization, that context has to be carried by the system.
Improvement without comparability
Consider a common scenario.
A team has a baseline:
M0 = 0.72
A later experiment produces:
M1 = 0.78
The natural conclusion is:
ΔM = +0.06
and therefore the intervention improved the system.
But suppose the evaluator changed between the two runs.
Or the test dataset changed.
Or both the model and preprocessing changed.
The subtraction is still mathematically correct. The scientific inference may not be.
What should have been represented as:
We observed 0.78 under a different experimental condition.
is gradually transformed into:
The new model improved performance by six points.
That statement enters a status report.
The report enters a roadmap discussion.
The roadmap influences a product decision.
Eventually, an uncertain comparison becomes organizational truth.
The problem is not dishonesty. It is that the system has no durable way to carry the conditions attached to evidence forward with the result.
This is how technical debt becomes decision debt.
Attribution is harder still
A related problem appears when several variables change together.
Imagine an experiment that changes:
- model architecture,
- training data,
- augmentation,
- optimizer settings,
- and evaluation configuration.
The result improves.
Which change caused the improvement?
Strictly speaking, the experiment may show only that the combined system performed better under those conditions. It does not establish which intervention deserves causal credit.
Yet organizations naturally compress complex findings.
A few weeks later:
“The additional training data gave us the improvement.”
Three months later:
“We know this data strategy works.”
Six months later, another team designs its roadmap around that conclusion.
This is how organizations accumulate beliefs faster than they accumulate evidence.
The issue is not that every experiment must be perfectly controlled. Real-world research often involves exploratory changes, messy systems, and practical constraints.
The issue is whether the organization can distinguish:
- what was observed,
- what was inferred,
- what remains confounded,
- and what should be tested next.
That distinction is essential for making good bets under uncertainty.
The scaling problem is organizational memory
Small research teams often handle these problems surprisingly well.
The person who ran the experiment remembers that the evaluator changed.
Someone remembers why a configuration was unusual.
The researcher who designed the study remembers which hypothesis was actually being tested.
Context lives in people’s heads.
That model works until:
- experiment volume increases,
- teams become distributed,
- projects last longer,
- researchers change roles,
- multiple organizations contribute to the same system,
- or decisions need to be revisited months later.
At that point, memory becomes part of the technical architecture whether we acknowledge it or not.
And human memory is a poor system of record.
The failure becomes visible through questions that sound operational but are actually strategic:
Why did we make this decision?
Which experiment established this?
Can we reproduce the best-performing version?
Are these results actually comparable?
What changed between these two runs?
Which hypothesis did this experiment resolve?
What evidence supports this claim?
Is this project genuinely progressing toward the original objective?
Most organizations can answer some of these questions.
Far fewer can answer all of them reliably, quickly, and without pulling senior researchers away from current work.
That is the hidden tax of weak scientific memory: every important decision requires archaeology.
Experiment tracking solves only part of this
We have built increasingly sophisticated infrastructure for tracking research:
- experiment trackers,
- model registries,
- source control,
- data versioning,
- feature stores,
- observability platforms,
- notebooks,
- dashboards,
- project-management systems,
- document stores.
These are necessary.
But collectively they still leave an important gap.
They tell us a great deal about what exists.
They do not necessarily establish:
what can validly be inferred from what exists.
An experiment tracker can tell me that two runs produced different scores.
A source repository can tell me that code changed.
A model registry can tell me which artifact was deployed.
A ticket can tell me why someone wanted to try something.
A document may contain the conclusion.
The missing layer is the relationship between all of them:
Intent → Hypothesis → Experiment → Conditions → Observation → Evidence → Claim → Decision
That chain is where much of the actual scientific knowledge lives.
It is also where research becomes legible to the rest of the company.
A strong AI organization cannot rely on researchers to repeatedly translate raw experiment history into decision-ready context by hand. The operating system for research has to preserve enough structure that technical judgment can scale beyond the people closest to the work.
Comparability should be a first-class object
One design principle follows from this:
A comparison should not be implicit.
If Run B is described as improving upon Run A, the relationship between those runs should itself have evidence.
At minimum, the system should know whether material dimensions remained constant:
- data,
- evaluation population,
- evaluator,
- protocol,
- environment,
- relevant configuration,
- intended intervention.
This allows a research system to distinguish between several fundamentally different statements.
Comparable
B outperformed A under sufficiently matched conditions.
Non-comparable
B produced a higher number, but material evaluation conditions differ.
Confounded
B differs from A across multiple interventions, so attribution is unresolved.
This sounds obvious when written explicitly.
It is remarkably easy to lose in day-to-day research operations, especially when teams are moving quickly and the pressure is to convert every positive result into a narrative of progress.
The goal is not to slow research down with bureaucracy. It is to make the confidence level of a result visible at the moment it is used to make a decision.
That is a leadership responsibility: create systems that preserve speed without allowing ambiguity to masquerade as certainty.
Claims should also be first-class objects
The next step is separating observations from claims.
Suppose an experiment establishes:
Metric X increased under configuration Y.
A researcher may infer:
Technique Y improves Metric X.
That may be justified.
But the second statement is a claim, not an observation.
A stronger research infrastructure should allow us to ask:
- Which observations support this claim?
- Which experiments contradict it?
- Are the supporting comparisons valid?
- Has it been reproduced?
- Under which conditions does it hold?
- How confident should we be?
- Has newer evidence invalidated it?
In other words, claims should have lineage.
This is not academic overhead. It is how an organization avoids repeatedly reopening settled questions—and how it notices when a previously accepted conclusion no longer holds.
Long-lived AI systems accumulate assumptions. Some are explicit. Many are not. A claim with visible lineage can be challenged, updated, or retired. A claim without lineage becomes folklore.
That distinction matters when deciding whether to:
- scale a model,
- invest in new data,
- change an evaluation standard,
- commit to a product capability,
- or stop funding a line of work.
Research progress must remain connected to intent
There is another failure mode that has little to do with reproducibility.
Teams can conduct excellent research and still drift away from the original objective.
This usually does not happen dramatically.
A proxy metric improves.
That metric becomes easier to optimize than the actual problem.
The team runs more experiments against it.
Progress begins to mean:
the metric moved.
instead of:
the evidence supporting our original objective became stronger.
Over time, an organization can become extraordinarily efficient at improving something that no longer captures the problem it set out to solve.
This is one of the central management challenges in AI: metrics are necessary for execution, but metrics can also narrow attention. The job of leadership is to keep local optimization connected to the outcome the organization actually cares about.
Evidence infrastructure should preserve another relationship:
Objective ↔ Hypotheses ↔ Evidence
Then progress can be discussed in terms of evidence coverage, not merely experimental activity.
That creates a more useful conversation between researchers, engineering leaders, and executives.
Instead of:
We ran 140 experiments this quarter.
the questions become:
What do we know now that we did not know before?
Which important uncertainties have actually been reduced?
Which assumptions still block the product or research objective?
What is the next experiment with the highest expected information value?
That is a much stronger operating cadence than counting runs.
The solution is not more documentation
It is tempting to respond to all of this with process.
Add another experiment template.
Require more fields.
Create a checklist.
Ask researchers to write better documentation.
That can help temporarily.
But there is a fundamental scaling problem:
Every additional requirement competes with the work researchers are actually trying to do.
The better approach is to make evidence capture and validation increasingly structural and automatic.
When an experiment runs, its relevant identity should travel with it.
When results are compared, material differences should be detected automatically.
When several variables move simultaneously, attribution risk should become visible.
When a claim is made, its evidence lineage should be inspectable.
When evidence changes, dependent conclusions should be reconsidered.
When an objective accumulates activity without corresponding evidence, that distinction should be visible.
This is the difference between documenting rigor and encoding rigor into the operating environment.
It is also the difference between a research process that depends on exceptional individuals and one that can support a larger organization.
The best systems do not ask people to remember everything. They make the important things difficult to forget.
Deterministic systems should establish facts
The rise of AI agents introduces an interesting temptation here.
Many of these questions can be phrased naturally:
Are these experiments comparable?
What caused this improvement?
Does the evidence support this claim?
It would be easy to put an LLM in front of the experiment history and ask it to answer.
I think that is the wrong abstraction.
When something can be established from structured evidence, it should be established deterministically.
If two experiments used different evaluators, we do not need a language model to speculate about comparability.
If a required configuration is absent, we do not need probabilistic reasoning to detect that.
If five controlled variables changed, that fact should not depend on a model’s interpretation.
AI becomes valuable after those facts have been established.
An agent can help answer harder questions:
- Which confound should we investigate first?
- What previous experiments are relevant?
- Which evidence appears contradictory?
- What experiment would reduce uncertainty most efficiently?
- Does this finding interact with another objective?
- What should the team investigate next?
This creates a useful division of responsibility:
Deterministic systems establish the evidence.
AI helps reason about what to do next.
That boundary is important for more than technical correctness. It is also a governance principle.
As AI systems become more capable, leaders need to be clear about where probabilistic judgment is appropriate and where the organization should require auditable facts. The system should not use an LLM to manufacture confidence where the underlying record is incomplete.
The emerging architecture: scientific memory
Putting these pieces together suggests a category of infrastructure that most R&D organizations do not yet explicitly have.
Not another model registry.
Not another experiment tracker.
Not another documentation system.
Something closer to organizational scientific memory.
A system capable of remembering:
- what was attempted,
- why it was attempted,
- under which conditions,
- what changed,
- what was observed,
- what could legitimately be compared,
- what remained confounded,
- which evidence supported which conclusion,
- what the organization decided,
- and what happened afterward.
The interesting property of such a system is not that it stores more information.
It preserves the relationships that give information scientific meaning.
That has direct implications for how an AI organization operates:
- research decisions become easier to review,
- engineering handoffs become less lossy,
- product commitments rest on clearer evidence,
- failed experiments remain useful,
- new researchers ramp faster,
- and leadership can see where uncertainty is real versus where it is merely an artifact of poor record-keeping.
This is not about replacing scientific judgment. It is about giving judgment a better substrate.
What this changes for research leadership
For an individual researcher, the immediate benefit is reproducibility.
For an organization, the implications are larger.
Research leadership should be able to distinguish:
Activity
from
evidence generation
from
knowledge creation.
Those are not the same thing.
A team may execute hundreds of experiments while resolving very little uncertainty.
Another team may run ten carefully designed experiments and fundamentally change what the organization knows.
Most operational systems are much better at measuring the former.
A mature evidence architecture gives leadership a better way to understand the latter.
It also changes how research survives organizational change.
When a researcher leaves, the experiments should remain.
More importantly, the reasoning behind the experiments should remain.
When a project pauses for a year, restarting it should not require reconstructing its intellectual history from Slack.
When leadership challenges a technical assumption, the answer should not depend primarily on who happens to remember the meeting where it was discussed.
This is where the role of an AI VP becomes concrete. The responsibility is not simply to sponsor more models or more infrastructure. It is to build an organization in which research, engineering, product, and leadership can make decisions from a shared and trustworthy understanding of what is known, what is uncertain, and what should happen next.
That requires technical architecture, operating discipline, and judgment about where to standardize and where to preserve autonomy.
Toward more predictable R&D
Research itself will never be fully predictable.
That would defeat the point of research.
We explore precisely because we do not know the answer.
But there is an important difference between uncertainty inherent in the problem and uncertainty created by our own infrastructure.
We cannot eliminate the first.
We should be much more ambitious about eliminating the second.
A failed experiment is useful knowledge.
A surprising result is useful knowledge.
A disproven hypothesis is useful knowledge.
A result whose configuration disappeared is simply lost knowledge.
A comparison whose conditions cannot be reconstructed is ambiguous knowledge.
A conclusion disconnected from its evidence is fragile knowledge.
As research organizations scale, reducing those failures becomes an infrastructure and leadership problem.
The long-term opportunity is not to make science less exploratory.
It is to ensure that exploration compounds.
Every experiment should leave the organization knowing something it can reliably build upon.
Every major decision should make clear which evidence supports it and which assumptions remain open.
Every AI system should be evaluated not only on whether it produces a better output, but on whether the organization can explain why it believes the output is better and under what conditions that belief holds.
That requires moving beyond experiment tracking toward an evidence architecture where provenance, comparability, claims, decisions, and objectives remain connected.
In other words:
Scientific rigor should not depend on remembering to be rigorous.
It should increasingly be a property of the system in which research happens—and a defining capability of the organization that leads it.
Disclosure: I created this piece of content for the purposes of entering the All Things Agentic Hackathon. The problems and perspectives discussed here are informed by my broader experience working with applied AI and R&D organizations.