As every disease area pins its hopes on blood biomarkers, it is instructive to see what these tests are actually being used for in Alzheimer’s.

It strikes me that the current usage of blood biomarkers in dementia will be a crutch that is immensely helpful for patients and clinicians to take patient’s future risk of dementia seriously. Patients can easily initiate conversations about it; tests become a re-imburseable expense and so forth.

From a technological standpoint, we are not doing a more precise job of measuring dementia risk, and we are nowhere closer to capturing a more narrow biological process that better differentiates heterogeneous paths to developing dementia. The scope of equivalence between blood-biomarker based decisions and PET imaging based decisions is still narrow. The utility is primarily in cost-reduction without reducing accuracy, which is a big deal in terms of accessibility.

In the long run, we don’t yet know the price we will pay in terms of causal specificity for substituting the tissue of interest with blood and how that intersects with a wide swarth of the general population being measured instead of a targeted population.

The recommendations in the new CPG — both of which apply only to patients with cognitive impairment being seen in specialized care for memory disorders — are:

• BBM tests with ≥90% sensitivity and ≥75% specificity can be used as a triaging test, in which a negative result rules out Alzheimer’s pathology with high probability. A positive result should also be confirmed with another method, such as a cerebral spinal fluid (CSF) or amyloid positron emission tomography (PET) test.

• BBM tests with ≥90% for both sensitivity and specificity can serve as a substitute for PET amyloid imaging or CSF Alzheimer’s biomarker testing.

“Not all BBM tests have been validated to the same standard or tested broadly across patient populations and clinical settings, yet patients and clinicians may assume these tests are interchangeable,” said alz.org/press/spokespeo…, Alzheimer’s Association vice president of scientific engagement and a co-author of the guideline. “This guideline helps clinicians apply these tools responsibly, avoid overuse or inappropriate use, and ensure that patients have access to the latest scientific advancements.”

Review Note

November 17, 2025

Box’s “Improving almost anything” applies to dieting too.

What would reduce the friction to actually implementing optimal experimental designs dynamically for personal health experiments?

• In reference to: “​​Optimal experimental design is difficult. I’ve usually tried only one new food at a time. In retrospect, this was not optimal, especially near the beginning of the process. If you eat five foods you are unsure of, wait a week, and get no symptoms, then all five are fine. If you get sick, at least one of them was a trigger, but you don’t know which one, and it may have been more than one. Then you could try two of them at a time, and so on. Proceeding this way would get you more information faster, at an upfront cost of being more sick more often. There’s whole subfields of statistical theory devoted to optimizing this sort of process. For a certain sort of geek, using that would be a lot of fun. I am that sort of geek! But it didn’t occur to me to try this until a year into the process.3 And there are several reasons it would be difficult. So I still haven’t done i”

November 17, 2025

Today: Research syntheses, quantitative systematic reviews, position papers usually take all previous reported research articles at face value.

Future: A research synthesis will dynamically re-analyze all prior work to make as both as relevant to the question at hand and as accurate given all that has been discovered since the original publication.

*Individual patient data re-analysis is done for certain kinds of meta-analyses but very restricted to clinical trials, usually same drug-outcome combination. Nobody does this for larger scale integration of different types of studies and levels of evidence.

November 17, 2025

If everyone is following the hottest trends written up in 5 year strategic research plans, it might definitely be important, but it is also mainstream.

• In reference to: “It can be easy to delude yourself into thinking that you are doing something radical and creative as an expression of your own deep interests, when in fact you are doing what everybody around you is doing.”

November 17, 2025

The 4 types of patients we can only distinguish in Lake Wobegon,

where all counterfactuals are fixed, (no noise)

no one has carry-over, (no history)

and

every patient is helpfully deterministic (stable individual effects)

November 17, 2025

The 4 types of patients we can only distinguish in Lake Wobegon- II

In practice, you cannot use data from a standard RCT to classify patients into these 4 buckets because we only ever observe outcome under placebo, R(0) or outcome under treatment R(1) but never both at the same time.

• Always responders and treatment responders can’t be distinguished

• Always responders and placebo responders can’t be distinguished.

Review Note

November 18, 2025

Why is legally hard to share clinical trial data when companies like EPIC can just strip personal information and re-sell it to so many other people?

5 months to get access to clinical trial data on VIVLI (likely more if it is multiple trials)

whereas NIH repos that I will not name just have the de-identified data downloadable on their website with a simple data use agreement!

November 18, 2025

Gobburu (2009) points out that biomarkers can act as surrogates for drug development decisions without having to be surrogates for regulatory approval. The scientific concerns of surrogacy remain the same but the degree of accuracy / risk-reward will be different.

Changes in fasting plasma glucose or international normalized ratio (INR) are good surrogates for dose selection but not for regulatory approval.

Caveat: A lot has changed since 2009 since we expect many more tools for evaluating drugs go through a qualification process now.

November 22, 2025

The vast majority of neuroimaging studies that estimate heritability of various biomarkers use kinship/pedigree or twin study estimates.

• In reference to: “Kinship-based models therefore provide us with a mushy estimate of narrow-sense heritability plus an unbounded amount of environmental confounding”

There are broadly two approaches to analyze data and distribute results when the data is not available to share for legal and privacy reasons.

• Federated learning

• Obscure the data via differential privacy or synthetic data generation

Where is the normative theory for when synthetic data is trustworthy?

There are no universal surrogate endpoints for therapeutic evaluation, because there is no way to produce a universal surrogate for every possible decision. If something is a good proxy, it has been shown to be a good proxy for a particular purpose. The same goes for synthetic datasets when used in-lieu of the real thing. How do you just generate synthetic data in a vacuum or pre-emptively before knowing what decisions are going to be made on it?

The only one I really understand is differential privacy due to its theoretical origins, but there are a wide variety of synthetic data generation schools of thought and approaches today. In practice, many of them are based on intuitive heuristics. But how do we what they are good for and when such data might be misleading?

• Counterfactual imputation of missing data is “synthetic” data

• LLM benchmarks: take a seed template problem and using language models to scale up many variations of the problem. This makes problems realistic but “synthetic” in that they are not written up by humans as in FrontierMath.

• Generative models trained on large biological datasets as a source of new synthetic observations. We see this right now in both brain imaging research and genomics.

In short, how do we know that a synthetically generated dataset produces optimal and valid answers for every kind of scientific question and decision that could be made on it?

Many years ago, I began calling this type of situation the Poincaré-Lippman gap. I first used it publicly in a very popular thread about many sciences being too empirical and anti-theory. One consequence of being anti-theory is that too many scientists, especially life-scientists, fail to test and evaluate assumptions needed to draw valid inferences about their studies.

History

In the early days of probability and statistics since the late 19th century, the normal or Gaussian distribution1 as a probability model grew in popularity across many fields.

At the time Poincaré discussed how mathematical assumptions about normality were a great confusion between empirical vs. theoretically minded researchers.

Everyone is sure of this [that errors are normally distributed], Mr. Lippman told me one day, since the experimentalists believe that it is a mathematical theorem, and the mathematicians that it is an experimentally determined fact.

Henri Poincaré, Calcul des probabilités (2nd ed., 1912), p. 171

Causal discovery edition of the Poincaré-Lippman gap

Scientists think quantifying causal effects and causal discovery is mathematically guaranteed via do-calculus and algorithms, while mathematicians think these tools offer principled scaffolding to aid empirical hypothesis testing in future studies.

If you don’t believe me, Stephen Gillman once exactly articulated what I’ve observed to be a ubiquituous misconception among scientists who read Judea Pearl’s work including Book of Why.

There are likely many such examples out there. shared another instance of this type of misunderstanding

Reminded of the situation in the late 1980s when philosophers of mind thought AI people had a coherent understanding of how representation worked and AI people thought philosophers of mind had a coherent understanding of that and suddenly around 1991 both disciplines realized neither did

If you have encountered such examples in your field, please share them in a comment!

Subscribe now

• Markers of psychiatric disorders are neither necessary nor sufficient

• Modern causal inference for psychiatric kinds

• Subjectivity is not the problem for psychiatric endpoints

• The papers that change you

• The power of estimands (for digital twins)

Markers of psychiatric disorders are neither necessary nor sufficient

Many people (even MDs) use flawed heuristics like

• X responded to stimulants → therefore must have ADHD (not necessarily)

• X improved in a different environment → therefore not ADHD (nope, not necessarily)

All this boils down to the fact that markers of complex psychiatric disorders are neither necessary nor sufficient for the condition.

Great illustrations of the problem in this post from

“So these stories of “I can function better because I am doing work that I am highly interested in” aren’t counterexamples to ADHD as a “real” clinical disorder. They demonstrate how the distress and impairment of ADHD is a product of individual-environment interactions and that sometimes environmental change can be a powerful intervention for individuals with ADHD.”

Modern Causal Inference for Psychiatric Kinds

In 2019, I gave a lab talk on how we need to embrace homeostatic property clusters for psychiatry and that modern biostatistics/ causal inference / theoretical epidemiology offer some clues. A few months before Ken Kendler made the same point! As far as I know, this has still not happened. I don’t think the computational psychiatry community reads this stuff, or takes “real category ≠ biological essence” seriously enough to change their methodological playbook.

You can’t just throw canonical ML/DL/foundation models at this stuff without thinking about changing what you are fundamentally asking them to do.

asked for a concrete example, so here was my response.

Clustering does not give you relevant psychiatric kinds

The standard playbook is do something like 1) measure biology 2) learn predictors of clinical phenotype (usually cross-sectional) 3) cluster representations of the predictive model

The problem then is that there is no guarantee that “statistical” clusters correspond to sufficient causes [1]. IMO thinking about sufficient causes or other equivalent frameworks that acknowledge that component causal factors maybe neither necessary nor sufficient for phenotype to manifest in one individual is important. If we truly want to learn then, then we should reverse engineer how we approach measurement + quantitative methods to get close to this.

Causal Pies / Sufficient Component Causation

One of the ideas in the causal inference literature is to acknowledge that forward causal inference i.e manipulating A, B, C, D, E and learning that they make a difference in outcome Y only identifies a component cause. But A maybe neither necessary nor sufficient to produce an outcome (e.g diagnosis of treatment resistant disorder). It can only be necessary if every possible way for outcome to manifest always has A as one of the causal factors. On top of this, the way we measure all these component causes may result in different kinds of confounding, differential measurement error, failure to observe all possible component causes, etc..

But thinking about modeling while explicitly keeping pluralistic causes in mind helps get around to the idea of psychiatric kinds as homeostatic property clusters i.e we distinguish subtypes of a disease by differences in shared causal mechanisms giving rise to it; but mechanism here includes everything not just biology. They are of course likely to be a bit more fuzzy IRL than what the models make it look like.

The causal pies formulation of Lewis’s counterfactual causes is popular in epidemiology. Incidentally, Kendler also seemed to be inspired by Rothman/epidemiology in his criticism of monocausalities in psychiatry [2]

It goes by learning probability of necessity/ probability of sufficiency, learning sufficient cause interactions, etc.. in more modern work.

To be fair the field does think about it conceptually — that is why there is so much more experimentation with what we measure (different modalities, clinical perturbation, etc..) , but then they just get excited about the standard ML playbook instead of changing it to suit their needs.

To make this concrete: why do we adopt such a narrow lens to understanding “treatment resistance” of any disorder? This confuses the pragmatic clinical stance of what to do with treatment resistance being a strictly biological kind. What would we learn if we considered that either genetics (e.g rare variant burden) or some neurobiological circuit dysfunction may sometimes make up one sufficient cause for some people, but may generally be neither necessary nor sufficient because this interacts with all the ways disease progresses over a lifetime in the presence of other causal factors as well e.g beliefs / belief level representations x maybe other kinds of psychosocial factors that also make up component causes. Finding neuriobiological predictors of “treatment resistant” status is likely to contain all kinds of misleading representations.

Also it is possible that I have missed more recent work directly tackling these kinds of issues. So I welcome being corrected on this. I do recall for example that

[1] https://www.sciencedirect.com/science/article/pii/S0167865515001269

[2] https://pubmed.ncbi.nlm.nih.gov/31215968/

Follow the rest of the thread here

Subjectivity is NOT the problem with psychiatric endpoints

I was recently at a biopharma <> stats workshop and was disappointed to see that statisticians on the one hand who are trained to identify problems of degeneracy, lack of causal identification, etc…, unquestioningly buy intomyths like > “psychiatry is challenging to study because patient endpoints and symptoms are subjective and we don’t have endpoints/biomarkers that are objective”.

This sloppy thinking is everywhere and really deserves to be challenged.

Just how illogical is it to

• accept that symptoms are “subjective” AND

• that we will find “objectivity” by correlating neurobiology with symptoms.

If your instruments for measuring symptoms are not measurement invariant and if the vast majority of them are simply screening questionnaires and other very non-specific measurements then yes, your whole approach is effed up and isn’t going to yield anything generalizable or more granular.

I had to challenge a panel to consider that perhaps the more useful thing would be a far richer set of patient-specific longitudinal measurements e.g like the idiographic approach to psychopathology instead of bemoaning the subjectivity of patient measurements. Everywhere else patient reported measurements are getting richer and taken more seriously while the recent resurgence in psychiatric drug development is embracing some impoverished narratives.

Personally I’m a bit of an activist methodologist — if you care about drawing correct scientific inferences then you have to be invested in the scientific question yourself and be prepared to tackle theoretical flaws end to end. We aren’t going to get much progress by strictly deferring on scientific questions to domain experts with a “throw things over the wall” approach. The virtue of being theoretical scientists is that you can spot conceptual problems a mile away. Every theoretical / conceptual problem will bleed over into the experimental design and quantitative evidence. To ignore all that amounts to rigor-washing and ultimately this does a dis-service to patients.

Awais Aftab’s writing in the last few years lays bare a lot more of the better thinking behind the scenes that is less well known but deserves to be more widely understood. I just don’t see how we are going to get the revolution in therapies we deserve until we stop attempting to do target discovery, biomarker development etc.. with old reductionist playbooks that just don’t correspond to how things really are. And the virtue of all this interest in neurotech is that we don’t have to be naive biological reductionists anymore. Everything kind of measurement is now on the table, we don’t have to choose out of methodological convenience.

• In reference to: “Treatments likewise tend to have broad, transdiagnostic effects across mental functions. Trials may be anchored to a target diagnosis, but the causal traffic usually runs through mechanisms that cut across our labels. “Physiological,” “psychological,” and “sociocultural” are not sealed ontological provinces; they are overlapping languages for a single, complicated reality. The neurophysiological strand is one thread among many —experiential, sociocultural, existential-and not always the most important one. Even so, because the mind is embodied, bodily mechanisms can be leveraged to produce desired effects, whether or not they count as “dysfunctional” in any simple factual sense. We should resist a priori privileges for either technological fixes or hermeneutic readings. The posture must be Jaspersian: causal explanation and meaningful understanding as partners, with their relevance varying case by case and…”

The papers that *change* you.

Those papers that you spent 1-12 months reading, re-reading, working through every result are the ones that make you.

And no you can’t read them like a novel or even a science paper.

Once you do this, you will not flinch at tossing out a paper with incomprehensible math in a science journal. Mathematistry is not impressive.

The power of estimands

If you are bullish about AI digital twins, virtual patients, virtual clinical trials, clinical trials in a dish and all that but you have never heard of estimands*, it is going to get a lot worse before it gets better.

* If you have heard of per-protocol effects vs. intention to treat effects, then you know just a tiny bit about clinical trial estimands. There is also the full universe of estimands for all science.

HDL serum cholesterol, for instance, is great for evaluating risk of heart disease but not for evaluating effectiveness of treatments to improve cardiovascular health. There is no such thing as a “good biomarker” in a vacuum. Decisions to use biomarkers are always dependent on the intended applications.1 Sadly, this is not something that most biomedical researchers think about when they do biomarker discovery.

Back to glossary

Biomarker guided therapeutic decisions require developing and validating biomarkers. Specifying what these criteria are requires constant meta-scientific innovation. It is easy to conflate the enterprise of biomarker validation with analytical validation2 , followed by reproducible clinical studies. But what constitutes clinical validation?

Analytical validation is all about evaluating the measurement process or assay. Many biomarker discovery studies will demonstrate test-retest reliability that looks for whether people can be reliably differentiated based on their biomarker measurements. One has to evaluate a far more thorough checklist of measurement issues that go well beyond test-retest reliability for analytical validation. We also need repeatability of measurements for any individual with good tolerance intervals, comparability of quantitative measurements in a wide variety of circumstances and many others. Yet, analytical validation is the easier part of the biomarker evaluation process with systematic criteria. Qualification on the other hand encompasses the full spectrum of validity problems across all the life-medical-health sciences — it includes all the possible “does it mean what you think it means” problems. Biomarker qualification includes assessing the clinical validity3 of the biomarker as well as other validation criteria specific to a therapeutic decision.4

Importantly, there is no easy way to preemptively specify what all the threats to validity are — construct validity, causal validities including internal validity and external validity, all the modern validities beyond reliability of the measurement that link it to disease and/or therapeutic outcomes.

Altar et. al. (2008)

Here is a concrete example of what a comprehensive understanding of biological and clinical validation looks like.

Credit: Altar, C.A. et al. (2008) ‘A prototypical process for creating evidentiary standards for biomarkers and diagnostics’, Clinical pharmacology and therapeutics, 83(2), pp. 368–371. https://doi.org/10.1038/sj.clpt.6100451.

However, this table reflects 20th century understanding. It could use significant updating given how far scientific and statistical methodology has come in 20 years.

Share

Checklist for a biomedical stakeholder

Analytical validation and biomarker qualification are terms of art when biomarkers are proposed for drug development decisions in clinical trials. Unfortunately, these terms are not widely used within mainstream biomedical research. It is easy to think these are regulatory concerns don’t matter until one wants to bring biomarkers to the clinic, as opposed to scientific concerns that need to be addressed. Every research community has its own epistemic norms around “validation”. When you visit premier conferences in different niches of life science where biomarker research occurs, these differences become apparent. No one actually owns the problem of understanding the full scope of scientific R&D that needs to occur.

If you read an article that calls for a large-scale validation for new biomarkers, here is what you should ask yourself —

1. Is it clear that the biologist or scientist’s notion of validation is distinct from analytical validation? Has it at least considered all the problems in Altar 2008, for instance?

2. Does the roadmap for biological plausibility and clinical validation cover the full spectrum of research designs and grades of evidence that need to be generated?

3. Does it address all threats to scientific validity known to methodologists for a particular decision or context of use, even outside one disease area?

If not, then the field might need a better specification of the validation roadmap. It is far too late to do the necessary R&D if you wait until someone is ready to initiate conversations with the FDA.

Subscribe now

Definition: Intercurrent events are post-baseline events (or post-randomisation events in randomised trials) that affect the interpretation or existence of outcome data. These events frequently affect receipt of treatment (eg, treatment switching or treatment discontinuation) or preclude existence of the outcome (eg, death, if it is not defined as part of the outcome).

There are a dizzying number of ways that your clinical trial might chose to handle intercurrent events. Historically these decisions were made implicitly. Clinical trial methodologists everywhere found the failure to specify how intercurrent events are handled to be problematic. So much so that good clinical trial hygiene demands such specification according to guidelines from the de-facto international community for standardizing clinical evidence generation. Intercurrent events are only one component amongst many other aspects of how a trial operationalizes the evaluation of their clinical hypothesis — how a trial is designed, what is measured, the quantitative objective, and the procedure to produce quantitative estimates.

Kahan et. al. 2024 have a nice introduction to 5 different ways one might handle an intercurrent event in their review.

Figure: Different strategies regarding intercurrent events. In this example, a randomized trial compares intervention with control to understand how outcomes differ at month 2. However, one participant stops treatment before month 2 (ie, an intercurrent event). The figure shows what happens to this participant under each intercurrent event strategy. Under a composite strategy, investigators have decided to assign a score of 0 to any participant who experienced an intercurrent event. Under a while-on-treatment strategy, because the participant experienced an intercurrent event before month 2, their month 1 score of 3 is used in place of their month 2 score. Under a hypothetical strategy, the participant’s outcome that would have occurred had they continued treatment at month 2 is used (here, it is a value of 9); but in practice, this value will not be known and so must be estimated. M=month.

Kahan B C, Hindley J, Edwards M, Cro S, Morris T P. The estimands framework: a primer on the ICH E9(R1) addendum BMJ 2024; 384 :e076316 doi:10.1136/bmj-2023-076316

Back to Glossary

• Contra Elad Gil, it isn’t surprising that biomarkers are still in such a primitive state — they are a complex construct and the knowledge to develop a valid one is distributed amongst people who rarely read each other’s work.

Subscribe now

• Ian Hacking’s ‘seeing through a microscope’ is a classic and worth re-reading to consider whether it will apply to effectively ‘seeing with AI’,

• Is it possible that the AI epistemic issues also apply to areas of biomedical research prone to stagnation?

• On the dangers of quantifauxcation, a word I have loved since I heard Philip Stark use it.

• Rare genetic variants are hard to find but we eventually figured out how to find a signal that is weak when averaged over a population and high effect for a few people. When will be understand learn to measure transformative introspective experiences rigorously and understand them?

The original use-case for assurance was to optimize sample size choices in clinical trials!

Assurance is the unconditional probability that the trial will yield a ‘positive outcome’. A positive outcome usually means a statistically significant result, according to some standard frequentist significance test. The assurance is then the prior expectation of the power, averaged over the prior distribution for the unknown true treatment effect.

We argue that assurance is an important measure of the practical utility of a proposed trial, and indeed that it will often be appropriate to choose the size of the sample (and perhaps other aspects of the design) to achieve a desired assurance, rather than to achieve a desired power conditional on an assumed treatment effect.

O'Hagan, A., Stevens, J.W. and Campbell, M.J. (2005), Assurance in clinical trial design. Pharmaceut. Statist., 4: 187-201. https://doi.org/10.1002/pst.175

Most major pharmaceutical companies like Novartis, GSK, and Roche all have teams who design methodology for assurance, sometimes even for clinical pharmacology and dosing decisions, for trials based on surrogate endpoints. The quality of a biomarker endpoint needs to be substantial to guarantee assurance.

Conditional assurance is a further extension of the original concept.

But what about analogous probabilistic forecasting for earlier stages of drug development? Scannell et. al (2022) point out that the lack of predictive validity of high throughput screening tools and translational models are in-part responsible for the inefficiency of biopharma R&D. One might think that simply correlating the outcomes of a translational model with binary outcomes like drug approvals/failures would provide a good assessment of the predictive validity of translational models. But this is mistaken for a few conceptual reasons

• Clinical trial failures have a file-drawer problem, thus we don’t have an unbiased estimate of even historical successes/failures of some category of molecules-indication combinations. Forget forecasting, we don’t have an unbiased prediction error of historical drug development. One major but not only component of the file-drawer here is right censoring.

• Right censoring problem: The sequential nature of drug development implies a sequential drop off in the number of drugs tested from phase 1 → phase 2 → phase 3 trials. Thus we we don’t get to assess the future clinical validity of all assays of therapeutic potency, toxicity, DMPK, disease models, putative surrogate endpoints and so on without severe omission biases.

• Underspecification of success: Defining what constitutes success is a huge outcome and metric-hacking problem. There is a reason so much time and resources are dedicated to choosing endpoints in clinical trials, so that evaluations cannot be gamed. Similarly, choosing what constitutes success for pharmaceutical forecasting at every stage of drug development is subject to external pressures.

One solution to resolving the omission bias problem is to move away from using binary measures of success such as drug approval/failure to using actual individual participant data from clinical trials which has much richer sources of variation to assess the clinical validity of in-vitro/in-silico/in-vivo models and assays. This was the basis of the program idea I developed last year during my time as a BRAINS Fellow with SpecTech, also on featured on ’s Gap Map database. But there are even more important reasons to use individual patient-level data to evaluate translational forecasts — to reduce the risk of surrogate paradoxes and other clinical trial failures that amount to getting the therapeutic benefit-adverse effect tradeoff wrong.

In principle, a translational analogs of clinical assurance for every tool and model used for drug development would be the principled approach to keeping track of their effectiveness. AI in drug development is not merely about evaluating therapeutic candidates during hit and lead optimization, but also useful to severely test the translational and clinical validity of drug development tools — all in-vitro, in-silico, in-vivo tools individually and jointly for their capacity to generate clinically valid forecasts. Even without raw preclinical and clinical trial datasets, however, there are a multitude of clever approaches to mitigating the biases from the sequential nature of clinical development.

• A classic paper in survival analysis from 2011 developed “virtual twins” for analyzing clinical trials via the potential outcomes. One school of thought thinks of causal inference as a missing data problem. The fundamental problem in causal inference, especially in therapeutic experiments, is that you can only evaluate one of 2 possible outcomes for an individual. Imputing one of the possible outcomes amounts to simulating or imputing counterfactuals. The important question is when can we get away with creating data without actually measuring it? Any virtual twin method should lay out exactly when such an operation still results in valid conclusions.

• Science progresses when we re-visit and overturn bad assumptions. The paper that launched the re-evaluation of motion artifacts in fMRI comes to mind. For all the criticism that fMRI gets about its limitations, the fMRI imaging community has always been a progressive, rather than degenerating research program in the Lakatosian sense.

  Share Neurostats

• Another example of science progressing by improving the measurement validity of brain clearance measurements. It is unclear to me who is right on this issue as both the statistical issues and measurement issues of a construct like clearance are legitimate. Not at all definitive that sleep clears plaques.

  

• The statistical science of transformations meets AI

Share

1. Links
2. Quick Takes

I’m also starting a private AI study group to learn about new AI papers in many different fields. Starting with a focus on “Evals”. I’m particularly interested in the gap between how tech thinks about evaluations in contrast to different parts of biotech/medicine. Send me an email with the subject “Manjari’s Secret Study Group”

Links

• Improving evaluations in AI for chemistry / drug discovery

• o1-preview finds a very nuanced problem with non-linear mendelian randomization in a now retracted paper on vitamin-D interventions from Lancet Endocrinology

• take on systemic gaps in innovation on the Good Science Project.

• My favorite articulation and critical evaluation of the Free Energy Principle (FEP) and predictive processing theories of the brain is from Dan Williams (@conspicuouscognition) . He gets to the bottom of what has always bugged me about FEP since I first read it as an EECS graduate student stumbling into cognitive neuroscience. He has a few more papers on this worth reading.

  
  

  Subscribe now

Quick Takes

• In which I strongly disagree with Ben Recht about statistical thinking at the FDA. The gatekeeping of new drugs via statistical hypothesis testing is not the primary or even most important use of statistical thinking at the FDA. It really isn’t about squeezing out the most rigorous p-value at the end, one can be pragmatic about that. It is everything else that comes before that.
  

• Anyone who proposes to replace a clinical endpoint with a biomarker or intermediate endpoint that hasn’t itself been tested in successful clinical trials doesn’t know what they are doing. Consider, the 20-year saga of the failed preterm birth prevention drug, 17-OHPC, that has finally been withdrawn. Yet another reason why we need the proper attitude to biomarkers should be “Let’s have more relevant clinical endpoints and more qualified biomarkers”.

  • I have been reading the clinical papers, FDA reviews, company submissions from 1999. A deeper dive into 25+ years of scientific drama is forthcoming.

• This is how we end up with underpowered clinical trials

Subscribe now

In 1952, Iowa State College and The Biometrics Society, ENAR (Eastern North American Region) sponsored a Biostatistics conference. I suspect this might have been the precursor to the annual ENAR conference. The work presented at this conference was later published as a series of book chapters in “Statistics and Mathematics in Biology” edited by Oscar Kempthorne, Theodore Bancroft, John Gowen and Jay Lush. One of the special sessions at this conference was titled “Correlation and Causation as Biometric Concepts” where both Sewall Wright and John Tukey were invited speakers. I was once reading David Freedman’s “A Case Study in Path Analysis” and he referenced results from a 1952 paper by Tukey. It was so unprecedented, that I just had to get the conference proceedings and get a copy of the paper.

In 2018, I tweeted out some excerpts from Tukey’s paper

John Tukey on Causation, Regression & Path Analysis in 1952.
He speaks favorably of Sewall Wright's path analysis and how structural regressions are different from predictive regressions. pic.twitter.com/pOvfMRjxF0

— Manjari Narayan (@NeuroStats) December 8, 2018

and Judea Pearl responded

Thanks Manjari for posting this paper. We should have mentioned Tukey in #Bookofwhy as a singularity (among statisticians) who stated publically that causation is NOT a species of correlation and that statistics can learn something from an outsider. Any link to the whole paper?

— Judea Pearl (@yudapearl) December 9, 2018

Here is a copy of Wright’s The Interpretation of Multivariate Systems and Tukey’s Causation, Regression and Path Analysis (excerpt below).

Causation, Regression & Path Analysis
John Tukey

When I began to prepare this paper a few weeks ago, I realized that the session title, "Correlation and Causation as Biometric Concepts," did not bring a very clear picture to my mind. So I got in touch with Professor Wright and learned roughly what he was going to discuss. It was no surprise that this was mainly path coefficient analysis, for I had heard of path coefficients repeatedly. The surprise came when I found that I did not know anything about path analysis, although after some study it seemed to be natural and useful.

After coming to the point where I thought I understood it moderately well, it occurred to me to wonder why I had not known about it before. The obvious answer was that it had not been used in the problems that I had been concerned with or discussed in the literature that I was used to reading. Was this because it was not useful? The evidence was strongly to the contrary. On further thought I came to the conclusion that the main reason for my ignorance was a tendency of many to confuse correlation and causation as biometric concepts.

What I shall try to do, therefore, is fourfold: (I) to set forth an opinion as to correlation, regression, and causation as methodological (and especially biometric) concepts and to relate path analysis to this point of view; (II) to discuss briefly the application of path analysis to inbreeding, pointing out the situations under which it may fail or require special attention; (III) to discuss briefly some of the applications of path analysis to observed data; and (IV) to discuss very briefly some topics which seem related to path analysis.

References

ENAR 1952 Announcement published in The American Statistician

*A previous version of this post originally appeared on my personal website.

TOC

• Notes

• Quick Takes

Notes

• https://notes.manjarinarayan.org/daily-notes
  Notes on counterfactual probabilities, inclusive of some recent work.

• https://notes.manjarinarayan.org/daily-notes/…

• https://notes.manjarinarayan.org/daily-notes/…

• https://notes.manjarinarayan.org/daily-notes/…

  

• https://notes.manjarinarayan.org/…

Quick Takes

• Artificial Intelligence, Scientific Discovery and Product Innovation

• An explanation algebra for counterfactual attributions!

• Trial results for the other muscarinic receptor drug, emraclidine are in

• Literature review of clozapine mechanisms of action

• Addiction as a brain disease revisited

• Calls for empirical ML in the same vein as calls in the mid 20th century for statistics as a science (instead of mathematics)

  Thanks for reading NeuroStats! Subscribe for free to receive new posts and support my work.

It is interesting that they call this perspective “empirical ML”. I rather think of their position as making ML/AI a proper branch of science instead of a branch of theoretical CS or as a branch of software engineering. The issue is quite analogous to how some practice statistics as a branch of mathematical science whereas many those of the British statistics tradition (Fisher, Box, etc..) practice statistics as a branch of science.

We’ve settled to be second rate mathematicians when we could be first-rate scientists — George Box.

I rather see the current crop of ML papers as too empirical (similar to what I see in many other sciences) without any formal methodology or normative principles about how to draw conclusions correctly.

We believe that one of the main reasons for this is that ML stands, like few other disciplines, at the interface between formal sciences and real-world applications. Because ML has strong foundations in formal sciences such as mathematics, (theoretical) computer science (CS), and mathematical statistics, many ML researchers are accustomed to reasoning mathematically about abstract objects – ML methods – using formal proofs. On the other hand, ML can also very much be considered a (software) engineering science, to create practical systems that can learn and improve their performance by interacting with their environment. Lastly, and especially concerning experimentation in ML, there exists an applied statistical perspective with a focus on thorough inductive reasoning. With its tradition in data analysis and design of experiments, it emphasizes the empirical aspects of ML research.

A statistical perspective, which we adopt here, is very sensitive to such empirical issues – explaining/analyzing real-world phenomena and generalizing beyond a specific context (inductive reasoning) – and thus particularly suited to explain 1) why ML is faced with non-replicable research, and 2) how a more complete and nuanced understanding of empirical research in ML can help to overcome this situation. With empirical ML we thus mean in a broad sense the systematic investigation of ML algorithms, techniques, and conceptual questions through simulations, experimentation, and observation. It deals with real objects: implementations of algorithms – which are usually more complex than their theoretical counterparts (e.g., Kriegel et al., 2017) – running on physical computers; data gathered and produced/simulated in the real world; and their interplay. Rather than focusing solely on theoretical analysis and proofs, empirical research emphasizes practical evaluations using real-world and/or synthetic data. Empirical ML, as understood here, requires a mindset very different from engineering and formal sciences and a different approach to methodology to allow for the full incorporation of the uncertainties inherent in dealing with real-world entities in experiments.

https://epub.ub.uni-muenchen.de/121738/1/herrmann24b__1_.pdf

We could nearly replace “statistics” with “ML” in Box’s Science and Statistics lecture would apply quite well.

[This is a repost of a note, before I realized how Substack Notes work]

Important work by @aweisstweets.

I am a fan of the power of causal inference, assuming it is done to the highest theoretically warranted & empirical standards. But this is a good reminder that papers that lack the capacity to infer the target causal effect, despite seeming to use the popular CI machinery, do not actually provide evidence of a causal effect.

My pet peeve in the last 4 years of bio/neuro science papers is that people continue to do the same old stuff they did before (where causal effects were not identifiable) but now just add "causal" or apply newfangled causal discovery algorithms or causal effect estimation algorithm that only work under unjustifiable assumptions.