Posts by Tags

DataViz

RNA-seq

alignment

bbduk

bbmap

bbmerge

bbtools

bioinformatics

biomarkers

Don’t Cut Your Continuous Biomarker in Half

7 minute read

Published:

A median split takes a variable that distinguishes a thousand shades of signal and reduces it to a coin that only knows heads from tails. We would never dilute a reagent tenfold to save pipetting. We do the analytic equivalent constantly, and we call it interpretability.

calibration

A Worked Example: 286 Tumours, 22,000 Genes, 69 Events, and an Honest Answer

10 minute read

Published:

On this dataset the popular pipeline reports an AUC of 0.89. The honest number is 0.59, and the model’s probabilities are so overfit that it predicts worse than saying “24%” to everyone. A penalised model on all the genes reaches 0.71 and is calibrated. And the bootstrap I recommended in the last post got the wrong answer.

You Probably Shouldn’t Balance Your Classes

6 minute read

Published:

Class imbalance is not a disease of your data. It is a fact about the world: the outcome you care about is rare. “Correcting” it by throwing away controls or synthesizing cases does not make your model better. It makes your probabilities wrong, and then you have to do more work to make them wrong in the opposite direction.

Accuracy Is a Bad Way to Judge a Predictor

6 minute read

Published:

If your accuracy score can be improved by a model that is calibrated worse, the score is broken. Proportion classified correctly is exactly such a score. It is not a minor imperfection to note in the discussion section. It is a metric that will, given the chance, hand you the wrong model and a straight face.

class-imbalance

You Probably Shouldn’t Balance Your Classes

6 minute read

Published:

Class imbalance is not a disease of your data. It is a fact about the world: the outcome you care about is rare. “Correcting” it by throwing away controls or synthesizing cases does not make your model better. It makes your probabilities wrong, and then you have to do more work to make them wrong in the opposite direction.

classification

Your Classifier Is Answering the Wrong Question

7 minute read

Published:

A classifier forces a choice. A probability model quantifies a tendency and hands the choice back to whoever is actually equipped to make it. In most of the problems I work on, the second thing is what I want, and the first thing is what I built by accident.

clumpify

data visualization

dichotomization

Don’t Cut Your Continuous Biomarker in Half

7 minute read

Published:

A median split takes a variable that distinguishes a thousand shades of signal and reduces it to a coin that only knows heads from tails. We would never dilute a reagent tenfold to save pipetting. We do the analytic equivalent constantly, and we call it interpretability.

dotplot

feature-selection

Your Gene Signature Probably Won’t Replicate

7 minute read

Published:

Take the same data, resample the patients, and rerun your selection. If the gene list comes back different every time, and it will, then the specific genes were never the finding. The finding was that many genes carry a little signal and your procedure picked some of them.

fun

genomics

A Worked Example: 286 Tumours, 22,000 Genes, 69 Events, and an Honest Answer

10 minute read

Published:

On this dataset the popular pipeline reports an AUC of 0.89. The honest number is 0.59, and the model’s probabilities are so overfit that it predicts worse than saying “24%” to everyone. A penalised model on all the genes reaches 0.71 and is calibrated. And the bootstrap I recommended in the last post got the wrong answer.

One Test Set Is Not Validation

7 minute read

Published:

A held-out test set answers one question: how did this model do on these particular held-out patients? You wanted a different answer, how it will do on the next patients, and a single split estimates that with a sample size of one experiment, run on the data you could least afford to set aside.

Count Your Events Before You Count Your Variables

7 minute read

Published:

A model does not learn from your samples. It learns from your events. If your rare outcome occurred forty times and you are considering twenty thousand genes, you are asking forty facts to adjudicate twenty thousand questions, and most of what comes back is the sound of the data overfitting.

Everything Correlates With Survival

5 minute read

Published:

The best gene I could find in this cohort has p = 0.001. After I shuffled the patients so that no gene could possibly matter, the best gene had p = 0.0003.

Your Gene Signature Probably Won’t Replicate

7 minute read

Published:

Take the same data, resample the patients, and rerun your selection. If the gene list comes back different every time, and it will, then the specific genes were never the finding. The finding was that many genes carry a little signal and your procedure picked some of them.

Don’t Cut Your Continuous Biomarker in Half

7 minute read

Published:

A median split takes a variable that distinguishes a thousand shades of signal and reduces it to a coin that only knows heads from tails. We would never dilute a reagent tenfold to save pipetting. We do the analytic equivalent constantly, and we call it interpretability.

You Probably Shouldn’t Balance Your Classes

6 minute read

Published:

Class imbalance is not a disease of your data. It is a fact about the world: the outcome you care about is rare. “Correcting” it by throwing away controls or synthesizing cases does not make your model better. It makes your probabilities wrong, and then you have to do more work to make them wrong in the opposite direction.

Accuracy Is a Bad Way to Judge a Predictor

6 minute read

Published:

If your accuracy score can be improved by a model that is calibrated worse, the score is broken. Proportion classified correctly is exactly such a score. It is not a minor imperfection to note in the discussion section. It is a metric that will, given the chance, hand you the wrong model and a straight face.

Your Classifier Is Answering the Wrong Question

7 minute read

Published:

A classifier forces a choice. A probability model quantifies a tendency and hands the choice back to whoever is actually equipped to make it. In most of the problems I work on, the second thing is what I want, and the first thing is what I built by accident.

ggplot2

interactive

Everything Correlates With Survival

5 minute read

Published:

The best gene I could find in this cohort has p = 0.001. After I shuffled the patients so that no gene could possibly matter, the best gene had p = 0.0003.

machine-learning

A Worked Example: 286 Tumours, 22,000 Genes, 69 Events, and an Honest Answer

10 minute read

Published:

On this dataset the popular pipeline reports an AUC of 0.89. The honest number is 0.59, and the model’s probabilities are so overfit that it predicts worse than saying “24%” to everyone. A penalised model on all the genes reaches 0.71 and is calibrated. And the bootstrap I recommended in the last post got the wrong answer.

One Test Set Is Not Validation

7 minute read

Published:

A held-out test set answers one question: how did this model do on these particular held-out patients? You wanted a different answer, how it will do on the next patients, and a single split estimates that with a sample size of one experiment, run on the data you could least afford to set aside.

Count Your Events Before You Count Your Variables

7 minute read

Published:

A model does not learn from your samples. It learns from your events. If your rare outcome occurred forty times and you are considering twenty thousand genes, you are asking forty facts to adjudicate twenty thousand questions, and most of what comes back is the sound of the data overfitting.

Your Gene Signature Probably Won’t Replicate

7 minute read

Published:

Take the same data, resample the patients, and rerun your selection. If the gene list comes back different every time, and it will, then the specific genes were never the finding. The finding was that many genes carry a little signal and your procedure picked some of them.

You Probably Shouldn’t Balance Your Classes

6 minute read

Published:

Class imbalance is not a disease of your data. It is a fact about the world: the outcome you care about is rare. “Correcting” it by throwing away controls or synthesizing cases does not make your model better. It makes your probabilities wrong, and then you have to do more work to make them wrong in the opposite direction.

Accuracy Is a Bad Way to Judge a Predictor

6 minute read

Published:

If your accuracy score can be improved by a model that is calibrated worse, the score is broken. Proportion classified correctly is exactly such a score. It is not a minor imperfection to note in the discussion section. It is a metric that will, given the chance, hand you the wrong model and a straight face.

Your Classifier Is Answering the Wrong Question

7 minute read

Published:

A classifier forces a choice. A probability model quantifies a tendency and hands the choice back to whoever is actually equipped to make it. In most of the problems I work on, the second thing is what I want, and the first thing is what I built by accident.

model-evaluation

One Test Set Is Not Validation

7 minute read

Published:

A held-out test set answers one question: how did this model do on these particular held-out patients? You wanted a different answer, how it will do on the next patients, and a single split estimates that with a sample size of one experiment, run on the data you could least afford to set aside.

Don’t Cut Your Continuous Biomarker in Half

7 minute read

Published:

A median split takes a variable that distinguishes a thousand shades of signal and reduces it to a coin that only knows heads from tails. We would never dilute a reagent tenfold to save pipetting. We do the analytic equivalent constantly, and we call it interpretability.

Accuracy Is a Bad Way to Judge a Predictor

6 minute read

Published:

If your accuracy score can be improved by a model that is calibrated worse, the score is broken. Proportion classified correctly is exactly such a score. It is not a minor imperfection to note in the discussion section. It is a metric that will, given the chance, hand you the wrong model and a straight face.

multiple-testing

Everything Correlates With Survival

5 minute read

Published:

The best gene I could find in this cohort has p = 0.001. After I shuffled the patients so that no gene could possibly matter, the best gene had p = 0.0003.

nextflow

overfitting

Count Your Events Before You Count Your Variables

7 minute read

Published:

A model does not learn from your samples. It learns from your events. If your rare outcome occurred forty times and you are considering twenty thousand genes, you are asking forty facts to adjudicate twenty thousand questions, and most of what comes back is the sound of the data overfitting.

paired-end

prediction

Your Classifier Is Answering the Wrong Question

7 minute read

Published:

A classifier forces a choice. A probability model quantifies a tendency and hands the choice back to whoever is actually equipped to make it. In most of the problems I work on, the second thing is what I want, and the first thing is what I built by accident.

reformat

reproducibility

Your Gene Signature Probably Won’t Replicate

7 minute read

Published:

Take the same data, resample the patients, and rerun your selection. If the gene list comes back different every time, and it will, then the specific genes were never the finding. The finding was that many genes carry a little signal and your procedure picked some of them.

sample-size

Count Your Events Before You Count Your Variables

7 minute read

Published:

A model does not learn from your samples. It learns from your events. If your rare outcome occurred forty times and you are considering twenty thousand genes, you are asking forty facts to adjudicate twenty thousand questions, and most of what comes back is the sound of the data overfitting.

statistics

A Worked Example: 286 Tumours, 22,000 Genes, 69 Events, and an Honest Answer

10 minute read

Published:

On this dataset the popular pipeline reports an AUC of 0.89. The honest number is 0.59, and the model’s probabilities are so overfit that it predicts worse than saying “24%” to everyone. A penalised model on all the genes reaches 0.71 and is calibrated. And the bootstrap I recommended in the last post got the wrong answer.

One Test Set Is Not Validation

7 minute read

Published:

A held-out test set answers one question: how did this model do on these particular held-out patients? You wanted a different answer, how it will do on the next patients, and a single split estimates that with a sample size of one experiment, run on the data you could least afford to set aside.

Count Your Events Before You Count Your Variables

7 minute read

Published:

A model does not learn from your samples. It learns from your events. If your rare outcome occurred forty times and you are considering twenty thousand genes, you are asking forty facts to adjudicate twenty thousand questions, and most of what comes back is the sound of the data overfitting.

Everything Correlates With Survival

5 minute read

Published:

The best gene I could find in this cohort has p = 0.001. After I shuffled the patients so that no gene could possibly matter, the best gene had p = 0.0003.

Your Gene Signature Probably Won’t Replicate

7 minute read

Published:

Take the same data, resample the patients, and rerun your selection. If the gene list comes back different every time, and it will, then the specific genes were never the finding. The finding was that many genes carry a little signal and your procedure picked some of them.

Don’t Cut Your Continuous Biomarker in Half

7 minute read

Published:

A median split takes a variable that distinguishes a thousand shades of signal and reduces it to a coin that only knows heads from tails. We would never dilute a reagent tenfold to save pipetting. We do the analytic equivalent constantly, and we call it interpretability.

You Probably Shouldn’t Balance Your Classes

6 minute read

Published:

Class imbalance is not a disease of your data. It is a fact about the world: the outcome you care about is rare. “Correcting” it by throwing away controls or synthesizing cases does not make your model better. It makes your probabilities wrong, and then you have to do more work to make them wrong in the opposite direction.

Accuracy Is a Bad Way to Judge a Predictor

6 minute read

Published:

If your accuracy score can be improved by a model that is calibrated worse, the score is broken. Proportion classified correctly is exactly such a score. It is not a minor imperfection to note in the discussion section. It is a metric that will, given the chance, hand you the wrong model and a straight face.

Your Classifier Is Answering the Wrong Question

7 minute read

Published:

A classifier forces a choice. A probability model quantifies a tendency and hands the choice back to whoever is actually equipped to make it. In most of the problems I work on, the second thing is what I want, and the first thing is what I built by accident.

survival

Everything Correlates With Survival

5 minute read

Published:

The best gene I could find in this cohort has p = 0.001. After I shuffled the patients so that no gene could possibly matter, the best gene had p = 0.0003.

tutorial

A Worked Example: 286 Tumours, 22,000 Genes, 69 Events, and an Honest Answer

10 minute read

Published:

On this dataset the popular pipeline reports an AUC of 0.89. The honest number is 0.59, and the model’s probabilities are so overfit that it predicts worse than saying “24%” to everyone. A penalised model on all the genes reaches 0.71 and is calibrated. And the bootstrap I recommended in the last post got the wrong answer.

validation

A Worked Example: 286 Tumours, 22,000 Genes, 69 Events, and an Honest Answer

10 minute read

Published:

On this dataset the popular pipeline reports an AUC of 0.89. The honest number is 0.59, and the model’s probabilities are so overfit that it predicts worse than saying “24%” to everyone. A penalised model on all the genes reaches 0.71 and is calibrated. And the bootstrap I recommended in the last post got the wrong answer.

One Test Set Is Not Validation

7 minute read

Published:

A held-out test set answers one question: how did this model do on these particular held-out patients? You wanted a different answer, how it will do on the next patients, and a single split estimates that with a sample size of one experiment, run on the data you could least afford to set aside.