Beyond Bar Charts: What I wish I knew as a benchtop biologist.
Published:
Published:
Published:
This is part of a series on the BBTools suite. If you haven’t read the overview post, that’s a good place to start.
Published:
This is part of a series on the BBTools suite. The overview introduces the tools and the BBDuk post builds the demo data used here.
Published:
This is part of a series on the BBTools suite. If you haven’t read the overview post, that’s a good place to start.
Published:
This is part of a series on the BBTools suite. The overview introduces the tools and the BBDuk post builds the demo data used here.
Published:
This is part of a series on the BBTools suite. The demo data come from the BBDuk post, and the BBMap post removed the phiX spike-in.
Published:
This closes the run of BBTools posts. The BBDuk, BBMap, BBMerge and Reformat/Clumpify posts each ran one tool by hand on the same simulated library. A pipeline is what you write once you have done that by hand enough times to know what the steps are, and Nextflow is the tool I use for it. Everything below was run, and the three files are downloadable at the end.
Published:
This is part of a series on the BBTools suite; the demo data are built in the BBDuk post and the BAM comes from the BBMap post.
Published:
This is part of a series on the BBTools suite. The demo data come from the BBDuk post, and the BBMap post removed the phiX spike-in.
Published:
This is part of a series on the BBTools suite. The overview introduces the tools and the BBDuk post builds the demo data used here.
Published:
This is part of a series on the BBTools suite. If you haven’t read the overview post, that’s a good place to start.
Published:
This closes the run of BBTools posts. The BBDuk, BBMap, BBMerge and Reformat/Clumpify posts each ran one tool by hand on the same simulated library. A pipeline is what you write once you have done that by hand enough times to know what the steps are, and Nextflow is the tool I use for it. Everything below was run, and the three files are downloadable at the end.
Published:
This is part of a series on the BBTools suite; the demo data are built in the BBDuk post and the BAM comes from the BBMap post.
Published:
This is part of a series on the BBTools suite. The demo data come from the BBDuk post, and the BBMap post removed the phiX spike-in.
Published:
This is part of a series on the BBTools suite. The overview introduces the tools and the BBDuk post builds the demo data used here.
Published:
This is part of a series on the BBTools suite. If you haven’t read the overview post, that’s a good place to start.
Published:
A median split takes a variable that distinguishes a thousand shades of signal and reduces it to a coin that only knows heads from tails. We would never dilute a reagent tenfold to save pipetting. We do the analytic equivalent constantly, and we call it interpretability.
Published:
On this dataset the popular pipeline reports an AUC of 0.89. The honest number is 0.59, and the model’s probabilities are so overfit that it predicts worse than saying “24%” to everyone. A penalised model on all the genes reaches 0.71 and is calibrated. And the bootstrap I recommended in the last post got the wrong answer.
Published:
Class imbalance is not a disease of your data. It is a fact about the world: the outcome you care about is rare. “Correcting” it by throwing away controls or synthesizing cases does not make your model better. It makes your probabilities wrong, and then you have to do more work to make them wrong in the opposite direction.
Published:
If your accuracy score can be improved by a model that is calibrated worse, the score is broken. Proportion classified correctly is exactly such a score. It is not a minor imperfection to note in the discussion section. It is a metric that will, given the chance, hand you the wrong model and a straight face.
Published:
Class imbalance is not a disease of your data. It is a fact about the world: the outcome you care about is rare. “Correcting” it by throwing away controls or synthesizing cases does not make your model better. It makes your probabilities wrong, and then you have to do more work to make them wrong in the opposite direction.
Published:
A classifier forces a choice. A probability model quantifies a tendency and hands the choice back to whoever is actually equipped to make it. In most of the problems I work on, the second thing is what I want, and the first thing is what I built by accident.
Published:
This is part of a series on the BBTools suite; the demo data are built in the BBDuk post and the BAM comes from the BBMap post.
Published:
Published:
A median split takes a variable that distinguishes a thousand shades of signal and reduces it to a coin that only knows heads from tails. We would never dilute a reagent tenfold to save pipetting. We do the analytic equivalent constantly, and we call it interpretability.
Published:
Published:
Take the same data, resample the patients, and rerun your selection. If the gene list comes back different every time, and it will, then the specific genes were never the finding. The finding was that many genes carry a little signal and your procedure picked some of them.
Published:
Published:
On this dataset the popular pipeline reports an AUC of 0.89. The honest number is 0.59, and the model’s probabilities are so overfit that it predicts worse than saying “24%” to everyone. A penalised model on all the genes reaches 0.71 and is calibrated. And the bootstrap I recommended in the last post got the wrong answer.
Published:
A held-out test set answers one question: how did this model do on these particular held-out patients? You wanted a different answer, how it will do on the next patients, and a single split estimates that with a sample size of one experiment, run on the data you could least afford to set aside.
Published:
A model does not learn from your samples. It learns from your events. If your rare outcome occurred forty times and you are considering twenty thousand genes, you are asking forty facts to adjudicate twenty thousand questions, and most of what comes back is the sound of the data overfitting.
Published:
The best gene I could find in this cohort has p = 0.001. After I shuffled the patients so that no gene could possibly matter, the best gene had p = 0.0003.
Published:
Take the same data, resample the patients, and rerun your selection. If the gene list comes back different every time, and it will, then the specific genes were never the finding. The finding was that many genes carry a little signal and your procedure picked some of them.
Published:
A median split takes a variable that distinguishes a thousand shades of signal and reduces it to a coin that only knows heads from tails. We would never dilute a reagent tenfold to save pipetting. We do the analytic equivalent constantly, and we call it interpretability.
Published:
Class imbalance is not a disease of your data. It is a fact about the world: the outcome you care about is rare. “Correcting” it by throwing away controls or synthesizing cases does not make your model better. It makes your probabilities wrong, and then you have to do more work to make them wrong in the opposite direction.
Published:
If your accuracy score can be improved by a model that is calibrated worse, the score is broken. Proportion classified correctly is exactly such a score. It is not a minor imperfection to note in the discussion section. It is a metric that will, given the chance, hand you the wrong model and a straight face.
Published:
A classifier forces a choice. A probability model quantifies a tendency and hands the choice back to whoever is actually equipped to make it. In most of the problems I work on, the second thing is what I want, and the first thing is what I built by accident.
Published:
Published:
Published:
The best gene I could find in this cohort has p = 0.001. After I shuffled the patients so that no gene could possibly matter, the best gene had p = 0.0003.
Published:
On this dataset the popular pipeline reports an AUC of 0.89. The honest number is 0.59, and the model’s probabilities are so overfit that it predicts worse than saying “24%” to everyone. A penalised model on all the genes reaches 0.71 and is calibrated. And the bootstrap I recommended in the last post got the wrong answer.
Published:
A held-out test set answers one question: how did this model do on these particular held-out patients? You wanted a different answer, how it will do on the next patients, and a single split estimates that with a sample size of one experiment, run on the data you could least afford to set aside.
Published:
A model does not learn from your samples. It learns from your events. If your rare outcome occurred forty times and you are considering twenty thousand genes, you are asking forty facts to adjudicate twenty thousand questions, and most of what comes back is the sound of the data overfitting.
Published:
Take the same data, resample the patients, and rerun your selection. If the gene list comes back different every time, and it will, then the specific genes were never the finding. The finding was that many genes carry a little signal and your procedure picked some of them.
Published:
Class imbalance is not a disease of your data. It is a fact about the world: the outcome you care about is rare. “Correcting” it by throwing away controls or synthesizing cases does not make your model better. It makes your probabilities wrong, and then you have to do more work to make them wrong in the opposite direction.
Published:
If your accuracy score can be improved by a model that is calibrated worse, the score is broken. Proportion classified correctly is exactly such a score. It is not a minor imperfection to note in the discussion section. It is a metric that will, given the chance, hand you the wrong model and a straight face.
Published:
A classifier forces a choice. A probability model quantifies a tendency and hands the choice back to whoever is actually equipped to make it. In most of the problems I work on, the second thing is what I want, and the first thing is what I built by accident.
Published:
A held-out test set answers one question: how did this model do on these particular held-out patients? You wanted a different answer, how it will do on the next patients, and a single split estimates that with a sample size of one experiment, run on the data you could least afford to set aside.
Published:
A median split takes a variable that distinguishes a thousand shades of signal and reduces it to a coin that only knows heads from tails. We would never dilute a reagent tenfold to save pipetting. We do the analytic equivalent constantly, and we call it interpretability.
Published:
If your accuracy score can be improved by a model that is calibrated worse, the score is broken. Proportion classified correctly is exactly such a score. It is not a minor imperfection to note in the discussion section. It is a metric that will, given the chance, hand you the wrong model and a straight face.
Published:
The best gene I could find in this cohort has p = 0.001. After I shuffled the patients so that no gene could possibly matter, the best gene had p = 0.0003.
Published:
This closes the run of BBTools posts. The BBDuk, BBMap, BBMerge and Reformat/Clumpify posts each ran one tool by hand on the same simulated library. A pipeline is what you write once you have done that by hand enough times to know what the steps are, and Nextflow is the tool I use for it. Everything below was run, and the three files are downloadable at the end.
Published:
A model does not learn from your samples. It learns from your events. If your rare outcome occurred forty times and you are considering twenty thousand genes, you are asking forty facts to adjudicate twenty thousand questions, and most of what comes back is the sound of the data overfitting.
Published:
This is part of a series on the BBTools suite. The demo data come from the BBDuk post, and the BBMap post removed the phiX spike-in.
Published:
A classifier forces a choice. A probability model quantifies a tendency and hands the choice back to whoever is actually equipped to make it. In most of the problems I work on, the second thing is what I want, and the first thing is what I built by accident.
Published:
This is part of a series on the BBTools suite; the demo data are built in the BBDuk post and the BAM comes from the BBMap post.
Published:
Take the same data, resample the patients, and rerun your selection. If the gene list comes back different every time, and it will, then the specific genes were never the finding. The finding was that many genes carry a little signal and your procedure picked some of them.
Published:
This closes the run of BBTools posts. The BBDuk, BBMap, BBMerge and Reformat/Clumpify posts each ran one tool by hand on the same simulated library. A pipeline is what you write once you have done that by hand enough times to know what the steps are, and Nextflow is the tool I use for it. Everything below was run, and the three files are downloadable at the end.
Published:
A model does not learn from your samples. It learns from your events. If your rare outcome occurred forty times and you are considering twenty thousand genes, you are asking forty facts to adjudicate twenty thousand questions, and most of what comes back is the sound of the data overfitting.
Published:
On this dataset the popular pipeline reports an AUC of 0.89. The honest number is 0.59, and the model’s probabilities are so overfit that it predicts worse than saying “24%” to everyone. A penalised model on all the genes reaches 0.71 and is calibrated. And the bootstrap I recommended in the last post got the wrong answer.
Published:
A held-out test set answers one question: how did this model do on these particular held-out patients? You wanted a different answer, how it will do on the next patients, and a single split estimates that with a sample size of one experiment, run on the data you could least afford to set aside.
Published:
A model does not learn from your samples. It learns from your events. If your rare outcome occurred forty times and you are considering twenty thousand genes, you are asking forty facts to adjudicate twenty thousand questions, and most of what comes back is the sound of the data overfitting.
Published:
The best gene I could find in this cohort has p = 0.001. After I shuffled the patients so that no gene could possibly matter, the best gene had p = 0.0003.
Published:
Take the same data, resample the patients, and rerun your selection. If the gene list comes back different every time, and it will, then the specific genes were never the finding. The finding was that many genes carry a little signal and your procedure picked some of them.
Published:
A median split takes a variable that distinguishes a thousand shades of signal and reduces it to a coin that only knows heads from tails. We would never dilute a reagent tenfold to save pipetting. We do the analytic equivalent constantly, and we call it interpretability.
Published:
Class imbalance is not a disease of your data. It is a fact about the world: the outcome you care about is rare. “Correcting” it by throwing away controls or synthesizing cases does not make your model better. It makes your probabilities wrong, and then you have to do more work to make them wrong in the opposite direction.
Published:
If your accuracy score can be improved by a model that is calibrated worse, the score is broken. Proportion classified correctly is exactly such a score. It is not a minor imperfection to note in the discussion section. It is a metric that will, given the chance, hand you the wrong model and a straight face.
Published:
A classifier forces a choice. A probability model quantifies a tendency and hands the choice back to whoever is actually equipped to make it. In most of the problems I work on, the second thing is what I want, and the first thing is what I built by accident.
Published:
The best gene I could find in this cohort has p = 0.001. After I shuffled the patients so that no gene could possibly matter, the best gene had p = 0.0003.
Published:
On this dataset the popular pipeline reports an AUC of 0.89. The honest number is 0.59, and the model’s probabilities are so overfit that it predicts worse than saying “24%” to everyone. A penalised model on all the genes reaches 0.71 and is calibrated. And the bootstrap I recommended in the last post got the wrong answer.
Published:
This closes the run of BBTools posts. The BBDuk, BBMap, BBMerge and Reformat/Clumpify posts each ran one tool by hand on the same simulated library. A pipeline is what you write once you have done that by hand enough times to know what the steps are, and Nextflow is the tool I use for it. Everything below was run, and the three files are downloadable at the end.
Published:
This is part of a series on the BBTools suite; the demo data are built in the BBDuk post and the BAM comes from the BBMap post.
Published:
This is part of a series on the BBTools suite. The demo data come from the BBDuk post, and the BBMap post removed the phiX spike-in.
Published:
This is part of a series on the BBTools suite. The overview introduces the tools and the BBDuk post builds the demo data used here.
Published:
This is part of a series on the BBTools suite. If you haven’t read the overview post, that’s a good place to start.
Published:
On this dataset the popular pipeline reports an AUC of 0.89. The honest number is 0.59, and the model’s probabilities are so overfit that it predicts worse than saying “24%” to everyone. A penalised model on all the genes reaches 0.71 and is calibrated. And the bootstrap I recommended in the last post got the wrong answer.
Published:
A held-out test set answers one question: how did this model do on these particular held-out patients? You wanted a different answer, how it will do on the next patients, and a single split estimates that with a sample size of one experiment, run on the data you could least afford to set aside.