First unlock from vibe science: public datasets

🔓 🔥 First big unlock from vibe science-ing: rapid, actionable access to publicly available datasets. Sounds basic. It isn't.

💽 If you've ever tried to pull raw data from a paper you care about, you know the reality: the metadata is a mess, supplementary tables are unstructured, file formats don't match, and by the time you've got it working you've lost half a day. For people without strong bioinformatics skills, it's often a dead end entirely.

I've always relied on public data to de-risk projects (does the signal even exist in a related dataset?) and to independently validate my own findings. In one of my past studies, I built whole figures based on 3 independent, published datasets to support observations on aging in microglia. Incredibly helpful. Still regularly took me days (at best) to retrieve, prepare and analyze it all.

⚡ That kind of activation energy has basically collapsed. You point an agent at the papers, tell it what you need, and it handles retrieval, decompression, and format wrangling largely on its own. This is pretty magical. 🤖

🧬🔍 Where it gets really powerful is combining datasets at scale. I recently aggregated Perturb-seq datasets from 5 independent sources into a single queryable collection: 107,219 perturbations, 20,929 unique gene targets, across studies from the Marson, Bhatt, and Weissman labs among others. Not to train a model. Just to ask: does knocking out gene X do something consistent across independent experiments, and how strong is the effect? You don't always need a new foundation model. Sometimes you just need a coherent map of what's already been measured. No more squinting at volcano plots in figure 5C

🧫 Same approach works across data types. For a hackathon project, I assembled about 90,000 T cell receptor sequences from 8 independent sources in a day. Once unified, 3,889 clonotypes appeared across multiple independent datasets. One CDR3 sequence from an Alzheimer's CSF dataset matched an EBV-reactive clone in an independent database. This was a rediscovery of a finding from Gate et al. 2020 that TCRs in the brain of Alzheimer's patients are clonal and appear to recognize EBV. The pipeline surfaced it without being asked. It just fell out of the aggregation.

📎 Most of this data was generated with public funding. It should be accessible. Often it isn't. It's buried in supplementary tables, poorly annotated, deposited in formats nobody maintains. These tools don't fix that incentive problem, but they dramatically lower the barrier to using what's already out there. And the easier it gets to access someone's data, the more important it becomes to properly cite and credit their work

If you're building AI science tools: this is not where you save on tokens - include those datasets & meta-analyzes in your scientific reasoning workflows. 🪙🪙🪙