This is one of my favorite LLM studies of the year. Not only because it showed that GPT4 outperforms physicians on these summary tasks (which is cool to see that parity), but because it showed how often humans and LLMs still make mistakes in summarization (~5-10% frequency)!

In every group I speak to, from business executives to scientists, including a group of very accomplished people in Silicon Valley last night, much less than 20% of the crowd has even tried a GPT-4 class model.

Less than 5% has spent the required 10 hours to know how they tick.

Claude 3 gets ~60% accuracy on GPQA. It's hard for me to understate how hard these questions are—literal PhDs (in different domains from the questions) with access to the internet get 34%.

PhDs *in the same domain* (also with internet access!) get 65% - 75% accuracy.

This whole piece by computer scientist Scott Aronson is interesting.

But I think these sections are helpful because they captures the level of surprise I often see among experts about how well LLMs work in practice & the uncertainty over what comes next. scottaaronson.blog

Sora by OpenAI is insane.

But it doesn't just generate AI videos from text, it can also change the styles and environments of input videos🤯

12 wild examples:

First, Input video 🧵👇

The finding that 54% of medical students in this global survey viewed medical training as a stepping stone to a non-clinical career is WILD.

There’s nothing wrong with wanting to take that path, but wow - that’s way more than I would have anticipated.

Use of #AI synthetic notes >300,000 clinic conversations, >3,400 PermanenteDoctors catalyst.nejm.org/doi/full/10.10…
—81% Physicians-less screen time during visits and less 'pajama time' work on EHR
—71% Patients-more time spent speaking w/ physicians
—Audit of random 35 notes-high quality

Halle Tecco MBA, MPH Sergei Polevikov Just this year through VC backed digital health:

1. Saved $300 on a prescription with GoodRx
2. Was FINALLY properly diagnosed with early menopause and experienced life altering symptom improvement on low dose estrogen through Midi
3. Using DTC lab ordering and a lipidoligist I…

#AI 's black box dissuades many in high-stakes (medical) applications. With purposeful prompting methods, large language models can mimic clinical reasoning processes on their way to delivering answers. Tom Savage, Ashwin Nayak, Rob Gallo, Ekanath Rangan
nature.com/articles/s4174…

Data from the Oscar Medical Group's AI-written messaging summaries: the time it takes to document visits to virtual care providers via secure messages dropped by around 30% with automated summaries. The chart below shows how providers are using the AI-written summaries.…

Evidence of AI accelerating AI: AMIE from @google is a tour de force for the AIM community bit.ly/3tTVLhE Most striking is the multiple places in which LLMs are used to critique/augment/fine-tune other LLM's bypassing hugely expensive human assessments PRIOR to RCT.

Inspired by recent posts, here is my Christmas gift for you: the worst postdoc interview ever (without gifs because I am old but not that old).

Large-scale AI models are a once-in-a-generation opportunity to improve healthcare. 28 of the most forward-thinking payers and providers got together to figure out how we leverage frontier AI models to drive the change we want to see in healthcare. Here are our commitments:…

Folks don’t give epi or vasopressin for someone “going into Afib.” 😂

I loved seeing this paper expanding on our prior work evaluating LLM diagnostic performance, and especially appreciated that the Google team gave me and Zahir Kanjee MD, MPH, FACP a shout out as the amorphous “human raters” who evaluated GPT-4’s ddx.

Alan Karthikesalingam call us anytime 😂

Ever wonder what an LLM actually looks like under the hood?

This visualization is mesmerizing.

Check it out yourself: bbycroft.net/llm

Please show this to anyone who says stuff like “because of Ozempic there won’t be overweight people anymore.” We’ve had plenty of miracle drugs before and can still barely get people to take them!

Very cool recreation (and improvement) of Zahir Kanjee MD, MPH, FACP Byron Crowe and my study on GPT-4 and NEJM CPCs! (but with a diagnostically fine-tuned PaLM-2)