you can reach out to me via DM or email: [email protected]

i am a 3rd-year PhD student & am looking to meet more academics interested in these research areas! giving talks has been a great way for me to do this, and i am so grateful for these opportunities ☺️

gave a talk at last week's Stanford MLSys seminar about evaluating LLM pipeline output quality, especially for custom tasks at scale: youtube.com/watch?v=eGVDKe…

if your lab is interested in data management x LLMs x HCI, i would love to give this talk & get your feedback! 🙏

Devin Petersohn from Snowflake talked to CMU Database Group yesterday about how dataframes are meeting databases. If you missed it, you can catch up here: youtu.be/7TyIjqvfWto?fe…

Lots of exciting technical nuggets in the talk. Another interesting part is Devin talking about hanging out…

since I'm deeply immersed in evals right now (and the process of building them) I got a kick out of this paper from Shreya Shankar J.D. Zamfirescu bjoern hartmann @[email protected] Aditya Parameswaran Ian Arawjo (@[email protected])

it addresses the challenge of time-efficiently coming up with evals that are aligned with practitioners

some…

Evals are arguably the hardest part of LLMOps. LLMs mess up, so we check them w/ other LLMs, but this feels icky. Who validates the validators??

We built an interface to align LLM-based evals with user preferences, learning a lot about why this is hard: arxiv.org/abs/2404.12272

Our work on extracting information from police records is helping with important news articles — here's one such article featuring our collaborator Lisa Pickoff-White at the California Reporting Project.

Hey data scientists and analysts, take two minutes to fill out this survey on searching for data from one of my friends at Berkeley.

If you do, comment here. I'll be giving away a digital copy of Effective Pandas 2 to one of the respondents.

Good luck to everyone doing PhD visit days this week! If you have any questions about Berkeley feel free to hmu.

We're trying to understand when, how, and why people search for datasets, to build better dataset search systems. Please RT this 2 minute survey!

cc BIDS - Berkeley Institute for Data Science Berkeley Computing, Data Science, and Society Berkeley School of Information UC Berkeley EECS

Congrats to MCB's Dipti Nayak Nayak Lab and Meng-meng Fu Meng-meng Fu on being named Sloan Foundation research fellows! 👏🎉

I’m clearly biased (as I work with two of the three co-chairs) but I think DEEM Workshop @ SIGMOD is a terrific workshop with a vibrant community! Do consider submitting!

⭐️ SPADE is a very intriguing work on testing software built on top of LLM prompts. The authors propose a method to automatically synthesize assertions for prompts to identify bad outputs. (1/n)

🚀 It was great fun integrating SPADE, a novel framework for synthesizing LLM evaluations, with UpTrain.

Big shoutout to the authors:

Shreya Shankar HaotianLi ParthAsawa Madelon Hulsebos YimingLin J.D. Zamfirescu Harrison Chase Will Fu-Hinthorn AdityaParameswaran wu

Berkeley Data Systems and Foundations team were a big presence at CIDR 2024 #cidr24 including two talks and two gong show wins. Happy to reunite with alumni and the godfather himself, Mike Stonebraker. Good times! UC Berkeley EECS UC Berkeley UC Berkeley EPIC Lab UC Berkeley Sky

Today at the CIDR conference, Marcin Zukowski, Niels Nes and I received the Test of Time award for our CWI DA paper on vectorized query execution!
Very thankful for this honor & for creating this award going forward.
We gave a talk: bit.ly/cidr2024-test-… (with speaker notes)

We all know LLMs make mistakes. One simply cannot deploy LLM pipelines without assertions, yet writing good assertions is tedious & hard. So, we built SPADE, a system that analyzes prompts & auto-generates custom assertions in low-data settings: arxiv.org/abs/2401.03038

It was an amazing edition of the Table Representation Learning workshop at NeurIPS. Familiar faces from ‘22 and many new ones; a persistent and growing community! ❤️ Below a brief reflection (1/6)