vibe eval is giving off good vibes!

Let’s just say that 1.5 Pro’s abilities have been severely underestimated.

Bunch of improvements to Gemini 1.5 Pro today:

* No waitlist on the API.
* Better at reasoning, multimodal, instruction-following and a bunch of other evals.
* Video+Audio support: Video inputs can now parse audio as well on AIStudio.

developers.googleblog.com/2024/04/gemini…

【Gemini 1.5 Proの性能が群を抜いてすごい件。GPT-4、Claude 3を凌駕】

目でも見えないくらい細かい生成AI企業のカオスマップをGemini 1.5 Pronで解析したところ、5分くらいずっとAIが動いて企業名を書き出した。

GPT-4：解析不能
Claude 3：本当に一部のみ

比較すると差は歴然。

Gemini 1.5 is free in AI Studio right now, with 1M context length. 😉

Waitlist should move pretty fast, sign up here: aistudio.google.com/app/waitlist/9…

Congrats to the anthropic team on shipping!

GPQA is kinda the only eval I trust on this table, and opus seems clearly better there than any other publicly available model.

For this simple retrieval task:
- short input: Gemini 1.5 wins
- medium input: methods other than Gemini 1.5 perform poorly
- long input: methods other than Gemini 1.5 fail totally

IMO, Gemini 1.5 Pro is the largest leap in long form video understanding since I started my PhD in long form video + language understanding back in 2016, and more will be coming 🔥

I gave Gemini 1.5 Pro Mr. Beast’s latest video.

It’s 22min and 347,849 tokens.

I asked it to “Reply with [NAME OF GUY ON BLUE FOOTBALL] has [TOTAL CASH PRIZE POOL] of [FOOD ITEM BEING ADVERTISED].”

Watch the model get it 100% correct.

Insane leap for AI.

Shit, Google wasn't kidding.

Gemini 1.5 Pro just went straight from a full movie to a summary in seconds.

No transcription, no intermediate steps. Just visual tokens -> summary.

Next up, validating the haystack tests.

To preempt any confusion: Multimodal queries don't go through Pro / Ultra yet, but that's coming soon too!

How do the SOTA large Vision-Language Models (or Multimodal LLMs) performance on this task? A striking performance gap from average humans (e.g., 99% vs. 39-72% on the real-world subset)!
Congrats to the Gemini team for the impressive lead over GPT-4V Jeff Dean (@🏡)! Curious about

🔥Breaking News from Arena

Google's Bard has just made a stunning leap, surpassing GPT-4 to the SECOND SPOT on the leaderboard! Big congrats to Google for the remarkable achievement!

The race is heating up like never before! Super excited to see what's next for Bard + Gemini

Elad Gil Patrick Collison In AI at least, the real 30 under 30 imo you have never heard of. They are 5 layers down the org chart from the CEO. They are usually not on Twitter, they have an unmaintained LinkedIn, they don’t go on podcasts, and they maybe published at one point but don’t do so anymore. They