Code for Let's Think Dot by Dot: Hidden LLM Computations
github.com/JacobPfau/fill…

Results show that additional tokens provide benefits independent of token choice

LLM Hidden Computation

-Does LLM chain-of-thought work due just to greater computation?
-LLMs can use meaningless filler (like '......') to solve hard algorithmic tasks they could not solve otherwise
-Raises issue of LLMs engaging in hidden computations

arxiv.org/abs/2404.15758

The Wall Street Journal featured quotes from Norm Ai CEO, John Nay, in an article on AI guardrails over the weekend.

The WSJ discussed how, with ever more powerful AI systems we may soon be in a situation where AI agents are able not only to act on our behalf, but also do…

In our recent article, Professor @jbruh and John Nay share their insights on AI's impact on future lawyers.

What is a 'language model?' Should students use AI in their studies? And, most importantly, who do we blame when things go wrong?

ow.ly/ktpV50RcJEC

Had a great time chatting AI & Law at Vanderbilt last week at public lectures, Philosophy of AI Class, Robots, AI, and Law Class, and Engineering School!

We're excited to release a recap of our recent AI Agents and the Law Summit at the NYSE.

After a fireside chat with Lawrence H. Summers (OpenAI Board Member & Harvard University Professor), our CEO, John Nay, moderated a panel with the Attorney General of New Jersey, Matthew…

LLM Agents Are Superhuman At Fact Checking

-LLM breaks down long texts into sets of individual facts
-Checks each fact w/ multi-step reasoning processes
-Using Google & determining whether fact is supported by the search results
-20x cheaper than humans

arxiv.org/abs/2403.18802

Sharing parts of the Norm Ai fireside chat w/ John Nay and Larry Summers (@Harvard Professor & OpenAI Board Member)

In this video, they discuss whether more capable AI agents could increase productivity of smaller firms more than larger firms, and the tumult expected from AI

LLMs Are More Persuasive Than Humans: A Randomized Controlled Trial

Debate experiments:
-Between 2 humans; or btwn human & LLM
-Opponent has access to basic info about other; or not

Humans debating GPT w/ info about them had 82% higher odds of agreement

arxiv.org/abs/2403.14380

At the Norm Ai Agents & Law Summit at the NYSE 🏛, we were lucky to hear from Megan Ma (Stanford Center for Legal Informatics & MIT Computational Law Report)

She explores how we are moving from AI-driven companies to an AI-native world & whether we need Chief AI Officers

LLM Prediction Capabilities Match Human Accuracy

-A crowd of 12 LLMs vs a crowd of 925 human forecasters on a 3-month forecasting tournament
-LLM crowd is statistically equivalent to the human crowd
-Replicates the 'wisdom of the crowd' effect for LLMs

arxiv.org/abs/2402.19379

LLM Human-Level Forecasting

-A RAG system automatically searches for relevant information, generates forecasts, & aggregates predictions
-On questions from competitive forecasting platforms published after LLM cut-offs, system can beat human forecasters

arxiv.org/abs/2402.18563

At the Norm Ai AI Agents & Law Summit at the NYSE 🏛:

Eric Vandevelde, Gibson Dunn Partner and AI Practice Lead (who has repeatedly been ranked as one of the top AI lawyers in CA), discusses the scramble for turf-claiming he is witnessing in the GenAI regulatory environment

Now releasing content from our NYSE 🏛 AI agents event

Here Attorney General Matt Platkin discusses large gap he sees in AI expertise between gov & private sector

He also underlines need for society to come together to decide where human responsibility should apply along complicated Gen AI stack

Search Dynamics Bootstrapping

-Toward LLMs as better planners/reasoners...

-Search dynamics expressed as a token sequence, then fine-tuned via expert iterations to perform fewer steps
-Transformer model optimally solves previously unseen Sokoban puzzles

arxiv.org/abs/2402.14083

Glad to have made (small) contributions to the LangChain community/codebase back in the day (years ago, feels like decades in LLM time)

Harrison Chase is building an important community and company

LLM Chain-of-Thought Reasoning Without Prompting

-CoT reasoning paths elicited from LLMs by simply altering decoding
-Enables assessing LLMs' intrinsic reasoning abilities
-More CoT in decoding path, more confidence in answer
-Outperforms greedy decoding

arxiv.org/abs/2402.10200

Language Feedback Models from LLMs

-Language Feedback Models (LFMs) identify actions that help achieve tasks specified in the instruction
-LFMs outperform using LLMs as experts to directly predict actions to take
-LFMs generalize to unseen environments

arxiv.org/abs/2402.07876

An Interactive LLM Agent

-Trains AI agents across diverse pre-training strategies (visual masked auto-encoders, language modeling, next-action prediction)
-Multimodal & multi-task
-Generates contextually relevant outputs in Robotics, Gaming, & Healthcare

arxiv.org/abs/2402.05929

The Attorney General Matt Platkin perspective on AI Agents & Law was insightful.

Thank you for being part of this event!