Large Language Models: A Series

Building products on LLMs and AI generally.

LLMs Aren’t Just “Trained On the Internet” Anymore

May 31, 2024

A path to continued model improvement.

I often see a misconception when people try to reason about the capability of LLMs, and in particular how much future improvement to expect. It’s frequently said that that LLMs are “trained on the internet,” and so they’ll always be bad at producing content that is rare on the web....

5 min read →

From Chatbot to Everything Engine

January 10, 2024

A curious design constraint signals an ambitious future.

This morning, OpenAI launched the GPT Store: a simple way to browse and distribute customized versions of ChatGPT. GPTs – awkwardly named to solidify OpenAI’s claim to the trademark “GPT” – consist of a custom ChatGPT prompt, an icon, and optionally some reference data or hookups to external APIs. In...

5 min read →

Going Way Beyond ChatGPT

June 30, 2023

Techniques for building products on LLMs today.

Modern instruction-tuned language models, or LLMs, are the latest tool in software engineers’ toolboxes. Joining classics like databases, networking, hypertext, and async web applications, we now have a new enabling technology that seems wickedly powerful, but whose best applications aren’t yet clear. ChatGPT lets you poke at those possibilities. You...

11 min read →

32K of Context in Your Pocket

March 15, 2023

A wild large-context LLM appears.

One month ago, I wrote about on the limits of 4K-token AI models, and the wild capabilities and costs that large-context language models may one day have. Today, OpenAI not only debuted GPT-4 with a doubly large 8K token limit, but demoed and began trials of a version that supports...

2 min read →

A 175-Billion-Parameter Goldfish

February 16, 2023

The problem and opportunity of language model context.

It has been a wild week in AI. By now, we’re getting used to the plot twist that rather than the cold Spock-like AIs of science fiction, large language models tend to be charismatic fabulists with a tenuous understanding of facts. Into that environment, last week Microsoft launched a Bing...

13 min read →

© Allen Pike. 👋🏼 You can contact me, or check out Steamclock.