Model Uncertainty Without Priors (Part 1)

This is Part 1 of a two-part post where I describe how to use martingales to quantify model uncertainty without a Bayesian prior. Part 1 is not very technical: I motivate the need for models that describe real-world phenomena, and then I describe the role of scientists in model creation. Part 2 is more technical: I describe and implement a technique for quantifying model uncertainty that is based on martingales and does not require Bayesian priors.

---

Systems (humans, robots, financial markets, weather, traffic) can be thought of as functions that take in input data and produce output data . Often, the function is very complicated and depends on a set of parameters that are unknown to us. I’ll refer to the combination of () as a “model.” Here are some examples:

System	Input	Output	Function	Parameters
Human	Sights, sounds, touch	Actions, decisions	The brain	Knowledge, mood, personality
Robot	Motor torques	Limb movement	Physics of motors and limbs	Limb mass, motor strength
Car	Steering angle, gas, brake	Position, speed, acceleration	Physics of the car	Car mass, road friction
Financial market	Number of shares bought and sold	Price per share	Supply and demand	How much people think it’s worth
Weather	Sunlight, wind, pressure	Temperature, rain	Physics of the atmosphere	Season, location

The job of a scientist is to answer the following questions:

1. What phenomenon do I care about?
2. How can I collect data for it?
3. What model (, ) describes this data?

Why do we want a model ( and )? Because knowing and allows us to make predictions, which in turn allows us to take better decisions. For example, when you’re driving a car, a sense of the relationship between the gas pedal and the position of the car is what keeps you from crashing.

For example, Isaac Newton cared about the way planets and stars move in the night sky. So he collected a bunch of data on planetary motions and came up with a function (his law of universal gravitation) and a parameter (the gravitational constant ) that matched the data better than ever before.

System	Input	Output	Function	Parameters
Gravity	Masses , and distance	Gravitational force		Gravitational constant

Finding good models is hard §

Unsurprisingly, the world is very complex and finding models that predict anything useful is very hard. For example, language models like Claude, Gemini, or ChatGPT can be thought of as functions that take in a prompt and output a response. For each of these models, the function is a version of a neural network, a big mathematical operation consisting mostly of matrix multiplications. The parameters are the matrices used in the matrix multiplications. It turns out that everything about modeling language is hard:

• What data do you need to describe language? Is it books, voice recordings, websites, video transcriptions? All of the above?
• How do you get this data?
• How do you set up your neural network? I.e., what is its “architecture”?
• Modern language models use trillions of parameters. How exactly do you learn/estimate/select the right ones? I.e., how do you train the network?

System	Input	Output	Function	Parameters
Language model	A prompt (e.g. “help me prepare for a job interview”)	Practice interview questions	A massive neural network	Trillions of numbers

Fortunately, at least for language, most of these questions have been answered, as evidenced by the success of modern language models. So in Part 2 of this post, I’ll explore a fourth question scientists often ask: once you have a model , how certain are you that your estimate is the right one?

Post 30/100 §