Walking neural networks with ChatGPT, Part 2

Part 1 of this article showed that a small neural network (one that approximates the square root function) can be “walked” to see how an input value such as 5 gets turned into an output y value such as 2.2318 (which, considering that random noise was deliberately added to the training data, is a reasonable approximation for the correct answer, 2.2361).Here in Part 2, we’ll extend this to walk a small neural network that represents text. The net still consists of numbers representing weights and biases, but the network was trained with vectors of numbers that are “embeddings” (which we’ll explain) representing text. Note that the net still consists of weights and biases, NOT the text itself, nor even the embedding vectors for the text.In Google Drive, I selected a copy of Part 1 of this article, and clicked on “Summarize this file”. Google Gemini responded with the following summary of the article that focused on ChatGPT (it’s often helpful to have different AI chatbots comment on each other’s work):

The PDF file `chatgpt_walk_net_part1.pdf` documents a project where the author worked with OpenAI’s ChatGPT and Anthropic’s Claude.ai to write Python code for inspecting neural networks. This was part of a larger project applying software reverse engineering to AI.

Here are the key takeaways from the document:

• “Walking” a Neural Net: The core concept is “walking” a neural network, meaning following an input value `x` through each element of the network to produce an output value `y`. This involves tracking how the input is multiplied by weights, added to biases, and affected by threshold functions (like ReLU).
• Interpretable NeuralNet: This work is related to a more complex project called “Interpretable NeuralNet,” which uses “surrogate polynomial coefficients” to understand the formulas learned by a neural network. However, this document focuses on a simpler method of direct inspection.
• Linear Networks: The document starts by examining purely linear networks (no hidden layers). In such cases, the learned formula (y = wx + b) can be directly extracted from the network’s weights (w) and biases (b).
• Hidden Layers and Non-linear Activations (ReLU): When hidden layers and non-linear activation functions like ReLU [rectified linear unit which == max(0,x)] are introduced, the process becomes more complex. The network’s function becomes piecewise linear with “kinks” or breakpoints introduced by the ReLU function. While it’s still mathematically possible to “walk” the network, the resulting formula becomes much more complex, especially with larger networks.
• ChatGPT’s Role: The document includes excerpts of conversations with ChatGPT, showcasing how the AI chatbot helped in writing the Python code, explaining concepts, and troubleshooting issues.
• Illustrative Examples: The document provides Python code examples that demonstrate these concepts, including creating synthetic data, training small neural networks, and “walking” their weights. There are also discussions about how random initialization, gradient descent, and other aspects of neural network training work in practice, and how they relate to the actual code being written and executed.
• Square Root Example: A significant portion of the document focuses on creating a small neural network that learns to approximate the square root function (y = sqrt(x)). This example further illustrates the complexities introduced by ReLU activations.
• Limitations: The document emphasizes that while the “walking” approach is possible for small networks, it becomes practically intractable for larger, more complex architectures due to the combinatorial explosion of possible piecewise regions.

In essence, the document details an exploration of how to understand what a neural network is doing internally, by directly examining its weights, biases and the effects of threshold functions, how this can be done in simple case, how this process is made dramatically more complex by non-linear activation functions in deeper networks, and illustrating how AI can assist the process of exploring this kind of reverse-engineering.

I’ve been getting heavily into AI-generated summaries, which often trigger a “hmm, this summary is very good, but it’s missing these three points which are more important” — and I likely wouldn’t have thought of those three points without first having seen the AI summary. So, here is how I summarized Part 1, after reading several AI-generated summaries, including the one above:

• Large language models (LLMs) such as ChatGPT, Claude, and Google Gemini, are, at bottom, neural networks.
• Even the largest neural net, like ChatGPT itself, after its has been trained with sample data, computes a function: y=f(x) .
• A neural net can approximate the effect of a function such as square root (and log, tan, etc.), without ever knowing that’s the function it is approximating, and based only on some sample (x,y) training data that was generated by the function.
• Neural networks are overly opaque (non-transparent) for many applications: while the net can amazingly compute all manner of things (including an appropriate next word in a sentence), they are currently much less good at telling us how/why they arrive at their y outputs, given x inputs.
• However, they are not totally opaque; neural networks can be reverse engineered.
• In addition to applying post-hoc analysis to neural networks, such as surrogate polynomial coefficients, in certain limited cases one can “walk” through the neural net to extract the formula a neural net has learned, directly from the net’s weights/biases.
• Any non-trivial neural net, once it uses nonlinear activation functions such as ReLU, will contain “kinks” or regions; each seem between regions is marked by an activation function.
• We don’t want a neural net to memorize its (x,y) training data; we want to net to learn the (x,y) relationship, which is very different; we don’t want the effect of a lookup table; instead, we want something that can predict y give input values x the net hasn’t seen before; we want neither “overfitting” nor “overfitting”; we want the net to learn both accuracy and generalization.
• We saw an extended example of interacting with an AI chatbot like ChatGPT to produce code and assess its output, when the user runs the code locally.

Topics discussed here in Part 2 include:

• Apply net-walking from small neural nets representing numeric data to those representing text
• Similarities and differences from LLMs
• Walking a tiny toy language model
• What text “embeddings” represent in the tiny language model, compared/contrasted with embeddings in LLMs
• Temperature, argmax/softmax, logits, sampling, and selecting the next token from a list of the most-probable next tokens
• Historical comparison of finding square root by guessing/iterating, with the neural net’s method of training/error-adjusting weights
• The centrality of error-correction (nets may be doing pattern recognition, but the way they get there is via error correction)
• The 768 “sweet spot”
• The separate roles of backpropagation vs. the optimizer
• Transformers and “attention” are NOT performing recurrence, though it can appear that way in a chat
• Transformers like ChatGPT are in some ways “just like” our tiny neural networks (except for the ways in which they’re not)
• Tweaking a neural net’s parameters to avoid local minima

Applying net-walking from small numeric neural nets to (equally small) text nets, with applicability to LLMs

Below, my prompts and commentary in italics. In addition to ChatGPT’s responses, I’ve sometimes also included material from the right-hand “chain of thought” (CoT) panel that appears if you click on e.g. “Thinking for 7 seconds”. [Whether such explicit CoT statements in prose truly reflect an underlying sequence of reasoning steps, and if so, whether prose is the best way for CoT to be implemented, is discussed in “Worries about latent reasoning at LLMs” at the LessWrong blog re: Meta’s COCONUT (Chain of Continuous Thought); see also my CoT page.]

In an earlier session, ChatGPT had built a nice small Python script for me as described in a summary I asked it to generate in an attempt  to carry over “state” or “memory” from one session to another:
“Multi-layer / Nonlinear: We demonstrated a small ReLU net with a handful of neurons and manually ‘walked’ the weights to show how an input flows through each layer, forming a piecewise (ReLU) expression.”

AS (long multi-paragraph prompt):

That earlier small net did a good job modeling y=sqrt(x), basically achieving the effect of regression on the (x,y) training numbers, but not using top-down regression formulas, and instead simply learning square root.

I now want to discuss the extent to which what is learned from a small numeric net like that can be applied to a LLM, such as yourself ChatGPT. Is it safe to say that you ChatGPT (or other LLMs like Claude) are doing essentially the same thing that our tiny toy neural network did, although on a monumentally larger scale? Might I describe you as performing a y=f(x) function, where x is e.g. text like what I’m typing right now, converted to numbers in some way (word or sentence embeddings??), input to your model, and coming out as predicted next word/phrase/sentence, then fed back in for more (using attention/transformer which I’d prefer to not get into at the moment if possible)?

This is probably very superficial. I haven’t mentioned how your y=f(x) gets trained in the first place (though I assume backpropagation, error correction), and I haven’t mentioned subsequence training that might occur in tool use, data analysis, even reasoning (to the extent that doesn’t simply come “for free” with your basic training).

But please walk me through what one could learn from small toy numeric neural nets might still be applicable to an LLM giant working with text or images, as well as numeric user input.

ChatGPT:
[Thought about small nets and LLMs for 31 seconds:
Drawign [sic] connections
I’m examining the similarity between “toy numeric networks” with ReLU regression and large language models like ChatGPT, focusing on how both predict outputs based on inputs.
Connecting concepts
Interestingly enough, small toy networks and large language models share core function approximation principles, despite their distinct inputs, architecture, and scale.
Understanding structures
I’m noting that neural networks, whether feed-forward with ReLUs or Transformer-based, both function as function approximators and are trained using gradient-based methods.
Mapping connections
I’m working through the iterative training of neural networks, their increasing complexity with layers, and constant progress towards sophisticated pattern recognition. Adjusting parameters and activation functions are crucial steps.
]

At a high level, all neural networks — be they tiny MLPs learning sqrt(x) or massive language models like ChatGPT — are doing the same fundamental thing: function approximation. In other words, given an input x, the network produces an output y = f(x). The differences arise mostly in:

1. What the inputs and outputs represent
2. How large/complex the network is
3. How it’s trained and what data it’s trained on
4. What architectural features help it handle very large or complex input domains (e.g. text sequences, images, etc.).

Below is a “walk-through” explaining what carries over from a small numeric net to a giant LLM, and what changes.

1. The Essence: A Learned Mapping

In your toy ReLU network for sqrt(x):

• Input: a single scalar x
• Hidden layers: a handful of neurons, each computing a linear transform plus ReLU
• Output: a single scalar ^y​ that the net tries to match to sqrt(x)​

Under the hood, you have parameters (weights, biases) that get adjusted via gradient descent and backpropagation to minimize some loss like (^y – sqrt(x))^2. [^y = hat y, or predicted y]

How LLMs Are Similar

• They also have inputs and outputs: The input is a sequence of tokens (words or subword pieces) converted to numeric embeddings; the output is a distribution over the next token(s).
• They also have layers of parameters: Instead of a few layers and a handful of neurons, LLMs have many layers and thousands (or billions) of hidden dimensions.
• They also learn via gradient-based optimization: They start with random weights, see vast amounts of text, and iteratively adjust parameters to minimize a next-word prediction loss.

Hence the fundamental concept is the same: find the parameter set W that best approximates a mapping f(x). Whether x is “the current text so far” or a single numeric value, the approach is mathematically quite similar. [This similarity also applies to the target/output value, whether y is sqrt(x) or the next word/token/phrase.]

2. Input Encoding

For the toy network: You fed in a plain number x.

For an LLM: The “plain text” (characters or tokens) is first turned into embeddings — vectors of real numbers. This is analogous to feeding sqrt(x)-net a single real number, but on a far grander scale: each token is represented by, say, a 768-dimensional embedding (or 1024, or more). [See embeddings discussed later in this web page; in our tiny text net here, we have only 2 dimensions (hidden_dim=2 in tiny_text_net.py .]

In both cases, you end up with vectors of real numbers as the raw “food” for the network. The difference is just dimensionality and how we construct those embeddings.

3. Output Interpretation

For your sqrt(x)-net: The output was also a single real number, and you interpret it as the approximate sqrt(x).

For an LLM:

• The network outputs a large vector of logits (one entry per possible token).
• We convert those logits to probabilities (via softmax) for each possible next token. [See logits and softmax discussed later in this web page.]
• The model “chooses” (or samples) the next token based on these probabilities.

In essence, it’s still a numeric output. It’s just that we interpret the numeric output as a probability distribution over a giant vocabulary. [See tiny_text_net.py]

4. Piecewise / Nonlinear Structure

A small ReLU net can be piecewise linear when we manually expand it. An LLM is typically built from many layers of multi-head attention, feedforward blocks (often with ReLU or GeLU activations), residual connections, etc. But the same principle holds:

• Each activation function (like ReLU or GeLU) introduces nonlinearity.
• Each linear or attention head is still a linear transform with a huge weight matrix.

Put enough of these together, and you can approximate extremely complicated functions.

[AS (later follow-up): You earlier said, “Each activation function (like ReLU or GeLU) introduces nonlinearity.” Give me a sentence or two on why that is desirable. Is it simply in order to be able to map non-linear x/y relations, such that it doesn’t serve a purpose for mapping linear relationships? Or does everything, even the simplest seemingly linear x/y relation, turn out in a neural net to benefit from non-linearity, or what?
ChatGPT:
Short Answer:
Neural networks must include nonlinear activations (ReLU, sigmoid, etc.) to capture nonlinear relationships in data. Without these nonlinearities, a network of stacked linear layers collapses to a single linear mapping, severely limiting the functions it can learn.
1. Why Nonlinearity Is Essential
Expressiveness:
— A stack of purely linear layers (like Linear -> Linear -> Linear) effectively collapses to one linear transformation.
— By adding a nonlinear activation (e.g., ReLU) between layers, the network can bend the input–output mapping into curves, corners, or more complex shapes.
Universal Approximation:
— The theoretical power of neural nets as “universal approximators” relies on having nonlinear elements.
— Without nonlinearity, the net can’t approximate complicated functions like y=sqrt(x), classification boundaries, or anything else that isn’t straight-line mapping.
Practical Impact:
— In many real problems (like image classification, language modeling, etc.), input–output relationships are far from linear. Nonlinear activations allow the net to learn those complexities.
— Even for simpler, mostly linear tasks, mild nonlinearity often helps the network converge faster or capture slight deviations from perfect linearity.
2. Do Simple Tasks Benefit from Nonlinearity?
— If your data truly is perfectly linear (say y=2x+3), technically a linear-only network (one layer, no activation) can learn that exactly.
— But in practice, real-world data rarely aligns to a perfect linear form. Even “simple” relationships might have nuances.
— Therefore, most neural nets keep a nonlinear activation by default, because it makes the model more versatile. If the data really is linear, the network can still approximate it. If it isn’t, the network can adapt to those extra complexities.
]

5. Iterative Training and Error Correction

Both systems use backpropagation and an optimizer (like SGD, Adam, etc.) to repeatedly: [See explanation below of the difference between backpropagation and the optimizer.]

1. Make a forward pass: compute ^y=f(x).
2. Compare ^y​ to the target (for LLMs, the target is “the next token in the sequence”). [During training, the next token, or a masked token, are known.]
3. Compute a loss.
4. Do a backward pass: compute gradients (∂loss / ∂W) ​.
5. Update the weights to reduce the loss.

Whether you’re learning sqrt(x)​ or next-word prediction, it’s the same cycle — just repeated millions or billions of times on massive text corpora in the LLM case.

6. Why the “Toy Net” View Is Still Instructive

1. Universal Approximation
   A tiny ReLU net (with enough hidden units) can approximate just about any continuous function in 1D, 2D, etc. [See here for brief discussion of the Universal Approximation Theorem (AUT); note “continuous” limitation.]
   LLMs extend that idea to function spaces mapping from sequences of tokens to next tokens. The “same rules” (linear transformations + activation + large data + iterative optimization) let the net approximate language patterns.
2. Layers as Feature Builders
   In a tiny net, each hidden layer might successively capture more refined “features” of $x$. In LLMs, each layer can be viewed as building richer contextual features of the entire input text. The concept—transformation of raw input into increasingly abstract features—remains the same. [See (forthcoming) for discussion of “features” captured by, or least said afterwards said to be captured by, neural nets.]
3. Overfitting vs. Generalization
   Even in your small net, you might worry about overfitting if you have too many neurons or train too long on too few data points. LLMs can overfit too (though typically they’re trained on so much data that they instead face issues of memorizing rare training examples).
4. Interpretability Challenges
   Even with a small net, expanding out all the piecewise expressions can get unwieldy. With an LLM, interpretability is even harder. The difference is mostly scale and architectural complexity, but the fundamental “black box” nature is similar.

7. The “Next Word” Aspect

One thing that distinguishes language models is the sequence aspect: they have to repeatedly apply the function for one token at a time, or sometimes for many tokens in parallel, to get the next token. It’s akin to feeding the output back in as part of the next input. But at each step, there’s still a big forward pass: “Given all tokens so far, produce the next token distribution.” This yields a chain of predictions that forms the conversation. [Somewhat akin to feeding the output back in, but NOT recurrent: see discussion below of transformers/attention.]

8. Omitted Topics (But Still Important)

You mentioned you’d rather not dive deeply into attention or the internal workings of the Transformer. That’s perfectly fine: just know that the basic building blocks—linear transformations, “activation” functions, many layers, big weight matrices—are conceptually the same as your toy net. The attention mechanism is a specialized layer that helps the model handle long-range dependencies in text more efficiently than a simple MLP. But it still stands on the same neural network foundation. [See below for more detailed discussion. Note here how ChatGPT makes sure I don’t completely sweep this under the rug.]

9. Summary

• Core Concept: Both your small numeric net and a large language model are neural nets that learn parameters to approximate a function from input (numbers or text embeddings) to output (predicted number or next-token distribution).
• Scale: LLMs have orders of magnitude more parameters, training data, and architectural complexity (many layers, attention blocks, massive parallelization).
• Training Loop: Fundamentally the same: forward pass, compute loss, backward pass, update weights.
• Applicability: Studying small nets clarifies how neural networks “build” a function internally, how weights are updated, and how nonlinearity + linear combos approximate functions. Those ideas absolutely carry over to LLMs, just scaled up dramatically.

Ultimately, everything you learn about how your toy net approximates sqrt(x)​ does apply to how giant models approximate “the next word.” The differences are mostly in the details of architecture, training scale, and the complexity of the input/output domains.

Walking a tiny toy language model

AS: Is there some way a tiny test could be written, as similar as possible to the toy ReLU network for sqrt(x), but that handled text input/output in some way? The toy ReLU network “walked” the weights/biases of the net to show how the net after training is used to predict sqrt(x). Let me show you what output looks like: [Showed this ChatGPT session the same “python walk_sqrt_net_2.py” output as shown above.] 
Is it foolish to think something like this could be done with a little bit of text?

ChatGPT:
[Thought about toy model for text for 5 seconds
Envisioning a neural network
I’m thinking through a small text-based neural network, using minimal embeddings and ReLU layers to map input tokens to output tokens, while avoiding overfitting.
Visualizing transformations
I’m mapping out a tiny text-based neural network with small embeddings and ReLU layers, demonstrating how it transforms input tokens through a brief code snippet and a mini dataset.
Coding up the solution
OK, let me see. I’m coding a small neural network in PyTorch, outlining each layer’s expansion. The example, though limited, effectively demonstrates the process.]

Not at all foolish! You can create a toy neural net for text that is analogous to your 2×2 ReLU net for sqrt(x). Of course, text is more complex than a single number, but the principle remains:

1. Convert text to a numeric representation (e.g., a small embedding).
2. Pass that through a couple of ReLU (or other) layers.
3. Produce an output distribution over tokens (or a single “predicted token”).
4. “Walk” the trained net by expanding the linear transformations + ReLUs exactly like you did with sqrt(x).

Below is a contrived end-to-end example in PyTorch to demonstrate the idea. We’ll keep the vocabulary extremely small (like 3–5 tokens) and train on a silly “dataset” just to illustrate. After training, we’ll do a forward pass manually in python, printing each intermediate activation. This is only to show that it’s possible—it’s definitely not a useful text model!

1) Setup and Dummy “Dataset”

We’ll imagine we have a vocabulary of just 5 tokens:

1. ["hello"]
2. ["world"]
3. ["cat"]
4. ["dog"]
5. ["!"]

We want a toy network that tries to map "hello" to "world" and maybe "cat" to "dog", so we have some minimal variety. We’ll train it as if it’s a next-token prediction. That is:

• Input: a single token index
• Output: distribution over next token index