La Shader is Shading

Why this is hard

Artists don't want to be copied. The question is: how do you use generative AI as a shader-making tool rather than a replacement?

Shader code is notoriously difficult to vibe-code. The gap between a visual idea and the GLSL math that produces it is enormous. Fragment shaders run per-pixel in parallel on the GPU -- there's no memory between pixels, no iteration history, and no way to "look at the whole image."

Current AI models fail at direct image-to-GLSL generation. The output bears little resemblance to the target.

Claude attempting to generate a shader from an image directly — Direct LLM shader generation: the output doesn't match the target.

GLSL fragment shaders execute per-pixel in parallel on the GPU. There is no memory between pixels, no iteration history, and no ability to "look at the whole image" from within the shader. Visual effects like flow, turbulence, and organic patterns must be expressed as pure mathematical functions of UV coordinates and time.

Current vision-language models understand image content but cannot reverse-engineer the mathematical procedures that would produce those visuals. The mapping from "soft organic flowing particles" to "layered fbm noise with curl-based advection" requires domain-specific knowledge that general models lack.

This is why a single-shot approach fails. You need an iterative loop: generate, render, compare, critique, and refine -- exactly what this system does.

How it works

A vision model + code-editing LLM iteratively improve a shader to match a target image.

Phase A: Discovery

Reference Text Anchor shader

+

Target Image User upload

→

Gap Analysis GPT-4.1-mini

→

Tailored Prompts SIMILAR | DIFFERENT | BRIDGE

↓

Phase B: Iteration Loop

Generate / Edit Shader GPT-4.1-mini

→

Render Frames ModernGL

↓

Score LPIPS / AlexNet

+

Critique GPT-4o-mini (Vision)

↓

Log & Trace Weave / W&B

↑ repeat N iterations · feedback drives the next edit ↓

↓

Best Output Lowest LPIPS score

MVP walkthrough

Upload a target, ground with a reference

Provide any image as the visual target. Optionally pair it with a reference shader (e.g. a ShaderToy effect) as a starting point. The Discovery phase analyzes what techniques to borrow, what to change, and what to adapt.

Discovery + Iteration

The agent runs gap analysis, classifying techniques as SIMILAR, DIFFERENT, or BRIDGE NEEDED. Then it iterates: generate GLSL, render frames via ModernGL, score with LPIPS, critique with GPT-4 Vision, and feed back. Watch the LPIPS scores converge.

Iterations grid showing LPIPS scores improving across iterations

VLM Critique

Like a studio critique: the vision model identifies COLOR DELTA, WHAT'S WORKING, and WHAT NEEDS TO CHANGE. Color matching is always the #1 priority. This structured feedback drives the next edit.

Critique modal showing structured VLM analysis

Evaluation with LPIPS

LPIPS (Learned Perceptual Image Patch Similarity) uses a pre-trained AlexNet to measure perceptual distance -- how different two images look to a human, not how different they are pixel-by-pixel. Lower is better.

Traced with Weave

Weave dashboard showing traced edit_shader calls across iterations

Full Trace Every LLM call -- discovery, generation, editing, repair -- is traced end-to-end.

Error Recovery Compile errors are caught, diagnosed, and auto-repaired. The agent consults learned GLSL rules before every generation.

Visibility Full observability into the agent's decision-making at every iteration: prompts, responses, scores, and critique.

GLSL compile rules the agent learned

GLSL COMPILE RULES -- check these before outputting any shader: 1. VECTOR CONSTRUCTORS: exact component count required. vec3(vec2, float) OK | vec3(vec2, vec2) WRONG (4 into 3) 2. MATRIX MULTIPLY: matrix on LEFT only. mat2*vec2 OK | vec2*mat2 WRONG 3. MISSING FUNCTIONS: every function call needs a definition above the call site. 4. CASCADING ERRORS: fix the FIRST error only. COMMON ERRORS: "Incompatible types in initialization" → wrong component count "'*' does not operate on 'vecN' and 'matN'" → put matrix on LEFT "undeclared identifier 'X'" → add missing function definition

## Vector Constructor Rules
GLSL constructors require exact component counts.
Mismatches cause silent failures that cascade into
"undeclared identifier" errors.

vec3 b0 = vec3(x.xy, y.xy);  // WRONG: 4 into vec3
vec3 b0 = vec3(x.xy, y.z);   // OK: 3 into vec3

## Matrix Multiplication Rules
GLSL matrix multiply is order-dependent.
Matrix must be on the LEFT.

vec2 s = vec2(x.x, x.y) * m;  // WRONG
vec2 s = m * vec2(x.x, x.y);  // OK

Safer alternative: use dot products instead of
matrix multiplication for hash functions.

## Missing Function Definitions
Every function call must have a corresponding
definition. Common causes:
- Incomplete copy-paste from reference code
- Missing utility/helper functions
- Typo in function name

## Debugging Cascading Errors
When you see multiple errors, the root cause is
always the FIRST error. Fix line N; the rest resolve.

## Common Error Summary
| Error | Fix |
| Incompatible types     | Count components |
| '*' does not operate   | Matrix on LEFT   |
| undeclared identifier  | Add function def |