Shader basics, and how to connect them to music

The mental model

Forget everything about drawing. A fragment shader is one function, and the GPU calls it once for every pixel it needs to fill — a few million times per frame, all at once, on hundreds of cores.

Each invocation is almost completely isolated. It knows:

• which pixel it is (gl_FragCoord),
• whatever constants you uploaded for the whole frame (uniforms),
• whatever it chooses to read from a texture.

It cannot see its neighbours, cannot see the previous frame, and cannot write anywhere except its own output colour. Every visual you have ever seen on Shadertoy is built out of that.

This sounds restrictive and it is, but it is also why it scales. There are no objects to iterate, no draw calls to batch, no scene graph to traverse. The cost is fragments — pixel count times the complexity of the function — and nothing else.

The smallest useful setup

Because we only want a fragment shader, the geometry is a formality: two triangles covering the clip-space square. The vertex shader does nothing interesting on purpose.

The vertex shader, in full

#version 300 es
in vec2 aPosition;
void main() {
    gl_Position = vec4(aPosition, 0.0, 1.0);
}

The JavaScript scaffolding

const canvas = document.querySelector('#visualizer');
const gl = canvas.getContext('webgl2', { antialias: false, alpha: false });
if (!gl) throw new Error('WebGL2 unavailable');

function compile(type, source) {
  const shader = gl.createShader(type);
  gl.shaderSource(shader, source);
  gl.compileShader(shader);
  if (!gl.getShaderParameter(shader, gl.COMPILE_STATUS)) {
    throw new Error(gl.getShaderInfoLog(shader));   // always read this log
  }
  return shader;
}

const program = gl.createProgram();
gl.attachShader(program, compile(gl.VERTEX_SHADER, VERTEX_SHADER));
gl.attachShader(program, compile(gl.FRAGMENT_SHADER, FRAGMENT_SHADER));
gl.linkProgram(program);
gl.useProgram(program);

// Two triangles covering the screen
const buffer = gl.createBuffer();
gl.bindBuffer(gl.ARRAY_BUFFER, buffer);
gl.bufferData(gl.ARRAY_BUFFER, new Float32Array([
  -1, -1,  3, -1,  -1, 3
]), gl.STATIC_DRAW);
const position = gl.getAttribLocation(program, 'aPosition');
gl.enableVertexAttribArray(position);
gl.vertexAttribPointer(position, 2, gl.FLOAT, false, 0, 0);

// One draw call per frame, forever
gl.drawArrays(gl.TRIANGLES, 0, 3);

That vertex array is a single oversized triangle rather than two — a standard trick. One triangle whose corners sit outside the viewport covers the screen with fewer vertices and avoids a seam along the diagonal where two triangles would meet.

Coordinates, the thing everyone gets wrong

gl_FragCoord.xy gives pixel coordinates — (0.5, 0.5) to (width - 0.5, height - 0.5), with the origin at the bottom left. Almost nothing you want to draw is expressed naturally in those units, so the first line of nearly every fragment shader converts them.

uniform vec2 u_resolution;

// 0..1 across the screen — good for gradients and texture lookups
vec2 uv01 = gl_FragCoord.xy / u_resolution;

// -1..1, centred, aspect-corrected — good for shapes
// Dividing by min() keeps circles circular and guarantees the
// shorter axis spans exactly -1..1 on any screen.
vec2 uv = (gl_FragCoord.xy * 2.0 - u_resolution) / min(u_resolution.x, u_resolution.y);

That second form is the one to internalise. Dividing by u_resolution.y alone is common and works, but on a phone in portrait the visible horizontal range collapses. Dividing by min() means the shorter axis always spans -1..1, so your composition survives rotation from landscape to portrait — which matters a great deal for something people hold up at a party.

From there, two derived values unlock most effects:

float radius = length(uv);            // distance from centre
float angle  = atan(uv.y, uv.x);      // -PI..PI around the centre

Shapes without geometry

You do not draw a circle. You ask, for this pixel, "how far am I from where the circle's edge should be?" and turn that distance into a colour. This is signed distance field thinking, and it is the core skill.

float d = length(uv) - 0.4;                    // negative inside, 0 on the edge

float disc = step(d, 0.0);                    // hard-edged, aliased
float smooth = smoothstep(0.01, -0.01, d);    // anti-aliased edge
float ring = smoothstep(0.02, 0.0, abs(d));   // abs() turns a disc into a ring
float glow = 0.02 / max(0.001, abs(d));       // 1/d falloff = neon bloom

Four one-liners, four completely different looks, from the same distance. Note the argument order in smoothstep(0.01, -0.01, d) — putting the larger edge first inverts the ramp, which is the idiomatic way to fade inward without writing 1.0 -.

The glow line is worth dwelling on. Dividing a small constant by distance produces the soft, blown-out neon falloff that defines this entire visual genre, and it costs one division. The max() guard is not optional: division by zero in GLSL yields infinity, which propagates into your colour and produces white pixels or driver-specific garbage.

Two more primitives and you can build almost anything:

// Repetition: tile space infinitely for free
vec2 tiled = fract(uv * 4.0) - 0.5;

// Rotation
vec2 rotate(vec2 p, float a) {
    float s = sin(a), c = cos(a);
    return mat2(c, -s, s, c) * p;
}

// Kaleidoscope: fold the angle into a wedge
float wedge = 6.2831853 / 8.0;
float a = mod(atan(uv.y, uv.x), wedge);
a = abs(a - wedge * 0.5);
vec2 folded = vec2(cos(a), sin(a)) * length(uv);

Colour: cosine palettes

Hand-picked RGB triples look flat and rarely transition well. Iñigo Quílez's cosine palette formula gives you an infinite family of smooth, harmonious gradients from twelve numbers:

vec3 palette(float t, vec3 a, vec3 b, vec3 c, vec3 d) {
    return a + b * cos(6.28318 * (c * t + d));
}

// Usage: t is any 0..1 value — radius, angle, time, or a spectrum sample
vec3 color = palette(radius + iTime * 0.1,
                     vec3(0.5), vec3(0.5),
                     vec3(1.0, 1.0, 0.5), vec3(0.8, 0.9, 0.3));

Because the palette is periodic, animating t loops seamlessly forever — no fade-out, no discontinuity. HSV is the other reliable option and is easier to reason about when you want "the same colour, brighter":

vec3 hsv(vec3 c) {
    vec4 k = vec4(1.0, 2.0 / 3.0, 1.0 / 3.0, 3.0);
    vec3 p = abs(fract(c.xxx + k.xyz) * 6.0 - k.www);
    return c.z * mix(k.xxx, clamp(p - k.xxx, 0.0, 1.0), c.y);
}

The audio side: what an FFT gives you

Now the interesting half. The Web Audio API's AnalyserNode runs an FFT over a sliding window and hands you the result as bytes.

const audioContext = new AudioContext();
const stream = await navigator.mediaDevices.getUserMedia({
  audio: { echoCancellation: false, noiseSuppression: false, autoGainControl: false }
});
const source = audioContext.createMediaStreamSource(stream);

const analyser = audioContext.createAnalyser();
analyser.fftSize = 2048;                 // → 1024 frequency bins
analyser.smoothingTimeConstant = 0.38;   // light temporal smoothing
source.connect(analyser);                // note: NOT connected to destination

const frequencyData = new Uint8Array(analyser.frequencyBinCount);  // 1024
const timeDomainData = new Uint8Array(analyser.fftSize);           // 2048

Several things in that snippet are deliberate.

Disable the microphone processing. Echo cancellation, noise suppression, and automatic gain control exist to make speech intelligible. Applied to music they are actively destructive — AGC in particular will fight your visuals, compressing exactly the dynamics you are trying to show.

Do not connect the analyser to the destination when the source is a microphone, or you will create a feedback loop through the speakers. An analyser is a passive tap; it does not need to be in the output path.

fftSize = 2048 is a trade. At a 48 kHz sample rate you get 1024 bins covering 0–24 kHz, so each bin is about 23 Hz wide, and the analysis window is ~43 ms. Larger FFTs give finer frequency resolution but sluggish response; smaller ones react instantly but cannot tell a kick from a bass note. For visuals, 2048 is the sweet spot.

smoothingTimeConstant is not your main smoothing. A little here (0.3–0.4) removes FFT noise. Push it to 0.8 and everything turns to mush. Do the expressive smoothing yourself, later, where you can make it asymmetric.

For an audio file rather than a microphone, the only change is the source node — and here you do connect to the destination, because the user expects to hear it:

const element = document.querySelector('#audioPlayer');
const source = audioContext.createMediaElementSource(element);
source.connect(analyser);
source.connect(audioContext.destination);   // the user wants to hear this one

Splitting into bands that mean something

1024 raw bins are not directly useful. What a visual wants is a handful of numbers that correspond to things a listener perceives: the kick, the body, the vocal range, the shimmer.

const BAND_RANGES = {
  bass:   [20, 180],
  lowMid: [180, 700],
  mid:    [700, 2500],
  treble: [2500, 12000]
};

Converting a frequency range to bin indices needs the Nyquist frequency, which depends on the actual sample rate — never hard-code it:

const nyquist = audioContext.sampleRate / 2;
const binHz = nyquist / frequencyData.length;

const band = (low, high) => {
  const start = Math.max(0, Math.floor(low / binHz));
  const end = Math.min(frequencyData.length - 1, Math.ceil(high / binHz));
  let sum = 0;
  let weightSum = 0;
  for (let index = start; index <= end; index += 1) {
    const hz = Math.max(binHz, index * binHz);
    const weight = 1 / Math.sqrt(Math.max(1, hz / 80));   // perceptual tilt
    const value = frequencyData[index] / 255;
    sum += value * value * weight;                         // RMS, not mean
    weightSum += weight;
  }
  return weightSum ? Math.sqrt(sum / weightSum) : 0;
};

Two choices in there do the heavy lifting.

RMS instead of an average. Squaring before summing and taking the square root at the end weights loud bins more than quiet ones. A plain mean lets a wide, mostly-empty band dilute a genuine peak into nothing.

The perceptual tilt. Real music has far more energy at low frequencies than high — spectra roughly follow a downward slope. Without compensation, bass dominates every measurement and treble never moves. The 1 / sqrt(hz / 80) weight is a cheap approximation of that tilt: it does not attempt to be a true equal-loudness contour, it just stops the low end from swamping everything.

Adaptive normalization (the part nobody tells you)

Here is where most tutorials stop and most visualizers disappoint. You now have band values between 0 and 1 — but a phone microphone across a room in a quiet flat might only ever produce values between 0.01 and 0.08. Map that straight to brightness and your visual barely moves. Point the same code at a loud club and everything is pinned at maximum.

The fix is to track a running floor and ceiling per band and normalize against them, so the visual always uses its full range regardless of input level:

const bandState = {
  bass:   { floor: 0.015, peak: 0.24 },
  lowMid: { floor: 0.012, peak: 0.20 },
  mid:    { floor: 0.010, peak: 0.18 },
  treble: { floor: 0.008, peak: 0.16 }
};

const normalize = (name, value) => {
  const current = bandState[name];
  // Floor drops quickly toward a new minimum, recovers slowly
  current.floor = lerp(current.floor, Math.min(value, current.floor),
                       value < current.floor ? 0.08 : 0.006);
  // Peak rises quickly to a new maximum, then decays gently
  current.peak = lerp(current.peak, Math.max(value, current.peak * 0.985),
                      value > current.peak ? 0.12 : 0.004);
  return clamp((value - current.floor) / Math.max(0.035, current.peak - current.floor));
};

The asymmetric rates are the whole trick. The envelope expands fast and contracts slow: a sudden loud passage widens the range immediately so nothing clips, while a quiet passage takes many seconds to narrow it, so the visual does not "pump" between songs. The peak * 0.985 term makes the ceiling sag continuously, so a one-off spike does not permanently deaden the response. And Math.max(0.035, ...) guards the denominator — in silence, floor and peak converge, and without that clamp you would divide by nearly zero and produce violent flicker.

Then smooth for display, asymmetrically — fast attack so beats land crisply, slow release so nothing strobes:

function lerp(a, b, amount) {
  return a + (b - a) * amount;
}

function smoothFeature(previous, next, attack = 0.46, release = 0.18) {
  return lerp(previous, next, next > previous ? attack : release);
}

features = {
  bass:   smoothFeature(features.bass,   bass,   0.58, 0.20),
  mid:    smoothFeature(features.mid,    mid,    0.46, 0.18),
  treble: smoothFeature(features.treble, treble, 0.70, 0.30),
  volume: smoothFeature(features.volume, volume, 0.54, 0.18),
  energy: smoothFeature(features.energy, energy, 0.50, 0.18)
};

Treble gets a faster release than bass because hi-hats are short and a lingering shimmer looks wrong; bass gets a slower one because a kick's body decays gradually.

Beat detection

Full beat tracking is a research problem. For visuals you do not need it — you need "something just hit," and a rolling comparison gets you most of the way:

bassHistory.push(rawBass);
if (bassHistory.length > 42) bassHistory.shift();          // ~0.7s at 60fps
const averageBass = bassHistory.reduce((sum, v) => sum + v, 0) / bassHistory.length;

const beat = rawBass > averageBass * 1.22          // well above recent average
  && normalizedBass > 0.52                        // loud in absolute terms
  && (rawBass - previousBass > 0.035              // and rising sharply
      || rawVolume - previousVolume > 0.025)
  && now - lastBeatAt > 135;                      // refractory period

if (beat) lastBeatAt = now;

Four conditions, four false-positive classes eliminated. The ratio test finds local prominence. The absolute threshold stops quiet passages from generating phantom beats — without it, near-silence produces constant triggers because everything is above a near-zero average. The rising-edge test catches the transient rather than the sustain. And the 135 ms refractory period caps detection at roughly 440 BPM, which stops one kick from registering three times across consecutive frames.

Rather than a boolean, expose a decaying float — a shader wants a value it can multiply, not a flag it has to branch on:

features.beat = beat ? 1 : features.beat * 0.78;   // sharp attack, exponential tail

Bridging audio into GLSL

Scalars go across as uniforms, once per frame:

gl.uniform1f(uniforms.uBass, features.bass);
gl.uniform1f(uniforms.uMid, features.mid);
gl.uniform1f(uniforms.uTreble, features.treble);
gl.uniform1f(uniforms.uVolume, features.volume);
gl.uniform1f(uniforms.uEnergy, features.energy);
gl.uniform1f(uniforms.uBeat, features.beat);
gl.uniform1f(uniforms.iTime, elapsedSeconds);
gl.uniform2f(uniforms.u_resolution, canvas.width, canvas.height);

Resolve every getUniformLocation once after linking and cache them. It is a string lookup into the driver, and calling it per frame per uniform is pure waste.

One subtlety about time. Do not derive it from a frame counter, and do not use raw wall-clock time either. Accumulate elapsed time yourself so you can scale it — that is what lets you speed the whole scene up with the music, or slow it down for reduced-motion users:

const deltaSeconds = Math.min(0.05, Math.max(0, (now - lastRenderAt) / 1000));
lastRenderAt = now;
const motionScale = reducedMotion ? 0.18 : 1;
travel += deltaSeconds * motionScale * (1.15 + features.bass * 1.35 + features.energy * 0.55);

Because travel only ever increases and is driven by the music, the shader gets a clock that literally runs faster during loud passages. Passing this as a separate uniform from iTime means an effect can use steady time for some elements and music-driven time for others.

The audio texture

Six scalars cannot express "draw a bar for each frequency." For that the shader needs the whole spectrum, and the way to hand a GPU a thousand numbers is a texture.

The layout: a 1024 × 2 single-channel texture. Row 0 is the frequency spectrum, row 1 is the raw waveform.

const audioTextureData = new Uint8Array(2048);
audioTextureData.fill(128, 1024);          // waveform silence sits at mid-grey

const audioTexture = gl.createTexture();
gl.bindTexture(gl.TEXTURE_2D, audioTexture);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
gl.texImage2D(gl.TEXTURE_2D, 0, gl.R8, 1024, 2, 0, gl.RED, gl.UNSIGNED_BYTE, audioTextureData);

Then update it once per frame — with texSubImage2D, which overwrites the existing allocation rather than making a new one:

analyser.getByteFrequencyData(frequencyData);
analyser.getByteTimeDomainData(timeDomainData);
audioTextureData.set(frequencyData.subarray(0, 1024), 0);      // row 0
audioTextureData.set(timeDomainData.subarray(0, 1024), 1024);  // row 1

gl.bindTexture(gl.TEXTURE_2D, audioTexture);
gl.texSubImage2D(gl.TEXTURE_2D, 0, 0, 0, 1024, 2, gl.RED, gl.UNSIGNED_BYTE, audioTextureData);

R8 is one byte per texel — 2 KB per frame, which is nothing. LINEAR filtering means the GPU interpolates between bins for free, so sampling at an arbitrary position gives you a smooth curve rather than a staircase.

Inside GLSL, wrap the lookups in two helpers so shader authors never think about rows:

uniform sampler2D uAudioTexture;

// 0..1 across the spectrum, returns 0..1 magnitude
float spectrum(float x) {
    return texture(uAudioTexture, vec2(clamp(x, 0.0, 1.0), 0.25)).r;
}

// 0..1 across the waveform, returns -1..1 amplitude
float waveform(float x) {
    return texture(uAudioTexture, vec2(clamp(x, 0.0, 1.0), 0.75)).r * 2.0 - 1.0;
}

The 0.25 and 0.75 are texel centres for a two-row texture — sampling at 0.0 or 0.5 would land on a boundary and blend the two rows together, mixing spectrum into waveform. The * 2.0 - 1.0 maps the unsigned byte back to a signed amplitude, since the time-domain data centres silence at 128.

Now an oscilloscope is three lines:

float w = waveform(uv01.x);
float d = abs(uv.y - w * 0.5);
color += vec3(0.2, 0.9, 1.0) * (0.01 / max(0.001, d));

A complete audio-reactive shader

This is a real shader from the Vibro Music library — a starter scene, small enough to read in one go, that uses every signal we have built.

Audio Rings — the full fragment body

// r = u_resolution, t = u_time, FC = gl_FragCoord.xy
vec2 uv = (FC * 2.0 - r) / min(r.x, r.y);
float radius = length(uv);
float angle = atan(uv.y, uv.x);

// Bass widens the ring spacing; volume drives how fast they travel outward
float pulse = sin(radius * (18.0 + uBass * 22.0) - t * (3.0 + uVolume * 5.0));

// Treble adds spokes — floor() makes the count snap rather than smear
float spokes = cos(angle * (8.0 + floor(uTreble * 10.0)) + t * (1.2 + uMid * 3.0));

// The ring itself, kicked outward on every beat
float ring = smoothstep(0.08, 0.0, abs(pulse * 0.035 + radius - 0.38 - uBeat * 0.08));

// Hue rotates with angle, drifts with time, shifts with overall energy
vec3 color = hsv(vec3(fract(angle / 6.2831853 + t * 0.05 + uEnergy * 0.2), 0.72, 0.96));

color *= ring * (1.2 + uBeat) + pow(max(spokes, 0.0), 6.0) * (0.16 + uTreble * 0.5);
color += vec3(0.02, 0.03, 0.06) / max(0.08, radius);   // centre glow

o = vec4(color, 1.0);

Read it as a mapping table, because that is what an audio-reactive shader really is:

Which musical signal drives which visual parameter, and why that pairing works.

The general principle: map bass to geometry, treble to detail, beat to punctuation, and energy to colour. Drive several parameters from one signal and the result looks like a single object being inflated. Drive different parameters from different bands and it looks like it is listening.

Taste, safety, and fallbacks

Never let a parameter reach zero or infinity

Write 18.0 + uBass * 22.0, not uBass * 40.0. The constant is the resting state — the scene should look composed in silence and intensify with music, not collapse. Every reactive term should be an addition to a sensible baseline.

Clamp the flashing

Rapid full-screen luminance changes can trigger photosensitive seizures. On a beat-reactive full-screen visual this is a genuine hazard, not a theoretical one. Limit per-frame brightness change, and default the limiter to on — make users opt into intensity rather than opt out of harm.

Honour reduced motion at the source

Scale the time increment rather than freezing the shader. A still image is not a good substitute for an animation; slow, gentle movement is. Watch for runtime changes to the preference too.

Have something to show when there is no audio

Microphone permission gets denied. Autoplay gets blocked. Have an idle animation ready, driven by a slow sine so the fallback still breathes:

const idle = 0.1 + Math.sin(now * 0.0016) * 0.055;
features = {
  bass:   smoothFeature(features.bass,   idle,        0.18, 0.08),
  mid:    smoothFeature(features.mid,    idle * 0.78, 0.16, 0.08),
  treble: smoothFeature(features.treble, idle * 0.62, 0.14, 0.08),
  volume: smoothFeature(features.volume, idle,        0.18, 0.08),
  energy: smoothFeature(features.energy, idle,        0.18, 0.08),
  beat:   features.beat * 0.86
};

Feeding the idle values through the same smoothing functions means there is no visible seam when real audio arrives — the visual simply wakes up.

Remember the browser's rules

An AudioContext starts suspended until a user gesture resumes it, and getUserMedia requires both a secure context and an explicit permission prompt. Start both from a real click, and handle denial as a normal path rather than an error state. Cap your device pixel ratio while you are at it — a full-screen fragment shader at 3× DPR is nine times the work of 1× for a difference nobody can see on a moving image.