Designing Voices Using 8 Perceptual Axes — Blending Speaker Embeddings Along the Semantic Axis

📝 Originally published (in Japanese) at forge.workstyle.tech.

Bridging the Gap: Continuous Voice Adjustment with Perceptual Sliders

When working with voice conversion models or TTS systems, you'll inevitably hit this wall: the assumption that the target voice must be specified by providing a full reference audio sample. But what users actually want isn't this discrete swap of voices.

"I want my voice to sound a bit younger and slightly huskier."

A reference-swap operation can't express that "a bit" or "slightly." While we need continuous knobs to finely tune voice characteristics, most models only accept discrete inputs like "someone else's voice."

This article introduces a general technique to bridge that gap. We'll explore how to convert human-understandable perceptual axes (younger↔older, clear↔harsh, etc.) into a target voice vector by blending multiple "anchor speakers" using weighted interpolation. This approach isn't tied to a specific product implementation—it's a framework that can be applied to any model that accepts speaker embeddings.

Prerequisite: Speaker Embeddings as "Voice Coordinates"

Modern voice conversion and TTS models typically represent speaker identity as a fixed-length vector called a speaker embedding. For example, a 192-dimensional vector might represent one person's voice.

Empirically, speaker embeddings exhibit two useful properties:

1. Proximity: Similar vectors produce similar voices
2. Additivity: The midpoint between two vectors produces a voice intermediate between the two

While this linearity isn't strictly guaranteed, blending nearby vectors often yields natural intermediate voices. This property forms the foundation for our blending technique.

The process involves two steps:

1. Slider values → weights for each anchor speaker (distributed via softmax)
2. Weights × anchor embeddings → blended embedding (weighted sum)

Anchors are a pre-defined set of "reference voices." We use clean recordings of multiple speakers as anchors. The more diverse your anchors, the broader the voice space you can reach through blending.

Step 0: Grounding Perceptual Axes in Measurable Acoustic Features

First, we define the axes we'll present to users. Here are eight we'll use:

• age (age perception)
• gender (gender perception)
• pitch (pitch level)
• build (body type perception)
• huskiness (huskiness level)
• clarity (clarity level)
• warmth (warmth level)
• roughness (roughness level)

The key insight is to define these subjective axes as linear combinations of measurable acoustic features. For each anchor recording, we extract features like F0 (fundamental frequency), formants, HNR (harmonic-to-noise ratio), spectral tilt, shimmer, and jitter. Then we represent each axis as a weighted sum of these features.

AXES = (
    Axis("age",       ..., APPROX,    {"f0": -1.0, "formant": -0.5}),
    Axis("gender",    ..., MEASURED,  {"f0": 1.0, "formant": 1.0}),
    Axis("pitch",     ..., MEASURED,  {"f0": 1.0}),
    Axis("build",     ..., MEASURED,  {"formant": 1.0}),
    Axis("huskiness", ..., MEASURED,  {"shimmer": 1.0, "hnr": -0.5}),
    Axis("clarity",   ..., MEASURED,  {"hnr": 1.0}),
    Axis("warmth",    ..., MEASURED,  {"tilt": -1.0}),
    Axis("roughness", ..., MEASURED,  {"jitter": 1.0, "shimmer": 0.5}),
)

The interpretation follows standard acoustics:

• Higher F0 = higher pitch
• Higher formants = "thinner" voice (associated with smaller body type)
• Higher shimmer + lower HNR = huskier voice
• Higher jitter + shimmer = rougher voice

Since F0 and formants are perceived logarithmically, we use log2 values.

By defining axes as formulas from features, we can automatically calculate any new voice's position along these axes when we add it as a new anchor—no lookup tables or case-by-case if statements needed. This consistent mapping from features is crucial.

Step 0.5: Axes Have Three Calibration States

Not all axes can be measured with equal reliability. Each axis has one of three states:

• measured — Cleanly defined by measurable acoustic attributes (pitch, gender, clarity, etc.)
• approx — Approximated from measurements (e.g., "age" is approximated by "low F0 and low formants = older," which doesn't fully capture age-related voice changes)
• needs_label — Requires human labeling (e.g., axes like "cute" that lack objective metrics)

This distinction matters because we shouldn't use uncalibrated axes. If an axis lacks labels, it's silently excluded from weight calculations. Faking "plausible" metrics for uncalibrated axes harms overall system reliability. In implementation, we maintain a set of active_axes containing only calibrated (measured/approx) axes or those with labels, and perform calculations only on this set.

Step 1: Slider Values → Anchor Weights

This is the core of the technique. When a user moves a slider, we get 0-1 values for each axis. We convert these to a target point in our voice space, then assign weights to each anchor based on how close they are to this target.

First, we z-normalize each anchor's axis values across the anchor population (mean=0, std=1). We store the population's mean and standard deviation so new voices can be evaluated on the same scale.

def design_weights(slider_values, bank, temp=0.3, spread=2.5):
    d2 = np.zeros(n)
    for key in bank.active_axes:
        s = clip(slider_values[key], 0, 1)
        target = (2.0 * s - 1.0) * spread    # Slider → target z-score
        d2 += (bank.axis_z[key] - target) ** 2
    d2 /= len(bank.active_axes)
    w = np.exp(-d2 / temp)                    # Closer anchors get higher weight
    return w / w.sum()                        # Softmax normalization

This does three things:

1. Slider → target z-score: The slider's center (0.5) maps to z=0, with endpoints at ±spread (default ±2.5σ). We use a wider spread to prevent slider "sticking" at extremes and allow more tolerance for extreme voices.

2. Distance aggregation: For all active axes, we sum squared differences between the anchor's z-coordinate and target z, then average across axes.

3. Softmax weighting: Anchors closer to the target get higher weights. A lower temp (default 0.3) concentrates weight on the single closest anchor, while a higher temp distributes weight more evenly across multiple anchors.

The key innovation is using softmax instead of argmin (nearest neighbor). This allows smooth interpolation even when no anchor perfectly matches the target, blending multiple anchors to achieve the desired voice characteristics.

Step 2: Weights → Blended Embedding

Once we have weights, blending the embeddings is straightforward:

def design_embedding(slider_values, bank, top_k=3, **kw):
    w = design_weights(slider_values, bank, **kw)
    emb = (w[:, None] * bank.embeddings).sum(axis=0)   # (192,) blended embedding
    order = np.argsort(w)[::-1][:top_k]                 # Top-k anchors by weight
    top = [bank.names[i] for i in order]
    return emb.astype(np.float32), top, w

This returns three things:

• Blended embedding (the voice vector to pass to your model)
• Top-k anchor names (anchors with highest weights, useful for reference prompts)
• Weight vector (for explainability and debugging)

Beyond just blending embeddings, being able to say "this voice is primarily a blend of anchors A and B" improves product explainability. Instead of a black-box vector, we present the voice as a "known combination of voices."

Bonus: Inverse Transform to Read "Slider Position" from Any Voice

By inverting our z-normalization mapping, we can analyze any voice (uploaded recording) to restore its slider position:

pct = (z / spread + 1.0) / 2.0 * 100.0   # z-score → slider 0..100%

This is the exact inverse of the "slider → target z" mapping in design_weights. If a user records their voice, we can reflect its position across our axes in the sliders, then allow fine-tuning from there. Adding this recording as a new anchor (after rebuilding the bank's statistics) expands the voice space itself. The ability to use the same mapping in both directions is a secondary benefit of defining axes as formulas.

Pitfalls & Learnings

• Additivity only holds "locally": Extreme blending of distant voices often produces unnatural results due to embedding space nonlinearities. Avoid over-mixing unrelated anchors by not setting temperature too high in softmax.

• z-normalization statistics are tied to the population: Always recalculate mean/std when adding new anchors. Evaluating new voices against old statistics shifts their axis positions.

• Don't fake unmeasurable axes: The needs_label state prevents us from pretending we can measure subjective axes. Sacrificing accuracy for "plausible" implementations harms long-term reliability.

• Define axes as feature mappings: Avoid hardcoded if-statements or lookup tables. Defining axes as linear combinations of acoustic features enables consistent forward mapping (design), inverse mapping (analysis), and anchor addition.

Summary

• Instead of swapping reference voices discretely, use perceptual sliders to continuously design voices
• Define each axis as a linear combination of measurable acoustic features (F0, formants, HNR, shimmer, jitter, etc.), then z-normalize across the anchor population
• Map slider values to target z-scores → calculate squared distances to anchors → convert to weights via softmax → blend speaker embeddings to get the target voice vector
• Each axis has one of three states: measured/approx/needs_label. Only calibrated axes participate in matching
• The same mapping can be inverted to restore slider positions from any voice, and adding new anchors expands the voice space