PRISM-V · A Multidimensional Evaluation of Pretrained Neural Vocoders for Speech Synthesis

        P · Perceptual
        R · Reconstruction
        I · Intelligibility
        S · Speaker
        M · Model Efficiency
      

A Multidimensional Evaluation of Pretrained Neural Vocoders for Speech Synthesis · Ranked by PRISM-V Score

The PRISM-V Leaderboard evaluates open-source and pretrained neural vocoders on English speech across diverse acoustic conditions, generative architectures, and edge deployment profiles.

Overview & Benchmark Motivation

Traditional neural vocoder evaluations predominantly report PESQ and MCD exclusively on clean, single-speaker studio recordings (e.g., LJSpeech). In production speech synthesis and voice conversion, however, practitioners deploy models on diverse voices, accented speech, and real-world noisy audio across hardware constrained to sub-realtime latencies.

PRISM-V establishes a comprehensive, zero-shot, training-free benchmark evaluating 15 pretrained vocoder architectures across four English acoustic corpora and five core dimensions: Perceptual, Reconstruction, Intelligibility, Speaker, and Model Efficiency.

Pretrained Models

GAN, Flow, SDE, Fourier

Acoustic Corpora

Studio, Multi, Accent, Noisy

PRISM Dimensions

Holistic deployment scoring

Max Throughput

370×

RT on Edge profiling

Reproducibility

100%

Open data & checkpoints

🎯

Studio PESQ Mismatch

Models scoring >4.4 PESQ on clean LJSpeech (e.g. BigVGAN-v2, PeriodWave) show variable degradation on mobile microphone noise (Free_ST) and accented speech (VCTK). Cross-corpus evaluation is required for true generalization.

⚡

The Flow-Matching Frontier

Modern flow matching models (Flow2GAN, WaveFM) and Brownian Bridge SDEs (BridgeVoC) compress diffusion generation into 1 to 4 steps, matching 30-step diffusion fidelity at 12–35× real-time throughput.

📱

Edge Deployment Realities

Fourier / iSTFT ConvNeXt models (Vocos) and pseudo-inverse GANs (FreeV) achieve >300× throughput with under 400 MB peak VRAM, unlocking viable on-device streaming neural synthesis on edge hardware.

Explore the Benchmark

Navigate directly to interactive components to inspect empirical metrics, listen to audio samples, or configure custom model rankings.

Open Neural Vocoder Analysis

🏆 Rank & PRISM-V (1–100)

📐 Objective Metrics (Blue)

👂 Perceptual / Subjective (Violet)

⚡ Efficiency & Edge (Amber)

🛡️ Robustness / Per-Corpus (Green)

✅ Code Open-Source

⭐ Pareto Optimal

            📊 Benchmarked Checkpoints 
          
            Click any column header to sort ↑↓

💻 Code ✅ Training & inference code publicly available

📦 Checkpoint ✅ Pretrained weights freely downloadable

⭐ Pareto Non-dominated Pareto frontier on quality & speed

📊 Cross-Corpus Robustness & Generalization Analysis

Evaluates degradation across clean studio (LJSpeech), multi-speaker narrative (LibriTTS), regional accents (VCTK), and mobile noisy recordings (Free_ST).

Metric: PESQ ↑ STOI ↑ LSD ↓ MCD ↓

View: Heatmap Grouped Bar

Cross-Corpus Numerical Breakdown Table

Model	Corpus	PESQ ↑	STOI ↑	LSD (dB) ↓	MCD (dB) ↓

⚡ Throughput, Latency & GPU Memory Footprint

All efficiency metrics measured under an Edge Profiling protocol using generator-only batch-size 1 FP32 inference with synchronous CUDA timing.

Full Computational Profile Table

Model	Architecture	RTF ↓	xRT ↑	Peak VRAM ↓	Params (M)	Edge Ready	Code OS	Ckpt OS

🧠 Architecture-Aware Taxonomy & Comparison

Examine generative paradigms: GAN, Flow Matching, Fourier / Transformer, Diffusion SDE, and Algorithmic baselines.

Model	Category	Family	Params	Track	Year	License	PESQ	UTMOS	RTF	xRT	VRAM	PRISM-V Score

🔬 Montreal Forced Aligner Phoneme Diagnostics

Decomposes reconstruction error across standard ARPAbet phonetic classes: Vowels, Stops, Fricatives, Affricates, Nasals, Liquids, and Glides.

Full Phoneme Diagnostics Data Table

Model	Phonetic Class	LSD (dB) ↓	F0 Error (¢)	Boundary Error (dB)

📈 Multi-Objective Quality vs. Efficiency Pareto Frontier

Models located on the dashed Pareto frontier ⭐ are non-dominated checkpoints simultaneously optimizing acoustic fidelity and execution speed.

X-axis (Cost):

Y-axis (Metric):

Log X-axis Highlight Pareto Frontier

Deployment Recommendations Guide

🖥️ Deployment Target	✅ Recommended Checkpoints	💡 Engineering Rationale
Edge / Mobile / Embedded	Vocos, FreeV, ComVo-Base	RTF < 0.015 (66×+ real-time), VRAM < 650 MB. Fits smartphone GPU & NPU envelopes.
Mid-Range GPU Server	Flow2GAN, ComVo-Large	SOTA fidelity (PESQ > 4.4) with 12–19× real-time throughput. Ideal for interactive TTS streaming APIs.
High-End Cloud / Studio Offline	BigVGAN-v2 (112M), PeriodWave	Uncompromised acoustic reconstruction quality, anti-aliased snake activations, fine spectral envelope detail.

🧪 Benchmark Protocol & Metric Definitions

Comprehensive empirical benchmarking specification, dataset partitions, and hardware profiling details.

Figure 1: PRISM-V Evaluation Taxonomy — Refracting neural vocoder synthesis across the 5 core PRISM dimensions: Perceptual, Reconstruction, Intelligibility, Speaker, and Model Efficiency.

Design Principles

PRISM-V is a strictly training-free, zero-shot evaluation of publicly released pretrained checkpoints.

Models are evaluated exactly as practitioners deploy them: no fine-tuning, no cherry-picked test sets, and fixed deterministic seeds.

Inference Protocol

• Batch size: 1 (realistic streaming inference)

• Precision: FP32 with generator-only timing (excludes frontend mel extraction)

• Hardware Profiling: Warmup iterations + measured loops with CUDA synchronization fences

Evaluation Corpora

Corpus	Acoustic Condition	Characteristics
LJSpeech	Clean studio recording	Single female speaker, professional recording booth, near-zero reverberation.
LibriTTS	Audiobook narrative	Multi-speaker (test-clean + test-other), expressive reading variations.
VCTK	Regional accented speech	109 speakers with British, Scottish, Irish, and Commonwealth accents.
Free_ST	Mobile microphone noisy	Real-world acoustic environments, mobile devices, background reverberation.

📐 Metric Definitions & Qualitative Interpretations

Metric	Formal Designation	Direction	Technical Description & Range

➕ Submit / Request a Pretrained Vocoder

We actively welcome vocoder authors and speech researchers to submit new pretrained neural vocoders to PRISM-V.

1. Public Checkpoint

GitHub release or project page accessible without authentication.

2. Open Inference Code

Open-source license (MIT, Apache-2.0, BSD, CC-BY) with clean pip/PyTorch installation.

3. Deterministic Mel Adapter

Compatible with the standard PRISM-V BaseVocoderAdapter interface.

PRISM-V Adapter Interface Example:

from benchmark.adapters.base import BaseVocoderAdapter
import torch

class MyVocoderAdapter(BaseVocoderAdapter):
    def load(self, device, precision):
        self.model = load_my_model(self.checkpoint_path).to(device).eval()

    def preprocess(self, waveform: torch.Tensor, sr: int) -> torch.Tensor:
        return my_mel_extractor(waveform, sr)

    @torch.no_grad()
    def infer(self, condition: torch.Tensor, seed: int) -> torch.Tensor:
        return self.model(condition)

Ready to submit? Submit a Pull Request with your adapter and preflight smoke test. Open PR on GitHub →

📜 Changelog

Platform release history, version notes, and updates.

Version History

Version	Release Date	Key Updates & Notes
v1.0.1	September 12, 2026	Static deployment engine for GitHub Pages (zero CPU quota exhaustion), high-contrast badges, interactive A/B audio explorer with sync waveforms, and custom weighting calculator.
v1.0.0	September 10, 2026	Initial public benchmark release: 15 pretrained neural vocoders evaluated across 4 English corpora.
v1.1.0	Upcoming	Additional vocoder families (CodecGAN, DAC-based) and streaming inference latency profiles.

📙 Cite PRISM-V

If you use the PRISM-V benchmark, evaluated checkpoints, or codebase in your scientific research, please cite:

@misc{purohit2026prismv,
  author       = {Ravindrakumar M. Purohit and Hemant A. Patil},
  title        = {{PRISM-V}: A Multidimensional Evaluation of Pretrained Neural Vocoders for Speech Synthesis},
  year         = {2026},
  howpublished = {\url{https://iamshreeji-copy2.github.io/open_vocoder_leaderboard/}},
  note         = {Open neural vocoder evaluation leaderboard}
}