PRISM-V · A Multidimensional Evaluation of Pretrained Neural Vocoders for Speech Synthesis

        P · Perceptual
        R · Reconstruction
        I · Intelligibility
        S · Speaker
        M · Model Efficiency
      

Benchmark Overview

Pretrained Models

GAN, Flow, SDE, Fourier

Acoustic Corpora

Studio, Multi, Accent, Noisy

PRISM Dimensions

Holistic deployment scoring

Max Throughput

370×

RT on Edge profiling

Reproducibility

100%

Open data & checkpoints

📦 Pretrained Vocoder Checkpoints & Configurations

Evaluated checkpoints, source repositories, training configurations, and inference protocols in PRISM-V.

14 Neural Vocoders Evaluated

ID	Model	Checkpoint	GitHub	Architecture	Training Data	Sampling Rate	Mel Bins	Hop Size	Inference
Baseline	Griffin–Lim	—	💻 Code	Iterative phase reconstruction	—	24 kHz	—	256	32 iterations
M1	RNDVoC	🤗 Hugging Face	💻 Code	Range–null T–F network	LibriTTS	24 kHz	100	256	1 pass
M2	Flow2GAN	🤗 Hugging Face	💻 Code	Flow matching and GAN	LibriTTS	24 kHz	100	256	4 NFEs
M3	Vocos	🤗 Hugging Face	💻 Code	ConvNeXt iSTFT GAN	LibriTTS	24 kHz	100	256	1 pass
M4	BridgeVoC	🤗 Hugging Face	💻 Code	Schrödinger bridge	LibriTTS	24 kHz	100	256	4 NFEs
M5	PeriodWave-Turbo	📁 Google Drive	💻 Code	Adversarial flow matching	LibriTTS	24 kHz	100	256	4 NFEs
M6	ComVo-Base	🤗 Hugging Face	💻 Code	Complex-valued iSTFT GAN	LibriTTS	24 kHz	100	256	1 pass
M7	BigVGAN-v2	🤗 Hugging Face	💻 Code	AMP-based GAN	Large-scale audio	24 kHz	100	256	1 pass
M8	BigVGAN-Base	🤗 Hugging Face	💻 Code	AMP-based GAN	LibriTTS	24 kHz	100	256	1 pass
M9	ComVo-Large	🤗 Hugging Face	💻 Code	Complex-valued iSTFT GAN	LibriTTS	24 kHz	100	256	1 pass
M10	WaveFM	💻 GitHub	💻 Code	Consistency-distilled flow matching	LibriTTS	24 kHz	100	256	1 pass
M11	HiFi-GAN Universal V1	📁 Google Drive	💻 Code	Time-domain GAN	Universal	22.05 kHz	80	256	1 pass
M12	FreeV	🤗 Hugging Face	💻 Code	Pseudo-inverse iSTFT GAN	LJSpeech	22.05 kHz	80	256	1 pass
M13	RFWave	📁 Google Drive	💻 Code	Multi-band rectified flow	LibriTTS	24 kHz	100	256	10 NFEs
M14	PeriodWave	📁 Google Drive	💻 Code	Multi-period flow matching	LibriTTS	24 kHz	100	256	16 NFEs

📊 Architectural Paradigm Analysis

Select any benchmark evaluation measure to analyze and compare performance across generative paradigms

Metric:

Quick Select:

            📊 Benchmarked Checkpoints 
          
            Click any column header to sort ↑↓

💻 Code ✅ Training & inference code publicly available

📦 Checkpoint ✅ Pretrained weights freely downloadable

🥇 🥈 🥉 Evaluation metric medalists (Gold / Silver / Bronze)

⭐ Pareto Non-dominated Pareto frontier on quality & speed

† Hardware Gated Score floored to 1.00 due to edge streaming constraint ($\text{RTF} > 1.0$ on GTX 1650 FP32 profile)

📊 Cross-Corpus Robustness & Generalization Analysis

Evaluates degradation across clean studio (LJSpeech), multi-speaker narrative (LibriTTS), regional accents (VCTK), and mobile noisy recordings (Free_ST).

Metric: PESQ ↑ STOI ↑ LSD ↓ MCD ↓

View: Heatmap Grouped Bar

Cross-Corpus Numerical Breakdown Table

ID	Model	Corpus	PESQ ↑	STOI ↑	LSD (dB) ↓	MCD (dB) ↓

⚡ Throughput, Latency & GPU Memory Footprint

All efficiency metrics measured under an Edge Profiling protocol using generator-only batch-size 1 FP32 inference with synchronous CUDA timing.

Computational Profile

ID	Model	Architecture	RTF ↓	xRT ↑	Peak VRAM ↓	Params (M)	Edge Ready	Code	Checkpoint	Paper

🧠 Architecture-Aware Taxonomy & Comparison

Examine generative paradigms: GAN, Flow Matching, Fourier / Transformer, Diffusion SDE, and Algorithmic baselines.

Category Filter:

Showing 15 of 15 models

ID	Model	Category	Family	Params	Track	Year	License	PESQ	UTMOS	RTF	xRT	VRAM	PRISM-V Score	Resources

🔬 Montreal Forced Aligner Phoneme Diagnostics

Decomposes reconstruction error across standard ARPAbet phonetic classes: Vowels, Stops, Fricatives, Affricates, Nasals, Liquids, and Glides.

Rotate / Zoom:

Batch 1/3 0°

🔬 Montreal Forced Aligner Phoneme Diagnostics Plotly Powered

Decomposes fine-grained reconstruction quality across 7 phonetic classes for all 15 architectures

Model:

Class:

ID	Model	Phonetic Class	LSD (dB) ↓	F0 Error (¢)	Boundary Error (dB)

🌲 PRISM-V Score & 95% Bootstrap Confidence Intervals

Marginal uncertainty estimates per system. Click any model to select it across the views below.

Observed PRISM-V scores are shown with 95% bootstrap confidence intervals; bootstrap mean ± SD summarizes the center and dispersion of the resampled score distribution.

* Note: Marginal/model-wise confidence intervals depict individual score uncertainty and are not equivalent to paired pairwise significance tests (evaluated via paired bootstrap in the 15×15 matrix below).

⬇ Download Summary (CSV)

Rank	Model	Observed PRISM-V	Bootstrap μ ± σ	Mean − Observed (Δ)	95% Bootstrap CI	Replicates (N)

🗺️ Statistical Significance Matrix

Signed difference in PRISM-V score (Row − Column). Diverging color scale centered at 0. Click any cell to inspect details in the Pairwise Explorer.

📥 Download 105 Comparisons (CSV)

💡

Proposed approach achieves statistically robust separation across 98 of 105 unique model pairs (excluding self-comparisons and duplicate symmetric pairs from total possible comparisons).

Blue = Row model higher (A > B) Red = Row model lower (A < B) ns Grey (ns) = Not statistically significant (p > 0.05) - "-" (Diagonal) = Self-comparison

🔍 Model-vs-Model Pairwise Comparison Explorer

Select any two models or choose a featured contrast to inspect paired effect size, 95% CI, and Holm-corrected p-value.

Featured Contrasts (Curated Key Pairs):

System A (Reference Model):

System B (Comparison Model):

A higher -->

← System B Higher Zero Difference (Δ = 0) System A Higher →

📊 Cross-Corpus Consistency Profile

Utility score $B$ across 4 evaluation corpora. Shows consistency and reveals worst-corpus bottleneck.

Model 1:

Model 2:

Model 3:

🎲 Bootstrap Uncertainty Distribution

Empirical bootstrap draw distribution across models.

Primary:

Comparison:

* Distribution visualization uses 1,000 stored bootstrap draws; reported CIs and p-values use the complete 10,000-replicate analysis.

📖 Statistical protocol

▼

PRISM-V uncertainty was estimated with 10,000 paired bootstrap replicates. LJSpeech was resampled at the utterance level, while Free_ST, LibriTTS, and VCTK used speaker-cluster resampling. Identical sampled units were applied across systems within each replicate. All C(15,2)=105 system pairs formed one multiple-comparison family, with Holm-Bonferroni correction at α=0.05. Pairwise results report PRISM-V score differences with paired 95% bootstrap confidence intervals.

Speaker-Cluster Resampling: Applied to Free_ST (10 speakers), LibriTTS (682 speakers), and VCTK (109 speakers). Preserves inter-speaker acoustic correlation structure.

Holm-Bonferroni Family-Wise Control: Step-down adjustment over sorted p-values $p_{(1)} \le \dots \le p_{(m)}$ testing $H_{(k)}$ at $\alpha / (m - k + 1)$, with $m = 105$. Controls FWER at 0.05 without conservatism of standard Bonferroni.

➕ Submit a Vocoder Model to the Leaderboard

We warmly welcome researchers and speech synthesis teams to submit neural vocoders to PRISM-V. Follow the open-source GitHub workflow below to fork the repository into your personal account, implement the adapter, and open an upstream Pull Request.

🍴 1. Fork to Your Account 🔀 2. Submit Pull Request 📦 View Repository

Submission Criteria

1. Public Model Checkpoints

Weights hosted publicly on Hugging Face Hub, GitHub Releases, or Zenodo accessible without proprietary credentials.

2. Open Inference Code

Open-source license (MIT, Apache-2.0, BSD, CC-BY) with reproducible PyTorch dependencies via standard pip install.

3. Standardized Mel Adapter

Conforming to the PRISM-V BaseVocoderAdapter interface (80-channel log-mel spectrogram input).

Community Contribution Workflow

1 Fork the Repository to Your GitHub Account

🍴 Fork on GitHub →

Click the button above or visit github.com/iamshreeji-copy2/open_vocoder_leaderboard/fork. GitHub will open the fork creation prompt under your personal profile (github.com/<your-username>/open_vocoder_leaderboard).

2 Clone Your Fork & Create a Branch

Pull your personal fork to your local workstation and create a new feature branch for your vocoder:

# 1. Clone your personal fork (replace <YOUR-USERNAME> with your GitHub username)
git clone https://github.com/<YOUR-USERNAME>/open_vocoder_leaderboard.git
cd open_vocoder_leaderboard

# 2. Create a dedicated branch for your model
git checkout -b add-my-vocoder-model

3 Implement the BaseVocoderAdapter Interface

Create your adapter file at src/vocoder_benchmark/models/adapters/my_vocoder.py implementing the three required methods:

from benchmark.adapters.base import BaseVocoderAdapter
import torch

class MyVocoderAdapter(BaseVocoderAdapter):
    """PRISM-V Standard Model Adapter Interface"""

    def load(self, device: str = "cuda", precision: str = "fp32"):
        """Load pretrained weights and set model to evaluation mode."""
        self.model = load_my_checkpoint(self.checkpoint_path).to(device).eval()

    def preprocess(self, waveform: torch.Tensor, sr: int) -> torch.Tensor:
        """Extract 80-band log-mel spectrogram (1024 FFT, 256 hop, 80 mels)."""
        return extract_mel_spectrogram(waveform, sr)

    @torch.no_grad()
    def infer(self, condition: torch.Tensor, seed: int = 42) -> torch.Tensor:
        """Synthesize audio waveform from conditioning mel tensor."""
        return self.model(condition)

Test your adapter locally with the self-contained verification suite:

pytest tests/test_adapters.py -k my_vocoder

4 Push to Your Fork & Open Upstream Pull Request

🔀 Open Pull Request (Compare) →

Push the branch to your personal fork on GitHub:

git add src/vocoder_benchmark/models/adapters/ configs/models/
git commit -m "feat(adapter): add MyVocoder adapter and benchmark config"
git push -u origin add-my-vocoder-model

Then click Open Pull Request (Compare Forks). GitHub will automatically compare your fork's branch against iamshreeji-copy2/open_vocoder_leaderboard:main. Review your changes and click Create pull request.

🤖 Automated Benchmark Verification & Leaderboard Publishing

Once submitted, CI verifies model loading, determinism, and sample reconstruction. Our GPU cluster then evaluates all 5 PRISM-V dimensions across the 4 speech corpora, automatically publishing your model's scores to the public leaderboard.

View Open PRs →

📜 Changelog

Platform release history, version notes, and updates.

Version History

Version	Release Date	Key Updates & Notes
v1.0.0	September 10, 2026	Initial public release of the open vocoder benchmark evaluating 15 pretrained neural vocoders across 4 diverse English speech corpora.
v1.0.1	September 12, 2026	Interactive A/B audio listening explorer with synchronized waveforms, customizable multidimensional weighting calculator, standardized PRISM-V color palette, and accessibility controls.

If you use the PRISM-V benchmark, evaluated checkpoints, or codebase in your scientific research, please cite:

@misc{purohit2026prismv,
  author       = {Ravindrakumar M. Purohit and Hemant A. Patil},
  title        = {{PRISM-V}: A Multidimensional Evaluation of Pretrained Neural Vocoders for Speech Synthesis},
  year         = {2026},
  howpublished = {\url{https://iamshreeji-copy2.github.io/open_vocoder_leaderboard/}},
  note         = {Open neural vocoder evaluation leaderboard}
}