Distributional quality (R1) is measured with Fréchet Audio Distance (FAD) using the
AudioLDM-eval implementation, computed per class and pooled. Because the shared ATA protocol is
reference-conditioned rather than text-driven, identity (R2) and diversity (R3) use
ImageBind audio embeddings: an audio-text metric such as CLAP would mainly reflect agreement
with the class label rather than preservation of the specific reference event.
Alignment. With $\hat{z}^{ref}_r$ and $\hat{z}^{gen}_{r,v}$ the $\ell_2$-normalised
embeddings of reference $r$ and variant $v$, alignment is the cosine similarity:
$$ A_{r,v} = \mathrm{cos\_sim}\!\left(\hat{z}^{ref}_r,\hat{z}^{gen}_{r,v}\right)
= \frac{\langle \hat{z}^{ref}_r, \hat{z}^{gen}_{r,v}\rangle}
{\lVert \hat{z}^{ref}_r\rVert_2\,\lVert \hat{z}^{gen}_{r,v}\rVert_2}. $$
Diversity. Following Seeing and Hearing, diversity is the mean pairwise
semantic distance among the $V_r=10$ variants of a reference:
$$ D_r = \frac{2}{V_r(V_r-1)} \sum_{1\le i\lt j\le V_r}
\left(1 - \mathrm{cos\_sim}\!\left(\hat{z}^{gen}_{r,i},\hat{z}^{gen}_{r,j}\right)\right). $$
High diversity must be read jointly with FAD and alignment, since a large spread can also indicate semantic
drift. ImageBind alignment measures reference–variant similarity, while ImageBind diversity measures variation
among generations from the same reference; temporal behaviour is evaluated separately via signal-level onset
diagnostics below.
From the log-mel spectrogram $\ell_{m,t}$ we keep only positive temporal increases, sum over mel bins, and
normalise to $[0,1]$ to obtain an onset-strength envelope $\tilde{e}_t$ focused on transient shape rather than
loudness:
$$ D_{m,t} = \max(\ell_{m,t+1}-\ell_{m,t},\,0), \qquad e_t = \sum_m D_{m,t}, \qquad
\tilde{e}_t = \frac{e_t - \min_t e_t}{\max_t(e_t-\min_t e_t)+\epsilon}. $$
Peaks are detected with a 40 ms minimum spacing and prominence $\rho=0.10$, yielding three diagnostics.
FWHM ratio — transient width preservation (→1 ideal; >1 smeared, <1 sharpened):
$$ \mathrm{FWHM}(p)=\omega_p\,\Delta_{\mathrm{ms}}, \qquad
R_{\mathrm{FWHM}}=\frac{\mathrm{FWHM}_{\mathrm{gen}}}{\mathrm{FWHM}_{\mathrm{ref}}+\epsilon}. $$
Pre-onset Δ — energy before vs. after the transient (40 ms vs. 80 ms windows; →0 ideal, positive = pre-echo):
$$ R_{\mathrm{pre}}(p)=\frac{\sum_{t=\max(0,p-q_{\mathrm{pre}})}^{p-1}\tilde{e}_t}
{\sum_{t=p}^{\min(T,p+q_{\mathrm{post}})-1}\tilde{e}_t+\epsilon}, \qquad
\Delta_{\mathrm{pre}}=R_{\mathrm{pre,gen}}-R_{\mathrm{pre,ref}}. $$
Onset error — median timing gap between matched reference and generated onsets (tolerance $\delta_{\max}=150$ ms):
$$ j^*(i)=\arg\min_{j\in U}\bigl|\tau^{gen}_j-\tau^{ref}_i\bigr|, \qquad
E_{\mathrm{onset}}=1000\cdot\mathrm{median}_i\bigl|\tau^{gen}_{j^*(i)}-\tau^{ref}_i\bigr|. $$
A Similarity Mean Opinion Score (S-MOS) was rated by 15 participants on a 1–5
identity scale over reference–variation pairs drawn from the 4,000 outputs per method. Participants used
headphones in a quiet environment, could replay each pair freely, and rated 100 randomised trials
per method. Trials were anonymised and randomised; scores are reported with 95% confidence intervals. The prompt
was: "For each trial: listen to the Reference, then the Candidate. Rate the identity fidelity (1–5), defined
as the similarity to the reference event/source (excluding loudness)."
All baselines are initialised from released pretrained checkpoints and adapted to ESC-50 with deliberately
lightweight fine-tuning, keeping encoders frozen when applicable and adapting only task-relevant
generation/conditioning layers. The ESC-50 test fold has 50 classes × 8 references; for each reference,
$N=10$ variants are generated (4,000 outputs/model).
- Latent diffusion (AudioLDM, ThinkSound): text/audio encoders frozen; only the diffusion
backbone (UNet/MMDiT) and conditioning projection layers fine-tuned. ThinkSound: 10 epochs × 150 steps;
AudioLDM: 200 steps. Updates kept low to limit overfitting and drift.
- Waveform diffusion (T-Foley): reduced from 500 to 25 epochs × 250 steps (6,250 steps);
class/MLP embeddings and FiLM layers updated; operates at 22 kHz (native rate preserved).
- AudioX: 20 epochs × 200 steps (4,000 steps) via stable-audio-tools. Fixed 11 s window —
each 5 s clip zero-padded with a padding-mask loss; conditioning uses only the class name (audio/video
modalities empty).
- A²SB: two runs matched to pretrained masking splits (0.0–0.5 s, 0.5–1.0 s) plus
extended masked-duration fine-tuning (short 0.3–1.0 s, long 0.3–2.0 s). STFT features with
n_fft=2048, hop=512, 44.1 kHz.
- Sampling rates: native rates preserved during fine-tuning (ThinkSound 44 kHz, T-Foley
22 kHz, A²SB 44.1 kHz); resampling to 16 kHz only at evaluation when needed for fair comparison.
