TASTE: Taxonomy-Aware Symbolic Piano Generation from Textual Description

Selected Demos

Abstract

Text-conditioned symbolic music generation lets users steer editable music with language, yet two questions remain underexplored: what a conditioning description should contain, and how data for such descriptions should be collected and used for training. We address both questions with TASTE, a taxonomy-aware framework for symbolic music generation with main focus on piano. First, we introduce a taxonomy that separates finite-vocabulary tags (key, tempo, emotion, genre, accompaniment, and performance style), a section plan for temporal form, and a free-form imagery description of the scene or atmosphere a piece evokes. Each part of a description can then be annotated, learned, and evaluated on its own terms. Second, we build a multi-tier annotation pipeline that draws each condition from suitable evidence and propagates annotations from 10K directly paired and 51K curated public performances to 371K weakly labeled MIDI files. Finally, we design a staged training recipe that moves through these tiers from broad weak supervision to direct pairs, and instantiate it in both an autoregressive decoder and a latent diffusion transformer. Both objective and subjective evaluations demonstrate substantial improvement on condition alignment at almost all fields over caption-conditioned baselines while maintaining competitive music quality. We will release the annotated datasets.

Overview

Taxonomy

Taxonomy: sources of musical intention, the representations used as controls (tags, section plan, imagery description), and the evaluation target for each.

TASTE splits a description into three parts that play different roles: six finite-vocabulary tags (key, tempo, emotion, genre, accompaniment, and performance style), a section plan that fixes the piece's form over time, and a free-form imagery description of the scene or atmosphere the music evokes. Each part is annotated, learned, and evaluated on its own terms.

Data pipeline

Multi-tier annotation pipeline: reference annotations on paired and curated pieces, then field-specific methods for tags, section plans, and imagery descriptions extend them to weakly labeled public MIDI.

A multi-tier pipeline draws each condition from the evidence best suited to it. About 10K directly paired pieces and 51K curated public performances, annotated from compositional context or rendered audio, anchor the labels; confidence-gated tag probes and retrieved imagery descriptions then extend them to 371K weakly labeled MIDI files. Section plans come from structure analysis at every tier.

Models

Model architectures: an autoregressive event decoder and a latent diffusion transformer, each conditioned on tags through adaptive layer normalization and on the imagery description through cross-attention.

The same staged recipe, moving from MIDI pretraining through the weakly labeled, curated, and directly paired tiers, trains two generators: an autoregressive decoder that writes its own section plan before the notes, and a latent diffusion transformer that generates a whole piece under a given plan. Tags enter through adaptive layer normalization and the imagery description through cross-attention over a frozen text encoder.

Release

TASTE pairs a taxonomy of music description (six tags, a section plan, and an imagery description) with a three-tier annotation pipeline and two model families. We release the annotated data.

TierPiecesSourceReleased
High-quality9,949Collected and annotated internallyMIDI + labels
Curated51,130Aria-MIDI, PIAST-YT, Pop2Piano, PiJAMA, POP909, EMOPIAMIDI + labels
Large public371,053Aria-MIDILabels only

Every piece carries key, tempo, emotion, genre, accompaniment, performance, an imagery description, and a timed section plan.

TASTE (AR) is a 1.235B autoregressive model (a 0.89B pretrained decoder with cross-attention over the imagery description), trained Pretrain → Large public → Curated → High-quality.
TASTE (DiT) is a 397.6M latent diffusion transformer over a window-VAE codec.