What is EDGE: Editable Dance Generation from Music?
EDGE is a powerful method for editable dance generation that creates realistic, physically-plausible dances while remaining faithful to arbitrary input music. It uses a transformer-based diffusion model paired with Jukebox, a strong music feature extractor, and confers powerful editing capabilities well-suited to dance.
Features of EDGE
EDGE has several features that make it stand out, including:
-
Joint-wise conditioning: Generate lower body from upper body or vice versa
-
Motion in-betweening: Dances that start and end with prespecified motions
-
Dance continuation: Dances that start with a prespecified motion
-
Arbitrarily long dances: Enforce temporal continuity between batches of multiple sequences
-
Physical plausibility: Avoids unintentional foot sliding and is trained with physical realism in mind
How does EDGE work?
EDGE uses a frozen Jukebox model to encode input music into embeddings. A conditional diffusion model learns to map the music embedding into a series of 5-second dance clips. At inference time, temporal constraints are applied to batches of multiple clips to enforce temporal consistency before stitching them into an arbitrary-length full video.
Benefits of EDGE
EDGE has several benefits, including:
-
High-quality dances: EDGE generates high-quality dances even for in-the-wild music samples
-
Powerful editing capabilities: EDGE supports arbitrary spatial and temporal constraints
-
Physical realism: EDGE learns when feet should and shouldn't slide using the Contact Consistency Loss
Frequently Asked Questions
How does EDGE compare to other methods?
EDGE is compared to recent methods Bailando and FACT, and human raters strongly prefer dances generated by EDGE.
Can EDGE generate dances of arbitrary length?
Yes, EDGE can generate dances of arbitrary length by imposing temporal constraints on batches of sequences.
Is EDGE physically plausible?
Yes, EDGE avoids unintentional foot sliding and is trained with physical realism in mind.