-
MusicRL: Aligning Music Generation to Human Preferences
Authors:
Geoffrey Cideron,
Sertan Girgin,
Mauro Verzetti,
Damien Vincent,
Matej Kastelic,
Zalán Borsos,
Brian McWilliams,
Victor Ungureanu,
Olivier Bachem,
Olivier Pietquin,
Matthieu Geist,
Léonard Hussenot,
Neil Zeghidour,
Andrea Agostinelli
Abstract:
We propose MusicRL, the first music generation system finetuned from human feedback. Appreciation of text-to-music models is particularly subjective since the concept of musicality as well as the specific intention behind a caption are user-dependent (e.g. a caption such as "upbeat work-out music" can map to a retro guitar solo or a techno pop beat). Not only this makes supervised training of such…
▽ More
We propose MusicRL, the first music generation system finetuned from human feedback. Appreciation of text-to-music models is particularly subjective since the concept of musicality as well as the specific intention behind a caption are user-dependent (e.g. a caption such as "upbeat work-out music" can map to a retro guitar solo or a techno pop beat). Not only this makes supervised training of such models challenging, but it also calls for integrating continuous human feedback in their post-deployment finetuning. MusicRL is a pretrained autoregressive MusicLM (Agostinelli et al., 2023) model of discrete audio tokens finetuned with reinforcement learning to maximise sequence-level rewards. We design reward functions related specifically to text-adherence and audio quality with the help from selected raters, and use those to finetune MusicLM into MusicRL-R. We deploy MusicLM to users and collect a substantial dataset comprising 300,000 pairwise preferences. Using Reinforcement Learning from Human Feedback (RLHF), we train MusicRL-U, the first text-to-music model that incorporates human feedback at scale. Human evaluations show that both MusicRL-R and MusicRL-U are preferred to the baseline. Ultimately, MusicRL-RU combines the two approaches and results in the best model according to human raters. Ablation studies shed light on the musical attributes influencing human preferences, indicating that text adherence and quality only account for a part of it. This underscores the prevalence of subjectivity in musical appreciation and calls for further involvement of human listeners in the finetuning of music generation models.
△ Less
Submitted 6 February, 2024;
originally announced February 2024.
-
Fluid of fused spheres as a model for protein solution
Authors:
M. Kastelic,
Yu. V. Kalyuzhnyi,
V. Vlachy
Abstract:
In this work we examine thermodynamics of fluid with "molecules" represented by two fused hard spheres, decorated by the attractive square-well sites. Interactions between these sites are of short-range and cause association between the fused-sphere particles. The model can be used to study the non-spherical (or dimerized) proteins in solution. Thermodynamic quantities of the system are calculated…
▽ More
In this work we examine thermodynamics of fluid with "molecules" represented by two fused hard spheres, decorated by the attractive square-well sites. Interactions between these sites are of short-range and cause association between the fused-sphere particles. The model can be used to study the non-spherical (or dimerized) proteins in solution. Thermodynamic quantities of the system are calculated using a modification of Wertheim's thermodynamic perturbation theory and the results compared with new Monte Carlo simulations under isobaric-isothermal conditions. In particular, we are interested in the liquid-liquid phase separation in such systems. The model fluid serves to evaluate the effect of the shape of the molecules, changing from spherical to more elongated (two fused spheres) ones. The results indicate that the effect of the non-spherical shape is to reduce the critical density and temperature. This finding is consistent with experimental observations for the antibodies of non-spherical shape.
△ Less
Submitted 23 March, 2016;
originally announced March 2016.
-
Salt-specific effects in lysozyme solutions
Authors:
T. Janc,
M. Kastelic,
M. Boncina,
V. Vlachy
Abstract:
The effects of additions of low-molecular-mass salts on the properties of aqueous lysozyme solutions are examined by using the cloud-point temperature, $T_{cloud}$, measurements. Mixtures of protein, buffer, and simple salt in water are studied at pH=6.8 (phosphate buffer) and pH=4.6 (acetate buffer). We show that an addition of buffer in the amount above $I_{buffer} = 0.6$ mol dm$^{-3}$ does not…
▽ More
The effects of additions of low-molecular-mass salts on the properties of aqueous lysozyme solutions are examined by using the cloud-point temperature, $T_{cloud}$, measurements. Mixtures of protein, buffer, and simple salt in water are studied at pH=6.8 (phosphate buffer) and pH=4.6 (acetate buffer). We show that an addition of buffer in the amount above $I_{buffer} = 0.6$ mol dm$^{-3}$ does not affect the $T_{cloud}$ values. However, by replacing a certain amount of the buffer electrolyte by another salt, keeping the total ionic strength constant, we can significantly change the cloud-point temperature. All the salts de-stabilize the solution and the magnitude of the effect depends on the nature of the salt. Experimental results are analyzed within the framework of the one-component model, which treats the protein-protein interaction as highly directional and of short-range. We use this approach to predict the second virial coefficients, and liquid-liquid phase diagrams under conditions, where $T_{cloud}$ is determined experimentally.
△ Less
Submitted 23 March, 2016;
originally announced March 2016.
