Kyutai lança o MuScriptor: um Transformer somente-decodificador de pesos abertos para transcrição de música multi-instrumental para MIDI
O MuScriptor é um Transformer somente-decodificador de pesos abertos, desenvolvido pela Kyutai e pela Mirelo. Treinado com 170 mil gravações reais mais 1,45 milhão de MIDIs sintéticos, ele transcreve mixagens completas de múltiplos instrumentos em MIDI. O artigo aborda seu pipeline de três estágios, comparações de desempenho com o YourMT3+, o condicionamento por instrumento e a configuração, além de uma demonstração explicativa interativa. O post Kyutai lança o MuScriptor: um Transformer somente-decodificador de pesos abertos para transcrição de música multi-instrumental para MIDI apareceu primeiro no MarkTechPost.
Automatic Music Transcription (AMT) converts an audio recording into symbolic notes, usually MIDI. Single-instrument transcription already works reasonably well. However, transcribing a full multi-instrument mix stays difficult. Kyutai and Mirelo team now release MuScriptor to close that gap. It is an open-weight model trained on real, multi-instrument recordings across many genres.
This article explains how MuScriptor works, what the benchmarks show, and how to run it.
What is MuScriptor?
At its core, MuScriptor is a decoder-only Transformer for music transcription. First, it reads a mel-spectrogram of a short audio segment. Then it autoregressively predicts MIDI-like tokens for pitch, timing, and instrument. In effect, transcription becomes a language-modeling task, following the MT3 tokenization scheme.
The release ships three weight variants on Hugging Face. Their sizes are small (103M), medium (307M, default), and large (1.4B). The inference code uses the MIT license. The weights use CC BY-NC 4.0, so commercial use is restricted.
How the Three-Stage Pipeline Works
MuScriptor’s main idea is data, not architecture. Accordingly, training moves through three stages, and each builds on the last.
• Pre-training uses DSynth, roughly 1.45M MIDI files. An on-the-fly pipeline synthesizes them during training. Augmentations include pitch shifting, tempo changes, velocity adjustment, and instrument randomization. Over 250 soundfonts plus random detuning yield near-infinite audio realizations.
• Fine-tuning uses DReal, an internal set of 170,000 recordings. Together they total more than 11,000 hours with aligned note annotations. Most alignments come from audio-symbolic synchronization using interpolation and dynamic time warping. Poor pairs are filtered by warping distance and a maximum time-dilation factor.
• Reinforcement learning post-training uses DRL, 300 manually verified tracks. The team applies a GRPO-like method combining REINFORCE with group-relative advantage normalization. The reward sums three F-scores: onset, frame, and offset. As a result, the model learns to favor cleaner transcriptions.
Transcribe
True positive (detected)
False negative (missed)
False positive (wrong)
Onset F1
Multi F1
Note pattern is illustrative. F1 scores are real, from the MuScriptor paper (1.3B model on DTest).
Built by Marktechpost
(function(){
var SVG="http://www.w3.org/2000/svg";
var roll=document.getElementById("roll");
var W=720,H=300,PADL=64,PADR=14,PADT=14,PADB=26;
var DUR=10; // seconds shown
{name:"Drums",inst:"drums",y:0},
{name:"E-Bass",inst:"e_bass",y:1},
{name:"Dist. E-Gtr",inst:"distorted_e_guitar",y:2},
{name:"Ac. Piano",inst:"acoustic_piano",y:3},
{name:"Voice",inst:"voice",y:4},
{name:"Strings",inst:"string_ensemble",y:5}
var NR=rows.length;
var rowH=(H-PADT-PADB)/NR;
function tx(t){return PADL+(t/DUR)*(W-PADL-PADR);}
function ry(r){return PADT+r*rowH;}
// Illustrative ground-truth notes: [row, start, dur, rank(0=easy..1=hard)]
[0,0.3,0.18,0.1],[0,0.8,0.18,0.15],[0,1.3,0.18,0.2],[0,1.8,0.18,0.1],[0,2.4,0.18,0.25],[0,3.0,0.18,0.2],[0,3.6,0.18,0.3],[0,4.2,0.18,0.2],[0,4.9,0.18,0.4],[0,5.5,0.18,0.3],[0,6.2,0.18,0.5],[0,6.9,0.18,0.4],[0,7.6,0.18,0.6],[0,8.3,0.18,0.5],[0,9.0,0.18,0.7],
[1,0.4,0.8,0.2],[1,1.6,0.8,0.25],[1,3.0,0.9,0.35],[1,4.4,0.8,0.4],[1,5.8,0.9,0.5],[1,7.2,0.9,0.6],[1,8.6,0.9,0.7],
[2,1.0,1.1,0.45],[2,2.6,1.0,0.55],[2,4.2,1.2,0.6],[2,6.0,1.1,0.7],[2,7.8,1.1,0.8],
[3,0.6,0.7,0.3],[3,1.7,0.6,0.35],[3,2.9,0.7,0.4],[3,4.1,0.6,0.5],[3,5.3,0.7,0.55],[3,6.6,0.6,0.65],[3,8.0,0.8,0.75],
[4,2.0,1.4,0.5],[4,4.0,1.5,0.6],[4,6.3,1.4,0.72],[4,8.4,1.2,0.85],
[5,1.2,1.8,0.55],[5,3.6,1.9,0.68],[5,6.0,1.9,0.8],[5,8.2,1.6,0.9]
// Illustrative false positives available: [row,start,dur,rank]
[0,2.1,0.16,0.9],[0,5.9,0.16,0.7],[0,7.9,0.16,0.85],
[1,2.4,0.6,0.8],[1,6.6,0.7,0.9],
[2,3.7,0.9,0.85],[2,8.9,0.9,0.95],
[3,3.5,0.5,0.9],[3,7.3,0.6,0.8],
[4,3.5,0.9,0.9],[4,7.7,0.8,0.95],
[5,5.1,1.2,0.88],[5,9.2,0.9,0.97]
// Real numbers from the paper (1.3B, DTest). recall/fpRate are illustrative thresholds.
var STAGES={
synth:{onset:34.5,multi:16.2,onBase:26.1,recall:0.42,fp:0.85,
note:"Trained only on synthetic MIDI. It finds coarse pitch activity, but misses many onsets and mislabels instruments."},
real:{onset:54.4,multi:41.6,onBase:52.5,recall:0.78,fp:0.35,
note:"Fine-tuning on 170k real recordings lifts every metric by roughly 20 points over synthetic-only training."},
rl:{onset:60.4,multi:48.2,onBase:60.4,recall:0.90,fp:0.12,
note:"GRPO-style RL post-training on 300 verified tracks reduces false negatives and sharpens onset timing."}
var current="rl", cond=true, playing=false, playhead=0, raf=null, streamed={};
function selectedRows(){
if(!cond) return rows.map(function(r){return r.y;});
// conditioning example: focus on the core rhythm section + piano
return [0,1,2,3];
function classify(stage){
var s=STAGES[stage];
var tp=[],fn=[],fp=[];
var sel=selectedRows();
GT.forEach(function(n){
if(cond && sel.indexOf(n[0])===-1) return; // hidden by conditioning
var boost=cond?0.08:0; // conditioning nudges recall up (illustrative)
if(n[3] <= s.recall+boost) tp.push(n); else fn.push(n);
FPS.forEach(function(n){
if(cond && sel.indexOf(n[0])===-1) return;
var fpr=cond?s.fp*0.7:s.fp; // conditioning trims spurious notes
if(n[3] > (1-fpr)) fp.push(n);
return {tp:tp,fn:fn,fp:fp};
function el(tag,attrs){var e=document.createElementNS(SVG,tag);for(var k in attrs)e.setAttribute(k,attrs[k]);return e;}
function drawGrid(){
while(roll.firstChild) roll.removeChild(roll.firstChild);
var sel=selectedRows();
for(var r=0;r var active=sel.indexOf(r)!==-1; roll.appendChild(el("rect",{x:PADL,y:ry(r),width:W-PADL-PADR,height:rowH, fill:(r%2? "#FAFBFD":"#FFFFFF"),opacity:active?1:0.4})); roll.appendChild(el("line",{x1:PADL,y1:ry(r),x2:W-PADR,y2:ry(r),stroke:"#EFF1F5","stroke-width":1})); var lbl=el("text",{x:PADL-8,y:ry(r)+rowH/2+4,"text-anchor":"end","font-size":11, fill:active?"#14161C":"#B6BCC7","font-weight":active?600:500,"font-family":"Inter,sans-serif"}); lbl.textContent=rows[r].name; roll.appendChild(lbl); // time gridlines every 2s (segment markers) for(var t=0;t<=DUR;t+=2){ roll.appendChild(el("line",{x1:tx(t),y1:PADT,x2:tx(t),y2:H-PADB,stroke:"#E7E9EE","stroke-width":1,"stroke-dasharray":"2 4"})); var tl=el("text",{x:tx(t),y:H-9,"text-anchor":"middle","font-size":10,fill:"#9AA1AE","font-family":"Inter,sans-serif"}); tl.textContent=t+"s"; roll.appendChild(tl); roll.appendChild(el("line",{x1:PADL,y1:H-PADB,x2:W-PADR,y2:H-PADB,stroke:"#E7E9EE","stroke-width":1})); var noteLayer,headLine; function render(reveal){ drawGrid(); var c=classify(current); noteLayer=el("g",{}); roll.appendChild(noteLayer); function bar(n,color,opacity){ var x=tx(n[1]),w=Math.max(6,tx(n[1]+n[2])-tx(n[1])); var y=ry(n[0])+rowH*0.28,h=rowH*0.44; var vis=(!reveal)|| (n[1]<=playhead); var rect=el("rect",{x:x,y:y,width:w,height:h,rx:3,fill:color,opacity:vis?opacity:0}); rect.style.transition="opacity .18s"; noteLayer.appendChild(rect); c.fn.forEach(function(n){bar(n,"#16A34A",0.55);}); c.fp.forEach(function(n){bar(n,"#DC2626",0.8);}); c.tp.forEach(function(n){bar(n,"#2563EB",0.95);}); headLine=el("line",{x1:tx(playhead),y1:PADT,x2:tx(playhead),y2:H-PADB,stroke:"#5B9A00","stroke-width":2,opacity:reveal?0.9:0}); roll.appendChild(headLine); postSize(); function setMetrics(){ var s=STAGES[current]; animateNum("mOnset",s.onset); animateNum("mMulti",s.multi); document.getElementById("bOnset").textContent="baseline YourMT3+ 32.5"; document.getElementById("bMulti").textContent="baseline YourMT3+ 21.9"; document.getElementById("barOnset").style.width=s.onset+"%"; document.getElementById("barMulti").style.width=(s.multi/60*100)+"%"; document.getElementById("stageNote").textContent=s.note; function animateNum(id,to){ var e=document.getElementById(id),from=parseFloat(e.textContent)||0,st=null,dur=550; function step(ts){if(!st)st=ts;var p=Math.min(1,(ts-st)/dur); e.textContent=(from+(to-from)*(1-Math.pow(1-p,3))).toFixed(1); if(p<1)requestAnimationFrame(step);} requestAnimationFrame(step); // event stream var stream=document.getElementById("stream"); function resetStream(){stream.innerHTML='
function pushEvent(n,kind){
var pitchNames=["C2","G2","E3","C4","A4","D5"];
var line=document.createElement("div");line.className="ln";
var p=pitchNames[n[0]]||"C4";var inst=rows[n[0]].inst;
if(kind==="start"){
line.innerHTML='NoteStart t='+n[1].toFixed(2)+'s pitch='+p+' inst='+inst+'';
line.innerHTML='NoteEnd t='+(n[1]+n[2]).toFixed(2)+'s ('+inst+')';
stream.appendChild(line);stream.scrollTop=stream.scrollHeight;
function play(){
if(playing)return; playing=true;
var btn=document.getElementById("playBtn");btn.disabled=true;btn.textContent="● Transcribing…";
playhead=0; resetStream();
var c=classify(current);
var visible=c.tp.concat(c.fp).sort(function(a,b){return a[1]-b[1];});
render(true);
var t0=null,SPEED=DUR/4200; // ms mapping
function frame(ts){
if(!t0)t0=ts;
playhead=Math.min(DUR,(ts-t0)*SPEED);
// reveal notes + emit events
visible.forEach(function(n,i){
var key="s"+i;
if(n[1]<=playhead && !streamed[key]){streamed[key]=1;
var rects=noteLayer.querySelectorAll("rect");
pushEvent(n,"start");
setTimeout((function(nn){return function(){pushEvent(nn,"end");};})(n),120);
// update opacities
Array.prototype.forEach.call(noteLayer.querySelectorAll("rect"),function(r){});
render(true);
if(playhead else{playing=false;btn.disabled=false;btn.textContent=" Transcribe again"; headLine.setAttribute("opacity","0");postSize();} raf=requestAnimationFrame(frame); // controls document.getElementById("stages").addEventListener("click",function(e){ var b=e.target.closest("button");if(!b)return; Array.prototype.forEach.call(this.children,function(c){c.classList.remove("on");}); b.classList.add("on");current=b.getAttribute("data-s"); playhead=DUR;render(false);setMetrics();postSize(); document.getElementById("condToggle").addEventListener("click",function(){ cond=!cond;this.classList.toggle("on",cond); playhead=DUR;render(false);postSize(); document.getElementById("playBtn").addEventListener("click",play); // pipeline shimmer var pnodes=document.querySelectorAll("#pipe .node"),pi=0; setInterval(function(){ pnodes.forEach(function(n){n.classList.remove("hot");}); pnodes[pi].classList.add("hot");pi=(pi+1)%pnodes.length; // resize to parent (WordPress embed) function postSize(){ try{var h=document.body.offsetHeight+40; window.parent.postMessage({muscriptorHeight:h},"*");}catch(e){} window.addEventListener("resize",function(){render(false);postSize();}); playhead=DUR;render(false);setMetrics();setTimeout(postSize,60);setTimeout(postSize,400); (function(){ window.addEventListener("message",function(e){ if(e.data && e.data.muscriptorHeight){ var f=document.getElementById("muscriptor-embed"); if(f) f.style.height = e.data.muscriptorHeight + "px"; Performance For evaluation, the research team use DTest, 372 held-out tracks with accurate annotations. They report instrument-agnostic metrics from the mir_eval library. Among them, Multi F1 is strictest, since it also requires the correct instrument. The table below traces each training stage against the YourMT3+ baseline, using the large (~1.3B) model. Model (DTest)Onset F1Frame F1Offset F1Drums F1Multi F1YourMT3+ (baseline)32.545.517.841.421.9MuScriptor · DSynth34.548.916.121.016.2MuScriptor · DSynth + DReal54.469.342.343.341.6MuScriptor · DSynth + DReal + DRL60.473.349.050.248.2 Clearly, every stage improves results, and real data matters most. Synthetic-only training reaches competitive frame F1 but weak onset and multi scores. Adding DReal then lifts all metrics by roughly 20 points. Finally, RL post-training reduces false negatives and sharpens onset timing. Cross-dataset tests point the same way. For example, frame F1 on Dagstuhl ChoirSet rises from 51.0 to 80.7. Even so, onset and offset stay lower on hard styles like chorals. Getting Started Installation takes one command, and inference streams note events directly. Copy CodeCopiedUse a different Browser # pip install muscriptor (or: uv add muscriptor) from pathlib import Path from muscriptor import TranscriptionModel # Downloads the default "medium" variant (also accepts "small" / "large") model = TranscriptionModel.load_model() # Stream note events; optionally condition on known instruments for event in model.transcribe("audio.wav", instruments=["acoustic_piano", "drums"]): print(event) # NoteStartEvent / NoteEndEvent / ProgressEvent # Or write a MIDI file directly Path("out.mid").write_bytes(model.transcribe_to_midi("audio.wav")) For the released models, keep cfg_coef at 1, since they are already RL post-trained. Additionally, uvx muscriptor serve launches a browser web UI with a live piano roll. Use Cases with Examples Because the output is standard MIDI, many workflows open up: • Producers can extract a MIDI bassline from a mix, then re-voice it in a DAW. • Musicologists can convert historical recordings into editable scores for analysis. • MIR researchers can feed transcriptions into chord or key recognition systems. • Educators can build practice tools showing a live piano roll during playback. • Developers can transcribe only drums by passing instrument conditioning. Strengths and Weaknesses • Trained on 170k real recordings spanning classical to heavy metal. • Open weights plus MIT-licensed inference code, in three size variants. • Multi F1 of 48.2 versus 21.9 for the YourMT3+ baseline on DTest. • Instrument conditioning customizes output and stabilizes cross-segment predictions. • A streaming API emits note events and MIDI, alongside a browser web UI. Weaknesses: • Weights are CC BY-NC 4.0, so commercial deployment is restricted. • The tokenizer drops velocity and cannot represent overlapping same-pitch, same-instrument notes. • Onset and offset accuracy stay lower on chorals and similar styles. • The large model wants a GPU for practical speed. • The 5-second segment size limits long-range context and inference speed. Check out the Paper, GitHub Repo and Model Weights. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Kyutai Releases MuScriptor: An Open-Weight Decoder-Only Transformer for Multi-Instrument Music Transcription to MIDI appeared first on MarkTechPost.