The other day, I wanted to load two user arps into my XF8 at the same time. Sadly, they were in two different .X3A files, and whilst the Motif can load one performance, song, waveform or pattern at a time, it can only load ‘all arpeggios’ which precludes such filing on the instrument itself. (Definitely an omission that would be good to fix...)

There exists proprietary third-party software which can do this (e.g. John Melas’ waveform editor), but I don’t really want to have to keep Windows or Mac OS boxes around and fiddle with GUI software just for simple library management tasks like this. Consequently, I’ve spent a little time this afternoon unpicking the X3A file format and writing a suitable arp selection utility.

In case it’s of interest to others, I’ve popped a prototype of my tool and a couple of other simple examples in Python, together with my notes on the Motif file format at

https://gist.github.com/2883877

list.py lists all the content in a Motif-format file on stdin. Run it as

python list.py <myfile.X3A

stripsys.py filters out the system settings from an X3A file. This is useful to allow files to be distributed without leaking private data like network settings and file server passwords, but also so that loading them doesn’t overwrite the settings already in your Motif. Run it as

python stripsys.py <original.X3A >stripped.X3A

arpsel.py pulls selected arpeggios out of a series of X3A/X3G files, and assembles them into a combined X3G file. Run it as

python arpsel.py one.X3A:1,2,3 two.X3A:10 three.X3G:4 >combined.X3G

Time permitting, I intend to evolve this into a more general library management tool which can extract and combine voices, performances and so on from Motif files, including the arpeggios and waveforms they depend upon and keeping track of renumbering issues.

Is there any official Yamaha file-format documentation available, or a suitable tech contact point for third-party developers wanting to write supporting software?

Whilst I have already pulled apart the file format enough to achieve most of what I want, I’d love to tie up a few loose ends, maximise interoperability with older instruments in the Motif series, and ensure my terminology and descriptions match the upstream versions.

Cheers, Chris.

cdw - 06 June 2012 05:04 PM

[...]Whilst I have already pulled apart the file format enough to achieve most of what I want, I’d love to tie up a few loose ends, maximise interoperability with older instruments in the Motif series, and ensure my terminology and descriptions match the upstream versions.

Thanks for your post and the link to your notes on the file format. It’s encouraging to see that someone is delving into such things and is willing to share information.

The interoperability issue is one that I’ve been looking into somewhat myself. It’s convenient that the XF has backward compatibility with XS files, but of course going the other way isn’t supported by the Motifs. In trying to deal with the files, it’s frustrating that although XS ROM waveforms and arpeggios are all found in the XF, the numbering (and sometimes even the naming) is mostly different. To that end, I’ve been working with another person to develop spreadsheets that list the equivalent XS/XF numbers, and also indicate where the XF has waveforms or arps not found in the XS. If you haven’t already determined the XS/XF wave and arp numbering “translation”, would you be interested in what I have?

Wow this is really great!!! Thank you so much for sharing!!!

5pinDIN - 06 June 2012 06:17 PM

In trying to deal with the files, it’s frustrating that although XS ROM waveforms and arpeggios are all found in the XF, the numbering (and sometimes even the naming) is mostly different. To that end, I’ve been working with another person to develop spreadsheets that list the equivalent XS/XF numbers, and also indicate where the XF has waveforms or arps not found in the XS. If you haven’t already determined the XS/XF wave and arp numbering “translation”, would you be interested in what I have?

Yes please! I don’t have good facilities for handling spreadsheet files; exported in CSV format (or similar) would be much easier for me to manage if possible?

Thinking about this, the two injective mappings you’re looking for are XSArp -> XFArp and XSWaveform -> XFWaveform. You’ve probably already devised a good method for automatically constructing these, but if not, I imagine what I’d do (given access to both an XS and an XF) is to write a small script to use the (documented) SysEx interface to set up all the voices USR1:1 -> USR3:128 with eight elements using waveforms 1-8 in USR1:1, 9-16 in USR1:2 and so on. Once this is done, I’d save an X0A, import that into the XF, and bulk dump the voice data out again. This will give voices with successive elements XStoXF(1), XStoXF(2), etc., and thus the mapping is determined. (You’d probably need two lots of USR1:1 -> USR3:128 voices to cover all the waveforms in the XS.)

Presumably the arps could be tackled by the same trick.

Looking at the examples of XS and XF files I’ve seen, I don’t think it would be especially hard to cook up a conversion utility given the mappings, although I haven’t unpicked the voice data far enough to spot the waveform numbers and arpeggio numbers yet. (I’ll need to do that for the dependency tracking in any case.)

cdw - 07 June 2012 04:31 AM

Yes please! I don’t have good facilities for handling spreadsheet files; exported in CSV format (or similar) would be much easier for me to manage if possible?

Might I suggest that you look into Open Office? It’s open source freeware, and a good alternative to Microsoft Office. See http://en.wikipedia.org/wiki/OpenOffice for a description.
.

cdw -

Thinking about this, the two injective mappings you’re looking for are XSArp -> XFArp and XSWaveform -> XFWaveform.

The spreadsheets I’ll make available (in a few days) will list all of the XS waveforms and the equivalent XF ones, and all of the XS arps and their XF equivalents. There will be two sets - the first will have the XS waves/arps in ascending numerical order, the second will have the XF waves/arps in ascending order. (My initial intent is to make it easy for someone who is manually converting XS->XF and XF->XS Voices, etc.) The names, etc., of the waves/arps will also be included.
.

cdw -

Looking at the examples of XS and XF files I’ve seen, I don’t think it would be especially hard to cook up a conversion utility given the mappings, although I haven’t unpicked the voice data far enough to spot the waveform numbers and arpeggio numbers yet. (I’ll need to do that for the dependency tracking in any case.)

As you’re likely aware, it’s possible to do MIDI SysEx bulk dumps of individual Voices, Performances, etc. Since sometimes Motifator members ask how to program a particular sound, I had hoped that it would be possible to post such example files (which are quite small in comparison to an “all” file), rather than just describe what could be done.

One problem is that I have only an XS, but the dumps aren’t compatible with an XF. Besides the Model ID byte (03 for the XS, 12 for the XF), there’s the differing wave/arp numbering, and the check sum would have to be recalculated. In addition, using information from the Data Lists and a file kindly sent to me by VikasSharma, I was able to determine that the Performance dump from an XF is eight bytes longer than that of an XS. Data for each of the four Parts of the Performance in the XF dump has two additional bytes appended.

I just recently started looking into the above, and I don’t have enough free time right now to dig deeper, but I’ll post the spreadsheets as soon as possible.

Hi. I’ve updated the notes at https://gist.github.com/2883877 to include information about the data offsets for arpeggio numbers in voices, performances and mixings (pattern/song/template). These can be read to include arp dependencies when extracting voices, performances or mixings, but also written to allow the user arps to be renumbered without gaps in the resulting file. There’s a similar section on voice dependencies for performances and mixings.

5pinDIN - 07 June 2012 09:45 AM

One problem is that I have only an XS, but the dumps aren’t compatible with an XF. Besides the Model ID byte (03 for the XS, 12 for the XF), there’s the differing wave/arp numbering, and the check sum would have to be recalculated. In addition, using information from the Data Lists and a file kindly sent to me by VikasSharma, I was able to determine that the Performance dump from an XF is eight bytes longer than that of an XS. Data for each of the four Parts of the Performance in the XF dump has two additional bytes appended.

Interesting. There are similar small differences between the sizes of Entr chunks in 1.0.1 (XS) and 1.0.2 (XF) data files. Presumably in both cases, they’re bytes for the settings of new features that have been added.

With the sysex dump, we do have documentation (albeit rather condensed): does a comparison of the relevant section of the XS and XF data sheet give any clues? I haven’t had a chance to look myself yet, because I’m mostly focussing on just the native data files. (I did look at the XF sysex documentation a bit early on, in the hope there was a direct connection between sysex dump format and the content of the data blocks in the native files, but if there is, it isn’t obvious and I don’t see it yet.)

5pinDIN - 07 June 2012 09:45 AM

I just recently started looking into the above, and I don’t have enough free time right now to dig deeper, but I’ll post the spreadsheets as soon as possible.

I’ll very much look forward to seeing your results. I should have a more polished ‘librarian’ tool ready to go shortly. If it doesn’t work out too horrible, I’ll teach it to use your waveform and arp mappings to convert X3A <-> X0A where possible.

I’ve also dropped Yamaha a note to ask officially about a developer contact point.

Cheers, Chris.

PS (and off-topic):

5pinDIN - 07 June 2012 09:45 AM

Might I suggest that you look into Open Office? It’s open source freeware, and a good alternative to Microsoft Office. See http://en.wikipedia.org/wiki/OpenOffice for a description.

Yes, I’ve come across OpenOffice, but the dependencies needed to build and run it makes compiling myself a copy into a whole afternoon’s job, and the resulting giant, graphical application isn’t much fun to use and doesn’t fit at all well with the rest of my text-based environment. I count myself lucky that I don’t need to suffer these horrible GUI ‘office software’ things for anything I do!

I’ve used your excellent reverse engineering information to come up with a Python script that prints the information in a Motif file. I appreciate all the work you did. Thanks!