Tuesday, 2 April 2019

Mod player in Rust - part 6. Creating and publishing the crate

The module music player has progressed quite well. It now handles most of the effects and unusual storage types.
I think it is now good enough to create and publish a crate so that anyone can use the mod player functionality. In order to publish a crate there are a couple of areas I need to address;
  • documentation
  • examples
  • creating and publishing the crate

Creating documentation

I like crates that come with good documentation that make it easy to understand what the crate does and explains how to use it. I should try to do the same for my crate.
Rust comes with all the functionality for building good documentation. Typing the following command in the project folder ( the same folder the .toml) builds the html documentation from the source files and opens it in a browser.
cargo doc --open
The doc compiler looks for special comments that start with /// or //! to use for building the documentation. Any lines starting with /// are assumed to document the item ( function, enum, structure etc.) that follows the comment.
So for example the following code adds documentation for the Note structure.
/// Describes what sound sample to play and an effect (if any) that should be applied. 
pub struct Note{
    sample_number: u8,
    period: u32,            // how many clock ticks each sample is held for
    effect: Effect,
In the above example the second comment does not get included in the documentation for two reasons; it uses normal // comment and it is not public. Only public items can be documented. It would make very little sense to have documentation for items that are not visible to the library users.
The other type of comment, //! is documentation for the surrounding module. When I add the following lines to the too of my lib.rs file they get added to documentation's index page.
//! # mod_player
//! mod_player is a crate that reads and plays back mod audio files. The mod_player decodes the audio one sample pair (left,right) at a time
//! that can be streamed to an audio device or a file. 

Documentation examples

Good libraries should make it easy to find working example code. Rust lets you insert tested example code right into the documentation.
Inserting ``` after the /// in the comment section tells Rust's doc compiler to treats it like a code block. So inserting the code following comment
//! To use the library to decode the a mod file and save it to disk
//! ```rust
//! fn main() {
//!     //...
//! } 
produces the following documentation:
To use the library to decode the a mod file and save it to disk
fn main() {
The really cool thing is that the included example code can be fully tested code. So if you ever change the way the library should be used, the tests will catch any outdated documentation examples.
Running these tests could not be simpler. Typing
cargo test
on the command line will instruct cargo to run all the tests in the folder, including the code examples in the documentation.
You can also use rustdoc directly but I found it much harder to get this to work correctly when the examples have external dependencies. For now, I have decided to let cargo handle this for me.
The example compiler does its best to make it easy to create and maintain examples. It will automatically wrap the example code inside a fn main() {} block if one does not exist.


In addition to the examples in the documentation I want to include a couple of more fully fleshed out examples. The examples are placed into the examples folder.
My examples use several dependencies to illustrate how to use the library; I use hound for writing WAV files and cpal for playing to an audio device. The library itself does not care about how the audio is used so I do not want my library to have a dependency on either of these crates. This is where the [dev-dependencies] section in the toml becomes useful.
I have the following section in my toml file
cpal = "0.8.2"
hound = "3.4.0"
This tells Rust that the examples and tests have a dependency on these crates. The crate itself does not have these dependencies and these will not be picked up by any project using the crate.
To run an example I use the cargo command run
cargo run --example streaming_player
The above commmand tells cargo to go into the examples folder and compiled and run the file streaming_player.rs.
The examples replace the various versions of main.rs I used for testing the library. By turning that code into examples I have converted previous throw-away code into a useful part of the library documentation.

Publishing the crate

This too turned out to be fairly easy. First I had to create an account at crates.io. I needed to use my github account to create a crates.io account and grant it access to my github account. It was not terribly clear exactly why the access was needed and what it would be used for. I wish crates.io would have a clearer explanation of this.
I also needed to create an a access token for crates.io that my installed cargo can use to access my crates. This just involved going to my crates.io Account settings and creating a new token buy giving it a name, clicking on create and copying the resulting command line into my local session
cargo login 
my cargo can access crates.io
Now I need to package up my crate. Unsurprisingly, the command line for this is
cargo package
This builds the package, runs all the tests and checks that the .toml file has all the relevant sections filled in correctly.
One of the required fields is license, so I needed to pick the license for my crate. The spdx.org has a long list of standard open source licenses and their identifiers. I picked the MIT license because it is widely used and permits the use of the code in open source and proprietary projects.
Once the package builds it can be published. The command for this is;
cargo publish
This uploads the crate to crates.io and makes the crate visible to all.

Crate documentation

This gets the crate published but the its page looks a bit barren. The only text is the description tag from the .toml file.
Turns out that crates.io does not extract the documentation from the source code comments but expects to find a separate readme.md file. This is a bit of a pain because the top level module documentation would be ideally suited for the crate description.
The crate is available at crates.io and the source code is at github

Next Steps

There are parts of the code that have become quite messy and could do with refactoring. Before I do any such thing I need to makes sure my changes don't break the functionality so I need to add tests.
As I am nor publishing the crate I think I should add options for different playing modes and what to do when encountering unhandled effects

Sunday, 10 March 2019

Mod player in Rust - part 5. Rust modules

For this post I am going to change my the style slightly. Instead of going through all the new code I am going to focus on the bits that were more interesting or challenging.
The mod player is now able to play most of the files that I throw at if although there are still some files that sound wrong or are missing some effects. Below a clip of the player playing an old school 90s demo tune.

Rust module basics

Getting my head around Rust's module system was a bit trickier than I thought. I am so used to the C-style model where each file is a translation unit that is compiled into its own .obj file which are then linked together that anything else seemed just wrong.
My initial mistake was to think that the key word mod was used to import other modules into a Rust file. This appeared to work as long as I was using it to 'import' modules into the main.rs file. As soon as I started using it in other .rs. files I got loads of unexpected errors.
My key to understanding the module system was to realize that modules form a tree which is rooted in the main.rs file. The key word mod is used to declare other sub-modules that are used in the current module. I split my project into three modules;
  • main the root module for the starting point and orchestrating all the calls into the library
  • mod_player for the code that parses and streams the audio mod file. This module has one sub-module of it's own;
    • textout that can be used to print info about the module and the players current stat
When the file main.rs has the following statement;
mod mod_player
it tells the compiler that the root module has a sub module mod_player. The compiler will locate this by looking for a file called mod_player.rs in the root folder or by looking for a file mod.rs in the folder /mod_player.
Because text_out is a sub-module of mod_player I need to declare it inside the file mod_player.rswith the line.
pub mod textout
In this case I have added the key word pub because I want anyone using mod_player to also be able to access textout. Without the pub keyword the textout module would be a private module only visible to mod_player.
I can now use public functions and structures from mod_player and textout by specifying the path to the items I want to use. To load an audio mod file using the functionality in mod_player I use,
let song = mod_player::read_mod_file("song_name.MOD");
and to print out some info about it using the textout sub-module I can write
mod_player::textout::print_song_info( &song );
Both functions read_mod_file and print_song_info have to be declared public with a pub modifier to visible. It is not enough fro the containing modules to be public.

Extending impl in sub-modules

For my library I wanted all the printing related functionality to be in the texout module while still preserving the clear relationship between data and the methods. Ideally, I could just extend the existing impl blocks.
Fortunately Rust lets me do exactly that by declaring a new impl block and adding the new methods. The only slight complications is in how to identify the structure that the impl block refers to. There are three equivalent ways of declaring the extension;
  • declare the full path from the crate root using the crate path specifier
impl crate::mod_player::Sample{
  • declare the the relative path using super path specifier
impl super::Sample{
  • use use to bring the structure into the current module
use super::Sample;      // at top of the file
impl Sample{
The three versions do the same thing; bring the Sample from the parent module into the scope of the current module.
I ended up using the last method as it brings the Sample structure into scope everywhere within the sub-module and means I don't need to use path specifiers elsewhere. The nice thing about the using use is that one statement can bring multiple items into the module. The line
use super::{Note,Song,Effect};
brings NoteSong and Effect into the current module.

Exporting to WAV

One of the challenges with debugging an audio player is that debugging consist of listening to the playback and trying to spot anything that doesn't sound right and then trying to figure out where exactly the anomaly took place. Writing the sound out into a wav file makes it easier to work out the exact play time.
The hound library provides exactly the functionality I needed for writing out WAV files. I am still impressed by how easy it is to add crates to a project. Once I decided I wanted the hound crate it took comfortably less than half an hour to get WAV export working ( and most of that was spent on modifying my code ).
Because exporting to WAV is not a part of mod_playing functionality I put the functionality into the root module in main.rs

Searching tuple arrays

I changed the way notes were printed from periods into actual notes. That meant having a function that takes a period value and converts it into a string. My first instinct was to use the match syntax but this becomes very ugly and unnecessarily verbose.
fn note_string( period : u32 ) -> &str{
    return match val{
        113 => "B-5", 
        120 => "A-5", 
With 60 notes this becomes a very ugly function. There is also the nagging feeling that the performance of this code will be pretty suboptimal. ( Granted, performance is not really an issue in this instance but the code is representative of situations where the performance would matter so I think its worth investigating)
What I really want is a map that is statically initialized. There doesn't seem to be a Rust native way of initializing a map. There is the lazy_static crate which would let me create the a static map but I want to learn to use vanilla Rust efficiently before starting to use too many other crates.
While Rust doesn't let me declare static maps I can declare static arrays of tuples and do a binary search on it. I can declare a static array of all of the (period, note) pairs as;
static NOTE_FREQUENCY_STRINGS : [ (u32, &str ); 60 ]= [
( 57,  "B-6" ), ( 60,  "A#6" ), ( 64,  "A-6" ),( 67,  "G#6" ), ( 71,  "G-6" ), ( 76,  "F#6" ), ( 80,  "F-6" ), ( 85 , "E-6" ), ( 90,  "D#6" ), ( 95 , "D-6" ), ( 101, "C#6" ), ( 107, "C-6"), 
I can use the slice function binary_search_by to search through the tuple array.
    fn note_string( &self ) -> &str{
        let idx = NOTE_FREQUENCY_STRINGS.binary_search_by( | val | val.0.cmp( &self.period) );
        if idx.is_ok() {
            return  NOTE_FREQUENCY_STRINGS[ idx.unwrap() ].1;
        } else { "..." }
Because I am searching through tuples I need to provide the search function with my own comparator function. That is what |val|val.0.cmp( &self.period) declares. For each tuples the comparator function is called, it takes the first part and compares it to the period value. I am using the standard cmp function that returns an Ordering enum that the comparator expects.
I am finding that it is really worth spending time understanding slice and all of its methods.

Casting bug

This was the first time in a long time that I had to deal with a casting bug. One of the effects in a mod file is tone portamento which will change the currently playing tone into the target tone at the specified speed. Handling the effect consists of changing the period counter which controls time between subsequent samples. Increasing or decreasing the period moves the tone lower or higher.
The original code which worked most of the time is below. The two checks are used to constrain the period to a range that is supported by the mod format.
fn change_note( current_period : u32, change : i32 ) -> u32 {
    let mut result : u32 = ( current_period as i32 + change ) as u32;
    if result > 856 {  result = 856;  }
    if result < 113 { result = 113; }
The current_period must always a positive number but change can be negative because the period can increase or decrease. The current_period is cast to i32 to make it compatible with change and the result is cast to u32 because that is the type of current_period and also the return type. This worked perfectly for all the mod files I initially tested with but then I came across one that had weird popping sounds.
The code breaks down when change is negative and larger than current_period. So if period is 120 and change is -200 the value of ( current_period as i32 + change ) is -80. When -80 is cast to u32 the result is a very large positive number.
The simple fix was to change the code to;
fn change_note( current_period : u32, change : i32 ) -> u32 {
    let mut result = current_period as i32 + change;
    if result > 856 {  result = 856;  }
    if result < 113 { result = 113; }
    result as u32
This lets Rust decide the type of result and only does the casting at the very end. This is not really a Rust specific bug but highlights the fact that by giving you more control, the compiler also give you more ways to shoot yourself in the foot.

Next Steps

All the code so far has been uploaded into https://github.com/janiorca/articles/tree/master/mod_player-5
The next step is to improve the effects coverage as there are still some files that sound wrong ( or won't play at all ). The second step is to turn this into a crate that can be used by other projects.

Mod player in Rust - part 6. Creating and publishing the crate

The module music player has progressed quite well. It now handles most of the effects and unusual storage types. I think it is now good ...