Decoders #14

Merged

MartinKavik merged 17 commits from decoders into main

2024-07-08 18:02:16 +00:00

MartinKavik commented

2024-07-08 18:02:07 +00:00

(Migrated from github.com)

Decoders are plugins/extensions/components adding custom functionality to FastWave, primarily extending waveform visualizer. Decoders have been implemented as Wasm components defined by WebAssembly Component Model in this PR. The Model and related libraries have been created by Bytecode Alliance, a nonprofit organization founded 2019 by Mozilla, Fastly, Intel and Red Hat with current members like Amazon, Cisco, Docker or Microsoft. The Component Model and tools for writing and using components are still very young but a major milestone has been achieved January 2024 when a stable release of WASI (WebAssembly System Interface) and the component model has been published. Most Bytecode Alliance tools are written in Rust and Rust seems to be the top priority that's why Rust components work best. Rust components are also most suitable for FastWave because both are written in Rust and the components don't need a runtime like Javascript or Python compoments. There are plans to share language-specific runtimes among components to reduce overhead and component loading time. The host system (Wasmtime by Bytecode Alliance in our case) communicates with Guests (aka Decoders in our case) through interfaces defined in *.wit files with WIT (Wasm Interface Type) language.

In practice, Rust component/decoder Wasm module size is around 27 KB, Javascript component has 10 MB and Python around 35 MB. Loading time of such components seems to correspond to the size of the component and it's much higher in debug mode. Javascript and Python components need linked WASI to work properly, Rust component can be compiled for both wasm32-unknown-unknown and wasm32-wasi` target. I also found a problem with calling host functions in Python, probably caused by a bug in a dispatch mechanism in the Componentize-py tool. That's why I would recommend to focus mainly on Rust components until support for other languages are better. Officially supported languages are currently Rust, Javascript, Python and Go. I wouldn't recommend to switch to alternative plugins systems like Extism because they often already use Wasmtime under the hood and plan to migrate to The Component Model once it's stable enough. Another advantage is that users need only tools native to the component language - e.g. users writing Javascript decoders need only tools published in NPM. Another option would be to integrate language runtimes directly into FastWave but I think it'll be costly in the long run because of maintenance and the need to add support for all languages one by one and it will be much less secure because there won't be Wasm "sandbox" enclosing the code in plugins.

There were two other implementation "surprises":

When calling component methods panics, it fails with error "Cannot start a runtime from within a runtime.". I think it's caused by running Wasmtime runtime inside Tauri runtime but it seems to be a Wasmtime problem and a workaround is to call component methods inside an std thread. It needs more investigation but it isn't a showstopper.
There is a helper function in signal_to_timeline.rs:

fn type_hint<F>(f: F) -> F
where
    F: for<'a> FnMut((u32, SignalValue<'a>)) -> (f64, SignalValue<'a>),
{
    f
}

that is used to only provide a type hint to a closure inside async function representing a Tauri command. It seems to be a very annoying Rust compiler issue already described in the main Rust repo. The error message is "Implementation of FnOnce is not general enough" pointing to a different code location.

Decoders are defined by this WIT file:

And there is a video of implemented decoders loaded through FastWave Javascript Command Interface:

(Both files are included in README)

Testing decoders are located in testing_files/components in the repo. Each component/decoder contains README describing how to create and build the particular component.

Decoders are plugins/extensions/components adding custom functionality to FastWave, primarily extending waveform visualizer. Decoders have been implemented as Wasm components defined by [WebAssembly Component Model](https://component-model.bytecodealliance.org/) in this PR. The Model and related libraries have been created by Bytecode Alliance, a nonprofit organization founded 2019 by Mozilla, Fastly, Intel and Red Hat with current members like Amazon, Cisco, Docker or Microsoft. The Component Model and tools for writing and using components are still very young but a major milestone has been achieved January 2024 when a stable release of WASI (WebAssembly System Interface) and the component model has been published. Most Bytecode Alliance tools are written in Rust and Rust seems to be the top priority that's why Rust components work best. Rust components are also most suitable for FastWave because both are written in Rust and the components don't need a runtime like Javascript or Python compoments. There are plans to share language-specific runtimes among components to reduce overhead and component loading time. The host system ([Wasmtime](https://docs.wasmtime.dev/) by Bytecode Alliance in our case) communicates with Guests (aka Decoders in our case) through interfaces defined in `*.wit` files with [WIT ](https://component-model.bytecodealliance.org/design/wit.html)(Wasm Interface Type) language. In practice, Rust component/decoder Wasm module size is around 27 KB, Javascript component has 10 MB and Python around 35 MB. Loading time of such components seems to correspond to the size of the component and it's much higher in debug mode. Javascript and Python components need linked WASI to work properly, Rust component can be compiled for both `wasm32-unknown-unknown and `wasm32-wasi` target. I also found a problem with calling host functions in Python, probably caused by a bug in a [dispatch mechanism](https://github.com/bytecodealliance/componentize-py/blob/e20d9e6706ff1421cd8001449acb51eb9c87d0c6/runtime/src/lib.rs#L405) in the Componentize-py tool. That's why I would recommend to focus mainly on Rust components until support for other languages are better. Officially supported languages are currently Rust, Javascript, Python and Go. I wouldn't recommend to switch to alternative plugins systems like Extism because they often already use Wasmtime under the hood and plan to migrate to The Component Model once it's stable enough. Another advantage is that users need only tools native to the component language - e.g. users writing Javascript decoders need only tools published in NPM. Another option would be to integrate language runtimes directly into FastWave but I think it'll be costly in the long run because of maintenance and the need to add support for all languages one by one and it will be much less secure because there won't be Wasm "sandbox" enclosing the code in plugins. There were two other implementation "surprises": 1. When calling component methods panics, it fails with error _"Cannot start a runtime from within a runtime."_. I think it's caused by running Wasmtime runtime inside Tauri runtime but it seems to be a Wasmtime problem and a workaround is to call component methods inside an `std` thread. It needs more investigation but it isn't a showstopper. 2. There is a helper function in `signal_to_timeline.rs`: ```rust fn type_hint<F>(f: F) -> F where F: for<'a> FnMut((u32, SignalValue<'a>)) -> (f64, SignalValue<'a>), { f } ``` that is used to only provide a type hint to a closure inside `async` function representing a Tauri command. It seems to be a very annoying Rust compiler issue already described in the main Rust repo. The error message is _"Implementation of `FnOnce` is not general enough"_ pointing to a different code location. Decoders are defined by this WIT file: ![screenshot_world_wit](https://github.com/JoyOfHardware/FastWave2.0/assets/18517402/0cecd07a-49af-4bd0-8bdf-0bf2dc8949b1) And there is a video of implemented decoders loaded through FastWave Javascript Command Interface: ![video_decoders](https://github.com/JoyOfHardware/FastWave2.0/assets/18517402/c3d3bc6a-16d1-4fe4-8c8d-72db021c65dc) (Both files are included in README) Testing decoders are located in `testing_files/components` in the repo. Each component/decoder contains README describing how to create and build the particular component.