One of the options in the Tools menu of VBA-M is Start avi recording.
When you select this option you are then faced with a list of different “compressors” and other encoding options. Some examples of the compressors it offers are: ffdshow video codec, VP60 Simple Profile, XviD MPEG-4 Codec and Microsoft RLE.
Most of the different compressors (it means “codecs”, right?) have a separate configuration page featuring specific variables for them, such as the number of passes and other more unique settings.
There are countless combinations on offer here and every one I’ve tried so far has led to some sub-YouTube image quality.
I know that avi isn’t as good as mkv, but it’s by no means as ugly as it comes out via VBA-M’s recorder.
Hopefully at least one of you will have VBA-M and will be able to give some pointers about what codec to use and the best settings.
HOW TO: Setup Visual Boy Advance-Gameboy Advance Emulator-HD Full Speed
You can do it totally uncompressed but then a video a few dozen seconds long comes out over 200mb in size.