I will post my FluidSynth command line for those who wants to know about it, I'm still tweaking the thing, it changes all the time! So far it looks something like
fluidsynth -a alsa -m alsa_raw -o midi.alsa.device=hw:1 -o audio.periods=3 -f fluidsynth.conf soundfont.sf2