I'd agree with only having two or three presenters (plus a special guest, maybe), and even then, their voices need to be easily distinguished. Any more than that and it gets harder to track who's talking.
Would it be worth trying a couple of different formats out? Maybe a totally free-form chat (though that could easily sound awkward), and then an agenda-led version?