I think that would be an interesting concept to try, but the problem is that so many ferns get pretty large . . . so you would have to have a larger terrarium. You could even take it a step further by utilizing only fern relatives (like the club mosses--different Selaginella's) as your base plant instead of moss.
Some ferns I have had good luck with in the terrarium are Nephrolepis cordifola 'duffi' (the lemon button fern), a small maidenhair fern (sorry, I have forgotten the name of this one), and the resurrection fern. I have also seen Nephrolepis exaltata 'Suzi Wong' do well in the terrarium. Of course, different Selaginella's commonly available include uncinata (rainbow or peacock club moss), plana, or kraussinea 'aurea' (gold tipped club moss). You really can't go wrong with these.
I know others have had good luck with other ferns, but I'll let them jump in. I know Alexander Stubbs has some interesting ferns . . . and I'd personally jump on the chance to trade for one of his. 
-----
Homer W. Faucett III, esq.
Purveyor of Trivialities and Fine Nonsense