Recently, Raspbian Gets Experimental OpenGL Driver (ici en Français).
So I build Gem and it works.
I didn’t test it deeply but few examples, from basics to shader passing through particule system work fine.
For now, I don’t know any way to create a Gem window without a X server, but maybe it’s somehow possible.
There are two versions in deken already, an APT one, and a more recent one.