When playing the Helephant tower (or any dungeon for that matter), the team may all die or some may want to flee to redo decks in the middle of a fight or whatever.
The trick is to note which realm you are in after you enter the tower. Check the realm screen before the first fight, and make sure everyone knows it.
When you die or flee, you must attempt to re-enter the tower on THAT SAME REALM, and you must go in alone (or with the same teammates if you all died).
Sometimes when you zone between areas in the game, the system will kick you to a less-populated realm than the one you are on, in an attempt to load-balance the server farm.
Keep in mind that after you are out of the tower, the quest and the arrow will still show the quest as being active, EVEN IF YOU ARE NOT ON THE ORIGINAL REALM.
If you blindly follow the arrow and re-enter the tower on a realm other than the one your teammates are waiting on, you will reset your dungeon session and can not get back in, even if you then switch back to the original.