Trust me I had trouble with this too. Here's what you do: you have to defeat the mander that serves his guard's food then poison the food. This should leve him alone so you can face him 1 on 1.
Following the quest arrow lead will not lead you to Taskmaster Djal. I followed quest line and still he was not in the area indicated by the arrow. Look at your map. It will show his chamber. That is where i found him. He was alone.