I'm going to take a wild guess and assume that you don't have to walk all the way. If you're attuned to Chiconis, then that is one of the closest portals to the Cleric's Tower. If not, then it's...