Once it's gone, the Horror and its penalties are removed. Try building a Torch Sconce to support expeditions far from the Keep; having an easy spot to gather and remove Horror will make things ...