The demons have a slow respawn time. So kill them before opening the doors. Also make sure you kill them near a wall, so theres a few tiles without demongoop. Finally, try to lure as many demons as possible and stand still while they drain their own hp. Its a pretty tough area, you consider coming back later. I don't think I got all the books there either.