There is something you construct to fix this difficulty. There is a damper that you can purchase which is installed at the peak of the chimney.
This damper acts like a storm door includes a seal so that the air cannot sneak around it.