Tests: Avoid leaving behind non-readable directories

Since commit 337bc5662c (if(): add operators IS_READABLE, IS_WRITABLE
and IS_EXECUTABLE., 2023-10-21) we create some non-readable directories.
CMake 3.28 and below do not know how to delete them, so some nightly
builds fail ctest_empty_binary_directory.  Add read permission to those
directories when we are finished with them.