Build: Do not use variables for sources lists

Instead use `add_library()` with unconditionally compiled sources and
conditional `target_sources()` with extra sources to compile.