The distinction is accountability. Source distributions don't
necessarily prevent attacks, but they allow you to detect them,
usually.
If you download source, presumably it has the author's email address.
Presumably that author is known to at least a few folks on the net.
If you compile the code, and it behaves antisocially, you can (1) look
at the source to find out why, (2) contact the author, or (3) sling
mud at his name publicly.
It's possible for an unnamed intruder to substitue source code with
an attack in it, but the file size or checksum of the distribution
will be different from all the other copies on the net. Granted...
not a lot of folks check that.
And in that case, the author can presumably diff the rogue sources
against his real sources and find exactly the difference.
So the same sorts of attacks that go with binaries can happen
with source distributions, but they're much more difficult to pull off.
The right answer is digitally signed distributions. Then only can
you be certain that the bits have not changed since they left
the author's hands.
Dan