An anonymous reader points out this recently published study (PDF) on detecting malicious (or at least suspicious) Tor exit relays.
From their conclusions: "After developing a scanner, we closely
monitored all ~1000 exit relays over a period of four months. Wed
discovered 25 relays which were either outright malicious or simply
misconfigured. Interestingly, the majority of the attacks were
coordinated instead of being isolated actions of independent
individuals. Our results further suggest that the attackers made an
active effort to remain under the radar and delay detection."
One of the authors, Philipp Winter, wrote a followup blog post to help clarify what the paper's findings mean for Tor users, including this clarification: "First,
it's important to understand that 25 relays in four months isn't a lot.
It is ultimately a very small fraction of the Tor network. Also, it
doesn't mean that 25 out of 1,000 relays are malicious or misconfigured
(we weren't very clear on that in the paper). We have yet to calculate
the churn rate of exit relays which is the rate at which relays join and
leave the network. 1,000 is really just the approximate number of exit
relays at any given point in time. So the actual number of exit relays
we ended up testing in four months is certainly higher than that. As a
user, that means that you will not see many malicious relays 'in the
wild."