Psychopathy Tweets: Too Much Statistics, Not Enough Proof of Concept
On Sunday Defcon 20 had a talk that I had previously written about on the idea of using statistical analysis of word use to determine psychopathy in individuals online. As I sat through the talk and steadily watched people get up and leave I too had the urge to walk away as well. However, I had a mission and that was to confirm if there was any evidence that would say to me this was a viable means of detection for psychopaths.
What I came out with, after many slides of numbers, was “nope not really” Which, I pretty much had thought before. There are just too many variables to this type of venture and you would, in the end, need to have a trained psychoanalyst to talk to the individual to determine whether or not they are a true psychopath.
Sorry Sugg.. It was an interesting idea and I am wondering just where this will go if the author of the original paper tries to expand upon this process. You see, for this to work online possibly, is that the trained individual would chat with the “patient” or “UNSUB” as the case may be, to ask specific questions to elicit responses. See, that would work I think, but it is a manual process not a big data solution. So, while it was an interesting trip into what psychopathy is and possibly how to spot it in word use, it was a failed experiment in my book.
Now, another twist on this idea might be to take the transcripts of anonymous and other IRC chats and wash that through your program… There’s a lot going on there mentally and might show some traits, but, are they really suffering from some sort of psychiatric illness or are they just maladjusted? This has been something I have written about before an the vernacular used as well as the mindset that seems to be prevalent warrants some looking at perhaps.
Maybe next year?
Overall though, I surely hope that the governments and law enforcement bodies out there do not take up this idea and begin to mine people’s chat logs for psychopathy
*shudder*
Ding Dong! It’s the forensic psychiatrist.. We saw your tweets and thought we’d have a chat? What? these cops? They’re just here to visit too!
K.
Krypt3ia 
Doesn’t sound too far off. Psychiatric diagnosis is only borderline scientific in it’s offline form.
Alex G. (@neurochancer)
2012/07/31 at 04:37
Firstly, thank you for attending the talk and staying through until the end. For me the most rewarding discussion occured in the Q&A afterwards, were you able to attend that? Some of the discussion touched on exactly your points.
I would tend to disagree that it’s a failed experiment because the aim was to critically (as critically as we were able), answer the question on the DNA blog and the performance stated in the 2011 Golbeck et al paper, which employed the same techniques that we used (That paper is here : http://www.cs.umd.edu/~golbeck/pubs/Golbeck%20et%20al.%20-%202011%20-%20Predicting%20Personality%20from%20Twitter.pdf ).
The Golbeck paper states “Our results show that we can predict personality to within just over 10%, a resolution that is likely fine-grained enough for many applications”.
We attempted to highlight the problem of using Average Error as a performance measure by demonstrating that our models also had ~90% accuracy over the whole model, but that still generates a large number of false positives.
There have been a relatively small number of papers focusing on personality in social media content through automated feature extraction, but almost all of them are followed by headlines such a “Facebook can serve as a personality test”. What we aimed to do is show that, it doesn’t look anywhere near that simple to us. That is, we tend to agree with what you’re saying in this post.
However, things get a little more complex if multiple measures of personality are brought together. A paper on just this was recently shared at the European Conference on Personality Psychology.
Finally, I’d be happy to share our draft paper with you if you’d be interested in providing some critical feedback/review. Our intention is to highlight the limitations with research of this nature and the pitfalls of choosing the wrong performance measures in machine learning. This is not to say that current research is entirely without merit (multiple interesting use-cases), it’s just not as good as the leadlines would have you believe. Sadly, that’s not a sexy story.
Cheers
Chris
Suggy (@TheSuggmeister)
2012/07/31 at 05:27