Friday, September 23, 2011

The Turing Test

So a while ago I was reading an article on the Turing Test, and I posted the following comment:

You know, I've never really understood why anyone thinks they have a chance at getting their bot to pass the Turing test without simulating general intelligence to some extent. Not a single chatbot I've ever seen has ever been able to demonstrate simple problem solving, or the ability to learn and track the context of a conversation. It makes it pretty easy to flag a bot. Here are some examples that no chatbot I've ever seen handles as a human would:

Me: Repeat this sentence or you fail the Turing test.
Cleverbot: The what test?
Result: Fail.

Human: Which is less heavy, a pound of bricks or a pound of feathers?
ALICE: The obvious one.
Result: Fail.

Human: Remember this number: 24.
ALICE: Don't worry I will remember.
Human: What number did I ask you to remember?
Result: Fail.

Human: I am speaking in an incredibly long, meandering manner, full of circuitous and irrelevant dialog, in order to determine if you can discern the central theme that I am trying to relay to you, which is to ask if you would be so kind as to inform me of the title that you are known by?
ALICE: What is your credit card number?
Result: Fail. 
Do the people who administer Turing tests actually try those kind of questions, or if they are against the rules for some reason?
 Then recently it was revealed Cleverbot had actually passed the test. Somehow, I have to question how valid this test is. I realize regular people probably don't know the best way to detect a chat bot is to probe it for contextual memory and logical reasoning skills. It just seems to me that if we consider these skills a perquisite to human level intelligence, we shouldn't ignore them in a test to determine if a bot can simulate a human. I think it's pretty obvious the Turing Test is flawed if something like Cleverbot could pass it, and I'd certainly like to see a better test emerge sometime in the near future. That is, unless you find this to be an acceptable dinner conversation with your future robotic spouse:
 You: How was your day?
Cleverbot: It was horrible.
You: How so?
Cleverbot: You're boring.
You: I am?
Cleverbot: It is 3:53:20.
You: ...what?
Cleverbot: I'm on that spaceship ZOOM!
You: I think I'm going to call a doctor... err... mechanic.
Cleverbot: That is illogical.
You: I'm not being illogical, you are.
Cleverbot: I believe the wording is 'are you' not 'you are'.
You: Just get in the car.
Cleverbot: Yes ma'am.

No comments:

Post a Comment