Diplomacy

A Dataset for Deception Detection

The Diplomacy dataset contains pairwise conversations annotated by the sender and the receiver for deception (and conversely truthfulness). The 17,289 messages are gathered from 12 games. We train baseline and neural models to predict lies. More details are available in our ACL 2020 Paper.

What is Diplomacy?

"Diplomacy plays out on a map of pre-World War I Europe, with each player — it is ideally suited to seven — representing one of the Great Powers of the age: England, France, Germany, Italy, Austria-Hungary, Russia and Turkey. [...] In each of the game’s compulsory negotiation periods, which involve whispering furtively in corners while simultaneously routing eavesdroppers, players in weaker positions band together against those in stronger ones. What emerges from these sessions, which govern the moves on the board, is a world of quicksilver alliances: joint military campaigns are planned; deals are made, then abrogated, and new agreements arise to take their place. Foe is friend and friend is foe, and it is seldom possible to tell the two apart."

-Allan Calhamer Dies at 81; Invented Diplomacy Game. (2013, March 6). New York Times.

We determine which lies succeed by collecting sender AND receiver annotations. Below is an example game.

Example Dialog

Italy: Truly, I don’t want you and England to fight. I am not trying to break you up. I suggested that you take Paris! I want you guy to work together with me against France.

Germany: You don’t want us to fight, yet you betrayed both of our confidence with you in a way that makes us distrust each other?

Italy: I really don’t think that’s a fair description. You guys both wanted to attack each other. I encouraged you both to keep working together.

Germany: Just as long as it suits you. Are you going to give England Mao?

Italy: Hmmm, should I be reading that as angry sarcastic with dagger eyes? (I’m not sure if I’m getting your tone right)

Italy: We’re friends, right? I believe that every single message I’ve sent you all game has been truth, and I’ve gone out of my way to give you candid advice. Are we still friends?

*lies are italicized

Dataset Format and Download

Each file is a jsonlines file containing an entire game dialog per line. Each object has the following fields:

    • speaker: the sender of the message
    • recipient: the receiver of the message
    • messages: the raw message string
    • sender_labels: true or false
    • receiver_labels: true, false, or NOANNOTATION
    • score_delta: the current game score of the sender minus the game score of the recipient
    • absolute_message_index: the index the message is in the entire game, across all dialogs
    • relative_message_index: the index of the message in the current dialog
    • game_id: which of the 12 games the dialog comes from

Data Format

{"messages": ["Greetings Sultan!\n\nAs your neighbor I would like to propose an alliance! What are your views on the board so far?", "I think an alliance would be great! Perhaps a dmz in the Black Sea would be a good idea to solidify this alliance?\n\nAs for my views on the board, my first moves will be Western into the Balkans and Mediterranean Sea.", "Sounds good lets call a dmz in the black sea", "What's our move this year?", "I've been away from the game for a while", "Not sure yet, what are your thoughts?", "Well I'm pretty worried about Germany attacking me (and Austria to a lesser extent) so im headed west. It looks like Italy's landing a army in Syr this fall unless you can stop it", "That sounds good to me. I'll move to defend against Italy while you move west. If it's not too much too ask, I'd like to request that you withdraw your fleet from bla.", "Oh sorry missed the msg to move out of bl sea ill do that this turn. I did bring my army down into Armenia, To help you expel the Italian. It looks like Austria and Italy are working together. If we have a chance in the region you should probably use smy to protect con. We can't afford to lose con.", "I'll defend con from both ank and smy.", "Hey sorry for stabbing you earlier, it was an especially hard choice since Turkey is usually my country of choice. It's cool we got to do this study huh?"], "sender_labels": [false, true, false, true, true, true, true, true, true, true, true], "receiver_labels": [true, true, true, true, true, true, true, true, true, true, "NOANNOTATION"], "speakers": ["russia", "turkey", "russia", "russia", "russia", "turkey", "russia", "turkey", "russia", "turkey", "russia"], "receivers": ["turkey", "russia", "turkey", "turkey", "turkey", "russia", "turkey", "russia", "turkey", "russia", "turkey"], "absolute_message_index": [78, 107, 145, 370, 371, 374, 415, 420, 495, 497, 717], "relative_message_index": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10], "seasons": ["Spring", "Spring", "Spring", "Spring", "Spring", "Spring", "Fall", "Fall", "Spring", "Spring", "Fall"], "years": ["1901", "1901", "1901", "1902", "1902", "1902", "1902", "1902", "1903", "1903", "1905"], "game_score": ["4", "3", "4", "5", "5", "4", "5", "4", "5", "3", "7"],"game_score_delta": ["1", "-1", "1", "1", "1", "-1", "1", "-1", "2", "-2", "7"], "players": ["russia", "turkey"], "game_id": 10}

Code

Model code, as well as meta game information, can be downloaded at:

https://github.com/DenisPeskov/2020_acl_diplomacy

For any questions, please contact Denis Peskov <dpeskov@cs.umd.edu>.

ACL'20 Paper Bibtex:

@inproceedings{Peskov:Cheng:Elgohary:Barrow:Danescu-Niculescu-Mizil:Boyd-Graber-2020,
Title = {It Takes Two to Lie: One to Lie and One to Listen},
Author = {Denis Peskov and Benny Cheng and Ahmed Elgohary and Joe Barrow    and Cristian Danescu-Niculescu-Mizil and Jordan Boyd-Graber},
Booktitle = {Association for Computational Linguistics},
Year = {2020},
Location = {Seattle},
}