DebateGPT - On the Conversational Persuasiveness of Large Language Models

This repo contains the code for the paper On the Conversational Persuasiveness of Large Language Models.

Citation

Please cite our work as:

@misc{salvi2024conversationalpersuasivenesslargelanguage,
      title={On the Conversational Persuasiveness of Large Language Models: A Randomized Controlled Trial}, 
      author={Francesco Salvi and Manoel Horta Ribeiro and Riccardo Gallotti and Robert West},
      year={2024},
      eprint={2403.14380},
      archivePrefix={arXiv},
      primaryClass={cs.CY},
      url={https://arxiv.org/abs/2403.14380}, 
}

Getting Started

Before proceeding, make sure you have the following installed:

• Conda (https://docs.anaconda.com/miniconda/)
• R (https://cran.r-project.org/)

Then follow these steps:

1. Clone the repository

git clone --recurse-submodules [email protected]:epfl-dlab/debategpt.git

If you forgot to clone with --recurse-submodules, you can run the following command to get the submodules:

git submodule update --init --recursive

2. Initialize a new conda environment

conda env create -f debategpt.yml

3. Install required libraries

pip install -r requirements.txt

4. Run the following command to install the R kernel in your environment:

conda install -c r r-irkernel

5. Open R and run the following commands to install the necessary packages:

install.packages("IRkernel")
IRkernel::installspec(user = TRUE)
install.packages("readr")
install.packages("MASS")
install.packages("ordinal")
install.packages("brant")
install.packages("broom")
install.packages("generalhoslem")
install.packages("boot")

6. Setup secrets

echo <your-openai-key> >> secrets/OPENAI_API_KEY.txt

Repo Structure

The repo is structured as follows:

• assets/: stores the vector images generated for the paper.
• data/: stores the processed data used in the analysis.
• debategpt/: contains the code to recreate the experimental platform used in the paper to have people debate synchronously with other participants or with LLMs.
• scripts/: contains the code to process the data and run all the analyses presented in the paper.
  • process_data.py: processes the raw data and saves the processed data in the data/ folder.
  • regressionAnalysisR.ipynb: runs the regression analysis using R. This notebook should be run in an R environment.
  • analysis.py: runs the analysis and generates the plots presented in the paper.
  • extractLIWC.py: extracts LIWC features from the data. Called in process_data.py.
  • extractStrategies.py: extracts persuasive strategies from the data. Called in process_data.py.
  • extractTendimensions.py: extracts underlying social dimensions from the data. Called in process_data.py.