HW3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CTtrdqGpCpIZ"
   },
   "source": [
    "# Homework 3\n",
    "This notebook provides some skeleton code to get you started on the homework. Add in your own code and markdown cells to answer the homework questions. If you want to submit the notebook as a PDF, make sure your code and markdowns are clear and concise to make grading easy for the TAs.\n",
    "This notebook can be opened in Colab Open In Colab\n",
    "\n",
    "Before starting, select \"Runtime->Factory reset runtime\" to start with your directories and environment in the base state.\n",
    "\n",
    "If you want to save changes to the notebook, select \"File->Save a copy in Drive\" from the top menu in Colab. This will save the notebook in your Google Drive.\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zr4vM1Sd4RQd"
   },
   "source": [
    "# Clones, imports, and **GPU**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mk6wms--l4qB"
   },
   "source": [
    "## Use a GPU\n",
    "\n",
    "You will find this homework much easier if you use a GPU.  Make sure your runtime is set to a GPU."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xxAnuxWhC8Dx"
   },
   "source": [
    "## Clone GitHub Repository\n",
    "This will clone the repository to your machine. This includes the code and data files. Then change into the directory of the repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "h7BAeHRvnyqE"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bhvV8ImiCpIa"
   },
   "source": [
    "## Import Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ZeGEQicZnzmN"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "H92DV5YyCpIc"
   },
   "source": [
    "# Problem 1. Tweet Sentiment (35 points)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DG9ifhLjCpIc"
   },
   "source": [
    "#### 1. (4 points) Load data\n",
    "\n",
    "Load the tweets in the file `\"data/Oreo.csv\"` into a dataframe.  How many tweets in this dataframe?  Print out your answer.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2bxI5u3vCpId"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Atsr_qBkCpIe"
   },
   "source": [
    "#### 2. (15 points) Calculate sentiment\n",
    "\n",
    "Calculate the sentiment of tweets using the BERT model `\"nlptown/bert-base-multilingual-uncased-sentiment\"`.  Make sure your sentiment is a number between 0 and 1.  To do this, find the mean value of the sentiment and then divide by 5.  Make sure you use the GPU provided in Colab to make things faster.   \n",
    "\n",
    "Save the sentiments to a list `Sentiment`. Then add this list as a column to your tweet dataframe called `\"sentiment\"`.  Print out the head of your dataframe of the `\"sentiment\"` column so we know you did it right.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "UQgB12S5CpIf"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "blOMzPjMCpIh"
   },
   "source": [
    "#### 3. (2 points)  What is the mean opinion of the tweets?  Make sure you print out your answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "wK48WUaXLxE6"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "x4SG1Fa5CpIi"
   },
   "source": [
    "#### 4. (4 points) Sentiment histogram\n",
    "\n",
    "Plot a histogram of the tweet sentiment using this `histplot` function.   Make sure to add axes labels and a title to the figure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2-cuG6gLCpIi"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eUMTofBfCpIj"
   },
   "source": [
    "#### 5. (5 points) Most postive sentiment tweets\n",
    "\n",
    "Print out the text and sentiment of the five highest sentiment tweets.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Kz_mlxfjCpIj"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RMaWTUjZCpIj"
   },
   "source": [
    "#### 6. (5 points)  Most negative sentiment tweets\n",
    "\n",
    "Print out the text and sentiment of the three lowest sentiment tweets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "GtkdJUWdCpIj",
    "scrolled": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VnNytKv5CpIk"
   },
   "source": [
    "# Problem 2:  Clustering Tweets (32 points)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AOeI4vbZCpIk"
   },
   "source": [
    "#### 1. (5 points)  Clean text\n",
    "\n",
    "Use the `text_clean` function to clean the tweet text in your dataframe.  Add a column to the tweet dataframe with the cleaned text.  Print the head of the dataframe so we know you did it right.\n",
    "\n",
    "Note: to avoid issues later in this assignment, don't remove cleaned tweets of length zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "S95M7zB7CpIk"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YlRrCq2HCpIl"
   },
   "source": [
    "#### 2. (5 points) TF Embedding\n",
    "\n",
    "Calculate the term frequency frequency (TF) embedding of the tweets. Make sure you convert the embedding to an array using the `.toarray()` function, as we need this format for the PCA we do next.\n",
    "\n",
    "How big is the vocabulary?  Print your answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Ug8u4-vVCpIl"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KRdrElrYCpIl"
   },
   "source": [
    "#### 3. (8 points) PCA Embedding\n",
    "\n",
    "Calculate the 2 dimensional PCA embedding of the tf-idf embedding.  Add columns to your dataframe called `\"pca_x\"` and `\"pca_y\"` for the x and y embedding coordinates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "rj_8wbyGCpIl"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "w49WyEzZCpIm"
   },
   "source": [
    "#### 4. (14 points) K-means clusters\n",
    "\n",
    "Calculate the k-means clusters of the PCA embedding using three clusters.  Add a column to the dataframe with the tweets called `\"kmeans_label\"` that contains the k-means cluster labels.\n",
    "\n",
    "Make a histogram showing the number of tweets in each cluster with the `countplot` function.  Make sure to add axes labels and a title to the figure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "uMw7uBbgCpIm"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8sJkROoXCpIo"
   },
   "source": [
    "# Problem 3:  Clusters, Sentiment, and Engagement (33 points)\n",
    "\n",
    "Now we will look at the relationship between the sentiment, engagement, and text in the clusters you found.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4-b3ur1OCpIo"
   },
   "source": [
    "#### 1. (6 points) Plot like count versus sentiment\n",
    "\n",
    "Create a column called `\"star\"` which is the sentiment column rounded to the nearest integer, and then put in bins of width 10 (so 0.09 would be 1, 0.10 would be 1, 0.22 would be 3, 0.50 would be 5,  0.94 would be 10,  etc.).  Make a barplot of the `\"like_count\"` versus `\"star\"` value, and make the hue be `\"kmeans_label\"`.  Make sure to add axes labels and a title to the figure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "l9XkTkROCpIo",
    "scrolled": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ssjb4vKGCpIo"
   },
   "source": [
    "#### 3. (6 points) Plot sentiment per cluster\n",
    "\n",
    "Make a `boxplot` of the  sentiment of tweets vs. k-means cluster.  Make sure to add axes labels and a title to the figure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ygFkLZDeCpIp"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TRZb18tQCpIp"
   },
   "source": [
    "#### 4. (6 points) Plot like count per cluster\n",
    "\n",
    "Make a `barplot` of the `\"like_count\"` of tweets vs. k-means cluster.  Make the y-axis log scale. Make sure to add axes labels and a title to the figure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ULoGoTRbCpIp"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Q-GKEuqUCpIp"
   },
   "source": [
    "#### 5. (10 points) Cluster word clouds\n",
    "\n",
    "Plot the word cloud for each cluster.  Don't forget to remove stopwords.  A `for` loop might be useful here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "mRST99FMCpIp"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QqU5egTFgRvA"
   },
   "source": [
    "#### 6. (5 points) Cluster analyis\n",
    "\n",
    "Calculate the median sentiment of each cluster. Which cluster has the lowest median sentiment?  Print your answer and its median sentiment value.\n",
    "\n",
    "Print the sentiment and text for 10 lowest sentiment tweets in this cluster.\n",
    "\n",
    "What is the common theme of these tweets?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "1_nqUsMVSp95"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "T4",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}