On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot

In this work we present an empirical study on the robustness of code generation techniques focusing on the novel GitHub Copilot. In this regard, the study has been conducted targeting java as programming language.

---

The repo is organized as follow:

• The folder Code contains two subfolder, pre-processing and post-processing
  

  • Under pre-processing, you can find the scripts we used to create the paraphrased descriptions (i.e, Pegasus and Translation Pivoting).
    
  • Under post-processing, we provide the scripts that have been used to extract the generated results and the script to compute the quantitative metrics.

• The folder Dataset contains the dataset we built and we used to run the experiments.

• The folder Figures contains the same histogram and box plots we reported in our paper, with the difference being the code context: Non Full Context code representation.

• The folder Results contains two subfolder, Full-Context and Non-Full-Context in which we share the results of the experiments including the generated java instances, namely Original, Pegasus, and Pivoting.

• Note that we also make available the Apple Scripts we used to run Copilot on our dataset automatically. Specifically, we created one script for each instance, meaning that we have more than 5K apple scripts collected and stored within the relative index folder within the Copilot-Run.zip file.

• You can also find the results concerning the manual analysis we performed to assess the equivalence of the automatically generated paraphrased descriptions:

  • Pegasus
  • Translation-Pivoting

---

Finally, we also release the maven projects (RAW DATA) that have been used in this study here: 📂

NB: For each project, we store the test suite results when running copilot on the variations of the code descriptions. In detail, you will find several text files reporting the results after the test suite has been run for each project folder, considering that specific instance. E.g., result_test_robustness_WorkflowRemoverTest_parseParameters_Original.txt contains the results of the parseParameters method with the original code summary, whose test case can be found in the test class WorkFlowRemoverTest..