Initial

README.md

@@ -21,9 +21,9 @@ For detailed description of each script, please refer to the Artifact  Appendix
 ```
 VEGA_AE
 |──dataset
-|──saved_models
-|   |──Fine_Tuned_Model
-|   |──New_Fine_Tuned_Model
+|──models
+|   |──FT_Model
+|   |──New_FT_Model
 |   └──UnixCoder
 |——Scripts
     |──Exp
@@ -47,7 +47,7 @@ VEGA_AE
 
 ## 4. Code Generation
 
-We have provided a fine-tuned model using data from ```./dataset/train.jsonl``` and ```./dataset/valid.jsonl``` in ```./saved_models/Fine_Tuned_Model```. The ```train.jsonl``` and ```valid.jsonl``` files contain code from 98 backends in our dataset.
+We have provided a fine-tuned model using data from ```./dataset/train.jsonl``` and ```./dataset/valid.jsonl``` in ```./models/FT_Model```.  The ```train.jsonl``` and ```valid.jsonl``` files contain function templates, feature vectors and ground truth for 98 backends in our dataset.
 
 We have also provided a script fot functionality test, which only generates a single function for RI5CY (Recorded as PULP in our dataset), taking less than 3 minutes with 8 Nividia Tesla V100 GPUs.
 
@@ -66,11 +66,11 @@ Upon completion of the code generation, the script outputs
 $ " Finished Function Inferencing."
 ``` 
 
-The inference result will be saved in ```./saved_models/Fine_Tuned_Model/result.jsonl```.
+The inference result will be saved in ```./models/FT_Model/result.jsonl```.
 
 Check the generated code with:
 ```
-$ cat ./saved_models/Fine_Tuned_Model/result.jsonl
+$ cat ./models/FT_Model/result.jsonl
 ```
 
 In the `result.jsonl` file, the meaning of each item in each entry corresponds as follows:
@@ -89,15 +89,18 @@ In the `result.jsonl` file, the meaning of each item in each entry corresponds a
 
 - **Run code generation with:**
 
+
+The fine-tuned model will take function templates and feature vectors for RISC-V, RI5CY, and xCORE from ```./dataset/test.jsonl``` as input, generating code and confidence scores automatically.
+
 ```
 $ bash run_test.sh
 ```
 
 Customize parameters for inferencing by modifying following options in the ```run_test.sh```.
 ```
- --model_name_or_path ../../saved_models/UnixCoder \
+ --model_name_or_path ../../models/UnixCoder \
  --test_filename ../../dataset/test.jsonl \
- --output_dir ../../saved_models/Fine_Tuned_Model \
+ --output_dir ../../models/FT_Model \
  --beam_size 4 \
  --train_batch_size 256 \
  --eval_batch_size 256 \
@@ -120,7 +123,7 @@ Upon completion of the code generation, the script outputs
 $ " Finished Inferencing."
 ``` 
 
-The inference result will be saved in ```./saved_models/Fine_Tuned_Model/result.jsonl```.
+The inference result will be saved in ```./models/FT_Model/result.jsonl```.
 
 
 
@@ -129,7 +132,7 @@ The inference result will be saved in ```./saved_models/Fine_Tuned_Model/result.
 
 Fine-tune the original UnixCoder-base-nine with provided ```./dataset/train.jsonl``` and ```./dataset/valid.jsonl```:
 
-We provide the original UnixCoder-base-nine in ```./saved_models/UnixCoder```. The original UnixCoder-base-nine can also be downloaded from HuggingFace: https://huggingface.co/microsoft/unixcoder-base-nine.
+We provide the original UnixCoder-base-nine in ```./models/UnixCoder```. The original UnixCoder-base-nine can also be downloaded from HuggingFace: https://huggingface.co/microsoft/unixcoder-base-nine.
 
 - **Run fine-tuning with:**
 ```
@@ -138,10 +141,10 @@ $ bash run_fine_tuning.sh
 
 Customize parameters for fine-tuning by modifying following options in the ```run_fine_tuning.sh```.
 ```
-  --model_name_or_path ../../saved_models/UnixCoder \
+  --model_name_or_path ../../models/UnixCoder \
   --train_filename ../../dataset/train.jsonl \
   --dev_filename ../../dataset/valid.jsonl \
-  --output_dir ../../saved_models/New_Fine_Tuned_Model \
+  --output_dir ../../models/New_FT_Model \
   --beam_size 4 \
   --train_batch_size 64 \
   --eval_batch_size 48 \
@@ -160,14 +163,14 @@ We provide the scripts to reproduce each Figure/Table from the paper, along with
 
 | Script | Description | Output | Figure/Table |
 | ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------ | -------------- |
-| ./Scripts/Exp/Time/calculate_time.py 	| Calculate the time overhead.                                                                                                            	| ./Scripts/Exp/Time/time_overhead.csv                 	| Fig.7        	|
-| ./Scripts/Exp/Accuracy/calculate_accuracy.py 	| Calculate the function-level accuracy.                                                                                                  	| ./Scripts/Exp/Accuracy/vega_result.csv               	| Fig.8        	|
-| ./Scripts/Exp/Accuracy/calculate_purple.py 	| Calculate the the percentage of functions accurately synthesized from the statements of various existing targets (Purple Bar in Fig.8). 	| ./Scripts/Exp/Accuracy/fig8_purple.csv          	| Fig.8        	|
-| ./Scripts/Exp/Accuracy/calculate_accuracy.py 	| Calculate the percentage of three types of error.                                                                                       	| ./Scripts/Exp/Accuracy/err_percentage.csv            	| Table.2      	|
-| ./Scripts/Exp/ForkFlow/calculate_forkflow.py 	| Calculate the statement-level accracy of VEGA and ForkFlow.                                                                             	| ./Scripts/Exp/ForkFlow/forkflow_result.csv           	| Fig.9        	|
-| ./Scripts/Exp/ForkFlow/calculate_forkflow.py 	| Calculate the number of accurate statements of VEGA.                                                                       	| ./Scripts/Exp/ForkFlow/mod_lines.csv 	                | Table.3      	|
-| ./Scripts/Exp/Correction/calculate_correction.py | Calculate time required by two developers to modify the VEGA-generated RISC-V backend.                                                	| ./Scripts/Exp/Correction/Correction.csv        	| Table. 4     	|
-| ./Scripts/Exp/Performance/calculate_perf.py  	| Calculate the speedup of LLVM-Base (-O3),and LLVM-VEGA (-O3) over LLVM-Base (-O0).                                    	| ./Scripts/Exp/Performance/Perf.csv             	| Fig. 10      	|
+| ./Scripts/Exp/Time/calculate_time.py 	| Calculate the time overhead.                                                                                                            	| ./Scripts/Exp/Time/Fig7.csv                 	| Fig.7        	|
+| ./Scripts/Exp/Acc/calculate_accuracy.py 	| Calculate the function-level accuracy.                                                                                                  	| ./Scripts/Exp/Acc/Fig8_Acc.csv               	| Fig.8        	|
+| ./Scripts/Exp/Acc/calculate_purple.py 	| Calculate the the percentage of functions accurately synthesized from the statements of various existing targets (Purple Bar in Fig.8). 	| ./Scripts/Exp/Acc/Fig8_Purple.csv          	| Fig.8        	|
+| ./Scripts/Exp/Acc/calculate_accuracy.py 	| Calculate the percentage of three types of error.                                                                                       	| ./Scripts/Exp/Acc/Table2.csv            	| Table.2      	|
+| ./Scripts/Exp/ForkFlow/calculate_forkflow.py 	| Calculate the statement-level accracy of VEGA and ForkFlow.                                                                             	| ./Scripts/Exp/ForkFlow/Fig9.csv           	| Fig.9        	|
+| ./Scripts/Exp/ForkFlow/calculate_forkflow.py 	| Calculate the number of accurate statements of VEGA.                                                                       	| ./Scripts/Exp/ForkFlow/Table3.csv 	                | Table.3      	|
+| ./Scripts/Exp/Correction/calculate_correction.py | Calculate time required by two developers to modify the VEGA-generated RISC-V backend.                                                	| ./Scripts/Exp/Correction/Table4.csv        	| Table. 4     	|
+| ./Scripts/Exp/Perf/calculate_perf.py  	| Calculate the speedup of LLVM-Base (-O3),and LLVM-VEGA (-O3) over LLVM-Base (-O0).                                    	| ./Scripts/Exp/Perf/Fig10.csv             	| Fig. 10      	|
 ### 6.1 Results for Fig. 7
 
 In the code generation process, we set a batch size of 256 on 8 Nvidia Tesla V100 GPU (each with 16GB memory), meaning each batch contains 256 statements. Since each batch may include statements from different function modules, we did not directly measure the generation time for each function modules of three targets (RISC-V, RI5CY, xCORE) during execution. Instead, we calculated the average inference time of each batch (25 seconds) and then derived the inference time of each statement (25/256 seconds). With the total number of statements within each function module of each target, we subsequently calculated the total inference time required for each function module of each target.
@@ -181,7 +184,7 @@ $ python ./Scripts/Exp/Time/calculate_time.py
 
 - Results:
 ```
-$ cat ./Scripts/Exp/Time/time_overhead.csv
+$ cat ./Scripts/Exp/Time/Fig7.csv
 ```
 
 ### 6.2 Results for Fig. 8
@@ -195,15 +198,15 @@ In this Exact Match evaluation, each statement is deemed correct if the VEGA-gen
 
 - Command:
 ```
-$ cp ./saved_models/Fine_Tuned_Model/result.jsonl ./Scripts/Exp/Accuracy
-$ python ./Scripts/Exp/Accuracy/calculate_accuracy.py
+$ cp ./models/FT_Model/result.jsonl ./Scripts/Exp/Acc
+$ python ./Scripts/Exp/Acc/calculate_accuracy.py
 ```
 
 This script will automatically analyze the VEGA's output from "result.jsonl" and compare the generated code and confidence scores with the ground truth. Based on this comparison, it will determine whether each function is correct.
 
 - Accuracy Results:
 ```
-$ cat ./Scripts/Exp/Accuracy/vega_result.csv
+$ cat ./Scripts/Exp/Acc/Fig8_Acc.csv
 ```
 
 
@@ -212,13 +215,13 @@ We also provide a script for calculating the proportion of "Accurate Functions w
 
 - Command:
 ```
-$ python ./Scripts/Exp/Accuracy/calculate_purple.py
+$ python ./Scripts/Exp/Acc/calculate_purple.py
 ```
 
 
 - Results:
 ```
-$ cat ./Scripts/Exp/Accuracy/fig8_purple.csv
+$ cat ./Scripts/Exp/Acc/Fig8_Purple.csv
 ```
 
 
@@ -272,13 +275,13 @@ Executing the script in 6.2 will also yield the proportion of the three types of
 
 - Command:
 ```
-$ python ./Scripts/Exp/Accuracy/calculate_accuracy.py
+$ python ./Scripts/Exp/Acc/calculate_accuracy.py
 ```
 
 
 - Results:
 ```
-$ cat ./Scripts/Exp/Accuracy/err_percentage.csv
+$ cat ./Scripts/Exp/Acc/Table2.csv
 ```
 
 
@@ -294,7 +297,7 @@ $ python ./Scripts/Exp/ForkFlow/calculate_forkflow.py
 
 - Results:
 ```
-$ cat ./Scripts/Exp/ForkFlow/forkflow_result.csv
+$ cat ./Scripts/Exp/ForkFlow/Fig9.csv
 ```
 
 ### 6.5 Results for Table. 3
@@ -310,7 +313,7 @@ $ python ./Scripts/Exp/ForkFlow/calculate_forkflow.py
 
 - Results:
 ```
-$ cat ./Scripts/Exp/ForkFlow/mod_lines.csv
+$ cat ./Scripts/Exp/ForkFlow/Table3.csv
 ```
 
 
@@ -328,7 +331,7 @@ $ python ./Scripts/Exp/Correction/calculate_correction.py
 
 - Results:
 ```
-$ cat ./Scripts/Exp/Correction/Correction.csv
+$ cat ./Scripts/Exp/Correction/Table4.csv
 ```
 
 ### 6.7 Results for Fig. 10
@@ -340,12 +343,12 @@ By executing the following script, the speedup for VEGA-generated LLVM backend (
 
 - Command:
 ```
-$ python ./Scripts/Exp/Performance/calculate_perf.py
+$ python ./Scripts/Exp/Perf/calculate_perf.py
 ```
 
 - Results:
 ```
-$ cat ./Scripts/Exp/Performance/Perf.csv
+$ cat ./Scripts/Exp/Perf/Fig10.csv
 ```
 
 

Scripts/Exp/{Accuracy → Acc}/calculate_accuracy.py

@@ -82,7 +82,7 @@ def calculate_accuracy():
                 
     all_func_lis = list(set(all_func_lis))
 
-    with open(folder+"/vega_result.csv", 'a', encoding='utf-8', newline="") as f:
+    with open(folder+"/Fig8_Acc.csv", 'a', encoding='utf-8', newline="") as f:
         f_csv = csv.writer(f)
         avg_dic = {}
         all_dic = {}
@@ -136,7 +136,7 @@ def calculate_accuracy():
 
 if __name__ == '__main__':
     get_wrong_list()
-    with open(folder+"/vega_result.csv", 'w', encoding='utf-8', newline="") as f:
+    with open(folder+"/Fig8_Acc.csv", 'w', encoding='utf-8', newline="") as f:
         f_csv = csv.writer(f)
         f_csv.writerow(["Target", "Module", "Correct", "Total", "Accurate", "Inaccurate", "Confidence Score≈1.00", "Confidence Score in [0.50, 1.00)"])
     total_dic = calculate_accuracy()
@@ -166,7 +166,7 @@ if __name__ == '__main__':
         else:
             target_func_num_dic[k.split(" ")[0].lower()] += len(list(set(total_dic[k])))
     
-    with open(folder+"/err_percentage.csv", 'w', encoding='utf-8', newline = "") as f:
+    with open(folder+"/Table2.csv", 'w', encoding='utf-8', newline = "") as f:
         f_csv = csv.writer(f)
         for k in target_func_num_dic:
             #print(target_func_num_dic[k])

Scripts/Exp/{Accuracy → Acc}/calculate_purple.py

@@ -30,7 +30,7 @@ def calculate_template():
                 res_dic[" ".join([row[-1], row[0]]).lower()] = 1
             else:
                 res_dic[" ".join([row[-1], row[0]]).lower()] += 1
-        with open(folder+"/fig8_purple.csv", 'w', encoding='utf-8', newline="") as f:
+        with open(folder+"/Fig8_Purple.csv", 'w', encoding='utf-8', newline="") as f:
             f_csv = csv.writer(f)
             for k in res_dic.keys():
                 f_csv.writerow([k.split(' ')[0].replace("pulp", "ri5cy"), k.split(' ')[1], round(float(res_dic[k])/float(len(list(total_dic[k]))), 3)])

Scripts/Exp/Correction/calculate_correction.py

@@ -26,7 +26,7 @@ result_A = sum_time_by_name("/Dev_A.csv")
 
 result_B = sum_time_by_name("/Dev_B.csv")
 
-with open(folder+"/Correction.csv", mode='w', newline='', encoding='utf-8') as out_file:
+with open(folder+"/Table4.csv", mode='w', newline='', encoding='utf-8') as out_file:
     csv_writer = csv.writer(out_file)
     for k in result_A.keys():
         csv_writer.writerow(["Dev A", k, round(result_A[k]/3600.0, 2)])

Scripts/Exp/ForkFlow/calculate_forkflow.py

@@ -153,7 +153,7 @@ def duplicate_data(tar):
         Mod_Result[module][1] +=  Mips_same
 
 
-    with open(folder+"/forkflow_result.csv", 'a', encoding='utf-8', newline="") as f:
+    with open(folder+"/Fig9.csv", 'a', encoding='utf-8', newline="") as f:
         f_csv = csv.writer(f)
         avg_vega = 0.0
         avg_mips = 0.0
@@ -163,7 +163,7 @@ def duplicate_data(tar):
             avg_mips += float(round(kv[1][1]*1.0 / kv[1][0], 3))
         f_csv.writerow([tar.replace("PULP", "RI5CY"), "Avg", round(avg_mips / len(Mod_Result), 3), round(avg_vega / len(Mod_Result), 3)])
 
-    with open(folder+"/mod_lines.csv", 'a', encoding='utf-8', newline="") as f:
+    with open(folder+"/Table3.csv", 'a', encoding='utf-8', newline="") as f:
         f_csv = csv.writer(f)
         all_vega = 0
         all_mips = 0
@@ -177,11 +177,11 @@ def duplicate_data(tar):
 
 if __name__ == '__main__':
     get_wrong_list()
-    with open(folder+"/forkflow_result.csv", 'w', encoding='utf-8', newline="") as f:
+    with open(folder+"/Fig9.csv", 'w', encoding='utf-8', newline="") as f:
         f_csv = csv.writer(f)
         f_csv.writerow(["Target", "Module", "Fork_Acc", "VEGA_Acc"])
 
-    with open(folder+"/mod_lines.csv", 'w', encoding='utf-8', newline="") as f:
+    with open(folder+"/Table3.csv", 'w', encoding='utf-8', newline="") as f:
         f_csv = csv.writer(f)
         f_csv.writerow(["Target", "Module", "VEGA_Accurate_Lines", "VEGA_Manual_Lines"])
 

Scripts/Exp/{Performance → Perf}/calculate_perf.py

@@ -5,7 +5,7 @@ import time
 
 folder = str(pathlib.Path(__file__).parent.resolve())
 
-with open(folder+"/Perf.csv", mode='w', newline='', encoding='utf-8') as out_file:
+with open(folder+"/Fig10.csv", mode='w', newline='', encoding='utf-8') as out_file:
     csv_writer = csv.writer(out_file)
     csv_writer.writerow(["Target", "Case", "LLVM-Base", "LLVM-VEGA"])
     

Scripts/Exp/Time/calculate_time.py

@@ -27,7 +27,7 @@ def calculate_time():
         else:
             Target_Module[dic["Target"]+" "+dic["Module"]] += 1
         Func_Lis.append(dic["File"]+" "+dic["Func"])
-    with open(folder+"/time_overhead.csv", "w",encoding="utf-8", newline = "") as f:
+    with open(folder+"/Fig7.csv", "w",encoding="utf-8", newline = "") as f:
         writer = csv.writer(f)
         for kv in Target_Module.items():
             writer.writerow(kv[0].replace("PULP", "RI5CY").split(" ") + [kv[1], math.ceil(kv[1] * 25.0 / 256)])

Scripts/UnixCoder/run_one_model.py

@@ -372,6 +372,7 @@ def vega_train_main():
     set_seed(args.seed)
 
     # make dir if output_dir not exist
+    args.output_dir = folder + "/" + args.output_dir
     if os.path.exists(args.output_dir) is False:
         os.makedirs(args.output_dir)
     args.model_name_or_path = folder + "/" + args.model_name_or_path
@@ -381,7 +382,6 @@ def vega_train_main():
         args.dev_filename = folder + "/" + args.dev_filename
     if args.test_filename:
         args.test_filename  = folder + "/" + args.test_filename
-    args.output_dir = folder + "/" + args.output_dir
     # build model
     tokenizer = RobertaTokenizer.from_pretrained(args.model_name_or_path)
     config = RobertaConfig.from_pretrained(args.model_name_or_path)

run_fine_tuning.sh

@@ -2,10 +2,10 @@
 python ./Scripts/UnixCoder/run_one_model.py \
   --do_train \
   --do_eval \
-  --model_name_or_path ../../saved_models/UnixCoder \
+  --model_name_or_path ../../models/UnixCoder \
   --train_filename ../../dataset/train.jsonl \
   --dev_filename ../../dataset/valid.jsonl \
-  --output_dir ../../saved_models/New_Fine_Tuned_Model \
+  --output_dir ../../models/New_FT_Model \
   --beam_size 4 \
   --train_batch_size 64 \
   --eval_batch_size 48 \

run_function_test.sh

@@ -2,9 +2,9 @@
 # do test
 python ./Scripts/UnixCoder/run_one_model.py \
  --do_function_test \
- --model_name_or_path ../../saved_models/UnixCoder \
+ --model_name_or_path ../../models/UnixCoder \
  --test_filename ../../dataset/test.jsonl \
- --output_dir ../../saved_models/Fine_Tuned_Model \
+ --output_dir ../../models/FT_Model \
  --beam_size 4 \
  --train_batch_size 256 \
  --eval_batch_size 256 \

run_test.sh

@@ -2,9 +2,9 @@
 # do test
 python ./Scripts/UnixCoder/run_one_model.py \
  --do_test \
- --model_name_or_path ../../saved_models/UnixCoder \
+ --model_name_or_path ../../models/UnixCoder \
  --test_filename ../../dataset/test.jsonl \
- --output_dir ../../saved_models/Fine_Tuned_Model \
+ --output_dir ../../models/FT_Model \
  --beam_size 4 \
  --train_batch_size 256 \
  --eval_batch_size 256 \