Hugging Face's logo

docz1105
/
ComBack_Models

code
C++/C Code
Compiler Backend
Model card Files
unknown commited on
Commit
ff842a9
·
1 Parent(s): 2405a27

add files

Browse files
Files changed (1) hide show
  1. README.md +152 -3
README.md CHANGED
@@ -1,3 +1,152 @@
1
- ---
2
- license: cc-by-4.0
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # ComBack: A Versatile Dataset for Enhancing Compiler Backend Development Efficiency
2
+
3
+ ComBack is a large-scale multi-platform compiler backend code dataset.
4
+ This repository contains all fine-tuned models for experiments with ComBack.
5
+
6
+ - language: C++/C
7
+ - metrics: Exact Match(EM), Edit Distance(ED), BLEU4
8
+ - tags: code; compiler backend
9
+ - license: CC-BY-4.0
10
+
11
+ ## Task Information
12
+
13
+ - Statement-Level Completion: complete current statement.
14
+ ```c++
15
+ //Inputs:
16
+ ...
17
+ adjustReg(MBB,LastFrameDestroy, DL, SPReg, FPReg, -StackSize+RVFI->getVarArgsSaveSize()
18
+ //Ground Truth:
19
+ MachineInstr::FrameDestroy);
20
+ ```
21
+
22
+ - Next-Statement Suggestion: predict the next statement.
23
+
24
+ ```c++
25
+ //Inputs:
26
+ ...
27
+ maxCallFrameSize = (maxCallFrameSize + AlignMask) & ~AlignMask;
28
+ //Ground Truth:
29
+ MFI -> setMaxCallFrameSize(maxCallFrameSize);
30
+ ```
31
+
32
+
33
+ - Code Generation: generate a function with function description in natrual language.
34
+
35
+ ```c++
36
+ //Inputs:
37
+ getPointerRegClass: Returns a TargetRegisterClass used for pointer values.
38
+ Target-Specific Value: Sparc, SP::I64RegsRegClass, SP::IntRegsRegClass.
39
+ //Ground Truth:
40
+ TargetRegisterClass *SparcRegisterInfo::getPointerRegClass(MachineFunction &MF ,unsigned Kind) {
41
+ return Subtarget.is64Bit() ? &SP::I64RegsRegClass : &SP::IntRegsRegClass;
42
+ }
43
+ ```
44
+
45
+ ## Organization
46
+
47
+ - `Existing_Targets/*`: **split data of all 178 backends into train/valid/test set in the ratio of 80%:10%:10%**
48
+ - Dataset Info
49
+
50
+ | Task | Train | Valid | Test |
51
+ | ---- | ---- | ---- | ---- |
52
+ | Statement-Level Comp. | 128,899(11.36M Token) | 16,112(1.43M Token) | 16,113(1.43M Token) |
53
+ | Next-Statement Sugg. | 173,052(15.69M Token) | 21,631(1.99M Token) | 21,632(1.98M Token) |
54
+ | Code Generation. | 36,236(5.10M Token) | 4,530(0.64M Token) | 4,530(0.64M Token) |
55
+
56
+ We fine-tuned six representative models across three tasks.
57
+
58
+ - Without Fine-Tuning
59
+ | | Stmt. Comp. | Stmt. Comp. | Next. Sugg. | Next. Sugg. | Code. Gen. | Code. Gen. |
60
+ |-------------|:-----------------:|:-----------------:|:----------------:|:----------------:|:----------:|:----------:|
61
+ | **Model** | EM | ED | EM | ED | BLEU4 | ED |
62
+ | CodeBert | 0.00 | 0.97 | 0.00 | 1.31 | 0.00 | 0.44 |
63
+ | GraphCodeBert | 0.00 | 0.35 | 0.00 | 0.54 | 0.00 | 2.41 |
64
+ | UnixCoder | 0.07 | 27.56 | 15.93 | 29.11 | 0.00 | 31.81 |
65
+ | CodeT5 | 0.65 | 21.45 | xxx | xxx | xxx | xxx |
66
+ | NatGen | 0.00 | 13.52 | 0.02 | 15.95 | 0.01 | 28.76 |
67
+ | CodeT5+ | 0.02 | 7.24 | 0.12 | 9.87 | 0.00 | 12.33 |
68
+
69
+ - Fine-Tuned
70
+ | | Stmt. Comp. | Stmt. Comp. | Next. Sugg. | Next. Sugg. | Code. Gen. | Code. Gen. |
71
+ |-------------|:-----------------:|:-----------------:|:----------------:|:----------------:|:----------:|:----------:|
72
+ | **Model** | EM | ED | EM | ED | BLEU4 | ED |
73
+ | CodeBert | 53.84 | 77.44 | 52.67 | 70.82 | xxx | xxx |
74
+ | GraphCodeBert | 43.00 | 71.89 | 47.10 | 61.31 | xxx | xxx |
75
+ | UnixCoder | **67.84** | **85.06** | 58.51 | 75.31 | 56.24 | 73.45 |
76
+ | CodeT5 | 66.38 | 84.34 | xxx | xxx | xxx | xxx |
77
+ | NatGen | 67.47 | 84.83 | **60.30** | **76.84** | 71.73 | 81.39 |
78
+ | CodeT5+ | 66.93 | 84.45 | 59.57 | 76.41 | **75.28** | **82.95** |
79
+
80
+
81
+ - `New_Targets/All_Types/*`: **Take data of RISC-V,ARC,NVPTX both in GCC and LLVM as test set, split train/valid set in the ratio of 85%:15% of other 171(178 - 2*3 - 1) targets excluding RI5CY(RI5CY is custmoized based on RISCV)**
82
+
83
+
84
+ - Datset Info
85
+
86
+
87
+ | Task | Train | Valid | Test |
88
+ | ---- | ---- | ---- | ---- |
89
+ | Statement-Level Comp. | 114,016(10.20M Token) | 20,121(1.81M Token) | 6,645(0.58M Token) |
90
+ | Next-Statement Sugg. | 152,114(14.10M Token) | 26,844(2.49M Token) | 9,313(0.83M Token) |
91
+ | Code Generation. | 30,633(4.44M Token) | 5,406(0.79M Token) | 2,819(0.37M Token) |
92
+
93
+
94
+ We only fine-tuned CodeT5+ across three tasks, and compare it with ChatGPT-3.5-Turbo and Code-LLaMA-34B with similar input.
95
+
96
+ - GCC
97
+
98
+ | | Stmt. Comp. | Stmt. Comp. | Stmt. Comp. | Stmt. Comp. | Stmt. Comp. | Stmt. Comp. | Next. Sugg. | Next. Sugg. | Next. Sugg. | Next. Sugg. | Next. Sugg. | Next. Sugg. | Code. Gen. | Code. Gen. | Code. Gen. | Code. Gen. | Code. Gen. | Code. Gen. |
99
+ |----------|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|
100
+ | | RISC-V | RISC-V | ARC | ARC | NVPTX | NVPTX | RISC-V | RISC-V | ARC | ARC | NVPTX | NVPTX | RISC-V | RISC-V | ARC | ARC | NVPTX | NVPTX |
101
+ | Model | EM | ED | EM | ED | EM | ED | EM | ED | EM | ED | EM | ED | BLEU4 | ED | BLEU4 | ED | BLEU4 | ED |
102
+ | ChatGPT | 10.34 | 38.41 | 15.35 | 42.94 | 12.01 | 41.47 | 6.44 | 12.9 | 9.75 | 20.79 | 7.97 | 17.79 | 7.33 | 30.83 | 7.35 | 32.34 | 8.12 | 32.71 |
103
+ | Code-LLaMA | 0.41 | 19.07 | 0.85 | 16.77 | 0.56 | 18.22 | 1.58 | 13.54 | 2.66 | 17.95 | 2.47 | 16.59 | 9.38 | 35.53 | 11.06 | 37.15 | 8.24 | 33.00 |
104
+ | CodeT5+ | **51.16** | **75.32** | **52.45** | **74.57** | **50.56** | **75.52** | xxx | xxx | xxx | xxx | xxx | xxx | xxx | xxx | xxx | xxx | xxx | xxx |
105
+
106
+
107
+ - LLVM
108
+
109
+ | | Stmt. Comp. | Stmt. Comp. | Stmt. Comp. | Stmt. Comp. | Stmt. Comp. | Stmt. Comp. | Next. Sugg. | Next. Sugg. | Next. Sugg. | Next. Sugg. | Next. Sugg. | Next. Sugg. | Code. Gen. | Code. Gen. | Code. Gen. | Code. Gen. | Code. Gen. | Code. Gen. |
110
+ |----------|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|
111
+ | | RISC-V | RISC-V | ARC | ARC | NVPTX | NVPTX | RISC-V | RISC-V | ARC | ARC | NVPTX | NVPTX | RISC-V | RISC-V | ARC | ARC | NVPTX | NVPTX |
112
+ | Model | EM | ED | EM | ED | EM | ED | EM | ED | EM | ED | EM | ED | BLEU4 | ED | BLEU4 | ED | BLEU4 | ED |
113
+ | ChatGPT | 12.08 | 41.39 | 16.77 | 42.02 | 14.73 | 43.72 | 9.80 | 21.86 | 10.81 | 20.66 | 11.39 | 22.82 | 9.24 | 32.13 | 11.96 | 35.33 | 10.07 | 32.90 |
114
+ | Code-LLaMA | 0.45 | 17.61 | 0.61 | 17.21 | 0.99 | 17.23 | 1.75 | 15.04 | 0.42 | 11.27 | 2.42 | 16.25 | 6.92 | 32.54 | 8.95 | 38.22 | 8.20 | 34.16 |
115
+ | CodeT5+ | **62.68** | **82.02** | **71.34** | **85.98** | **64.45** | **81.53** | xxx | xxx | xxx | xxx | xxx | xxx | xxx | xxx | xxx | xxx | xxx | xxx |
116
+
117
+
118
+
119
+
120
+ - `New_Targets/CPU_Only/*`: **Take data of ARC,NVPTX both in GCC and LLVM as test set, split train/valid set in the ratio of 85%:15% of CPU targets excluding RISC-V and RI5CY**
121
+
122
+ - Datset Info
123
+
124
+
125
+ | Task | Train | Valid | Test |
126
+ | ---- | ---- | ---- | ---- |
127
+ | Statement-Level Comp. | 87,018(7.78M Token) | 15,357(1.37M Token) | 2,764(0.26M Token) |
128
+ | Next-Statement Sugg. | 113,684(10.65M Token) | 20,063(1.87M Token) | 4,029(0.38M Token) |
129
+ | Code Generation. | 21,184(3.14M Token) | 3,739(0.55M Token) | 1,372(0.18M Token) |
130
+
131
+ We only fine-tuned CodeT5+ across three tasks, and compare it with accuracy in `New_Targets/All_Types/*`.
132
+
133
+ - `New_Targets/Itr_Expansion/*`: **Take data of RI5CY in LLVM as test set, split train/valid set in the ratio of 85%:15% of CPU targets excluding RISC-V and including RISC-V**
134
+
135
+ - Datset Info
136
+ - Excluding RISC-V
137
+
138
+ | Task | Train | Valid | Test |
139
+ | ---- | ---- | ---- | ---- |
140
+ | Statement-Level Comp. | 87,018(7.78M Token) | 15,357(1.37M Token) | 721(0.04M Token) |
141
+ | Next-Statement Sugg. | 113,684(10.65M Token) | 20,063(1.87M Token) | 1,035(0.06M Token) |
142
+ | Code Generation. | 21,184(3.14M Token) | 3,739(0.55M Token) | 219(0.02M Token) |
143
+
144
+ - Including RISC-V
145
+
146
+ | Task | Train | Valid | Test |
147
+ | ---- | ---- | ---- | ---- |
148
+ | Statement-Level Comp. | 90,316(8.06M Token) | 15,940(1.42M Token) | 721(0.04M Token) |
149
+ | Next-Statement Sugg. | 118,175(11.04M Token) | 20,856(1.94M Token) | 1,035(0.06M Token) |
150
+ | Code Generation. | 22,413(3.30M Token) | 3,957(0.58M Token) | 219(0.02M Token) |
151
+
152
+ We only fine-tuned CodeT5+ across three tasks, and compare it with accuracy in `New_Targets/All_Types/*`.