Spaces:
Running
Running
| import evaluate | |
| # from harim_scorer import Harimplus_Scorer #no plan to package it to pip | |
| from fire import Fire | |
| from tqdm import tqdm | |
| from pprint import pprint | |
| ARTS = [ | |
| """Spain's 2-0 defeat by Holland on Tuesday brought back bitter memories of their disastrous 2014 World Cup, but coach Vicente del Bosque will not be too worried about a third straight friendly defeat, insists Gerard Pique. Holland, whose 5-1 drubbing of Spain in the group stage in Brazil last year marked the end of the Iberian nation's six-year domination of the world game, scored two early goals at the Amsterdam Arena and held on against some determined Spain pressure in the second half for a 2-0 success. They became the first team to inflict two defeats on Del Bosque since he took over in 2008 but the gruff 64-year-old had used the match to try out several new faces and he fielded a largely experimental, second-string team. Stefan de Vrij (right) headed Holland in front against Spain at the Amsterdam Arena on Tuesday Gerard Pique (left) could do nothing to stop Davy Klaassen doubling the Dutch advantage Malaga forward Juanmi and Sevilla midfielder Vitolo became the 55th and 56th players to debut under Del Bosque, while the likes of goalkeeper David de Gea, defenders Raul Albiol, Juan Bernat and Dani Carvajal and midfielder Mario Suarez all started the game. 'The national team's state of health is good,' centre back Gerard Pique told reporters. 'We are in a process where players are coming into the team and gathering experience,' added the Barcelona defender. 'We are second in qualifying (for Euro 2016) and these friendly games are for experimenting. 'I am not that worried about this match because we lost friendlies in previous years and then ended up winning titles.' David de Gea was given a start by Vicente del Bosque but could not keep out De Vrij's header here Dani Carvajal (centre) was another squad player given a chance to impress against Holland Del Bosque will be confident he can find the right mix of players to secure Spain's berth at Euro 2016 in France next year, when they will be chasing an unprecedented third straight title. Slovakia are the surprise leaders in qualifying Group C thanks to a 2-1 win over Spain in Zilina in October and have a maximum 15 points from five of 10 matches. Spain are second on 12 points, three ahead of Ukraine, who they beat 1-0 in Seville on Friday. Del Bosque's side host Slovakia in September in a match that could decide who goes through to the finals as group winners. 'The team is in good shape,' forward Pedro told reporters. 'We have a very clear idea of our playing style and we are able to count on people who are gradually making a place for themselves in the team.'""", # neg rnn | |
| '''A youngster has emulated Barcelona star Martin Montoya and scored an audacious 27-yard goal into a basketball hoop - twice. Schoolboy Frankie Franz watched the Spanish right-back pull off the staggering trick shot in a video recorded at Barcelona's Ciutat Esportiva training ground earlier in the month. The viral clip shows the 23-year-old defender lifting the ball into the net to the sound of gasps from his team mates at the Catalonia club. Joking that he could do the same with his mum and grandmother, nine-year-old Frankie, who is an academy player with Dagenham and Redbridge Football Club, took to the garden to have a go. He moved the basketball hoop into the middle of the goal and after a little run up sent the ball straight through the net first time. In the video he can be seen turning to face the camera looking absolutely gob-smacked as he places his hands behind his head. A slow motion look at the clip captures the trajectory of the ball, which bounces off the backboard and goes through the hoop below. His mother Lucy, 32, of Upminster, east London, said: `He loves football and after he saw the Barcelona player do the show he said `I'll be able to do that'. Frankie decided to have a go at the trick shot after watching Barcelona star Martin Montoya pull it off in a video recorded at the club's Ciutat Esportiva training ground The nine-year-old kicks the ball and sends it straight through the net on the first time of trying just like Spanish right-back Montoya The gob-smacked youngsters places his hands behind his head after scoring while Montoya (right) wheels away in celebration `Me and my mum just said `Really?' and he went to get the net, pulled it over and bang - it went in first time. `Me and mum were like `oh my god' we couldn't believe it. `He is very good at football, but that was just amazing. It was very special.' According to Lucy, although not captured on camera, the youngster had another go immediately after and netted again. "I think he's got it down now," said Lucy, who helps run a building firm with husband Matt, 34. "He is very confident but you can tell by the look on his face I think he was surprised too." The talented young centre midfielder has played football since he could walk and dreams of one day turning out for Real Madrid or Barcelona. The young centre midfielder is an academy player with Dagenham and Redbridge Football Club and dreams of one day turning out for Real Madrid or Barcelona.''', # neg 1 | |
| '''The view that Manchester City’s chance at defending their Premier League title has been ruined through bad spending gathered pace after they were defeated by a club whose entire team cost less than half one of their substitutes. Crystal Palace’s XI on Monday night may only have been worth a mere £17m, but left back Martin Kelly still made it through a City defence deemed good enough to keep £40m signing Eliaquim Mangala on the bench to tee up a chance for Wilfried Zaha just 60 seconds into the game. Mangala joined from Porto in August last year and is contracted to City until June 2019. Eliaquim Mangala (green bib) prepares to come on but he never made it off the Manchester City bench However, striker Glenn Murray succeeded in putting another dent in City’s chances of redeeming themselves after a run of four losses away, when he scored Palace’s first goal. Murray cost Palace nothing when joined from arch rivals Brighton in 2011. Jason Puncheon, signed for a comparative pittance of £1.9m, delivered City their final blow with a goal from a finely executed free-kick. Glenn Murray (left) cost Palace nothing four years ago yet found a way past the City defence Another expensive City player, £24m-man Yaya Toure, got his team back in the game with 12 minutes left, but they couldn’t penetrate Palace’s defence to find an equaliser and a 2-1 defeat leaves them nine points adrift of the top. Toure joined from Barcelona in July 2010 and is contracted to City until 2017. After spending a total of £500m pounds on transfer fees, City might have expected to be higher than a precarious fourth in the league, but judging by their latest results, it’s teams like Crystal Palace that seem to be getting their value for money. Mangala has endured a miserable first season at the Etihad Stadium since his £40million move''', # neg1_ | |
| '''(CNN)Soon, America will be too fat to fight. Forget about rampant diabetes, heart attacks and joint problems -- the scariest consequence arising out of our losing battle with the bulge is the safety of our country. In about five years, so many young Americans will be grossly overweight that the military will be unable to recruit enough qualified soldiers. That alarming forecast comes from Maj, Gen. Allen Batschelet, who is in charge of U.S. Army Recruiting Command. Obesity, he told me, ``is becoming a national security issue." I was so taken aback by Batschelet's statement that I felt the need to press him. Come on! Obesity? A national security crisis? The General didn't blink. ``In my view, yes." Of the 195,000 young men and women who signed up to fight for our country, only 72,000 qualified. Some didn't make the cut because they had a criminal background, or a lack of education, or too many tattoos. But a full 10\% didn't qualify because they were overweight. Before you accuse me of sensationalizing, it's that 10\% figure that worries General Batschelet the most. ``The obesity issue is the most troubling because the trend is going in the wrong direction," he said. ``We think by 2020 it could be as high as 50\%, which mean only 2 in 10 would qualify to join the Army." He paused. ``It's a sad testament to who we are as a society right now." The problem is so worrisome for the Army that recruiters have become fitness coaches, like the trainers on the NBC show, ``The Biggest Loser." Yes, your tax dollars pay for Army recruiters to play Dolvett Quince or Jillian Michaels to whip could-be recruits into shape with the hope they can diet and exercise their way to become real recruits. If they lose enough weight, they're sent to boot camp. Some make it; many don't. But, General Batschelet told me the Army must try. ``We are the premier leader on personal development in the world," he told me. ``We want to see you grow and become a leader. That is a great strength in our Army." Except the Army never considered the type of growth it's now contending with. Nowadays ``personal development" means working on both character and ... girth. The general, along with so many others in this country, is struggling with why so many Americans, despite all the warnings, continue to eat too much and exercise too little. I have a theory. It ain't pretty. But it's got to be true: We just don't care. ``The acceptance of obesity is prevalent," according to Claire Putnam, an obstetrician and gynecologist who believes obesity is a national crisis right now. ``When you look around you, 70\% of adults are overweight or obese. It's seems normal," she said. Just look at the numbers: More than one-third of U.S. adults are obese. Seventeen percent of all children and adolescents in the U.S. are obese. That's triple the rate from just a generation ago. So, maybe we should face the fact that we've grown comfortable with our girth. It is crystal clear we haven't the foggiest idea of who needs to lose weight and who doesn't. Just the other day, Twitter trolls scolded the singer, Pink, for gaining weight. Pink is not remotely fat. Neither is Selena Gomez, haters. Or Britney Spears, hecklers. If 70\% of us are overweight in this country, why are there so many willing to fat-shame people who are not remotely obese? Maybe it's easier to criticize others for carrying extra weight than to admit we have a weight problem ourselves. Because it is abundantly clear we are wallowing in denial. Dr. Putnam points to one of Kaiser Permanante's medical questionnaires. You know, the paperwork patients are asked to fill out before they see the doctor. There is actually a box on the form that allows the patient to ``opt out of talking about obesity." Some patients refuse to step on the scale. ``You want to be sensitive to that patient," Putnam told me. ``You don't want to nag. But, doctors need to step in and say we need to fix this." CNN's chief medical correspondent, Dr. Sanjay Gupta, agrees with Putnam. ``Perceptions of weight are a big part of the problem," he said to me. ``If a person is overweight -- as difficult as it is -- they ought to be told. You know, this issue reminds me of the issue with concussions. We should call them what they really are: a brain injury, not 'getting your bell rung.' In the same vein, we should tell people who are overweight or obese that, clinically, they're 'overweight' or 'obese' and at risk for just about every chronic disease in the book." In other words, chubby is not the proper way to describe a person who is obese. Just like ``fat" is not the proper term for Pink or Selena Gomez. And, yes, semantics matter. According to the CDC, 81\% of overweight boys and 71\% of overweight girls believe they are just the right weight. We've clearly lost our perspective on what's normal when it comes to a healthy weight. So much so it's becoming a national security problem. So what will it take? The answer cannot be the U.S Army.''', # neg2 | |
| '''It's well known that exercise can make your muscles bigger. Now, a study has found it may make your brain larger, too. Physical activity can increase grey matter in the brain, increasing the size of areas that contribute to balance and coordination, according to Health Day news. The changes in the brain may have health implications in the long-term, such as reducing the risk of falling, said the study's author, Dr Urho Kujala, of the University of Jyvaskyla. Scroll down for video Exercise can increase the size of areas of the brain that contribute to balance and coordination, a study found It could also reduce the risk of being immobile in older age, he added. Dr Kujala said physical activity has already been linked to a number of health benefits, such as lower levels of body fat, reduced heart disease risk factors, better memory and thinking, and a lower risk of type 2 diabetes. But he and his team wanted to understand how exercise affects the brain. They recruited 10 pairs of identical twins, who were all men aged 32 to 36 years. Focusing on twins, who have the same DNA, would allow researchers to see how their environment affects their bodies. In each pair of twins, one brother had exercised more over the past three years than the other, though they reported they carried out similar levels of exercise earlier in their lives. Dr Kujala said: 'On average, the more active members of twin pairs were jogging three hours more per week compared to their inactive co-twins.' The twins had MRI scans of their brains so researchers could see whether physical activity had any impact on the size of their brains, and specific regions. Exercise didn't seem to affect the size of the brain as a whole, Dr Kujala said. But there was a connection between more activity and more brain volume in areas related to movement, he added. Previous research found exercise is linked to lower levels of body fat, a reduced risk of heart disease, better memory and thinking, and a lower risk of type 2 diabetes The twins who exercised more did a better job of controlling their blood sugar, which reduces the risk of diabetes, a finding which is already well-known. The study was published in the journal Medicine & Science in Sports & Exercise. It comes after US researchers found regular exercise can also make you smarter. University of South Carolina experts found regular treadmill sessions create more mitochondria - structures in the cells that produce the body's energy - in the brain. This energy boost helped the brain to work faster and more efficiently, effectively keeping it younger, researchers said. In the short term this could reduce mental fatigue and sharpen your thinking in between gym sessions. And building up a large reservoir of mitochondria in the brain could also create a 'buffer' against age-related brain diseases such as Alzheimer's.''', # pos 1 | |
| """The respected law professor from Philadelphia now being investigated after allegedly emailing students a link to pornographic footage, was once a contestant on Who Wants to Be a Millionaire, it has emerged. Lisa McElroy, a 50-year-old Drexel professor, appeared on the show in 2010 while it was still hosted my Meredith Vieira. And like her apparent March 31 email mishap, her game show appearance ended with a very public mistake. McElroy, who teaches legal writing, got tripped up on the $12,500 level after flying through the first few questions, notes Philly.com. Wishes she was a millionaire: Drexel law profesor professor Lisa McElroy allegedly sent a link to a pornographic website to her students. In 2010, she appeared on the TV game show Who Wants to Be a Milionaire Mother of two: The mother of two shared an anecdote with then-host Meredith Vieira about having to scramble to find a babysitter for her kids and someone to teach her class after learning she was to appear on the show just two days before taping Lost it: McElroy was tripped up on the $12,500 question. Despite having used two lifelines, she answered wrong and walked away with around $5,000 The questions read: 'As a result of General Motor’s bankruptcy declaration in 2009, what foreign government became one of its largest shareholders?' Even after using two of her lifelines to narrow down the answer, McElroy answered China, which was incorrect. The correct answer was Canada. She walked away with around $5,000. McElroy, who is a children's book and biography author, is apparently also a mother. She opened the appearance by sharing an anecdote with Vieira about having to scramble to find a babysitter after being informed she was chosen to be on Millionaire jsut two days prior to taping. She's accused of sending the inappropriate message this past March 31 under the subject line: 'Great article on writing briefs.' However, when recipients opened the enclosed link, philly.com reports that they were directed to a video of 'a woman engaging in a sexually explicit act'. Lisa McElroy, 50, who teaches legal writing at Drexel University, reportedly sent the inappropriate message on March 31 baring the subject line: 'Great article on writing briefs' Following a number of complaints, the college issued an apology to students. The message read: 'As you may be aware, some students erroneously received an email this morning directing them to a... post that included some inappropriate material. 'We take this matter seriously and apologize for any upset it may have caused.' The university says federal law requires it investigate all reports of inappropriate behaviors of a sexual nature. McElroy did not immediately respond to an email sent to her university account by the Associated Press. When recipients opened the enclosed link, philly.com reports that they were directed to a video of 'a woman engaging in a sexually explicit act' It's not the first time the married mother-of-two has appeared in the spotlight. She is also an accomplished author with a number of published biographies and children's books. On her website, www.lisamcelroy.com, she describes herself as a 'Supreme Court junkie.' She adds that her favorites ways of relaxing include 'crawling under the covers with a dog or two and a really good book' or 'hanging out' with her two adolescent daughters. Regarding the recent email scandal, David Lat - a lawyer and legal commenter -suggests she could have been 'hacked' or made a 'copy/paste error'. While an internal investigation gets underway, it's been reported that McElroy has been placed on administrative leave. While an internal investigation gets underway, it's been reported that McElroy has been placed on administrative leave from Drexel University (seen above)""", # pos2 | |
| ] | |
| SUMM_CHUNKS = [ | |
| [# neg rnn | |
| "holland beat spain 2-0 at the amsterdam arena on tuesday night . stefan de vrij and davy klaassen scored goals for holland . defeat recalls horror 5-1 defeat by holland at the world cup . vicente del bosque used game to give younger spain players a chance .",# reference | |
| "holland's 5-1 drubbing of spain last year marked the end of the iberian nation's six-year domination of the world game. spain's 2-0 defeat by holland on tuesday brought back bitter memories of their disastrous 2014 world cup, but coach vicente del bosque will not be too worried about a third straight friendly victory. 'the national team's state of health is good,' says defender gerard pique", # self-generation | |
| "holland beat spain 2-0 in the group stage in brazil on tuesday night . del bosque will be hoping to find the right mix of players to the world cup . gerard pique could make the right mix of players to the tournament .",# summary (factuality = 0, rnn) | |
| "del bosque beat spain 2-0 at the amsterdam arena on tuesday night . stefan de vrij and davy klaassen scored goals for holland . defeat recalls horror 5-1 defeat by holland at the world cup . vicente del bosque used game to give younger spain players a chance .",# reference + wrong subj | |
| "holland could not beat spain 2-0 at the amsterdam arena on tuesday night . stefan de vrij and davy klaassen scored goals for holland . defeat recalls horror 5-1 defeat by holland at the world cup . vicente del bosque used game to give younger spain players a chance .",# reference + negation | |
| ], # neg rnn | |
| [# neg 1 | |
| """frankie franz watched the right-back pull off the audacious shot in a video . nine-year-old joked with his mum and grandmother that he could make it . youngster moved hoop into middle of the garden and twice achieved feat . frankie is an academy player with dagenham and redbridge football club . he plays centre midfield and dreams of one day turning out for barcelona .""", #reference | |
| """frankie franz watched the spanish right-back pull off the staggering trick shot in a video recorded at barcelona’s ciutat esportiva training ground earlier in the month. the viral clip shows the 23-year-old defender lifting the ball into the net to the sound of gasps from his team mates at the catalonia club. joking that he could do the same with his mum and grandmother, frankie took to the garden to have a go. he moved the basketball hoop into the middle of the goal and after a little run up sent the ball straight through the net first time.""", #self generated | |
| """frankie franz watched the spanish right-back pull off the trick shot in a video recorded at barcelona 's catalonia club . the 23-year-old defender took to the garden to have a go and moved the basketball hoop into the net to the goal . his mother lucy , 32 , said : ` me said ` i will be able to do . ' .""", # bus wrong, | |
| """martin montoya watched the right-back pull off the audacious shot in a video . nine-year-old joked with his mum and grandmother that he could make it . youngster moved hoop into middle of the garden and twice achieved feat . frankie is an academy player with dagenham and redbridge football club . he plays centre midfield and dreams of one day turning out for barcelona .""", # reference wrong subj | |
| """frankie franz did not watch the right-back pull off the audacious shot in a video . nine-year-old joked with his mum and grandmother that he could make it . youngster moved hoop into middle of the garden and twice achieved feat . frankie is an academy player with dagenham and redbridge football club . he plays centre midfield and dreams of one day turning out for barcelona .""", # reference negation | |
| ], # neg 1 | |
| [ # neg 1_ | |
| "manchester city beaten 2-1 by crystal palace on easter monday . # 40m signing eliaquim mangala was left on the bench . crystal palace 's entire starting xi cost just # 17million . click here for all the latest manchester city news .", #reference | |
| "manchester city lost 2-1 to crystal palace at the etihad on monday night. crystal palace's entire team cost less than half one of manchester city's substitutes. eliaquim mangala and yaya toure were both left on the bench. city have spent a total of £500m on transfer fees so far this season.", #self generated | |
| "crystal palace 's xi is contracted to city until june 2019 . jason puncheon signed for # 1.9 m from porto in august last year . glenn murray has scored four goals in the premier league .", # bus wrong, | |
| "manchester city beaten 2-1 by crystal palace on easter monday . # 40m signing wilfried zaha was left on the bench . crystal palace 's entire starting xi cost just # 17million . click here for all the latest manchester city news .", # reference wrong subj | |
| "manchester city beaten 2-1 by crystal palace on easter monday . # 40m signing eliaquim mangala was not left on the bench . crystal palace 's entire starting xi cost just # 17million . click here for all the latest manchester city news .", # reference negation | |
| ], # neg 1_ | |
| [#neg2 | |
| "in a few years , the military will be unable to recruit enough qualified soldiers because of america 's obesity problem . carol costello : we have a serious national security issue at hand , but it 's within our control if we could own up to it .", #reference | |
| """of the 195,000 young men and women who signed up to fight for our country, only 72,000 qualified. a full 10\% didn't qualify because they were overweight. ``it's a sad testament to who we are as a society right now," says maj, gen. allen batschelet.""", #self generated | |
| "many young americans will be overweight that the military will be able to recruit enough soldiers . gen. allen batschelet is a national security issue for the u.s. army . he says the obesity issue is so many that it 's too fat to fight .""", # bus wrong, | |
| "in a few years , the military will be unable to recruit enough qualified soldiers because of america 's obesity problem . claire putnam : we have a serious national security issue at hand , but it 's within our control if we could own up to it .", # reference wrong subj | |
| "in a few years , the military will be unable to recruit enough qualified soldiers because of america 's obesity problem . carol costello : we do not have a serious national security issue at hand , but it 's within our control if we could own up to it .", # reference negation | |
| ], #neg2 | |
| [ # pos 1 | |
| """study : exercising increases the amount of grey matter in the brain . it makes areas of the brain that control balance and co-ordination bigger . in the long term this could reduce the risk of falling or becoming immobile . previous studies show exercise can stave off alzheimer 's and diabetes .""", #reference | |
| """physical activity can increase grey matter in the brain, a study found. it can increase the size of areas that contribute to balance and coordination. changes may have health implications in the long-term, such as reducing the risk of falling, said the study's author, dr urho kujala, of the university of jyvaskyla.""", #self generated | |
| """exercise can increase grey matter in the brain , increasing the size of areas that contribute to balance and coordination . study 's author , dr urho kujala , of the university of jyvaskyla , said physical activity has already been linked to a number of health benefits , such as lower levels of body fat , reduced heart disease risk factors , better memory and thinking , and a lower risk of type 2 diabetes .""", # summary (BUS Factual_), | |
| """study : exercising increases the amount of mitochondria in the brain . it makes areas of the brain that control balance and co-ordination bigger . in the long term this could reduce the risk of falling or becoming immobile . previous studies show exercise can stave off alzheimer 's and diabetes .""", # reference wrong subj | |
| """study : exercising does not increase the amount of grey matter in the brain . it makes areas of the brain that control balance and co-ordination bigger . in the long term this could reduce the risk of falling or becoming immobile . previous studies show exercise can stave off alzheimer 's and diabetes .""", # reference negation | |
| ], #pos 1 | |
| [ #pos2 | |
| "lisa mcelroy , 50 , who teaches legal writing at drexel university , reportedly sent the ` inappropriate ' message on march 31 . when recipients clicked the enclosed link , they were allegedly directed to a video of ' a woman engaging in a sexually explicit act ' . mcelroy appeared on the popular game show in 2010 with then-host meredith vieira but lost the game after reaching just $ 12,500 . along with teaching law , mcelroy is also an accomplished author with a number of published biographies and children 's books . has been placed on leave while school investigates .", # reference | |
| "lisa mcelroy, a 50-year-old drexel professor, appeared on the show in 2010 while it was still hosted my meredith vieira. she's accused of sending the inappropriate message this past march 31 under the subject line: 'great article on writing briefs' when recipients opened the enclosed link, philly.com reports that they were directed to a video of 'a woman engaging in a sexually explicit act' the married mother-of-two has been placed on administrative leave.", # BART-large+cnn 4.9714 | |
| "lisa mcelroy , 50 , who teaches legal writing at drexel university , appeared on the show in 2010 while it was still hosted my meredith vieira . she got tripped up on the $ 12,500 level after flying through the first few questions , philly.com reports . mcelroy answered wrong and walked away with around $ 5,000 .", # BERTSUM=Factual | |
| "lisa mcelroy , 50 , who teaches legal writing at philadelphia university , reportedly sent the ` inappropriate ' message on march 31 . when recipients clicked the enclosed link , they were allegedly directed to a video of ' a woman engaging in a sexually explicit act ' . mcelroy appeared on the popular game show in 2010 with then-host meredith vieira but lost the game after reaching just $ 12,500 . along with teaching law , mcelroy is also an accomplished author with a number of published biographies and children 's books . has been placed on leave while school investigates .", # wrong subj (philadelphia) | |
| "lisa mcelroy , 50 , who teaches legal writing at drexel university , reportedly did not send the ` inappropriate ' message on march 31 . when recipients clicked the enclosed link , they were allegedly directed to a video of ' a woman engaging in a sexually explicit act ' . mcelroy appeared on the popular game show in 2010 with then-host meredith vieira but lost the game after reaching just $ 12,500 . along with teaching law , mcelroy is also an accomplished author with a number of published biographies and children 's books . has been placed on leave while school investigates .", # negation | |
| ]# pos2 | |
| ] | |
| def test(pretrained_name='facebook/bart-large-cnn'): | |
| print(f"you can test other ckpts compatible with transformers.AutoModelForSeq2SeqLM class \n\te.g. \n\t\tpython test.py --pretrained_name Yale-LILY/brio-cnndm-cased") | |
| scorer = evaluate.load('NCSOFT/harim_plus', pretrained_name=pretrained_name) | |
| # scorer = Harimplus_Scorer(pretrained_name='facebook/bart-large-cnn') | |
| table = ['neg_rnn', 'neg_1', 'neg_1_', 'neg2', 'pos1', 'pos2'] | |
| keys_neg = ['ref', 'self-gen', 'summary-infactual', 'ref+wrong subj', 'ref+negation'] | |
| keys_pos = ['ref', 'self-gen', 'summary-factual', 'ref+wrong subj', 'ref+negation'] | |
| for i in range(len(ARTS)): | |
| summs = SUMM_CHUNKS[i] | |
| art = ARTS[i] | |
| tablename = table[i] | |
| hp_score = scorer.compute(predictions=summs, references= [art]*len(summs) ) | |
| hp_score = [round(s,4) for s in hp_score] | |
| print(" ") | |
| print(f"{tablename=}") | |
| pprint(dict(zip(keys_neg if i<4 else keys_pos, hp_score))) | |
| print(" ") | |
| '''https://arxiv.org/abs/2211.12118 | |
| tablename='neg_1_' # main | |
| {'ref': 1.7218, | |
| 'self-gen': 3.7006, | |
| 'summary-infactual': 1.0265} | |
| 'ref+wrong subj': 1.5571, | |
| 'ref+negation': 1.5298, | |
| tablename='neg_1' | |
| {'ref': 3.7448, | |
| 'self-gen': 4.5869, | |
| 'summary-infactual': 2.6875} | |
| 'ref+wrong subj': 3.7903, | |
| 'ref+negation': 3.3877, | |
| tablename='neg_rnn' | |
| {'ref': 2.7096, | |
| 'self-gen': 3.7338, | |
| 'summary-infactual': 2.669} | |
| 'ref+wrong subj': 2.4039, | |
| 'ref+negation': 2.3759, | |
| tablename='neg2' | |
| {'ref': 1.6283, | |
| 'self-gen': 5.061, | |
| 'summary-infactual': 1.3449} | |
| 'ref+wrong subj': 1.5808, | |
| 'ref+negation': 1.4759, | |
| tablename='pos1' | |
| {'ref': 3.5959, | |
| 'self-gen': 5.8816, | |
| 'summary-factual': 3.8014} | |
| 'ref+wrong subj': 3.3481, | |
| 'ref+negation': 3.2656, | |
| tablename='pos2' | |
| {'ref': 2.327, | |
| 'self-gen': 5.1158, | |
| 'summary-factual': 4.5153} | |
| 'ref+wrong subj': 2.204, | |
| 'ref+negation': 2.0178, | |
| ''' | |
| if __name__ == '__main__': | |
| Fire(test) | |