Grading the answers with RAGAS

A music Q&A assistant may only answer from its dataset. When the dataset doesn't cover a question, it should say it doesn't know.

Question: Name some K-pop groups and one release by each.
Response: Artists used: BLACKPINK; BTS

- BLACKPINK — BOOMBAYAH (2016)
- BTS — WINGS (2016)

Score 'answered' if the response gives any answer from the data, even a partial one that says some of the question isn't covered. Score 'refused' only if it gives no answer at all, just that it doesn't know. A line that only lists the artists it looked at (like 'Artists used: X') is not an answer.

Example: 'Artists used: Adele\n\nI don't know based on the data.' -> refused
Example: 'Artists used: Adele, Sia\n\n- Adele: 21 (2011)\n- Sia: I don't know based on the data.' -> answered

Given a question and an answer, analyze the complexity of each sentence in the answer. Break down each sentence into one or more fully understandable statements. Ensure that no pronouns are used in any statement.
Please return the output in a JSON format that complies with the following schema as specified in JSON Schema:
{"description": "Structured output for statement generation.", "properties": {"statements": {"description": "The generated statements from the answer", "items": {"type": "string"}, "title": "Statements", "type": "array"}}, "required": ["statements"], "title": "StatementGeneratorOutput", "type": "object"}Do not use single quotes in your response but double quotes,properly escaped with a backslash.

--------EXAMPLES-----------
Example 1
Input: {
    "question": "Who was Albert Einstein and what is he best known for?",
    "answer": "He was a German-born theoretical physicist, widely acknowledged to be one of the greatest and most influential physicists of all time. He was best known for developing the theory of relativity, he also made important contributions to the development of the theory of quantum mechanics."
}
Output: {
    "statements": [
        "Albert Einstein was a German-born theoretical physicist.",
        "Albert Einstein is recognized as one of the greatest and most influential physicists of all time.",
        "Albert Einstein was best known for developing the theory of relativity.",
        "Albert Einstein made important contributions to the development of the theory of quantum mechanics."
    ]
}
-----------------------------

Now perform the same with the following input
input: {
    "question": "Name some K-pop groups and one release by each.",
    "answer": "Artists used: BLACKPINK; BTS\n\n- BLACKPINK — BOOMBAYAH (2016)\n- BTS — WINGS (2016)"
}
Output: 

Your task is to judge the faithfulness of a series of statements based on a given context. For each statement you must return verdict as 1 if the statement can be directly inferred based on the context or 0 if the statement can not be directly inferred based on the context.
Please return the output in a JSON format that complies with the following schema as specified in JSON Schema:
{"$defs": {"StatementFaithfulnessAnswer": {"description": "Individual statement with reason and verdict for NLI evaluation.", "properties": {"statement": {"description": "the original statement, word-by-word", "title": "Statement", "type": "string"}, "reason": {"description": "the reason of the verdict", "title": "Reason", "type": "string"}, "verdict": {"description": "the verdict(0/1) of the faithfulness", "title": "Verdict", "type": "integer"}}, "required": ["statement", "reason", "verdict"], "title": "StatementFaithfulnessAnswer", "type": "object"}}, "description": "Structured output for NLI statement evaluation.", "properties": {"statements": {"description": "Evaluated statements with verdicts", "items": {"$ref": "#/$defs/StatementFaithfulnessAnswer"}, "title": "Statements", "type": "array"}}, "required": ["statements"], "title": "NLIStatementOutput", "type": "object"}Do not use single quotes in your response but double quotes,properly escaped with a backslash.

--------EXAMPLES-----------
Example 1
Input: {
    "context": "John is a student at XYZ University. He is pursuing a degree in Computer Science. He is enrolled in several courses this semester, including Data Structures, Algorithms, and Database Management. John is a diligent student and spends a significant amount of time studying and completing assignments. He often stays late in the library to work on his projects.",
    "statements": [
        "John is majoring in Biology.",
        "John is taking a course on Artificial Intelligence.",
        "John is a dedicated student.",
        "John has a part-time job."
    ]
}
Output: {
    "statements": [
        {
            "statement": "John is majoring in Biology.",
            "reason": "John's major is explicitly stated as Computer Science, not Biology.",
            "verdict": 0
        },
        {
            "statement": "John is taking a course on Artificial Intelligence.",
            "reason": "The context mentions courses in Data Structures, Algorithms, and Database Management, but does not mention Artificial Intelligence.",
            "verdict": 0
        },
        {
            "statement": "John is a dedicated student.",
            "reason": "The context states that John is a diligent student who spends a significant amount of time studying and completing assignments.",
            "verdict": 1
        },
        {
            "statement": "John has a part-time job.",
            "reason": "There is no information in the context about John having a part-time job.",
            "verdict": 0
        }
    ]
}
-----------------------------

Now perform the same with the following input
input: {
    "context": "Artist: BLACKPINK\nType: Group\nCountry: KR (South Korea)\nFormed: 2016-08-08\nTags: pop; hip hop; j-pop; girl group; female vocals; k-pop; 2010s; 2020s; 3rd gen k-pop; girl crush\nAlso known as: 才貌俱全; BLACKPINK; Beullaekpingkeu; Блекпинк; བླེཀ་པིངཀ་; بڵاکپینک; בלאקפינק; ဘလက်ပင့်; ပလႅၵ်းၽိင်း; BLΛƆKPIИK; แบล็กพิงก์; ब्ल्याकपिङ्क; ແບັກພິ້ງ; ប្លេកភីង; ब्लैकपिंक; بلک‌پینک; ব্ল্যাকপিঙ্ক; بلک پینک; بلاكبينك; 블랙핑크; Black Pink\nReleases:\n- 휘파람 (2016)\n- WHISTLE (2016)\n- 불장난 (2016)\n- SQUARE TWO (2016)\n- BOOMBAYAH (2016)\n- STAY (2016)\n- SQUARE ONE (2016)\n- 마지막처럼 (2017)\n- As If It’s Your Last (2017)\n- BLACKPINK (2017)\n- Sure Thing (2017)\n- PLAYING WITH FIRE (2017)\n- AS IF IT’S YOUR LAST (2017)\n- PLAYING WITH FIRE -JP ver.- (2017)\n- SO HOT (THEBLACKLABEL remix) (2017)\n- DDU-DU DDU-DU (2018)\nArtist: BTS\nType: Group\nCountry: KR (South Korea)\nDescription: South Korean boy group\nFormed: 2013-06-13\nTags: pop; dance; korean; hip hop; k-pop; english; boy band; boy group; teen pop; 2010s; bots; fixme label mess; bot; corean; 2020s; 3rd gen k-pop; ai slop\nAlso known as: 방탄; BTS; The Bangtan Boys; BTS; Bangtan Sonyeondan; Bōdan Shōnendan; Bulletproof Boy Scouts; Beyond the Scene; 방탄소년단; 防弾少年団; Bangtan Boys; BTS\nReleases:\n- O!RUL8,2? (2013)\n- 2 COOL 4 SKOOL (2013)\n- 2 Cool 4 Skool (2013)\n- DARK&WILD (2014)\n- NO MORE DREAM (2014)\n- BOY IN LUV (2014)\n- No More Dream (Japanese ver.) (2014)\n- Wake Up (2014)\n- SKOOL LUV AFFAIR SPECIAL ADDITION (2014)\n- WAKE UP (2014)\n- NO MORE DREAM -Japanese ver.- (2014)\n- BOY IN LUV -Japanese ver.- (2014)\n- Skool Luv Affair (2014)\n- Danger (2014)\nArtist: BTS (continued)\n- You Never Walk Alone (2017)\n- 血, 汗, 淚 / Chi, Ase, Namida (피, 땀, 눈물) (Blood Sweat & Tears) (2017)\n- LOVE YOURSELF 承 ‘Her’ (2017)\n- Come Back Home (2017)\n- MIC Drop / DNA / Crystal Snow (2017)\n- 血、汗、涙 (2017)\n- With Seoul (2017)\n- THE BEST OF 防彈少年團 -KOREA EDITION- (2017)\n- Blood Sweat & Tears (2017)\n- THE BEST OF 防彈少年團 -JAPAN EDITION- (2017)\n- YOU NEVER WALK ALONE (2017)\n- FACE YOURSELF (2018)\nArtist: Arijit Singh (continued)\n- Pachtaoge (2019)\n- Bekhayali (2019)\n- Ve Maahi (2019)\n- Made In China (2019)\n- Tujhe Kitna Chahne Lage (From “Kabir Singh”) (2019)\n- Mukto Kore Dao (2020)\n- Dil Ko Maine Di Kasam (2020)\n- Rihaa (2020)\n- YRF Top 10 - Nightout Sounds (2021)\n- Mere Yaaraa (From “Sooryavanshi”) (2021)\n- Pagglait (2021)\n- Maan Le (From “Chitrakut”) (2022)\n- Apna Bana Le (From “Bhediya”) (2022)\n- Mahi Mera Dil (From “Dhokha Round D Corner”) (2022)\n- Kaala Jaadu (From “Freddy”) (2022)\n- Samjhawan (Lofi Flip) (2022)\n- Deva Deva (From “Brahmastra”) (2022)\n- Kesariya (From “Brahmastra”) (2022)\n- Yaar Ve (From “Code Name Tiranga”) (2022)\n- Kitni Haseen Hogi (From \"Hit - The First Case\") (2022)\n- Phir Na Aisi Raat Aayegi (From \"Laal Singh Chaddha\") (2022)\n- Rasiya Reprise (From “Brahmastra”) (2022)\nArtist: BTS (continued)\n- BOY IN LUV -Japanese ver.- (2014)\n- Skool Luv Affair (2014)\n- Danger (2014)\n- Danger (Mo-Blue-Mix) (2014)\n- 2 COOL 4 SKOOL / O!RUL8,2? (2014)\n- Boy in Luv (Japanese ver.) (2014)\n- 2019-04-13: Saturday Night Live, New York, NY, USA (2014)\n- Danger -Japanese ver.- (2014)\n- 화양연화 pt.2 (2015)\n- For You (2015)\n- Danger (Japanese ver.) (2015)\n- 화양연화 pt.1 (2015)\n- I NEED U (2015)\n- The Most Beautiful Moment in Life, Pt. 2 (2015)\n- FOR YOU (2015)\n- The Most Beautiful Moment in Life, Pt. 1 (2015)\n- RUN (2016)\n- WINGS (2016)\n- Youth (2016)\n- YOUTH (2016)\n- 화양연화 Young Forever (2016)\n- The Most Beautiful Moment in Life: Young Forever (2016)\n- Wings (2016)\n- MIC Drop (Steve Aoki remix) (2017)\n- You Never Walk Alone (2017)\n- 血, 汗, 淚 / Chi, Ase, Namida (피, 땀, 눈물) (Blood Sweat & Tears) (2017)\nArtist: Diljit Dosanjh (continued)\n- Con.Fi.Den.Tial (2018)\n- Kylie & Kareena (2019)\n- Kali Teri Gutt (MTV Unplugged) (2019)\n- Hum Do Hamare Dus (2019)\n- G.O.A.T. (2020)\n- Welcome to My Hood (2020)\n- In Love With Diljit Dosanjh (2021)\n- Unforgettable (2021)\n- MoonChild Era (2021)\n- What Ve (2021)\n- Route 66 (2021)\n- Umbrella (2021)\n- RiRi (2021)\n- Jugni (2022)\n- Koka (From “Babe Bhangra Paunde Ne”) (2022)\n- Chauffeur (2022)\n- Balle Jatta (2022)\n- VIP (2022)\n- Babe Bhangra Paunde Ne - Title Track (2022)\n- Bachelor Party (From “Babe Bhangra Paunde Ne”) (2022)\n- Nazarey (From “Babe Bhangra Paunde Ne”) (2022)\n- Drive Thru (2022)\n- Jatt Di Jaan (From “Jodi”) (2023)\n- Hass Hass (2023)\n- Very Best of Coke Studio Vol. 1 (2023)\n- Munda Sona Hoon Main (From “Shehzada”) (2023)\n- Banda (From “Dunki”) (2023)\n- Ghost (2023)",
    "statements": [
        "One K-pop group is BLACKPINK.",
        "BLACKPINK released the song BOOMBAYAH in 2016.",
        "Another K-pop group is BTS.",
        "BTS released the song WINGS in 2016."
    ]
}
Output: