Revesis/setfit-mpnet-nli-rag-hallucination

SetFit with lizchu414/mpnet-base-all-nli-triplet

This is a SetFit model trained on the Revesis/rag_truth_hallucination_binary dataset that can be used for Text Classification. This SetFit model uses lizchu414/mpnet-base-all-nli-triplet as the Sentence Transformer embedding model. A SetFitHead instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a Sentence Transformer with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

• Model Type: SetFit
• Sentence Transformer body: lizchu414/mpnet-base-all-nli-triplet
• Classification head: a SetFitHead instance
• Maximum Sequence Length: 384 tokens
• Number of Classes: 2 classes
• Training Dataset: Revesis/rag_truth_hallucination_binary

Model Sources

• Repository: SetFit on GitHub
• Paper: Efficient Few-Shot Learning Without Prompts
• Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
0	"Premise: GameStop Horror: Employee Faces Charges for Stabbing Customer During Argument\nA seemingly routine visit to a GameStop store in Potomac Mills, Virginia, took a shocking turn when an employee allegedly stabbed a customer during an argument. The incident left both the victim and the accused facing dire consequences, shaking the community and raising questions about in-store safety.\nThe incident unfolded when an argument between a GameStop employee and a customer escalated to a horrifying level. The employee, identified as a D.C. resident, Courtney, reportedly stabbed the customer during the dispute. The victim suffered a non-life-threatening injury and was promptly provided with first aid by officers before being transported to an area hospital for further treatment.\nLaw enforcement swiftly arrived on the scene and took Courtney into custody without further incident. Thankfully, there were no additional injuries reported during the incident.\nThe seriousness of the alleged crime led to police charging Courtney with aggravated malicious wounding. The specifics of any bond arrangements were not immediately available at the time of reporting, leaving questions about Courtney's legal status unanswered.\nThis incident underscores the importance of maintaining a safe and respectful environment for both employees and customers in public spaces, such as retail stores. The GameStop incident serves as a reminder that disagreements can sometimes escalate to levels of violence, and the importance of maintaining composure and finding non-violent ways to resolve conflicts. It also highlights the essential role of law enforcement in promptly responding to such incidents to protect the safety of the public.\nAs the legal process unfolds, the Potomac Mills community will be closely following this case, hoping for justice and measures to ensure the safety of all individuals in public spaces. This incident reminds us that maintaining respectful and safe interactions is crucial in our day-to-day lives, and violence should never be an option for dispute resolution. Hypothesis: An employee at a GameStop store in Virginia stabbed a customer during an argument, leading to charges of aggravated malicious wounding. The victim suffered non-life-threatening injuries and was taken to a hospital. The employee, a D.C. resident named Courtney, was taken into custody. The incident highlights the need for maintaining a safe and respectful environment in public spaces and the importance of non-violent conflict resolution." "Premise: When a drug user feels safe talking with you, they may explain honestly about what compels them to use their drug. There is such a variety of the types or classes of drugs. Some are stimulants while others help one relax.\n\nPrecautions. Cholinergic drugs should be avoided when the patient has any sort of obstruction in the urinary or digestive tracts, such a a tumor, or severe inflammation which is causing blockage. They should be used with caution in patients with asthma, epilepsy, slow heart beat, hyperthyroidism, or gastric ulcers.\n\nHow to Understand Why People Use Drugs. If there are people around, then there are drug users around. It is simple as that, people have been using drugs to get high since prehistoric times, and they aren't going to stop anytime soon. Chances are you might be addicted to a drug, like nicotine or caffeine. Hypothesis: Unable to answer based on given passages."
1	'Premise: {'name': "Crazy Jim's Tacos Y Más", 'address': '505 State St', 'city': 'Santa Barbara', 'state': 'CA', 'categories': 'Tex-Mex, Mexican, Tacos, Restaurants', 'hours': {'Monday': '11:0-20:0', 'Tuesday': '11:0-20:0', 'Wednesday': '11:0-20:0', 'Thursday': '11:0-20:0', 'Friday': '11:0-21:0', 'Saturday': '11:0-21:0', 'Sunday': '11:0-20:0'}, 'attributes': {'BusinessParking': None, 'RestaurantsReservations': None, 'OutdoorSeating': True, 'WiFi': 'free', 'RestaurantsTakeOut': True, 'RestaurantsGoodForGroups': True, 'Music': None, 'Ambience': None}, 'business_stars': 4.0, 'review_info': [{'review_stars': 2.0, 'review_date': '2021-05-21 18:02:15', 'review_text': "Really overpriced, greasy fried tacos. There is a good amount of meat inside but the meat isn't so tasty. We really had no other options so had to eat this. Left me feeling bloated and unsatisfied. Would not recommend this place."}, {'review_stars': 4.0, 'review_date': '2021-03-31 19:59:01', 'review_text': "I decided to try this spot today. I ordered a rajas burrito. Had crazy Flavor! \n\nThe menu is well rounded. From burgers to Nachos. All the food looked pretty good coming out. My burrito came with Gauc and sour cream. \n\nI asked for the spiciest salsa. She gave all 3 to try. I'd have to say the red/orange Salsa had the best flavor and was the spiciest. Not really over the top. All were good. \n\nWas a beautiful day to sit outside in SB."}, {'review_stars': 5.0, 'review_date': '2021-03-06 21:28:05', 'review_text': "Crazy Jim's made you makes you feel good crazy. Don't miss this amazing tacos especially the fish taco and the L Pastore taco and steak and shrimp taco and the Rahas tacos. All amazing with a negro modelo."}]}\nOverview: Hypothesis: Crazy Jim's Tacos Y Más is a popular Tex-Mex and Mexican restaurant located in Santa Barbara, CA. With a 4.0-star rating and over 100 reviews, it's clear that locals and visitors alike enjoy dining at this establishment. The menu offers a variety of options, including tacos, burritos, nachos, and burgers, ensuring that there's something for everyone.\n\nOne reviewer praised the flavorful food, specifically mentioning the rajas burrito and the spiciest salsa, which they described as having the best flavor. Another reviewer enjoyed their fish taco, L Pastore taco, steak and shrimp taco, and Rahas tacos, all of which they found to be amazing. However, one customer had a less positive experience, finding the tacos to be overpriced and the meat to be unimpressive.\n\nCrazy Jim's Tacos Y Más offers outdoor seating, free Wi-Fi, and takeout services, making it a convenient and comfortable spot for a casual meal. The restaurant also has a relaxed atmosphere, perfect for groups or families. While some customers have noted that the prices are a bit high, the quality of the food and the variety of options make it a great choice for those looking for delicious Tex-Mex and Mexican cuisine in the area.'

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Revesis/setfit-mpnet-nli-rag-hallucination")
# Run inference
preds = model("Premise: When a drug user feels safe talking with you, they may explain honestly about what compels them to use their drug. There is such a variety of the types or classes of drugs. Some are stimulants while others help one relax.

Precautions. Cholinergic drugs should be avoided when the patient has any sort of obstruction in the urinary or digestive tracts, such a a tumor, or severe inflammation which is causing blockage. They should be used with caution in patients with asthma, epilepsy, slow heart beat, hyperthyroidism, or gastric ulcers.

How to Understand Why People Use Drugs. If there are people around, then there are drug users around. It is simple as that, people have been using drugs to get high since prehistoric times, and they aren't going to stop anytime soon. Chances are you might be addicted to a drug, like nicotine or caffeine. Hypothesis: Unable to answer based on given passages.")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	152	318.3333	434

Label	Training Sample Count
0	2
1	1

Training Hyperparameters

• batch_size: (16, 16)
• num_epochs: (2, 2)
• max_steps: -1
• sampling_strategy: unique
• num_iterations: 1
• body_learning_rate: (1e-05, 1e-05)
• head_learning_rate: 0.01
• loss: BatchAllTripletLoss
• distance_metric: cosine_distance
• margin: 0.25
• end_to_end: True
• use_amp: False
• warmup_proportion: 0.1
• l2_weight: 0.0
• seed: 42
• eval_max_steps: -1
• load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
1.0	1	0.2533	-
2.0	2	0.2532	-

Framework Versions

• Python: 3.13.8
• SetFit: 1.1.3
• Sentence Transformers: 5.1.2
• Transformers: 4.57.3
• PyTorch: 2.9.1+cu130
• Datasets: 4.4.1
• Tokenizers: 0.22.1

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}