hassaangyt/compliance-emb-v1

SentenceTransformer based on sentence-transformers/all-mpnet-base-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-mpnet-base-v2 on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-mpnet-base-v2
• Maximum Sequence Length: 384 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 384, 'do_lower_case': False, 'architecture': 'MPNetModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'tation. A copy of the assessment, denial, or other determination you are disputing must be attached. Representative Information (complete only if you want someone to represent you during the reconsideration process) Power of Attorney (complete only if you want someone to represent you during the reconsideration process) By signing below, the petitioner(s) named in Section 1 appoints the individual named in Section 6 to act as their representative with full authority to receive confidential information and to perform any and all acts the petitioner can perform in connection with matters associated with this petition, except, the representative may not delegate their authority to another individual. If you wish to limit the authority granted by this Power of Attorney, please describe the limitation: A copy of the assessment, denial, or other determination (collectively referred to throughout these instructions as a “Determination”) you are disputing must be attached.',
    'Criminals are targeting human resources and financial professionals across Maine with a new phishing scheme. Don’t fall victim. If you get an email asking you to send employee W-2 or other private/sensitive information, stop to confirm if the request is legitimate before you hit send. Criminals have perfected techniques to trick you into thinking an email is coming from a person you work with. Don’t fall victim to this scam. Connect with the person who you believe sent you the request by phone or by walking over to see them. Do not respond to the email to confirm the sender’s request. The sender could be a criminal, disguising their identity with a fake email address. If you confirm a legitimate request, take steps to protect the information before you send it. and place W-2 Scam in the subject line. • File a complaint with the Internet Crime Complaint Center (IC3), operated by the Federal Bureau of Investigation.',
    'noise',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000, -0.2313, -0.0125],
#         [-0.2313,  1.0000,  0.0933],
#         [-0.0125,  0.0933,  1.0000]])

Training Details

Training Dataset

json

• Dataset: json
• Size: 16,975 training samples
• Columns: sentence1, sentence2, score, and type
• Approximate statistics based on the first 1000 samples:

  	sentence1	sentence2	score	type
  type	string	string	float	string
  details	min: 36 tokens mean: 179.93 tokens max: 270 tokens	min: 21 tokens mean: 94.85 tokens max: 217 tokens	min: 0.05 mean: 0.49 max: 1.0	min: 3 tokens mean: 4.03 tokens max: 5 tokens

• Samples:

  sentence1	sentence2	score	type
  additional cost incurred providing job coaching, training and supervision on a supported work site. : "Supported Employment/Work" is a consumer-oriented, integrated, and non-segregated employment which is based on the individual's informed choice and which provides appropriate ongoing services to individual s with most significant disabilities in order for the individual to work productively in the community. Specifically, individuals in supported employment/work must: A. be engaged in part-time or full-time employment which pays wages and benefits commensurate with the individuals ability to produce goods or render services and which is based on current competitive rates at or above the minimum wage .	All industrial facilities must implement a confined space entry program, including written procedures, employee training, and rescue plans, to comply with workplace safety requirements.	0.3	soft_negative
  additional cost incurred providing job coaching, training and supervision on a supported work site. : "Supported Employment/Work" is a consumer-oriented, integrated, and non-segregated employment which is based on the individual's informed choice and which provides appropriate ongoing services to individual s with most significant disabilities in order for the individual to work productively in the community. Specifically, individuals in supported employment/work must: A. be engaged in part-time or full-time employment which pays wages and benefits commensurate with the individuals ability to produce goods or render services and which is based on current competitive rates at or above the minimum wage .	“Supported Employment/Work” permits payment of wages below the minimum wage and allows supported workers to be paid without regard to current competitive rates.	0.6	hard_negative
  vices and which is based on current competitive rates at or above the minimum wage . hat is individualized, and customized, consistent with the unique strengths, abilities, interests, and informed choice of the individual, including with ongoing support services for individuals with the most significant disabilities – For whom competitive integrated employment has not historically occurred, or for whom competitive integrated employment has been interrupted or intermittent as a result of a significant disability; and Who, because of the nature and severity of their disabilities, need intensive supported employment services and extended services after the transition from support provided by the designated State unit, in order to perform this work.	Licensed businesses that discharge process water must install and operate best available technology to limit nutrient discharges and must submit semiannual effluent monitoring reports to the environmental agency.	0.3	soft_negative

• Loss: CoSENTLoss with these parameters:

  {
      "scale": 40.0,
      "similarity_fct": "pairwise_cos_sim"
  }
  

Evaluation Dataset

json

• Dataset: json
• Size: 2,139 evaluation samples
• Columns: sentence1, sentence2, score, and type
• Approximate statistics based on the first 1000 samples:

  	sentence1	sentence2	score	type
  type	string	string	float	string
  details	min: 44 tokens mean: 181.52 tokens max: 329 tokens	min: 22 tokens mean: 86.31 tokens max: 227 tokens	min: 0.05 mean: 0.5 max: 1.0	min: 3 tokens mean: 3.98 tokens max: 5 tokens

• Samples:

  sentence1	sentence2	score	type
  rce, after an individual has made the transition from support provided by the designated State unit. can produce a diminished or altered state of consciousness resulting in impairment of cognitive abilities or physical functioning; C. can result in the disturbance of behavioral or emotional functioning; E. can cause partial or total functional disability or psychological maladjustment. An applicant for or recipient of Extended Support services who is dissatisfied with any determination made by the Bureau of Rehabilitation Services concerning the furnishing or denial of services may request a timely review of the determination. The Bureau shall make reasonable accessibility accommodations for the individual with disabilities during the appeals process. Whenever possible, the Bureau will attempt to resolve conflicts through Informal Review or through Mediation. An individual may request a Due Process Hearing immediately without having to go through other appeal steps.	Conditions described can impair consciousness, cognition, physical functioning, behavior, or emotional functioning and may cause partial or total functional disability or psychological maladjustment. An applicant or recipient dissatisfied with a Bureau of Rehabilitation Services decision about providing or denying Extended Support may request a timely review. The Bureau must provide reasonable accessibility accommodations during the appeals process, will attempt to resolve disputes through Informal Review or Mediation when possible, and an individual may request a Due Process Hearing immediately without first completing other appeal steps.	1.0	positive
  The purpose of this program is to provide ongoing extended supports to individuals with brain injuries who are VR consumers with the most significant disabilities once training has been completed. The program provides financial assistance to providers of ongoing support and/or employers to help defray the additional cost incurred providing job coaching, training and supervision on a supported work site.	The program provides financial assistance only to individuals prior to completing training and does not fund supports for vocational rehabilitation consumers after training completion.	0.6	hard_negative
  r assessments and determinations. After I file a petition, do I still have to pay the amount due? No. However, interest, but not penalties, will continue to accrue during the appeals process. You can minimize additional interest by helping to get your case decided as quickly as possible. For example, if you have any documents that you want MRS to consider, you should attach copies to your petition. You should also be as specific as possible in explaining why you believe the assessment is not correct. However, if you have received a “jeopardy” assessment, you must immediately pay the assessed amount, or file a bond or other security, to prevent immediate collection proceedings. You may still file a petition within 60 days as with any other assessment. For any assessment, if you pay the amount due and MRS later determines that you do not owe some or all of the assessment, MRS will issue you a refund. Yes, if you think meeting with MRS would be helpful.	No, filing a petition does not generally require you to pay the assessed amount while the appeal is pending. However, interest (but not penalties) will continue to accrue during the appeals process. You can help minimize further interest by promptly providing any documents you want MRS to consider and by explaining specifically why the assessment is incorrect. If you received a “jeopardy” assessment, you must immediately pay the assessed amount or post a bond or other security to avoid immediate collection; you may still file a petition within 60 days. If you pay an assessment and MRS later determines you did not owe some or all of it, MRS will issue a refund. If you believe a meeting with MRS would help, you may request one.	1.0	positive

• Loss: CoSENTLoss with these parameters:

  {
      "scale": 40.0,
      "similarity_fct": "pairwise_cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 16
• learning_rate: 1e-05
• weight_decay: 0.01
• num_train_epochs: 6
• warmup_ratio: 0.1
• fp16: True
• load_best_model_at_end: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 1e-05
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 6
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss
0.4713	500	6.2293	-
0.9425	1000	1.8318	-
1.0	1061	-	0.5277
1.4138	1500	1.3701	-
1.8850	2000	0.9076	-
2.0	2122	-	0.4623
2.3563	2500	0.4613	-
2.8275	3000	0.3688	-
3.0	3183	-	0.3232
3.2988	3500	0.1959	-
3.7700	4000	0.1412	-
4.0	4244	-	0.3016
4.2413	4500	0.0838	-
4.7125	5000	0.0647	-
5.0	5305	-	0.3073
5.1838	5500	0.0338	-
5.6550	6000	0.0124	-
6.0	6366	-	0.2879

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.12.12
• Sentence Transformers: 5.1.2
• Transformers: 4.57.2
• PyTorch: 2.9.0+cu126
• Accelerate: 1.12.0
• Datasets: 4.4.1
• Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CoSENTLoss

@article{10531646,
    author={Huang, Xiang and Peng, Hao and Zou, Dongcheng and Liu, Zhiwei and Li, Jianxin and Liu, Kay and Wu, Jia and Su, Jianlin and Yu, Philip S.},
    journal={IEEE/ACM Transactions on Audio, Speech, and Language Processing},
    title={CoSENT: Consistent Sentence Embedding via Similarity Ranking},
    year={2024},
    doi={10.1109/TASLP.2024.3402087}
}