erichuber/finetuned-medical-model

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("erichuber/finetuned-medical-model")
# Run inference
sentences = [
    'Does the early adopter of drugs exist?',
    "Within drug groups, indicators of drug adoption, except for adoption time, correlate reasonably well. However, the theory that physicians' early adoption of new drugs is a personal trait independent of the type of drug could not be confirmed. The notion of the early-drug-adopting general practitioner may be mistaken.",
    'How is pars planitis diagnosed? Pars planitis is typically diagnosed based on a specialized eye examination. During the exam, the ophthalmologist will typically see clusters of white blood cells trapped within the eyeball that are called snowballs (or "inflammatory exudate"). If these clusters are located on the pars plana, they are known as snowbanks. Snowbanks are considered a "hallmark" sign of pars planitis. It is often recommended that people over age 25 with pars planitis have an MRI of their brain and spine to rule out multiple sclerosis.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

• Size: 2,000 training samples
• Columns: sentence_0, sentence_1, and label
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	label
  type	string	string	float
  details	min: 7 tokens mean: 15.1 tokens max: 39 tokens	min: 12 tokens mean: 166.48 tokens max: 256 tokens	min: 0.0 mean: 0.51 max: 1.0

• Samples:

  sentence_0	sentence_1	label
  What is (are) Psoriasis ?	How might Fanconi Bickel syndrome be treated? Management of Fanconi Bickel syndrome (FBS) generally focuses on the signs and symptoms of the condition. Treatment includes replacement of water and electrolytes, and vitamin D and phosphate supplements for prevention of hypophosphatemic rickets. Although there is limited data on the effectiveness of dietary treatment for this condition, it is recommended that affected individuals follow a galactose-restricted diabetic diet, with fructose as the main source of carbohydrate. Diet and supplements may alleviate some of the signs and symptoms of the condition but generally do not improve growth, resulting in short stature in adulthood.	0.0
  Is Adenoid cystic carcinoma inherited ?	What are the signs and symptoms of Kaufman oculocerebrofacial syndrome? The Human Phenotype Ontology provides the following list of signs and symptoms for Kaufman oculocerebrofacial syndrome. If the information is available, the table below includes how often the symptom is seen in people with this condition. You can use the MedlinePlus Medical Dictionary to look up the definitions for these medical terms. Signs and Symptoms Approximate number of patients (when available) Abnormality of calvarial morphology 90% Arachnodactyly 90% Cognitive impairment 90% Long toe 90% Microcephaly 90% Optic atrophy 90% Respiratory insufficiency 90% Upslanted palpebral fissure 90% Abnormality of the palate 50% Aplasia/Hypoplasia of the eyebrow 50% Blepharophimosis 50% Epicanthus 50% Long face 50% Microcornea 50% Microdontia 50% Muscle weakness 50% Myopia 50% Narrow face 50% Nystagmus 50% Preauricular skin tag 50% Short philtrum 50% Strabismus 50% Telecanthus 50% Thin vermilion border 50% Wide mouth 50% C...	0.0
  Is protein C deficiency inherited ?	The heart has an internal electrical system that controls the rhythm of the heartbeat. Problems can cause abnormal heart rhythms, called arrhythmias. There are many types of arrhythmia. During an arrhythmia, the heart can beat too fast, too slow, or it can stop beating. Sudden cardiac arrest (SCA) occurs when the heart develops an arrhythmia that causes it to stop beating. This is different than a heart attack, where the heart usually continues to beat but blood flow to the heart is blocked. There are many possible causes of SCA. They include coronary heart disease, physical stress, and some inherited disorders. Sometimes there is no known cause for the SCA. Without medical attention, the person will die within a few minutes. People are less likely to die if they have early defibrillation. Defibrillation sends an electric shock to restore the heart rhythm to normal. You should give cardiopulmonary resuscitation (CPR) to a person having SCA until defibrillation can be done. If...	0.0

• Loss: CosineSimilarityLoss with these parameters:

  {
      "loss_fct": "torch.nn.modules.loss.MSELoss"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• num_train_epochs: 1
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin

Framework Versions

• Python: 3.13.1
• Sentence Transformers: 4.1.0
• Transformers: 4.52.4
• PyTorch: 2.6.0+cpu
• Accelerate: 1.7.0
• Datasets: 3.6.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}