Bongsoo

Rank:

Average Model Cost: $0.0000

Number of Runs: 3,658

Models by this creator

moco-sentencebertV2.0

bongsoo

Platform did not provide a description for this model.

$-/run

3.0K

Huggingface

klue-cross-encoder-v1

bongsoo

Platform did not provide a description for this model.

$-/run

343

Huggingface

kpf-cross-encoder-v1

bongsoo

kpf-cross-encoder-v1 jinmang2/kpfbert 모델을 훈련시켜 cross-encoder로 파인튜닝한 모델 This model was trained using SentenceTransformers Cross-Encoder class. Training sts(10)-sts(10)훈련 시킴 STS : seed=111,epoch=10, lr=1e-4, eps=1e-6, warm_step=10%, max_seq_len=128, train_batch=128(small 모델=32) (albert 13m/7G) 훈련코드 평가코드,테스트코드 모델 korsts klue-sts glue(stsb) stsb_multi_mt(en) albert-small-kor-cross-encoder-v1 0.8455 0.8526 0.8513 0.7976 klue-cross-encoder-v1 0.8262 0.8833 0.8512 0.7889 kpf-cross-encoder-v1 0.8799 0.9133 0.8626 0.8027 Usage and Performance Pre-trained models can be used like this: The model will predict scores for the pairs ('Sentence 1', 'Sentence 2') and ('Sentence 3', 'Sentence 4'). You can use this model also without sentence_transformers and by just using Transformers AutoModel class

$-/run

125

Huggingface

moco-sentencedistilbertV2.1

bongsoo

Platform did not provide a description for this model.

$-/run

Huggingface

kpf-sbert-v1.1

bongsoo

kpf-sbert-v1.1 This is a sentence-transformers model: It maps sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search. jinmang2/kpfbert 모델을 sentencebert로 파인듀닝한 모델 (kpf-sbert-v1 에서 NLI-STS 훈련을 1번 더 시킴) Evaluation Results 성능 측정을 위한 말뭉치는, 아래 한국어 (kor), 영어(en) 평가 말뭉치를 이용함 한국어 : korsts(1,379쌍문장) 와 klue-sts(519쌍문장) 영어 : stsb_multi_mt(1,376쌍문장) 와 glue:stsb (1,500쌍문장) 성능 지표는 cosin.spearman 평가 측정 코드는 여기 참조 모델 korsts klue-sts glue(stsb) stsb_multi_mt(en) distiluse-base-multilingual-cased-v2 0.7475 0.7855 0.8193 0.8075 paraphrase-multilingual-mpnet-base-v2 0.8201 0.7993 0.8907 0.8682 bongsoo/albert-small-kor-sbert-v1 0.8305 0.8588 0.8419 0.7965 bongsoo/klue-sbert-v1.0 0.8529 0.8952 0.8813 0.8469 bongsoo/kpf-sbert-v1.0 0.8590 0.8924 0.8840 0.8531 bongsoo/kpf-sbert-v1.1 0.8750 0.8900 0.8863 0.8554 For an automated evaluation of this model, see the Sentence Embeddings Benchmark: https://seb.sbert.net Training jinmang2/kpfbert 모델을 sts(10)-distil(10)-nli(3)-sts(10)-nli(3)-sts(10) 훈련 시킴 The model was trained with the parameters: 공통 do_lower_case=1, correct_bios=0, polling_mode=mean 1.STS 말뭉치 : korsts(5,749) + kluestsV1.1(11,668) + stsb_multi_mt(5,749) + mteb/sickr-sts(9,927) + glue stsb(5,749) (총:38,842) Param : lr: 1e-4, eps: 1e-6, warm_step=10%, epochs: 10, train_batch: 128, eval_batch: 64, max_token_len: 72 훈련코드 여기 참조 2.distilation 교사 모델 : paraphrase-multilingual-mpnet-base-v2(max_token_len:128) 말뭉치 : news_talk_en_ko_train.tsv (영어-한국어 대화-뉴스 병렬 말뭉치 : 1.38M) Param : lr: 5e-5, eps: 1e-8, epochs: 10, train_batch: 128, eval/test_batch: 64, max_token_len: 128(교사모델이 128이므로 맟춰줌) 훈련코드 여기 참조 3.NLI 말뭉치 : 훈련(967,852) : kornli(550,152), kluenli(24,998), glue-mnli(392,702) / 평가(3,519) : korsts(1,500), kluests(519), gluests(1,500) () HyperParameter : lr: 3e-5, eps: 1e-8, warm_step=10%, epochs: 3, train/eval_batch: 64, max_token_len: 128 훈련코드 여기 참조 Citing & Authors bongsoo

$-/run

Huggingface

albert-small-kor-sbert-v1.1

bongsoo

albert-small-kor-sbert-v1.1 This is a sentence-transformers model: It maps sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search. albert-small-kor-v1 모델을 sentencebert로 만든 모델. Usage (Sentence-Transformers) Using this model becomes easy when you have sentence-transformers installed: Then you can use the model like this: Usage (HuggingFace Transformers) Without sentence-transformers, you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings. Evaluation Results 성능 측정을 위한 말뭉치는, 아래 한국어 (kor), 영어(en) 평가 말뭉치를 이용함 한국어 : korsts(1,379쌍문장) 와 klue-sts(519쌍문장) 영어 : stsb_multi_mt(1,376쌍문장) 와 glue:stsb (1,500쌍문장) 성능 지표는 cosin.spearman 평가 측정 코드는 여기 참조 모델 korsts klue-sts glue(stsb) stsb_multi_mt(en) distiluse-base-multilingual-cased-v2 0.7475 0.7855 0.8193 0.8075 paraphrase-multilingual-mpnet-base-v2 0.8201 0.7993 0.8907 0.8682 bongsoo/albert-small-kor-sbert-v1 0.8305 0.8588 0.8419 0.7965 bongsoo/klue-sbert-v1.0 0.8529 0.8952 0.8813 0.8469 bongsoo/kpf-sbert-v1.1 0.8750 0.8900 0.8863 0.8554 bongsoo/albert-small-kor-sbert-v1.1 0.8526 0.8833 0.8484 0.8286 For an automated evaluation of this model, see the Sentence Embeddings Benchmark: https://seb.sbert.net Training albert-small-kor-v1 모델을 sts(10)-distil(10) 훈련만 시킴 (nli-sts추가 훈련시키면 정합도가 떨어짐) 교사모델은 kpf-sbert-v1.1 이용함. The model was trained with the parameters: 공통 do_lower_case=1, correct_bios=0, polling_mode=cls 1.STS 말뭉치 : korsts(5,749) + kluestsV1.1(11,668) + stsb_multi_mt(5,749) + mteb/sickr-sts(9,927) + glue stsb(5,749) (총:38,842) Param : lr: 1e-4, eps: 1e-6, warm_step=10%, epochs: 10, train_batch: 32, eval_batch: 64, max_token_len: 72 훈련코드 여기 참조 2.distilation 교사 모델 : kpf-sbert-v1.1(max_token_len:128) 말뭉치 : news_talk_ko_en_train.tsv (한국어-영어 대화-뉴스 병렬 말뭉치 : 1.38M) Param : lr: 5e-5, epochs: 10, train_batch: 32, eval/test_batch: 64, max_token_len: 128(교사모델이 128이므로 맟춰줌) 훈련코드 여기 참조 Full Model Architecture Citing & Authors bongsoo

$-/run

Huggingface

albert-small-kor-v1

bongsoo

Platform did not provide a description for this model.

$-/run

Huggingface

albert-small-kor-sbert-v1

bongsoo

Platform did not provide a description for this model.

$-/run

Huggingface

moco-sentencedistilbertV2.0

bongsoo

Platform did not provide a description for this model.

$-/run

Huggingface

mdistilbertV3.1

bongsoo

mdistilbertV3.1 distilbert-base-multilingual-cased 모델에 moco-corpus-kowiki2022 말뭉치(kowiki202206 + MOCOMSYS 추출 3.2M 문장)로 vocab 추가하여 학습 시킨 모델 vocab: 159,552개 (기존 bert 모델 vocab(119,548개)에 40,004개 (한글단어30,000개+영문10,000개+수동 4개)vocab 추가 mdistilbertV2.1 보다 약 7,000개 단어가 더 많고, 한글단어는 mecab를 이용하여 추출함. epoch은 12번 학습함(mdistilbertV2.1은 8번) Usage (HuggingFace Transformers) 1. MASK 예시 결과 2. 임베딩 예시 평균 폴링(mean_pooling) 방식 사용. (cls 폴링, max 폴링) 결과 Training MLM(Masked Langeuage Model) 훈련 입력 모델 : distilbert-base-multilingual-cased 말뭉치 : 훈련 : bongsoo/moco-corpus-kowiki2022(7.6M) , 평가: bongsoo/moco_eval HyperParameter : LearningRate : 5e-5, epochs: 12 , batchsize: 32, max_token_len : 128 vocab : 159,552개 (기존 bert 모델 vocab(119,548개)에 40,004개 (한글단어30,000개+영문10,000개+수동 4개)vocab 추가 출력 모델 : mdistilbertV3.1 (size: 634MB) 훈련시간 : 90h/1GPU (24GB/16.5 use) 훈련loss: 2.1154, 평가loss: 2.5275 훈련코드 여기 참조 perplexity 평가 코드는 여기 참조 Model Config Citing & Authors bongsoo

$-/run

Huggingface

Bongsoo

Models by this creator

moco-sentencebertV2.0

klue-cross-encoder-v1

kpf-cross-encoder-v1

moco-sentencedistilbertV2.1

kpf-sbert-v1.1

albert-small-kor-sbert-v1.1

albert-small-kor-v1

albert-small-kor-sbert-v1

moco-sentencedistilbertV2.0

mdistilbertV3.1

Similar creators