SimCSE: Simple Contrastive Learning of Sentence Embeddings
Tianyu Gao, Xingcheng Yao, Danqi Chen
This paper presents SimCSE, a simple contrastive learning framework that greatly advances the state-of-the-art sentence embeddings. We first describe an unsupervised approach, which takes an input sentence and predicts itself in a contrastive objective, with only standard dropout used as noise. This simple method works surprisingly well, performing on par with previous supervised counterparts. We find that dropout acts as minimal data augmentation and removing it leads to a representation collapse. Then, we propose a supervised approach, which incorporates annotated pairs from natural language inference datasets into our contrastive learning framework, by using "entailment" pairs as positives and "contradiction" pairs as hard negatives. We evaluate SimCSE on standard semantic textual similarity (STS) tasks, and our unsupervised and supervised models using BERT base achieve an average of 76.3% and 81.6% Spearman's correlation respectively, a 4.2% and 2.2% improvement compared to previous best results. We also show-both theoretically and empirically-that contrastive learning objective regularizes pre-trained embeddings' anisotropic space to be more uniform, and it better aligns positive pairs when supervised signals are available. 1
this paper
works it cites
works citing it
node size = global citations · hover for the full title
Citation edge evidence
- Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing→SimCSE: Simple Contrastive Learning of Sentence EmbeddingsLegacy edge — source detail unavailable
- SimCSE: Simple Contrastive Learning of Sentence Embeddings→Exploiting Generative AI to Scale up Intelligent Tutoring SystemsLegacy edge — source detail unavailable
- SimCSE: Simple Contrastive Learning of Sentence Embeddings→Dropout: a simple way to prevent neural networks from overfittingLegacy edge — source detail unavailable
- SimCSE: Simple Contrastive Learning of Sentence Embeddings→Glove: Global Vectors for Word RepresentationLegacy edge — source detail unavailable
- SimCSE: Simple Contrastive Learning of Sentence Embeddings→BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingLegacy edge — source detail unavailable
- SimCSE: Simple Contrastive Learning of Sentence Embeddings→Distributed Representations of Words and Phrases and their CompositionalityLegacy edge — source detail unavailable
- SimCSE: Simple Contrastive Learning of Sentence Embeddings→HISTORIAE, History of Socio-Cultural Transformation as Linguistic Data Science. A Humanit…Legacy edge — source detail unavailable
- SimCSE: Simple Contrastive Learning of Sentence Embeddings→Sentence-BERT: Sentence Embeddings using Siamese BERT-NetworksLegacy edge — source detail unavailable
- SimCSE: Simple Contrastive Learning of Sentence Embeddings→Transformers: State-of-the-Art Natural Language ProcessingLegacy edge — source detail unavailable
- SimCSE: Simple Contrastive Learning of Sentence Embeddings→Mining and summarizing customer reviewsLegacy edge — source detail unavailable
- SimCSE: Simple Contrastive Learning of Sentence Embeddings→A Simple Framework for Contrastive Learning of Visual RepresentationsLegacy edge — source detail unavailable
- SimCSE: Simple Contrastive Learning of Sentence Embeddings→Recursive Deep Models for Semantic Compositionality Over a Sentiment TreebankLegacy edge — source detail unavailable
- SimCSE: Simple Contrastive Learning of Sentence Embeddings→A sentimental educationLegacy edge — source detail unavailable
- SimCSE: Simple Contrastive Learning of Sentence Embeddings→Seeing starsLegacy edge — source detail unavailable
- SimCSE: Simple Contrastive Learning of Sentence Embeddings→Supervised Learning of Universal Sentence Representations from Natural\n Language Inferen…Legacy edge — source detail unavailable
- SimCSE: Simple Contrastive Learning of Sentence Embeddings→Annotating Expressions of Opinions and Emotions in LanguageLegacy edge — source detail unavailable
- Seeing stars→A sentimental educationLegacy edge — source detail unavailable
- Glove: Global Vectors for Word Representation→Distributed Representations of Words and Phrases and their CompositionalityLegacy edge — source detail unavailable
- Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank→Seeing starsLegacy edge — source detail unavailable
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding→Distributed Representations of Words and Phrases and their CompositionalityLegacy edge — source detail unavailable
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding→Glove: Global Vectors for Word RepresentationLegacy edge — source detail unavailable
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding→Recursive Deep Models for Semantic Compositionality Over a Sentiment TreebankLegacy edge — source detail unavailable
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding→Supervised Learning of Universal Sentence Representations from Natural\n Language Inferen…Legacy edge — source detail unavailable
- Supervised Learning of Universal Sentence Representations from Natural\n Language Inferen…→Distributed Representations of Words and Phrases and their CompositionalityLegacy edge — source detail unavailable
- Supervised Learning of Universal Sentence Representations from Natural\n Language Inferen…→Glove: Global Vectors for Word RepresentationLegacy edge — source detail unavailable
- HISTORIAE, History of Socio-Cultural Transformation as Linguistic Data Science. A Humanit…→Recursive Deep Models for Semantic Compositionality Over a Sentiment TreebankLegacy edge — source detail unavailable
- HISTORIAE, History of Socio-Cultural Transformation as Linguistic Data Science. A Humanit…→BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingLegacy edge — source detail unavailable
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks→Annotating Expressions of Opinions and Emotions in LanguageLegacy edge — source detail unavailable
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks→A sentimental educationLegacy edge — source detail unavailable
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks→Mining and summarizing customer reviewsLegacy edge — source detail unavailable
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks→Seeing starsLegacy edge — source detail unavailable
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks→Glove: Global Vectors for Word RepresentationLegacy edge — source detail unavailable
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks→Recursive Deep Models for Semantic Compositionality Over a Sentiment TreebankLegacy edge — source detail unavailable
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks→BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingLegacy edge — source detail unavailable
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks→Supervised Learning of Universal Sentence Representations from Natural\n Language Inferen…Legacy edge — source detail unavailable
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks→HISTORIAE, History of Socio-Cultural Transformation as Linguistic Data Science. A Humanit…Legacy edge — source detail unavailable
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks→Exploiting Generative AI to Scale up Intelligent Tutoring SystemsLegacy edge — source detail unavailable
- Transformers: State-of-the-Art Natural Language Processing→HISTORIAE, History of Socio-Cultural Transformation as Linguistic Data Science. A Humanit…Legacy edge — source detail unavailable
- Transformers: State-of-the-Art Natural Language Processing→Exploiting Generative AI to Scale up Intelligent Tutoring SystemsLegacy edge — source detail unavailable
What this paper cites, inside the corpus
What cites it, inside the corpus
| Paper | Year | Cited |
|---|---|---|
| Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing | 2021 | 1,687 |
Links
Download BibTeX · Download RIS · DOI · Open access copy · OpenAlex record
Topics
| Topic Modeling | Computer Science |
| Natural Language Processing Techniques | Computer Science |
| Sentiment Analysis and Opinion Mining | Computer Science |
Is this record sound?
complete
Nothing in this record contradicts itself and no field we check is missing.
- supports3 author record(s) attached.
- supports70 reference(s) recorded.
- supportsThe DOI's year agrees with the publication year.
- supportsA title is present.
Provenance
sha256 46f669157f96ea35…