Efficient Long-range Language Modeling with Self-supervised Causal Retrieval
Published in arXiv, 2024
An efficient approach to train a dense retriever under the supervision from auto-regressive loss.
Publications
You can also find my articles on my Google Scholar profile.
Preprints
Unsupervised Morphological Tree Tokenizer
Published in arXiv, 2024
A tokenizer based on an unsupervised tree-based morphological parser.
Conference Papers
Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at Scale
Published in ACL, 2024
We propose GPST, a syntactic language model which could be pre-trained on raw text efficiently without any human-annotated trees. When GPST and GPT-2 are both pre-trained on OpenWebText from scratch, GPST can outperform GPT-2 on various downstream tasks. Moreover, it significantly surpasses previous methods on generative grammar induction tasks, exhibiting a high degree of consistency with human syntax.
Augmenting Transformers with Recursively Composed Multi-Grained Representations
Published in ICLR, 2024
We reduce the space complexity of the deep inside-outside algorithm from cubic to linear and further reduce the parallel time complexity to approximately log N thanks to the new pruning algorithm proposed in this paper. Furthermore, we find that joint pre-training of Transformers and composition models can enhance a variety of NLP downstream tasks. We push unsupervised constituency parsing performance to 65% and demonstrate that our model could outperform vanillar Trasformers around 5% on span-level tasks.
A Multi-Grained Self-Interpretable Symbolic-Neural Model For Single/Multi-Labeled Text Classification
Published in ICLR, 2023
We explore the interpretability of the structured encoder and find that the induced alignment between labels and spans is highly consistent with human rationality.
Fast-R2D2: A Pretrained Recursive Neural Network based on Pruned CKY for Grammar Induction and Text Representation
Published in EMNLP, 2022
Improve the heuristic pruning module used in R2D2 to model-based pruning.
R2D2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Modeling
Published in ACL, 2021
Proposing an unsupervised structured encoder able to compose low-level constituents into high-level constituents without gold trees. The learned trees are highly consistent with human-annotated ones. The backbone of the encoder is a neural inside algorithm with heuristic pruning, thus the time and space complexity are both in linear.
Interactive Question Clarification in Dialogue via Reinforcement Learning
Published in COLING, 2020
Question clarification based on MSTC.