Abstract:

LLMs have solidified their place as a key component of most of today’s state of the art methods for language technology development. However, despite their impressive abilities in a number of languages, LLM-based models continue to face challenges that prevent their responsible deployment in real-world applications, particularly in low-resource domains. Sara will present ongoing work she’s conducting for her PhD dissertation to develop a potential framework for mitigating the risks and harms associated with LLM-based systems for translation, taking Maltese-English translation as a case study. The proposed experimental pipeline draws on work using retrieval-augmented generation (RAG) and iterative refinement methods inspired by human translation workflows and theories of morphology and the mental lexicon. The ultimate goal is a language-universal framework for a machine translation system that can be easily and safely deployed and improved upon by the very people it is intended to serve.

Next week, I will be giving a talk on sparse autoencoders (SAEs), their use in interpretability for LLMs, and my work applying them to vision models. I will cover Anthropic’s core line of work (Toy Models of Superposition, Towards Monosemanticity, Scaling Monosemanticity), the core challenges that still exist, why I’m excited about them for vision, and interesting applications of the technology.

Language emergence studies have explored interaction among agents in a network, using a game-theoretic approach (e.g., Lewis signaling games) and reinforcement learning frameworks. Prior research has demonstrated that emergent languages exhibit compositionality (Chaabouni et al., 2020), linguistic conventions shaped by network structure (Lipowska & Lipowski, 2018), and population-driven changes such as improved generalization due to cultural transmission (Cogswell et al., 2019). However, these studies make use of unrealistic tasks and unrealistic agents incapable of reproducing natural language interactions. Recent advancements have expanded multi-agent modeling with large language models capable of reproducing natural language for a range of domains and tasks, including negotiation, consensus seeking, and problem-solving (Guo et al., 2024; Sun et al., 2024). In spirit of this work, I am brainstorming ideas for a project: I am curious to investigate language contact in a multi-agent setting with agents as language models that interact using natural language. I am interested in whether (1) agents develop hybrid languages similar to language change induced by contact among humans, (2) their communication strategies shift toward simplification or complexity over time, and (3) network topology influences linguistic change. This is a nascent idea, so all kind of suggestions are welcomed.