Research
Text-to-SQL
As the size and complexity of databases continue to grow, managing and querying them manually becomes increasingly infeasible. Our lab is designing advanced large language model (LLM)-powered multi-agent systems that serve as natural language interfaces to databases, making it possible to query and retrieve information seamlessly using everyday language. Two notable contributions from our lab in this domain are CHESS and CHASE-SQL, each introducing innovative approaches to tackling challenges in text-to-SQL synthesis.
Chess: Contextual Harnessing for Efficient SQL Synthesis
CHESS framework addresses key challenges such as large database schemas, ambiguous natural language questions, and ensuring functional validity of queries. CHESS leverages a multi-agent architecture, including components like the Information Retriever, Schema Selector, Candidate Generator, and Unit Tester, to generate accurate, privacy-preserving SQL queries. It efficiently handles industrial-scale databases by narrowing schemas, reducing computational cost, and supporting deployment in privacy-sensitive environments. CHESS demonstrates remarkable scalability, achieving near state-of-the-art performance with significantly fewer computational resources.
Paper: https://arxiv.org/abs/2405.16755
Code: https://github.com/ShayanTalaei/CHESS
Chase-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQL
The CHASE-SQL framework further advances text-to-SQL research through multi-path reasoning and preference-optimized candidate selection. By employing diverse LLM generators, including methods such as divide-and-conquer query decomposition, chain-of-thought reasoning, and instance-aware synthetic example generation, CHASE-SQL generates robust SQL candidates. A dedicated selection agent then identifies the optimal query through pairwise comparisons. This framework achieves state-of-the-art execution accuracy on the BIRD benchmark, setting new standards in query quality and diversity while outperforming other methods.
Paper: https://arxiv.org/abs/2410.01943
LLM Verification
Large Language Models (LLMs) sometimes produce outputs that are factually incorrect or inconsistent with their training data, a phenomenon commonly referred to as "hallucination." This project focuses on enhancing the reliability and trustworthiness of LLMs by integrating tools and verifiers into the generation process. By equipping LLMs with mechanisms to cross-check their outputs against verified data sources and logical frameworks, our approach seeks to ensure factual accuracy and reduce misinformation. This research not only addresses one of the most critical limitations of current LLMs but also paves the way for their safer deployment in applications requiring high reliability, such as scientific research, logical reasoning, coding, and mathematics.
AI Friend/Biographer
The AI Friend/Biographer project aims to develop a personalized AI companion designed to support emotional well-being, cognitive health, and memory preservation, particularly for aging individuals or those facing memory challenges. Beyond being a conversational assistant, this AI acts as a long-term companion, helping users recall, reflect upon, and document their life stories. By guiding users in narrating their personal experiences and weaving them into a cohesive memoir or biography, the AI preserves cherished memories for future generations. Combining emotional support with practical functionality, this innovative project not only enhances mental sharpness and promotes emotional resilience but also creates enduring records of personal histories. Through tailored interactions, the AI adapts to individual preferences and writing styles, offering a unique and empathetic experience that empowers users to celebrate and share their journeys.
LLMs as Persuasion Agents
The Persuasion Agent project explores the persuasive capabilities of large language models (LLMs), aiming to understand, enhance, and ethically deploy these systems. While LLMs have demonstrated significant potential in generating persuasive text across domains like political discourse, marketing, and ideology, this project addresses the challenges of systematically evaluating their persuasive effectiveness and ensuring ethical safeguards. Key objectives include developing systems to measure the persuasive power of LLM-generated content, detect bias, and verify factual accuracy to mitigate risks of misuse, such as spreading misinformation or manipulating public opinion. Additionally, the project explores subtle and context-aware persuasion techniques through the design of a "persuasion agent" (PA) that assists users in making informed decisions in a supportive and unobtrusive manner. By utilizing user preferences and profiles to personalize interactions, the PA enhances its ability to provide relevant and meaningful guidance. This approach prioritizes ethical use, transparency, and user empowerment, ensuring that persuasive capabilities are applied responsibly and constructively to foster positive outcomes.
MAGNOLIA: Matching Algorithms via GNNs
Online Bayesian bipartite matching is a central problem in digital marketplaces and exchanges, including advertising, crowdsourcing, ridesharing, and kidney exchange. We introduce a graph neural network (GNN) approach that emulates the problem's combinatorially-complex optimal online algorithm, which selects actions (e.g., which nodes to match) by computing each action's value-to-go (VTG) -- the expected weight of the final matching if the algorithm takes that action, then acts optimally in the future. We train a GNN to estimate VTG and show empirically that this GNN returns high-weight matchings across a variety of tasks. Moreover, we identify a common family of graph distributions in spatial crowdsourcing applications, such as rideshare, under which VTG can be efficiently approximated by aggregating information within local neighborhoods in the graphs. This structure matches the local behavior of GNNs, providing theoretical justification for our approach.
Paper: https://arxiv.org/abs/2406.05959
A Local Limits Perspective on GNNs
We propose a theoretical framework for training Graph Neural Networks (GNNs) on large input graphs via training on small, fixed-size sampled subgraphs. This framework is applicable to a wide range of models, including popular sampling-based GNNs, such as GraphSAGE and FastGCN. Leveraging the theory of graph local limits, we prove that, under mild assumptions, parameters learned from training sampling-based GNNs on small samples of a large input graph are within an ϵ-neighborhood of the outcome of training the same architecture on the whole graph. We derive bounds on the number of samples, the size of the graph, and the training steps required as a function of ϵ. Our results give a novel theoretical understanding for using sampling in training GNNs. They also suggest that by training GNNs on small samples of the input graph, practitioners can identify and select the best models, hyperparameters, and sampling algorithms more efficiently. We empirically illustrate our results on a node classification task on large citation graphs, observing that sampling-based GNNs trained on local subgraphs 12× smaller than the original graph achieve comparable performance to those trained on the input graph.