Learning Language Games through Interaction
Paper
•
1606.02447
•
Published
UI Agent
a collection of algorithmic agents for user interfaces/interactions, program synthesis, and robots
-
-
Naturalizing a Programming Language via Interactive Learning
Paper • 1704.06956 • Published
Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration
Paper • 1802.08802 • PublishedNote Introducing the Workflow-Guided Exploration (WGE) framework, and a new synthetic dataset called MiniWoB++ (an extension of MiniWoB, a dataset proposed by "World of Bits: An Open-Domain Platform for Web-Based Agents" at ICML 2017). This work was published at ICLR 2018.
-
Mapping Natural Language Commands to Web Elements
Paper • 1808.09132 • Published -
Learning to Navigate the Web
Paper • 1812.09195 • Published -
Interactive Task and Concept Learning from Natural Language Instructions and GUI Demonstrations
Paper • 1909.00031 • Published -
Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning
Paper • 1909.01871 • Published -
Mapping Natural Language Instructions to Mobile UI Action Sequences
Paper • 2005.03776 • Published -
FLIN: A Flexible Natural Language Interface for Web Navigation
Paper • 2010.12844 • Published -
ActionBert: Leveraging User Actions for Semantic Understanding of User Interfaces
Paper • 2012.12350 • Published -
Grounding Open-Domain Instructions to Automate Web Support Tasks
Paper • 2103.16057 • Published -
Mobile App Tasks with Iterative Feedback (MoTIF): Addressing Task Feasibility in Interactive Visual Environments
Paper • 2104.08560 • Published -
AndroidEnv: A Reinforcement Learning Platform for Android
Paper • 2105.13231 • Published -
UIBert: Learning Generic Multimodal Representations for UI Understanding
Paper • 2107.13731 • Published -
Benchmarking the Spectrum of Agent Capabilities
Paper • 2109.06780 • Published -
Compositional Generalization for Natural Language Interfaces to Web APIs
Paper • 2112.05209 • Published -
VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modeling
Paper • 2112.05692 • Published -
A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility
Paper • 2202.02312 • Published -
Inferring Rewards from Language in Context
Paper • 2204.02515 • Published -
META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI
Paper • 2205.11029 • Published -
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge
Paper • 2206.08853 • Published • 1 -
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
Paper • 2207.01206 • Published • 2 -
Enabling Conversational Interaction with Mobile UI using Large Language Models
Paper • 2209.08655 • Published -
Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus
Paper • 2209.14927 • Published -
ReAct: Synergizing Reasoning and Acting in Language Models
Paper • 2210.03629 • Published • 18 -
Understanding HTML with Large Language Models
Paper • 2210.03945 • Published • 1 -
Lexi: Self-Supervised Learning of the UI Language
Paper • 2301.10165 • Published -
WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semantics
Paper • 2301.13280 • Published -
Self-Refine: Iterative Refinement with Self-Feedback
Paper • 2303.17651 • Published • 2 -
Language Models can Solve Computer Tasks
Paper • 2303.17491 • Published -
Multimodal Web Navigation with Instruction-Finetuned Foundation Models
Paper • 2305.11854 • Published • 4 -
Gorilla: Large Language Model Connected with Massive APIs
Paper • 2305.15334 • Published • 5 -
Voyager: An Open-Ended Embodied Agent with Large Language Models
Paper • 2305.16291 • Published • 9 -
Decision-Oriented Dialogue for Human-AI Collaboration
Paper • 2305.20076 • Published -
From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces
Paper • 2306.00245 • Published -
Mind2Web: Towards a Generalist Agent for the Web
Paper • 2306.06070 • Published • 19 -
Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control
Paper • 2306.07863 • Published -
Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners
Paper • 2307.01928 • Published • 11 -
Android in the Wild: A Large-Scale Dataset for Android Device Control
Paper • 2307.10088 • Published • 11 -
A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis
Paper • 2307.12856 • Published • 36 -
WebArena: A Realistic Web Environment for Building Autonomous Agents
Paper • 2307.13854 • Published • 24 -
Learning to Model the World with Language
Paper • 2308.01399 • Published • 35 -
AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework
Paper • 2308.08155 • Published • 6 -
Never-ending Learning of User Interfaces
Paper • 2308.08726 • Published • 2 -
Empowering LLM to use Smartphone for Intelligent Task Automation
Paper • 2308.15272 • Published • 1 -
LASER: LLM Agent with State-Space Exploration for Web Navigation
Paper • 2309.08172 • Published • 12 -
You Only Look at Screens: Multimodal Chain-of-Action Agents
Paper • 2309.11436 • Published • 1 -
SmartPlay : A Benchmark for LLMs as Intelligent Agents
Paper • 2310.01557 • Published • 13 -
AXNav: Replaying Accessibility Tests from Natural Language
Paper • 2310.02424 • Published -
SteP: Stacked LLM Policies for Web Actions
Paper • 2310.03720 • Published • 8 -
Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models
Paper • 2310.04406 • Published • 8 -
Reinforced UI Instruction Grounding: Towards a Generic UI Task Automation API
Paper • 2310.04716 • Published -
Automatic Macro Mining from Interaction Traces at Scale
Paper • 2310.07023 • Published -
A Zero-Shot Language Agent for Computer Control with Structured Reflection
Paper • 2310.08740 • Published • 15 -
CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization
Paper • 2310.10134 • Published • 1 -
OpenAgents: An Open Platform for Language Agents in the Wild
Paper • 2310.10634 • Published • 8 -
WebWISE: Web Interface Control and Sequential Exploration with Large Language Models
Paper • 2310.16042 • Published -
PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task Completion
Paper • 2311.01767 • Published • 19 -
GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation
Paper • 2311.07562 • Published • 13 -
GAIA: a benchmark for General AI Assistants
Paper • 2311.12983 • Published • 192 -
Language Model Agents Suffer from Compositional Generalization in Web Automation
Paper • 2311.18751 • Published -
Explore, Select, Derive, and Recall: Augmenting LLM with Human-like Memory for Mobile Task Automation
Paper • 2312.03003 • Published -
CogAgent: A Visual Language Model for GUI Agents
Paper • 2312.08914 • Published • 30 -
UINav: A Practical Approach to Train On-Device Automation Agents
Paper • 2312.10170 • Published -
ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation
Paper • 2312.13108 • Published • 3 -
AppAgent: Multimodal Agents as Smartphone Users
Paper • 2312.13771 • Published • 53 -
WebVLN: Vision-and-Language Navigation on Websites
Paper • 2312.15820 • Published -
If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents
Paper • 2401.00812 • Published • 6 -
MobileAgent: enhancing mobile control via human-machine interaction and SOP integration
Paper • 2401.04124 • Published -
GPT-4V(ision) is a Generalist Web Agent, if Grounded
Paper • 2401.01614 • Published • 22 -
SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
Paper • 2401.10935 • Published • 4 -
TroVE: Inducing Verifiable and Efficient Toolboxes for Solving Programmatic Tasks
Paper • 2401.12869 • Published -
WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models
Paper • 2401.13919 • Published • 30 -
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
Paper • 2401.13649 • Published • 1 -
Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception
Paper • 2401.16158 • Published • 19 -
Executable Code Actions Elicit Better LLM Agents
Paper • 2402.01030 • Published • 67 -
DRED: Zero-Shot Transfer in Reinforcement Learning via Data-Regularised Environment Design
Paper • 2402.03479 • Published -
Graph-enhanced Large Language Models in Asynchronous Plan Reasoning
Paper • 2402.02805 • Published • 1 -
RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback
Paper • 2402.03681 • Published -
AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls
Paper • 2402.04253 • Published -
Dual-View Visual Contextualization for Web Navigation
Paper • 2402.04476 • Published -
ScreenAI: A Vision-Language Model for UI and Infographics Understanding
Paper • 2402.04615 • Published • 41 -
ScreenAgent: A Vision Language Model-driven Computer Control Agent
Paper • 2402.07945 • Published -
WebLINX: Real-World Website Navigation with Multi-Turn Dialogue
Paper • 2402.05930 • Published • 39 -
OS-Copilot: Towards Generalist Computer Agents with Self-Improvement
Paper • 2402.07456 • Published • 44 -
Training Language Model Agents without Modifying Language Models
Paper • 2402.11359 • Published • 1 -
The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning
Paper • 2402.12527 • Published -
On the Multi-turn Instruction Following for Conversational Web Agents
Paper • 2402.15057 • Published -
Language Agents as Optimizable Graphs
Paper • 2402.16823 • Published • 3 -
OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web
Paper • 2402.17553 • Published • 23 -
ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL
Paper • 2402.19446 • Published -
SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Code
Paper • 2403.01248 • Published • 2 -
KnowAgent: Knowledge-Augmented Planning for LLM-Based Agents
Paper • 2403.03101 • Published • 1 -
Towards General Computer Control: A Multimodal Agent for Red Dead Redemption II as a Case Study
Paper • 2403.03186 • Published • 5 -
WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?
Paper • 2403.07718 • Published • 2 -
BAGEL: Bootstrapping Agents by Guiding Exploration with Language
Paper • 2403.08140 • Published -
Tur[k]ingBench: A Challenge Benchmark for Web Agents
Paper • 2403.11905 • Published -
AgentStudio: A Toolkit for Building General Virtual Agents
Paper • 2403.17918 • Published -
Octopus v2: On-device language model for super agent
Paper • 2404.01744 • Published • 57 -
AutoWebGLM: Bootstrap And Reinforce A Large Language Model-based Web Navigating Agent
Paper • 2404.03648 • Published • 25 -
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs
Paper • 2404.05719 • Published • 82 -
VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?
Paper • 2404.05955 • Published -
Autonomous Evaluation and Refinement of Digital Agents
Paper • 2404.06474 • Published • 2 -
OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
Paper • 2404.07972 • Published • 48 -
MMInA: Benchmarking Multihop Multimodal Internet Agents
Paper • 2404.09992 • Published -
AgentKit: Flow Engineering with Graphs, not Coding
Paper • 2404.11483 • Published • 2 -
UIClip: A Data-driven Model for Assessing User Interface Design
Paper • 2404.12500 • Published -
AutoCrawler: A Progressive Understanding Web Agent for Web Crawler Generation
Paper • 2404.12753 • Published • 43 -
Benchmarking Mobile Device Control Agents across Diverse Configurations
Paper • 2404.16660 • Published -
Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning
Paper • 2405.00451 • Published -
Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning
Paper • 2405.10292 • Published • 1 -
Latent State Estimation Helps UI Agents to Reason
Paper • 2405.11120 • Published -
AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents
Paper • 2405.14573 • Published -
V-Zen: Efficient GUI Understanding and Precise Grounding With A Novel Multimodal LLM
Paper • 2405.15341 • Published -
Devil's Advocate: Anticipatory Reflection for LLM Agents
Paper • 2405.16334 • Published -
The Importance of Directional Feedback for LLM-based Optimizers
Paper • 2405.16434 • Published -
Position: Foundation Agents as the Paradigm Shift for Decision Making
Paper • 2405.17009 • Published -
Can Graph Learning Improve Planning in LLM-based Agents?
Paper • 2405.19119 • Published -
Large Language Models Can Self-Improve At Web Agent Tasks
Paper • 2405.20309 • Published • 2 -
Controlling Large Language Model Agents with Entropic Activation Steering
Paper • 2406.00244 • Published -
Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration
Paper • 2406.01014 • Published • 32 -
On the Effects of Data Scale on Computer Control Agents
Paper • 2406.03679 • Published -
NATURAL PLAN: Benchmarking LLMs on Natural Language Planning
Paper • 2406.04520 • Published • 12 -
AgentGym: Evolving Large Language Model-based Agents across Diverse Environments
Paper • 2406.04151 • Published • 19 -
Synthetic Query Generation using Large Language Models for Virtual Assistants
Paper • 2406.06729 • Published -
VersiCode: Towards Version-controllable Code Generation
Paper • 2406.07411 • Published -
UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedback
Paper • 2406.07739 • Published • 2 -
StreamBench: Towards Benchmarking Continuous Improvement of Language Agents
Paper • 2406.08747 • Published • 1 -
GUI Odyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices
Paper • 2406.08451 • Published • 24 -
DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning
Paper • 2406.11896 • Published • 20 -
VideoGUI: A Benchmark for GUI Automation from Instructional Videos
Paper • 2406.10227 • Published • 9 -
GUI-WORLD: A Dataset for GUI-oriented Multimodal LLM-based Agents
Paper • 2406.10819 • Published -
GUICourse: From General Vision Language Models to Versatile GUI Agents
Paper • 2406.11317 • Published • 1 -
AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoning
Paper • 2406.11200 • Published -
WebCanvas: Benchmarking Web Agents in Online Environments
Paper • 2406.12373 • Published -
τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Paper • 2406.12045 • Published • 7 -
Adversarial Attacks on Multimodal Agents
Paper • 2406.12814 • Published • 4 -
ICAL: Continual Learning of Multimodal Agents by Transforming Trajectories into Actionable Insights
Paper • 2406.14596 • Published • 5 -
Trace is the New AutoDiff -- Unlocking Efficient Optimization of Computational Workflows
Paper • 2406.16218 • Published • 2 -
Symbolic Learning Enables Self-Evolving Agents
Paper • 2406.18532 • Published • 12 -
APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets
Paper • 2406.18518 • Published • 24 -
Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding
Paper • 2406.19263 • Published • 10 -
BMW Agents -- A Framework For Task Automation Through Multi-Agent Collaboration
Paper • 2406.20041 • Published -
Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents
Paper • 2407.00993 • Published • 1 -
CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents
Paper • 2407.01511 • Published -
AI Agents That Matter
Paper • 2407.01502 • Published • 3 -
Tree Search for Language Model Agents
Paper • 2407.01476 • Published -
Agentless: Demystifying LLM-based Software Engineering Agents
Paper • 2407.01489 • Published • 59 -
AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents
Paper • 2407.17490 • Published • 31 -
AgentInstruct: Toward Generative Teaching with Agentic Flows
Paper • 2407.03502 • Published • 51 -
WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks
Paper • 2407.05291 • Published • 2 -
LLM-Based Open-Domain Integrated Task and Knowledge Assistants with Programmable Policies
Paper • 2407.05674 • Published -
Hypothetical Minds: Scaffolding Theory of Mind for Multi-Agent Tasks with Large Language Models
Paper • 2407.07086 • Published -
GTA: A Benchmark for General Tool Agents
Paper • 2407.08713 • Published • 15 -
TLDR: Unsupervised Goal-Conditioned RL via Temporal Distance-Aware Representations
Paper • 2407.08464 • Published -
IDAT: A Multi-Modal Dataset and Toolkit for Building and Evaluating Interactive Task-Solving Agents
Paper • 2407.08898 • Published -
Security Matrix for Multimodal Agents on Mobile Devices: A Systematic and Proof of Concept Study
Paper • 2407.09295 • Published -
Sibyl: Simple yet Effective Agent Framework for Complex Real-world Reasoning
Paper • 2407.10718 • Published • 18 -
Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?
Paper • 2407.10956 • Published • 6 -
Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systems
Paper • 2407.13032 • Published -
MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains
Paper • 2407.18961 • Published • 40 -
WebRPG: Automatic Web Rendering Parameters Generation for Visual Presentation
Paper • 2407.15502 • Published -
AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?
Paper • 2407.15711 • Published • 9 -
OpenDevin: An Open Platform for AI Software Developers as Generalist Agents
Paper • 2407.16741 • Published • 70 -
AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents
Paper • 2407.18901 • Published • 33 -
Diffusion Augmented Agents: A Framework for Efficient Exploration and Transfer Learning
Paper • 2407.20798 • Published • 24 -
OmniParser for Pure Vision Based GUI Agent
Paper • 2408.00203 • Published • 24 -
AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation
Paper • 2408.00764 • Published • 1 -
Caution for the Environment: Multimodal Agents are Susceptible to Environmental Distractions
Paper • 2408.02544 • Published -
A Taxonomy of Architecture Options for Foundation Model-based Agents: Analysis and Decision Model
Paper • 2408.02920 • Published -
ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities
Paper • 2408.04682 • Published • 18 -
VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents
Paper • 2408.06327 • Published • 17 -
Automated Design of Agentic Systems
Paper • 2408.08435 • Published • 39 -
WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Exploration
Paper • 2408.15978 • Published -
xLAM: A Family of Large Action Models to Empower AI Agent Systems
Paper • 2409.03215 • Published • 4 -
Game On: Towards Language Models as RL Experimenters
Paper • 2409.03402 • Published -
UI-JEPA: Towards Active Perception of User Intent through Onscreen User Activity
Paper • 2409.04081 • Published • 3 -
Robot Utility Models: General Policies for Zero-Shot Deployment in New Environments
Paper • 2409.05865 • Published • 14 -
Agent Workflow Memory
Paper • 2409.07429 • Published • 29 -
Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale
Paper • 2409.08264 • Published • 44 -
EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage
Paper • 2409.11295 • Published -
The Impact of Element Ordering on LM Agent Performance
Paper • 2409.12089 • Published -
Updating Robot Safety Representations Online from Natural Language Feedback
Paper • 2409.14580 • Published -
MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning
Paper • 2409.20566 • Published • 55 -
Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scale
Paper • 2409.15637 • Published -
DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation
Paper • 2410.00201 • Published -
Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents
Paper • 2410.02644 • Published -
NNetscape Navigator: Complex Demonstrations for Web Agents Without a Demonstrator
Paper • 2410.02907 • Published -
MLLM as Retriever: Interactively Learning Multimodal Retrieval for Embodied Agents
Paper • 2410.03450 • Published • 36 -
Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents
Paper • 2410.05243 • Published • 18 -
EVOLvE: Evaluating and Optimizing LLMs For Exploration
Paper • 2410.06238 • Published • 1 -
TinyClick: Single-Turn Agent for Empowering GUI Automation
Paper • 2410.11871 • Published • 2 -
AppBench: Planning of Multiple APIs from Various APPs for Complex User Instruction
Paper • 2410.19743 • Published • 1 -
Agent S: An Open Agentic Framework that Uses Computers Like a Human
Paper • 2410.08164 • Published • 24 -
Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents
Paper • 2410.13886 • Published -
From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating UI Operation Impacts
Paper • 2410.09006 • Published • 1 -
Automated Rewards via LLM-Generated Progress Functions
Paper • 2410.09187 • Published -
Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance
Paper • 2410.12361 • Published -
Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation
Paper • 2410.13232 • Published • 41 -
Harnessing Webpage UIs for Text-Rich Visual Understanding
Paper • 2410.13824 • Published • 30 -
AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agents
Paper • 2410.13825 • Published • 1 -
MobA: A Two-Level Agent System for Efficient Mobile Task Automation
Paper • 2410.13757 • Published • 32 -
GUIDE: Real-Time Human-Shaped Agents
Paper • 2410.15181 • Published -
Improve Vision Language Model Chain-of-thought Reasoning
Paper • 2410.16198 • Published • 25 -
Beyond Browsing: API-Based Web Agents
Paper • 2410.16464 • Published -
AdvWeb: Controllable Black-box Attacks on VLM-powered Web Agents
Paper • 2410.17401 • Published -
MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Control
Paper • 2410.17520 • Published -
MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark
Paper • 2410.19168 • Published • 19 -
AgentStore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistant
Paper • 2410.18603 • Published • 32 -
Infogent: An Agent-Based Framework for Web Information Aggregation
Paper • 2410.19054 • Published -
Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms
Paper • 2410.18967 • Published • 1 -
VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Tasks
Paper • 2410.19100 • Published • 6 -
OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization
Paper • 2410.19609 • Published • 17 -
EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Data
Paper • 2410.19461 • Published -
OS-ATLAS: A Foundation Action Model for Generalist GUI Agents
Paper • 2410.23218 • Published • 47 -
Evaluating Cultural and Social Awareness of LLM Web Agents
Paper • 2410.23252 • Published -
DynaSaur: Large Language Agents Beyond Predefined Actions
Paper • 2411.01747 • Published • 28 -
DesignRepair: Dual-Stream Design Guideline-Aware Frontend Repair with Large Language Models
Paper • 2411.01606 • Published -
Learning to Assist Humans without Inferring Rewards
Paper • 2411.02623 • Published -
Attacking Vision-Language Computer Agents via Pop-ups
Paper • 2411.02391 • Published -
CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environments
Paper • 2411.02305 • Published -
WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning
Paper • 2411.02337 • Published • 35 -
HtmlRAG: HTML is Better Than Plain Text for Modeling Retrieved Knowledge in RAG Systems
Paper • 2411.02959 • Published • 68 -
CodeTree: Agent-guided Tree Search for Code Generation with Large Language Models
Paper • 2411.04329 • Published -
AgentOps: Enabling Observability of LLM Agents
Paper • 2411.05285 • Published -
Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks
Paper • 2411.04468 • Published • 1
Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents
Paper • 2411.06559 • Published • 13Note Developing an agent (WebDreamer) that synthesizes a single-step planning horizon to improve agent planning capabilities.
-
The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Use
Paper • 2411.10323 • Published • 32 -
CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrieval
Paper • 2411.12644 • Published • 3 -
AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations
Paper • 2411.13451 • Published -
ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow Data
Paper • 2411.15004 • Published • 1 -
ShowUI: One Vision-Language-Action Model for GUI Visual Agent
Paper • 2411.17465 • Published • 80 -
Large Language Model-Brained GUI Agents: A Survey
Paper • 2411.18279 • Published • 29 -
SketchAgent: Language-Driven Sequential Sketch Generation
Paper • 2411.17673 • Published • 18 -
Challenges in Human-Agent Communication
Paper • 2412.10380 • Published • 2 -
Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction
Paper • 2412.04454 • Published • 61 -
The BrowserGym Ecosystem for Web Agent Research
Paper • 2412.05467 • Published • 20 -
Unraveling the Complexity of Memory in RL Agents: an Approach for Classification and Evaluation
Paper • 2412.06531 • Published • 71 -
From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons
Paper • 2412.08442 • Published -
Falcon-UI: Understanding GUI Before Following User Instructions
Paper • 2412.09362 • Published -
AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials
Paper • 2412.09605 • Published • 28 -
Large Action Models: From Inception to Implementation
Paper • 2412.10047 • Published • 32 -
Attention-driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models without Fine-Tuning
Paper • 2412.10840 • Published -
GUI Agents: A Survey
Paper • 2412.13501 • Published • 25 -
Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agents
Paper • 2412.13194 • Published • 12 -
Agent-SafetyBench: Evaluating the Safety of LLM Agents
Paper • 2412.14470 • Published • 12 -
TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks
Paper • 2412.14161 • Published • 51 -
Aria-UI: Visual Grounding for GUI Instructions
Paper • 2412.16256 • Published • 1 -
PC Agent: While You Sleep, AI Works -- A Cognitive Journey into Digital World
Paper • 2412.17589 • Published • 12 -
AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generation
Paper • 2412.18116 • Published -
OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis
Paper • 2412.19723 • Published • 82 -
Training Software Engineering Agents and Verifiers with SWE-Gym
Paper • 2412.21139 • Published • 21 -
AutoPresent: Designing Structured Visuals from Scratch
Paper • 2501.00912 • Published • 8 -
A3: Android Agent Arena for Mobile GUI Agents
Paper • 2501.01149 • Published • 22 -
PPTAgent: Generating and Evaluating Presentations Beyond Text-to-Slides
Paper • 2501.03936 • Published • 19 -
InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection
Paper • 2501.04575 • Published • 23 -
WebWalker: Benchmarking LLMs in Web Traversal
Paper • 2501.07572 • Published • 19 -
Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environments
Paper • 2501.10893 • Published • 24 -
Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks
Paper • 2501.11733 • Published • 28 -
EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agents
Paper • 2501.11858 • Published • 5 -
Agent-R: Training Language Model Agents to Reflect via Iterative Self-Training
Paper • 2501.11425 • Published • 91 -
Optimizing Return Distributions with Distributional Dynamic Programming
Paper • 2501.13028 • Published -
GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration
Paper • 2501.13896 • Published -
CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigation
Paper • 2501.16609 • Published • 6 -
Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitate
Paper • 2501.17703 • Published • 54 -
Reinforcement Learning for Long-Horizon Interactive LLM Agents
Paper • 2502.01600 • Published -
Teaching Language Models to Critique via Reinforcement Learning
Paper • 2502.03492 • Published • 22 -
ReachAgent: Enhancing Mobile Agent via Page Reaching and Operation
Paper • 2502.02955 • Published -
Self-Regulation and Requesting Interventions
Paper • 2502.04576 • Published -
Towards Internet-Scale Training For Agents
Paper • 2502.06776 • Published • 4 -
Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks
Paper • 2502.08586 • Published