Asynchronous CLI Agents in GitHub Actions

The CLI strikes back — your guide for delegating coding work to CLI agents running asynchronously in CI

This guide explores an alternative to cloud-hosted products for asynchronous AI-assisted coding: running command-line interface (CLI) agents within a Continuous Integration (CI) workflow. Specifically, we'll examine how tools like Claude Code, Gemini CLI, and opencode can be deployed as GitHub Actions to function as autonomous agents.

Through a practical journey of refactoring a repository, this report details the setup, user experience, and outcomes of using these three agents. The conclusion is that while this approach requires more initial configuration than their SaaS counterparts, it offers significant advantages in flexibility, control, and model choice, making it a powerful option for developers and engineering teams.

See our previous article on using cloud-hosted asynchronous AI coding agents

CLI Coding Agents in GitHub Actions

While cloud-hosted platforms like GitHub Copilot Agent offer a streamlined experience, integrating a CLI coding agent into your CI pipeline presents a compelling alternative. This method involves configuring a tool to run non-interactively as part of a workflow, typically triggered by an event like a comment on a GitHub issue.

Advantages

• Choice and Flexibility: You can select from a wide range of open-source and commercial CLI agents. Open-Source tools like opencode further expand your options by allowing you to use virtually any model from any provider, from open models like Qwen to proprietary ones like Claude or GPT-5.

• Control and Security: The agent operates within your own CI environment — whether it's a GitHub-hosted runner or your own self-hosted infrastructure. This gives you complete control over the execution environment and can address specific security or compliance requirements.

• Configurability: The agent's behavior is defined in a workflow .yml file, giving you fine-grained control over its prompts, triggers, permissions, and the specific commands it executes.

Trade-Offs

• Setup Complexity: Installation is more involved than simply enabling a cloud service. It typically requires adding API keys as repository secrets, setting refined permissions and access control for the execution environment, and committing workflow files to your repository.

• User Experience: The user interface is limited to the output of the GitHub Action run and comments posted to issues or pull requests. This is generally less polished than the dedicated UIs of cloud-hosted products. However, as these agents operate asynchronously, an elaborate UI is often unnecessary once the system is functioning correctly.

A Refactoring Journey with CLI Agents

To evaluate this approach, I used several CLI agents to perform a series of well-defined refactoring tasks on a project repository. The setup for each involved adding the necessary API key(s) as secrets to the GitHub repository.

Claude Code

Claude Code was the first agent that popularized this CI-based workflow.

Setup

The setup process was remarkably smooth.

1. I ran the /install-github-app in Claude Code command locally.

2. This command opened a browser window, guiding me through the process of installing the Claude GitHub App on my repository.

3. Once authorized, Claude automatically created a pull request to add the necessary GitHub Actions workflow files to my repository.

4. After reviewing and merging the PR, the setup was complete.

Execution and Experience

To trigger the agent, I commented @claude on a GitHub issue, followed by my request.

The user experience was highly polished. Claude promptly replied with a comment confirming it was working, including a link to the live job run. It then posted another comment with a detailed, itemized to-do list, which it updated in real-time as it progressed. This provided excellent visibility into its process. The commits and comments were made by a dedicated "claude" user, making its contributions easy to identify.

While the raw output in the GitHub Actions log was a verbose stream of JSON, the well-formatted updates in the issue itself were more than sufficient for monitoring its work.

Outcome

After about 15 minutes, Claude commented that it had finished and provided a link to create a pull request. The initial PR had failing CI tests. However, by commenting on the PR and asking Claude to fix the errors, it successfully identified the problem, created a new plan, and pushed a fix. The final PR was correct and ready to merge. The only significant limitation is that this tool works exclusively with Anthropic's Claude models.

Gemini CLI

Next, I tested the GitHub Action for Google's Gemini CLI, which is currently in beta.

Setup

1. I ran the /setup-github command in Gemini CLI locally.

2. This created the workflow .yml file in my local repository.

3. I then had to manually commit and push this file to GitHub.

This process was straightforward but lacked the automation provided by Claude Code.

Execution and Experience

I invoked the agent by commenting @gemini-cli on an issue. It replied that it had received the request. The experience felt less refined than Claude's; comments came from the generic github-actions bot user rather than a dedicated Gemini user.

Like Claude, Gemini posted a to-do list in a comment and updated it as it worked. However, it did not provide a direct link to the job run. When I located the run manually, I found that it produced no log output, making it impossible to observe its internal process. While not critical for an asynchronous task, this lack of visibility was a drawback.

Outcome

Gemini completed the task successfully. The generated code was correct, and all tests passed on the first attempt. This was a notable success, especially when contrasted with a previous poor experience using Google's cloud-hosted agent, Jules, which failed on a similar task. The Gemini CLI installation also included an optional workflow for automatically reviewing new pull requests, which provided a brief but accurate summary of the changes.

opencode

opencode positions itself as the premier open-source alternative to Claude Code, and its GitHub Action integration is a core feature.

Setup

The opencode github install command provides an excellent interactive setup experience.

1. The CLI prompts you to choose a provider (e.g., OpenAI, OpenRouter, Groq).

2. It then prompts you to choose a specific model.

3. Based on your selections, it generates a customized workflow .yml file locally.

4. After committing and pushing the file, the setup is complete.

For the first test, I configured it to use Qwen3 Coder via OpenRouter to experiment with an open model.

Execution and Experience

I triggered the agent with a /oc comment on an issue. The user experience was excellent. The agent, posting as a distinct opencode-agent user, provided a link to the job run. The log output was clean, structured, and easy to follow, clearly indicating which tools were being used at each step.

A standout feature was the link provided upon completion to a shared conversation on the opencode web platform. This view presented the entire agent run — including thoughts, tool calls, and file diffs — in a beautifully formatted, readable interface.

Outcome

The combination of opencode and Qwen3 Coder (likely running on a fast inference provider) was extremely fast, completing the task in just a few minutes. The initial results were perfect. However, subsequent attempts with the same open model combination were mixed, encountering some timeout errors.

To test its flexibility, I re-ran the opencode github install command, this time configuring it for OpenAI's GPT-5. The agent again performed flawlessly, demonstrating the power of being able to easily switch between different models and providers to find the best fit for a given task.

Other CLI Agents

I considered testing other agents like OpenHands, Codex CLI, and Qwen Code. However, none offered a guided installation for GitHub Actions. While it is certainly possible to configure them manually, the effort required was beyond the scope of this particular evaluation.

Conclusion and Summary

The experience of using CLI agents within GitHub Actions as asynchronous, autonomous developers was positive. This approach provides a viable and powerful alternative to fully-managed cloud products.

Here is a comparative summary of the tested tools:

• Claude Code: Offers the most polished and seamless user experience, with excellent feedback mechanisms built into the GitHub issue workflow. Its primary limitation is its exclusivity to Anthropic's models.

• Gemini CLI: A promising beta product that delivered a high-quality result. While its user experience is currently less refined, it is easy to set up and works effectively. It is a promising tool that will likely improve over time. And like Claude, it is limited to Google's Gemini models.

• opencode: Delivers an outstanding combination of user experience and flexibility. Its interactive setup, clear logging, web viewer, and ability to use any model from any provider make it an exceptionally powerful open-source choice.

Ultimately, the decision between a cloud-hosted product and a CI-integrated CLI agent depends on your priorities.

• Cloud-hosted agents offer superior ease-of-use and a richer out-of-the-box user interface.

• CLI agents in CI offer unparalleled flexibility, configurability, and control, allowing you to tailor the solution precisely to your needs and infrastructure.