Project Log #15: My AI Agent Just Completed Its First Multi-App Task

The Milestone

I gave the agent a command: "Copy my account balance from my banking app and send it to Mom on WhatsApp."

It opened the banking app. It read the balance from the screen using OCR (since banking apps have zero accessibility labels-F score, remember?). It copied the number. It switched to WhatsApp. It found Mom. It typed the balance. It hit send.

Two apps. One task. Completed autonomously.

What Made This Possible

The key was building a task memory system. The agent now stores values extracted from one app in a simple key-value store (a Python dictionary that persists for the duration of the task). When it switches to the next app, it retrieves that value and uses it.

The flow works like this:

• Parse the command into a plan with Gemma 4.
• Execute Step 1: Open banking app. Read the balance. Store it in memory as account_balance.
• Execute Step 2: Open WhatsApp. Find the contact. Retrieve account_balance from memory. Type it. Send.

The memory system is simple-just a dictionary that gets passed between steps. But it's the bridge between single-app automation and true multi-app workflows.

Today's Progress

What Still Breaks

• The banking app is slow to load on my phone. Sometimes the agent times out waiting for the balance screen to appear. I added a retry loop, but it's not perfect.
• App switching is clunky. The agent uses adb shell am start to launch apps, but if the app was already open in the background, it sometimes opens to the wrong screen. I'm working on a "reset to home screen" step between app switches.
• OCR on banking app text is unreliable. The font is small. The contrast is low. The agent sometimes misreads the balance by a digit or two. For a task like "send my balance to Mom," that's embarrassing but not catastrophic. For a task like "transfer money," that's dangerous. I need higher accuracy before I can trust this with anything financial.

The Bigger Picture

Multi-app workflows are the difference between a cool prototype and a useful tool. An agent that can only operate in one app is a novelty. An agent that can move data between apps is an assistant.

The next step is chaining more complex workflows: "Find my last three transactions, summarise them, and email them to my accountant." That's three apps. That's the goal.

What's Next (Day 16)

• Add a "reset to home screen" step between app switches
• Improve OCR accuracy on banking apps
• Test a 3-app workflow

The Repo

👉 github.com/Dexter2344/phone-agent

agent.py now includes a task memory system. Multi-app workflows are supported. README updated with Day 15 status.

This is Day 15. The agent is no longer trapped in one app.