21 build logs. Months of work. One shipped project. Here's the full journey—and how to set up the Phone Agent on your own device.
The first build log was published months ago. "I'm building an AI agent that controls a phone." No code. No repo. Just an idea and a cracked phone.
Today, after 21 build logs spread across months, the project is shipped.
This wasn't a straight line. There were gaps. Weeks where the log went silent—not because the work stopped, but because life doesn't pause for build logs. I took breaks to survive exam season. I paused to ship 9 portfolio websites. I stepped away when the code refused to cooperate and my brain needed rest.
But every time I came back, the agent was still there. Waiting. And every log picked up where the last one left off.
What We Built
An autonomous AI agent that controls an Android phone using natural language commands. It can parse your words into actions, read the screen, tap buttons, type text, switch between apps, verify financial data, and serve a web interface—all offline.
The Real Timeline
| Phase | What Happened |
|---|---|
| Days 1-4 | Foundation. Gemma 4 + ADB. First working pipeline. |
| Days 5-8 | Vision overhaul. UI tree. OCR. Template matching. |
| Days 9-12 | Accessibility audit. 30 apps scored. |
| Days 13-16 | Multi-app workflows. Task memory. Home reset. |
| Days 17-19 | Financial verification. Accuracy from 80% to 94%. |
| Breaks | Exams. Portfolio sites. Life. |
| Days 20-21 | Web interface. Flask backend. Shipped. |
📖 FULL SETUP GUIDE: How to Install and Use the Phone Agent
Follow these steps to get the agent running on your own Android phone.
Prerequisites
- An Android phone (Android 7 or later)
- At least 6GB of free storage space
- A WiFi connection for the initial download
Step 1: Install Termux
Do NOT install Termux from the Google Play Store—that version is outdated. Install it from F-Droid instead.
- Open your phone's browser
- Go to
f-droid.org - Download and install the F-Droid app
- Open F-Droid and search for "Termux"
- Install Termux from F-Droid
Step 2: Set Up Termux
Open Termux and run these commands one by one. Wait for each to finish before typing the next.
bash
pkg update
pkg upgrade
pkg install python git curl wget
pkg install tesseract
pkg install android-tools
Step 3: Install Ollama
Ollama is the tool that runs Gemma 4 locally on your phone.
bash
curl -fsSL https://ollama.com/install.sh | sh
After installation, start Ollama:
bash
ollama serve
Open a new Termux session (swipe from left, tap "New session") and pull the Gemma 4 model:
bash
ollama pull gemma4:4b
This downloads about 2-3GB. Use WiFi. Be patient.
Step 4: Clone the Phone Agent Repository
bash
git clone https://github.com/Dexter2344/phone-agent.git
cd phone-agent
Step 5: Install Python Dependencies
bash
pip install flask flask-cors requests numpy PyPDF2 lancedb
Step 6: Enable Developer Options and ADB
Go to Settings → About Phone
Tap "Build Number" 7 times until you see "You are now a developer"
Go back to Settings → Developer Options
Enable "USB Debugging"
Connect your phone to itself via ADB:
bash
adb devices
You may need to approve a prompt on your phone. Once approved, you'll see your device listed.
Step 7: Start the Phone Agent
Using the Web Interface (Recommended):
bash
python server.py
Then open your phone's browser and go to: http://localhost:5000
You'll see a chat interface. Type commands like:
"Open WhatsApp and send a message to Mom saying I'll call later"
"Copy my bank balance and send it to Mom on WhatsApp"
Using the Terminal (Alternative):
bash
python agent.py
Then type commands directly into the terminal.
Troubleshooting
Problem Fix
"Ollama not found" Make sure you ran ollama serve in a separate Termux session
"ADB device not found" Check USB Debugging is enabled. Run adb kill-server then adb devices again
"Module not found" Run pip install for the missing module
"Tesseract not found" Run pkg install tesseract
Phone gets hot Close other apps. The agent is CPU-intensive. Take breaks.
Agent misreads numbers The verification layer will catch most errors. For financial data, always double-check.
Using the Web Interface
Once server.py is running and you've opened http://localhost:5000:
Status bar: Green dots mean Ollama and ADB are connected. Red dots mean something is wrong.
Command input: Type what you want the agent to do. Be specific.
Example commands: Click any example to auto-fill and send.
History: Your last 20 commands and responses are stored.
The Repo
👉 github.com/Dexter2344/phone-agent
21 build logs. Months of work. One shipped project. Full setup guide included.
Thank You
To everyone who followed this log—through the daily posts and the weeks of silence—thank you. This was never about shipping fast. It was about shipping honestly.
This was Log #1:
"I'm starting a new project. It's the most ambitious thing I've attempted from a phone."
This is Log #21. We shipped.
Top comments (6)
How did you check that there was demand for your value proposition and MVP scope?
I validated through the build itself. The accessibility audit post generated inbound questions from developers asking how to fix their apps. Followers consistently asked for the same features a real interface, easier setup, multi-app support and those directly became the MVP scope. Plus the broader market momentum around AI agents confirmed the direction was sound.
Interesting project. The verification layer for financial data caught my attention, especially the jump from 80% to 94% accuracy.
I’ve been working on a somewhat similar problem in AI agent tooling (MSCodeBase): an agent can execute the correct-looking action, and the verification can also succeed, while the underlying target or source was actually wrong. In our case, we had a case where symbol resolution selected a test fixture instead of the intended source — the execution itself was valid, but the conclusion was wrong.
That made me curious about your financial-data verification: when you double/triple-read an OCR result and the readings agree, how do you distinguish correctness from a consistent OCR error?
For example, if OCR reads the same wrong value twice, the verification layer would see strong agreement — but not necessarily truth.
Have you tested the verification layer with deliberately introduced but repeatable OCR errors?
Do you keep the evidence behind each verification (what was observed, when, and by which method)? I ask because a previously verified state can become stale when the UI changes, and that evidence helps distinguish “still valid” from “needs re-verification.”
I think there’s an interesting distinction here between verification of an observation and verification of the conclusion drawn from that observation.
This is a brilliant point and honestly, you've nailed the exact blind spot in the current verification system.
Right now, the double-read confirmation only checks for consistency, not truth. If OCR consistently misreads "₦15,000" as "₦15.000" on both passes, the verification layer sees perfect agreement and passes wrong data downstream. It verified the observation, not the conclusion.
To your questions:
I haven't yet tested with deliberately introduced but repeatable OCR errors. That's a gap I need to close.
The agent currently stores the final verified value, but not the raw OCR output, screenshots, or confidence scores. So there's no audit trail to distinguish "still valid" from "needs re-verification."
Your distinction between verifying an observation and verifying the conclusion is exactly where this needs to go next.
This conversation is actually pushing me toward a v2.0 with evidence logging, semantic verification, and staleness detection. Thanks for the thoughtful critique this is the kind of feedback that makes building in public worth it.
"Really glad that resonated. The fact that you're thinking about v2.0 with evidence logging is exactly the right step.
For the semantic verification (distinguishing truth from a consistent OCR error), the most effective pattern I’ve found is an independent negative control. Instead of just re-reading the screen with the same OCR engine, introduce a deliberately broken state or a second, fundamentally different method of extraction (e.g., parsing the accessibility tree or UI hierarchy instead of pixels) and require the two methods to agree. If they disagree, you hit UNKNOWN rather than passing wrong data downstream.
Keeping the raw evidence (screenshot + bounding boxes + OCR confidence) behind every verified state is what gives you that audit trail. It turns a silent failure into a debuggable one.
Looking forward to seeing how v2.0 shapes up. You’ve built something really cool here."
This is gold. The independent negative control idea just clicked for me don't just re-read with the same OCR engine, cross-check with a completely different method. If OCR says one thing and the UI hierarchy tree says another, flag it as UNKNOWN instead of passing potentially wrong data. That's the missing piece.
I'm already thinking about implementation: the agent already has access to both OCR (pixel-based) and UI tree inspection (accessibility-based). The two methods rarely agree perfectly, but a significant divergence is a strong signal that something is wrong. Combining that with raw evidence storage screenshot, bounding boxes, OCR confidence scores turns silent failures into debuggable ones.
This conversation literally shaped the architecture for v2.0. Really appreciate you taking the time to share what you've learned from MSCodeBase.