I Built an Android App That Tracks Screens Using Computer Vision

I Built an Android App That Tracks Screens Using Computer Vision

Most of my earlier projects were fairly conventional applications.

They had screens.

They had data.

They had buttons.

They had users interacting with information.

Then I wanted to build something different.

I wanted the application to look at the physical world through a camera and actually understand something about what it was seeing.

That became ScreenFrame, also called Warp_Flow.

The Idea

ScreenFrame is an Android application focused on real-time screen detection and alignment.

The basic idea is:

Camera
↓
Analyze Frames
↓
Find Edges
↓
Find Corners
↓
Track Screen
↓
Calculate Perspective

The application attempts to identify the boundaries of a display in the camera view.

This sounds straightforward until you realize that the camera doesn't always see a perfect rectangle.

The Screen Isn't Always a Rectangle

Point a phone directly at a monitor and the geometry looks relatively simple.

Move the phone to the side and perspective changes everything.

The physical screen is still rectangular.

But the camera sees something closer to:

  __________
 /         /
/         /

/_________/

The four corners have shifted.

That's where homography becomes useful.

It allows the application to reason about how points in one plane relate to another perspective.

For ScreenFrame, that means using detected screen corners to help understand and align the display.

Edge Detection

The application uses edge detection to find possible boundaries.

But a camera frame contains much more than the screen.

There can be:

Reflections
Text
Images
Furniture
Objects
Lighting changes

So simply detecting every edge isn't enough.

ScreenFrame exposes Canny low and high thresholds that can be calibrated to help identify useful edges while filtering some unwanted visual information.

That turned image processing into a much more practical problem.

Real-World Lighting Is Messy

Computer vision works very differently on a controlled image compared with a real camera.

A screen might be bright.

It might be dim.

It might reflect the room.

The surrounding environment might be darker.

To deal with these conditions, ScreenFrame includes an adaptive compensation filter designed to help with ambient and low-light conditions.

That made one thing very clear to me:

Computer vision isn't just about getting the algorithm to work once.

It's about getting it to continue working when the environment changes.

Real-Time Means Real Performance

This was probably the biggest difference from my other Android applications.

A normal app might do something after a user action.

A camera application can be processing continuously.

Frame → Analyze
Frame → Analyze
Frame → Analyze
Frame → Analyze
Frame → Analyze
...

At 60 FPS, that's a lot of incoming data.

A small performance problem can happen dozens of times every second.

So I had to think about performance much earlier.

Compose and Real-Time State

The interface uses Jetpack Compose and Material 3.

The application uses reactive state with Kotlin StateFlow inside a ViewModel.

The simplified architecture looks like:

Vision Pipeline
↓
ViewModel
↓
StateFlow
↓
Compose HUD

I also use techniques such as remember, state hoisting, and controlled recomposition to avoid unnecessary UI work.

This was interesting because the computer vision pipeline and the UI aren't separate performance problems.

They affect each other.

Battery Is Part of Performance

There's another constraint on mobile devices:

battery.

Running camera analysis continuously isn't free.

ScreenFrame therefore includes an automatic battery-saving mode.

When the battery becomes low, such as below 20%, the application can reduce the analysis refresh frequency.

So instead of thinking:

Maximum performance all the time.

the application can think:

How much performance does the device actually need right now?

That's a much more practical approach for mobile software.

30 FPS vs 60 FPS

ScreenFrame supports both 30 FPS and 60 FPS workflows.

That difference matters.

At 30 FPS:

30 frames per second

At 60 FPS:

60 frames per second

The application potentially has twice as many frames to process.

That means performance problems become much easier to expose.

Building the HUD

I also wanted the UI to match the technical nature of the project.

Instead of creating a traditional Android interface, I designed a dark HUD-style control deck using Jetpack Compose and Material 3.

The application uses centralized design tokens for:

Spacing
Shapes
Typography
Visual accents

There are also multiple neon-inspired themes, including:

Neon Green
Cyber Orange
Hot Pink
Electric Purple
Electric Blue
Amber Gold

The UI is intentionally designed to feel more like a technical instrument than a typical Android application.

What I Learned

ScreenFrame changed how I think about Android development.

Before this project, I mostly thought about:

How do I make this screen work?

Now I also think about:

How many frames am I processing?

How often does this state change?

What causes recomposition?

What happens when the battery gets low?

What happens when lighting changes?

How expensive is this operation when repeated 60 times per second?

Those are very different questions.

And they pushed me toward thinking more about systems rather than individual screens.

Why I Like Building Projects

This is probably the biggest lesson I got from ScreenFrame.

When you build something from scratch, the problems aren't neatly organized into lessons.

You discover them.

You start with:

"I want to detect a screen."

Then you discover:

"I need edge detection."

Then:

"Perspective is a problem."

Then:

"I need homography."

Then:

"The camera is processing too much."

Then:

"Now I have a performance problem."

And eventually the project becomes much more technically interesting than the original idea.

That's one of the reasons I enjoy building.

The project forces you to learn things you didn't initially know you needed.

What's Next?

ScreenFrame pushed me into computer vision, real-time processing, geometry, and Android performance.

For the next project, I wanted to take everything I've been learning about application architecture and apply it to another real-world problem.

That's where the next project in my portfolio begins.