01 / Multimodal AI · Computer vision
Roya.
Detection, OCR, maps and a language model, wired into one app.
Read the diagram
- Visual input
- Detection + OCR
- Geospatial context
- LLM interpretation
- React application
01 / Context
The problem
One photo holds several kinds of information at once: the objects in it, any text in the scene, and the place it sits in the world. Most tools only read one of those. Roya was my attempt to read them together, inside a single piece of software.
02 / Contribution
What I did
I built the platform on both sides: the Python inference backend and the React interface. Most of that work was wiring YOLO, OCR, geospatial analysis and an LLM into one flow that a person could actually use.
03 / Decisions
The approach
No single model was the hard part. The hard part was getting them to behave like one application instead of a pile of separate experiments. FastAPI sits in front as the service layer, and OpenCV and PyTorch do the visual processing underneath.
- Visual input
- Detection + OCR
- Geospatial context
- LLM interpretation
- React application
How the main pieces fit together.
04 / Evidence
What I can show
The pipeline has handled roughly 1,000 images, and the frontend and backend code are both public. Treat that number as a sense of scale. It says how much the workflow processed, not how accurate or fast it was.
05 / Reflection
Limits & lessons
It’s a prototype. The camera feeds are simulated and the profiling data is mock, so this isn’t a system running in the field. The lesson I kept from it: hold what the camera actually saw apart from what the language model says about it. An LLM’s summary is an interpretation, and it shouldn’t be dressed up as evidence.