Skip to content
Skip to content

Nathan Espejo

Software engineer building applied AI tools. 3rd-year Software Engineering at Western.

Ask about me

all projects

AWARE

A solo Hack the 6ix build: an $80 Arduino that sees, hears and follows plain-English rules, with every model on the board.

Jul 2026Hack the 6ix 2026solo

tl;dr

  • Every model runs on one $80 Arduino UNO Q. No cloud inference, but not offline: the camera and mic are processed on the board, and alerts go out over Telegram.
  • You type a rule like "when glass breaks after 10pm alert me". An on-device LLM turns it into a deterministic rule.
  • A rules engine checks every active rule each 500 ms. No LLM in that loop.
  • YOLOv8n runs on the board's Adreno GPU through ONNX Runtime QNN.

Why I built it

My goal for the edge track: a device where you describe what to watch for in plain English, and every model, including the language model, runs on the board. Nothing goes to a cloud model.

What I built

I built it solo, so every part is mine: the perception pipeline, the LLM rule compiler and parser, the rules engine, memory digests, the dashboard, the bridge to the microcontroller, and deployment.

How it works

A camera, microphone and sensors feed an always-on perception pipeline. A plain-English command goes through the on-device language model to become a rule. The rules engine checks every rule every 500 milliseconds and drives a speaker, LEDs, Telegram alerts and video clips.
  1. Perception, always on. YOLOv8n object detection at 2 Hz, plus BlazeFace, MediaPipe pose and hands, and MiDaS depth. YAMNet sorts sounds into 521 classes at 4 Hz. The STM32 microcontroller reads temperature, distance and motion, with Edge Impulse anomaly detection.
  2. Reasoning, on demand. The LLM reads your command and calls create_rule() with "when" and "then" fields. A plain parser compiles those into triggers and actions.
  3. Rules, every 500 ms. The engine checks each rule against the latest perception data. The LLM is not involved.
  4. Action. Piper TTS on a Bluetooth speaker, LEDs, servo or relay control, Telegram alerts, video clips, and live dashboard updates.
  5. Memory. Every event goes into SQLite. Every 5 minutes the LLM compresses recent events into a short digest. Ask "what happened today?" and it answers from the log, on the device.

Models and cost

The on-device models AWARE runs on the board, and what each one does.
ModelJobRuntime
YOLOv8nobject detectionONNX on the Adreno 702 GPU, via QNN
YAMNetsound events, 521 classesONNX
Whisper-tinyspeech to texton-device
Piper TTStext to speechon-device
MiniCPM5-1B (GGUF Q4_K_M)commands to rules, memoryon-device

The LLM only runs when you add a rule, ask a question, or a digest is due. The 500 ms checks are plain code. No API costs, and no camera or mic data goes to a cloud model.

Results

  • Built at Hack the 6ix 2026, with all inference on one $80 board.
  • Python 3.12
  • YOLOv8n
  • YAMNet
  • Whisper-tiny
  • MiniCPM5-1B (GGUF)
  • Arduino UNO Q

Next case study: Campus