At NURDspace, we have a very specific problem: nobody remembers to put the bins out. Or bring them back in. There are two garbage containers, a blue one for paper and yellow for general waste. On collection days someone has to wheel them outside, and back in after pickup. Simple enough in theory.

We had Home Assistant firing automations based on the municipal garbage calendar: yelling on IRC every few hours, spamming Signal and Telegram, and if the space was open, triggering the TTS system to yell at people in person. Passive-aggressive, but effective enough.

We also had ultrasonic distance sensors on the containers. An ESPHome binary sensor would check whether the distance reading was within a calibrated range and report back to HA. The problem: the sensors were flaky, and if anyone bumped a container it’d spam IRC with movement alerts. This got especially unbearable once renovations started. Something had to change.

Klikos
Klikos in their natural habitat.

The Attempts

BLE Beacons

My first idea: cheap BLE beacons from AliExpress, paired with an ESP32. The plan was to estimate distance from RSSI. A container outside would read weaker than one inside. Simple enough. Mounting was the problem. The beacon baseplate got screwed directly into the container (sorry Multipacity!), with the PCB housing clipped on top. We lost the first beacon when the garbage truck emptied the bin. The second one fell off before we even got that far.

YOLO Object Detection

So I figured: why not run YOLO on the frames? Easier said than done, it turned out.

I found a garbage container dataset on Roboflow, trained a custom YOLO model on my workstation (RTX 5090, so at least training was fast), and deployed it. The results were… not great. Our camera is mounted in a suboptimal position too high, not pointed directly at the containers but down at them. I briefly considered putting my own dataset together, but I found that it was more troublesome than worth it to find images of Dutch containers and the right angles.

Klikos
YOLO thinking we put our frikadellen in here.
Klikos
YOLO after training on containers

The angle was just bad enough that the bounding boxes were nearly useless. Sometimes it’d see one container and miss the other entirely, sometimes barely clip either of them. Nothing reliable.

I did consider a workaround: since YOLO gives you a bounding box, you could crop just the top portion of the detected object to isolate the lid and check the dominant colour. But the complexity was spiralling fast and the underlying detection was still shaky, so results would’ve been shaky too.

Why not ask a VLM?

Instead of trying to localise the bins geometrically, why not just ask a vision language model what it sees? VLMs are surprisingly good at this kind of fuzzy “tell me about the colours in this image” task. Far better than fighting with bounding boxes and colour histograms.

KlikoAI now uses mistral3:14b via llama-swap with llama.cpp as the backend. It runs on gpu.vm.nurd.space, a VM at the space running NixOS with an Nvidia RTX 3080.

The prompt is very simple:

Look carefully at this image.
I can see garbage containers. Tell me which colored LIDS are visible:

1. Is there a YELLOW or cream colored lid?
2. Is there a BLUE or dark blue colored lid?

Reply ONLY with valid JSON using double quotes:
{"yellow": true, "blue": true}

Detection isn’t 100% perfect though. Occasionally it still misses a container. But it’s a massive improvement over everything we tried before. The JSON output parses cleanly and the false positive rate is low enough to actually trust.

Inference takes around 40 seconds, but that includes loading and unloading the model dynamically via llama-swap since it’s a shared GPU machine. The actual vision inference is much faster than it sounds.

Klikos
What the camera sees
Klikos
What we see

Illuminating

The bins live in a fairly dark corner of the space. Poor lighting → bad photos → confused model. So I added a couple of bright LED strips controlled by an ESP8266 and an N-channel MOSFET. When KlikoAI triggers an inference run, it turns the lights on via Home Assistant, snaps the photo, then turns them back off.

Side effect: the lights occasionally scare the hell out of people walking past the space at night. I’ve added a strobe mode for gabber parties. These things happen.

Klikos
The lights on at max brightness.
Klikos
The controller box housing the mosfet and a ESP8266.

Automation

KlikoAI publishes its state as an MQTT sensor to Home Assistant:

  • Paper: Visible / Paper: Not Visible
  • Waste: Visible / Waste: Not Visible

Home Assistant uses these to gate the garbage notification: PUT THE BINS OUTSIDE keeps firing until the relevant bin is not visible, i.e., it’s been taken out. The automation shuts up the moment someone actually does the thing. For the time being there’s no automation yet to yell at people to bring the bin back inside, but getting them out is the more pressing problem. That’s next on the list.

KlikoAI also integrates with NURDbot, our IRC bot, via two commands:

  • !klikos — display current container states
  • !klikoupdate — trigger a manual inference run immediately
<@Melan> !klikoupdate
<@nurdbot> [Klikos] Running kliko inference task...
<@nurdbot> [Klikos] Paper: Visible Waste: Visible | Last Update: 2026-04-14 19:52:04 | Avg time: 41.48 secs

Schedule

KlikoAI uses the Home Assistant calendar integration to know when collection days are. On normal days it scans every 6 hours, enough to catch if someone forgot to bring a bin back or moved it. On collection days it switches to a much more frequent schedule to catch the “bin is now outside” state quickly and kill the reminder notifications.

There’s also the idea to add logic to detect significant frame changes (in case something moves in front of the camera, or the lights change, triggering an unscheduled scan).

What’s Next

  • Coding/testing the frame-change detection.
  • Test how the performance/error rate detection is with newer models like Gemma4 or Qwen3.5.
  • Yeet bin back indoors messages.

NURDspace Project page: https://nurdspace.nl/KlikoAI