AI testing agent · Mock board, made-up tickets

Passed CI. Failed the ticket.

Lookout is an AI agent I built that watches the “Ready for Test” column on a project board. When a ticket lands there, it checks the build against what the ticket asked for and shows its work. It’s a personal project, built from scratch on my own time. The demo runs on a mock board with made-up tickets.

The full demo, 3 minutes. No sound; the captions are on screen.

When a ticket lands, it

What it caught

  1. 01

    Ready for test

    A developer marks “Gangway scans are never lost” ready for test. CI passes all 6 unit tests.

    CI · 6 OF 6 UNIT TESTS PASSED
  2. 02

    Two of three

    Lookout finds that only 2 of the 3 acceptance criteria are met. A day-return pass gets scanned twice, once going out and once coming back, but only the first scan reaches the office.

    VERDICT · fix-not-verified · 2 OF 3 CRITERIA MET
  3. 03

    Why CI missed it

    A unit test counted the second scan as a duplicate, so the test passed while the requirement failed.

    test_same_pass_never_sent_twice · PASSED
  4. 04

    A person decides

    Lookout drafts a bug with exact steps to reproduce it and an explanation of why CI missed it. A person reviews it, files it, and reopens the ticket.

    TRACKER WRITES BY LOOKOUT · 0

Design choices

AI

The agent can be wrong, so it has to show its work. Each claim either cites the code or is marked as a guess, and every log is kept.

Product

Lookout can read the tracker but can’t post to it. It drafts, and a person decides. The tool does the legwork; people keep the judgment.

Quality

Code, not the model, picks the verdict from a fixed list: reproduced, not reproduced, partially reproduced, fix verified, fix not verified, or inconclusive. The model can’t make up a softer word like “mostly works.”

“It passed CI” and “it does what the ticket asked” are two different claims. Lookout is built to check the second.