I looked, and it seems like there aren’t enough LLM evals out there!

Haha!

So I made a new eval, (don’t) sue me. Like making an album, it’s just a thing that people should do at least once, at some point in their lives. It’s not a thing that’s wrong to do.

I’m unsure of why I made it. Beyond the obvious “because” and “I wonder”.

But yeah - during my work on the Minivac 601 Simulator in the grand old year of 2025, I noticed that LLMs were interesting to watch (and fail) whenever they tried to follow / resolve the result of a logical circuit.

You’d say, “Hey LLM, I have a few NAND gates, and they’re connected so, and I’m going to put 0 1 1 1 across their inputs – can you tell me what are the output values? And then as the values cycle through, what are the new output values?”

LLMs (from 2025) would often go into wild crazes while trying to solve this, poorly attempting to follow flip flops through its fl(i/o)ps. Flip flop is an incredible clown duo name btw.

The (good/new) news is that in 2026, LLMs are less nutty when trying to solve this. Also, why would you (or I) even ask an LLM to “solve” logical gate circuits? Well… I think it’s interesting to see them struggle and try to remember/figure out “where they are” state-/cycle-/gate-wise. And it’s also somewhat easy to make new circuits that are just hard enough to make all LLMs fail. Being parameterizable and ostensibly correlated with other things that LLMs try to do - that’s good? Maybe.

Have fun reading about it and talking about it, tweaking your new models to be better at it, building startups around it, and considering me for you know what… Haha!

Be well, xx