I could not find my multimeter. Looked everywhere I could think of. Eventually, gave up and sat back down at my desk… and immediately found it. It was right next to my left elbow.
With a piece of paper on top of it.
The 3.6V battery I wanted to test in fact read about 2mV, so yes, replacement ordered.