Maximum used connections forever growing

[counterpoint]

We’ve a situation where munin monitoring is showing that the maximum used connections grows (by periodic jumps) but never falls. It continues to do this until the mysql server is rebooted, when it goes back to the starting level and the process starts all over again.

It is puzzling why the number of connections is growing - numbers of threads connected or new connections are pretty steady. And running “show processlist” frequently shows only a handful of active processes.

What could be causing the apparently never ending growth in maximum used connections?

[xaprb]

This variable is a “high-water mark” so it never decreases.

[counterpoint]

OK, that makes sense - up to a point.

But it’s not clear why the maximum should be pushed up further as time goes by rather than hitting a peak within, say, 24 hours when a complete cycle of activities has been completed.

A graph of connections per second is attached. The maximum used figure can stay constant for a couple of days, but then bumps up. The same thing appears to keep happening - it’s not clear why the server hasn’t reached a steady state. The increase seems to go on and on. We’re concerned about the apparent memory requirement on a server that on the face of it is over specified for the load imposed on it.

Average figures from munin can hide spikes. But why would there be ever higher spikes over a period of many days?