Can't create thread error

MySQL & MariaDB Percona Server for MySQL 5.7
1

Hi,

We recently upgraded to Percona Server 5.7 on Ubuntu 16.04 (from 5.6 on 12.04) and started getting the

Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug

error.

This error is very well reported but all the solutions I find are so far related to sort_buffer_size being too large or ulimit max processes set too low, and I can’t find any evidence for either on our server.

Does anyone know of any other reasons for getting this error?

Regards,
Patrik

The, from what I can tell, relevant system information:


$ cat /proc/`pidof mysqld`/limits | egrep "(processes|files)"
Max processes 515161 515161 processes 
Max open files 65536 65536 files 

$ cat /proc/sys/kernel/threads-max
1030322

$ free -h
total used free shared buff/cache available
Mem: 125G 86G 18G 62M 21G 38G
Swap: 4.7G 2.2G 2.5G

mysql> SHOW GLOBAL VARIABLES LIKE 'key_buffer_size'; 
+-----------------+-----------+
| Variable_name | Value |
+-----------------+-----------+
| key_buffer_size | 536870912 |
+-----------------+-----------+
1 row in set (0.00 sec)

mysql> SHOW GLOBAL VARIABLES LIKE 'read_buffer_size'; 
+------------------+---------+
| Variable_name | Value |
+------------------+---------+
| read_buffer_size | 2097152 |
+------------------+---------+
1 row in set (0.01 sec)

mysql> SHOW GLOBAL VARIABLES LIKE 'sort_buffer_size';
+------------------+--------+
| Variable_name | Value |
+------------------+--------+
| sort_buffer_size | 262144 |
+------------------+--------+
1 row in set (0.00 sec)

mysql> SHOW GLOBAL VARIABLES LIKE 'max_connections'; 
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| max_connections | 1000 |
+-----------------+-------+
1 row in set (0.01 sec)

mysql> SHOW GLOBAL STATUS LIKE 'Threads%'; 
+-------------------+-------+
| Variable_name | Value |
+-------------------+-------+
| Threads_cached | 71 |
| Threads_connected | 35 |
| Threads_created | 95096 |
| Threads_running | 2 |
+-------------------+-------+
4 rows in set (0.00 sec)

mysql> SHOW GLOBAL STATUS LIKE 'open_files'; 
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| Open_files | 696 |
+---------------+-------+
1 row in set (0.00 sec)

mysql> SHOW GLOBAL STATUS LIKE 'max_used_connections';
+----------------------+-------+
| Variable_name | Value |
+----------------------+-------+
| Max_used_connections | 474 |
+----------------------+-------+
1 row in set (0.00 sec)

mysql> SHOW GLOBAL STATUS LIKE 'connections';
+---------------+-----------+
| Variable_name | Value |
+---------------+-----------+
| Connections | 288807853 |
+---------------+-----------+
1 row in set (0.00 sec)

mysql> SHOW GLOBAL VARIABLES LIKE 'open_files_limit'; 
+------------------+-------+
| Variable_name | Value |
+------------------+-------+
| open_files_limit | 65536 |
+------------------+-------+
1 row in set (0.00 sec)