Proxysql Error Lost connection to mysql server

I am getting a lost connection error in proxysql. What should I be looking at to fix/troubleshoot the error?

Info:

ProxySQL version 2.0.14-70-g7b02b19c, codename Truls

Proxysql is an ec2 instance in AWS, I have 2 RDS servers and 1 Mysql local instance (db4).

I am getting regular errors in proxsql.log:

2020-12-15 14:29:25 MySQL_Monitor.cpp:836:monitor_ping_thread(): [ERROR] Error after 0ms on server db4:6033 : Lost connection to MySQL server during query

Looking at monitor.mysql_server_ping_log, it is connecting, but not all the time:

| db2-aws| 3306 | 1608042425144773 | 1126 | NULL |

| db4 | 6033 | 1608042434961571 | 34070 | NULL |

| db2-aws| 3306 | 1608042435078947 | 1061 | NULL |

| db1-aws| 3306 | 1608042435196261 | 1132 | NULL |

| db4 | 6033 | 1608042444961715 | 0 | Lost connection to MySQL server during query |

| db2-aws| 3306 | 1608042445085211 | 1021 | NULL |

| db1-aws| 3306 | 1608042445208458 | 1121 | NULL |

timeouts:

±-------------------------------------------------------------±---------------+

| variable_name | variable_value |

±-------------------------------------------------------------±---------------+

| mysql-connect_timeout_client | 10000 |

| mysql-connect_timeout_server | 3000 |

| mysql-connect_timeout_server_max | 10000 |

| mysql-default_query_timeout | 36000000 |

| mysql-monitor_connect_timeout | 600 |

| mysql-monitor_galera_healthcheck_max_timeout_count | 3 |

| mysql-monitor_galera_healthcheck_timeout | 800 |

| mysql-monitor_groupreplication_healthcheck_max_timeout_count | 3 |

| mysql-monitor_groupreplication_healthcheck_timeout | 800 |

| mysql-monitor_ping_timeout | 2000 |

| mysql-monitor_query_timeout | 100 |

| mysql-monitor_read_only_max_timeout_count | 3 |

| mysql-monitor_read_only_timeout | 500 |

| mysql-monitor_replication_lag_timeout | 1000 |

| mysql-monitor_wait_timeout | true |

| mysql-ping_timeout_server | 500 |

| mysql-poll_timeout | 2000 |

| mysql-poll_timeout_on_failure | 100 |

| mysql-wait_timeout | 28800000 |

Hi,

The fact that the ones failing are to port 6033 is strange (this is most likely the mysql interface from proxysql), the entries that are not failing are connecting directly to port 3306. How do you have the MySQL servers configured?

select * from runtime_mysql_servers;

Also send outputs from SHOW VARIABLES (and remember to filter out fields that have passwords or users, please).

select * from runtime_mysql_servers ;

±-------------±-------------------------------------------------±-----±----------±-------±-------±------------±----------------±--------------------±--------±---------------±--------+

| hostgroup_id | hostname | port | gtid_port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |

±-------------±-------------------------------------------------±-----±----------±-------±-------±------------±----------------±--------------------±--------±---------------±--------+

| 10 | db1-aws | 3306 | 0 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |

| 30 | db4 | 6033 | 0 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |

| 20 | db2-aws| 3306 | 0 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |

±-------------±-------------------------------------------------±-----±----------±-------±-------±------------±----------------±--------------------±--------±---------------±--------+

±-------------------------------------------------------------±-------------------------------------+

| Variable_name | Value |

±-------------------------------------------------------------±-------------------------------------+

| admin-admin_credentials | admin:pass |

| admin-checksum_mysql_query_rules | true |

| admin-checksum_mysql_servers | true |

| admin-checksum_mysql_users | true |

| admin-cluster_check_interval_ms | 1000 |

| admin-cluster_check_status_frequency | 10 |

| admin-cluster_mysql_query_rules_diffs_before_sync | 3 |

| admin-cluster_mysql_query_rules_save_to_disk | true |

| admin-cluster_mysql_servers_diffs_before_sync | 3 |

| admin-cluster_mysql_servers_save_to_disk | true |

| admin-cluster_mysql_users_diffs_before_sync | 3 |

| admin-cluster_mysql_users_save_to_disk | true |

| admin-cluster_password | |

| admin-cluster_proxysql_servers_diffs_before_sync | 3 |

| admin-cluster_proxysql_servers_save_to_disk | true |

| admin-cluster_username | |

| admin-hash_passwords | true |

| admin-mysql_ifaces | 0.0.0.0:6032 |

| admin-read_only | false |

| admin-refresh_interval | 2000 |

| admin-restapi_enabled | false |

| admin-restapi_port | 6070 |

| admin-stats_credentials | stats:stats |

| admin-stats_mysql_connection_pool | 60 |

| admin-stats_mysql_connections | 60 |

| admin-stats_mysql_query_cache | 60 |

| admin-stats_mysql_query_digest_to_disk | 0 |

| admin-stats_system_cpu | 60 |

| admin-stats_system_memory | 60 |

| admin-telnet_admin_ifaces | (null) |

| admin-telnet_stats_ifaces | (null) |

| admin-vacuum_stats | true |

| admin-version | 2.0.14-70-g7b02b19c |

| admin-web_enabled | false |

| admin-web_port | 6080 |

| mysql-add_ldap_user_comment | |

| mysql-auditlog_filename | |

| mysql-auditlog_filesize | 104857600 |

| mysql-aurora_max_lag_ms_only_read_from_replicas | 2 |

| mysql-auto_increment_delay_multiplex | 5 |

| mysql-autocommit_false_is_transaction | false |

| mysql-autocommit_false_not_reusable | false |

| mysql-automatic_detect_sqli | 0 |

| mysql-binlog_reader_connect_retry_msec | 3000 |

| mysql-client_found_rows | true |

| mysql-client_multi_statements | true |

| mysql-client_session_track_gtid | true |

| mysql-commands_stats | true |

| mysql-connect_retries_delay | 1 |

| mysql-connect_retries_on_failure | 10 |

| mysql-connect_timeout_client | 10000 |

| mysql-connect_timeout_server | 3000 |

| mysql-connect_timeout_server_max | 10000 |

| mysql-connection_delay_multiplex_ms | 0 |

| mysql-connection_max_age_ms | 0 |

| mysql-connection_warming | false |

| mysql-connpoll_reset_queue_length | 50 |

| mysql-default_charset | utf8 |

| mysql-default_collation_connection | utf8_general_ci |

| mysql-default_max_latency_ms | 1000 |

| mysql-default_query_delay | 0 |

| mysql-default_query_timeout | 36000000 |

| mysql-default_reconnect | true |

| mysql-default_schema | information_schema |

| mysql-default_session_track_gtids | OFF |

| mysql-default_tx_isolation | READ-COMMITTED |

| mysql-enforce_autocommit_on_reads | false |

| mysql-eventslog_default_log | 0 |

| mysql-eventslog_filename | |

| mysql-eventslog_filesize | 104857600 |

| mysql-eventslog_format | 1 |

| mysql-firewall_whitelist_enabled | 0 |

| mysql-firewall_whitelist_errormsg | Firewall blocked this query |

| mysql-forward_autocommit | false |

| mysql-free_connections_pct | 10 |

| mysql-handle_unknown_charset | 1 |

| mysql-have_compress | true |

| mysql-have_ssl | false |

| mysql-hostgroup_manager_verbose | 1 |

| mysql-init_connect | |

| mysql-interfaces | 0.0.0.0:3306;0.0.0.0:6033 |

| mysql-keep_multiplexing_variables | tx_isolation,version |

| mysql-kill_backend_connection_when_disconnect | true |

| mysql-ldap_user_variable | |

| mysql-log_unhealthy_connections | true |

| mysql-long_query_time | 1000 |

| mysql-max_allowed_packet | 1073741824 |

| mysql-max_connections | 2048 |

| mysql-max_stmts_cache | 10000 |

| mysql-max_stmts_per_connection | 20 |

| mysql-max_transaction_time | 14400000 |

| mysql-min_num_servers_lantency_awareness | 1000 |

| mysql-mirror_max_concurrency | 16 |

| mysql-mirror_max_queue_length | 32000 |

| mysql-monitor_connect_interval | 60000 |

| mysql-monitor_connect_timeout | 600 |

| mysql-monitor_enabled | true |

| mysql-monitor_galera_healthcheck_interval | 5000 |

| mysql-monitor_galera_healthcheck_max_timeout_count | 3 |

| mysql-monitor_galera_healthcheck_timeout | 800 |

| mysql-monitor_groupreplication_healthcheck_interval | 5000 |

| mysql-monitor_groupreplication_healthcheck_max_timeout_count | 3 |

| mysql-monitor_groupreplication_healthcheck_timeout | 800 |

| mysql-monitor_groupreplication_max_transactions_behind_count | 3 |

| mysql-monitor_history | 600000 |

| mysql-monitor_password | pass |

| mysql-monitor_ping_interval | 10000 |

| mysql-monitor_ping_max_failures | 3 |

| mysql-monitor_ping_timeout | 2000 |

| mysql-monitor_query_interval | 60000 |

| mysql-monitor_query_timeout | 100 |

| mysql-monitor_read_only_interval | 1500 |

| mysql-monitor_read_only_max_timeout_count | 3 |

| mysql-monitor_read_only_timeout | 500 |

| mysql-monitor_replication_lag_count | 1 |

| mysql-monitor_replication_lag_interval | 10000 |

| mysql-monitor_replication_lag_timeout | 1000 |

| mysql-monitor_replication_lag_use_percona_heartbeat | |

| mysql-monitor_slave_lag_when_null | 60 |

| mysql-monitor_threads_max | 128 |

| mysql-monitor_threads_min | 8 |

| mysql-monitor_threads_queue_maxsize | 128 |

| mysql-monitor_username | proxy_monitor |

| mysql-monitor_wait_timeout | true |

| mysql-monitor_writer_is_also_reader | true |

| mysql-multiplexing | true |

| mysql-ping_interval_server_msec | 120000 |

| mysql-ping_timeout_server | 500 |

| mysql-poll_timeout | 2000 |

| mysql-poll_timeout_on_failure | 100 |

| mysql-query_cache_size_MB | 256 |

| mysql-query_cache_stores_empty_result | true |

| mysql-query_digests | true |

| mysql-query_digests_lowercase | false |

| mysql-query_digests_max_digest_length | 2048 |

| mysql-query_digests_max_query_length | 65000 |

| mysql-query_digests_no_digits | false |

| mysql-query_digests_normalize_digest_text | false |

| mysql-query_digests_replace_null | false |

| mysql-query_digests_track_hostname | false |

| mysql-query_processor_iterations | 0 |

| mysql-query_processor_regex | 1 |

| mysql-query_retries_on_failure | 1 |

| mysql-reset_connection_algorithm | 2 |

| mysql-server_capabilities | 569899 |

| mysql-server_version | 5.5.30 |

| mysql-servers_stats | true |

| mysql-session_idle_ms | 1000 |

| mysql-session_idle_show_processlist | true |

| mysql-sessions_sort | true |

| mysql-set_query_lock_on_hostgroup | 1 |

| mysql-show_processlist_extended | 0 |

| mysql-shun_on_failures | 5 |

| mysql-shun_recovery_time_sec | 10 |

| mysql-ssl_p2s_ca | |

| mysql-ssl_p2s_cert | |

| mysql-ssl_p2s_cipher | |

| mysql-ssl_p2s_key | |

| mysql-stacksize | 1048576 |

| mysql-stats_time_backend_query | false |

| mysql-stats_time_query_processor | false |

| mysql-tcp_keepalive_time | 0 |

| mysql-threads | 4 |

| mysql-threshold_query_length | 524288 |

| mysql-threshold_resultset_size | 4194304 |

| mysql-throttle_connections_per_sec_to_hostgroup | 1000000 |

| mysql-throttle_max_bytes_per_second_to_client | 0 |

| mysql-throttle_ratio_server_to_client | 0 |

| mysql-use_tcp_keepalive | 0 |

| mysql-verbose_query_error | false |

| mysql-wait_timeout | 28800000 |

Can you explain a bit more your setup? Do you have the db4 mysql instance running in the same node as the proxysql service? I see you are using port 6033 for db4, is MySQL listening on it, or ProxySQL? From the config outputs you sent, it may be ProxySQL itself:

| mysql-interfaces | 0.0.0.0:3306;0.0.0.0:6033 |

We can get more information on this node’s timeouts in particular with:

SELECT * FROM monitor.mysql_server_ping_log WHERE hostname = ‘db4’;

Do you see any timeout patterns there?

ec2 instance with proxysql connecting to local server with mysql and proxysql. Mysql listening on 3306 proxysql listening on 6033

±---------------±-----±-----------------±---------------------±---------------------------------------------+

| hostname | port | time_start_us | ping_success_time_us | ping_error |

±---------------±-----±-----------------±---------------------±---------------------------------------------+

|db4 | 6033 | 1609257997551350 | 33664 | NULL |

|db4 | 6033 | 1609258007723383 | 33634 | NULL |

|db4 | 6033 | 1609258017649089 | 272729 | NULL |

|db4 | 6033 | 1609258027786446 | 33961 | NULL |

|db4 | 6033 | 1609258037791764 | 33709 | NULL |

|db4 | 6033 | 1609258047702139 | 33685 | NULL |

|db4 | 6033 | 1609258057552203 | 33712 | NULL |

|db4 | 6033 | 1609258067680075 | 33712 | NULL |

|db4 | 6033 | 1609258077780261 | 0 | Lost connection to MySQL server during query |

|db4 | 6033 | 1609258087753865 | 33801 | NULL |

|db4 | 6033 | 1609258097552738 | 33995 | NULL |

|db4 | 6033 | 1609258107552894 | 35086 | NULL |

|db4 | 6033 | 1609258117756236 | 33917 | NULL |

|db4 | 6033 | 1609258127648349 | 33870 | NULL |

|db4 | 6033 | 1609258137703918 | 33845 | NULL |

|db4 | 6033 | 1609258147553461 | 33946 | NULL |

|db4 | 6033 | 1609258157675198 | 33918 | NULL |

|db4 | 6033 | 1609258167816745 | 33908 | NULL |

|db4 | 6033 | 1609258177747507 | 35135 | NULL |

|db4 | 6033 | 1609258187675816 | 33922 | NULL |

|db4 | 6033 | 1609258197554081 | 0 | Lost connection to MySQL server during query |

|db4 | 6033 | 1609258207731842 | 33716 | NULL |

|db4 | 6033 | 1609258217554423 | 33873 | NULL |

|db4 | 6033 | 1609258227554546 | 33782 | NULL |

|db4 | 6033 | 1609258237714476 | 33960 | NULL |

|db4 | 6033 | 1609258247554870 | 33932 | NULL |

|db4 | 6033 | 1609258257803847 | 33792 | NULL |

|db4 | 6033 | 1609258267727837 | 34043 | NULL |

|db4 | 6033 | 1609258277681727 | 33969 | NULL |

|db4 | 6033 | 1609258287686055 | 272153 | NULL |

|db4 | 6033 | 1609258297555507 | 33820 | NULL |

|db4 | 6033 | 1609258307653576 | 33897 | NULL |

|db4 | 6033 | 1609258317714103 | 0 | Lost connection to MySQL server during query |

|db4 | 6033 | 1609258327555883 | 34047 | NULL |