I currently have an application that runs a daily process with heavy read-write loads onto the database.
The application was developed with a DigitalOcean Managed MySQL instance (2 vCPUs and 4GB RAM). Over time, the cost of resizing the managed MySQL instance became prohibitive, so I picked up a droplet with 8 vCPUs and 16GB RAM to host my own instance of MySQL. Even with over 2x the CPU and RAM, my droplet MySQL instance shows much slower performance.
This is evidenced by the old managed MySQL being able to ingest records at 1400 records per minute, while the current MySQL only takes in 800 items per minute. For reference, these are python scrapy items that I insert into the database, using SQLAlchemy as an ORM.
I followed recommendations from MySQL_tuner_perl and online sources and have set the following values in my mysqld.cnf
# Updated Configs
innodb_buffer_pool_size = 10G
innodb_buffer_pool_chunk_size = 256
innodb_buffer_pool_instances = 10
innodb_log_file_size = 1G
innodb_log_buffer_size = 512M
innodb_flush_method = O_DIRECT
innodb_io_capacity = 1000
innodb_io_capacity_max = 2000
innodb_thread_concurrency = 0
innodb_read_io_threads = 32
innodb_write_io_threads = 32
max_connections = 100
table_open_cache = 1000
thread_cache_size = 12
skip_name_resolve = ON
binlog_cache_size = 16M
Are there any recommended troubleshooting tips to find the bottleneck? With my new DB having superior hardware specs, I believe it’s only normal to expect better performance. As far as MySQL configs go, I don’t see any big red flags, but I could be wrong. Open to any help or suggestions on how to speed up my DB operations!
Thanks in advance :)
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.
Sign up for Infrastructure as a Newsletter.
Working on improving health and education, reducing inequality, and spurring economic growth? We'd like to help.
Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.
Hey!
What I could suggest here is to check your resources utilization, you can use
htop
,top
, andiostat
to check CPU and I/O utilization.This will give you enough information if there are any other processes consuming too much resources not leaving enough for the MySQL service to perform better.
Another option here is to try and use another MySQL optimization tool like Releem for example:
Also, you could try enabling the slow query log to identify and optimize slow-running queries:
Let me know how it goes!
- Bobby