Comments on: MySQL JDBC Memory Usage on Large ResultSet http://benjchristensen.com/2008/05/27/mysql-jdbc-memory-usage-on-large-resultset/ Software Development and Other Random Stuff Mon, 06 Sep 2010 12:52:33 +0000 hourly 1 http://wordpress.com/ By: Große (riesige!) ResultSets und MySQL « msaure.com http://benjchristensen.com/2008/05/27/mysql-jdbc-memory-usage-on-large-resultset/#comment-396 Große (riesige!) ResultSets und MySQL « msaure.com Tue, 23 Feb 2010 15:59:38 +0000 http://benjchristensen.wordpress.com/?p=59#comment-396 [...] http://benjchristensen.com/2008/05/27/mysql-jdbc-memory-usage-on-large-resultset/ wird eine einfache Lösung angeboten, die im Wesentlichen zu folgender Codezeile [...] [...] http://benjchristensen.com/2008/05/27/mysql-jdbc-memory-usage-on-large-resultset/ wird eine einfache Lösung angeboten, die im Wesentlichen zu folgender Codezeile [...]

]]> By: Abhishek Anand http://benjchristensen.com/2008/05/27/mysql-jdbc-memory-usage-on-large-resultset/#comment-388 Abhishek Anand Sat, 16 Jan 2010 15:29:39 +0000 http://benjchristensen.wordpress.com/?p=59#comment-388 The deadlock problem I described above was solved when I restarted my computer which also runs the mysql server The deadlock problem I described above was solved when I restarted my computer which also runs the mysql server

]]> By: Abhishek Anand http://benjchristensen.com/2008/05/27/mysql-jdbc-memory-usage-on-large-resultset/#comment-387 Abhishek Anand Sat, 16 Jan 2010 04:29:41 +0000 http://benjchristensen.wordpress.com/?p=59#comment-387 I need to read from one huge table, do some processing and insert into another small table. I opened 2 connections, one for reading and another for updating. My program starts correctly but very soon encounters a deadlock. I need to read from one huge table, do some processing and insert into another small table.
I opened 2 connections, one for reading and another for updating.
My program starts correctly but very soon encounters a deadlock.

]]> By: Abhishek Anand http://benjchristensen.com/2008/05/27/mysql-jdbc-memory-usage-on-large-resultset/#comment-386 Abhishek Anand Fri, 15 Jan 2010 16:38:18 +0000 http://benjchristensen.wordpress.com/?p=59#comment-386 Thanks a lot for this post. Thanks a lot for this post.

]]> By: Ben Christensen http://benjchristensen.com/2008/05/27/mysql-jdbc-memory-usage-on-large-resultset/#comment-380 Ben Christensen Wed, 18 Nov 2009 17:52:13 +0000 http://benjchristensen.wordpress.com/?p=59#comment-380 It could be many things, row locking, new indexes, thread-thrashing in MySQL, CPU or IO constraints. Anytime you're doing something in parallell on a shared resource any of these or other issues can cause performance issues. If you do the query without the parallel inserts does it perform in the 10 minute range? If I was in your position I'd start profiling and testing the different individual components and then combining them to see the impact of parallel execution to isolate what the issue really is. It could be many things, row locking, new indexes, thread-thrashing in MySQL, CPU or IO constraints.

Anytime you’re doing something in parallell on a shared resource any of these or other issues can cause performance issues.

If you do the query without the parallel inserts does it perform in the 10 minute range?

If I was in your position I’d start profiling and testing the different individual components and then combining them to see the impact of parallel execution to isolate what the issue really is.

]]> By: Ben Christensen http://benjchristensen.com/2008/05/27/mysql-jdbc-memory-usage-on-large-resultset/#comment-379 Ben Christensen Wed, 18 Nov 2009 17:50:00 +0000 http://benjchristensen.wordpress.com/?p=59#comment-379 I don't know anything that would prevent you from doing this as long as each resultset has a separate connection. Are you experiencing table locks when doing so? I don’t know anything that would prevent you from doing this as long as each resultset has a separate connection. Are you experiencing table locks when doing so?

]]> By: Amit http://benjchristensen.com/2008/05/27/mysql-jdbc-memory-usage-on-large-resultset/#comment-378 Amit Fri, 13 Nov 2009 20:11:33 +0000 http://benjchristensen.wordpress.com/?p=59#comment-378 Great article, Thanks! So is there no away to allow concurrent access to the tables while streaming the result set? Why this limitation if i dont care about dirty reads? Great article, Thanks!
So is there no away to allow concurrent access to the tables while streaming the result set? Why this limitation if i dont care about dirty reads?

]]> By: Marc http://benjchristensen.com/2008/05/27/mysql-jdbc-memory-usage-on-large-resultset/#comment-377 Marc Thu, 05 Nov 2009 11:47:16 +0000 http://benjchristensen.wordpress.com/?p=59#comment-377 I have been using fetchSize(Integer.MIN_VALUE) for a long time and it was working perfect until now. I use MySQL, java 1.6, mysql-connector-java-5.1.7-bin.jar and InoDB tables. Since a month ago when the query is executed it will take a long time untill it starts processing the results from the resultSet. The query matches about 2M rows. It use to take 10 min untill rows processing started. Now it's taking about 2 hours. Parallelly I have another process wich is doing lots of inserts to that table (I also had it before but with less number of inserts). Could this be causing some bloking that makes the query take that long? Thanks in advance. I have been using fetchSize(Integer.MIN_VALUE) for a long time and it was working perfect until now. I use MySQL, java 1.6, mysql-connector-java-5.1.7-bin.jar and InoDB tables.
Since a month ago when the query is executed it will take a long time untill it starts processing the results from the resultSet. The query matches about 2M rows. It use to take 10 min untill rows processing started. Now it’s taking about 2 hours.
Parallelly I have another process wich is doing lots of inserts to that table (I also had it before but with less number of inserts). Could this be causing some bloking that makes the query take that long?
Thanks in advance.

]]> By: Emm http://benjchristensen.com/2008/05/27/mysql-jdbc-memory-usage-on-large-resultset/#comment-374 Emm Tue, 06 Oct 2009 21:42:41 +0000 http://benjchristensen.wordpress.com/?p=59#comment-374 Nice suggestion. Nice suggestion.

]]> By: Myšlenky dne otce Fura » Blog Archive » Import velkých dat do MySQL http://benjchristensen.com/2008/05/27/mysql-jdbc-memory-usage-on-large-resultset/#comment-372 Myšlenky dne otce Fura » Blog Archive » Import velkých dat do MySQL Tue, 06 Oct 2009 12:14:58 +0000 http://benjchristensen.wordpress.com/?p=59#comment-372 [...] Zdrojový článek, který mě navedl na řešení OOM v případě rozsáhlých ResultSetů [...] [...] Zdrojový článek, který mě navedl na řešení OOM v případě rozsáhlých ResultSetů [...]

]]>