ws_prod tables are not being cleaned
Our ws_prod tables are not being cleaned to the point that REST
queries seem slow.
e.g.:
mysql> use ws_prod
mysql> select count(*) from resource;
+----------+
| count(*) |
+----------+
| 6164157 |
+----------+
1 row in set (0.00 sec)
mysql> select count(*) from request;
+----------+
| count(*) |
+----------+
| 21043313 |
+----------+
1 row in set (0.00 sec)
The /home/saffron/smb/logs/ws.log file has the following entries:
2010-03-05 03:00:48,879 [quartzScheduler_Worker-3] ERROR core.JobRunShell - Job GRAILS_JOBS.com.saffrontech.ws.job.ResourceCleanupJob threw an unhandled Exception:
java.lang.OutOfMemoryError: Java heap space
at com.mysql.jdbc.Buffer.<init>(Buffer.java:60)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1418)
at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2708)
at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:459)
at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2411)
at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1657)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2036)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2543)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1737)
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1888)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:92)
at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208)
at org.hibernate.loader.Loader.getResultSet(Loader.java:1808)
at org.hibernate.loader.Loader.doQuery(Loader.java:697)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
at org.hibernate.loader.Loader.doList(Loader.java:2228)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2125)
at org.hibernate.loader.Loader.list(Loader.java:2120)
at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:118)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1596)
at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:306)
at org.codehaus.groovy.grails.orm.hibernate.metaclass.FindAllByPersistentMethod$1.doInHibernate(FindAllByPersistentMethod.java:75)
at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:419)
at org.springframework.orm.hibernate3.HibernateTemplate.executeFind(HibernateTemplate.java:343)
at org.codehaus.groovy.grails.orm.hibernate.metaclass.FindAllByPersistentMethod.doInvokeInternalWithExpressions(FindAllByPersistentMethod.java:49)
at org.codehaus.groovy.grails.orm.hibernate.metaclass.AbstractClausedStaticPersistentMethod.doInvokeInternal(AbstractClausedStaticPersistentMethod.java:489)
at org.codehaus.groovy.grails.orm.hibernate.metaclass.AbstractStaticPersistentMethod.invoke(AbstractStaticPersistentMethod.java:56)
at sun.reflect.GeneratedMethodAccessor223.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:229)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:52)
2010-03-05 03:00:48,880 [quartzScheduler_Worker-3] ERROR core.ErrorLogger - Job (GRAILS_JOBS.com.saffrontech.ws.job.ResourceCleanupJob threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.lang.OutOfMemoryError: Java heap space]
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525)
Caused by: java.lang.OutOfMemoryError: Java heap space
2010-03-05 03:00:48,880 [quartzScheduler_Worker-3] ERROR core.ErrorLogger - Job (GRAILS_JOBS.com.saffrontech.ws.job.ResourceCleanupJob threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.lang.OutOfMemoryError: Java heap space]
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525)
Caused by: java.lang.OutOfMemoryError: Java heap space
Thanks
Harry
Support Staff 2 Posted by Jared Peterson on 09 Mar, 2010 07:38 PM
Harry,
We'll get this fixed up, but for now you can delete these resources/jobs using the MySQL command-line. First, make sure that there are no new jobs arriving that are being processed. Then execute the following commands:
delete from resource where persistent='false';
delete from resource_job where status='passed';
I'll created a bug in lighthouse and attach it here.
Thanks,
Jared