ws_prod tables are not being cleaned

Harry Stovall's Avatar

Harry Stovall

09 Mar, 2010 07:24 PM via web

Our ws_prod tables are not being cleaned to the point that REST queries seem slow.

e.g.:

 mysql> use ws_prod
 mysql> select count(*) from resource;
 +----------+
 | count(*) |
 +----------+
 |  6164157 |
 +----------+
 1 row in set (0.00 sec)

 mysql> select count(*) from request;
 +----------+
 | count(*) |
 +----------+
 | 21043313 |
 +----------+
 1 row in set (0.00 sec)

The /home/saffron/smb/logs/ws.log file has the following entries:

 2010-03-05 03:00:48,879 [quartzScheduler_Worker-3] ERROR core.JobRunShell  - Job      GRAILS_JOBS.com.saffrontech.ws.job.ResourceCleanupJob threw an unhandled Exception:
 java.lang.OutOfMemoryError: Java heap space
         at com.mysql.jdbc.Buffer.<init>(Buffer.java:60)
         at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1418)
         at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2708)
         at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:459)
         at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2411)
         at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1657)
         at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2036)
         at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2543)
         at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1737)
         at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1888)
         at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:92)
         at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208)
         at org.hibernate.loader.Loader.getResultSet(Loader.java:1808)
         at org.hibernate.loader.Loader.doQuery(Loader.java:697)
         at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
         at org.hibernate.loader.Loader.doList(Loader.java:2228)
         at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2125)
         at org.hibernate.loader.Loader.list(Loader.java:2120)
         at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:118)
         at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1596)
         at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:306)
         at org.codehaus.groovy.grails.orm.hibernate.metaclass.FindAllByPersistentMethod$1.doInHibernate(FindAllByPersistentMethod.java:75)
         at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:419)
         at org.springframework.orm.hibernate3.HibernateTemplate.executeFind(HibernateTemplate.java:343)
         at org.codehaus.groovy.grails.orm.hibernate.metaclass.FindAllByPersistentMethod.doInvokeInternalWithExpressions(FindAllByPersistentMethod.java:49)
         at org.codehaus.groovy.grails.orm.hibernate.metaclass.AbstractClausedStaticPersistentMethod.doInvokeInternal(AbstractClausedStaticPersistentMethod.java:489)
         at org.codehaus.groovy.grails.orm.hibernate.metaclass.AbstractStaticPersistentMethod.invoke(AbstractStaticPersistentMethod.java:56)
         at sun.reflect.GeneratedMethodAccessor223.invoke(Unknown Source)
         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
         at java.lang.reflect.Method.invoke(Method.java:597)
         at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:229)
         at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:52)
 2010-03-05 03:00:48,880 [quartzScheduler_Worker-3] ERROR core.ErrorLogger  - Job (GRAILS_JOBS.com.saffrontech.ws.job.ResourceCleanupJob threw an exception.
 org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.lang.OutOfMemoryError: Java heap space]
         at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
         at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525)
 Caused by: java.lang.OutOfMemoryError: Java heap space

 2010-03-05 03:00:48,880 [quartzScheduler_Worker-3] ERROR core.ErrorLogger  - Job (GRAILS_JOBS.com.saffrontech.ws.job.ResourceCleanupJob threw an exception.
 org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.lang.OutOfMemoryError: Java heap space]
         at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
         at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525)
 Caused by: java.lang.OutOfMemoryError: Java heap space

Thanks
Harry

  1. Support Staff 2 Posted by Jared Peterson on 09 Mar, 2010 07:38 PM

    Jared Peterson's Avatar

    Harry,

    We'll get this fixed up, but for now you can delete these resources/jobs using the MySQL command-line. First, make sure that there are no new jobs arriving that are being processed. Then execute the following commands:

    delete from resource where persistent='false';

    delete from resource_job where status='passed';

    I'll created a bug in lighthouse and attach it here.

    Thanks,
    Jared

Reply to this discussion

Internal reply

Formatting help or Preview

Attached Files

You can attach files up to 10MB

What month comes before July?

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.