[resolved] CommCare HQ Slowness

Hi all,

At this time we have finished all of our backend database upgrades and have
observed response times that are the same or better than they were before
the performance issues across the board. We will be continuing to look into
performance issues and gains that can be made over the course of the next
few weeks and months, but at this time we believe the site has been
restored to stability.

Thanks for being patient,

··· On Thu, Aug 22, 2013 at 10:13 PM, Cory Zue wrote:

Hi everyone,

Just a quick update from our side on this. Our database provider is still
working to upgrade our machines and restore performance to CommCare HQ. We
expect the site will continue to be significantly slower than normal for
the next 12 hours while this is happening. We should have another update at
that time.


On Tue, Aug 20, 2013 at 7:38 PM, Cory Zue czue@dimagi.com wrote:

Hi all,

Another follow up - we are still gathering information and working on
this. We have taken some temporary precautions to make things more load
tolerant and CommCare HQ appears to be functioning normally (perhaps a bit
slower than usual) although we are not 100% sure that it will continue to
be performant during peak load hours (which typically occur from 2am - 10am

Many of the fixes that need to be made are quite complicated, so we will
be continuing to be monitoring this closely and incrementally making
changes to restore performance to the site over the course of the next few
days (possibly weeks).

In the meantime, you can use CommCare HQ normally and hopefully the
changes we have made so far will hold up.

thanks for being patient,

On Mon, Aug 19, 2013 at 7:01 PM, Cory Zue czue@dimagi.com wrote:

Hi all,

Just to follow up on the original thread, we are still working with the
database provider to resolve these issues. We anticipate that CommCare HQ
will continue to be much slower than usual for the next 12-24 hours until
we are both able to do a more thorough audit of the problem and begin
working towards a resolution. In the meantime there is no need to report
additional slowness via this mailing list or our issue reporting systems.
We will provide an additional update as soon as we know more information.

In the meantime unless it is absolutely critical to your workflow we
kindly ask that you refrain from doing heavy, data intensive operations.
These include all form, case or metadata exports, viewing reports
over large time ranges
, bulk case reassignment, deleting users, bulk
imports of users, cases, or fixtures. This will allow the system to be more
responsive to the most crucial workflows like front line worker syncs and

We are doing everything in our power to restore the normal functionality
of the site, and please bear with us and be patient through that process.

HQ team

On Mon, Aug 19, 2013 at 8:53 AM, Cory Zue czue@dimagi.com wrote:

Hi all,

We are aware of extreme slowness on CommCare HQ. Our database server is
experiencing very heavy load and we are working urgently with our provider
to understand and address the issue. Please bear with us as we work to
resolve this as fast as we can.

the CommCare HQ team