Hi folks... I was just wondering if you had any thoughts on the page
loading times that I'm experiencing this past few weeks on CommCareHQ.
We have a strong internet connection here in Bangkok, and most sites load
as fast as one tends to get in a good normal office connection.
But as I administer and build on CommCare, there are some times of a
typical day when the loading time for the form editors and other CommCareHQ
features are really slow.
-- On occasion, it has gotten *prohibitively *bad (May 25 around
noon Bangkok time, and May 29 around 9am Bangkok time) ... where the site
would basically freeze for 10 minutes... so I just had to go do other work
and then CommCare would come back.
-- But other times, like today, it's just quite slow, though I can still
work (with decreased efficiency ) Pages take quite a while to start
loading in the browser. (again -- other sites on the web are loading just
fine for us.) It might event take 10-15 seconds for an AJAX-like button
(for example the Save button in a Case Management configuration tab.)
Any thoughts on this? Is there a chance your servers are doing maintenance
or something in the USA night times, and so the site is slower for me
during those times? Or any other thoughts? I usually try to avoid whining
on items at that level, but the experiences accumulate and I figured I
should give it a mention.
Just to provide more context, this current moment -- 5-5:30pm on June 2
bangkok time -- is very slow... 10-20 seconds to perform a load or save,
sometimes.
Over the past week or so, I've been having the same "quite unresponsive
system" issues that I had a few months ago in the thread below. It is
interesting, because there is never a problem in the morning hours of
Bangkok time, but then starting around 2pm onward, very regularly,
CommCareHQ becomes ultra slow. And then, today, it's become worse in that
syncing phones is also taking many minutes, and half of the time, the
phones create red-text messages about "didn't receive a response from the
server in time" or similar. (of course I have no idea if those two issues
are linked or not.)
Is there any information about this slowdown? Also, more generally, as we
chatted below, is there any news about progress on the capacity of the
system that might alleviate these daily slowdowns that happen after 2pm BKK
time?
Thanks for raising this. We're quite aware of ongoing performance problems
with HQ and are actively working on a number of fixes to address them.
There were a few operations that were being taken on the servers that were
causing the database to be busy for long periods of time and resulted in
site wide slowness/outages. We've addressed a number of those in the last
two weeks and expect the site to be generally more reliable now than the
last two weeks. There are also a set of other fixes we are taking that are
in various stages of testing/development and we will continue to roll those
out as soon as they are ready which should continue to improve reliability
and speed over the coming days.
Longer term we are also working on a number of small and large changes to
make the site faster and more reliable. We have 3-4 developers working on
this right now and are hoping to see steady improvements. As some of the
changes are quite large it is going to take some time before we are able to
fully test and release everything.
We do get notified when the site is working slowly (as it is now) and if
there are any steps that can be taken to reduce load, our operations team
takes them.
We appreciate your patience as we work to deal with these issues.
thanks,
Cory
···
On Tue, Jun 2, 2015 at 12:25 PM, Eric Stephan wrote:
Just to provide more context, this current moment -- 5-5:30pm on June 2
bangkok time -- is very slow... 10-20 seconds to perform a load or save,
sometimes.
Thanks for the note. We're aware of the issues (there is a new crop that
have started arising recently) and working with our database provider to
address them. I understand this is very frustrating to you and everyone
else who relies on CommCare HQ on a daily basis, and appreciate your
patience as we continue to try to address things as fast as we can.
I can't quote any timelines at the moment, but I can say that we are
continuing to treat performance at scale as our top priority.
thanks,
Cory
···
On Mon, Sep 7, 2015 at 5:02 AM, Eric Stephan wrote:
Hello CommCare folks --
Over the past week or so, I've been having the same "quite unresponsive
system" issues that I had a few months ago in the thread below. It is
interesting, because there is never a problem in the morning hours of
Bangkok time, but then starting around 2pm onward, very regularly,
CommCareHQ becomes ultra slow. And then, today, it's become worse in that
syncing phones is also taking many minutes, and half of the time, the
phones create red-text messages about "didn't receive a response from the
server in time" or similar. (of course I have no idea if those two issues
are linked or not.)
Is there any information about this slowdown? Also, more generally, as we
chatted below, is there any news about progress on the capacity of the
system that might alleviate these daily slowdowns that happen after 2pm BKK
time?
By the way, yesterday I was looking at the status page of my email
provider, and I saw they were using a site status app called Town Crier. In
case you guys were looking into that. (you probably already know it well.)
You can see it in use: http://fastmailstatus.com/ ... I know that your
considerations of using such a thing come down to more questions than
simply the technical infrastructure to do it.
Thanks for the pointer Eric! We're definitely keen to expose more
visibility into the site operations to our users and this looks like an
interesting option.
···
On Wed, Jun 3, 2015 at 4:30 AM, Eric Stephan wrote:
Great info, appreciate it much.
By the way, yesterday I was looking at the status page of my email
provider, and I saw they were using a site status app called Town Crier. In
case you guys were looking into that. (you probably already know it well.)
You can see it in use: http://fastmailstatus.com/ ... I know that
your considerations of using such a thing come down to more questions than
simply the technical infrastructure to do it.
) Pages take quite a while to start