You may have noticed some continued slowness and out of date reports today
on CommCareHQ. I would like to reassure you that there is still no risk of
data loss for submitted data.
There is also some good news I can share. We have identified the cause of
some of the problems, and have set changes in motion to make HQ faster and
keep web reports up to date.
We will continue to update you on our progress. Please accept our apologies
for the inconvenience and thanks again for your patience!
Sincerely,
···
--
Nate Haduch
Technical Support Analyst
Dimagi, Inc | 585 Massachusetts Ave | Suite 3 | Cambridge, MA 02139
web: http://www.dimagi.com/
Just wanted to mention that we are still working with our database provider
to resolve these issues. HQ should be fully stable, all data is being saved
fine, and all mobile interactions should work smoothly. However, many parts
of the site will not show the most recent data, including the submit
history and case list reports.
For a definitive source of data, we recommend using the "Raw Forms, Errors,
and Duplicates" report with the "normal submissions" option checked, or
data exports.
Apologies again for the continued inconvenience. We will update out as soon
as anything changes.
best,
Cory
···
On Tue, Feb 10, 2015 at 10:33 PM, Nate Haduch wrote:
Hi CommCare Users,
You may have noticed some continued slowness and out of date reports today
on CommCareHQ. I would like to reassure you that there is still no risk of
data loss for submitted data.
There is also some good news I can share. We have identified the cause of
some of the problems, and have set changes in motion to make HQ faster and
keep web reports up to date.
We will continue to update you on our progress. Please accept our
apologies for the inconvenience and thanks again for your patience!
Sincerely,
--
Nate Haduch
Technical Support Analyst
Dimagi, Inc | 585 Massachusetts Ave | Suite 3 | Cambridge, MA 02139
web: http://www.dimagi.com/
···
On Wednesday, 11 February 2015 05:34:16 UTC+2, nhaduch wrote:
>
> Hi CommCare Users,
>
> You may have noticed some continued slowness and out of date reports today
> on CommCareHQ. I would like to reassure you that there is still no risk of
> data loss for submitted data.
>
> There is also some good news I can share. We have identified the cause of
> some of the problems, and have set changes in motion to make HQ faster and
> keep web reports up to date.
>
> We will continue to update you on our progress. Please accept our
> apologies for the inconvenience and thanks again for your patience!
> Sincerely,
>
> --
> Nate Haduch
>
> Technical Support Analyst
> Dimagi, Inc | 585 Massachusetts Ave | Suite 3 | Cambridge, MA 02139
> web: http://www.dimagi.com/
>
>
Just a quick update on this issue. Everything is progressing smoothly but
the rebuild operation is taking some time. Many reports are back to normal,
including the case list report, however submit history and others are still
not showing all the latest data.
Based on the current progress rate and barring any unforeseen issues, we
expect things to be 100% restored at some point early in the week. In the
meantime, please continue to use the workarounds below to get your data
from HQ.
Appreciate your ongoing patience as we work to resolve this as fast as we
can.
Cory
···
On Wed, Feb 11, 2015 at 10:19 PM, Cory Zue wrote:
Hey all,
Just wanted to mention that we are still working with our database
provider to resolve these issues. HQ should be fully stable, all data is
being saved fine, and all mobile interactions should work smoothly.
However, many parts of the site will not show the most recent data,
including the submit history and case list reports.
For a definitive source of data, we recommend using the "Raw Forms,
Errors, and Duplicates" report with the "normal submissions" option
checked, or data exports.
Apologies again for the continued inconvenience. We will update out as
soon as anything changes.
You may have noticed some continued slowness and out of date reports
today on CommCareHQ. I would like to reassure you that there is still no
risk of data loss for submitted data.
There is also some good news I can share. We have identified the cause of
some of the problems, and have set changes in motion to make HQ faster and
keep web reports up to date.
We will continue to update you on our progress. Please accept our
apologies for the inconvenience and thanks again for your patience!
Sincerely,
--
Nate Haduch
Technical Support Analyst
Dimagi, Inc | 585 Massachusetts Ave | Suite 3 | Cambridge, MA 02139
web: http://www.dimagi.com/
Thank you for your patience as we resolve the remaining CommCareHQ reports
delays. We have another update for you with some good news.
The following reports that were recently experiencing delays are now
up-to-date and displaying data in real-time:
The Submit History report (where you can view all form submissions)
All worker monitoring reports except the Case Activity report
The following reports are still experiencing delays and may not be
displaying data in real-time:
The Case Activity report
Mobile reports that rely on the advanced "Call Center" functionality
All other reports and features on CommCareHQ have been functioning
normally.
We have pinpointed the issues causing the performance problems, and taken
measure to address them. We are confident we have addressed the problem, so
we anticipate any functionality that is now working and up-to-date will
reliably remain.
Our apologies to those still affected; at this point, we are waiting for
the measures we put in place to take effect on the Case Activity and Call
Center reports. We will update you when the remaining out-of-date reports
listed above are also back to normal.
Sincerely,
Amelia
···
On Sun, Feb 15, 2015 at 11:55 AM, Cory Zue wrote:
Hi all,
Just a quick update on this issue. Everything is progressing smoothly but
the rebuild operation is taking some time. Many reports are back to normal,
including the case list report, however submit history and others are still
not showing all the latest data.
Based on the current progress rate and barring any unforeseen issues, we
expect things to be 100% restored at some point early in the week. In the
meantime, please continue to use the workarounds below to get your data
from HQ.
Appreciate your ongoing patience as we work to resolve this as fast as we
can.
Cory
On Wed, Feb 11, 2015 at 10:19 PM, Cory Zue czue@dimagi.com wrote:
Hey all,
Just wanted to mention that we are still working with our database
provider to resolve these issues. HQ should be fully stable, all data is
being saved fine, and all mobile interactions should work smoothly.
However, many parts of the site will not show the most recent data,
including the submit history and case list reports.
For a definitive source of data, we recommend using the "Raw Forms,
Errors, and Duplicates" report with the "normal submissions" option
checked, or data exports.
Apologies again for the continued inconvenience. We will update out as
soon as anything changes.
You may have noticed some continued slowness and out of date reports
today on CommCareHQ. I would like to reassure you that there is still no
risk of data loss for submitted data.
There is also some good news I can share. We have identified the cause
of some of the problems, and have set changes in motion to make HQ faster
and keep web reports up to date.
We will continue to update you on our progress. Please accept our
apologies for the inconvenience and thanks again for your patience!
Sincerely,
--
Nate Haduch
Technical Support Analyst
Dimagi, Inc | 585 Massachusetts Ave | Suite 3 | Cambridge, MA 02139
web: http://www.dimagi.com/
We're happy to report that the remaining reports are now up to date (the
Case Activy report and Call Center mobile reports).
We are investigating some reported issues related to the case and submit
history to determine if they are linked to the same underlying issue. We
will communicate directly with those who have reported issues once the root
cause is determined.
We believe all site-wide performance issues related to the iniatial outtage
are resolved and will confirm over the coming days once our team has
completed verification.
At this point, please report any issues you are having through the normal
"report an issue" channel.
Thanks again for bearing with us as we take the final steps to resolve HQ's
performance issues.
Sincerely,
Amelia
···
On Tue, Feb 17, 2015 at 5:31 PM, Amelia Sagoff wrote:
Dear Users,
Thank you for your patience as we resolve the remaining CommCareHQ reports
delays. We have another update for you with some good news.
The following reports that were recently experiencing delays are now
up-to-date and displaying data in real-time:
The Submit History report (where you can view all form submissions)
All worker monitoring reports except the Case Activity report
The following reports are still experiencing delays and may not be
displaying data in real-time:
The Case Activity report
Mobile reports that rely on the advanced "Call Center" functionality
All other reports and features on CommCareHQ have been functioning
normally.
We have pinpointed the issues causing the performance problems, and taken
measure to address them. We are confident we have addressed the problem, so
we anticipate any functionality that is now working and up-to-date will
reliably remain.
Our apologies to those still affected; at this point, we are waiting for
the measures we put in place to take effect on the Case Activity and Call
Center reports. We will update you when the remaining out-of-date reports
listed above are also back to normal.
Sincerely,
Amelia
On Sun, Feb 15, 2015 at 11:55 AM, Cory Zue czue@dimagi.com wrote:
Hi all,
Just a quick update on this issue. Everything is progressing smoothly but
the rebuild operation is taking some time. Many reports are back to normal,
including the case list report, however submit history and others are still
not showing all the latest data.
Based on the current progress rate and barring any unforeseen issues, we
expect things to be 100% restored at some point early in the week. In the
meantime, please continue to use the workarounds below to get your data
from HQ.
Appreciate your ongoing patience as we work to resolve this as fast as we
can.
Cory
On Wed, Feb 11, 2015 at 10:19 PM, Cory Zue czue@dimagi.com wrote:
Hey all,
Just wanted to mention that we are still working with our database
provider to resolve these issues. HQ should be fully stable, all data is
being saved fine, and all mobile interactions should work smoothly.
However, many parts of the site will not show the most recent data,
including the submit history and case list reports.
For a definitive source of data, we recommend using the "Raw Forms,
Errors, and Duplicates" report with the "normal submissions" option
checked, or data exports.
Apologies again for the continued inconvenience. We will update out as
soon as anything changes.
You may have noticed some continued slowness and out of date reports
today on CommCareHQ. I would like to reassure you that there is still no
risk of data loss for submitted data.
There is also some good news I can share. We have identified the cause
of some of the problems, and have set changes in motion to make HQ faster
and keep web reports up to date.
We will continue to update you on our progress. Please accept our
apologies for the inconvenience and thanks again for your patience!
Sincerely,
--
Nate Haduch
Technical Support Analyst
Dimagi, Inc | 585 Massachusetts Ave | Suite 3 | Cambridge, MA 02139
web: http://www.dimagi.com/
Great to hear that we're in the "home stretch" in terms of getting past
this phase.
But, just to provide more input -- for the past hour or so, here in
Bangkok, I've been trying to log in for some regular use of CommCareHQ, and
I basically can not get in. I'm getting long, long loading time and have
not gotten further into the system than what I believe is a half-loaded
version of the big dashboard-like page with the six large blue items
navigating to the different project elements. Sometimes not even that far.
Did you have a bit of site slowdown around noon on 26 Feb, Bangkok time?
Also, I wonder if it would be tricky to consider one of those site-status
pages, like status.commcarehq.com, which could help us troubleshoot,
sometimes possibly indicating that problems lie elsewhere than your
servers...
We have completed final verification that the performance issues related to
the initial outage are all resolved. All reports should be up-to-date and
displaying data in real-time as usual.
Thanks again for your patience.
Sincerely,
Amelia
···
On Mon, Feb 23, 2015 at 11:42 AM, Amelia Sagoff wrote:
Dear Users,
We're happy to report that the remaining reports are now up to date (the
Case Activy report and Call Center mobile reports).
We are investigating some reported issues related to the case and submit
history to determine if they are linked to the same underlying issue. We
will communicate directly with those who have reported issues once the root
cause is determined.
We believe all site-wide performance issues related to the iniatial
outtage are resolved and will confirm over the coming days once our team
has completed verification.
At this point, please report any issues you are having through the normal
"report an issue" channel.
Thanks again for bearing with us as we take the final steps to resolve
HQ's performance issues.
Sincerely,
Amelia
On Tue, Feb 17, 2015 at 5:31 PM, Amelia Sagoff asagoff@dimagi.com wrote:
Dear Users,
Thank you for your patience as we resolve the remaining CommCareHQ
reports delays. We have another update for you with some good news.
The following reports that were recently experiencing delays are now
up-to-date and displaying data in real-time:
The Submit History report (where you can view all form submissions)
All worker monitoring reports except the Case Activity report
The following reports are still experiencing delays and may not be
displaying data in real-time:
The Case Activity report
Mobile reports that rely on the advanced "Call Center"
functionality
All other reports and features on CommCareHQ have been functioning
normally.
We have pinpointed the issues causing the performance problems, and taken
measure to address them. We are confident we have addressed the problem, so
we anticipate any functionality that is now working and up-to-date will
reliably remain.
Our apologies to those still affected; at this point, we are waiting for
the measures we put in place to take effect on the Case Activity and Call
Center reports. We will update you when the remaining out-of-date reports
listed above are also back to normal.
Sincerely,
Amelia
On Sun, Feb 15, 2015 at 11:55 AM, Cory Zue czue@dimagi.com wrote:
Hi all,
Just a quick update on this issue. Everything is progressing smoothly
but the rebuild operation is taking some time. Many reports are back to
normal, including the case list report, however submit history and others
are still not showing all the latest data.
Based on the current progress rate and barring any unforeseen issues, we
expect things to be 100% restored at some point early in the week. In the
meantime, please continue to use the workarounds below to get your data
from HQ.
Appreciate your ongoing patience as we work to resolve this as fast as
we can.
Cory
On Wed, Feb 11, 2015 at 10:19 PM, Cory Zue czue@dimagi.com wrote:
Hey all,
Just wanted to mention that we are still working with our database
provider to resolve these issues. HQ should be fully stable, all data is
being saved fine, and all mobile interactions should work smoothly.
However, many parts of the site will not show the most recent data,
including the submit history and case list reports.
For a definitive source of data, we recommend using the "Raw Forms,
Errors, and Duplicates" report with the "normal submissions" option
checked, or data exports.
Apologies again for the continued inconvenience. We will update out as
soon as anything changes.
You may have noticed some continued slowness and out of date reports
today on CommCareHQ. I would like to reassure you that there is still no
risk of data loss for submitted data.
There is also some good news I can share. We have identified the cause
of some of the problems, and have set changes in motion to make HQ faster
and keep web reports up to date.
We will continue to update you on our progress. Please accept our
apologies for the inconvenience and thanks again for your patience!
Sincerely,
--
Nate Haduch
Technical Support Analyst
Dimagi, Inc | 585 Massachusetts Ave | Suite 3 | Cambridge, MA 02139
web: http://www.dimagi.com/
(P.S. As a follow-up, now, at about 3:45, I logged in and the entire site
sprang to much faster response. So whatever the issue was is OK at the
moment. But it would be interesting to know if you guys see that you had a
slowdown around the time I mentioned above -- it wasn't functioning for at
least an hour. Thanks!)
Thanks for the good thoughts. We have an internal status page that we use
and are in the process of figuring out how to make more public. Also, we
are very aware of site slowness and performance issues and have made
addressing that one of our team's top priorities. In fact (ironically) many
of the recent issues with HQ have been a result of measures we are trying
to take to improve performance - measures which have largely not been fully
rolled out and thus haven't had any noticeable effect yet. Unfortunately
due to the volume of data on CommCare HQ and our desire to fully test
everything we do - these changes are taking some time to fully role out.
thanks,
Cory
···
On Thu, Feb 26, 2015 at 11:35 AM, Amelia Sagoff wrote:
Dear Users,
We have completed final verification that the performance issues related
to the initial outage are all resolved. All reports should be up-to-date
and displaying data in real-time as usual.
Thanks again for your patience.
Sincerely,
Amelia
On Mon, Feb 23, 2015 at 11:42 AM, Amelia Sagoff asagoff@dimagi.com wrote:
Dear Users,
We're happy to report that the remaining reports are now up to date (the
Case Activy report and Call Center mobile reports).
We are investigating some reported issues related to the case and submit
history to determine if they are linked to the same underlying issue. We
will communicate directly with those who have reported issues once the root
cause is determined.
We believe all site-wide performance issues related to the iniatial
outtage are resolved and will confirm over the coming days once our team
has completed verification.
At this point, please report any issues you are having through the normal
"report an issue" channel.
Thanks again for bearing with us as we take the final steps to resolve
HQ's performance issues.
Sincerely,
Amelia
On Tue, Feb 17, 2015 at 5:31 PM, Amelia Sagoff asagoff@dimagi.com wrote:
Dear Users,
Thank you for your patience as we resolve the remaining CommCareHQ
reports delays. We have another update for you with some good news.
The following reports that were recently experiencing delays are now
up-to-date and displaying data in real-time:
The Submit History report (where you can view all form submissions)
All worker monitoring reports except the Case Activity report
The following reports are still experiencing delays and may not be
displaying data in real-time:
The Case Activity report
Mobile reports that rely on the advanced "Call Center"
functionality
All other reports and features on CommCareHQ have been functioning
normally.
We have pinpointed the issues causing the performance problems, and
taken measure to address them. We are confident we have addressed the
problem, so we anticipate any functionality that is now working and
up-to-date will reliably remain.
Our apologies to those still affected; at this point, we are waiting for
the measures we put in place to take effect on the Case Activity and Call
Center reports. We will update you when the remaining out-of-date reports
listed above are also back to normal.
Sincerely,
Amelia
On Sun, Feb 15, 2015 at 11:55 AM, Cory Zue czue@dimagi.com wrote:
Hi all,
Just a quick update on this issue. Everything is progressing smoothly
but the rebuild operation is taking some time. Many reports are back to
normal, including the case list report, however submit history and others
are still not showing all the latest data.
Based on the current progress rate and barring any unforeseen issues,
we expect things to be 100% restored at some point early in the week. In
the meantime, please continue to use the workarounds below to get your data
from HQ.
Appreciate your ongoing patience as we work to resolve this as fast as
we can.
Cory
On Wed, Feb 11, 2015 at 10:19 PM, Cory Zue czue@dimagi.com wrote:
Hey all,
Just wanted to mention that we are still working with our database
provider to resolve these issues. HQ should be fully stable, all data is
being saved fine, and all mobile interactions should work smoothly.
However, many parts of the site will not show the most recent data,
including the submit history and case list reports.
For a definitive source of data, we recommend using the "Raw Forms,
Errors, and Duplicates" report with the "normal submissions" option
checked, or data exports.
Apologies again for the continued inconvenience. We will update out as
soon as anything changes.
You may have noticed some continued slowness and out of date reports
today on CommCareHQ. I would like to reassure you that there is still no
risk of data loss for submitted data.
There is also some good news I can share. We have identified the
cause of some of the problems, and have set changes in motion to make HQ
faster and keep web reports up to date.
We will continue to update you on our progress. Please accept our
apologies for the inconvenience and thanks again for your patience!
Sincerely,
--
Nate Haduch
Technical Support Analyst
Dimagi, Inc | 585 Massachusetts Ave | Suite 3 | Cambridge, MA 02139
web: http://www.dimagi.com/