Emergency maintenance, update

Users
1

Hi everyone,

Just wanted to provide hopefully a final update on this.

  1. It looks like the task processor is a lot more stable with the changes.
    The issue with reminders and exports looks to be resolved (though do let us
    know if you notice more issues).
  2. Due to an oversight on our part the implications of the work were
    actually further reaching than just the APIs. For the last 24 hours many
    reports including the Submit History, the single Case History, and the
    Worker Activity report were not displaying the latest data. The data is
    fine, and the reports should all be fixed within the next few hours.

Again, sorry about the oversights here. It is our top priority to keep the
site running and the information accurate. We're doing an internal
retrospective to prevent this type of error from happening again.

thanks,
Cory

ยทยทยท On Mon, Jul 29, 2013 at 6:40 PM, Cory Zue wrote:

Hi all,

We had to do some emergency maintenance to the server just now in hopes of
fixing this problem. We are hoping we have created enough processing power
to improve the performance of exports so that you will no longer see this
message under current load.

This maintenance should also improve delivery times on reminders and form
and case forwarding, for those of you who have recently reported delays in
those parts of the system.

We can't guarantee this problem is fully addressed, but we are still
tracking it closely and will continue to do what we can to increase
performance.

As a side effect of this emergency maintenance, we had to temporarily
disable new forms from showing up in the version 4 form API (if you don't
know what that means you can ignore it). We expect these forms will be
available via that API within 24 hours. Sorry about any inconvenience that
this causes - all other areas of HQ should not be affected.

best,
Cory

On Mon, Jul 15, 2013 at 8:57 AM, Cory Zue czue@dimagi.com wrote:

Hi all,

You may have noticed getting a "task processor not detected" error when
trying to download exports in the last 24 hours. This is due to a change in
our infrastructure that reduced the amount of resources we have to generate
exports. As a result we are starting to see that when we have a higher than
normal load of people creating exports they get backed up and this error
shows up. If you see this error please note the following:

  1. Please be patient and leave the window open. Your export will still
    eventually complete. It might take a long time to do so, though.
  2. Please do not close and reopen a new export. This will only add to the
    backlog and make everyone's (including your) exports take longer.
  3. We are looking into alternative solutions, including increasing our
    export-processing power and sending an email when exports are ready so you
    don't have to leave the window open. We'll update when there's a more
    satisfying solution in place.

thanks,
Cory