Problem with case list reports and case APIs

Hi all,

We're having a problem with one of the pieces of infrastructure used to
generate the case list reports (including case reassignment), and apis. We
are looking into it urgently and will let you know as soon as things are
back up and running.

Cory

Hi again,

This has been resolved now. Sorry about the inconvenience!

Cory

··· On Thu, Mar 7, 2013 at 7:48 AM, Cory Zue wrote:

Hi all,

We're having a problem with one of the pieces of infrastructure used to
generate the case list reports (including case reassignment), and apis. We
are looking into it urgently and will let you know as soon as things are
back up and running.

Cory

Hi Cory,

I'm now getting a 500 error?

Bird

··· On Thu, Mar 7, 2013 at 5:46 AM, Cory Zue wrote:

Hi again,

This has been resolved now. Sorry about the inconvenience!

Cory

On Thu, Mar 7, 2013 at 7:48 AM, Cory Zue czue@dimagi.com wrote:

Hi all,

We're having a problem with one of the pieces of infrastructure used to
generate the case list reports (including case reassignment), and apis. We
are looking into it urgently and will let you know as soon as things are
back up and running.

Cory

--
You received this message because you are subscribed to the Google Groups
"commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

Sorry, to clarify, that's when I'm trying to get cases through the API.

CommcareHQ is saying "No data available to display. Please try changing
your filters."

Thanks,

Bird

··· On Thu, Mar 7, 2013 at 5:51 AM, Sarah Bird wrote:

Hi Cory,

I'm now getting a 500 error?

Bird

On Thu, Mar 7, 2013 at 5:46 AM, Cory Zue czue@dimagi.com wrote:

Hi again,

This has been resolved now. Sorry about the inconvenience!

Cory

On Thu, Mar 7, 2013 at 7:48 AM, Cory Zue czue@dimagi.com wrote:

Hi all,

We're having a problem with one of the pieces of infrastructure used to
generate the case list reports (including case reassignment), and apis. We
are looking into it urgently and will let you know as soon as things are
back up and running.

Cory

--
You received this message because you are subscribed to the Google Groups
"commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

Hi Sarah,

Sorry, it looks like the problem has resurfaced again. We're looking into
the root cause again.

Cory

··· On Thu, Mar 7, 2013 at 8:53 AM, Sarah Bird wrote:

Sorry, to clarify, that's when I'm trying to get cases through the API.

CommcareHQ is saying "No data available to display. Please try changing
your filters."

Thanks,

Bird

On Thu, Mar 7, 2013 at 5:51 AM, Sarah Bird sbird@alum.mit.edu wrote:

Hi Cory,

I'm now getting a 500 error?

Bird

On Thu, Mar 7, 2013 at 5:46 AM, Cory Zue czue@dimagi.com wrote:

Hi again,

This has been resolved now. Sorry about the inconvenience!

Cory

On Thu, Mar 7, 2013 at 7:48 AM, Cory Zue czue@dimagi.com wrote:

Hi all,

We're having a problem with one of the pieces of infrastructure used to
generate the case list reports (including case reassignment), and apis. We
are looking into it urgently and will let you know as soon as things are
back up and running.

Cory

--
You received this message because you are subscribed to the Google
Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

--
You received this message because you are subscribed to the Google Groups
"commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Just so you also know Cory, I generated a case list before your email. With 16 cases, 3 of which were closed, but now regenerating that same user cases now has 14 cases with 2 closed users. Meaning 2 cases not appearing.

-Emeka Chukwu-

··· Sent from my iPhone

On Mar 7, 2013, at 2:56 PM, Cory Zue czue@dimagi.com wrote:

Hi Sarah,

Sorry, it looks like the problem has resurfaced again. We're looking into the root cause again.

Cory

On Thu, Mar 7, 2013 at 8:53 AM, Sarah Bird sbird@alum.mit.edu wrote:
Sorry, to clarify, that's when I'm trying to get cases through the API.

CommcareHQ is saying "No data available to display. Please try changing your filters."

Thanks,

Bird

On Thu, Mar 7, 2013 at 5:51 AM, Sarah Bird sbird@alum.mit.edu wrote:
Hi Cory,

I'm now getting a 500 error?

Bird

On Thu, Mar 7, 2013 at 5:46 AM, Cory Zue czue@dimagi.com wrote:
Hi again,

This has been resolved now. Sorry about the inconvenience!

Cory

On Thu, Mar 7, 2013 at 7:48 AM, Cory Zue czue@dimagi.com wrote:
Hi all,

We're having a problem with one of the pieces of infrastructure used to generate the case list reports (including case reassignment), and apis. We are looking into it urgently and will let you know as soon as things are back up and running.

Cory

--
You received this message because you are subscribed to the Google Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

You received this message because you are subscribed to the Google Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

500 again?

··· On Thu, Mar 7, 2013 at 6:18 AM, Daniel Myung wrote:

To clarify:

The case api and search service is fully back online, we are now running
tests to ensure our search index is up to date with the database to ensure
that the two are in sync.

Dan

On Thu, Mar 7, 2013 at 9:14 AM, Daniel Myung dmyung@dimagi.com wrote:

Hi Users,
Since the resurfaced problem a few minutes ago, we've again restarted our
search service system which is slowly coming back online - when it's in a
partial state - it may return data that it has available.

OK, the services are fully back online - can you test again?
If there is missing data, there is a chance that newly registered cases
that came in during the outage were not picked up.

Dan

On Thu, Mar 7, 2013 at 9:10 AM, Emeka2015 emeka2015@gmail.com wrote:

Just so you also know Cory, I generated a case list before your email.
With 16 cases, 3 of which were closed, but now regenerating that same user
cases now has 14 cases with 2 closed users. Meaning 2 cases not appearing.

-Emeka Chukwu-
Sent from my iPhone

On Mar 7, 2013, at 2:56 PM, Cory Zue czue@dimagi.com wrote:

Hi Sarah,

Sorry, it looks like the problem has resurfaced again. We're looking
into the root cause again.

Cory

On Thu, Mar 7, 2013 at 8:53 AM, Sarah Bird sbird@alum.mit.edu wrote:

Sorry, to clarify, that's when I'm trying to get cases through the API.

CommcareHQ is saying "No data available to display. Please try changing
your filters."

Thanks,

Bird

On Thu, Mar 7, 2013 at 5:51 AM, Sarah Bird sbird@alum.mit.edu wrote:

Hi Cory,

I'm now getting a 500 error?

Bird

On Thu, Mar 7, 2013 at 5:46 AM, Cory Zue czue@dimagi.com wrote:

Hi again,

This has been resolved now. Sorry about the inconvenience!

Cory

On Thu, Mar 7, 2013 at 7:48 AM, Cory Zue czue@dimagi.com wrote:

Hi all,

We're having a problem with one of the pieces of infrastructure used
to generate the case list reports (including case reassignment), and apis.
We are looking into it urgently and will let you know as soon as things are
back up and running.

Cory

--
You received this message because you are subscribed to the Google
Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

--
You received this message because you are subscribed to the Google
Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

No worries. Difficult day :frowning: Thanks for all your hard work.

··· On Thu, Mar 7, 2013 at 7:00 AM, Daniel Myung wrote:

Yes, sorry - it's gone offline again - am working on getting it back up.

On Thu, Mar 7, 2013 at 9:58 AM, Sarah Bird sbird@alum.mit.edu wrote:

500 again?

On Thu, Mar 7, 2013 at 6:18 AM, Daniel Myung dmyung@dimagi.com wrote:

To clarify:

The case api and search service is fully back online, we are now running
tests to ensure our search index is up to date with the database to ensure
that the two are in sync.

Dan

On Thu, Mar 7, 2013 at 9:14 AM, Daniel Myung dmyung@dimagi.com wrote:

Hi Users,
Since the resurfaced problem a few minutes ago, we've again restarted
our search service system which is slowly coming back online - when it's in
a partial state - it may return data that it has available.

OK, the services are fully back online - can you test again?
If there is missing data, there is a chance that newly registered cases
that came in during the outage were not picked up.

Dan

On Thu, Mar 7, 2013 at 9:10 AM, Emeka2015 emeka2015@gmail.com wrote:

Just so you also know Cory, I generated a case list before your email.
With 16 cases, 3 of which were closed, but now regenerating that same user
cases now has 14 cases with 2 closed users. Meaning 2 cases not appearing.

-Emeka Chukwu-
Sent from my iPhone

On Mar 7, 2013, at 2:56 PM, Cory Zue czue@dimagi.com wrote:

Hi Sarah,

Sorry, it looks like the problem has resurfaced again. We're looking
into the root cause again.

Cory

On Thu, Mar 7, 2013 at 8:53 AM, Sarah Bird sbird@alum.mit.edu wrote:

Sorry, to clarify, that's when I'm trying to get cases through the
API.

CommcareHQ is saying "No data available to display. Please try
changing your filters."

Thanks,

Bird

On Thu, Mar 7, 2013 at 5:51 AM, Sarah Bird sbird@alum.mit.eduwrote:

Hi Cory,

I'm now getting a 500 error?

Bird

On Thu, Mar 7, 2013 at 5:46 AM, Cory Zue czue@dimagi.com wrote:

Hi again,

This has been resolved now. Sorry about the inconvenience!

Cory

On Thu, Mar 7, 2013 at 7:48 AM, Cory Zue czue@dimagi.com wrote:

Hi all,

We're having a problem with one of the pieces of infrastructure
used to generate the case list reports (including case reassignment), and
apis. We are looking into it urgently and will let you know as soon as
things are back up and running.

Cory

--
You received this message because you are subscribed to the Google
Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

--
You received this message because you are subscribed to the Google
Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

--
You received this message because you are subscribed to the Google Groups
"commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

Looking good from here. Thanks!

··· On Thu, Mar 7, 2013 at 8:24 AM, Daniel Myung wrote:

Dear Users,

We have completed a rebalance and reallocation of resources on our
search+api machines and restored its stability, but we'll be continuing to
keep a close eye on things. We have resumed synchronization for the past
hour with the database, and it is back in sync, and the search+api service
is responding well. So we think the issues have been resolved and will be
OK. The issue stemmed from our case API/search index being crowded on just
1 of our 2 servers causing it to not respond in a timely manner. I have
spread the case index over the 2 machines and it is behaving significantly
better now.

We're also going to be focusing on longer term fixes to address the
underlying problem - and will keep you posted should anything require
attention again.
Thanks for the quick feedback and alerts when issues did come up, and
apologies again for the inconvenience.

Dan

On Thu, Mar 7, 2013 at 10:03 AM, Sarah Bird sbird@alum.mit.edu wrote:

No worries. Difficult day :frowning: Thanks for all your hard work.

On Thu, Mar 7, 2013 at 7:00 AM, Daniel Myung dmyung@dimagi.com wrote:

Yes, sorry - it's gone offline again - am working on getting it back up.

On Thu, Mar 7, 2013 at 9:58 AM, Sarah Bird sbird@alum.mit.edu wrote:

500 again?

On Thu, Mar 7, 2013 at 6:18 AM, Daniel Myung dmyung@dimagi.com wrote:

To clarify:

The case api and search service is fully back online, we are now
running tests to ensure our search index is up to date with the database to
ensure that the two are in sync.

Dan

On Thu, Mar 7, 2013 at 9:14 AM, Daniel Myung dmyung@dimagi.comwrote:

Hi Users,
Since the resurfaced problem a few minutes ago, we've again restarted
our search service system which is slowly coming back online - when it's in
a partial state - it may return data that it has available.

OK, the services are fully back online - can you test again?
If there is missing data, there is a chance that newly registered
cases that came in during the outage were not picked up.

Dan

On Thu, Mar 7, 2013 at 9:10 AM, Emeka2015 emeka2015@gmail.comwrote:

Just so you also know Cory, I generated a case list before your
email. With 16 cases, 3 of which were closed, but now regenerating that
same user cases now has 14 cases with 2 closed users. Meaning 2 cases not
appearing.

-Emeka Chukwu-
Sent from my iPhone

On Mar 7, 2013, at 2:56 PM, Cory Zue czue@dimagi.com wrote:

Hi Sarah,

Sorry, it looks like the problem has resurfaced again. We're looking
into the root cause again.

Cory

On Thu, Mar 7, 2013 at 8:53 AM, Sarah Bird sbird@alum.mit.eduwrote:

Sorry, to clarify, that's when I'm trying to get cases through the
API.

CommcareHQ is saying "No data available to display. Please try
changing your filters."

Thanks,

Bird

On Thu, Mar 7, 2013 at 5:51 AM, Sarah Bird sbird@alum.mit.eduwrote:

Hi Cory,

I'm now getting a 500 error?

Bird

On Thu, Mar 7, 2013 at 5:46 AM, Cory Zue czue@dimagi.com wrote:

Hi again,

This has been resolved now. Sorry about the inconvenience!

Cory

On Thu, Mar 7, 2013 at 7:48 AM, Cory Zue czue@dimagi.com wrote:

Hi all,

We're having a problem with one of the pieces of infrastructure
used to generate the case list reports (including case reassignment), and
apis. We are looking into it urgently and will let you know as soon as
things are back up and running.

Cory

--
You received this message because you are subscribed to the
Google Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

--
You received this message because you are subscribed to the Google
Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

--
You received this message because you are subscribed to the Google
Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

--
You received this message because you are subscribed to the Google Groups
"commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

Hi Users,
Since the resurfaced problem a few minutes ago, we've again restarted our
search service system which is slowly coming back online - when it's in a
partial state - it may return data that it has available.

OK, the services are fully back online - can you test again?
If there is missing data, there is a chance that newly registered cases
that came in during the outage were not picked up.

Dan

··· On Thu, Mar 7, 2013 at 9:10 AM, Emeka2015 wrote:

Just so you also know Cory, I generated a case list before your email.
With 16 cases, 3 of which were closed, but now regenerating that same user
cases now has 14 cases with 2 closed users. Meaning 2 cases not appearing.

-Emeka Chukwu-
Sent from my iPhone

On Mar 7, 2013, at 2:56 PM, Cory Zue czue@dimagi.com wrote:

Hi Sarah,

Sorry, it looks like the problem has resurfaced again. We're looking into
the root cause again.

Cory

On Thu, Mar 7, 2013 at 8:53 AM, Sarah Bird sbird@alum.mit.edu wrote:

Sorry, to clarify, that's when I'm trying to get cases through the API.

CommcareHQ is saying "No data available to display. Please try changing
your filters."

Thanks,

Bird

On Thu, Mar 7, 2013 at 5:51 AM, Sarah Bird sbird@alum.mit.edu wrote:

Hi Cory,

I'm now getting a 500 error?

Bird

On Thu, Mar 7, 2013 at 5:46 AM, Cory Zue czue@dimagi.com wrote:

Hi again,

This has been resolved now. Sorry about the inconvenience!

Cory

On Thu, Mar 7, 2013 at 7:48 AM, Cory Zue czue@dimagi.com wrote:

Hi all,

We're having a problem with one of the pieces of infrastructure used
to generate the case list reports (including case reassignment), and apis.
We are looking into it urgently and will let you know as soon as things are
back up and running.

Cory

--
You received this message because you are subscribed to the Google
Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

--
You received this message because you are subscribed to the Google Groups
"commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

To clarify:

The case api and search service is fully back online, we are now running
tests to ensure our search index is up to date with the database to ensure
that the two are in sync.

Dan

··· On Thu, Mar 7, 2013 at 9:14 AM, Daniel Myung wrote:

Hi Users,
Since the resurfaced problem a few minutes ago, we've again restarted our
search service system which is slowly coming back online - when it's in a
partial state - it may return data that it has available.

OK, the services are fully back online - can you test again?
If there is missing data, there is a chance that newly registered cases
that came in during the outage were not picked up.

Dan

On Thu, Mar 7, 2013 at 9:10 AM, Emeka2015 emeka2015@gmail.com wrote:

Just so you also know Cory, I generated a case list before your email.
With 16 cases, 3 of which were closed, but now regenerating that same user
cases now has 14 cases with 2 closed users. Meaning 2 cases not appearing.

-Emeka Chukwu-
Sent from my iPhone

On Mar 7, 2013, at 2:56 PM, Cory Zue czue@dimagi.com wrote:

Hi Sarah,

Sorry, it looks like the problem has resurfaced again. We're looking into
the root cause again.

Cory

On Thu, Mar 7, 2013 at 8:53 AM, Sarah Bird sbird@alum.mit.edu wrote:

Sorry, to clarify, that's when I'm trying to get cases through the API.

CommcareHQ is saying "No data available to display. Please try changing
your filters."

Thanks,

Bird

On Thu, Mar 7, 2013 at 5:51 AM, Sarah Bird sbird@alum.mit.edu wrote:

Hi Cory,

I'm now getting a 500 error?

Bird

On Thu, Mar 7, 2013 at 5:46 AM, Cory Zue czue@dimagi.com wrote:

Hi again,

This has been resolved now. Sorry about the inconvenience!

Cory

On Thu, Mar 7, 2013 at 7:48 AM, Cory Zue czue@dimagi.com wrote:

Hi all,

We're having a problem with one of the pieces of infrastructure used
to generate the case list reports (including case reassignment), and apis.
We are looking into it urgently and will let you know as soon as things are
back up and running.

Cory

--
You received this message because you are subscribed to the Google
Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

--
You received this message because you are subscribed to the Google
Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Yes, sorry - it's gone offline again - am working on getting it back up.

··· On Thu, Mar 7, 2013 at 9:58 AM, Sarah Bird wrote:

500 again?

On Thu, Mar 7, 2013 at 6:18 AM, Daniel Myung dmyung@dimagi.com wrote:

To clarify:

The case api and search service is fully back online, we are now running
tests to ensure our search index is up to date with the database to ensure
that the two are in sync.

Dan

On Thu, Mar 7, 2013 at 9:14 AM, Daniel Myung dmyung@dimagi.com wrote:

Hi Users,
Since the resurfaced problem a few minutes ago, we've again restarted
our search service system which is slowly coming back online - when it's in
a partial state - it may return data that it has available.

OK, the services are fully back online - can you test again?
If there is missing data, there is a chance that newly registered cases
that came in during the outage were not picked up.

Dan

On Thu, Mar 7, 2013 at 9:10 AM, Emeka2015 emeka2015@gmail.com wrote:

Just so you also know Cory, I generated a case list before your email.
With 16 cases, 3 of which were closed, but now regenerating that same user
cases now has 14 cases with 2 closed users. Meaning 2 cases not appearing.

-Emeka Chukwu-
Sent from my iPhone

On Mar 7, 2013, at 2:56 PM, Cory Zue czue@dimagi.com wrote:

Hi Sarah,

Sorry, it looks like the problem has resurfaced again. We're looking
into the root cause again.

Cory

On Thu, Mar 7, 2013 at 8:53 AM, Sarah Bird sbird@alum.mit.edu wrote:

Sorry, to clarify, that's when I'm trying to get cases through the API.

CommcareHQ is saying "No data available to display. Please try
changing your filters."

Thanks,

Bird

On Thu, Mar 7, 2013 at 5:51 AM, Sarah Bird sbird@alum.mit.edu wrote:

Hi Cory,

I'm now getting a 500 error?

Bird

On Thu, Mar 7, 2013 at 5:46 AM, Cory Zue czue@dimagi.com wrote:

Hi again,

This has been resolved now. Sorry about the inconvenience!

Cory

On Thu, Mar 7, 2013 at 7:48 AM, Cory Zue czue@dimagi.com wrote:

Hi all,

We're having a problem with one of the pieces of infrastructure
used to generate the case list reports (including case reassignment), and
apis. We are looking into it urgently and will let you know as soon as
things are back up and running.

Cory

--
You received this message because you are subscribed to the Google
Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

--
You received this message because you are subscribed to the Google
Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

--
You received this message because you are subscribed to the Google Groups
"commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Dear Users,

We have completed a rebalance and reallocation of resources on our
search+api machines and restored its stability, but we'll be continuing to
keep a close eye on things. We have resumed synchronization for the past
hour with the database, and it is back in sync, and the search+api service
is responding well. So we think the issues have been resolved and will be
OK. The issue stemmed from our case API/search index being crowded on just
1 of our 2 servers causing it to not respond in a timely manner. I have
spread the case index over the 2 machines and it is behaving significantly
better now.

We're also going to be focusing on longer term fixes to address the
underlying problem - and will keep you posted should anything require
attention again.
Thanks for the quick feedback and alerts when issues did come up, and
apologies again for the inconvenience.

Dan

··· On Thu, Mar 7, 2013 at 10:03 AM, Sarah Bird wrote:

No worries. Difficult day :frowning: Thanks for all your hard work.

On Thu, Mar 7, 2013 at 7:00 AM, Daniel Myung dmyung@dimagi.com wrote:

Yes, sorry - it's gone offline again - am working on getting it back up.

On Thu, Mar 7, 2013 at 9:58 AM, Sarah Bird sbird@alum.mit.edu wrote:

500 again?

On Thu, Mar 7, 2013 at 6:18 AM, Daniel Myung dmyung@dimagi.com wrote:

To clarify:

The case api and search service is fully back online, we are now
running tests to ensure our search index is up to date with the database to
ensure that the two are in sync.

Dan

On Thu, Mar 7, 2013 at 9:14 AM, Daniel Myung dmyung@dimagi.com wrote:

Hi Users,
Since the resurfaced problem a few minutes ago, we've again restarted
our search service system which is slowly coming back online - when it's in
a partial state - it may return data that it has available.

OK, the services are fully back online - can you test again?
If there is missing data, there is a chance that newly registered
cases that came in during the outage were not picked up.

Dan

On Thu, Mar 7, 2013 at 9:10 AM, Emeka2015 emeka2015@gmail.com wrote:

Just so you also know Cory, I generated a case list before your
email. With 16 cases, 3 of which were closed, but now regenerating that
same user cases now has 14 cases with 2 closed users. Meaning 2 cases not
appearing.

-Emeka Chukwu-
Sent from my iPhone

On Mar 7, 2013, at 2:56 PM, Cory Zue czue@dimagi.com wrote:

Hi Sarah,

Sorry, it looks like the problem has resurfaced again. We're looking
into the root cause again.

Cory

On Thu, Mar 7, 2013 at 8:53 AM, Sarah Bird sbird@alum.mit.eduwrote:

Sorry, to clarify, that's when I'm trying to get cases through the
API.

CommcareHQ is saying "No data available to display. Please try
changing your filters."

Thanks,

Bird

On Thu, Mar 7, 2013 at 5:51 AM, Sarah Bird sbird@alum.mit.eduwrote:

Hi Cory,

I'm now getting a 500 error?

Bird

On Thu, Mar 7, 2013 at 5:46 AM, Cory Zue czue@dimagi.com wrote:

Hi again,

This has been resolved now. Sorry about the inconvenience!

Cory

On Thu, Mar 7, 2013 at 7:48 AM, Cory Zue czue@dimagi.com wrote:

Hi all,

We're having a problem with one of the pieces of infrastructure
used to generate the case list reports (including case reassignment), and
apis. We are looking into it urgently and will let you know as soon as
things are back up and running.

Cory

--
You received this message because you are subscribed to the Google
Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

--
You received this message because you are subscribed to the Google
Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

--
You received this message because you are subscribed to the Google
Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
skype: birdsarah
email: sbird@alum.mit.edu
web: www.sarahbird.org | www.bonvaya.com

--
You received this message because you are subscribed to the Google Groups
"commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.