Full excel export doesn't work on production

Pawel_Reise · May 19, 2014, 6:09pm

Hi,

it seems like full excel export doesn't work on production (for example
for this report:
: Upgrade Required - CommCare HQ).
I got e-mail with the link, but when I click it, 500 Error appears.
However when I tried to reproduce it on my computer it just worked fine.
Did you notice any error messages in production logs?

Best regards
Pawel

···

--- Ta wiadomość e-mail jest wolna od wirusów i złośliwego oprogramowania, ponieważ ochrona avast! Antivirus jest aktywna. http://www.avast.com

dannyroberts · May 20, 2014, 6:24pm

Hi Pawel,

The problem is that you're using the file system to save the report. This
works on your local machine, but doesn't work in production where we have
different machines that don't share a file system. If the report is saved
on the celery machine for example, the django worker machine won't have
access to it. In all other places that do this kind of thing, we've gotten
around that by saving that file to redis.

I think that should get you started, but let me know if you need more
context.

Cheers,
Danny

···

On Mon, May 19, 2014 at 2:09 PM, Pawel Reise wrote:

Hi,

it seems like full excel export doesn't work on production (for example
for this report: Log In :: CommCare HQ - CommCare HQ
custom/mcct_project_review_page/). I got e-mail with the link, but when I
click it, 500 Error appears. However when I tried to reproduce it on my
computer it just worked fine. Did you notice any error messages in
production logs?

Best regards
Pawel

Ta wiadomość e-mail jest wolna od wirusów i złośliwego oprogramowania,
ponieważ ochrona avast! Antivirus jest aktywna.
http://www.avast.com

--

--- You received this message because you are subscribed to the Google
Groups "CommCare Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Pawel_Reise · May 20, 2014, 6:57pm

Hi Danny,

actually, as far as I see, I think we do save it to redis - we execute
export_all_rows_task which executes _store_excel_in_redis method:
https://github.com/dimagi/commcare-hq/blob/master/custom/m4change/reports/mcct_project_review.py#L280

github.com

dimagi/commcare-hq/blob/master/corehq/apps/reports/tasks.py#L189


      
              if not recipient_list:
                  recipient_list = [report.request.couch_user.get_email()]
              for recipient in recipient_list:
                  _send_email(report.request.couch_user, report, hash_id, recipient=recipient, subject=subject)
                  logger.info(f'Sent {report.name} with hash {hash_id} to {recipient}')
          
          

          
def _send_email(user, report, hash_id, recipient, subject=None):
              link = absolute_reverse("export_report", args=[report.domain, str(hash_id),
                                                             report.export_format])
          
          
    send_report_download_email(report.name, recipient, link, subject)
          
          

          
def _store_excel_in_blobdb(report_class, file, domain, report_slug):
              key = uuid.uuid4().hex
              expired = 60 * 24 * 7  # 7 days
              db = get_blob_db()
          
          
    kw = {
                  "domain": domain,

Is there something I miss here? Should we also override excel_response
method in report?

Best regards
Pawel

···

On 5/20/2014 8:24 PM, Danny Roberts wrote: > Hi Pawel, > > The problem is that you're using the file system to save the report. > This works on your local machine, but doesn't work in production where > we have different machines that don't share a file system. If the > report is saved on the celery machine for example, the django worker > machine won't have access to it. In all other places that do this kind > of thing, we've gotten around that by saving that file to redis. > > I think that should get you started, but let me know if you need more > context. > > Cheers, > Danny > > > On Mon, May 19, 2014 at 2:09 PM, Pawel Reise <preise@soldevelo.com > wrote: > > Hi, > > it seems like full excel export doesn't work on production (for > example for this report: > https://www.commcarehq.org/a/test-pathfinder/reports/custom/mcct_project_review_page/). > I got e-mail with the link, but when I click it, 500 Error > appears. However when I tried to reproduce it on my computer it > just worked fine. Did you notice any error messages in production > logs? > > Best regards > Pawel > > --- > Ta wiadomość e-mail jest wolna od wirusów i złośliwego > oprogramowania, ponieważ ochrona avast! Antivirus jest aktywna. > http://www.avast.com > > -- > > --- You received this message because you are subscribed to the > Google Groups "CommCare Developers" group. > To unsubscribe from this group and stop receiving emails from it, > send an email to commcare-developers+unsubscribe@googlegroups.com > . > For more options, visit https://groups.google.com/d/optout. > > > -- > > --- > You received this message because you are subscribed to the Google > Groups "CommCare Developers" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to commcare-developers+unsubscribe@googlegroups.com > . > For more options, visit https://groups.google.com/d/optout.

dannyroberts · May 20, 2014, 7:04pm

Yeah but look at

https://github.com/dimagi/commcare-hq/blame/master/corehq/apps/reports/tasks.py#L219

it stores the filename in redis; but that tmp file won't be accessible from
the other machine. Instead you have to save the file's contents in redis.
You also need to change how you access it:

github.com

dimagi/commcare-hq/blob/master/corehq/apps/reports/views.py#L1021-1021


      
                      messages.success(request, _("Report sent to this report's recipients"))
          
              if send_count:
                  return HttpResponse(reverse("reports_home", args=(domain,)) + '#scheduled-reports')
              else:
                  return HttpResponseRedirect(reverse("reports_home", args=(domain,)) + '#scheduled-reports')
          
          
          def _can_send_test_report(report_id, user, domain):
              try:
                  report = ReportNotification.get(report_id)
              except ResourceNotFound:
                  return False
          
              if report.domain != domain:
                  return False
          
              return user._id == report.owner._id or user.is_domain_admin(domain)
          
          
          def get_scheduled_report_response(couch_user, domain, scheduled_report_id,

.

Cheers,
Danny

···

On Tue, May 20, 2014 at 2:57 PM, Pawel Reise wrote:

Hi Danny,

actually, as far as I see, I think we do save it to redis - we execute
export_all_rows_task which executes _store_excel_in_redis method:

https://github.com/dimagi/commcare-hq/blob/master/custom/m4change/reports/mcct_project_review.py#L280

commcare-hq/corehq/apps/reports/tasks.py at master · dimagi/commcare-hq · GitHub

Is there something I miss here? Should we also override excel_response
method in report?

Best regards
Pawel

On 5/20/2014 8:24 PM, Danny Roberts wrote:

Hi Pawel,

The problem is that you're using the file system to save the report. This
works on your local machine, but doesn't work in production where we have
different machines that don't share a file system. If the report is saved
on the celery machine for example, the django worker machine won't have
access to it. In all other places that do this kind of thing, we've gotten
around that by saving that file to redis.

I think that should get you started, but let me know if you need more
context.

Cheers,
Danny

On Mon, May 19, 2014 at 2:09 PM, Pawel Reise preise@soldevelo.com wrote:

Hi,

it seems like full excel export doesn't work on production (for example
for this report:
: Upgrade Required - CommCare HQ).
I got e-mail with the link, but when I click it, 500 Error appears. However
when I tried to reproduce it on my computer it just worked fine. Did you
notice any error messages in production logs?

Best regards
Pawel

Ta wiadomość e-mail jest wolna od wirusów i złośliwego oprogramowania,
ponieważ ochrona avast! Antivirus jest aktywna.
http://www.avast.com

--

--- You received this message because you are subscribed to the Google
Groups "CommCare Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups
"CommCare Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups
"CommCare Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

dannyroberts · May 23, 2014, 4:15pm

It's fine to use a temp file while building it, but you should then dump
the contents to redis, because that will be available later whereas the
tempfile won't be.

See
https://github.com/dimagi/commcare-hq/blame/master/corehq/apps/reports/tasks.py#L226
.

···

On Wed, May 21, 2014 at 11:32 AM, Krzysztof Kaczmarczyk < krzysztof.kaczmarczyk@soldevelo.com> wrote:

Hi Daniel,

I believe there was suggestion that using tmp files would be more memory
efficient for large exports.
See the discussion on PR:
Bihar reports: Added possibility to export all rows to excel file using ... by msienkiewicz · Pull Request #2574 · dimagi/commcare-hq · GitHub
Do you think it should be changed to save file's contents in redis?
Also it turns out that the export feature doesn't work on bihar domain on
production as well.

Thanks,
Krzysztof

W dniu wtorek, 20 maja 2014 21:04:59 UTC+2 użytkownik Daniel Roberts
napisał:

Yeah but look at

https://github.com/dimagi/commcare-hq/blame/master/
corehq/apps/reports/tasks.py#L219

it stores the filename in redis; but that tmp file won't be accessible
from the other machine. Instead you have to save the file's contents in
redis. You also need to change how you access it:
https://github.com/dimagi/commcare-hq/blob/master/
corehq/apps/reports/views.py#L1021-1021.

Cheers,
Danny

On Tue, May 20, 2014 at 2:57 PM, Pawel Reise pre...@soldevelo.comwrote:

Hi Danny,

actually, as far as I see, I think we do save it to redis - we execute
export_all_rows_task which executes _store_excel_in_redis method:
https://github.com/dimagi/commcare-hq/blob/master/
custom/m4change/reports/mcct_project_review.py#L280
https://github.com/dimagi/commcare-hq/blob/master/
corehq/apps/reports/tasks.py#L189

Is there something I miss here? Should we also override excel_response
method in report?

Best regards
Pawel

On 5/20/2014 8:24 PM, Danny Roberts wrote:

Hi Pawel,

The problem is that you're using the file system to save the report.
This works on your local machine, but doesn't work in production where we
have different machines that don't share a file system. If the report is
saved on the celery machine for example, the django worker machine won't
have access to it. In all other places that do this kind of thing, we've
gotten around that by saving that file to redis.

I think that should get you started, but let me know if you need more
context.

Cheers,
Danny

On Mon, May 19, 2014 at 2:09 PM, Pawel Reise pre...@soldevelo.comwrote:

Hi,

it seems like full excel export doesn't work on production (for example
for this report: Log In :: CommCare HQ - CommCare HQ
custom/mcct_project_review_page/). I got e-mail with the link, but
when I click it, 500 Error appears. However when I tried to reproduce it on
my computer it just worked fine. Did you notice any error messages in
production logs?

Best regards
Pawel

Ta wiadomość e-mail jest wolna od wirusów i złośliwego oprogramowania,
ponieważ ochrona avast! Antivirus jest aktywna.
http://www.avast.com

--

--- You received this message because you are subscribed to the Google
Groups "CommCare Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google
Groups "CommCare Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google
Groups "CommCare Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups
"CommCare Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Lech_Rozanski · May 26, 2014, 9:43am

Hello,

I have submitted a pull request which should solve this issue:
fix export all feature saving to local storage instead of redis by lrozanski · Pull Request #3490 · dimagi/commcare-hq · GitHub.

Regards
Lech

···

On 05/23/2014 06:15 PM, Danny Roberts wrote: > It's fine to use a temp file while building it, but you should then > dump the contents to redis, because that will be available later > whereas the tempfile won't be. > > See > https://github.com/dimagi/commcare-hq/blame/master/corehq/apps/reports/tasks.py#L226. > > > On Wed, May 21, 2014 at 11:32 AM, Krzysztof Kaczmarczyk <krzysztof.kaczmarczyk@soldevelo.com > wrote: > > Hi Daniel, > > I believe there was suggestion that using tmp files would be more > memory efficient for large exports. > See the discussion on PR: > https://github.com/dimagi/commcare-hq/pull/2574#discussion_r9876509 > Do you think it should be changed to save file's contents in redis? > Also it turns out that the export feature doesn't work on bihar > domain on production as well. > > Thanks, > Krzysztof > > W dniu wtorek, 20 maja 2014 21:04:59 UTC+2 użytkownik Daniel > Roberts napisał: > > Yeah but look at > > https://github.com/dimagi/commcare-hq/blame/master/corehq/apps/reports/tasks.py#L219 > > it stores the filename in redis; but that tmp file won't be > accessible from the other machine. Instead you have to save > the file's contents in redis. You also need to change how you > access it: > https://github.com/dimagi/commcare-hq/blob/master/corehq/apps/reports/views.py#L1021-1021. > > Cheers, > Danny > > > On Tue, May 20, 2014 at 2:57 PM, Pawel Reise wrote: > > Hi Danny, > > actually, as far as I see, I think we do save it to redis > - we execute export_all_rows_task which executes > _store_excel_in_redis method: > https://github.com/dimagi/commcare-hq/blob/master/custom/m4change/reports/mcct_project_review.py#L280 > https://github.com/dimagi/commcare-hq/blob/master/corehq/apps/reports/tasks.py#L189 > > Is there something I miss here? Should we also override > excel_response method in report? > > Best regards > Pawel > > > On 5/20/2014 8:24 PM, Danny Roberts wrote: >> Hi Pawel, >> >> The problem is that you're using the file system to save >> the report. This works on your local machine, but doesn't >> work in production where we have different machines that >> don't share a file system. If the report is saved on the >> celery machine for example, the django worker machine >> won't have access to it. In all other places that do this >> kind of thing, we've gotten around that by saving that >> file to redis. >> >> I think that should get you started, but let me know if >> you need more context. >> >> Cheers, >> Danny >> >> >> On Mon, May 19, 2014 at 2:09 PM, Pawel Reise wrote: >> >> Hi, >> >> it seems like full excel export doesn't work on >> production (for example for this report: >> https://www.commcarehq.org/a/test-pathfinder/reports/custom/mcct_project_review_page/). >> I got e-mail with the link, but when I click it, 500 >> Error appears. However when I tried to reproduce it >> on my computer it just worked fine. Did you notice >> any error messages in production logs? >> >> Best regards >> Pawel >> >> --- >> Ta wiadomość e-mail jest wolna od wirusów i >> złośliwego oprogramowania, ponieważ ochrona avast! >> Antivirus jest aktywna. >> http://www.avast.com >> >> -- >> >> --- You received this message because you are >> subscribed to the Google Groups "CommCare Developers" >> group. >> To unsubscribe from this group and stop receiving >> emails from it, send an email to >> commcare-developers+unsubscribe@googlegroups.com. >> For more options, visit >> https://groups.google.com/d/optout. >> >> >> -- >> >> --- >> You received this message because you are subscribed to >> the Google Groups "CommCare Developers" group. >> To unsubscribe from this group and stop receiving emails >> from it, send an email to >> commcare-developers+unsubscribe@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. > > -- > > --- > You received this message because you are subscribed to > the Google Groups "CommCare Developers" group. > To unsubscribe from this group and stop receiving emails > from it, send an email to > commcare-developers+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > > > -- > > --- > You received this message because you are subscribed to the Google > Groups "CommCare Developers" group. > To unsubscribe from this group and stop receiving emails from it, > send an email to commcare-developers+unsubscribe@googlegroups.com > . > For more options, visit https://groups.google.com/d/optout. > > > -- > > --- > You received this message because you are subscribed to the Google > Groups "CommCare Developers" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to commcare-developers+unsubscribe@googlegroups.com > . > For more options, visit https://groups.google.com/d/optout.