Data migrations during application updates

Hi,

I’m trying to figure out what I need to keep in mind once the first version
of my CommCare application is in production. Of course ideally I could
prevent introducing incompatible updates to my forms, but am not sure if I
can always avoid this. One of my previous mails regarding the birthdate
could be an example. I might come to the conclusion that instead of using
an ordinary date field, I now want to break down the birthdate into
separate day, month, and year fields.

I’ve found this page
(https://confluence.dimagi.com/display/commcarepublic/Updating+CommCare+Application+Content).
But unlike I’ve expected it doesn’t include the updates of content.

Are there other resources that I didn’t find covering this topic?

Thanks,
christian

Hi Christian

This is a tricky problem and one that we don’t have any automated tools to
support. There are two possibilities that I can suggest:

  1. Make all changes to your app backward’s compatible.
  2. e.g. Changing the name of a case property would be require you to
    load both the old and new version into a form and then
    determining which is
    correct e.g. coalesce(/data/new, data/old)
    2. You could make your data compatible after exporting it to Excel
    1. This would involve converting legacy data to new formats etc.
··· On 16 June 2016 at 15:45, Christian Neumann wrote:

Hi,

I’m trying to figure out what I need to keep in mind once the first
version of my CommCare application is in production. Of course ideally I
could prevent introducing incompatible updates to my forms, but am not sure
if I can always avoid this. One of my previous mails regarding the
birthdate could be an example. I might come to the conclusion that instead
of using an ordinary date field, I now want to break down the birthdate
into separate day, month, and year fields.

I’ve found this page (
https://confluence.dimagi.com/display/commcarepublic/Updating+CommCare+Application+Content).
But unlike I’ve expected it doesn’t include the updates of content.

Are there other resources that I didn’t find covering this topic?

Thanks,
christian


You received this message because you are subscribed to the Google Groups
"commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Simon Kelly
Senior Engineer | Dimagi South Africa

Thanks Simon,

Good (and important) to know. I didn’t come across it (so I assume it doesn’t exist), but to verify: Compare does not inform me whenever I have potentially non-backwards compatible changes in my forms definitions, or does it? I know it is impossible to predict this for every case, but some cases are: E.g. when I remove a question from a form where data was already captured. Or change the names of properties. Or add/remove entries from multiple choice questions.

And regarding your suggestion #2 about exporting it to Excel. Do you refer to the bulk-import feature so bring back the converted data into CommCare?

Thanks!
christian

··· > On 23 Jun 2016, at 11:46, Simon Kelly wrote: > > Hi Christian > > This is a tricky problem and one that we don't have any automated tools to support. There are two possibilities that I can suggest: > Make all changes to your app backward's compatible. > e.g. Changing the name of a case property would be require you to load both the old and new version into a form and then determining which is correct e.g. coalesce(/data/new, data/old) > You could make your data compatible after exporting it to Excel > This would involve converting legacy data to new formats etc. > > > On 16 June 2016 at 15:45, Christian Neumann <christian.neumann@gmail.com > wrote: > Hi, > > I'm trying to figure out what I need to keep in mind once the first version of my CommCare application is in production. Of course ideally I could prevent introducing incompatible updates to my forms, but am not sure if I can always avoid this. One of my previous mails regarding the birthdate could be an example. I might come to the conclusion that instead of using an ordinary date field, I now want to break down the birthdate into separate day, month, and year fields. > > I've found this page (https://confluence.dimagi.com/display/commcarepublic/Updating+CommCare+Application+Content ). But unlike I've expected it doesn't include the updates of content. > > Are there other resources that I didn't find covering this topic? > > Thanks, > christian > > -- > You received this message because you are subscribed to the Google Groups "commcare-users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to commcare-users+unsubscribe@googlegroups.com . > For more options, visit https://groups.google.com/d/optout . > > > > -- > Simon Kelly > Senior Engineer | Dimagi South Africa > > -- > You received this message because you are subscribed to a topic in the Google Groups "commcare-users" group. > To unsubscribe from this topic, visit https://groups.google.com/d/topic/commcare-users/OsU76WmXrP0/unsubscribe . > To unsubscribe from this group and all its topics, send an email to commcare-users+unsubscribe@googlegroups.com . > For more options, visit https://groups.google.com/d/optout .

CommCare will not tell you if you’ve made non-backwards compatible changes.
One thing that may help somewhat with that is the “App Summary” (see image):

[image: Inline images 1]

This view will give you an outline of all forms and also cases. For each
case it will list all the case properties as well as where each property is
read from and written to.

What it won’t show is you is how the current version of the app differs
from the previous version.

Regarding #2, you can use the bulk-import feature to update case data but
forms are considered ‘read-only’ so there is no way to bulk update them or
to change their structure once they have been submitted.

··· On 23 June 2016 at 15:40, Christian Neumann wrote:

Thanks Simon,

Good (and important) to know. I didn’t come across it (so I assume it
doesn’t exist), but to verify: Compare does not inform me whenever I have
potentially non-backwards compatible changes in my forms definitions, or
does it? I know it is impossible to predict this for every case, but some
cases are: E.g. when I remove a question from a form where data was already
captured. Or change the names of properties. Or add/remove entries from
multiple choice questions.

And regarding your suggestion #2 about exporting it to Excel. Do you refer
to the bulk-import feature so bring back the converted data into CommCare?

Thanks!
christian

On 23 Jun 2016, at 11:46, Simon Kelly skelly@dimagi.com wrote:

Hi Christian

This is a tricky problem and one that we don’t have any automated tools to
support. There are two possibilities that I can suggest:

  1. Make all changes to your app backward’s compatible.
  2. e.g. Changing the name of a case property would be require you to
    load both the old and new version into a form and then determining which is
    correct e.g. coalesce(/data/new, data/old)
    2. You could make your data compatible after exporting it to Excel
    1. This would involve converting legacy data to new formats etc.

On 16 June 2016 at 15:45, Christian Neumann christian.neumann@gmail.com wrote:

Hi,

I’m trying to figure out what I need to keep in mind once the first
version of my CommCare application is in production. Of course ideally I
could prevent introducing incompatible updates to my forms, but am not sure
if I can always avoid this. One of my previous mails regarding the
birthdate could be an example. I might come to the conclusion that instead
of using an ordinary date field, I now want to break down the birthdate
into separate day, month, and year fields.

I’ve found this page (
https://confluence.dimagi.com/display/commcarepublic/Updating+CommCare+Application+Content).
But unlike I’ve expected it doesn’t include the updates of content.

Are there other resources that I didn’t find covering this topic?

Thanks,
christian


You received this message because you are subscribed to the Google Groups
“commcare-users” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Simon Kelly
Senior Engineer | Dimagi South Africa


You received this message because you are subscribed to a topic in the
Google Groups “commcare-users” group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/commcare-users/OsU76WmXrP0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups
“commcare-users” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Simon Kelly
Senior Engineer | Dimagi South Africa

HI Christian,

You may also find this page useful:
https://confluence.dimagi.com/display/commcarepublic/Making+Changes+to+a+Deployed+Application

Jeremy

··· On Thu, Jun 23, 2016 at 9:59 AM, Simon Kelly wrote:

CommCare will not tell you if you’ve made non-backwards compatible
changes. One thing that may help somewhat with that is the “App Summary”
(see image):

[image: Inline images 1]

This view will give you an outline of all forms and also cases. For each
case it will list all the case properties as well as where each property is
read from and written to.

What it won’t show is you is how the current version of the app differs
from the previous version.

Regarding #2, you can use the bulk-import feature to update case data but
forms are considered ‘read-only’ so there is no way to bulk update them or
to change their structure once they have been submitted.

On 23 June 2016 at 15:40, Christian Neumann christian.neumann@gmail.com wrote:

Thanks Simon,

Good (and important) to know. I didn’t come across it (so I assume it
doesn’t exist), but to verify: Compare does not inform me whenever I have
potentially non-backwards compatible changes in my forms definitions, or
does it? I know it is impossible to predict this for every case, but some
cases are: E.g. when I remove a question from a form where data was already
captured. Or change the names of properties. Or add/remove entries from
multiple choice questions.

And regarding your suggestion #2 about exporting it to Excel. Do you
refer to the bulk-import feature so bring back the converted data into
CommCare?

Thanks!
christian

On 23 Jun 2016, at 11:46, Simon Kelly skelly@dimagi.com wrote:

Hi Christian

This is a tricky problem and one that we don’t have any automated tools
to support. There are two possibilities that I can suggest:

  1. Make all changes to your app backward’s compatible.
  2. e.g. Changing the name of a case property would be require you to
    load both the old and new version into a form and then determining which is
    correct e.g. coalesce(/data/new, data/old)
    2. You could make your data compatible after exporting it to Excel
    1. This would involve converting legacy data to new formats etc.

On 16 June 2016 at 15:45, Christian Neumann christian.neumann@gmail.com wrote:

Hi,

I’m trying to figure out what I need to keep in mind once the first
version of my CommCare application is in production. Of course ideally I
could prevent introducing incompatible updates to my forms, but am not sure
if I can always avoid this. One of my previous mails regarding the
birthdate could be an example. I might come to the conclusion that instead
of using an ordinary date field, I now want to break down the birthdate
into separate day, month, and year fields.

I’ve found this page (
https://confluence.dimagi.com/display/commcarepublic/Updating+CommCare+Application+Content).
But unlike I’ve expected it doesn’t include the updates of content.

Are there other resources that I didn’t find covering this topic?

Thanks,
christian


You received this message because you are subscribed to the Google
Groups “commcare-users” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Simon Kelly
Senior Engineer | Dimagi South Africa


You received this message because you are subscribed to a topic in the
Google Groups “commcare-users” group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/commcare-users/OsU76WmXrP0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups
“commcare-users” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Simon Kelly
Senior Engineer | Dimagi South Africa


You received this message because you are subscribed to the Google Groups
“commcare-users” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Jeremy Wacksman
Dimagi http://www.dimagi.com, Inc.

Yes, this is helpful - and what I somehow missed when scanning the documentation. Sorry for that.

Might be helpful to add a link to the page about Updating Application Content.

christian

··· > On 23 Jun 2016, at 16:00, Jeremy W. wrote: > > HI Christian, > > You may also find this page useful: https://confluence.dimagi.com/display/commcarepublic/Making+Changes+to+a+Deployed+Application > > Jeremy > > On Thu, Jun 23, 2016 at 9:59 AM, Simon Kelly <skelly@dimagi.com > wrote: > CommCare will not tell you if you've made non-backwards compatible changes. One thing that may help somewhat with that is the "App Summary" (see image): > > > > This view will give you an outline of all forms and also cases. For each case it will list all the case properties as well as where each property is read from and written to. > > What it won't show is you is how the current version of the app differs from the previous version. > > Regarding #2, you can use the bulk-import feature to update case data but forms are considered 'read-only' so there is no way to bulk update them or to change their structure once they have been submitted. > > On 23 June 2016 at 15:40, Christian Neumann <christian.neumann@gmail.com > wrote: > Thanks Simon, > > Good (and important) to know. I didn’t come across it (so I assume it doesn’t exist), but to verify: Compare does not inform me whenever I have potentially non-backwards compatible changes in my forms definitions, or does it? I know it is impossible to predict this for every case, but some cases are: E.g. when I remove a question from a form where data was already captured. Or change the names of properties. Or add/remove entries from multiple choice questions. > > And regarding your suggestion #2 about exporting it to Excel. Do you refer to the bulk-import feature so bring back the converted data into CommCare? > > Thanks! > christian > >> On 23 Jun 2016, at 11:46, Simon Kelly <skelly@dimagi.com > wrote: >> >> Hi Christian >> >> This is a tricky problem and one that we don't have any automated tools to support. There are two possibilities that I can suggest: >> Make all changes to your app backward's compatible. >> e.g. Changing the name of a case property would be require you to load both the old and new version into a form and then determining which is correct e.g. coalesce(/data/new, data/old) >> You could make your data compatible after exporting it to Excel >> This would involve converting legacy data to new formats etc. >> >> >> On 16 June 2016 at 15:45, Christian Neumann <christian.neumann@gmail.com > wrote: >> Hi, >> >> I'm trying to figure out what I need to keep in mind once the first version of my CommCare application is in production. Of course ideally I could prevent introducing incompatible updates to my forms, but am not sure if I can always avoid this. One of my previous mails regarding the birthdate could be an example. I might come to the conclusion that instead of using an ordinary date field, I now want to break down the birthdate into separate day, month, and year fields. >> >> I've found this page (https://confluence.dimagi.com/display/commcarepublic/Updating+CommCare+Application+Content ). But unlike I've expected it doesn't include the updates of content. >> >> Are there other resources that I didn't find covering this topic? >> >> Thanks, >> christian >> >> -- >> You received this message because you are subscribed to the Google Groups "commcare-users" group. >> To unsubscribe from this group and stop receiving emails from it, send an email to commcare-users+unsubscribe@googlegroups.com . >> For more options, visit https://groups.google.com/d/optout . >> >> >> >> -- >> Simon Kelly >> Senior Engineer | Dimagi South Africa >> >> -- >> You received this message because you are subscribed to a topic in the Google Groups "commcare-users" group. >> To unsubscribe from this topic, visit https://groups.google.com/d/topic/commcare-users/OsU76WmXrP0/unsubscribe . >> To unsubscribe from this group and all its topics, send an email to commcare-users+unsubscribe@googlegroups.com . >> For more options, visit https://groups.google.com/d/optout . > > > -- > You received this message because you are subscribed to the Google Groups "commcare-users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to commcare-users+unsubscribe@googlegroups.com . > For more options, visit https://groups.google.com/d/optout . > > > > -- > Simon Kelly > Senior Engineer | Dimagi South Africa > > -- > You received this message because you are subscribed to the Google Groups "commcare-users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to commcare-users+unsubscribe@googlegroups.com . > For more options, visit https://groups.google.com/d/optout . > > > > -- > Jeremy Wacksman > Dimagi , Inc. > > > -- > You received this message because you are subscribed to a topic in the Google Groups "commcare-users" group. > To unsubscribe from this topic, visit https://groups.google.com/d/topic/commcare-users/OsU76WmXrP0/unsubscribe . > To unsubscribe from this group and all its topics, send an email to commcare-users+unsubscribe@googlegroups.com . > For more options, visit https://groups.google.com/d/optout .

Thanks, Christian - good idea, I just added a link.

Jeremy

··· On Thu, Jun 23, 2016 at 10:07 AM, Christian Neumann < christian.neumann@gmail.com> wrote:

Yes, this is helpful - and what I somehow missed when scanning the
documentation. Sorry for that.

Might be helpful to add a link to the page about Updating Application
Content.

christian

On 23 Jun 2016, at 16:00, Jeremy W. jwacksman@dimagi.com wrote:

HI Christian,

You may also find this page useful:
https://confluence.dimagi.com/display/commcarepublic/Making+Changes+to+a+Deployed+Application

Jeremy

On Thu, Jun 23, 2016 at 9:59 AM, Simon Kelly skelly@dimagi.com wrote:

CommCare will not tell you if you’ve made non-backwards compatible
changes. One thing that may help somewhat with that is the “App Summary”
(see image):

<image.png>

This view will give you an outline of all forms and also cases. For each
case it will list all the case properties as well as where each property is
read from and written to.

What it won’t show is you is how the current version of the app differs
from the previous version.

Regarding #2, you can use the bulk-import feature to update case data but
forms are considered ‘read-only’ so there is no way to bulk update them or
to change their structure once they have been submitted.

On 23 June 2016 at 15:40, Christian Neumann christian.neumann@gmail.com wrote:

Thanks Simon,

Good (and important) to know. I didn’t come across it (so I assume it
doesn’t exist), but to verify: Compare does not inform me whenever I have
potentially non-backwards compatible changes in my forms definitions, or
does it? I know it is impossible to predict this for every case, but some
cases are: E.g. when I remove a question from a form where data was already
captured. Or change the names of properties. Or add/remove entries from
multiple choice questions.

And regarding your suggestion #2 about exporting it to Excel. Do you
refer to the bulk-import feature so bring back the converted data into
CommCare?

Thanks!
christian

On 23 Jun 2016, at 11:46, Simon Kelly skelly@dimagi.com wrote:

Hi Christian

This is a tricky problem and one that we don’t have any automated tools
to support. There are two possibilities that I can suggest:

  1. Make all changes to your app backward’s compatible.
  2. e.g. Changing the name of a case property would be require you to
    load both the old and new version into a form and then determining which is
    correct e.g. coalesce(/data/new, data/old)
    2. You could make your data compatible after exporting it to Excel
    1. This would involve converting legacy data to new formats etc.

On 16 June 2016 at 15:45, Christian Neumann <christian.neumann@gmail.com wrote:

Hi,

I’m trying to figure out what I need to keep in mind once the first
version of my CommCare application is in production. Of course ideally I
could prevent introducing incompatible updates to my forms, but am not sure
if I can always avoid this. One of my previous mails regarding the
birthdate could be an example. I might come to the conclusion that instead
of using an ordinary date field, I now want to break down the birthdate
into separate day, month, and year fields.

I’ve found this page (
https://confluence.dimagi.com/display/commcarepublic/Updating+CommCare+Application+Content).
But unlike I’ve expected it doesn’t include the updates of content.

Are there other resources that I didn’t find covering this topic?

Thanks,
christian


You received this message because you are subscribed to the Google
Groups “commcare-users” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Simon Kelly
Senior Engineer | Dimagi South Africa


You received this message because you are subscribed to a topic in the
Google Groups “commcare-users” group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/commcare-users/OsU76WmXrP0/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google
Groups “commcare-users” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Simon Kelly
Senior Engineer | Dimagi South Africa


You received this message because you are subscribed to the Google Groups
"commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Jeremy Wacksman
Dimagi http://www.dimagi.com/, Inc.


You received this message because you are subscribed to a topic in the
Google Groups “commcare-users” group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/commcare-users/OsU76WmXrP0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups
"commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Jeremy Wacksman
Dimagi http://www.dimagi.com, Inc.