Hello all --
I am working with a developer that runs a web app that we send our commcare
data through via the Form Forwarding API.
For various reasons, sometimes the form forwarding gets a bit backed up...
and when the congestion ends, the result is that there are many forms
waiting in CommCare to be sent all in a bunch to him via form forwarding...
and they go rapidly.
Here's the question. Let's imagine that there are 150 forms in commcareHQ
queued up to be sent, and suddenly they are all able to be sent through...
----> Are all those 200 forms sent somewhat "concurrently," all in the
same immediate moment? OR are they sent sequentially? If your answer is
"sequentially," how rapidly are they sent, what's the time gap between
sends?
This will make it much easier for him to understand the capacity of the
server/database setup on his end, for handling these rapid incoming items.
And any other relevant knowledge on this topic and type of issue are also
appreciated!
I'm sure I'm not phrasing this gracefully.
Thanks!
Eric
Hello everyone -- Any input for me on this question about the API sending
method? Thanks!
Hi Eric
The form forwarding records are processed by a task queue with a
concurrency of 200. This may be increased in the future as we continue to
grow. We don't have any facility for rate limiting on data forwarding at
the moment.
Simon Kelly
Director of Server Engineer | Dimagi
···
On 16 June 2017 at 04:16, Eric Stephan wrote:
Hello everyone -- Any input for me on this question about the API sending
method? Thanks!
--
You received this message because you are subscribed to the Google Groups
"commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Hi Eric,
And to be clear on your other question: The forms are sent sequentially,
since CommCare's processing semantics are order specific and we anticipate
a repeater target will need to be as well.
-Clayton
···
On Mon, Jun 26, 2017 at 3:32 AM, Simon Kelly wrote:
Hi Eric
The form forwarding records are processed by a task queue with a
concurrency of 200. This may be increased in the future as we continue to
grow. We don't have any facility for rate limiting on data forwarding at
the moment.
Simon Kelly
Director of Server Engineer | Dimagi
On 16 June 2017 at 04:16, Eric Stephan estephan500@gmail.com wrote:
Hello everyone -- Any input for me on this question about the API sending
method? Thanks!
--
You received this message because you are subscribed to the Google Groups
"commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Hello to both of you --
This is good info! But I want to make sure I can translate this to my less
techie mind...
You say that:
1-- it is processed in a queue with a concurrency of 200 (which might go up
later)
2-- but they are sent sequentially
So... since I guess I don't know the full meaning of those two stats side
by side: My main reason for asking is that a developer on the recieving end
is having issues about his capacity to recieve rapid submissions... So is
it possible to tell him the timing between API transmissions he might get
(some kind of nanosecond or something) ... OR the number that might happen
in 60 seconds or so?
OR, if you put yourself in his place (he is upgrading his hardware) is
there any other measure or stat that might be useful for him?
Eric
Hi you guys -- Sorry, I am just a little unclear on the answers above:
1-- You said that, when the API forwards get queued up in a bunch, that
when they do get sent, the they are processed with a concurrency of 200, *BUT
*they are sent sequentially... I am confused about what the final result
is. Note that I am asking for a developer that makes the app/server that
will receive these API form forwards, and he has had problems with them
sometimes coming in all at once and thus overloading his side. So does your
reply mean that they are indeed sent sequentially, but super fast? If fast,
any measure that he can expect?
2-- Also, it's smart for me to make sure I understand what causes them to
get queued up in a bunch. (because, in a wonderful ideal world, they would
go right through the moment that the user completed the form, then they
might come slightly quickly sometimes, but very easily within the capacity
of the receiving server!) --> I believe that I know about two reasons that
API forwards get placed in the queue to get re-sent later and therefore get
bunched up. Can you tell me if I am forgetting any:
- (A) if a form forward fails for any reason, and it then gets stored
for its "second try" or "third try," that's one reason for the
queue/bunching
- (B) Once I think there was a technical congestion on commcare's side,
perhaps fairly rare, which caused items to queue for a few days and then
flushed out later when the problem was resolved?
- Any other thing that causes queueing (and therefore bunching up
resulting in super rapid succession)?
Hi Eric
Responses inline:
1-- You said that, when the API forwards get queued up in a bunch, that
when they do get sent, the they are processed with a concurrency of 200,
*BUT *they are sent sequentially... I am confused about what the final
result is. Note that I am asking for a developer that makes the app/server
that will receive these API form forwards, and he has had problems with
them sometimes coming in all at once and thus overloading his side. So does
your reply mean that they are indeed sent sequentially, but super fast? If
fast, any measure that he can expect?
All form forwarding records are queued in the order that they were received
but since the queue is processed by a system of parallel workers I think it
would be hard to guarantee ordering. However under normal circumstances
they should be received in order barring any retry attempts. The process
that reads from the queue and sends the forward requests can process 200
records 'concurrently'. This means that assuming there were at least 200
adjacent records in the queue which were to be sent to your system then
it's possible that you could get 200 simultaneous requests with records but
not more than that.
2-- Also, it's smart for me to make sure I understand what causes them to
get queued up in a bunch. (because, in a wonderful ideal world, they would
go right through the moment that the user completed the form, then they
might come slightly quickly sometimes, but very easily within the capacity
of the receiving server!) --> I believe that I know about two reasons that
API forwards get placed in the queue to get re-sent later and therefore get
bunched up. Can you tell me if I am forgetting any:
- (A) if a form forward fails for any reason, and it then gets stored
for its "second try" or "third try," that's one reason for the
queue/bunching
- (B) Once I think there was a technical congestion on commcare's
side, perhaps fairly rare, which caused items to queue for a few days and
then flushed out later when the problem was resolved?
- Any other thing that causes queueing (and therefore bunching up
resulting in super rapid succession)?
The most common reason for a backup in the queue is because of a process
error on our side which causes the queue not to be consumed. The other
possible reason is that records from other project space can 'block' the
queue if they take a long time to process (the receiving system is slow to
respond to the requests). This is also fairly rare.
I hope I've answered your questions. We do not have any rate limiting built
into the data forwarding beyond the '200 concurrency'.
···
> --
> You received this message because you are subscribed to the Google Groups
> "commcare-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to commcare-users+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>