Recovering from gaps early in list orders etc.
Kevin Houstoun
5 Feb 1998 12:43PM ETWe've been testing our fix engine with large list orders. These messages by their nature are sent in large packs. We been testing with lists of about 150 orders. (These list are based on real programme trades, indeed we are replaying programmes done historically with counterparties). There are two ways of handling message processing in the fix protocol and these both have problems as I understand them.
Method 1
If you miss a message, ignore subsequent messages and request resends. In a situation where the message is one of the first in a large list order. This generates huges amounts of resend requests as can be seen from considering the following list of 5
Msg 2 - List item 1
Msg 3 - Lost for whatever reason
Msg 4 - List item 3, Discarded triggers resend request for 3 - 4
Msg 5 - List item 4, Discarded triggers resend request for 3 - 5
Msg 6 - List item 5, Discarded triggers resend request for 3 - 6
Msg 3 - List item 3 poss dup set in response to 4
Msg 4 - List item 4 poss dup set in response to 4
Msg 3 - List item 3 poss dup set in response to 5
Msg 4 - List item 4 poss dup set in response to 5
Msg 5 - List item 5 poss dup set in response to 5
Msg 3 - List item 3 poss dup set in response to 6
Msg 4 - List item 4 poss dup set in response to 6
Msg 5 - List item 5 poss dup set in response to 6
Msg 6 - List item 6 poss dup set in response to 6
The message traffic increases as a sum of series type thing and this is clearly unacceptable. Eg missing the first message in a list of 150 items causes the 150th message to be retransmitted 150 times. The 149th message to be transmitted 149 times etc. Total about 150 * 75 messages.
Some implementations used the latency of thier back ends to group responses to the resends together and this damps down the number of messages.
Method 2
Once you've missed your first message, store but don't process all further messages
Msg 2 - List item 1
Msg 3 - Lost for whatever reason
Msg 4 - List item 3, stored triggers resend request for 3 - 3
Msg 5 - List item 4, stored
Msg 6 - List item 5, stored
Msg 7 - List item 6, stored
Msg 8 - List item 3, poss dup set in response to 4, processes 3 - 7
Fine, least volume, probably quickest but the stored messages may not be ones that the counterparty would have resent (as allowed by the standard) thus causing a business error as two different fix implimentations would give two different results.
What we're currently trying
Once you've missed your first resend message ignore all further out of sequence messages until you've resynced.
Msg 2 - List item 1
Msg 3 - Lost for whatever reason
Msg 4 - List item 3, Discarded triggers resend request for 3 - 4
Msg 5 - List item 4, Discarded
Msg 6 - List item 5, Discarded
Msg 3 - List item 3 poss dup set in response to 4
Msg 4 - List item 3 poss dup set in response to 4
Msg 7 - Heartbeat, triggers resend request for 5 - 7
Msg 5 - List item 5
Msg 6 - List item 6
Msg 7 - GapFill
Fine but again the retransmit may be a whole list of 150 programs and relies on heart beat to trigger the resend. This is quite slow as the size of the programmes increase. Total a little over 300 messages. We also put a test message on the end, after a delay, to flush out the situation where both sides are stalled.
What have other people done about this, have I misunderstood something. Any thoughts.
Kevin J Houstoun
106333.2651@compuserve.com
