|
FAST Protocol
< Previous Next >
Re: Fast SCP
Jacob Northey / The LaSalle Technology Group
1 Aug 2008 8:21AM ETI think the easiest approach is to create a FAST best practices guide that includes instructions for achieving support for multiple message template versions at the same time. Summarizing what has been said before, this would require:
1) Length prefixed blocks
2) Fields cannot be added to sequences
3) Fields can only be added to a group if the group is the last field instruction in a template and the added fields do not affect the groups use of a presence map (e.g. a group that does not need a presence map can only have fields added that don't require entries in the presence map).
4) New fields cannot use operators that would affect shared dictionary values.
Jake
> I don't think we should label such a device inheritance and I think it
> should be sliced in another way if we in the end decide that the added
> complexity is worth the effort.
>
> First, let's summarize a few scenarios:
>
> 1) Fully Dynamic - The most generic way of solving template evolution is
> to use non mutating TID and SCP Level 3 (i.e, dynamic template
> exchange). The base for this is already defined, but additional work is
> required for multicast applications.
>
> 2) Non Mutating TID and Accessible Extension - You want to add new data
> without modifying or sending out a new template. You also want all
> receivers to be able to discover and use the new data. To solve this
> you'd have to tack on an explicit extension field. It could look
> something like
>
>
>
>
> The benefits of this soultion is that it is already available. The
> downside is that it is not particular flexible, it is intrusive, and it
> requires explicit processing at the application level.
>
> 3) Mutating TID and Opaque Extension - You have different versions of
> the same template (same identifier). The versions are compatible.
> Receivers only knowing of V1 will simply ignore content available to
> those receivers that use V2.
>
> Scenario 3 is what I beleive you tried to solve with your inheritance
> proposal. This is an outline of how I'd do it:
>
> Add a new template construct named 'extension':
>
> > name="OrdinaryField">
>
> > name="OrdinaryField"> > name="ExtensionField">
>
>
> > name="OrdinaryField"> > name="ExtensionField" presence="optional"> > presence="optional">
>
>
> The semantics of this would be that messages of all these three
> templates would be fully interchangable (with a few constraints, see
> below). That is, the templates are both backward and forward compatible.
> V1 - V3 decoders can decode any message encoded with V1 - V3.
>
> The fine print:
>
> The wire representation would need a new predefined template identifier
> in the reserved range. Let's use 121 for now. The definition of 121
> would look something like this.
>
> > ns="http://www.fixprotocol.org/ns/fast"
> templateNs="http://www.fixprotocol.org/ns/fast"> > name="Extension"/>
>
> The predefined template is assumed to be known in advanced by any
> decoder supporting the feature.
>
> When an encoder encodes an section, it wraps it in an
> Extension message:
>
> PMAP TID=121 LEN [... encoded extension section ...]
>
> If fields in the extension section must utlilize bits in a presence map,
> the encoded extension section starts with PMAP.
>
> Encoding backward compatibility: Let's say a V1 decoder receives an
> message encoded with V2. The decoder first reads the V1 fields (all it
> knows). The next thing in the stream is the extension secition, which
> starts with a PMAP and the tid 121. It recognizes this as an Extension
> message and just skips it.
>
> Encoding forward compatibility: Let's say a V2 decoder receives an
> message encoded with V1. The decoder first read the non-extension
> part. It then looks for the next message in the stream. Since this
> message is not the 121 - Extension message, it knows the extension
> section is not present. The additional constraint here is that the
> fields in the extension section must be optional in the corresponding
> application type.
>
> For this to work in general, there are few constraints that must be met:
>
> * elements may only appear as the last components of a top
> level template (after resolving any static template refereces).
>
> * Any field operator in an extension section must either specify no
> dictionary or must specify the special dictionary "extension". An
> extension dictionary is local to a template and a specific extension
> section. This way we avoid that operators in extension sections
> pollute dictionaries seen by ordinary fields (otherwise it would be
> dangerous to skip 121 messages).
>
> * The 121 Extension messsage must not be used in any other situation
> than as a result of encoding a template with an section.
> If this constraint is not met, the forward compatibility feature
> won't work.
>
> * Dynamic template references are not allowed in an extension section.
> There are two reasons: They'd pollute the previous value of the
> implicit copy operator for TIDs, and the actual contained templates
> could possible pollute arbitrary dictionaries. The dictionary
> constraint described above should be possible to enforce by static
> examination of a template.
>
> This solution might look complex at first but wouldn't be particulary
> hard to implement. The simplest level of support would be for a decoder
> to recognize the 121 tid and know how to skip that message.
>
> /David
>
> > May be we could create a special relation between templates -- like
> > "Russian nesting dolls", or template inheritance.
> >
> > If template T3 <-- T2 <-- T1 then if message M3 has been T3-encoded,
> > it then can be decoded by T2 and or T1.
> >
> > What you get when you decode M3 following T2 or T1 is not specified.
> >
> > That would solve Hanno's problem. That would also allow the notion of
> > template specialization etc.
> >
> > What do you think???
> >
> > > Hanno,
> > >
> > > There is no way to know how long the message is without the correct
> > > template, so a new message with an unknown or 'extended' template
> > > can't be fully decoded (a result of the decision to not include a
> > > message length in FAST or SCP).
> > >
> > > Perhaps an optional length for messages requiring a new template
> > > would maintain the integrity of the transport layer without
> > > affecting the message contents.
> > >
> > > Then you could create a revised template (instead of 7 it would
> > > 7.1?) that could have additional fields that would be ignored until
> > > you get a hold of the new template.
> > >
> > > Greg.
> > >
> > > > Hanno, this is an interesting problem.
> > > >
> > > > The short answer is no. There is no way for the receiver to detect
> > > > that there is another field in the message. As you mention, you
> > > > could put a placeholder into the template, maybe an optional field
> > > > at the end of the template. This would maybe work in some
> > > > instances, but you would be restricted in choice of operator etc.
> > > >
> > > > We did discuss this some when we were updating the spec from
> > > > 1.0 to
> > > > 1.1, but nothing came out of that discussion if I recall
> > > > correctly.
> > > >
> > > > It's worth discussing again, though. Anyone has ideas on this?
> > > >
> > > > /Rolf
> > > >
> > > > > Rolf,
> > > > >
> > > > > I have a related question. Is there a way to send a new message
> > > > > without having sent a new template definition, i.e. if I add a
> > > > > field to my message, is there a way for the receiver to ignore
> > > > > that new field until he receives a new template definition from
> > > > > me, perhaps by appending it to the end of my message and having
> > > > > a placeholder in my initial template definition? Or is it a
> > > > > strict requirement to always send a new template definition
> > > > > prior to sending a message for it.
> > > > >
> > > > > Thanks, Hanno.
Fast SCP Dimitry London / Morgan Stanley 30 Jul 2008 10:16PM ET Re: Fast SCP David Rosenborg / Pantor Engineering AB 31 Jul 2008 3:28AM ET Re: Fast SCP Rolf Andersson / Pantor Engineering 31 Jul 2008 6:17AM ET Re: Fast SCP Rolf Andersson / Pantor Engineering 31 Jul 2008 9:05AM ET Re: Fast SCP Hanno Klein / Deutsche Börse Systems 31 Jul 2008 10:06AM ET Re: Fast SCP Rolf Andersson / Pantor Engineering 31 Jul 2008 10:15AM ET Re: Fast SCP Greg Orsini / Cameron Systems 31 Jul 2008 11:44AM ET Re: Fast SCP Rolf Andersson / Pantor Engineering 1 Aug 2008 2:06AM ET Re: Fast SCP Rolf Andersson / Pantor Engineering 1 Aug 2008 2:39AM ET Re: Fast SCP Rolf Andersson / Pantor Engineering 1 Aug 2008 3:13AM ET Re: Fast SCP Hanno Klein / Deutsche Börse Systems 1 Aug 2008 3:33AM ET Re: Fast SCP Rolf Andersson / Pantor Engineering 1 Aug 2008 3:57AM ET Re: Fast SCP Rolf Andersson / Pantor Engineering 1 Aug 2008 4:10AM ET Re: Fast SCP Rolf Andersson / Pantor Engineering 1 Aug 2008 3:06AM ET Re: Fast SCP David Rosenborg / Pantor Engineering AB 1 Aug 2008 6:28AM ET Re: Fast SCP
Jacob Northey / The LaSalle Technology Group 1 Aug 2008 8:21AM ET Re: Fast SCP David Rosenborg / Pantor Engineering AB 1 Aug 2008 9:33AM ET Re: Fast SCP Greg Orsini / Cameron Systems 1 Aug 2008 10:01AM ET Re: Fast SCP David Rosenborg / Pantor Engineering AB 1 Aug 2008 10:16AM ET Re: Fast SCP Greg Orsini / Orc Software w/CameronFIX 1 Aug 2008 10:32AM ET Re: Fast SCP David Rosenborg / Pantor Engineering AB 1 Aug 2008 10:58AM ET Re: Fast SCP Jean-Marie Sulmont / RTS 31 Jul 2008 10:19AM ET Re: Fast SCP Jean-Marie Sulmont / RTS 31 Jul 2008 10:41AM ET Re: Fast SCP David Rosenborg / Pantor Engineering AB 31 Jul 2008 11:10AM ET Re: Fast SCP Dimitry London / Morgan Stanley 7 Aug 2008 10:53PM ET
|
