|
FAST Protocol
< Previous Next >
Re: Fast SCP
Dimitry London / Morgan Stanley <>
7 Aug 2008 10:53PM ETThanks everyone for responses.
Like Rolf noted, I am looking for a template distribution mechanism with minimum amount of overhead. It is also important not to restrict certain areas of template to be changed (end of the group, sequence, etc) - in practice, it hard to predict what part of the template will change in future.
I like the idea of using template id/template name resolution, with a new template id requiring a corresponding template structure. However, it is also important not to tie the template distribution mechanism to a particular transmission protocol.
Is SCP the right mechanism for the actual template distribution in both UDP and TCP environments?
Thanks,
Dimitry
> Thank you David -- this is what I thought. If this is acceptable for
> receiver, then I see indeed no reasons to bother with virtual
> synchrony etc.
>
> > When the decoder sees a tid it hasn't seen before, it must resolve
> > that tid into a template name and further resolve that template name
> > into the actual template definition before it can continue.
> >
> > The exact steps involved in the resolution are dependent on how the
> > particular service is setup. In a pure multicast scenario I guess
> > you'd have a "template beacon" periodically scrolling through the
> > current set of templates. Then you'd have to suspend your main decoder
> > until the template decoder sees the requested template definition pass
> > by if it hasn't seen it already.
> >
> > In other setups you might have a TCP back channel or FTP or web site
> > where you can request new templates (think this was what Darshan
> > recommended).
> >
> > /David
> >
> > > Cheers for this.
> > >
> > > As long as there is no compatibility between template version (which
> > > boils down to Hanno's question), then I don't understand how what
> > > you describe works.
> > >
> > > Say template foo, version associated with tid=7 and version 2
> > > associated with tid=27, as per your example. If I receive a message
> > > with tid=27 and I have not build the corresponding codec, then I'm
> > > doomed. Unless I know that 27 the id map to the next incarnation for
> > > template_id=7
> > >
> > >
> > >
> > > > I was maybe a bit terse.
> > > >
> > > > By using immutable template ids (in the short term) you can relax
> > > > the requirements on synchrony between template definitions and
> > > > messages using template ids. As a receiver, you still need a new
> > > > template definition before you can decode a message with the new
> > > > template id, but the existing definitions don't change (within a
> > > > time window).
> > > >
> > > > This allows for a lower overhead template distribution solution,
> > > > which is what Dimitry was asking about, I think.
> > > >
> > > > An example; Let's assume you have a template named "foo/1" where 1
> > > > is the version of the template. Further let's assume the Template
> > > > Name "foo/1" is associated with the Id 7. Now, if you want to
> > > > modify this template (extend or change), you can create "foo/2"
> > > > which differs in some aspect from "foo/1" and, depending on other
> > > > Ids in use your first unused Id is 27, so you associate the name
> > > > "foo/2" with the id 27.
> > > >
> > > > Now receivers may see different representations of a message type.
> > > > Template id 7 will still refer to the old layout and id 27 will be
> > > > unknown until receivers see / retrieve the definition for id 27.
> > > > The important difference is that you can relax strict ordering as
> > > > id 7 is immutable for some period of time before it may be reused.
> > > > Not seeing the new definition of id 27 can be detected.
> > >
> > > > You still need a level of synchronization between templates and
> > > > messages in order to consume messages with newly created
> > > > templates, but you don't need a strict ordering; new definitions
> > > > can be sent out earlier as there is no (short-term) change to the
> > > > meaning of an existing template.
> > > >
> > > > There needs to be additional synchronization if several sources
> > > > are allowed to allocate template ids, but that is a separate
> > > > issue.
> > > >
> > > > /Rolf
> > > >
> > > > >> The remaining problem is then to make sure that a receiver
> > > > >> recognizes and acts on template id rollover. Implementors need
> > > > >> to make sure that senders receivers agree on the current
> > > > >> meaning of a template id (ie. the current association between a
> > > > >> template name and a template id). It is still an instance of
> > > > >> the reliable messaging problem, but with drastically lower
> > > > >> latency requirements.
> > > > >
> > > > > That's precisely the hard part: IMHO this is more than an
> > > > > instance of the Reliable messaging problem -- because it
> > > > > involves some ordering considerations. Given a sender and a
> > > > > possibly large number of receivers: no receiver should deliver a
> > > > > message prior to delivering its related template.
> > > > > e.g., ... [T_i] [M_i_1] ... [M_i_j] ... [T_(i+1)] [M_(i+1)_1]
> > > > > ....
> > > > >
> > > > > Not sure that's an instance of the distributed consensus, but
> > > > > rather instance of the virtual synchrony problem. That is,
> > > > > something (SCP) should guarantee that template changes within a
> > > > > group of receiver, are observed in the same order by all
> > > > > receivers. Furthermore, template changes in SCP must be totally
> > > > > ordered with respect to all regular FAST messages.
> > > > >
> > > > > BTW there is a typo in Fast SCP draft 1.1: page 15 section 7.3
> > > > > third item one should read "...defined in 6.1" and not "defined
> > > > > in
> > > > > 5.1"
> > > > >
> > > > > I couldn't find mention of this in the SCP spec.
Fast SCP Dimitry London / Morgan Stanley 30 Jul 2008 10:16PM ET Re: Fast SCP David Rosenborg / Pantor Engineering AB 31 Jul 2008 3:28AM ET Re: Fast SCP Rolf Andersson / Pantor Engineering 31 Jul 2008 6:17AM ET Re: Fast SCP Rolf Andersson / Pantor Engineering 31 Jul 2008 9:05AM ET Re: Fast SCP Hanno Klein / Deutsche Börse Systems 31 Jul 2008 10:06AM ET Re: Fast SCP Rolf Andersson / Pantor Engineering 31 Jul 2008 10:15AM ET Re: Fast SCP Greg Orsini / Cameron Systems 31 Jul 2008 11:44AM ET Re: Fast SCP Rolf Andersson / Pantor Engineering 1 Aug 2008 2:06AM ET Re: Fast SCP Rolf Andersson / Pantor Engineering 1 Aug 2008 2:39AM ET Re: Fast SCP Rolf Andersson / Pantor Engineering 1 Aug 2008 3:13AM ET Re: Fast SCP Hanno Klein / Deutsche Börse Systems 1 Aug 2008 3:33AM ET Re: Fast SCP Rolf Andersson / Pantor Engineering 1 Aug 2008 3:57AM ET Re: Fast SCP Rolf Andersson / Pantor Engineering 1 Aug 2008 4:10AM ET Re: Fast SCP Rolf Andersson / Pantor Engineering 1 Aug 2008 3:06AM ET Re: Fast SCP David Rosenborg / Pantor Engineering AB 1 Aug 2008 6:28AM ET Re: Fast SCP Jacob Northey / The LaSalle Technology Group 1 Aug 2008 8:21AM ET Re: Fast SCP David Rosenborg / Pantor Engineering AB 1 Aug 2008 9:33AM ET Re: Fast SCP Greg Orsini / Cameron Systems 1 Aug 2008 10:01AM ET Re: Fast SCP David Rosenborg / Pantor Engineering AB 1 Aug 2008 10:16AM ET Re: Fast SCP Greg Orsini / Orc Software w/CameronFIX 1 Aug 2008 10:32AM ET Re: Fast SCP David Rosenborg / Pantor Engineering AB 1 Aug 2008 10:58AM ET Re: Fast SCP Jean-Marie Sulmont / RTS 31 Jul 2008 10:19AM ET Re: Fast SCP Jean-Marie Sulmont / RTS 31 Jul 2008 10:41AM ET Re: Fast SCP David Rosenborg / Pantor Engineering AB 31 Jul 2008 11:10AM ET Re: Fast SCP
Dimitry London / Morgan Stanley 7 Aug 2008 10:53PM ET
|
