re: [PC20020211_1] Modification of encoding of binary fields
Geoff Kratz / Alberta Market Solutions Ltd.
22 Jan 2004 8:29PM ET
This change would mean that encoding of some values would differ from FIXML, which would also be a significant deviation from 4.4. today. I don't see this applying to FIXML, only to tag=value. XML parsers are generally not happy when non-character data appears in the XML data stream.
Right now I'm of two minds on this myself. On one hand, a change that makes parsing easier, separates the meaning of the content from the form, and potentially reduces message size, would be nice. On the other hand, we're talking about a new format, and if the real goal is to ultimately move to FIXML (which I believe it should be), then this may not be worth the effort.
--
Geoff Kratz
Director
Alberta Market Solutions Ltd.
> But doesn't encoding numeric data violate one of the foundation guidelines of XML of having human-readable data?
>
> > Actually, in any case we are talking about changes to parsers, so if we're going to change the protocol, maybe its time to make it more self-describing, and possibly more efficient. This goes beyond binary data fields, and could include more efficient encoding of numeric data (using agreed upon formats to represent numbers in binary form rather than strings and avoid a cumbersome string to number conversion, etc), and providing information so that parsers could know what type a field is without having to look up the tag somewhere. This could allow the parsers to be more independent of the "meaning" of the tags, and might simplify some things. It could allow parsers to handle tags they don't know about in a more elegant fashion. Going this far would be a pretty radical change though, so I'm not sure if that would work well for the community.
> >
> > > The real issue here is one of in-band signalling vs. out-of-band signalling. In-band signalling is risky because the data can mimic the signal and the two can be confused.
> > >
> > > As was mentioned, Tag=Length[SOH]Value[SOH] isn't useable because Value could begin with a number and = thus being confused for a new Tag=Value pair.
> > >
> > > Similarly, Tag=Length[US]Value[SOH] isn't useable. At present, FIX makes *NO* restrictions on character set except that normal fields can't contain an [SOH]. [US] is a valid character in a FIX field. So if someone sends a normal Text field containing it, i.e.
> > >
> > > ...58=1000[US]Foo[SOH]10=123[SOH]8=FIX.4.5[SOH]9=100[SOH]35=...
> > >
> > > then a FIX parser will think that a 1000 byte binary field follows, and consider the checksum and next message(s) part of that binary field.
> > >
> > > Out-of-band signalling, i.e. keeping the length out of the data field itself, eliminates this problem, such as:
> > >
> > > Tag+Length=Value[SOH] or
> > > Tag:Length=Value[SOH]
> > >
> > > are completely unambiguous in the receiving engine. While I proposed the first initially, I'm thinking the second might be better because some library functions might consider the + as a sign and part of the number.
> > >
> > > Further, backwards compatibility with existing parsers is not an issue. We're proposing this for FIX 4.5 (or whatever the next FIX version is called) and later. FIX engines generally have to be changed to support new FIX versions anyway, so this would be part of all 4.5 engines, and could be made invisible to the business logic.
> > >
> > >
> >
>
@FIXProtocol
