X-Git-Url: http://scm.dxcluster.org/gitweb/gitweb.cgi?a=blobdiff_plain;ds=sidebyside;f=techdoc%2Fprotocol.pod;h=3c3ccdcc4af5bb5539d17b4600ecd5851a4380ef;hb=2a4700c098adc015869d76ab959ffc47e27c6276;hp=de523c9df3bf6bfcbd883ff73c5e5703bacba5de;hpb=1a3106f748d8123a2b88572227f18147019c61c5;p=spider.git
diff --git a/techdoc/protocol.pod b/techdoc/protocol.pod
index de523c9d..3c3ccdcc 100644
--- a/techdoc/protocol.pod
+++ b/techdoc/protocol.pod
@@ -1,6 +1,8 @@
=head1 NAME
-DXSpiderWeb Orthogonal Communications Protocol
+Aranea Orthogonal Communications Protocol
+
+$Revision$
=head1 SYNOPSIS
@@ -9,9 +11,18 @@ DXSpiderWeb Orthogonal Communications Protocol
=head1 ABSTRACT
For many years DX Clusters have used a protocol which was designed
-for a non-looped tree of nodes. This has probably never, reliably,
-been achieved in practice; certainly not recently. This document
-describes a complete replacement for that protocol. It allows a
+for a non-looped tree of nodes. This environment has probably never, reliably,
+been achieved in practice; certainly not recently.
+
+There have always been loops, sometimes bringing the network to its
+knees. In modern usage, both in order to get some resilience and also
+to expedite information flow, we use internet based, deliberately
+looped networks with filtering. Whilst this works, after a fashion, there
+are all sorts of problems that the current PC protocol can never
+address.
+
+This document
+describes a complete replacement for the PC protocol. It allows a
fully looped network, is inherently extensible and should be simple
to implement (especially in perl).
@@ -20,21 +31,78 @@ for inter-node communications.
=head1 DESCRIPTION
-This protocol is encoded in UTF8 with HTTP style escaping. It is
-designed to be an extensible basis for any type of one to many
+This protocol is
+designed to be an extensible basis for any type of one too many
"instant" line-based communications tasks.
This protocol is designed to be flood routed in a meshed network in
-as efficient a manner as possible.
+as efficient a manner as possible. The reason we have chosen this
+mechanism is that most L need to be broadcast to all nodes.
-The protocol consists of a L and a L.
-The two sections are separated with the '|' character.
+Experience has shown that nodes will appear and (more infrequently)
+disappear without much (or any) notice.
+Therefore, the constantly changing and uncoordinated
+nature of the network doesn't lend itself to fixed routing policies.
+
+Having said that: directed routing is available where routes have
+been learned through past traffic.
+Those L that could be routed (mainly single line one to
+one "talk" L)
+happen sufficiently infrequently that, should they need to be flood routed
+(because no route has been learned yet) it is a small cost overall.
+
+=head1 Messages
-Most of this document is concerned with the L, however
-some L which all implementation should issue and
+A message is a single line of UTF8 encoded and HTTP escaped text
+terminated in the standard internet manner with a .
+
+Each message consists of a L and a L.
+The two sections are separated with the '|' character.
+It follows that these
+characters (as well as non-printable characters, , and
+a small number of other reserved characters)
+can only be sent escaped. This is described further in the
+L and L.
+
+Most of this document is concerned with the L, however
+some L which all implementation should issue and
must accept are described.
-=head2 Routing Section
+=head1 Applications
+
+In the past messaging applications such as DX Cluster software have maintained
+a fairly strict division between "nodes" and "users". This protocol attempts
+to get away from that distinction by allowing any entity to connect to any
+other.
+
+Applications that use this protocol are essentially all peers and therefore
+nodes the only real difference between a "node" and a "user" (using this
+protocol) is that a "node" has one or more listeners running that will,
+potentially, allow incoming connections. A "user" simply becomes an end
+point that never uses the L or L slots in the
+L.
+
+The reason for this is that modern clients are more intelligent than simple
+character based connections such as telnet or ax25. They wish to be able to
+distinguish between the various classes of message, such as: DX spots,
+announces, talk, logging info etc. It is a pain to have to do it, as now,
+by trying to make sense of the (slightly different for each piece of node
+software) human readable "user" version of the output. Far better to pass on
+regular, specified, easily computer decodable versions of the message,
+i.e. in this protocol, and leave
+the human presentation to the client software.
+
+Having said that, the protocol allows for traditional, character based,
+connections, as in the past. But it is up to applications
+to service and control that type of connection and to provide human readable
+"user" output.
+
+One of the legacy, character based connections that will probably have to be
+serviced is that of existing PC protocol based nodes. They should be treated
+as local clients, B as peers in this protocol. It is likely that, in order
+to do this, some extra Ls will need to be defined at application level.
+
+=head1 Routing Section
The application that implements this protocol is essentially a line
oriented message router. One line equals one message. Each line is
@@ -42,8 +110,8 @@ effectively a datagram.
It is assumed that nodes are connected to
each other using a "reliable" streaming protocol such as TCP/IP or
-AX25. Having said that: in context, elements of the protocol could be
-multi or broadcast, either "as is" or wrapped in some other framing
+AX25. Having said that: in context, L in this protocol could be
+multi/broadcast, either "as is" or wrapped in some other framing
protocol.
Because this is an unreliable, best effort, "please route my packets
@@ -51,22 +119,22 @@ through your node" protocol, there is no guarantee that a message
will get to the other side of a mesh of nodes. There may be a
discontinuity either caused by outage or deliberate filtering.
-However, as it is envisaged that most messages will be flood routed or,
-in the case of directed messages (those that have a EtonodeE or
-EtouserE) down all interfaces showing a route for that
-direction, it is unlikely that messages will be lost in practice.
+However, as it is envisaged that most L will be flood routed or,
+in the case of directed L (those that have L and/or
+L fields) down some/most/all interfaces showing a route for that
+direction, it is unlikely that L will be lost in practice.
-=head3 Field Description
+=head2 Field Description
-Only the first three fields in the L are compulsory
+Only the first three fields in the L are compulsory
and indicate that this is a broadcast to be sent to all nodes coming
-from the L. If the message needs to be identified as coming
-from a user on a node, then the L field is added.
+from the L. If the message needs to be identified as coming
+from a user on a node, then the L field is added.
-Adding a L and/or L field will restrict the destinations
+Adding a L and/or L field will restrict the destinations
or recipients that receive this message.
-The L field is incremented on receipt of a message on a node.
+The L field is incremented on receipt of a message on a node.
Fields are separated by the comma ',' character with the last field
required followed by the vertical bar '|' character.
@@ -83,7 +151,7 @@ More detailed descriptions of the fields follow:
=over
-=item Origin
+=item B
This is a compulsory field. It is the name of the originating node.
The field can contain up to 12 characters in the set [-A-Z0-9_] in
@@ -91,16 +159,18 @@ any order. Higher layers may restrict this further.
The field must not be changed by any other node.
-=item TimeSeq
+=item B
This is a compulsory field. It is a 10 hexadecimal digit string which
-consists of a day no (1-31), seconds within that day (0-86399) [6
-hex digits] that are concatenated with a sequence number (0-65535)
-[4 hex digits] making the total of 10.
+consists of a day no (1-31),
+a flag to indicate NTP syncronisation in use,
+seconds within that day (0-86399) [total of 6 hex digits]
+that are concatenated with a sequence number (0-65535)
+[4 hex digits] making the total of 10 hexadecimal digits.
The date portion is constructed as:
- my $date = ((gmtime)[3] << 18) | (time % 86400);
+ my $date = ((((gmtime)[3] < 1) | $ntpflag) < 18) | (time % 86400);
The sequence number is simply an unsigned short (or 16 bit) number
starting at 0.
@@ -108,21 +178,21 @@ starting at 0.
Each message originated at this node will increment the sequence
number.
-=item Hop
+=item B
This is a compulsory field. It is the number of hops from the
originating node. It is incremented immediately on receipt and
before determining its value.
-So the originating node sends a message with a L of 0, the
+So the originating node sends a message with a L of 0, the
neighbouring nodes must increment this field before passing
it on to higher layers for onward processing.
Implementations may have an upper limit to this field and may
-silently drop incoming messages with a L count greater than the
+silently drop incoming L with a L count greater than the
limit.
-=item FrmUser
+=item B
This field is optional. It is the identifier of the originating
user. If it is missing then the message is
@@ -131,19 +201,19 @@ assumed to come from the originating node itself.
It can consist of up to 12 characters in the set [-A-Z0-9_]
in any order. Higher layers may restrict this further.
-=item To
+=item B
This field is optional. It is a string of up to 12 characters
in the set [-A-Z0-9_] in any order.
This field is used either to indicate particular node destination
or to differentiate this broadcast in some way by making this
-message as a member of a L. Any message can be sent
-down any L. The names of Ls and their usage
+message as a member of a L. Any message can be sent
+down any L. The names of Ls and their usage
is entirely up to the implementor.
It is assumed that node names can be differentiated from user
-names and L names.
+names and L names.
If the field is set to a particular node destination, it will
be routed (rather than broadcast) to that node. However, any
@@ -151,48 +221,48 @@ intervening nodes are free to duplicate the message and send
it down more than one, likely looking, interface - depending on any
network policies that may pertain.
-=item ToUser
+=item B
This field is optional. It is a string of up to 12 characters
in the set [-A-Z0-9_] in any order. Higher layers may restrict
this further.
Conventionally this field is used to indicate the user to whom
-this message is directed. In an ideal world the L field
+this message is directed. In an ideal world the L field
will be set, by the originating node, to the identifier of the node
on which this user resides.
-If the L field is not set then this message will be
+If the L field is not set then this message will be
broadcast. However, should a node become apparent (on route)
-then nodes are free to fill in the L field and proceed
+then nodes are free to fill in the L field and proceed
with a more directed approach.
If it becomes apparent (on route) that there may be more than
-one possible L destination for a L then a node
-may duplicate the message (keeping the same L) and
-route it onwards. Because of the L inherent in
+one possible L destination for a L then a node
+may duplicate the message (keeping the same L) and
+route it onwards. Because of the L inherent in
the system, it is indeterminate as to which destination will
receive the message. It is possible for all or just some
-destinations to receive the message. The tuple (L,
-L) will determine uniqueness.
+destinations to receive the message. The tuple (L,
+L) will determine uniqueness.
-This field can, in the case where L
-is set to the name of a node, be set to a L. If this
+This field can, in the case where L
+is set to the name of a node, be set to a L. If this
is the case then this will cause this message to be sent to
-a L on the L node only.
-
+a L on the L node only.
+
=back
-=head3 Channel
+=head2 Channel
Channels are a concept very similar to that on IRC. It is a
way of segregating data flows in a network. In principle, subject
to local policy or application requirements, any data (or
-L) can be sent down any channel.
+L) can be sent down any channel.
It is up to the implementation whether to use this feature or not.
-=head3 Routing
+=head2 Routing
It is assumed that nodes will be connected in a looped network with
more than one route available (in many cases) to another node.
@@ -200,12 +270,12 @@ more than one route available (in many cases) to another node.
In anycase, most traffic is not directed, but broadcast to all users
on all nodes.
-Each message is uniquely identified by the (L,L)
+Each message is uniquely identified by the (L,L)
tuple. The basic system will learn which interfaces can see what nodes
-by looking at the tuple and merging that with the L count.
-Each interface remembers the latest L with the lowest L
-for each L that arrives on that interface. It also remembers
-the number of messages for that L that has been received on
+by looking at the tuple and merging that with the L count.
+Each interface remembers the latest L with the lowest L
+for each L that arrives on that interface. It also remembers
+the number of L for that L that has been received on
that interface.
Any message for onward broadcast is duplicated and sent out on all
@@ -216,10 +286,10 @@ the "best" interface based on routing information gathered so far. If there
is more than one possible route then, depending on network or local
policy, the message may be duplicated and sent on other interfaces
as well.
-
-=head3 DeDuplication
-On receipt of a message, its unique tuple (L,L) is
+=head2 DeDuplication
+
+On receipt of a message, its unique tuple (L,L) is
checked against a hash table. If it exists: the message is silently
dropped. If it does not exist in the hash table then the tuple is
added.
@@ -232,18 +302,53 @@ required.
This mechanism only ensures that a message broadcast around the network
travels the least distance and through the fewest nodes possible. It
is up to higher layers to make sure that data carried is not, itself,
-duplicated!
-
-=head2 Command Section
+duplicated!
+
+=head2 Examples
+
+ # on link startup from GB7BAA (both sides hello)
+ GB7TLH,3D02350001,0|HELLO,Aranea,1.2,24.123
+ GB7BAA,3D02355421,1|HELLO,Aranea,1.1,23.245
+
+ # on user startup to GB7TLH
+ GB7TLH,3D042506F2,0,G1TLH|HELLO,PClient,1.3
+
+ # on user disconnection
+ GB7TLH,3D9534F32D,0,G1TLH|BYE
+
+ # a talk (actually 'text') message to a user (some distance away
+ # from the origin node)
+ GB7TLH,3D03450019,3,G1TLH,GB7BAA,G8TIC|T,Hiya Mike what's happening?
+
+ # a talk/chat/text message to a channel or group
+ GB7TLH,0413525F23,2,G1TLH,VHF|T,2m is opening on MS
+
+ # a ping to find the whereabouts and distance of a user from a node
+ # the hex number on the end is the ping ID
+ GB7TLH,1512346543,0,,,G7BRN|PING,9F4D
-The Command Section of the message contains the actual data being
+ # the same from a user on GB7TLH
+ GB7TLH,1512346543,0,G1TLH,,G7BRN|PING,23
+
+ # this effectively asks whether the user is on-line on a particular node
+ GB7TLH,1512346543,0,G1TLH,GB7DJK,G7BRN|PING,35DE
+
+ # A possible reply, same ID as ping followed by the no of hops on the
+ # ping that was received
+ GB7DJK,1512450534,3,G7BRN,GB7TLH,G1TLH|PONG,35DE,3
+
+
+=head1 Command Section
+
+The L of the message contains the actual data being
passed. It is called the Command Section because all commands
-are identified with a L which is implemented by
-the software using this protocol.
+are identified with a L each of which is implemented by
+the software using this protocol. Each (usually) is followed by one
+or more L.
-=head3 Command Tag
+=head2 Tag
-The Command Tag consists of string of uppercase letters and digits, starting
+The L consists of string of uppercase letters and digits, starting
with a leading, uppercase, letter. Tags should be as short as is meaningful.
Valid tags would be:
@@ -258,9 +363,118 @@ Invalid tags include:
dx
Ann
-There are a number of standard commands which must be accepted by
+The L is separated from its data L by a comma ','.
+
+=head2 Fields
+
+All fields
+in any subsequent data shall be separated by a comma ','.
+All fields shall
+be HTTP encoded such that reserved characters (comma ',',
+vertical bar '|',
+percent '%',
+equals '='
+and non printable characters less than 127 (or %7F in hex)
+[including newline and carraige return] are translated to
+their two hex digit equivalent preceeded by the percent '%' character.
+
+For example:
+
+ "%0D%0A" is "".
+ "hello%2C there" is "hello, there"
+
+This is not standard CSV, fields are not quoted (delimited with either
+' or ").
+
+All national characters above 127 are UTF8 encoded in the
+standard perl 5.8.x way. It follows that all (perl) programs that
+are written according to this specification must say:
+
+ use UTF8;
+
+A message (or line) is terminated with
+0x0d 0x0a. Incoming L must be accepted even when terminated
+with just .
+
+Care must be taken to make sure that fields have any reserved characters
+encoded. In particular: it is perfectly permissible to have
+characters in a field - so long as they are escaped.
+
+Fields come in two styles: either simple fields (just containing
+data) or B=B pairs. Each pair must be separated from
+the next by a comma ','. The B must consist of the set of
+characters [a-z0-9_] (ie lowercase letters, digits and underscore),
+with a leading letter. The B must be HTTP encoded as
+specified above and can otherwise contain any character.
+
+There is no maximum size specified for a message. It is up to each
+implimentation to enforce one (if only for their own protection).
+
+=head2 Standard Commands
+
+There are a number of L which must be accepted by
all implementations.
+=over
+
+=item B
+
+ HELLO,,,,
+
+Command sent on connection to another node. Both sides send their information
+to the other. All the possible arguments are optional, although some of the
+arguments should be sent in order to help diagnose problems. This command is
+broadcast.
+
+=item B
+
+ BYE,
+
+Command sent to all connections when the software is shutting down. This is sent
+by the node just before shutdown occurs. This is really only used to help the
+network prune its routing tables. It isn't a requirement. The field
+is optional.
+
+=item B
+
+ DISC,,
+
+Command sent when a node has disconnected from this node. This message is sent when
+an interface shuts down. It need not be sent if a L from an interface for
+that node has just been received. This command should be broadcast.
+
+The is mandatory and is the name of the interface that has just
+disconnected.
+
+=item B
+
+ PING,
+
+Command to send a ping to a node or user. This command is used both by the software
+and users to determine a) whether a node or user exists and b) how good the path is
+between them.
+
+The is a unique string which is usually the hexadecimal equivalent of an
+integer that is incremented every time it is used. But it can be anything that
+will identify this ping using the tuple (L,) as unique.
+
+=item B
+
+ PONG,,
+
+Command to reply to a ping. This is sent as a reply to an incoming ping command.
+The is the one supplied and the is the number of
+hops it took for the ping to arrive.
+
+=item B
+
+ T,
+
+All implementations must be able to send "text" (encoded as specified in
+L). There would be little point in doing all this otherwise!
+
+=back
+
=head1 AUTHOR
Dirk Koopman, G1TLH, Edjk@tobit.co.ukE
@@ -272,6 +486,8 @@ Copyright 2004 by Dirk Koopman, G1TLH
This library is free software; you can redistribute it and/or modify
it under the same terms as Perl itself.
+$Revision$
+
=cut