NNFEEDG

Synopsis

Description

The NNFEEDG command feeds news articles from the source NNTP server at from to the target NNTP server at to scanning the source server newsgroup by newsgroup and transferring articles not listed in its database of already transferred articles. Neither server need be local. NNFEEDG can be used to feed articles between any two arbitrary NNTP servers.

NNFEEDG uses a database of newsgroups, recording which articles it has already successfully transferred. This database comprises a series of files in a single directory, where the name of each file is the name of a newsgroup. Each file records, in machine-readable form, the article numbers of articles in the newsgroup that have been successfully transferred.

The database files are self-initialising. To add a new newsgroup to the database, or to clear the read article list of a newsgroup in order to force a re-scan of old articles, simply copy a zero-length file to the newsgroup file. To remove a newsgroup from the database permanently, delete the file. To remove a newsgroup from the database temporarily, move it to another directory.

NNFEEDG locates servers from the from and to strings supplied in the normal way, using nntp as the service name, tcp as the transport name, and 119 as the well-known port number.

groups is a search wildcard specification that comprises the directory containing the database of read article numbers (which, if omitted, will be the current directory) and a pattern for which newsgroups to scan. NNFEEDG iterates over all of the newsgroup files in the database, and for each one that matches the search wildcard specification attempts to select the group on the source server and scan it for articles not yet recorded in its database.

The set of newsgroups that are actually scanned, therefore, is the intersection of the set of newsgroups in the database and the set of newsgroups specified by the search wildcard specification in the arguments to NNFEEDG.

NNFEEDG updates each file after it has finished scanning its newsgroup. Therefore, if a temporary server error causes NNFEEDG to fail partway through, the next time that it is invoked it will quickly skip through newsgroups that it has already completed scanning. (Other NNTP news exchanger softwares use "news.rc" files as their database, and only update those files when the entire process has completed. A temporary server failure causes a restart from the beginning.)

NNFEEDG only opens one database file at a time, which it holds open as it scans the newsgroup and transfers articles. It opens the file with write access, denying write access to others, in order to prevent multiple instances of itself from attempting to update the same database file.

NNFEEDG relies upon no NNTP commands that are not listed in RFC 977. It requires the source server to support the GROUP command, the STAT command selecting by number, the ARTICLE command selecting by number, and the NEXT command. It requires the target server to support the STAT command selecting by Message-ID, the POST command, and the IHAVE command.

NNFEEDG uses the STAT command with an article number when it determines that it can skip one or more next articles because it has already transferred them. It also attemps to reduce the amount of time taken to scan a sparsely populated newsgroup by using the NEXT command to skip to the next valid article number in a newsgroup. However, the repudiation of prior protocol made by RFC 3977 limits the circumstances where this is possible. See the Bodges section for further details.

NNFEEDG may also send the SLAVE commands to the source and target servers, but does not require that either server support those commands.

NNFEEDG attempts to reduce the bandwidth that it uses. It does this by checking, with the STAT command, whether the target server already has an article before it retrieves the entire article header and body from the source server. (Other NNTP news exchanger softwares rely on the IHAVE command alone to determine whether the target server already has the article. The nature of IHAVE requires that they obtain the entire article headers and body from the source server first, even if the target server already has the article.)

In order to allow the passage of a news article to be traced, NNFEEDG prepends a Received: header to each article that it transfers, describing its receipt of the article from the source server. (It is presumed that the target server will add a header describing the receipt of the article in turn from NNFEEDG.)

To assist in tracing the passage of news articles NNFEEDG includes the message IDs of all articles that it transfers in its log output.

Bodges for broken servers and softwares

In some badly designed NNTP server softwares (e.g. Lyris ListManager) article numbers are globally unique across the whole database, rather than locally unique within a single newsgroup. On such servers, newsgroups are always sparsely populated. Sparsely populated newsgroups cause extra traffic as NNFEEDG attempts to retrieve messages for the many article numbers in each newsgroup that have never existed and will never exist in the newsgroup.

RFC 977 requires that the STAT command always update the current article pointer, even if an article does not exist. Some NNTP server softwares support this behaviour. With such softwares, it is possible to use the NEXT command to speed up the processing of sparsely populated newsgroups.

However, this is an aspect of RFC 977 that is repudiated by RFC 3977. By default, NNFEEGD does not rely upon it but instead will always explicitly STAT every article number that it would like, and only issue the NEXT command if the current article number is successfully updated by STAT. This results in a lot of extra STAT commands.

NNFEEDG can be configured to employ the old RFC 977 behaviour, if it is determined that the source NNTP server supports it, with the /BROKENSTAT option. NNFEEDG will set the current article pointer to the first article number that it wants with STAT, and use NEXT to find the number of the next actually available article that is greater than that number.

Some egregiously broken NNTP server softwares (e.g. Lyris ListManager) violate RFC 977 and do not support the ARTICLE command without a number. To cope with such softwares, NNFEEDG always explicitly supplies the number to the ARTICLE command, even though it only ever uses it to retrieve the article pointed to by the server's current article pointer.

Some badly designed NNTP server softwares complain and reject articles if they see Received: headers in articles, despite the clear injunction in RFC 1036 that any valid RFC 822 header should be allowed in an article. The stamping of articles with Received: headers must be turned off, with /ADDSTAMP-, if the target server is running such a software.

Example RUN file

This is a RUN file invoking NNFEEDG that is suitable to be scheduled by RUNWHEN from the OS/2 Command Line Utilities version 2.2. It feeds news from the IBM NNTP server to the local host's NNTP server, using a "read article" database that is unique name for this source and target pair, and selecting only the IBM VisualAge C/C++ compiler and tools newsgroups from that database.

  setenv SOURCE news.software.ibm.com.
  setenv TARGET localhost.
  setenv WILDCARD ibm.software.vacpp.{compiler,tools}
  program %APPS%\JdeBP\IU\bin\NNFeedG.exe
  argument NNFeedG
  argument %SOURCE%
  argument %TARGET%
  argument %TARGET%\%SOURCE%\Unread\%WILDCARD%

Example command-line invocation

This command feeds news from the Novell NNTP server to the local host's NNTP server, using a "read article" database in the current directory, and selecting all newsgroups from that database:

  [c:\]nnfeedg developer-forums.novell.com. localhost. *

Command-specific options

/V: Send the entire NNTP dialogue to the log.
/SKIPHEADER: Remove the named header from in-transit articles.
/PATHALIAS: Skip any messages where the given path occurs in the message's Path: header.
/BROKENSTAT: Take advantage of servers whose STAT command strictly obeys RFC 977.
/BACKFILL: Attempt to back fill any previously missing articles in sparsely populated newsgroups.
/ADDSTAMP: Add a Recieved: header to the top of each article transferred. This is enabled by default.
/FROMIP: If enabled, the source server is being specified by IP address rather than by name.
/TOIP: If enabled, the target server is being specified by IP address rather than by name.
/POST: If enabled, use the POST command rather than the IHAVE command.
/SLAVE: If enabled, send a SLAVE command to both source and target.
Note: This command has no effect on many modern NNTP server softwares.
/READER: If enabled, send a MODE READER command to both source and target.
Note: This command has no effect on many modern NNTP server softwares.