You can download Leafnode from http://www.leafnode.org/. Click the Download link and click the link to download the latest tar.gz or tar.bz2 file. When you've come offline, change to the directory where you saved the file. If you downloaded the tar.gz file, enter 'tar zxvf leafnode-1.9.22.tar.gz' (but change the filename you enter if it's a different version). If you downloaded the .tar.bz2 file, enter 'tar yxvf leafnode-1.9.22.tar.gz' (the 'y' option works for .bz2 files on my Mandrake system; if you're using a different distro, you might need to use a different option - do 'man tar' to find out which). This will create a subdirectory with the same name as the package you downloaded, but without the '.tar.gz' or '.tar.bz2' (although in some versions the directory might have a '.rel' on the end of the name).

Change to that directory and read any README.FIRST file if there is one, then read the README file. It will tell you a fair bit about Leafnode and how it works. The instructions here on this web page are meant as an adjunct to the text in the README file, and to simplify things as much as possible, so you only need to read as much of the README file as you want (or understand). If you're already using Leafnode, you'll need to take note of any upgrading instructions in the README file. You might also want to read the INSTALL file, just in case it contains any instructions that are not covered in this web page.

Enter 'sh ./configure' on the command line (when you see instructions like this, don't enter the quotes, just the text between them, and press Enter when you've finished typing). This will configure the program to run with certain defaults. For example, the executable programs will be placed into /usr/local/sbin. If you're already running Leafnode and the executables are somewhere else - /opt, for example - you can enter 'sh ./configure --prefix=opt' to overwrite the existing files. Now enter 'make' to compile the various programs that make up the Leafnode package.

You now need to be root to do the next part, so su to root (enter 'su' then root's password when asked). You need to have a user called 'news', so enter 'grep news /etc/group' and see if you get something like news:x:13:news. If not, you'll need to make a group called news. The simplest way is to enter 'groupadd news'. Now you can install the various parts of Leafnode by entering 'make install'. If the INSTALL file instructs you to (for example, as when upgrading from versions older than 1.9.3), enter 'make upgrade'.

Now you need to configure Leafnode so that it knows where your news server is, amongst other things. As root, fire up your favourite editor and load /etc/leafnode/config. Have a look through the file to get an idea of what goes where. Enter the name of your news server, together with your username and password. This section of the config file (at the top) will look something like this, but with your username rather than mine:

server = news.claranews.com
username = garryknight
password = mypassword

If you use other news servers, you can add them below this. You can follow each news server name with a username and password, but only if the server requires this. You might want to change the default 'expire = 7' line if you want articles to be available for longer than 7 days. You'll see a line in the config file that says # port = 8000. Leave this commented out (with the # in front of it) unless you know that Leafnode must use a different port. The default port for Leafnode is 119.

You'll probably want to set the initialfetch to somewhere between 500 and 2000, depending on which groups you intend to read and how much traffic there is on those groups. Likewise, set maxfetch to the maximum number of articles you want Leafnode's fetchnews program to download in any one go; somewhere between 300 and 1000 might be a suitable amount. You can also set the maxage option to something like 14 days if you keep getting old postings. You might want to impose a limit on the maximum size of article that can be downloaded; you can do this by setting the maxbytes option to a suitable size. Two other options that are worth setting are timeout_short and timeout_long. The comments in the config file tell you what they're used for. Once you're done, save the config file.

It's a good idea at this point to add an environment variable to your ~/.bash_profile (or the equivalent if you use a different shell). Add a line that simply says 'export NNTPSERVER=localhost'. This won't be needed for some newsreaders (e.g. Pan and KNode) but will be needed by others (e.g. slrn).

You'll need to set up a cron job to expire old articles from the local news spool, so create a file called texpire in /etc/cron.daily that contains the following:

    #!/bin/sh
    # Leafnode texpire cron job. To be run daily.
    # Expire old news.
    su - root -c /usr/sbin/texpire
    

If you're not using Mandrake, you can use the cron line suggested in the INSTALL file. You can also follow the instructions in the INSTALL file and add a line to /etc/hosts.deny that says 'leafnode: ALL' and another to /etc/hosts.allow that says leafnode: 127.0.0.1. This will allow only connections to Leafnode from your own machine. [Using Leafnode on a LAN is beyond the scope of the instructions in this web page. I'm assuming that if you've got your LAN up and running, you'll know how to configure your firewall and your hosts.deny and hosts.allow files the way you need them.]

The final piece of configuration that's needed is to set up Leafnode as a server daemon running under the xinetd network services superdaemon (there's an article here on how xinetd works and how to configure it). To get Leafnode running, you'll need to create a file called 'leafnode' (or you could call it 'nntp' in your /etc/xinetd.d directory. But check first to see if you already have this file. It should look like this:

    service nntp
    {
            disable         = no
            flags           = REUSE
            socket_type     = stream
            wait            = no
            user            = news
            server          = /usr/sbin/leafnode
            log_on_success  += DURATION
    }
    

Note that if you've just upgraded and the leafnode executable is somewhere other than /usr/sbin, you'll need to change the server line. And if you've decided to use tcp wrappers (setting the lines in /etc/hosts.deny and hosts.allow as described above, you'll need to change the server line to '/usr/sbin/tcpd' and add a line below it with 'server_args = /usr/sbin/leafnode'. Also note that if you're not using Mandrake, you'll need to check the location of the tcpd executable. And if your distro doesn't use xinetd, but uses inetd instead, you won't create the /etc/xinetd.d/leafnode file at all. Instead you'll edit your /etc/inetd.conf file and add a line that looks like this:

    nntp stream tcp nowait news /usr/sbin/tcpd /usr/sbin/leafnode
    

You'll need to restart the xinetd daemon, and you can do this as root with 'service xinetd restart'. If that gives you an error message, use '/etc/init.d/xinetd restart'. On a non-Mandrake system, the xinetd script might be somewhere else. And if you're using inetd instead of xinetd, you'll need to do kill -HUP `cat /var/run/inetd.pid` instead (note the use of `backticks` in that command).

You can now test that the Leafnode daemon is running and listening on port 119. Enter (as root or an ordinary user) 'telnet localhost 119'. You should see something like:

    [garry]$ telnet localhost 119
    Trying 127.0.0.1...
    Connected to localhost.localdomain (127.0.0.1).
    Escape character is '^]'.
    200 Leafnode NNTP Daemon, version 1.9.19 running at localhost.localdomain
    

This shows you that Leafnode is, indeed, listening. Enter 'quit' to get back to the command prompt.

You're now ready to fetch the newsgroup list from your ISP's server, after which you can subscribe to some newsgroups and set things up to automate the process of downloading articles and uploading your own posts. As root, connect to the net in your usual way, then enter '/usr/sbin/fetchnews -vv'. You should see something like this response:

  1.9.19: verbosity level is 2
  Trying to connect to news.claranews.com ... Connected.
  

Leafnode's fetchnews will take some time downloading the group list, so you might as well go and make coffee. Once you've got the list, you can disconnect from the net and fire up your news client and configure it to fetch news from the Leafnode daemon.

You need to tell your news reader to fetch its articles from 'localhost' rather than from your ISP's server. I suggest that, rather than altering the existing configuration, you add a new account. I'll give you instructions on how to do this for KNode, Pan and slrn; from that you should be able to work out how to configure other news clients.

Knode
Click Settings, Configure KNode. Click the Accounts section, then News, then click the New button to create a new account. On the Server tab, enter Leafnode into the Name box, localhost into the Server box, and make sure that the Port box contains 119. You can select the Fetch news descriptions item if you like, but don't select the Server requires authentication setting. You can set up the Identity tab as well, if you wish to. Click OK and you'll see the new account listed. Click OK again and you'll see Leafnode listed as a server in the left-hand column.

Pan
Click Edit, then Preferences, then Servers - News, then the New button. Enter Leafnode in the Profile Name box, localhost in the Server Address box, and make sure the Server Port box contains 119. The My server requires my username and password setting should not be selected. Click OK and you'll see the new account in the dialog.

slrn
Edit your ~/.slrnrc file and find the server line. Edit it so it says: server "localhost" ".jnewsrc-leafnode". You might use a different newsrc file, but make sure the server is set to localhost.

You can now subscribe to newsgroups in the usual way. You'll need to get KNode and Pan to 'download' the group list from the Leafnode server. With slrn, you'll need to run it like this: slrn -f /home/garry/.jnewsrc-leafnode --create. This will get the group list and write it to your newsrc file. If you've set a different newsrc file in ~/.slrnrc, use it in this command line too. Once you have a list of subscribed newsgroups, you can download all articles. In KNode, you use the Get New Articles in All Groups option in the Account menu. In Pan, after you've used Get List of All Groups in the Servers menu, highlight all of the groups, then right-click and select Get All Articles. In slrn, all you need to do is enter a subscribed group and it will automatically fetch all articles.

What you should see is just one article in each group. This is a placeholder for the group. You need to 'read' this article to make Leafnode remember that you want to download future articles for each group. If you look in /var/spool/news/leaf.node you should see your group list. If you look in /var/spool/news/interesting.groups you should see one empty file for each group you've subscribed to. In future, when you subscribe to a new group, remember to 'fetch all articles' and to 'read' the single placeholder article. You're now ready to download your first batch of posts.

You can now manually download new articles if you like, and I suggest you do this for a first run-through just to make sure everything's working. Soon I'll show you how you can automate the process to some degree. To fetch your first batch of posts, su to root and enter /usr/sbin/fetchnews -vv. You'll see fetchnews connect to your news server, check for new groups, then download articles from each group. Once it's finished, you can fire up your news client and 'fetch all articles' for all of your subscribed groups and you should see the ones you've just downloaded. If you can, everything is now working as it should be. If not, you need to go back over everything in this page (and read the Leafnode documentation as well) to find what you've inadvertently misconfigured. But if it's working, from now on all you need to do is to subscribe to new groups, do a 'fetch all' (if you're not using slrn) and read the placeholder article, and new articles for that group will come down next time you do a 'fetchnews'. When you unsubscribe, Leafnode will monitor the unsubscribed group for a while (the time set in timeout_long) and will then stop fetching new articles for the group.

You can automate the download process in at least three ways: you can run fetchnews from a cron job, from /etc/ppp/ip-up.local, or manually every time you connect to the net. I'll cover each in turn, but before I do I want to show you how you can avoid having to type in root's password every time you run fetchnews. There's a command called 'sudo' which allows you to run commands that require root privileges, and it can be set up to allow you to do this without having to enter root's password. You configure it by editing (as root) the /etc/sudoers file. Here are the relevant lines from my /etc/sudoers:

    # User privilege specification
    root    ALL=(ALL) ALL
    garry   ALL=(ALL) NOPASSWD:ALL
  

This last line says that I can run all commands that require root privileges without having to enter root's password. However, I do have to enter the sudo command instead:

    sudo /usr/sbin/fetchnews -vv
  

Downloading using a script
Here's a script you can use that simply downloads new articles:

  #!/bin/bash
  echo Fetching new articles...
  sudo /usr/sbin/fetchnews -vv
  # Let the user know we've finished
  echo Done.
  

As you can see, it simply lets you know what's going on as it does the job. You can enhance this by getting it to play a sound when it's finished. Later on in this article I'll show you the script I use, which does a fair bit more than just downloading the news.

Downloading using a cron job
Rather than manually downloading, if you're on a permanent connection to the Internet, you can download every so often by adding a line like this to your /etc/crontab:

  * */4 * * * root /usr/sbin/fetchnews -vv &>/home/garry/logs/fetchnews.log
  

This downloads new articles every 4 hours and writes any output to /home/garry/logs/fetchnews.log. Any error messages are written to /var/log/messages in the usual way, so if anything goes wrong, you should check both files.

Downloading from ip-up.local
Each time you connect to the net using a modem, the /etc/ppp/ip-up script is run. This script calls another script called ip-up.local which you can use for your own purposes. For example, you can upload and download mail, get a time check from an NTP server, initialise your web proxy, and so on. Obviously, you can also use it to download news. If you don't have an ip-up.local in /etc/ppp you can create one. Here's one I made earlier:

  #!/bin/bash
  /usr/sbin/fetchnews -vv
  /usr/bin/play /data/misc/sound/complete.wav
  

There's no way of knowing when the download has finished, so I've included a line that plays a sound to let me know.

Downloading from the command line
Finally, you can simply download from the command line. Once you've connected to the net, run your script or simply enter sudo /usr/sbin/fetchnews -vv.

However you do the downloading, once it's finished, just fire up your newsreader and tell it to get new articles for all groups.

This section contains some ideas on how to set things up so you get the best use from Leafnode. You can skip this (or skim-read it) for now and come back later when you've got more experience using Leafnode, if you like.

Where Leafnode keeps its 'stuff'
Have a browse around in the /var/spool/news directory. Amongst other things, you'll see directories called 'failed.postings', 'interesting.groups', 'leaf.node' 'message.id', and 'out.going'. If you've subscribed to some groups and downloaded articles for those groups, you'll also see directories with names like 'alt' and 'comp'.

Have a look in 'interesting.groups' and you'll see one zero-length file for each group to which you've subscribed. Leafnode uses these files to keep track of how long you've been reading each group. If you haven't read any articles in one of these groups (for example because you've unsubscribed) then when you run fetchnews you might see a message like 'Ignoring alt.boring.group from now on' and the alt.boring.group file will disappear from the 'interesting.groups' directory.

When you run fetchnews, any new articles you've composed will be sent to your ISP's news server. If for any reason one or more articles can't be uploaded, they will be moved to the 'failed.postings' directory. You should check this directory every now and again. It's also a good idea to check 'out.going' every now and then, too. I've seen some posts not get uploaded because the Newsgroups field contained something that wasn't a newsgroup. If this happens to you, you can edit the post (see below) and check that it gets sent next time you upload.

Once you've set up Leafnode and downloaded the groups list from your ISP's news server, the 'leaf.node' directory will hold at least two files: 'groupinfo' which contains details of all of the groups available on all of the servers you've named in your /etc/leafnode/config file, and another file that has the same name as your ISP's news server and which contains a list all the groups you are interested in downloading. For each group that is carried by that server, you'll see the latest article number. If any group will be downloaded from one of the other servers, you'll see article number 0.

When new articles are downloaded, they are stored in subdirectories that are named according to the newsgroup they're in. For example, if you're downloading the alt.os.linux.mandrake group you'll have an alt/os/linux/mandrake hierarchy of directories under /var/spool/news. Copies of the articles are also placed in subdirectories under the 'message.id' directory. I haven't had time to check out the program source code for Leafnode, so I don't know how these subdirectories map to the article numbers.

When you compose a new article in your newsreader and hit Send, it's placed into the 'out.going' directory, from where it will later be uploaded when you run fetchnews. So you can 'ls /var/spool/news/out.going' at any time to see what's waiting in the queue.

Re-editing articles before uploading
Have you ever had one of those times when you've hit Send then wondered whether you really should have written what you did? Or you've hit Send then come across a later article that either makes your article superfluous or makes you wish you could edit what you'd written? Well, you can delete any article you like (or rather, don't like) from /var/spool/news/out.going. If you're not sure which one it is, you can do 'less /var/spool/news/out.going/article-id' in a console or load each article into your favourite GUI text viewer/editor. One quick way of viewing article contents is to fire up 'mc' (Midnight Commander) in a console, browse to the out.going directory, move the cursor to each article in turn and press F3 to view it.

But if you want to edit an article before sending it, there's one thing you need to be aware of: articles that conform to the RFCs have DOS line endings. That is, each line ends in a #13#10 (or cr/lf) pair of characters. Your text editor needs to be able to load and save the text while preserving the line endings. The emacs editor will do this transparently. If you prefer a GUI editor, both KWrite and Kate are suitable. In fact, both of them allow you to change between Unix, Windows/Dos and Mac line ending formats. One other thing you should be aware of is that if you use emacs to edit a file, it might leave a backup copy (with a ~ added to the end of the filename) lying around and you'd best delete that in case it inadvertently gets uploaded.

Upload before download
Fetchnews downloads new articles before uploading any that you've composed. This means that you won't see your carefully-crafted pieces of literature next time you read the news, but the time after that. However, fetchnews also has a -P parameter that tells it just to upload and not to download. So, if you're using a script to download, you can put a 'sudo /usr/sbin/fetchnews -Pvv' on the line before the one that downloads new articles. I'll show you a sample script later on.

Backing up your posts
The chances are that your news client has a way of saving a copy of any articles that you upload. For example, KNode has its 'Sent' folder. But if you're like me you'll want a backup copy of each article, one per file, each named with the message ID. (If you're not like me, you can skip this section, of course.) I use KNode, but I can only export my uploaded articles in mbox format (all articles in one file), or I can select each article in turn and use the Save item on the File menu to back them up one at a time. If you're happy backing them up in mbox format, all well and good, but if you're not then you can simply copy the articles out of the 'out.going' directory before running fetchnews. What I've done is to create a bash script which does everything I need to have done, all in one place, like this:

  #!/bin/bash
  # Fetch news using Leafnode

  # if we have any outgoing articles...
  if dirhasfiles /var/spool/news/out.going
  then
  # copy them to backup database
    echo Backing up articles...
    cp /var/spool/news/out.going/* /data/news/out.going
  # now upload any waiting articles
    echo Uploading articles...
    sudo /usr/sbin/fetchnews -Pvv
  fi

  # fetchnews has a habit of unsubscribing for some reason
  echo Touching interesting.groups ...
  sudo /bin/touch /var/spool/news/interesting.groups/*

  # now download new articles
  echo Fetching new articles...
  sudo /usr/sbin/fetchnews -vv

  # Let the user know we've finished
  echo Done.
  /usr/local/bin/beep
  play /data/misc/sound/complete.wav
  

You'll see that the first thing it does is to check whether I actually have any articles waiting to be uploaded. It uses my dirhasfiles Perl script (which you can download here) to detect whether there are any files queued. If there are, it backs them up to my chosen directory then uploads the queued files. I've put a line in there that 'touches' the time stamps of the files in the 'interesting.groups' directory. The reason for this is that I found that fetchnews had a habit of telling me it was 'Ignoring most.of.your.interesting.groups from now on'. Until I find out why, the line stays in the script. Then, after downloading new articles, it tells me it's finished. I run this script from the command line but there's no reason why you can't run a similar script from a cron job or from /etc/ppp/ip-up.local. You'll notice the command '/usr/local/bin/beep' near the end. If you don't have a beep command you can create a bash script that just does this: echo -e "\a" >/dev/tty12

Forcing article expiry
If, as I do, you back up your news cache (everything in /var/spool/news) then you'll find that your cron job to expire old articles doesn't do what you expect. The reason for this is that when you back them up the atime (last access time) of each file changes to the time you did the backup. This makes it look to texpire as if you'd recently read the article, so it doesn't get expired. The way round that is to change the command line in your cron job so that it passes the -f option to texpire. This makes texpire check the actual file creation time instead, so the articles get to be expired at the proper time.

  su - root -c /usr/sbin/texpire -f
  

Using the 'killfile'
Many news clients have the ability to 'killfile' posts that are on subjects you don't care for or from people you don't like. With some of them (e.g. Pan) you can create an extensive rule set that results in the ignoring or actual deletion of posts based on filters that you define. With others (e.g. KNode) you can either ignore an entire thread or set a view filter that prevents certain articles from showing up in the list. Leafnode comes with a program called 'applyfilter' that can be used to delete articles from the cache if any of the article's header fields matches any of a set of regular expressions that you define. The definitions go in /etc/leafnode/filters, or in any other file you like - just remember to put the full path to the file in the 'filterfile =' section in your /etc/leafnode/config file. Here are some examples of lines you can put in your /etc/leafnode/filters file:

  ^Newsgroups:.*[, ].+windows.*$
  ^Subject:.*[Mm]ake money fast.*$
  ^From:.*Mr Spock.*$
  

The first example will delete any article that is posted or cross-posted to any newsgroup with 'windows' in the name. The second will delete any article with 'Make money fast' or 'make money fast' in the subject line. The third will delete any articles posted by 'Mr Spock'. I'm going to be writing an article on the use of regular expressions at some time in the future, so check the Tutorials section every now and again for further details.

The 'newsq' command
As I mentioned earlier, you can check to see if any articles are waiting to be uploaded by doing 'ls /var/spool/news/out.going' in a console. That will only give you a list of the article IDs and not the contents. Leafnode comes with the 'newsq' command which will show you output like this:

  [garry]$ newsq
  28596-1021940638-1:     1247 bytes, spooled Tue May 21 01:23:58 2002
        From: Garry Knight <garryknight@gmx.net>
        Ngrp: freeserve.help.unix
        Subj: Re: cant get into internet