10. Monitoring and administration

Once the Usenet News system is in place and running, the news administrator is then aided in monitoring the system by various reports generated by it. Also, he needs to make regular checks in specific directories and files to ascertain the smooth working of the system.

10.1. The newsdaily report

This report is generated by the script newsdaily which is typically run through cron. I shall enumerate some of the problems that are reported by it, based on my observations .

10.2. Crisis reports from newswatch

Most of the problems reported to me are those with either space shortage or persistent locks. There are instances when the scripts have created locks files and have aborted/terminated without removing them. Sometimes they are innocuous enough to be deleted but this should be determined after a careful analysis. They could be an indication of some part of the system not working correctly. For e.g. I would receive this error message when sendbatches would abnormally terminate trying to transmit huge togo files. I had to determine why sendbatches was failing this often.

The space shortage issue has to be addressed immediately. You could delete unwanted articles by running doexpire or add more disk space at the OS level.

10.3. Disk space

The $NEWSBIN area occupies space that is fixed. Since the binaries do not grow once installed, you do not have to worry about disk shortage here. The areas that take up more space as feeds come in are $NEWSCTL and $NEWSARTS. The $NEWSCTL has log files that keep growing with each feed. As the articles are digested in huge numbers, the $NEWSARTS area continues to grow. Also, you will need space if you have chosen to archive articles on expiry. Allocate a few GB of disk space for $NEWSARTS depending on the number of hierarchies you are subscribing and the feeds that come in everyday. $NEWSCTL grows to a lesser proportion as compared to $NEWSARTS. Allocate space for this accordingly.

10.4. CPU load and RAM usage

With modern C-News and NNTPd, there is very little usage of these system resources for processing news article flow. Key components like newsrun or sendbatches do not load the system much, except for cases where you have a very heavy flow of compressed outgoing batches and the compression utility is run by sendbatches frequently. newsrun is amazingly efficient in the current C-News release. Even when it takes half an hour to digest a large consignment of batches, it hardly loads the CPU of a slow Pentium 200 MHz CPU or consumes much RAM in a 64 MB system.

One thing which does slow down a system is a large bunch of users connecting using NNTP to browse newsgroups. We do not have heuristic based figures off-hand to provide a guidance figure for resource consumption for this, but we have found that the load on the CPU and RAM for a certain number of active users invoking nntpd is more than with an equal number of users connecting to the POP3 port of the same system for pulling out mailboxes. A few hundred active NNTP users can really slow down a dual-P-III Intel Linux server, for instance. This loading has no bearing on whether you are using INN or nntpd; both have practically identical implementations for NNTP reading and differ only in their handling of feeds.

Another situation which will slow down your Usenet news server is when downstream servers connect to you for pulling out NNTP feeds using the pull method. This has been mentioned before. This can really load your server's I/O system and CPU.

10.5. The in.coming/bad directory

The in.coming directory is where the batches/articles reside when you have received feeds from your NDN and before processing happens. Checking this directory regularly to see if there are batches is a good way of determining that feeds are coming in. The batches and articles have different nomenclature. Batches, typically, have names like nntp.GxhsDj and individual articles are named beginning with digits like 0.10022643380.t

The bad sub-directory under in.coming holds batches/articles that have encountered errors when they were being processed by relaynews. You will have to look at the individual files in this directory to determine the cause . Ideally speaking, this directory should be empty.

10.6. Long pending queues in out.going


10.7. Problems with nntpxmit and nntpsend


10.8. The junk and control groups

Control messages are those that have a newgroup/rmgroup/cancel/checkgroup in their subject line. Such messages result in relaynews calling the appropriate script and on execution a message is mailed to the admin about the action taken. These control messages are stored in the control directory of $NEWSARTS. For the propogation of such messages, one must subscribe to the control hierarchy.

When your news system determines that a certain article has not been subscribed by you, it is "junked" i.e. such articles appear in the junk directory. This directory plays a key role in transferring articles to your NDNs as they would subscribe to the junk hierarchy to receive feeds. If you are a leaf node, there is no reason why articles should pile here. Keep deleting them on a daily basis.