
	tips on daily routine works

------------------------------------------------------------
1	TIPS
1.1	compress articles in spool directory
1.2	remove too old articles in spool directory.
1.3	see logs
1.4	checks fml log

2	examples
2.1	The directory where FML creates archives
2.2	Archive Unit
2.3	get command and search path
2.4	Re-create summary file when expire

3	archive
3.1	Introduction
3.2	Automatic archive and Expire 
3.3	Why "gzip" articles?
4.1	Conversion to "gzip"ed file (archive file)
4.2	Automatic archive creation (run archive.pl by cron)

5	Expire and Newsyslog(8)
5.1	Introduction
5.2	Automatic Archive and Expire 
5.3	Config.ph Configurations
5.4	Expire.Pl Command Line Option (bin/expire.pl)

6	other tips
6.1	Cron Runs Newsyslog (obsolete)
6.2	Turn over actives/members files: .bak => .0 =>.1 
6.3	Turn over log file ($DIR/log)
6.4	Newsyslog; To Maintenance Log Files
------------------------------------------------------------


1	TIPS

1.1	compress articles in spool directory

use $USE_ARCHIVE variable. It compresses 100 articles to one tar.gz
file.

	$USE_ARCHIVE

1.2	remove too old articles in spool directory.

try $USE_EXPIRE to be 1 to remove too old articles in spool directory.

pay attention when you both expire and $USE_ARCHIVE. 

1.3	see logs

the last 10 lines

% makefml log elena

the last 100 lines

% makefml elena::tail -100 

1.4	checks fml log

/usr/local/fml/bin/daily.pl shows invalid lines of the given fml
logfile.

    daily.pl [-hd] [-f patfile] [-t trap_patfile] [-p days]
    
2	examples

2.1	The directory where FML creates archives

	$ARCHIVE_DIR = 'var/archive'; (default is 'var/archive')

The search path for archive directories is @ARCHIVE_DIR. 
$ARCHIVE_DIR is set to $ARCHIVE_DIR[0].

2.2	Archive Unit

The unit of packing articles to one file is 

	$ARCHIVE_UNIT 

If not defined, FML uses

	$DEFAULT_ARCHIVE_UNIT (default 100)

2.3	get command and search path

Consider "get 1", "mget 1" command.  The search path when FML
interprets "get" and "mget" command follows:

* Case: "get 1" and "mget 1"

1	$SPOOL_DIR (=> $FP_SPOOL_DIR)

2	@ARCHIVE_DIR

	If e.g. var/archive/100.tar.gz is found, FML extracts the
	article 1 from it and send back the article.

* Case: "mget 1-10"  command

The search path algorithm is the same. If plain article are not found
in spool and 100.tar.gz is found, FML extracts 1-10 articles from it
and send back them.

* Case: "get 100.tar.gz" and "mget 100.tar.gz"  command

send back var/archive/100.tar.gz itself ("uuencode"d in default).

2.4	Re-create summary file when expire

	$EXPIRE_SUMMARY = 1;

to re-create summary. This process removes older articles entries in
summary.

3	archive

3.1	Introduction

Articles in $DIR/spool requires a lot of spaces if the number of
articles are many (e.g. 1000 articles may need 2M). You can use "gzip"
(GNU zip) 100 articles to be 1/4 size tar.gz archived file. In my
experience 100 articles to be gzip-ped to be 50K size tar.gz. 50K
seems an appropriate size for one mail. You can save spaces a lot by
archiving articles.

3.2	Automatic archive and Expire 

Spooling $DIR/spool must be a kind of "archive" (plain format), but in
this document we distinct "archive" and "spooling". Here "archive" is
to pack several articles to one file (e.g. tar.gz) and to store them
in some directories.

Running archive and expire BY HAND must be safe. FML can expire or
archive articles automatically.  The variables are $USE_EXPIRE
(expire) and $USE_ARCHIVE (archive).

Notions of $USE_EXPIRE (expire) and $USE_ARCHIVE (archive) conflicts
each other.  If you set both variables, pay attention on the values.
Default parameters can be considered for this possibility.

Usually we want not to lose ML articles. Hence running both expire
and archive at the same time conflicts each other. Firstly it is
difficult to choose parameters. Secondly archive depends on "gzip"
and "tar", external system commands. It may be broken and may be
mis-replaced for version up. On the other hand expire code does not
depends such a thing. You can run "expire" without archive safely.
Running both functions depends on your confidence :) or tore-lance
that articles may be lost.

To do both archive and expire by hand is best.  To do automatic
archive and expire by hand must be second best. 
If you want to use both, firstly use only archive and check it.
After checking archive works well, set expire.

Configuration parameters require $EXPIRE_LIMIT is "number" (articles
left in $DIR/spool) NOT "DAYS" (expire date). It also requires

	$EXPIRE_LIMIT > $ARCHIVE_UNIT 

We recommend such as conditions

	$EXPIRE_LIMIT = $ARCHIVE_UNIT * 2;

3.3	Why "gzip" articles?

1. Why gzip articles? It is to save directory usage. Compression rate
is very high since each headers are similar each other.

2. to avoid a large directory cache in the unix kernel.

4.1	Conversion to "gzip"ed file (archive file)

If you would like to create archives by hand, please run archive.pl.

	archive.pl [-dh] [-A ARCHIVE_DIR][-u unit] range

	-d	debug mode
	-h	help 

	-d	debug mode
	-h	help
	-A	archive directory (overwrite $ARCHIVE_DIR)
	-u	unit	(overwrite $ARCHIVE_UNIT)
	range	scan 1-range articles and pack them

Please run thig program in $DIR, a mailing list home (config.ph
location).  E.g.

	% cd /var/spool/ml/Elena
	% perl bin/archive.pl

Run "archive.pl -u 100 2000" to crate 20 tar.gz files in $ARCHIVE_DIR. 
Each tar.gz file contains 100 articles. $DIR/spool has original
articles in tar.gz, which are not removed. Please remove them by hand
after checking tar.gz contents.

% ls 

MSendrc		actives		config.ph	deny
guide		help		list		log
members		objective	seq		spool
summary		welcome

% test -d var || mkdir var
% test -d var/archive || mkdir var/archive
% perl archive.pl 3800

	a lot of messages ...

% ls archive

100.tar.gz   200.tar.gz 

	a lot of messages ...

4.2	Automatic archive creation (run archive.pl by cron)

For example, you can run the following command to run archive.pl by
cron once in a week (morning on Sun) in the following:

   (chdir /var/spool/ml/Elena; /usr/local/bin/perl bin/archive.pl)

It is useful to set up a shell script for plural ML's.

crontab example:

	0 * * * * /bin/sh /usr/libexec/fml/archive.sh

[/usr/libexec/fml/archive.sh]

	#!/bin/sh

	(cd /var/spool/ml/Elena;    /usr/local/bin/perl bin/archive.pl)
	(cd /var/spool/ml/Freekick; /usr/local/bin/perl bin/archive.pl)

5	Expire and Newsyslog(8)

5.1	Introduction

Expire is to remove old articles in $DIR/spool. If you set $USE_EXPIRE
in config.ph, FML checks the spool periodically and remove old ones.

5.2	Automatic Archive and Expire 

Spooling $DIR/spool must be a kind of "archive" (plain format), but in
this document we distinct "archive" and "spooling". Here "archive" is
to pack several articles to one file (e.g. tar.gz) and to store them
in some directories.

Running archive and expire BY HAND must be safe. FML can expire or
archive articles automatically.  The variables are $USE_EXPIRE
(expire) and $USE_ARCHIVE (archive).

Notions of $USE_EXPIRE (expire) and $USE_ARCHIVE (archive) conflicts
each other.  If you set both variables, pay attention on the values.
Default parameters can be considered for this possibility.

Usually we want not to lose ML articles. Hence running both expire
and archive at the same time conflicts each other. Firstly it is
difficult to choose parameters. Secondly archive depends on "gzip"
and "tar", external system commands. It may be broken and may be
mis-replaced for version up. On the other hand expire code does not
depends such a thing. You can run "expire" without archive safely.
Running both functions depends on your confidence :) or tore-lance
that articles may be lost.

To do both archive and expire by hand is best.  To do automatic
archive and expire by hand must be second best. 
If you want to use both, firstly use only archive and check it.
After checking archive works well, set expire.

Configuration parameters require $EXPIRE_LIMIT is "number" (articles
left in $DIR/spool) NOT "DAYS" (expire date). It also requires

	$EXPIRE_LIMIT > $ARCHIVE_UNIT 

We recommend such as conditions

	$EXPIRE_LIMIT = $ARCHIVE_UNIT * 2;

5.3	Config.ph Configurations

If you run automatic expire by FML, please set in config.ph

	$USE_EXPIRE = 1; (default is 0, "not expire")

In default, expire time is "7days". It means FML removes articles
older than 7 days (one week).

	$EXPIRE_LIMIT = "7days";

$EXPIRE_LIMIT syntaxes accept "number" or "days" e.g. 100, 7days.

5.4	Expire.Pl Command Line Option (bin/expire.pl)

expire.pl [-h] [-e expire_days] [-s spool_directory] [-n]

    -h this help

    -h this help
    -e expire limit; days
    -n expire limit; the number of articles left in the spool
    -s spool directory ($DIR/spool)

6	other tips

6.1	Cron Runs Newsyslog (obsolete)

Example(Vixie cron):

0 6 * * 0 (chdir dir-of-fml; sh bin/newsyslog.sh log var/log)

FYI: msend.pl runs newsyslog in default.  msend.pl does not run
newsyslog if $NOT_USE_NEWSYSLOG = 1; is defined.

	$NOT_USE_NEWSYSLOG = 1; (default 0)

6.2	Turn over actives/members files: .bak => .0 =>.1 

When fml operates actives and members file, fml saves the previous
content in file.bak e.g. actives.bak and members.bak. In default 
fml appends content to file.bak in one week. If the file size is over

	$AMLIST_NEWSYSLOG_LIMIT	(default 50*3000 = 150K bytes)

fml turns over the file like this:

	file.bak       -> var/log/file.0
	var/log/file.0 -> var/log/file.1
	...

Independtly from this turn over, fml can save the previous content by
RCS. If you use this, please define

	$AMLIST_BACKUP_TYPE = "rcs"; (default "")

6.3	Turn over log file ($DIR/log)

If the log file excees the threshold $LOGFILE_NEWSYSLOG_LIMIT size
(bytes), fml turns over the log file
	log -> log.0 -> log.1 ...
The threshold is defined by this variable

	$LOGFILE_NEWSYSLOG_LIMIT (default "")

The unit is bytes. You can use 10K (10 Kiro bytes) and 1M (1 Mega
bytes) syntaxes.

6.4	Newsyslog; To Maintenance Log Files

newsyslog(8) is a log maintainer program. FML provides simplified
version of it.

       Newsyslog  is  a  program  that should be scheduled to run
       periodically by crontab.  When it is executed it  archives
       log  files  if  necessary.  If a log file is determined to
       require archiving, newsyslog rearranges the files so  that
       ``logfile''  is empty, ``logfile.0'' has the last period's
       logs in it, ``logfile.1'' has the next  to  last  period's
       logs  in  it,  and so on, up to a user-specified number of
       archived logs.  Optionally the archived logs can  be  com-
       pressed to save space.

msend.pl (digest program) runs newsyslog for files defined by
@NEWSYSLOG_FILES.

	@NEWSYSLOG_FILES = 
	   ("$MSEND_RC.bak", "$MEMBER_LIST.bak", "$ACTIVE_LIST.bak")

	rm -f var/log/members.4
	mv -f var/log/members.3 var/log/members.4
	mv -f var/log/members.2 var/log/members.3
	mv -f var/log/members.1 var/log/members.2
	mv -f var/log/members.0 var/log/members.1
	ln -s members.bak 	var/log/members.0

To apply newsyslog to another files, please add them to
@NEWSYSLOG_FILES.

@NEWSYSLOG_FILES = 
	("$MSEND_RC.bak", "$MEMBER_LIST.bak", "$ACTIVE_LIST.bak", "log")


		INDEX

$AMLIST_BACKUP_TYPE                        ...   6.2 
$AMLIST_NEWSYSLOG_LIMIT                    ...   6.2 
archive                                    ...   3 
archive.pl                                 ...   4.1 
$ARCHIVE_DIR                               ...   2.1 
$ARCHIVE_UNIT                              ...   2.2 3.2 5.2 
archive by cron                            ...   4.2 
$DEFAULT_ARCHIVE_UNIT                      ...   2.2 
expire                                     ...   5 
expire.pl                                  ...   5.4 
$EXPIRE_LIMIT                              ...   3.2 5.2 5.3 
$EXPIRE_LIMIT > $ARCHIVE_UNIT              ...   3.2 5.2 
$EXPIRE_SUMMARY                            ...   2.4 
$FML_EXIT_HOOK                             ...   5.4 
libnewsyslog.pl                            ...   6.4 
$LOGFILE_NEWSYSLOG_LIMIT                   ...   6.3 
@NEWSYSLOG_FILES                           ...   6.4 
$NOT_USE_NEWSYSLOG                         ...   6.1 6.4 
@StoredSpool_DIR                           ...   2.2 
$USE_EXPIRE                                ...   5.3 
