Differences between revisions 6 and 7
Deletions are marked like this. Additions are marked like this.
Line 6: Line 6:
See [:Nntp.club.cc.cmu.edu:nntp] See [:Infrastructure/Nntp.club.cc.cmu.edu:nntp]
Line 11: Line 11:
See [:Selenium.club.cc.cmu.edu:selenium]. See [:Infrastructure/Selenium.club.cc.cmu.edu:selenium]
Line 16: Line 16:
See [:Indium.club.cc.cmu.edu:indium] See [:Infrastructure/Indium.club.cc.cmu.edu:indium]
Line 39: Line 39:
Posts to CMU BBoards are spooled directly to the reader machines. To set this up, put appropriate key-value pairs in the hashes at the tops of /root/bbfeed.pl /root/bbspooler.pl on [:Nntp.club.cc.cmu.edu:nntp]. Posts to CMU BBoards are spooled directly to the reader machines. To set this up, put appropriate key-value pairs in the hashes at the tops of /root/bbfeed.pl /root/bbspooler.pl on [:Infrastructure/Nntp.club.cc.cmu.edu:nntp].

1. Machines

1.1. NNTP

Our transit/spool machine.

See [:Infrastructure/Nntp.club.cc.cmu.edu:nntp]

1.2. Selenium

Our old, deprecated for several years, news reader machine.

See [:Infrastructure/Selenium.club.cc.cmu.edu:selenium]

1.3. Indium

Replacement for selenium. Work in progress.

See [:Infrastructure/Indium.club.cc.cmu.edu:indium]

2. Software

We run [http://www.openusenet.org/diablo/ Diablo USENET Software] on both the transit/spool machine and the reader machine.

Apparently we had bad experiences with [http://www.isc.org/index.pl?/sw/inn/ INN] in the past.

2.1. Patches

We have a patch against Diablo that allows us to stow the executables into /usr/local/bin and keep configuration files and spool directories in /var/diablo (instead of having everything in /news).

See /afs/club.cc.cmu.edu/system/src/local/diablo/005/paths.patch

Diablo attempts to use AIO by default on Linux. This breaks dreaderd, so make sure USE_AIO is #define'd to 0 in dreaderd/defs.h.

See /afs/club.cc.cmu.edu/system/src/local/diablo/005/aio.patch

3. Setup

3.1. Transit/Spool

At the present, I (Keith) don't know all that much about the specifics.

3.1.1. BBFeed/BBSpool

Posts to CMU BBoards are spooled directly to the reader machines. To set this up, put appropriate key-value pairs in the hashes at the tops of /root/bbfeed.pl /root/bbspooler.pl on [:Infrastructure/Nntp.club.cc.cmu.edu:nntp].

3.2. Reader

On the reader machines, we run diablo to aggregate the news and bboard feeds from the transit/spool machine, and keep a local article cache. We also run dreaderd, so users can read from, list, and post to news groups.

3.2.1. Configuration Files

  • dexpire.ctl
    • Determines how overview information (article headers) is expired.
    • We keep more headers for local articles (cmu.*, assocs.*, graffiti.*, org.*, and official.*) than for other articles.
  • diablo.config
    • Configuration file used by all diablo programs.
    • Important things: paths, logging, contact email addresses, connection and process limits, data formats.
  • dnewsfeeds
    • Defines the newsfeeds consumed and produced by the diablo process.
  • dreader.access
    • Defines access controls for connection made to the dreaderd process.
    • Determine whether access is local (I used system:campusnet, and some aarons hosts from selenium's file). Allow local posting, and local reading of bboards, and alt.binaries*. Allow limited non-local reading.
  • dserver.hosts
    • Defines the transit servers backending the dreader process.
    • This should specify the diablo process running on the same machine as the trasit backend. Make sure the port number is correct.
  • dspool.ctl
    • Determines how articles are filed into spools.
    • Local articles are given their own spool so they are retained as long as possible. Text articles in alt.* are put in a separate spool. Next, text articles in alt.* are placed in a third spool. Finally, binary articles are placed in a fourth spool, so they do not adversely affect the retention of text articles.
  • moderators
    • Sets email addresses for group moderators.
    • Sets moderator to bogus-[groupname]@club.cc.cmu.edu for bboards. Sets moderator to [groupname]@moderators.isc.org otherwise.

3.2.2. Cron Jobs

What do the sample adm scripts provided with diablo do?

  • biweekly.atrim
    • Trims history file.
    • Trims active file.
  • daily.atrim
    • Rotates log files.
  • daily.reader
    • Removes cached articles (not necessary).
    • Expires overview information (light cleaning).
  • hourly.expire
    • Expires articles from the spool.
  • weekly.reader
    • Expires overview information (deep cleaning)

What modifications are necessary to make them work in our environment?

  • all
    • Look for executables in /usr/local/bin rather than ~news/dbin.
  • daily.atrim
    • Don't attempt to rotate logs that don't exist; maybe an issue with our /bin/csh. The sample script croaks because parts of the list of log files in curly braces don't exist.
  • weekly.reader
    • Uncomment the dexpireover line with -O180, since it works fine for us.

There are also cronjobs for root for the bofh script dealing with lusers.

  • Remove old badusers.2* files from /var/diablo/bofh
    • find ~news/bofh/badclients.2* -cmin +5760 | xargs rm -f

  • Find rejection messages in the diablo general log, and set iptables rules appropriately if there were too many
    • cd ~news/bofh && ./findbadclients.sh && cat ./badclients.* | awk -- '{if($1 > 1000) print $2}' | ./bofh.sh

Services/News (last edited 2014-03-27 21:43:00 by areese@CLUB.CC.CMU.EDU)