Kurobox NAS-Central Forums

Welcome to the Kuro Community
It is currently Fri Apr 20, 2018 10:44 am

All times are UTC-06:00




Forum locked  This topic is locked, you cannot edit posts or make further replies.  [ 12 posts ] 
Author Message
PostPosted: Thu Feb 17, 2005 6:52 pm 
- used the original kuro installation, but stripped out www
- installed ntpd (time service) to listen to home-network ntpd
- configured syslog to send to home-network syslog server
- run sshd
- upgraded netatalk to 2.0
- installed howl's zeroconf mDNSresponder to advertise all services
- installed mt-daapd to serve up music to home-network
- installed pure-ftpd to export anonymous ftp access to file system
- upgraded samba and built to send output to syslog
- run nfs to export anonymous nfs access to file system
- installed smartmontools to monitor IDE for problems
- installed subversion server for home-network
- installed webfsd to export anonymous http access to file system
- installed symon for remote logging of cpu/mem/interface stats

Now I have a proper NAS box: ftp, http, svn, nfs, afs, cifs/smb; with centralised storage for syslog and symon log, and reporting of ATA SMART problems.

What I need next is a proper PHP based backup scheduler to use my USB2 DVDRW; currently I just nero the contents to DVD using a desktop PC.

Suggestions for anything else I could do?

Code:
USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.8  1452  528 ?        S    Feb16   0:04 init
root         2  0.0  0.0     0    0 ?        SW   Feb16   0:00 [keventd]
root         3  0.0  0.0     0    0 ?        SWN  Feb16   0:00 [ksoftirqd_CPU0]
root         4  0.0  0.0     0    0 ?        SW   Feb16   0:01 [kswapd]
root         5  0.0  0.0     0    0 ?        SW   Feb16   0:00 [bdflush]
root         6  0.0  0.0     0    0 ?        SW   Feb16   0:00 [kupdated]
root         9  0.0  0.0     0    0 ?        SW   Feb16   0:01 [kjournald]
root        47  0.0  0.0     0    0 ?        SW   Feb16   0:02 [kjournald]
root        96  0.0  0.8  1440  516 ?        S    Feb16   0:00 dhcpcd -h nasbox -t 10 eth0
root       129  0.0  1.0  1812  636 ?        S    Feb16   0:00 /sbin/syslogd
root       131  0.0  0.7  1504  460 ?        S    Feb16   0:00 /sbin/klogd
root       140  0.0  0.0     0    0 ?        SW   Feb16   0:00 [khubd]
daemon     154  1.2  1.0  2716  632 ?        S    Feb16  19:25 /usr/sbin/ntpd -s -f /etc/ntpd.conf
root       155  0.0  0.9  2952  568 ?        S    Feb16   0:00 /usr/sbin/ntpd -s -f /etc/ntpd.conf
root       161  0.0  0.8  3592  548 ?        S    Feb16   0:02 /usr/sbin/sshd
root       164  0.0  1.0  1604  628 ?        S    Feb16   0:00 /usr/sbin/cron
root       176  0.0  0.9  3072  592 ?        S    Feb16   0:00 /usr/sbin/atalkd -f /etc/atalk/atalkd.conf
root       177  0.0  1.0  2716  664 ?        S    Feb16   0:00 /usr/bin/nifd -n -i 300
daemon     179  0.0  1.1  5232  736 ?        S    Feb16   0:00 /usr/bin/mDNSResponder -f /etc/mDNSResponder.conf
daemon     181  0.0  1.1  5232  736 ?        S    Feb16   0:00 /usr/bin/mDNSResponder -f /etc/mDNSResponder.conf
daemon     182  0.0  1.1  5232  736 ?        S    Feb16   0:00 /usr/bin/mDNSResponder -f /etc/mDNSResponder.conf
daemon     187  0.0  1.1  2252  708 ?        S    Feb16   0:00 /usr/sbin/mt-daapd
root       188  0.0  0.7  1908  496 ?        S    Feb16   0:00 /usr/sbin/pure-ftpd (SERVER)
daemon     194  0.0  1.0  1808  656 ?        S    Feb16   0:00 /usr/sbin/symon -u -f /etc/symon.conf
daemon     198  0.0  0.9  3172  608 ?        S    Feb16   0:00 /usr/sbin/webfsd -s -p 80 -r /mnt -u daemon -g daemon
daemon     201  0.0  1.8  7716 1152 ?        S    Feb16   0:00 /usr/sbin/mt-daapd
root       202  0.0  1.3  6256  844 ?        S    Feb16   0:00 /usr/sbin/smbd -s /etc/samba/smb.conf -l /var/tmp -D
root       204  0.0  0.9  6256  608 ?        S    Feb16   0:00 /usr/sbin/smbd -s /etc/samba/smb.conf -l /var/tmp -D
root       205  0.0  1.6  4508 1060 ?        S    Feb16   0:01 /usr/sbin/nmbd -s /etc/samba/smb.conf -l /var/tmp -D
daemon     208  0.0  0.6  1692  392 ?        S    Feb16   0:00 /sbin/portmap
root       212  0.0  0.8  1844  540 ?        S    Feb16   0:00 /usr/sbin/rpc.statd
root       215  0.0  1.4  2292  900 ?        S    Feb16   0:00 /usr/sbin/rpc.nfsd
root       217  0.0  0.8  2112  552 ?        S    Feb16   0:00 /usr/sbin/rpc.mountd
daemon     219  0.0  1.8  7716 1152 ?        S    Feb16   0:00 /usr/sbin/mt-daapd
daemon     221  0.0  1.8  7716 1152 ?        S    Feb16   0:00 /usr/sbin/mt-daapd
root       227  0.0  1.2  1796  808 ?        S    Feb16   0:00 /usr/sbin/smartd --pidfile=/var/run/smartd.pid --configfile=/etc/smartd.conf
root       231  0.0  0.8  3584  560 ?        S    Feb16   0:00 /usr/sbin/afpd -c 1000 -n nasbox
root       232  0.0  1.4  4788  888 ?        S    Feb16   0:04 /usr/sbin/sshd
root       233  0.0  2.1  2828 1368 pts/0    S    Feb16   0:00 -bash
root      1137  0.0  1.4  4788  920 ?        S    Feb16   0:04 /usr/sbin/sshd
root      1196  0.0  2.3  2852 1448 pts/1    S    Feb16   0:01 -bash
daemon   25580  0.0 14.5 16028 9168 ?        S    Feb17   0:00 /usr/bin/svnserve --daemon --root /mnt/repository --threads
daemon   25588  0.0 14.5 16028 9168 ?        S    Feb17   0:00 /usr/bin/svnserve --daemon --root /mnt/repository --threads
root     25620  0.0  0.8  1428  540 ?        S    Feb17   0:00 /usr/sbin/ppc_uartd


Code:
smartctl version 5.33 [powerpc-unknown-linux-gnu] Copyright (C) 2002-4 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Device Model:     ST3160021A
Serial Number:    XXXXXXXX
Firmware Version: 3.06
User Capacity:    160,041,885,696 bytes
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   6
ATA Standard is:  ATA/ATAPI-6 T13 1410D revision 2
Local Time is:    Fri Feb 18 00:48:27 2005 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED


Top
   
 Post subject:
PostPosted: Fri Feb 18, 2005 10:06 am 
Offline

Joined: Wed Jan 19, 2005 1:36 pm
Posts: 299
Nice post!

I'm in just about the same spot you are. I went for plain old daapd instead of mt-daapd because mt-daapd isn't in the official portage tree yet. I don't have any Macs any more so I left off netatalk.

Edna came up in my search for a web-based music browser. I just needed a quick way to browse the MP3 files with ID3 tags. It works but is not really as nice as I would prefer. The ID3 tag reading does not distinguish between v1 and v2 tags. I would prefer a web interface that showed both sets of tags side-by-side and even one that allowed editing of the tags. Any suggestions? Even better would be one that allowed browsing the tags similar to how iTunes does. But as a bonus, Edna does stream the music to Windows Media Player and WinAMP etc.

I was looking to access the entire filesystem too. I ended up setting a read only share in Samba to / that can only be accessed by my user account and root. That works but I know it's probably not 100% safe. It's also annoying that Windows has to ask how to open every file since it doesn't recognize them as text files. That and Notepad.exe can't read the UNIX linebreaks so you have to use Writepad or something else to read the text files.

Could you tell more about webfsd? Where is the home page?

Backup is next on my list. But I'll probably just stick with something easy like writing to a archive file such as tar or whatever.


Last edited by awx on Fri Feb 18, 2005 1:05 pm, edited 1 time in total.

Top
   
 Post subject:
PostPosted: Fri Feb 18, 2005 10:12 am 
Offline

Joined: Wed Jan 19, 2005 1:36 pm
Posts: 299
PS- Also, it sounds like you have already taken care of your logging requirements, but I'd like a web based log viewer. I've seen one before when I was looking but have since forgotten the name.

I'm also tempted to do something foolish like send all my logs to mysql and see how that runs on the kuro.


Top
   
 Post subject:
PostPosted: Sat Feb 19, 2005 12:11 pm 
This is webfs: http://freshmeat.net/projects/webfs/ -- it's plain and simple.

With regard to logging. I have about 8 machines on my home/soho network. They are a mix of Linux, BSD and Windows, either as servers or clients (including wireless notebooks for the wife and children). One of them is a dedicated development machine that I get paid to do work with. I use eventlog to syslog (https://engineering.purdue.edu/ECN/Reso ... IX/evtsys/) to send Windows events to a central logging server. All unix machines also send events to the same central logging server either with stock syslog or syslog-ng. This logging server is also the named, dhcpd and monitoring server for the network. In fact, it's just an old notebook whose battery acts as a UPS if the power/fuse/etc goes.

The logging server runs syslog-ng. You can easily make it log to different files like this:

Code:
destination d_hosts {
        file( "/opt/host/$HOST/logs/syslog-$YEAR$MONTH.log"
                create_dirs(yes) );
};
destination d_daily {
        file( "/opt/run/syslog-ng.log" perm(0644) );
};


So that each host log is stored in its own directory, but it maintains a running daily log that can be inspected easily. The logging server (a NetBSD machine) also runs thttpd and I use a simple php script to read and dump out the daily log. I compress old logs when they roll over using this rc.daily.local script:

Code:
_CUR=syslog-`date +'%Y%m'`.log
find $_OPT/host \( -name syslog\*.log -and \! -name $_CUR \) -exec gzip -9 {} \;


Each night, I also run dshield processing and logsentry:

Code:
_LOG=$_RUN/syslog-ng.log
_TMP=$_LOG.$$
_PID=$_RUN/syslog-ng.pid
mv -f $_LOG $_TMP
kill -HUP `cat $_PID`
$_SHR/logsentry/logcheck \
        $_TMP
$_SHR/dshield/dshield-pf.pl \
        -config=$_SHR/dshield/dshield.cnf \
        -log=$_TMP
rm -f $_TMP


logsentry (http://sourceforge.net/projects/sentrytools/) checks the log file for security or other anomalous behaviour, and sends an email report. You can customise what events are filter out and what aren't. For example, this is a daily email message:

Code:
Security Events
---------------
Feb 18 17:58:21 hostxyz su: userxyz to root on /dev/ttyp1

Unusual Events
--------------
Feb 18 04:34:03 hostpqr ntpd[882]: time reset -0.165641 s
Feb 18 12:38:39 nasbox smartd[227]: Device: /dev/hda, SMART Prefailure Attribute: 3 Spin_Up_Time changed from 96 to 97
Feb 18 17:55:35 hostabc MsiInstaller: N/A: Product: Symantec Network Drivers Update -- Configuration completed successfully.


dshield (www.dshield.org) is a global port scanning database. I use OpenBSD pflogd (the dsl firewall and wireless gateway for the network runs OpenBSD) with a custom patch to send the "blocked" output to syslog. I then use a modified dshield script to extract out the blocked output and mail it off to dshield. It keeps them informed about global threats.

This is syslog.conf on the kuro:

Code:
*.emerg                                                 *
*.*                                                     /var/log/messages
*.*                                                     @loghost


I changed logrotate to "rotate 0"; the /var/log/messages only needs one day of history; as the loghost is easier to look at if anything else needs checking.

All of these machines have local customisation in the /opt directory; which (was cvs, but now subversion) version controlled. I migrated them from cvs to subversion and hosted the subversion server on the kuro (in /mnt/repository).

In addition, I needed a good way to backup local machines. I recently decided to use g4u (http://www.feyrer.de/g4u/) to create raw images of my client machine harddrives (notebooks, desktops, etc): the images are compressed and uploaded to the pureftpd running on the kuro, and stored in a backup directory (in /mnt/backups). This is now the backup server for the network. For the unix machines, I use dump or a custom script to generate the backup images. I have no need to do incremental backups on the windows clients, as all the user data/documents sit on the network.

The kuro is a good starting point, but it needs a bit of help to become a real nasbox, which is what I did. What I'm lacking, as I mentioned, is a good backup process for it (I have good backup process for all of the other machines).

I hope this is useful for anyone, and would like to hear of others experiences as well, maybe I can learn a few tricks.


Top
   
 Post subject:
PostPosted: Wed Feb 23, 2005 12:53 am 
Offline

Joined: Thu Jan 27, 2005 2:32 am
Posts: 21
Location: Bay Area, California
awx wrote:
It's also annoying that Windows has to ask how to open every file since it doesn't recognize them as text files. That and Notepad.exe can't read the UNIX linebreaks so you have to use Writepad or something else to read the text files.


Not sure what you mean there?
I use the Kuro/Samba as home directories for my Windows laptops.
And no issue on that side - just have to dbl-click on any .txt file to open it.
Took care of EOL problem by replacing notepad (with Wordpad or Crimson Editor).

@Guest: you mention you got rid of WWW but now looking for a PHP-based backup solution?!

On my side, I use a simple cron job to mount my USB drive, mirror my Kuro (rsync) and dismount. Thinking about using "hotplug" to streamline this: plug in USB drive to kick off backup, let it run overnight, check the LEDs (drive station conveniently turns red when volume is unmounted), unplug the drive, bring it to another location (and dbl check the backup logs). Any creative ideas on backup side would be welcome.

Thanks for pointer to webfs - will use this for remote access to my mp3's
(I can just paste the m3u URL into my media player at work and am all done!). Was thinking about re-installing thpptd just for this.

Apart from that, my setup is very similar to yours:

- Samba (inside only)
- logsentry
- syslog-ng
- Subversion (inside and outside, via SSH)
- rsync
- unison (for synchronization of certain data dirs that are updated
from various clients).

Make sure you configure SSH for certificates and download Putty/Pageant
- this makes everything so convenient (and of course SVNTortoise for subversion).


Top
   
PostPosted: Wed Feb 23, 2005 1:33 am 
Offline

Joined: Thu Jan 27, 2005 2:32 am
Posts: 21
Location: Bay Area, California
Oh yes, one thing I'd like to add:
being able to enforce some time restrictions on file access.

Example scenario: allow family to access my photos but only off hours,
e.g. user X can use webfs only between 2am and 8am
(need to manage that precious 256Kb upload bw!).

Any recommendation?
(haven't researched too much yet, know that individual packages have different options for this; sshd might also be used as a gatekeeper - but do not want family to have to deal with SSH; firewall can do this at service-level - but unfortunately doesn't know about users, so not ideal).


Top
   
 Post subject:
PostPosted: Wed Feb 23, 2005 5:04 am 
For time constraints, you may need something more sophisticated: the best bet would be to use Apache and modrewrite: you can enforce time based rules. The other possible option, I'm less certain, is if TCP wrappers supports time based rules, as it could be used for multiple TCP services. AFAIK there is no general solution for network service time constraints, unless xinetd or other superservers support it, mostly it's on a per application basis.

I mean PHP solution for backup because I want to be able to run it from a UI style front end.


Top
   
 Post subject:
PostPosted: Wed Feb 23, 2005 7:47 pm 
Offline

Joined: Wed Jan 19, 2005 1:36 pm
Posts: 299
DMD wrote:
Not sure what you mean there?
I use the Kuro/Samba as home directories for my Windows laptops.
And no issue on that side - just have to dbl-click on any .txt file to open it.
Took care of EOL problem by replacing notepad (with Wordpad or Crimson Editor).

As you probably know, Windows uses the file extension to determine how to open the file. So a text file must be named with a ".txt" on the end of the file in order for Windows to recognize it. So most of the text files stored from the Linux environment will not get recognized by Windows.

For example, the file "/etc/samba/smb.conf" won't be recognized. You can tell Windows to associate the ".conf" extension with a text editor but you can't do that for files with no extension, such as "hosts" and "fstab".


Top
   
 Post subject:
PostPosted: Fri Feb 25, 2005 8:05 am 
Offline
Site Admin

Joined: Fri Oct 29, 2004 10:08 am
Posts: 177
Location: MA
Guest,
This is a really nice bit of information. Would you be willing to put it up as a How-To on the wiki? If you aare uncomfortable with the wiki PM me and maybe I can be your hands.

Thanks
Brian


Top
   
 Post subject:
PostPosted: Tue May 22, 2007 10:36 pm 
Offline

Joined: Thu May 03, 2007 1:22 pm
Posts: 29
Location: Belo Horizonte
Indeed, would like to see that on the wiki.


Top
   
 Post subject:
PostPosted: Wed May 23, 2007 9:46 am 
Offline

Joined: Fri Oct 29, 2004 6:00 am
Posts: 91
Location: UK
A quick note regarding backups:

'cron' does my nightly backups, using rsync to an "rsync.net" account. The "rsync.net" service is good, the people there are good. And the whole cron/rsync/rsync.net arrangement is good because I'm too lazy to perform regular manual backups.

I just make sure that everything I need backed up is stored on the Kuro (and exposed via NFS), rather than on the various PCs dotted around my house.

(I have no connection with "rsync.net" by the way)

Matthew


Top
   
PostPosted: Thu May 24, 2007 10:19 pm 
Offline

Joined: Sat Nov 25, 2006 2:16 pm
Posts: 83
[quote="Anonymous"]- used the original kuro installation, but stripped out
Suggestions for anything else I could do?
[\quote]

Why not postgres, a mail server, and spamassassin?


Top
   
Display posts from previous:  Sort by  
Forum locked  This topic is locked, you cannot edit posts or make further replies.  [ 12 posts ] 

All times are UTC-06:00


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Limited