Server Log 2K3

Tape Issues

2006-12-13T18:38:00.000-05:00

There have been some tape issues recently. Please check the message boards for a full discussion and more information.

IE 7 Installed

2006-12-13T18:37:00.000-05:00

I installed IE 7 which came in today through the standard windows updates.

Windows updates

2006-08-16T14:27:00.000-05:00

It's been a while since this log has been updated...

Windows updates were installed today and the auto restart is set for tonight...

Ruby Stuff

2006-03-17T17:43:00.000-05:00

Installed Ruby 1.8.2-14 and RubyForApache-1.3.1. The latter had an error when installing, but may have gone through ok anyway...I'll know more soon.

New Tape

2006-02-28T10:49:00.000-05:00

I got a new tape a couple of weeks ago to replace the old one. Apparently, you can't just drop it in and expect the server to detect the new tape and make a new full backup accordingly. So I had to go into the backup wizard and do a full backup manually.

CC.net Installed and Configured

2006-02-02T14:16:00.000-05:00

Cruise Control .net has been installed on this server. We've finally got a good config file running. The web portion requires Apache support for Asp.net which was obtained with modaspdotnet. Two problems we've encountered thus far:

1) CC.net log file gets real big real fast with debug output enabled. Had to change setting in service.exe.config or something like that.

2) Also, sometimes the state files can get re-written as UTF-16 when they should be UTF-8. To fix this, I just deleted all the state files and it seems to work.

Installed Visual Studio .NET 2005

2006-01-24T13:04:00.000-05:00

Installed VS.net 2005 with a minimal installation. No problems encountered. Split the install across c: and d:. In the future, please try to install as much stuff on d: as possible, since we're running low on room on c:.

Thanks.

Windows Patches Installed

2005-12-20T09:55:00.000-05:00

Ran windows update to install the newest batch of patches. Restart set for tonight.

Upgraded CVSNT

2005-11-01T13:27:00.000-05:00

Upgraded CVSNT to version 2.5.02.2115. There was a new version out that looked like it had some imporved functionality plus it seems that the upgrade was necesary for more advanced options with the CVSMailer plug-in. I did not even bother installing the pserver protocol this time, though there may still be issues with CVS version spoofing. I guess our ISP will let us know in a couple days.

Server Down?

2005-10-25T09:10:00.000-05:00

The server was down this morning. I have not determined the reason yet, but I turned it back on. The web server part seems to be acting kind of flaky right now. I will continue to investigate and post updates as I have them.

MediaWiki Installed

2005-10-25T09:07:00.000-05:00

Installed MediaWiki as per the installation instructions here. The installation was easy as pie and went off without a hitch. Becuase we already have PHP, MySQL, and Apache installed and configured correctly, all I had to do was copy the files into a new directory and visit the page to crank on the init script. The new wiki can be reached via http://blah.blaaah.com/Wiki.

I have not configured any users or massive protections yet, but seeings as how we are behind stong authentication already, it may not be necesary. Talk to me if you have any questions.

Installed CVSMailer

2005-10-24T13:01:00.000-05:00

I installed CVS mailer based on the instructions at http://web.telia.com/~u86216121/cvsmailer/CVSMailer.html. This software will send out emails to developers whenever commits have been made. Currently, I did not set up groups or anything, so it's just m and myself getting the notices right now. The three non-intuitive things about installation: 1) I had to change all \ to / in the config files. 2) I had to do servernameusername for the usernames, cause the server seems pretty insistent about prepending the name of itself to the beginning of all the user names. 3) You need to checkout CVSROOT in order to change the configuration files.

Internal Website

2005-09-07T10:47:00.000-05:00

I started an internally avaiable website on a port that is not forwarded to the outside. This web site is served by IIS's web server. I also learned that the default application pool must be running for IIS to serve web pages.

Automated Reboots

2005-08-15T17:10:00.000-05:00

I can't figure out the memory leak so I've scheduled 4 automated reboots per week. All are very early in the morning.

Memory Leak Investigation

2005-08-11T12:45:00.000-05:00

I've started using memtriage.exe to try to troubleshoot the memory leak. I just killed the sql server process, but it didn't seem to do anything. more to come soon.

Removed More Stuff

2005-08-08T12:08:00.000-05:00

I turned off jusched.exe, a java application that checks for java updates. Now java updates must be checked for manually. i also disabled some sql server manager that we obviously aren't using.

disabled another app for the time being to see if it was the leaker.

The network path was not found

2005-08-04T09:20:00.000-05:00

One of our employees was having trouble connecting to the server via the network this morning. I'm not on-site to see first hand. Firewall enabled/disabled on client side makes no difference in this case.

Removed Anti Spyware App

2005-08-01T11:04:00.000-05:00

It's in beta, so it might be the culprit. We'll see.

Memory Leak

2005-08-01T10:51:00.000-05:00

As mkeeling stated, the server problems stem from lack of resources which stems from a memory leak in one or more applications. Performance was monitored throughout the past week and the log files show a continual decline in the amount of available memory. I will remove questionable applications one at a time to try to isolate the leakers. If I am not successful I will schedule automated reboots.

Limited System Resources on Monday Morning

2005-08-01T08:58:00.000-05:00

Once again, unable to log in after the weekend either remotely or directly with monitor/keyboard plugged in to the server. Hard reset becuase it was the only option available. Everything works fine now, at least as far as I can tell.

Weekly maintanence resets of the server are recomended to keep some level of stability.

Spoke Too Soon

2005-07-27T11:54:00.000-05:00

Well, CSVNT started flipping out and I saw a bunch of these in the Event Log:

"Windows cannot determine the user or computer name. (Not enough storage is available to complete this operation. ). Group Policy processing aborted."

So I rebooted and everything seems fine now. I am very seriously considering the automated weekly reboot.

Disabled Even More Services

2005-07-27T09:14:00.000-05:00

Sharepoint Timer Service
SMTP

------------------------------------

On a side note, I think that something I did in the past week or two has improved the stability of the server. I will continue to monitor the Event Logs.

Something New in Event Log

2005-07-21T10:29:00.000-05:00

IPBOOTP was unable to receive an incoming message on the local interface with IP address [server]. The data is the error code.

------------------------

Not sure what this means or if it is harmful.

Wireless network issues

2005-07-21T08:25:00.000-05:00

Three employees, including myself, had difficulties connecting to the wireless router again today. I tried to get some of the cabling to work for a few other offices but was unable to. Eventually I just cycled power on the router and all of a sudden everything worked. This is the second time this has happened. It would be nice to know what is going on... It would be nice if we connnected the rest of the wires for the rest of the offices so that if wireless does go down again in the future it will be no problem for people to still have internet access.

Connectivity to the server is fine, and it still has internet access. Also, the printer displays a "Fatal Print Spool Error" still when you remote in.

Disabled More Services

2005-07-20T10:37:00.000-05:00

Microsoft Exchange Information Store
Microsoft Exchange Management
Microsoft Exchange Routing Engine
Microsoft Exchange System Attendant

and I also scheduled a reboot tonight because of an update to our anti-spyware that requires it.

Enabled Bi-Directional Printer Communication

2005-07-18T13:36:00.000-05:00

with the color printer.

Stopped WINS

2005-07-18T13:31:00.000-05:00

Windows Internet Name Service has been stopped because I don't think we need it. This was done under services.msc

Application Pools

2005-07-18T13:27:00.000-05:00

I've stopped all the Application Pools because it doesn't look like we need them.

This was done by going to: Server Management > Advanced Management > computer Management > Services and Applications > IIS Manager > Application Pools

More Monday Morning Blues

2005-07-18T11:50:00.000-05:00

the server was not responsive this morning and had to be rebooted. two possible things to consider:

1) Schedule a weekly (or bi-weekly?) automated reboot during the wee hours of the morning.

2) Find out what the heck is causing this

An I/O operation initiated by the Registry failed unrecoverably. The Registry could not read in, or write out, or flush, one of the files that contain the system's image of the Registry.

and fix it.

How To Retension

2005-07-12T17:13:00.000-05:00

To retension a tape

Open Backup.
The Backup or Restore Wizard starts by default, unless it is disabled.
Click the Advanced Mode link on the Backup or Restore Wizard.
On the Restore and Manage Media tab, click the tape you want to retension.
On the Tools menu, click Media Tools, and then click Retension.

Performed on 7/12

-----------------------

Retensioning fast forwards a tape to the end, and then rewinds it to the beginning to ensure that the tape flows smoothly over the heads for a clean backup. I do not currently know how to automate this.

Standard Monday Problems

2005-07-11T09:53:00.000-05:00

When I attempted to log on to the server this morning I was unable to connect. Assuming it was just the standard weekend "This thing has been on for more than two days strait" and being too lazy to plug in the monitor, keyboard, and mouse, I did a hard reset. The hard reset did no solve the problem. I then plugged in the monitor, keyboard, and mouse to further diagnose. It was discovered that there was no internet conectivity (no surprise) and also I was unable to reach the router (big surprise). Low and behold, the problem resulted from a loose connection to the router. Wiggling the cat5 and reinserting it immediately solved the problem.

Did someone mess with the routers/switches on Friday? Do be aware that the wires may be a little loose for future trouble shooting.

MySQL choke and strange windows error

2005-07-05T09:36:00.000-05:00

MySQL choked over the weekend at some point, or possibly did not restart after server reset on friday evening (lmortenson verify: did you run windows update?). I attempted to restart the mySQL server only to be thwarted by a windows error: "Not enough quota to process command." I am not sure what this means. Is there a swap space issue?

After rebooting the server, everything went back to normal.

I also noticed that the Microsoft Anti-Spyware beta says it will be expiring on July 31. This could be the update I installed when I logged on the see what was going on. I am unsure what this will mean for our protection scheme. Will MAS be going officially live soon?

CVS Security Updates

2005-06-28T13:24:00.000-05:00

The other day I received the following warnings from our ISP:

Vulnerabilities:
Problem: CVS pserver heap overflow
Risk: HIGH
Source: nessus
VID: 12240
CVE: CAN-2004-0396
BugTraqID: 10384
Source: nessus
Service: cvspserver (2401/tcp)
Status: HOLE
Category: General
Discussion: The remote CVS server, according to its version number, might allow an attacker to execute arbitrary commands on the remote system because of a heap overflow in the cvs pserver code.
Solution: Upgrade to CVS 1.12.8 or 1.11.16
Type: infos

The problem is that we don't run CVS < v1.12.8. In fact we don't run CVS at all, we run the most recent version of CVSNT (v2.5.01.1976). It turns out that CVSNT spoofs that it is a version of CVS for non-CVSNT clients. As it turns out that spoofed version is less than 1.12.8. Regardless, PServer was an active proticol and all the documentation reccomends that it be disabled.

The CVSNT documentation states that the PServer protocol can be disabled without the need to recompile. If anyone knows the official way to do that I would greatly appreciate it. To turn off PServer I did the following
-- changed the extension on the pserver protocol DLL to .XXX
-- changed the PServer port to -1. (I tried removing the registry entry but it came back...)

In addition I turned off support for all non-CVSNT clients. Since we are all using TortoiseCVS (which uses a CVSNT client) this will be no big deal.

It may be the case that some or all of these measures will need to be taken when we upgrade CVS in the future.

Java PHP Bridge

2005-06-20T18:16:00.000-05:00

I installed the Java-PHP bridge. You should now be able to call Java classes directly from PHP. The service will not stay running and I will have to investigate that further but when it is running it is pretty sweet.

This website has all the information you would need to know to integrate java into PHP. I hope to have some examples up soon and figure out why the extension does not stay up when a user is not logged in.

How to Create Apache User

2005-05-26T10:11:00.000-05:00

Go to DOS

c:\path_to\apache2\bin\htpasswd ../conf/dot.htpasswd newUserName

Server Move / Apache Troubleshooting / Missing HD

2005-05-24T16:15:00.000-05:00

The server was physically moved this afternoon. After booting up in its new location, I noticed that the Apache service had failed to start. I checked Computer Management -> Event Viewer -> System. "The Apache2 service terminated with service-specific error 1 (0x1)." This was rather incomprehensible, but I found out you could check the Application Log. This yielded better information: "The Apache service named reported the following error: >>> DocumentRoot must be a directory." Why was the DocumentRoot gone? I tried to access the directory, but I couldn't see it, nor the drive it was on. Apparently the new hard drive had been knocked loose during the move. I opened the box and pushed everything in tight and it booted up fine.

CVS Installation, Phase 2 (final)

2005-05-17T10:28:00.000-05:00

See me if you need more information on how to access the new repository.

Two repositories have been defined in F:\projects -- one for legacy code, one for a fresh start. Move legacy code to the fresh repostiory only if it is clean and in good standing with current standards and best practices. If you move anything between repositories, please hack CVS to do so. If you need help, ask me. There is also a temp directory in there. This directory is for CVS to read/write temp files to while checking out and committing data. Do not touch this folder directly.

Port forwarding has been opened so the repositories are accessible via LAN and the internet.

All should be running well, but if you have any issues, let me know.

Enabled SSI

2005-05-10T12:56:00.000-05:00

Added the following lines to httpd.conf:

AddType text/html .php
AddOutputFilter INCLUDES .php

Added the following to the lowest level <directory> possible

Options +Includes

Fixed phpMyAdmin Error

2005-05-09T14:50:00.000-05:00

Fixed a problem with phpMyAdmin by enabling the php mbstring extension in php.ini.

Scheduled Reboot

2005-05-09T11:43:00.000-05:00

Server is scheduled to reboot at 3 AM Tuesday May 10th. I was able to re-use the same "Scheduled Task" as before...I just set the new time/date.

Where Stuff Goes

2005-05-09T11:06:00.000-05:00

C Drive (12 gigs):
OS
Apache
PHP
Smarty
CVSNT
Tortoise
---------------------------
D Drive (25 gigs):
mySQL
Other Applications
Installers
--------------------------
F Drive (80 gigs):
CVS Repository
mySQL table data
User data
Web staging area

Please do not put anything new on the C drive if you can avoid it.

CVS Installation, Phase 1

2005-05-06T12:50:00.000-05:00

Installed CVS v2.5.01.1927
- installed to C: drive
- Accesible via Control Panel --> CVS for NT
- Requires restart to complete installation
- All services are currently off
Installed TortoiseCVS v1.8.14
- Installed into C:\Program Files
- Requires restart to complete installation
Moved old CVS repository data onto new server
- I didn't konw where to put it so it is sitting on the Desktop right now.
- Requires internet access and tape backup
Things left to complete the installation:
- Restart Server
- Define and configure repository(s)
- Put old repository data in correct place and clean up
- Configure and test both over Web and LAN
- Open port forward on router

Reinstalled MySQL

2005-05-04T15:28:00.000-05:00

Reinstalled mySQL. Application now resides on the d drive, while data is stored on the f drive. Got kind of messed up in the process. Had to totally uninstall, manually remove directories, and reboot. Then removed a recalcitrant service with "sc \\server delete MySQL".

Installed RAID card / new hard drive

2005-05-04T15:26:00.000-05:00

Installed a new hard drive (Seagate Barracuda 80G 7200.7 IDE) using SX4000 fasttrack RAID card as PCI -> IDE converter. Ran in JBOD mode. Using driver downloaded from promise.com.

Installed New Antivirus

2005-05-03T11:15:00.000-05:00

AVG 7.0 File Server Edition (trial)

Seems to be working without a reboot so far...

Uninstalled Antivirus

2005-04-28T14:28:00.000-05:00

Well, all was going smoothly until I enabled IMON in NOD32. Pretty much the system flipped out and wouldn't boot up correctly. I had to go into safe mode and uninstall NOD32 and things started working again.

Unattended Reboot

2005-04-28T12:46:00.000-05:00

shutdown.exe can take an optional parameter to specify how far in the future to shutdown, but unfortunately the limit is 600 seconds, making it largely useless.

so for an unattended reboot, i used the scheduled task wizard to run shutdown /r /f at 11:59pm. seems to have gone off with no trouble.

Installed Antivirus

2005-04-27T15:48:00.000-05:00

Experimenting with the trial version of NOD32. System needs to be rebooted at some point in the future for changes to take effect.

Changed Sharing Permissions

2005-04-27T12:22:00.000-05:00

Gave "Everybody" full control over c:\Docs when shared over the network.

Disabled Guest Account

2005-04-27T12:05:00.000-05:00

Automated Backup to Tape

2005-04-21T16:52:00.000-05:00

I have configured the server to automatically make an incremental backup to the tape drive every Friday at 11pm EST. I went to the DOS prompt and typed ntbackup /?. This brought up a screen that I couldn't seem to find elsewhere. I walked through the wizard and put in the options as you'd expect. At the end it gave me an option to run the comand as a scheduled task. I went with this. The command that the wizard generated was:

C:\WINDOWS\system32\ntbackup.exe backup "@C:\Documents and Settings\Administrator\Local Settings\Application Data\Microsoft\Windows NT\NTBackup\data\Tape Backup.bks" /a /d "Set created 4/21/2005 at 4:31 PM" /v:yes /r:no /rs:no /hc:on /m incremental /j "Tape Backup" /l:s /g "ABCDEFGH-aIJF-123-456-JKFHDSFJDS"

(I changed some of the numbers and letters at the bottom in case you couldnt tell.)

Changed Apache Password Protection

2005-04-21T12:58:00.000-05:00

Previously, I used a dot.htaccess file in every directory to force user authentication. However, this increased processing time for pages and was difficult to maintain due to its decentralized nature. Today I removed all the dot.htaccess files and transferred the authentication logic to httpd.conf. I added a section similar to the following:

<Directory "C:/Docs">
Options Indexes
IndexOptions FoldersFirst FancyIndexing NameWidth=60
AllowOverride None
AuthName "Invalid"
AuthType Basic
AuthUserFile "C:\Program Files\Apache Group\Apache2\conf\dot.htpasswd"
Require user User0
</Directory>

<Directory "C:/Docs/Business">
Require user User1 User2
</Directory>

<Directory "C:/Docs/Project">
Require user User3 User4 User5
</Directory>

All of the options from the root carry down to the subfolders, except the require users part, which is replaced. Thus User0 does not have access to Business or Project. This is a better solution, but unfortunately, FireFox seems to have a problem automatically re-submitting credentials. IE6 does not have the same problem.