Record-breaking uptime is over – 1003 days

Please, a moment of silence, for one of longest uptimes for a actively used server.

When we started many years ago and moved into an office, our first server was a white-box desktop. We scrambled to build it out of components we had… some memory from here, a motherboard from over there, and hard drives (software RAID) from who knows what. It was by no means anything comparable to our current arsenal made out of stacks of PowerEdge servers running vSphere. Anyway, we have moved a few times and it has faithfully followed us. It has occupied our current location for about 3 years.

The other day, it got jealous. Well actually, I think there was a sharp voltage drop when we plugged a 4U PowerEdge server into the UPS it was sharing. The high-quality components it’s made out of apparently showed their true colors this time causing …wait for it…. a reboot!

So now we’re back to 0… it’ll be a long journey. No one has committed to upgrading the critical software it holds, so it won’t be decommissioned anytime soon.

See you again in 2.747945205479452 years.

Before and After the 4U server was plugged into the UPS. Ouch!

BEFORE 4U PowerEdge
LINEV    : 117.0 Volts
LOADPCT  :  23.9 Percent Load Capacity
BCHARGE  : 100.0 Percent
TIMELEFT :  85.0 Minutes
LASTXFER : Automatic or explicit self test

AFTER 4U PowerEdge
LINEV    : 113.7 Volts
LOADPCT  :  50.4 Percent Load Capacity
BCHARGE  : 100.0 Percent
TIMELEFT :  39.0 Minutes
LASTXFER : Unacceptable line voltage changes

Troubleshooting/Debugging BSOD errors

What happens when you get a Blue Screen of Death (BSOD)?  I’m sure almost everyone just says something like “____ Microsoft!”  Unfortunately, most of the time, you would just be using Microsoft as a scape goat.  Why?  According to Microsoft and other gurus, about 70-80% of crashes are caused by 3rd party drivers.  Yep, all those great toys you have hooked up to your computer and the software that control them are most likely responsible.

I have probably just blown your mind or you are probably full of skeptism.  Hopefully these debugging techniques can make you a believer….

Step 1:  Disable auto-reboot on a crash

Step 2:  Create a memory dump versus a Mini crash dump..  This will allow you to get more information from the dumps.

Step3:  Install Windows Debugger tools

Step4:  Set environment variable to automatically download symbols from the Microsoft symbol servers (WinDBG->Source Symbol Path->”srv*C:symbols*http://msdl.microsoft.com/download/symbols”)

Step5: Open the crash dump file located in C:Windows or C:Windowsminidump

Step6: Run “analyze -v” to get list of drivers in the stack text.  If the driver points to one of the Windows core system files (ntoskrnl.exe, win2k32.sys, etc), then you probably have to dig a little deeper.

Step7: Additional helpful debug commands to run to find the culprit

kv – Looks at stack of current thread.  This is used for misdiagnosed analysis.  Look for suspicious drivers

lm kv – Shows version information (dates, etc) of currently loaded drivers to find updates for.

!vm – Check pool usage (if close to maximum, then it’s a leaky driver)

!thread – looks at currently running threads

!process 0 0 – summary level display of processes during crash

!irp <irp from IRP List from !thread> – Associates drivers thread (it’s a hint to investigate)

!poolused (needs to enable on xp and earlier) – Use with Strings

!deadlock

 

 

Debugging mode (F8) – Use when no crash dump created…, needs to connect using usb (modify boot.ini) or serial from another system running windbg

Windbg – File->Kernel Debug

Debug -> Break to connect to crashed system

.dump (saves dump information)

 

Hung system troubleshooting (computer freeze)

– Use crash on control-scrl-scrl (registry setting)

– Check other processors on multiple processors

lm kv <driver name from stack>

Help for Asterisk AA50 including issues, how to rebuild compact flash filesystem, and workarounds

First, I would like to say that the AA50 is not a recommended product.  Actually, I think it's the opposite of it.  I would recommend an analog Phone with a voicemail recorder before I would recommend one of these things.  Why do I have such harsh feelings towards it?  Well, support personnel is unable to realize that a PBX has major issues if it reboots randomly and prevents you from leaving voicemails or getting voice prompts.  I even tried to make them understand by explaining to them that the problem is not an advance or unsupported feature, but one that's critical to the basic intended functionality of the device itself.  My response was "It's not meant to be used as a full PBX".  Secondly, they told me the issues are being worked on, but they haven't figured it out yet.  Uhh… my support ticket was created about a year ago!  Response "Do you know how hard it is to rewrite a firmware?"  I'm a very patient and understanding person, but if you fail to recognize a critical issue with a product at such a simple level, I feel my point will never be accepted.  Just imagine if Toyota took a year to fix their brake problems or say the cars weren't suppose to be fully used that way…. 

I'm proud to do Digium's job for everyone by providing the public community a work-around and documenting what I've learned.  Hope this help others.  As for the AA50, I will never buy anything solely and directly made by Digium again.  Buy Sangoma and use open-source Asterisk.

Background: http://www.keycruncher.com/blog/2009/11/02/digium-confirms-major-issues-with-aa50-voip-appliance-spotaneous-reboots-and-memory-card-write-lock-a-review/

Symptoms:

  1. The system reboots randomly and frequently
  2. The system loses access to the compaq flash filesystem frequently, thus no voicemails or voicemenu prompts or even backups.
  3. The system prevents you from deleting voicemails due to the issue with Symptom 2.

Detail Description:

Basically, the reasons are:  Memory leak(s) (Symptoms 1) and Memory card write-locks (Symptoms 2,3)

Work-around:

Create an automated cronjob to reboot the system on a nightly basis.

  1. Create a script (reboot-24hrs.sh) in /etc/config (use this directory because it's backed up to the local storage; not flash storage)
    #!/bin/sh
    sleep 86400
    /bin/asterisk -rx
    reboot

Edit /etc/config/rc.local and add /etc/config/reboot-24hrs.sh &

What if you wanted to rebuild your compact flash card?  The answer is simple:

  • The appliance on startup (/etc/rc) mounts the compact flash using this command:  "mount -t ext3 /dev/hda1 /var/lib/asterisk/sounds"
  1. /sbin/create_sounds (Formats the compact flash memory card and creates the proper sounds directory.  It also downloads the files from the Internet)
  2. /sbin/update_tz (Downloads time zone files from the Internet)
  3. /sbin/update_phoneprov (Downloads phone provisioning files from the Internet)

A useful print server configuration tool

Have you ever wanted to make a backup of all your printers, it’s shares, the permissions for them, and the drivers on your print server?  Well, Microsoft has a very useful tool that does this.  Furthermore, it also does restores!  I couldn’t believe my eyes either!  It’s great for when you need to setup redundant print server configurations or when you are migrating print servers!

Here it is:

http://www.microsoft.com/WindowsServer2003/techinfo/overview/printmigrator3.1.mspx

Malware,Spyware,Scareware – How to detect and prevent infection…

What is malware and how do I get it?

Generally speaking, malware are malicious software designed to infiltrate a computer system without the owner knowingly allowing it to.  It’s intent is to perform devious acts on or using your computer.   These are programs that generate misleading alerts and false detections in order to convince users to purchase illegitimate security software.

Additional Malware Info

What are the symptoms?

Pop-ups, website redirection, network configuration changes, unresponsive computer, etc…

Information regarding Antivirus2009 Malware

Information regarding Internet Security 2010

How did I get it?

The source usually comes from emails, websites, pirated software downloads, P2P applications, fake video codecs, software exploits (ie. acrobat), etc… The typical scenario is a pop-up that asks you to download and install something.  Once the download and install happens, the malware will take over the computer.

How do I protect myself?

  1. We still live in a world where humans can usually make the best decisions.  This means user training is one of the best method to prevent infections.  Below are a list of things to train users on that doesn’t require a lot of time.
    • Users should be a little paranoid and skeptical when it comes to reading the emails they receive, especially emails requesting actions to be taken. If it sounds important, take the time to read and verify it carefully!
    • Users should make sure they have an SSL connection when making transactions online or logging into banking sites.
    • Exercise caution with e-mail and files received from unknown sources, or received unexpectedly from known sources.  If the email is from someone they know, make sure it has relevant content specific to that person (ie. writing style, context of message, etc.)
    • Users should know sometimes a pop-up can be made to look like a Windows error message. Recognizing legitamite software interfaces can help (Antivirus software, Windows Security Center, Windows Defender, Anti-malware software)
    • Don’t download random software from the Internet until you know it has a valid homepage and user base (look for software reviews for it). Once that’s verified, make sure you download directly from the vendor’s website.
    • Users should understand how a website can be spoofed to go to the wrong website using the HOSTS files.
    • Users should understand that a text link can have a different URL embedded.
    • Don’t install software unless you were intentionally trying to.
  2. Keep Windows and your browser software up-to-date by downloading and applying security updates.
  3. Use an active and updated antivirus and anti-malware application that detects harmful websites, files, and emails. There are many applications out there that are free. Some highly recommended ones are Spybot Webroot, Search and Destoy, MalwareBytes, SuperAntispyware, PC Tools Spyware Doctor.

Removal Tips:

  1. Boot into SAFE MODE. It will give you a more effective platform to work with.
  2. The key is to get the system to allow you to install anti-malware software with the latest updates to slowly remove the programs.
  3. Fix infections and reboot often will get you further along in the removal process.
  4. There is no perfect anti-malware software, therefore, you should run scans using multiple anti-malware software to make sure all malware is removed.
  5. Can’t run/install software due to access permissions – This is usually due to the software restriction in your local security policy or your registry has malicious group policies regarding software restrictions configured.
  6. Can browse website or weird website redirections – Check the Internet Explorer proxy settings. 95% of the time, it shouldn’t be using a proxy. Also, make sure your HOSTS file doesn’t have malicious entries in it.

Windows Server 2008 SSL VPN (SSTP)

Now-a-days, every business is mobile, which means a VPN connection is most likely needed.  The problem is when clients travel to hotels or other countries, where firewall compatiblity and configuration can cause connectivity issues.  Fortunately, there is an answer for this: SSL VPNs.  Since an SSL VPN connection is secure and allowed on almost all firewalls, remote users will have a much more reliable connection mechanism; no matter where they are.  To top it off, SSL VPN (SSTP) is a feature natively bundled with Windows Server 2008.  How cool is that?

How does SSL VPNs help?

  • A NAT device doesn’t need to support PPTP in order for it to work.
  • Specific ports aren’t required to be open on the firewall (think hotels and other countries).
  • Connectivity can be made through web proxy servers.
  • The small footprint VPN client is easily accessible.

Clients supported: Vista SP1+, Windows 7, Windows Server 2008

What are the high level steps involved to setting up Windows Server 2008 SSL VPN connections (SSTP)?

  • Obtain a certificate to be used for your connections (just as if you are installing an SSL certificate for your website)
  • Install IIS on the VPN server
  • Request a certificate for the VPN server using the IIS Certificate Request Wizard
  • Install the SSL certificate
  • Install the RRAS server role on the VPN server
  • Enable the RRAS Server and configure it to be a VPN server
  • Configure the User Account to allow dial-up connections
  • Update DNS (ie. vpn.company.com)
  • Configure the Client to use SSTP and Connect to the VPN Server using SSTP

How-To configure Windows 2008 for SSTP VPN
1.  Install IIS on VPN server with all security settings marked for installation
2.  Create a Certificate Request in IIS console
a.  Make sure common name is actual Internet Hostname clients will connect to (e.g. vpn.company.com)
3.  Cut and Paste the certificate request into your SSL provider’s website
4.  Install any Intermediary certificates and your SSL certificate per your SSL provider’s instructions (DO NOT bind the certificate to a website in IIS)
5.  Install Routing and Remote Access
6.  Load the Routing and Remote Access MMC and run the wizard to enable it (Select Custom -> VPN if you are using only 1 NIC)
8.  Enable Dial-In access for the remote user’s AD account
9.  Enabled SSL connection (port 443) from the outside
10. Update DNS for the domain with the common name of the certificate
11. Test the SSL VPN connection by choosing SSTP in the vpn network connection properties

If the connection doesn’t work, make sure the proper certificate is bound following:
1.  Make sure the right certificate hash is bound (netsh http show  ssl)
a. If necessary, delete and readd the correct certificate binding…
1. Remove binding from IPv4 (netsh http delete ssl 0.0.0.0:443)
2. Remove binding from IPv6 (netsh http delete ssl [::]:443)
3. Delete registry entry for the hash (reg delete hklmsystemcurrentcontrolsetservicessstpsvcparameters /v sha256certificatehash /f)
4. Add binding from IPv4 (netsh http add sslcert ipport=0.0.0.0:443 certhash=<replace with your cert hash> appid={ba195980-cd49-458b-9e23-c84ee0adcd75} certstorename=MY)
5. Add binding from IPv6 (netsh http add sslcert ipport=[::]:443 certhash=<replace with your cert hash> appid={ba195980-cd49-458b-9e23-c84ee0adcd75} certstorename=MY)

Helpful links to configuring SSTP VPN:

  • A step-by-step guide to setting up your own SSTP server is to be found here.
  • Troubleshooting help can be found here, here, and here.

The case of Windows 7 not wanting to install on your hard drive partition

As an IT Consultant, I often find myself mucking with the latest and greatest things before I would even consider recommending them to clients.  The down side to this is the time spent during a Saturday afternoon trying to install Windows 7 Professional 64-bit on my home desktop.  I was so impressed with the RC1 version, that I wanted to deploy it to my single desktop (can’t be worse than Vista… heheh)

Apparently, Windows 7 is very picky about the hardware/harddrive/partition/MBR it’s installed on.  When I ran the installation, my hard drive and the system partition would be displayed, but I could not get it to actually start the installation..

I tried the following, but to no avail:

  • Installed latest drivers
  • Deleted partition
  • Reformatted newly created partition
  • Removed external hard drive and USB key so the only thing left was a single 200GB HD and an IDE cdrom.
  • Using diskpart to set the new partition as an ACTIVE partition

Looking at the setup logs (Shift-F10 -> notepad windowspanthersetupact.log), I saw a bunch of “not system disk”, “not primary partition”, “not enough space”, “not good enough” errors…

Finally, I tried to think outside of the box.  I decided to repair my MBR and Boot record the old fashion way.  I booted off a Windows XP SP2 CD and ran the recovery console.  Once in, I ran “fixmbr” and “fixboot”.  I then rebooted into Windows 7 setup and was able to click Next to continue with the Windows 7 Professional installation.

Hopefully, this technique would work for most of you.  If it still doesn’t work for you, try making sure there’s no USB keys or any other storage device connected when you install.

Sarbanes-Oxley Act update…

Since we have approximately 4 more months until the Sarbanes-Oxley Act (SOX) will be applied to all publicly traded businesses, I’d like to give everyone a little refresher course.  Although it won’t apply to most of our clients, the act proves to be a solid compliance guideline for any company.  Due to the trickling effects of larger businesses and the tightening of requirements from lenders and such, it’s not a bad thing to follow.

SOX Compliance Summary

Designed to:

– protect investors and creditors of public companies

Applies to:

– Publicly traded corporations with larger than $75 Million in Public Equity (expires Dec 2009)

– Documents and files pertaining to financial statement generation (atleast 5 years)

Provisions:

Section 404 – Requires management to report on the effectiveness of their internal financial controls and for outside auditors to attest to the management reports.

Section 802 – Makes it a criminal violation to alter, destroy, mutilate, conceal or make a false entry in a record, document or tangible object with the intent to impede, obstruct or influence any investigation or bankruptcy matter.

Section 1107 – Provides criminal penalties for retaliation related to an employee’s whistle blowing activities.

Section 301 – Requires the independence of audit committees.

Section 302 – Mandates that CEOs and financial officers certify financial statements.

Section 406 – Requires public companies to disclose whether they have adopted a code of ethics governing the behavior of senior financial officers.

General Business Process recommendations:

– Segregation of duties

– Establish a policy of archival and backup (onsite and offsite)

– Have independent party review books on a regular basis (CPA)

– Have good documented procedures

Quickbooks recommendations:

– Review Financial data using reports

– Setup user accounts with only necessary priviledges

– Implement audit trails

– Backup regularly (minimum retention time of 3 periods)

General Info:

Sarbanes-Oxley regulations require that an audit trail of log files and all pertinent documentation must be retained for five years. SOX defines which records are to be stored and for how long, focusing specifically on retention of audit and accounting records that relate to the generation of financial statement that will be submitted to shareholders and the SEC. Both paper and electronic versions of this documentation must be retained. SOX does not, however, specify how they are to be stored — best practices for data protection, disaster recovery and storage management pertain. That means the impact of Sarbanes-Oxley can be felt by nearly every component of IT operations, including messaging, storage, virtualization and even networking, so long as financial data or activity occurs on them. In turn, IT must be able to produce electronic records of these audit trails for compliance audits.

The IT departments of all public companies must be aware of the key requirements of SOX, including log management, backups and all relevant electronic communications. New platforms for communication enabled by Web 2.0 technologies like blogs, wikis and social networking are introducing all-new compliance headaches, as gigabytes of data are generated through messaging and sharing. If it pertains to finance and accounting, enterprise IT professionals must track and archive it for the inevitable visit by a compliance auditor looking for log files. Increasingly, compliance officers are using event log management software to track key moments where data enters or exits an enterprise, like email systems or the addition or departure of employees with access to sensitive financial data.

What does Server 2008 SP2 and Vista SP2 have to offer?

Below is a quick rundown of all the goodies in the new Service Pack for Windows 2008 and Windows Vista.  Of course, SP2 includes all hotfixes and other updates post SP1, but unlike traditional methods, Windows Server 2008 SP2 and Windows Vista SP2 uses the same Service Pack executable, which makes it easier to deploy.

Although this update provides minimal changes, it’s always recommended to do a test rollout before deploying in full.

Warning:

This was taken from Information about Service Pack 2 for Windows Vista and for Windows Server 2008

After you install SP2, a sound device or some other hardware device may no longer work. If this behavior occurs, just install an updated driver for the device. To do this for a sound device, follow the steps in the following Microsoft Knowledge Base article:

948481 How to troubleshoot sound problems that you experience after you install a Windows Vista Service Pack

What it has to offer:

  • Bluetooth 2.1 Support
  • Wi-fi – Uses Windows Connect Now technologies for wireless connectivity and improves on resuming wireless connectivity from a sleep state
  • Power Management – Microsoft touts a 10% increase in power management efficiencies
  • exFat file system extension – Now includes UTC timestamps which helps with file synchronization across time zones
  • Blu-ray data burning capability – Not for making movies, but for backing up files
  • Windows Search 4.0 – faster, better support for Group Policy, able to index encrypted files
  • VIA 64-bit processor support

Windows Server 2008 offerings:

  • Hyper-V 1.0 versus a prelease version
  • Addresses Terminal Server license key issues

Most useful part about SP2:

Service Pack Clean-up Tool (compcln.exe) – Deletes older versions of the RTM and SP1 based files

Requirements:

SP1 has to be installed prior to updating.

Where to get it:

Windows Server 2008 SP2/Vista SP2 (x86, 32bit)

Windows Server 2008 SP2/Vista SP2 (x86, 64bit)

What the buzz is about: Virtualization and Consolidation

I know everyone has been hearing this continuously for the last 3 years or so, but what does it actually mean?  How does it help the IT department and how does that help the business?  While the two words of Virtualization and Consolidation are separate topics, they often go hand in hand.  I believe the reason for this are to take advantage of new hardware capabilities and new virtualization technologies.  Lets say you bought a new server and wanted to consolidate and migrate all the data from your file server and your mail server.  You’ll essentially end up with a server with more utilization.  Sure, you’ll save a little on time and the electric bill, but that isn’t going to give you the “WOW” factor when it comes to analyzing your ROI. You’ll soon realize that consolidation usually will not fully utilize the full capabilities of your new hardware.  Unlike oil and water, implementation of Virtualization creates a symbiotic relationship with Consolidation.  One could also say it’s the catalyst to a higher ROI.

Why is there a push now?

  • 64-bit operating systems allow for a significant increase in memory
  • Multi-core CPUs creates effective use of processing power
  • VT-enabled CPUs support virtualization specific instruction sets which increase the effectiveness of hardware.
  • Microsoft is now in the market, which usually means it’s growing and here to stay.

Summary of benefits of Consolidation and Virtualization:

  • Lower TCO (power and cooling requirements, less physical assets, reduce time and resources need by IT staff, reduce licensing requirements)
  • Increased flexibility (backups, snapshots, migration, quicker provisioning of new servers)
  • Space savings
  • Makes use of the full capabilities of your new hardware
  • You already have the capabilities, you just need to implement it.
  • Fail-overs can be performed practically instantaneously
  • Upgrades to a new server in the future is greatly simplified
  • It’s much easier  to make  the  resulting  infrastructure resilient  for  business  recovery  and  continuity  solutions

When will you know it’s “GO TIME”?

  • You are trying to cut costs
  • You are trying to increase performance
  • You will be purchasing a new server
  • You spend too much time focused on maintenance
  • You are implementing a software “refresh”
  • Your servers have multiplied to where you have a management problem
  • You need IT to work projects that drive the business
  • Your backup solution is inefficient and ineffective
  • You have a need for a development environment