What happens when you get a Blue Screen of Death (BSOD)?  I'm sure almost everyone just says something like "____ Microsoft!"  Unfortunately, most of the time, you would just be using Microsoft as a scape goat.  Why?  According to Microsoft and other gurus, about 70-80% of crashes are caused by 3rd party drivers.  Yep, all those great toys you have hooked up to your computer and the software that control them are most likely responsible.

I have probably just blown your mind or you are probably full of skeptism.  Hopefully these debugging techniques can make you a believer….

Step 1:  Disable auto-reboot on a crash

Step 2:  Create a memory dump versus a Mini crash dump..  This will allow you to get more information from the dumps.

Step3:  Install Windows Debugger tools

Step4:  Set environment variable to automatically download symbols from the Microsoft symbol servers (WinDBG->Source Symbol Path->"srv*C:\symbols*http://msdl.microsoft.com/download/symbols")

Step5: Open the crash dump file located in C:\Windows or C:\Windows\minidump\

Step6: Run "analyze -v" to get list of drivers in the stack text.  If the driver points to one of the Windows core system files (ntoskrnl.exe, win2k32.sys, etc), then you probably have to dig a little deeper.

Step7: Additional helpful debug commands to run to find the culprit

kv – Looks at stack of current thread.  This is used for misdiagnosed analysis.  Look for suspicious drivers

lm kv – Shows version information (dates, etc) of currently loaded drivers to find updates for.

!vm – Check pool usage (if close to maximum, then it's a leaky driver)

!thread – looks at currently running threads

!process 0 0 – summary level display of processes during crash

!irp <irp from IRP List from !thread> – Associates drivers thread (it's a hint to investigate)

!poolused (needs to enable on xp and earlier) – Use with Strings 

!deadlock

Debugging mode (F8) – Use when no crash dump created…, needs to connect using usb (modify boot.ini) or serial from another system running windbg

   Windbg – File->Kernel Debug

   Debug -> Break to connect to crashed system

   .dump (saves dump information)

Hung system troubleshooting (computer freeze)

- Use crash on control-scrl-scrl (registry setting)

- Check other processors on multiple processors

   lm kv <driver name from stack>

First, I would like to say that the AA50 is not a recommended product.  Actually, I think it's the opposite of it.  I would recommend an analog Phone with a voicemail recorder before I would recommend one of these things.  Why do I have such harsh feelings towards it?  Well, support personnel is unable to realize that a PBX has major issues if it reboots randomly and prevents you from leaving voicemails or getting voice prompts.  I even tried to make them understand by explaining to them that the problem is not an advance or unsupported feature, but one that's critical to the basic intended functionality of the device itself.  My response was "It's not meant to be used as a full PBX".  Secondly, they told me the issues are being worked on, but they haven't figured it out yet.  Uhh… my support ticket was created about a year ago!  Response "Do you know how hard it is to rewrite a firmware?"  I'm a very patient and understanding person, but if you fail to recognize a critical issue with a product at such a simple level, I feel my point will never be accepted.  Just imagine if Toyota took a year to fix their brake problems or say the cars weren't suppose to be fully used that way…. 

I'm proud to do Digium's job for everyone by providing the public community a work-around and documenting what I've learned.  Hope this help others.  As for the AA50, I will never buy anything solely and directly made by Digium again.  Buy Sangoma and use open-source Asterisk.

Background: http://www.keycruncher.com/blog/2009/11/02/digium-confirms-major-issues-with-aa50-voip-appliance-spotaneous-reboots-and-memory-card-write-lock-a-review/

Symptoms:

1. The system reboots randomly and frequently
2. The system loses access to the compaq flash filesystem frequently, thus no voicemails or voicemenu prompts or even backups.
3. The system prevents you from deleting voicemails due to the issue with Symptom 2.

Detail Description:

Basically, the reasons are:  Memory leak(s) (Symptoms 1) and Memory card write-locks (Symptoms 2,3)

Work-around:

Create an automated cronjob to reboot the system on a nightly basis.

1. Create a script (reboot-24hrs.sh) in /etc/config (use this directory because it's backed up to the local storage; not flash storage)
   #!/bin/sh
   sleep 86400
   /bin/asterisk -rx
   reboot

Edit /etc/config/rc.local and add /etc/config/reboot-24hrs.sh &

What if you wanted to rebuild your compact flash card?  The answer is simple:

• The appliance on startup (/etc/rc) mounts the compact flash using this command:  "mount -t ext3 /dev/hda1 /var/lib/asterisk/sounds"

1. /sbin/create_sounds (Formats the compact flash memory card and creates the proper sounds directory.  It also downloads the files from the Internet)
2. /sbin/update_tz (Downloads time zone files from the Internet)
3. /sbin/update_phoneprov (Downloads phone provisioning files from the Internet)

Have you ever wanted to make a backup of all your printers, it's shares, the permissions for them, and the drivers on your print server?  Well, Microsoft has a very useful tool that does this.  Furthermore, it also does restores!  I couldn't believe my eyes either!  It's great for when you need to setup redundant print server configurations or when you are migrating print servers!

Here it is:

http://www.microsoft.com/WindowsServer2003/techinfo/overview/printmigrator3.1.mspx

Now-a-days, every business is mobile, which means a VPN connection is most likely needed.  The problem is when clients travel to hotels or other countries, where firewall compatiblity and configuration can cause connectivity issues.  Fortunately, there is an answer for this: SSL VPNs.  Since an SSL VPN connection is secure and allowed on almost all firewalls, remote users will have a much more reliable connection mechanism; no matter where they are.  To top it off, SSL VPN (SSTP) is a feature natively bundled with Windows Server 2008.  How cool is that?

How does SSL VPNs help?

• A NAT device doesn’t need to support PPTP in order for it to work.
• Specific ports aren’t required to be open on the firewall (think hotels and other countries).
• Connectivity can be made through web proxy servers.
• The small footprint VPN client is easily accessible.

Clients supported: Vista SP1+, Windows 7, Windows Server 2008

What are the high level steps involved to setting up Windows Server 2008 SSL VPN connections (SSTP)?

• Obtain a certificate to be used for your connections (just as if you are installing an SSL certificate for your website)
• Install IIS on the VPN server
• Request a certificate for the VPN server using the IIS Certificate Request Wizard
• Install the SSL certificate
• Install the RRAS server role on the VPN server
• Enable the RRAS Server and configure it to be a VPN server
• Configure the User Account to allow dial-up connections
• Update DNS (ie. vpn.company.com)
• Configure the Client to use SSTP and Connect to the VPN Server using SSTP

How-To configure Windows 2008 for SSTP VPN
1.  Install IIS on VPN server with all security settings marked for installation
2.  Create a Certificate Request in IIS console
a.  Make sure common name is actual Internet Hostname clients will connect to (e.g. vpn.company.com)
3.  Cut and Paste the certificate request into your SSL provider’s website
4.  Install any Intermediary certificates and your SSL certificate per your SSL provider’s instructions (DO NOT bind the certificate to a website in IIS)
5.  Install Routing and Remote Access
6.  Load the Routing and Remote Access MMC and run the wizard to enable it (Select Custom -> VPN if you are using only 1 NIC)
8.  Enable Dial-In access for the remote user’s AD account
9.  Enabled SSL connection (port 443) from the outside
10. Update DNS for the domain with the common name of the certificate
11. Test the SSL VPN connection by choosing SSTP in the vpn network connection properties

If the connection doesn’t work, make sure the proper certificate is bound following:
1.  Make sure the right certificate hash is bound (netsh http show  ssl)
a. If necessary, delete and readd the correct certificate binding…
1. Remove binding from IPv4 (netsh http delete ssl 0.0.0.0:443)
2. Remove binding from IPv6 (netsh http delete ssl [::]:443)
3. Delete registry entry for the hash (reg delete hklm\system\currentcontrolset\services\sstpsvc\parameters /v sha256certificatehash /f)
4. Add binding from IPv4 (netsh http add sslcert ipport=0.0.0.0:443 certhash=<replace with your cert hash> appid={ba195980-cd49-458b-9e23-c84ee0adcd75} certstorename=MY)
5. Add binding from IPv6 (netsh http add sslcert ipport=[::]:443 certhash=<replace with your cert hash> appid={ba195980-cd49-458b-9e23-c84ee0adcd75} certstorename=MY)

Helpful links to configuring SSTP VPN:

• A step-by-step guide to setting up your own SSTP server is to be found here.
• Troubleshooting help can be found here, here, and here.