DNSChanger Malware on Monday, July 9th, 2012

If you’ve browsed Facebook or Google lately, you may have come across a few articles with the warning that “millions of Americans will lose their internet connections” on Monday, July 9th. Some articles claim this so-called ‘DNSChanger’ malware is set to go off like a timed bomb; others claim the FBI is forcefully causing the shutdown. Regardless of the reason, there has been much concern about a possible internet outage this Monday, and whether or not it affects you both at work and at home. All of us here at NetCal would like to save you the headache, and break down the facts from the fiction.

Q: Is this issue real?

A: Yes, but the facts are greatly distorted.

The ‘DNSChanger’ malware is not lying dormant on your computer until Monday, and the FBI is not cutting off your internet access forcefully. The malware was real however, and may have infected your computer 4-5 years ago.

Computers use something called a DNS (Domain Name System) in order to translate ‘internet names’ into ‘internet numbers’. When websites like ‘www.google.com’ are typed into your browser, a request goes to a server which translates the name into the proper IP address (74.125.224.65). Your computer is normally setup to acquire the DNS server automatically from your ISP (Internet Service Provider), or from a DNS server set up in your business.

The ‘DNSChanger’ malware, widely released in 2007, changed the settings on the computers it infected and redirected the DNS address to private servers run by scam artists and identity thieves. Instead of www.google.com translating to 74.125.224.65, it would translate to their private IP addresses instead!

The scam was so widespread (half a million computers infected in the US), the FBI was forced to get involved to shut the criminals down. The criminals were caught, their equipment confiscated, and computers were rid of the infection in record time. There was just one catch: Getting rid of the DNSChanger infection did not change the computer’s DNS settings back to normal!

The FBI decided to setup real DNS servers using the IP Addresses that the criminals used. In the end, even if you were infected by the malware, your internet access was no longer compromised. Fast forward 5 years later to 2012, and the FBI are now retiring these servers. As a result, the previously infected computers will be without DNS services.

Q: How can I find out if I was infected?

A: You can visit ‘dcwg.org’ and have your computer tested online.

Click on “Detect” towards the top and see if you are using the FBI’s DNS servers.

Q: How severe is this infection? Can it be fixed?

A: It is very quick to fix, and does not permanently harm any systems.

For more information please visit the following:

http://www.slashgear.com/dnschanger-malware-for-dummies-sophos-video-explains-it-all-06237487/

Exchange 2007-2010: Brief Overview of Changes

Exchange 2007

– Routing groups are tied with Active Directory sites and services

– Replication is done using Active Directory replicattion

– Bridgehead server role was eliminated and replaced with the Hub Transport seerver

– Outlook Web Access (OWA) was dramatically improved to similar to 32-bit version of Outlook

– Direct file access (Access shares on servers through OWA)

– OWA provides access to mailbox rules, out-of-office rules, provisioning of Mobile devices, access to digital rights managed content

– LCR – two databases replicated on separate drives on the same server

– CCR – users mailbox replication across servers and sites (fail-over and fail-back capabilities)

Exchange 2007 SP1

– Public folders available in OWA

– Standby Continuous Replication (SCR) allowed for offsite, over-the-wan replication of databases with 20 minute replication delays.

– Geo-cluster is possible for remote CCR

Exchange 2010

– Server Licensing

– Standard supports 5 database stores

– Enterprise supports up to 150 stores

– User Licensing (non-relating/exclusive to server licensing)

– Enterprise license provides unified messaging, per-user journaling for compliance support, and use of Exchange Server hosted services for message filtering

– No more Recovery Storage Groups (RSG)

– No more STM databases

– OWA enhanced features available to other browsers

– Database Availability Group (DAG, Basically CCR, No more LCR, CCR, SCR)

– Remote execution of EMS commands

Record-breaking uptime is over – 1003 days

Please, a moment of silence, for one of longest uptimes for a actively used server.

When we started many years ago and moved into an office, our first server was a white-box desktop. We scrambled to build it out of components we had… some memory from here, a motherboard from over there, and hard drives (software RAID) from who knows what. It was by no means anything comparable to our current arsenal made out of stacks of PowerEdge servers running vSphere. Anyway, we have moved a few times and it has faithfully followed us. It has occupied our current location for about 3 years.

The other day, it got jealous. Well actually, I think there was a sharp voltage drop when we plugged a 4U PowerEdge server into the UPS it was sharing. The high-quality components it’s made out of apparently showed their true colors this time causing …wait for it…. a reboot!

So now we’re back to 0… it’ll be a long journey. No one has committed to upgrading the critical software it holds, so it won’t be decommissioned anytime soon.

See you again in 2.747945205479452 years.

Before and After the 4U server was plugged into the UPS. Ouch!

BEFORE 4U PowerEdge
LINEV    : 117.0 Volts
LOADPCT  :  23.9 Percent Load Capacity
BCHARGE  : 100.0 Percent
TIMELEFT :  85.0 Minutes
LASTXFER : Automatic or explicit self test

AFTER 4U PowerEdge
LINEV    : 113.7 Volts
LOADPCT  :  50.4 Percent Load Capacity
BCHARGE  : 100.0 Percent
TIMELEFT :  39.0 Minutes
LASTXFER : Unacceptable line voltage changes

Troubleshooting/Debugging BSOD errors

What happens when you get a Blue Screen of Death (BSOD)?  I’m sure almost everyone just says something like “____ Microsoft!”  Unfortunately, most of the time, you would just be using Microsoft as a scape goat.  Why?  According to Microsoft and other gurus, about 70-80% of crashes are caused by 3rd party drivers.  Yep, all those great toys you have hooked up to your computer and the software that control them are most likely responsible.

I have probably just blown your mind or you are probably full of skeptism.  Hopefully these debugging techniques can make you a believer….

Step 1:  Disable auto-reboot on a crash

Step 2:  Create a memory dump versus a Mini crash dump..  This will allow you to get more information from the dumps.

Step3:  Install Windows Debugger tools

Step4:  Set environment variable to automatically download symbols from the Microsoft symbol servers (WinDBG->Source Symbol Path->”srv*C:symbols*http://msdl.microsoft.com/download/symbols”)

Step5: Open the crash dump file located in C:Windows or C:Windowsminidump

Step6: Run “analyze -v” to get list of drivers in the stack text.  If the driver points to one of the Windows core system files (ntoskrnl.exe, win2k32.sys, etc), then you probably have to dig a little deeper.

Step7: Additional helpful debug commands to run to find the culprit

kv – Looks at stack of current thread.  This is used for misdiagnosed analysis.  Look for suspicious drivers

lm kv – Shows version information (dates, etc) of currently loaded drivers to find updates for.

!vm – Check pool usage (if close to maximum, then it’s a leaky driver)

!thread – looks at currently running threads

!process 0 0 – summary level display of processes during crash

!irp <irp from IRP List from !thread> – Associates drivers thread (it’s a hint to investigate)

!poolused (needs to enable on xp and earlier) – Use with Strings

!deadlock

Debugging mode (F8) – Use when no crash dump created…, needs to connect using usb (modify boot.ini) or serial from another system running windbg

Windbg – File->Kernel Debug

Debug -> Break to connect to crashed system

.dump (saves dump information)

Hung system troubleshooting (computer freeze)

– Use crash on control-scrl-scrl (registry setting)

– Check other processors on multiple processors

lm kv <driver name from stack>

Help for Asterisk AA50 including issues, how to rebuild compact flash filesystem, and workarounds

First, I would like to say that the AA50 is not a recommended product.  Actually, I think it's the opposite of it.  I would recommend an analog Phone with a voicemail recorder before I would recommend one of these things.  Why do I have such harsh feelings towards it?  Well, support personnel is unable to realize that a PBX has major issues if it reboots randomly and prevents you from leaving voicemails or getting voice prompts.  I even tried to make them understand by explaining to them that the problem is not an advance or unsupported feature, but one that's critical to the basic intended functionality of the device itself.  My response was "It's not meant to be used as a full PBX".  Secondly, they told me the issues are being worked on, but they haven't figured it out yet.  Uhh… my support ticket was created about a year ago!  Response "Do you know how hard it is to rewrite a firmware?"  I'm a very patient and understanding person, but if you fail to recognize a critical issue with a product at such a simple level, I feel my point will never be accepted.  Just imagine if Toyota took a year to fix their brake problems or say the cars weren't suppose to be fully used that way…. 

I'm proud to do Digium's job for everyone by providing the public community a work-around and documenting what I've learned.  Hope this help others.  As for the AA50, I will never buy anything solely and directly made by Digium again.  Buy Sangoma and use open-source Asterisk.

Background: http://www.keycruncher.com/blog/2009/11/02/digium-confirms-major-issues-with-aa50-voip-appliance-spotaneous-reboots-and-memory-card-write-lock-a-review/

Symptoms:

  1. The system reboots randomly and frequently
  2. The system loses access to the compaq flash filesystem frequently, thus no voicemails or voicemenu prompts or even backups.
  3. The system prevents you from deleting voicemails due to the issue with Symptom 2.

Detail Description:

Basically, the reasons are:  Memory leak(s) (Symptoms 1) and Memory card write-locks (Symptoms 2,3)

Work-around:

Create an automated cronjob to reboot the system on a nightly basis.

  1. Create a script (reboot-24hrs.sh) in /etc/config (use this directory because it's backed up to the local storage; not flash storage)
    #!/bin/sh
    sleep 86400
    /bin/asterisk -rx
    reboot

Edit /etc/config/rc.local and add /etc/config/reboot-24hrs.sh &

What if you wanted to rebuild your compact flash card?  The answer is simple:

  • The appliance on startup (/etc/rc) mounts the compact flash using this command:  "mount -t ext3 /dev/hda1 /var/lib/asterisk/sounds"
  1. /sbin/create_sounds (Formats the compact flash memory card and creates the proper sounds directory.  It also downloads the files from the Internet)
  2. /sbin/update_tz (Downloads time zone files from the Internet)
  3. /sbin/update_phoneprov (Downloads phone provisioning files from the Internet)