DNS Debugging for new ISP

Another very important thing to learn as a Sysadmin working with different ISP's.

We recently got a killer offer from an ISP for a big chunk of bandwidth (You know more bandwidth at cheaper cost doesn't hurt ;) more movies, videos, audio are more fun). I am a big fan of recession at these times.

After installing the MUX in our side we configured a local DHCP server serving requests and moved a small portion of the network to the new connection. This connection was fast initially and generally grew slower and worser. We were not sure on what caused this. Digging deeper i could see a initial name resolution was taking a hell lot of time. After a lot of trouble shooting found a caching name server on the local network solved this problem.

We hit this as a result of DNS requests from all clients flooding the DNS of the service provider, which in turn was delaying using DNS delay for around 4 secs. Taken into account the number of connections the part of the network would make (18 clients connected to one server) almost all requests were being dropped or timed out by the DNS.

This a very tricky problem to solve. With a team of developers breating down my neck, added to the pressure.