If it's a time of day issue, likely due to netflix usage of others on your ISP... About all you can do is work around it (time-wise), see if your provider has a faster package to upgrade to, or switch providers.

Ahh just saw your message about others in the house... if it's more than 10 computers, or if they are bit-torrenting or streaming something like netflix, you need a faster package from your isp. Or, get your own dedicated connection