Strangecode Hosting Status

Balsa experiencing traffic spike

We're experiencing heavy traffic on balsa.strangecode.com; some sites may be inaccessible at the moment. We're investigating to determine the cause of the traffic and apply mitigation. Please stand by.

UPDATE 05:34—We've blocked a few IP addresses that seem to be associated with botnet activity from China at the firewall. This should help with the load. We'll continue monitoring to ensure things quiet down.

Hardware upgrade on Sunday May 13

We're taking advantage of recent advances in high-density memory technology, and will be upgrading the host hypervisor for servers Oak, Balsa, and Walnut to 64 GB of RAM. This should significantly improve performance for most websites, especially those which are database intensive.

• Sunday, May 13 at 1-2 am PST (i.e. late Saturday night)
• Anticipated downtime: 45-60 minutes
• Affected servers: Oak, Balsa, Walnut

Websites and email will be inaccessible during this time, but will promptly return to their normal working state once the servers are back online. No incoming email should be lost. Thanks for tolerating this necessary downtime, we hope you'll all be dreaming sweetly during the operation.

UPDATE—01:31 PST: Upgrades completed; all services are up and running. Now we'll spend some time tuning mysqld's configuration to take best advantage of the new memory. Things should be lightning fast when you wake up in the morning.

Reboot scheduled for 1am PST Dec 20

We're rebooting all servers 1am PST on Dec 20 for system updates. Downtime should be about 1-3 minutes.

Announcing the Strangecode Hosting Controls, our customer portal

Today we launched a shiny new management interface for our customers, the Strangecode Hosting Controls. Access it at https://control.strangecode.com/ and you can:

• view and update your account details, contact info, and payment method,
• view and manage your hosted websites,
• add new, transfer, and cancel websites,
• link to email, database and SFTP controls,
• view and pay invoices,
• open support tickets,
• and view support documents.

As always we’re available by phone and email to help you perform these tasks, yet with this site you’ll be able to do all these things on your own. We’d love your comments and suggestions, or if there are any other features you’d like to see.

Reboot scheduled for 5 Nov 2011

We'll be rebooting all servers tonight 10pm PST to install system updates. Downtime will be about 2 minutes.

Intermittent switch connectivity issues

Rackspace has informed us that one of the switches in front of our servers woke up having a bad hair day. It is having intermittent connectivity issues which seems to be affecting access to server pine (aka dev.strangecode.com). The switch will be rebooted, or replaced, to fix the problem. This will all be resolve within an hour.

Reboot scheduled Oct 3 at 10pm PST.

We'll be rebooting all servers tonight 10pm PST to install system updates.

In case you didn't notice our new telephone number

Please note: we updated our support line number to +1.530.636.2633 – a while ago in fact. Our old number, +1.530.554.9555, will be deactivated this month.

You can always visit www.strangecode.com/contact for current contact details.

Compromised mail accounts and spammers

Today a few email accounts on our mail server were compromised and the accounts used to send spam. The credentials for these accounts were somehow revealed (possibly a keystroke recorder on a client's computer) and were accessed from Nigerian IP addresses. We discovered over 250000 messages in our mail queue being sent from these accounts.

Due to the flood of spam messages leaving our mail server its IP address has been listed on some blacklists (so far we're aware of Comcast, Yahoo, AOL, Road Runner, Verizon, and Barracudacentral) which prevent messages sent from our server being accepted by these networks.

To solve the problem we've done the following:

• We've blocked the Nigerian IP addresses at our firewall
• The passwords have been changed on the compromised email accounts
• Spam messages which did not yet get a chance to leave the server have been deleted
• We've submitted unlisting requests for the blacklists we're aware we're on
• UPDATE: We've also changed our IP address for outgoing mail (see update 07 Sep 11:45 below for details)

If mail you send comes back undelivered with a failure notice indicating our IP is blacklisted, please forward the notice to us at hosting@strangecode.com.

In the meantime, you are welcome to use the SMTP server provided by your ISP (AT&T, Comcast) to send messages instead of the Strangecode mail server.

We're serious about email security and reliability, and are working our best to resolve this issue. Thanks for your understanding.

UPDATE—07 Sep 11:45: We're still waiting for Yahoo and other blacklists to remove our mail server's main IP. There isn't a way to expedite these types of requests; we're at the mercy of Yahoo's expediency. But instead of waiting, we've decided to change our mail server's IP address from 67.192.236.116 to 98.129.251.228, and update the DNS zones for all domains we manage. This should have zero impact for our clients, except perhaps email client software may display a security message indicating that the IP address has changed.

UPDATE—08 Sep 01:14: The new IP address is in place, and all DNS changes have taken place. Email is now being delivered successfully to all hosts, including Yahoo.

UPDATE—08 Sep 17:55: Apparently this issue has also effected receiving email; some customers have reported email delivery failures to their accounts due to the IP address being blacklisted. We've never known a blocklist to be used this way. In any case our change of the IP address has solved this issue too. Of course if any further issues are noticed don't hesitate to email hosting@strangecode.com.

Migrating one.strangecode.com

We're moving our dev server to a new data center tonight, midnight PST. This will require an IP address change. You won't even notice it moved. Thanks for listening.

UPDATE: Importing huge database dumps is taking longer than expected. We'll update here when things are finished.

UPDATE: All finished. The new server's name is officially pine.strangecode.com, but can be accessed still at one.strangecode.com, or any of your dev domains which are hosted on it.