WTF? Pagestackr was down!

Not anymore (I hope).

Last Sunday, we experienced more than one type of failure that caused Pagestackr to be down for several hours, intermittently.

Firstly, a friend said to us that our signup wasn't working. That's odd, since we are using exception notifier to send us any exception by email. Checking the server's log and we found out what happened: our external SMTP server was denying access from our server.

We use a SMTP server at railsplayground.net. We were using it in some project's development and we thought it would be a goood idea to use it with Pagestackr.

Well. We hurry to railsplayground support and the response was that their SMTP server was OK. "Check your app", they said. But it was working before. And we didn't change any line of code (due to the Rails Rumble restriction).

Oddly, after a few hours of agony, the SMTP connection resume working and has been working since then.

The second failure has to do with our del.icio.us import. A bad implementation on our side when treating del.icio.us error messages caused mongrel to lock in this thread. Combining that with an almost-infinite loop, and now we can lock ALL our mongrel instances. Nice, ahn?

This bug is very simple to solve (and we did have a solution for this), but, of course, we are not allowed to make changes on our code.

The solution? We used a iptables rule to limit access to del.icio.us' api to one request per 500 seconds:

echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -I OUTPUT -p tcp -d api.del.icio.us -m state --state NEW -m recent --set
iptables -I OUTPUT -p tcp -d api.del.icio.us -m state --state NEW -m recent --update --seconds 500 -j REJECT

(actually, we used api.del.icio.us' IP address. We don't want to resolve the hostname with packet filtering.)

This way, when our loop sends a new request that would lock one mongrel, it's instantly denied by iptables and our mongrel can keep serving other requests.

So, our del.icio.us import IS working, but in a time-limited way. (yeah, it sucks... but we have to do that to keep Pagestackr running).


Finally, I want to make a clarification on voting. We are a bunch of college guys, and our friends heard about pagestackr and it spread out in the entire campus. We have emailed some people too, but we explicitly said to vote as fairly as possible. And to give a look on other apps. But, Carter, feel free to remove any votes that you consider to be invalid.


We (me, Hugo, Jose and Mario) are available to talk, and we promise to be more present at #railsrumble. You can reach me at george.guimaraes@gmail.com (email or gtalk).


PS: Congratulations to irksome, linga and rubybrigade. Really loved your apps.