Surprise Visit from Hacker News!

What it was like having a surprise visit from the Hacker News community!

Screen Shot 2020-02-05 at 12.49.11 AM

Last Tuesday night was a very exciting night in the Space Frigates journey! Somebody posted a link to Space Frigates on Hacker News and it was upvoted to as high as the #2 spot and held a position on the first page most of the night.

I had heard about people being featured on sites like Hacker News or Reddit and receiving a large volume of unexpected traffic but never even thought about it happening to me.

So, what did it mean in terms of traffic? Well, I started using Google Analytics on November 1, 2018 and between that date and February 3, 2020 Space Frigates had received visits from abut 160 unique visitors. On February 4, 2020 Space Frigates received about 14,000 unique visitors!!!

I’m not even sure who posted it but I suspect it was a member of Indie Hackers since I have been actively posting there for the last couple of months. That is the only place I have really talked about Space Frigates online.

I’ve reached out to that user to thank them and ask how they heard about the game.

So how did the event start?

So, over the past year I have had so little traffic on the site that there was never anybody playing when I logged in. And I really wanted to play with somebody other than family and friends. I did occasionally get a random visitor but had no way to know when they were online.

So I created a trial account at Twilio and updated Space Frigates in such a way that I would receive an SMS message whenever somebody started playing the game. Well, at first I had the alert set up to fire whenever anybody connected but I was getting false alarms, likely from web crawlers, so I changed it to fire whenever a new player ship was actually created.

I talked to my brother about this and he joked about the possible implications if I started received a huge amount of traffic. We both laughed and forgot about it.

Fast forward to last Tuesday. I had just arrived in Florida to visit my parents and got an SMS message that somebody was playing the game. I thought it might be a cool chance to show them the game anyway so quickly jumped on their WiFi to see what was going on.

There was actually a real person playing and we began to do battle. I told my parents that this was pretty cool because it was really the first time it had happened. Then a second player jumped in. Then a third. Then a fourth.

I said right away that something weird was going on, I never get that much traffic. Within a few minutes I had a dozen players playing.

I wanted to try to figure out what was going on so I opened Google Analytics and looked at my referrals and  saw them coming from some site called news.ycombinator.com. I opened the link and realized that is was Hacker News!

Honestly, I had never even been to Hacker News before but had heard of it. Folks at Indie Hackers talk about it from time to time. And I knew that it gets a lot of traffic. And some how, some way, my little startup game was currently ranked #4!

Now what!

At this point, my phone was just blowing up. Sure enough I was getting hundreds of texts via my Twilio integration. Nice! I jumped back to the game and saw that there were now dozens of players and the performance had ground down to an unplayable crawl. Ouch!

I didn’t want to miss the chance to have all these folks try the game so I quickly logged into Digital Ocean to resize my VM. Space Frigates was running on their smallest, $5 a month droplet and now I was getting some serious traffic!

Resizing requires you to stop the droplet, which I hated to do but figured that it didn’t matter much since the game was crashing anyway. I picked a much larger size and restarted.

The restart kicked everybody off but at this point I had been upvoted to #2 and users were just pouring in so the server quickly become unplayable, again. Back to the console and and another vertical resize. To a much, much larger droplet this time.

It was about this time that I had to laugh at myself a bit because I work in an enterprise IT shop and know full well that I would never design a VM based solution for app like this at work. I would have gone straight to a Kubernetes based deployment model. Heck, I was already using Docker anyway! I had literally done tech talks with live demos of autoscaling web applications! The shame!!!

This is just one of those things were, as a solo founder, you just have to make choices about your time. Setting up the VM based deployments was super easy and I figured problems of scale were so far off that I didn’t need to be wasting time on it anyway.

Anyway, back to the app. Still super slow. At this point I figure maybe I needed to go horizontal. I already had my one VM behind a load balancer (totally lucked out here… the only reason I was using a load balancer at all was to gain the SSL Cert auto provisioning that comes with it) So I decided to spin up two more huge VMs to match the one, now huge, VM.

This normally would have taken a lot longer but I use a super cool tool called Meteor Up Meteor Up is an open source, Node based deployment tool for the Meteor JS stack that I use. It automates Docker based MeteorJs deployments and will even build out the VM for you too! So, using MUP, I was able to very quickly build out the two new servers behind the load balancer.

Back to the game. I was able to see all three servers quickly fill up with players again. It was still slow but at least running. And by this time it was like 3:00am in the morning and everybody else had long since gone to bed. So I figured at that point that without a real autoscaling solution or standing up a ridiculous amount of VMs I would never be able to keep up with the thousands of users logging in and decided to get a couple of hours of sleep.

The next day

I got up a few hours later and traffic was still heavy but starting to ramp down. Space Frigates had dropped off the front page at HN and I could tell almost exactly when by my traffic. It really makes a huge difference!

As that Wednesday wore on the traffic steadily decreased as the post grew stale. But I still had several hundred visitors for the next three days in a row. Way more than I had gotten in the last year combined!

By the end of Wednesday I went ahead and destroyed two of the VMs. With exponentially less traffic I didn’t need the power anymore and splitting the audience into three shards was actually decreasing the experience for the players coming in because there wasn’t as many players available to play against.

The comments

I also spent time on Wednesday reading the comments that people had been posting.

The funniest one said “I’m getting a 503, they must be getting the HN hug!”

There was one person, of course, he hated the “amateur hour piece of trash” and proceeded to troll other commenters who had anything nice to say until they gave up and left. Interestingly, they actually did have some fair criticisms. But deriding other users for liking the game seemed a little overboard.

But there were several people who left positive comments or observations about the game that were really awesome!

There was also some really long threads about other games people loved that Space Frigates reminded them of. At first I kind of blew it off but then realized that that kind of information is absolute gold! I’ve learned on Indie Hackers that it is really important to understand what your potential customers like so this gave me lots of research material!

Things that I wish had been ready

What to do and what order to do them in for a solo founder is so hard because you kind of need to do everything at once. But you can’t. So you make choices and constantly adjust.

In fairness, I did not expect this to happen at all and I consider the game to really be in early alpha so I’m pretty pleased that it worked at all under the load. My previous load testing had included five concurrent users.

Having said that, there were a couple of things that I really wished I had done before this happened and want to get knocked out before it happens again. Hopeful some thing like this will happen again…

  1. I need a real landing page. The fact that I had all those eyes without any social links or news letter sign up or really anything at all on the landing page makes me sad.
  2. I had actually just started the mobile controls on the way down to Florida but they were not done. So nobody hitting the app from mobile could play. I’m trying not to think about how many users that was.
  3. I didn’t have any autoscaling in place, or even a Kubernetes based deployment to quickly scale horizontally. I wouldn’t feel so bad about this if I didn’t know better and preach it at work all the time.
  4. My core game engine is not refined enough yet to handle such a load. Real time, javascript, multiplayer game engines like this are not trivial to build and I’m not an expert JS developer by any means. But this has to work or the game will never scale. So, a lot more work to do on that.

Final thoughts

Even though the app did not perform super well and I didn’t have my landing page ready it was still a super exciting event for me! It felt like enough people tried it and actually played for awhile to help validate the idea.

Also, there were other benefits:

  • Even though the traffic went back down I do have a steady trickle that is much higher than before the HN post. I had about 80 visitors today which is up from about 1 that I was getting before.
  • I was contacted by a couple of people who saw the game and wanted to trade ideas.
  • Space Frigates was picked up by a couple of game portal sites, which I hope will start generating more traffic
  • I did get an uptick of traffic here on the blog and a few more twitter followers so that is cool!

Over all it was really exciting, gave me some things to prioritize and just got me even more excited about the project and the possibilities if I keep working at it!

P.S. One thing I seriously did right!

Remember the Twilio integration? Had I not done that I would not have known that people were logging in and would not have been able to experience the event happening live!

Being there live allowed me to at least bump up the servers and keep the game somewhat playable.

And it was just exciting to see it happening!

So, even though SMS is probably not the recommending way to do it, always setup some kind of basic monitoring from the very start!!!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s