Happy Friday Everyone!
This is the single biggest signup week for AuxParty. Welcome to all of our new users! 💃🕺
We launched on Product Hunt Wednesday and are super happy with the results! Thanks to everyone who upvoted us and gave us a review 🙏
This week, we are going to:
get a little meta about AuxParty
share some great songs people discovered this week
share new features we shipped
do a deep dive into some disruptive bugs
Who Runs AuxParty?
AuxParty is built & run by two people: Ben Williams & Mike Guida. We write the code, answer support emails, tweet bad jokes, and write this newsletter. We met a little over 5 years ago and have been making stuff together since. Now, Mike lives in Portland and Ben lives in San Francisco. We built AuxParty to listen to music together and are excited to see other people using it to hang with friends & coworkers who don’t live near each other.
Right now we are bootstrapped and do contract software engineering work on the side to make sure AuxParty stays around for a long time. If you’re interested in helping us grow AuxParty sustainability or need some extra engineering help, you can reach us by responding to this email.
Songs & Albums Discovered This Week 🎧
Start your weekend off right with an upbeat track that everyone will be rocking to and asking to replay. Enjoy:
This next track is a good example of Little Brother’s lyrics, creativity, and production quality. It’s off of their new album, the first one they’ve released in 9 years. While it may be a new track, if you’re into old school hip hop, you’ll enjoy Little Brother:
Azrael is longer track, taking the listener on a long journey through different sounds. It’s mellow yet powerful - great for focusing on work.
Developer Log
Autoplay — If no one is playing music in a room, autoplay will take over and shuffle through previously played songs in the room. We are keeping this simple for now, but keep an eye out as we will be building out this functionality so that users can customize what autoplay does.
Updated Sidebar — We updated all of the sidebar to match the dark theme of the chat and give better controls when looking at a single song.
Backend Refactor — we did a lot of work to clean up how we handle events sent to and from the server. We still have more to do, but feel like we are in a much better position than were at the start of the week. Stability is the goal.
Larger roll out of permanent rooms — if you would like one, DM us.
Vote to Skip Bug Fix — We had an issue where the count of votes was used for skipping, not the number of users. Things should work as expected now.
Show current room name — when you’re in other views, you’ll see the room name that you’re in.
Admins must confirm ban — banning users was a little too easy, now you’ll get a warning before banning.
Other minor bug fixes.
Server Crash Post-Mortem(s)
On Nov 13 around 1 pm PT, we started to notice unusual CPU usage on our server. While we were expecting a larger load due to the Product Hunt launch, we were experiencing 100% usage on two cores of our CPU. At 1:15 pm PT, we decide to update our server with our hosting provider to handle the extra load with the goal of restoring service ASAP. At 1:27 pm PT, we confirmed that we were up and running with our new server. After upgrading, we noticed the load was still high but manageable with the new hardware. We did not have issues for the rest of the day.
The next day Nov. 14, around 10:30 pm PT, we noticed the server was unresponsive. After digging into the logs, it was clear that we had gone over the cap on our error management service, which was throwing errors in an infinite loop. We pushed a fix by 11:30 pm PT to not allow the error service to throw fatal errors in the future.
The next day Nov. 15, around 11:00 pm PT, we noticed high CPU usage resulting in an unusable service. At this point, we realized that while there were issues the previous two days, there was still an underlying issue that was still not fixed. After looking at our logs, we realized that our issue was due to the way we sent room updates on the homepage. Due to the table joins in our SQL database, the rate at which the client was sent the updates, and the lack of caching, we were essentially DDOSing our server when people viewed our room list. After a quick rewrite of how we sent rooms, our CPU usage went from 90% on average down to 8%. We were able to push these changes live around 1:00 pm PT.
We will continue to monitor logs and CPU usage as we learn to scale with our new userbase. We appreciate everyone’s support and kind words as we raced to get things fixed. We will be investing time next week to make sure these issues disappear.
Feedback
Shoutout to all of the users who gave us feedback this week! We’ve already made some updates from those discussions and there will be more to come.
As always, if you have feedback or ideas, we’d love to hear from you.
You can reach us by:
- replying to this email
- emailing us at contact@auxparty.com
- texting us at +1 (415) 417-1943
- sliding into our Twitter DMs
We look forward to hearing from you!