New Database System For 2015

, Posted By Anthony Rossbach

So far during 2015 the systems that power Twii have been getting some updates, and the biggest one is a few weeks away. Most sites like Facebook, Youtube, Twitter, Tumblr, and Flickr store images on normal servers and they are loaded on demand, but we want to do something different. We are moving to DB ONLY storage of images, removing the physical files per image for a few massive database's holding images in tables. This will allow for quicker image fetching, all future images are saved in raw, and generated on the spot. So if render quality goes up in the future old photo quality will also improve. We can also then provide download of the RAW uploaded photo in the future for the user.

Images currently take around 10GB of storage, with them being in the database from RAW we expect the images to use 30+ GB of database storage, so this is something that we are not just jumping into.

How will we start?

To start we are working on converting scripts in the background on a test server using image backups. We want to get the conversion down to a few hours so we can apply the storage switch and have all the old images converted while the system is down and uploaded back into the new database.

Downtime?

We are hoping for no system wide downtime, but we are still a few weeks away from the conversion date. The image uploads during this time will not work as a result of the move, but the rest of the site will function. Images will still load as the conversion happens, until we hit the last 10 minutes of the process. Then everything will be back to normal.