Author Topic: offload work to your database server  (Read 608 times)

Pinako

  • Hero Member
  • *****
  • Posts: 1263
    • View Profile
    • inportb
offload work to your database server
« on: January 18, 2011, 04:00:43 PM »
Okay, this may seem strange to some of you, since I'm recommending that you move work from one process to another, without gain of performance. But if you have a computationally intensive task that you need to do over and over again, and especially if you're doing it yourself in a language such as Python or Ruby, this would probably make your life a little bit easier.

Basically, when you saturate one CPU core to the max, you're going to want to put the other cores to work; and when you saturate all the cores on one server, you're going to want to add more servers. It can be difficult to design good concurrency in your application, but your database server is probably already capable of doing things in parallel. By converting your logic into a databasing problem, you essentially gain parallel processing "for free," and all you need to do is throw more hardware at the database server.

For example, I have been writing a little game where I need to compute a field of view (or fog of war) for every character/monster in the game. I can run the computation almost 2000 times a second on my netbook, and a bit faster on my VPS. This is a good start, but I'll probably be keeping track of a lot more creatures than can be handled using one CPU core. Then I realized that I was essentially doing a query (given the set of tiles and shadows within the sight radius, select all tiles that do not overlap with the shadows) that most database engines should handle with ease.

Of course, the way you would transform your logic depends on your application; but if the problem is CPU-bound rather than network-bound, it might be a good idea to put that database server to work...

daz

  • Full Member
  • ***
  • Posts: 240
    • View Profile
Re: offload work to your database server
« Reply #1 on: January 19, 2011, 11:00:18 AM »
(See how long you can keep a couple thousand computations / second running on your VPS before they get mad)

Is this all theory? Or have you actually tried implementing this yet? I'm curious as to the results.

Pinako

  • Hero Member
  • *****
  • Posts: 1263
    • View Profile
    • inportb
Re: offload work to your database server
« Reply #2 on: January 19, 2011, 04:33:37 PM »
The result is that while MongoDB (running on the same netbook) was a bit slower than Python, it made up for the deficit after I added a couple of shards. I tried that job against a SQLite database and it was slow (about 50 op/s, and SQLite can't handle concurrent loads anyway, but I wanted to try RDBMS without actually installing MySQL). Map-reduce across more machines via MongoDB is obviously more awesome, but it is still expensive. Therefore, while I'm thinking up ways to make do with fewer operations, I'm still trying to code with database/concurrency in mind :P

ultimatebuster

  • Hero Member
  • *****
  • Posts: 790
  • +2 Points/Stars
    • View Profile
    • KKSNetwork
Re: offload work to your database server
« Reply #3 on: February 05, 2011, 02:18:09 PM »
didn't we all know this like a long time ago? Offloading calculation to db... no wonder why MSSQL even has IF-ELSE IF-ELSE structure, which are used in crossfire servers.

Pinako

  • Hero Member
  • *****
  • Posts: 1263
    • View Profile
    • inportb
Re: offload work to your database server
« Reply #4 on: February 05, 2011, 03:37:37 PM »
Of course we all knew this since kindergarten. I was posting for the benefit of pre-kindergarteners ;p