I want to provide an example that will comprise my two last posts: "Thinking Inside the Box but Out of the Context" and "Resource optimization for the Virtual infrastructure".
In a large enterprise system or ERP, there are normally some batch jobs that you need to run, daily, weekly, monthly or on occasion.
These batch runs could handle millions of records in a single batch. It also common that the records are not dependent on each other e.g. customer details update from HQ. You probably know that each record could take different time to process e.g. one customer has multiple accounts in our organization while the other is just historical data.
There are many methods to optimize the process. Most of these methods are based on some kind of record distribution to several servers that perform the same process. For example, you break the million records into smaller batches of 10,000, than, send each 10,000 to a different server. There are two major issues with this solution:
- The breaking into 10,000 is arbitrary so you cannot predict the processing time
- You have to develop a control software to control the start and end of each 10,000.
So, what do I propose?
Still within the Box but out of the normal context, this is my proposal.
You prepare servers that can perform the process. You can use servers that are used for other purposes during the day. Then, expose a web services that gets a single record to process. In front of the servers you place a standard web load balancer.
Now, you run on the million records, one by one and call the service with each record.
The load balancing will do the optimization for you and you do not need to control anything beside the start and end of the million records.
This is the architecture:
Let me know if you used it!
Noam
Leave a comment