In the system that I am responsible on, we use SAP ERP as our back office for payments and charge.
Since we have produce a monthly batch of about 1,000,000 lines we need to pass a large amount of data in one batch via interface.
There are two main challenges:
1. Moving large amount of data between different technologies
2. Control of the success/fail status on the line level
3. Manage a complex scenario that involves updates from both systems in the process
My solution was to use elements to cover all challenges.
The first element was a database table as a staging place for the data and recording line level status. The table was designed to hold the basic line data and two additional fields for status from each system.
The other element was a middleware system to trigger actions on the systems on both sides of the interface. The other task of the middleware was to midiate the technologies (.NET and SAP).
Just to to complete the picture I want to mention other alternatives that we discussed:
1. Transfer files (it's so 90's)
2. Line-by-line call and process (good idea in general but not with large data)
3. Web service (again, problematic with large data)
I invite you to comment on your own experience.