WorldPay Module version 3.0 - Support thread

[bottyz]

Hi all,

Has anyone noticed the size of their database whilst running this module?

I notice that it stores all transactions as blob data which have now mounted to quite a size..... +200mb. Even my email achieve isn't that big...….

Due to the size of the worldpay table I have been unable to restore the database successfully, I have manually restored each table (except worldpay) and can use my store still but things must be missing! Does anyone know how to calm this module down a little???? Is there a better way of storing the data for this module?

Any input greatfully received.

Hi SpaceMonkey,

it is something I've noticed before, but not to your extremes.

How many records do you have? I've over 2500 with a table size of approx. 20Mb. An average blob in my db is only 2-3kb.
Each blob saves anywhere upto 6,000 characters. Is there a more space efficient data type within mysql to store that number of characters?
TBH I'm not sure if it is essential to record all of the various blobs, i.e. could we drop the 'session blob'. Is it ever used by anyone and for what purpose?

I've been tweaking a version of the module to try and incorporate a few improvements, but dealing with the support agents at Worldpay to ensure we're utilising the best options is not only time consuming, but confusing too (I don't think half their tech support team know how to communicate with their gateway).

If there's no reason to keep all the blob info, we could look to ditch some of it in a newer module.


Rich

[SpaceMonkey]

How many records do you have? I've over 2500 with a table size of approx. 20Mb. An average blob in my db is only 2-3kb.

Hi Bottyz,

I have approx. 14000 entries, each blob about 2-3kb for Post, Server & Request but I note the Session is about 9-12kb each.

I have never referred to the data here. If a callback error happens the best source of data for me is the Callback Failure email sent from worldpay. As for their support staff...…… They are administrators and not support (in my opinion), they appear to have no technical knowledge unless you want help using their control panel.

My development zone hasn't had this table data loaded and appears fine. I am thinking about dropping 10000 of the entries (from the live site) but I'm waiting for my hosting support to respond first (checking they can restore if necessary).

Slightly of topic I have had databases larger than this in the past and managed to restore with "import" in phpmyadmin no problem... Every table except worldpay restores fine. I will let you know what my host say regarding this and how the cull goes.

Thanks for response.

[SpaceMonkey]

I've been tweaking a version of the module to try and incorporate a few improvements, but dealing with the support agents at Worldpay to ensure we're utilising the best options is not only time consuming, but confusing too (I don't think half their tech support team know how to communicate with their gateway).

If there's no reason to keep all the blob info, we could look to ditch some of it in a newer module.


Rich

I have WorldPay v4 available (created by Niccol not me) if you wanted to review the work as part of your process. PM if you want a copy.

Regards

[niccol]

Here are my thoughts, just out of interest...

(1) I am not sure why those columns are mediumblob anyway. This seems like a really poor choice. (So, if someone rewrites they could consider this)

(2) Anyway, I don't think (but may be wrong) that all that data that is stored in blob format is ever used. I think it is just record keeping, of the state when the payment was processed, which is a good thing in some circumstances and may be a compliance issue .... maybe possibly.

(3) If it is decided that you need to store that information then it could be stored in a different table or even dumped to log files. Log files might be a nice solution and then the feature could be turned on/off as is done for some other payment modules logging.

(4) There is not a huge amount to recommend Worldpay as a web payment gateway. The interface and communication are just plain nasty in my opinion. I understand that the POS stuff works better and Worldpay is cheap, but I would really consider another option.

It is (4) that is the kicker for me. I got involved in trying to understand and streamline the module and ran away screaming. Happy to give some advice/opinions but I do not want to get into the nitty-gritty of this module. I value my sanity to highly :-)

[niccol]

Out of interest what are the factors that make you both choose Worldpay rather than, for instance, Sagepay?

[SpaceMonkey]

Out of interest what are the factors that make you both choose Worldpay rather than, for instance, Sagepay?

My Boss! lol

No seriously it's cost. With the added benefit of PCI certification in 10 minutes rather than days. Sage (at the time) didn't compete.

[SpaceMonkey]

(1) I am not sure why those columns are mediumblob anyway. This seems like a really poor choice. (So, if someone rewrites they could consider this)

Unusable data to me.

(2) Anyway, I don't think (but may be wrong) that all that data that is stored in blob format is ever used. I think it is just record keeping, of the state when the payment was processed, which is a good thing in some circumstances and may be a compliance issue .... maybe possibly.

Possibly maybe.... Have you ever sat and had a read of the PCI DSS SAQ level 4 plus guidance notes? Unless it's your subject it can really make you feel incompetent.

I'm going to backup my tables by exporting the data and then dropping table data older than 12mths, for me it then makes it manageable, GDPR compliant (I think) whilst keeping PCI compliance data available and complying to the full length of the law (UK). So yes If someone did add a trim function please consider an export function too.

Good luck!

[gilby]

Could a lot of that data be "debug" data
Is there a debug setting, and is it turned on?

[SpaceMonkey]

Could a lot of that data be "debug" data
Is there a debug setting, and is it turned on?

This module emails debug data with an option in Admin for mail distribution list. I remember I used it previously but not great reporting. Also worldpay email callback dumps with response data only on failure. Debug always emails... Everything.

The data table names are request, session, post, & server. Remembering this is an off site payment module I think this is the raw data encrypted!

[SpaceMonkey]

Looked a bit deeper..... its the data for the admin > customers > worldpay payments

example:

Original Billing Address: 
17 Any Road
Suburb
Town
 
WorldPay CC Address: 
17 Any Road
Another Town
Warning: Address Mismatch

Which of course is part of your AVS and other cross checks with worldpay.... or so my understanding of this goes.