Forums / General Questions / Error: Illegal mix of collations using ETL Framework

Error: Illegal mix of collations using ETL Framework

Locked

Started by jonny75904 on 28 Nov 2010, 12:26 AM · Views: 1,846

Results 1 to 3 of 3
This thread is locked. New replies are disabled.
28 Nov 2010, 12:26 AM
#1
jonny75904 avatar

jonny75904

New Zenner

Join Date:
Jul 2010
Posts:
9
Plugin Contributions:
0

Error: Illegal mix of collations using ETL Framework

I am writing an ETL transform to load product reviews from a vendor feed

Kettle Spoon 3.2.0 ETL Framework.
Zen Cart 1.3.8

The transform scans the reviews_description table to perform an upsert and return the primary key values to the stream. Here is an error I get when attempting to operate on the table:

2010/11/26 22:23:47 - Database lookup.0 - Checking row: [ALE96007], [ 5], [was a good movie], [robbie], [09/29/2004]
2010/11/26 22:23:47 - Database lookup.0 - Field [SKU] has nr. 0
2010/11/26 22:23:47 - CSV file input.0 - Signaling 'output done' to 1 output rowsets.
2010/11/26 22:23:47 - CSV file input.0 - Finished processing (I=5664, O=0, R=0, W=5663, U=0, E=0)
2010/11/26 22:23:47 - Combination lookup/update.0 - preparing combi-lookup statement:
2010/11/26 22:23:47 - Combination lookup/update.0 - SELECT reviews_id
2010/11/26 22:23:47 - Combination lookup/update.0 - FROM zen_reviews_description
2010/11/26 22:23:47 - Combination lookup/update.0 - WHERE ( ( reviews_text = ? ) OR ( reviews_text IS NULL AND ? IS NULL ) )
2010/11/26 22:23:47 - Combination lookup/update.0 - AND ( ( languages_id = ? ) OR ( languages_id IS NULL AND ? IS NULL ) )
2010/11/26 22:25:41 - Combination lookup/update.0 - ERROR (version 3.2.0-GA, build 10572 from 2009-05-12 08.45.26 by buildguy) : Because of an error, this step can't continue:
2010/11/26 22:25:41 - Combination lookup/update.0 - ERROR (version 3.2.0-GA, build 10572 from 2009-05-12 08.45.26 by buildguy) : Error looking up row in database
2010/11/26 22:25:41 - Combination lookup/update.0 - ERROR (version 3.2.0-GA, build 10572 from 2009-05-12 08.45.26 by buildguy) : Illegal mix of collations (latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE) for operation '='
2010/11/26 22:25:41 - Database lookup.0 - Stopped while putting a row on the buffer.

The problem appears to be with coercing encoding types between the csv file and the database? Maybe??

I found this thread in the Pentaho archives, but I'd like to bring this up over here:
forums.pentaho.com/archive/in...p/t-67651.html

I have not been able to modify the connection string as suggested, so I am interested to know if there is a Zen Cart based solution for this.

Please help!

28 Nov 2010, 4:56 AM
#2
drbyte avatar

drbyte

Sensei

Join Date:
Jan 2004
Posts:
63,515
Plugin Contributions:
176

Re: Error: Illegal mix of collations using ETL Framework

That message usually happens when you're trying to join 2 tables that have different collations -- it can't do a compare of two values if their collations are different. That often happens when you add your own extra tables using a different collation rule than what was used to create the other tables already there.
You'll need to make sure the columns you're comparing are using the same collation/charset if you want to proceed. And, yes, that means you'll likely have to alter your new table fields accordingly. If you're using any extended characters in those tables, you may mangle those details if you're not careful. Be sure to make good backups before proceeding. There are many online tutorials on how to alter table structures and collations.
You may also find some of the Similar Threads listed below to be of aid.

.
Zen Cart - putting the dream of business ownership within reach of anyone!
Donate to: DrByte directly or to the Zen Cart team as a whole

Remember: Any code suggestions you see here are merely suggestions. You assume full responsibility for your use of any such suggestions, including any impact ANY alterations you make to your site may have on your PCI compliance.
Furthermore, any advice you see here about PCI matters is merely an opinion, and should not be relied upon as "official". Official PCI information should be obtained from the PCI Security Council directly or from one of their authorized Assessors.

28 Nov 2010, 4:03 PM
#3
jonny75904 avatar

jonny75904

New Zenner

Join Date:
Jul 2010
Posts:
9
Plugin Contributions:
0

Re: Error: Illegal mix of collations using ETL Framework

Thank you for the feedback, I was able to resolve the problem - because I am reading from a csv file encoded with UFT-8 it is mostly coercible, but some rows contained chars that simply aren't coercible.

*You will most definitely want to back up your database before you start changing collation settings.

There are collation settings at the database, table, and column levels. These settings are NOT inherited, so if you change it at the database level, the change does NOT propagate down.

There are two ways to address the collation problems: Manually change the settings, or use an application.

The application can be found at: phoca.cz/phoca-changing-collation

Otherwise, in phpMyAdmin there are two places to edit collation:

  1. For databases and tables, it is in the Operations Tab
  2. For columns, select the table and view the Structure tab. Each column shows its collation type. Edit the column and you can change the collation type.

Here is sample script to create a table with different collation settings:
CREATE TABLE t1
(
col1 VARCHAR(5)
CHARACTER SET latin1
COLLATE latin1_german1_ci
);

ALTER TABLE t1 MODIFY
col1 VARCHAR(5)
CHARACTER SET latin1
COLLATE latin1_swedish_ci;