Future Japanese Language Pack Status, version 1.5.5 and beyond...

[gernot]

Note: solved issue of non-controllability of language by browser settings, by activating language sidebox.
Confirmed that categories and products all have English and Japanese settings available now, so all in all, pretty happy about the progress.
I need to check the contents of includes/languages/japanese.php and admin/includes/languages/japanese.php more to see if I need to add back any defines I chopped out, but apart from that, the langauges files are done.
The core files issue still remains, I'll post where I put each file (if I could find a working override) or if I had to resort to replacing core files directly.
Then, upload my work for others to make use of as well.

[gernot]

Progress report:

1. As part of handling the override requirements, I added a custom template override based on responsive_classic, and made sure that the English and Japanese appeared correctly.
2. I then installed the EZPages multili-language plugin, and made sure I could translate the EZPages, leading to a completely bilingual site (keeping backups of ezpages-related original files that had to be replaced by this plugin).
3. Then, the tagline was translated using a language override.
4. I've not swapped the name and surname in the core files for Japanese customization (those mention in previous comments, related to user registration and account management), so I will need to check how registering users works. If really needed for Japan use (i.e., not possible to work around with various templates), this will be a bit of a pain, since again it is not easily revertable.

So now what is left is to continue the overrides possible with the cutom template, and document as needed if any core files need to be replaced.

[gernot]

I attach the database changes and the extra configs needed for Japanese address book changes, zone definitions, and kana configuration. Contents of the ZIP file are as follows:

1. Japan-address-format.sql: when run, address Japanese address format and assigns it to use for country Japan (107).
2. Japan-address-additions.sql: when run, modified address_book, customers and orders table for Japanese standard, adds kana fields, and adds an admin configuration where languages requiring kana usage should be registered. Japanese is registered by default.
3. Japan-zones-en.sql: when run, adds the ISO 3166-2:JP code (JP01 through JP-47) and Engish name for the 47 Japanese prefectures.
4. zen-extra-configs.txt: contains the contents of set_kana_support.php, which should be placed in includes/extra_configures/set_kana_support.php. This extends definitions in application_top.php withouth modifying the file. Note I have changed the key to be FURIGANA_NECESSARY_LANGUAGES rather than countries, as the test is for browser language rather than country.

Using procedure definitions, I have tested that running the SQL files ("source filename" or "\. filename" from the mysql prompt, or with "mysql DBname < filename" from the command prompt) multiple times should cause no problems, so no duplicate entries (e.g., for zone definitions) will result.

[gernot]

One additional SQL file with additional settings for Japan.
This one includes the admin configuration for Privacy Policy agreement when contacting store owner, and adds the tax rate and tax zone definitions (assuming tax_rates id=1 and geo_zone id=1). Again, running multiple times should have no bad effects.

So now we have the following:
- language files translated and working
- cutom template defined for template overrides
- EZPages multi-language module installed and working (needs custom template)
- database changes and admin configs added for Japan usage

The program files to actually make use of the Japan-centric definitions and fields have been created already (see previous posts), and I will endeavour to use as much of the override system as possible (no notifiers written, just custom files for now, since virtually all the issues are with the construction of queries for addresses and names).

[mc12345678]

I've made my own address format addition SQL statement, to insert the new format and set for use by country Japan (countires_id=107), checking for existing ones with a procedure, without modifying the address_format table with unique keys.

# Japan address format addition if not already set

# (1) simple method based on Japanese localization in 1.5.1 assuming 8 will be the next available record in address_format (7 existing formats in 1.5.5e by default)

INSERT INTO address_format VALUES (8, '$firstname $lastname$cr$postcode$cr$state$city$cr$streets$cr$country$cr$telephone$cr$fax','$statename $city');
UPDATE countries SET address_format_id=8 WHERE countries_id=107;

# (2) more complex method that uses a precedure to insert a new address format for Japan only if the values of the two fields do not yet exist
#     Note this does not modify the address_format table to use unique keys (as would be required by DUPLICATE KEYS or IGNORE statements)
#     The update to the coutries table is done at the same time, using the values of the address format

DELIMITER $$
DROP PROCEDURE IF EXISTS insert_jp_address_format $$
CREATE PROCEDURE insert_jp_address_format()
BEGIN
IF NOT EXISTS (SELECT * FROM address_format WHERE address_format = '$firstname $lastname$cr$postcode$cr$state$city$cr$streets$cr$country$cr$telephone$cr$fax' AND address_summary = '$statename $city') THEN
    INSERT INTO address_format (address_format,address_summary) VALUES ('$firstname $lastname$cr$postcode$cr$state$city$cr$streets$cr$country$cr$telephone$cr$fax','$statename $city');
    UPDATE countries SET address_format_id=(SELECT address_format_id FROM address_format WHERE address_format = '$firstname $lastname$cr$postcode$cr$state$city$cr$streets$cr$country$cr$telephone$cr$fax' AND address_summary = '$statename $city') WHERE countries_id=107;
END IF;
END $$
DELIMITER ;

CALL insert_jp_address_format();

# Utility/Reference:
#
# Direct update:
# update countries set address_format_id=(select address_format_id from address_format where address_format = '$firstname $lastname$cr$postcode$cr$state$city$cr$streets$cr$country$cr$telephone$cr$fax' and address_summary = '$statename $city') where countries_id=107;
#
# To get back to original (address_format_id=1):
# update countries set address_format_id=(select address_format_id from address_format where address_format = '$firstname $lastname$cr$streets$cr$city, $postcode$cr$statecomma$country' and address_summary = '$city / $country') where countries_id=107;

Had seen this post earlier and had thought then to reply, but you've been so busy I didn't want to introduce too much noise.

My recommendation regarding #1 and #2 above is to go more towards #2. Additionally, if you were to incorporate an auto-installer into this process you could use more php related functions/queries to gather the information possibly "easier" than pushing against the database. Whatever works though. :) The reason I recommend something like #2 is try to think about what it is you're doing. You are trying to make one language system more compatible and easier to work with ZC, there are other languages/destinations that may have the same situation. By reading and writing to the database rather than casting a hard value, you are making this plugin/language more pliable so that there is not a conflict between this one and another. It is a win-win.

[mc12345678]

After reading up on notifiers/observers, I tried to find a notifier to attach in include/application_top.php. However, there does not appear to be one.
On the other hand, it seems I could use includes/extra_configures for exactly this purpose?
So I made a new file as follows, using the (slightly corrected spelling) logic from the Japanese localization for now. I don't know what to put in the header for the package, constants does not seem right, so I made up "langauges" until I learn what would be best:

<?php
/** 
 * define for use with Japanese: furigana support based on language chosen
 *
 * @package languages
 * @copyright Copyright 2003-2005 Zen Cart Development Team
 * @license http://www.zen-cart.com/license/2_0.txt GNU Public License V2.0
 * @version $Id: media_manager.php 2842 2006-01-13 06:21:11Z drbyte $
 */
/**
  * is Furigana necessary?
**/
if (defined('FURIGANA_NECESSARY_COUNTRIES') &&
  !is_bool(strpos(strtolower(FURIGANA_NECESSARY_COUNTRIES), strtolower($_SESSION['language']))))
  define('FURIGANA_NECESSARY', true);
else
  define('FURIGANA_NECESSARY', false);

If this is correct, then this extra file would be added to the Japanese language pack now in preparation, and application_top.php has no changes.
For the code where FURIGANA_NECESSARY is used, I don't know how to handle that yet, but if it can be done by notifiers that would of course be great.

As an aside, the Japanese localization mades some changes to the database fields too for kana support obviously, but also for privacy of customer's telephone number:

# add telephone number to address, but remove from private information
ALTER TABLE address_book ADD COLUMN entry_telephone varchar(32) NOT NULL;
ALTER TABLE address_book ADD COLUMN entry_fax varchar(32);
ALTER TABLE orders ADD COLUMN delivery_telephone varchar(32);
ALTER TABLE orders ADD COLUMN delivery_fax varchar(32);
ALTER TABLE orders ADD COLUMN billing_telephone varchar(32);
ALTER TABLE orders ADD COLUMN billing_fax varchar(32);
ALTER TABLE orders ADD COLUMN customers_fax varchar(32);
ALTER TABLE customers CHANGE customers_telephone customers_telephone VARCHAR(32);
ALTER TABLE orders CHANGE customers_telephone customers_telephone VARCHAR(32);

# Furigana support added
ALTER TABLE address_book ADD entry_firstname_kana     varchar(32) NOT NULL default '';
ALTER TABLE address_book ADD entry_lastname_kana      varchar(32) NOT NULL default '';
ALTER TABLE customers    ADD customers_firstname_kana varchar(32) NOT NULL default '';
ALTER TABLE customers    ADD customers_lastname_kana  varchar(32) NOT NULL default '';
ALTER TABLE orders       ADD customers_name_kana      varchar(64) NOT NULL default '';
ALTER TABLE orders       ADD delivery_name_kana       varchar(64) NOT NULL default '';
ALTER TABLE orders       ADD billing_name_kana        varchar(64) NOT NULL default '';

Is this revertable at all, if one decides to use Japanese language with kana support (and Japan-centric privacy setting of telephone number)? Once data is entered, I presume this is not removable again if one uninstalls the language pack. It would be up to the user to decide how to dispose of the data first?

I saw this one as well...

Privacy concerns??? At what stage or what portion of the overall operation is privacy being protected? By this I mean... Is it that the data should never be collected? Is it that it should not be displayed to a user when they are logged in? Is it that a store should never know the information? If the above (and I think a long time ago I had attempted installation of this language pack just to see what it did and was a little "shocked") is intended to wipe some or all of the data from the server, well there is a problem then. The problem is that the store may operate in more than one environment. Clearing/removing existing data would have a negative impact on the existing collected data. Complete prevention of collecting data from any location in the future would have the same concern. The expectation of the store's operation is that when the service is properly operated and maintained that the data not be available to others. If the desire or expectation is to not collect certain data because of where the person lives or where the item is being sent, then that should be controlled outside of the database and should be prevented from being added to the database when it meets those requirements.

If the concerns relate to the additional fields in the database, well, yes the fields would persist until they are removed (uninstall script which could be broken up to accommodate particular portions of the program so that historical data could be maintained for any collected). Another option which is a bit more extensive is to use/create tables outside of the above tables so that they are independent of the remainder of the store's code. If the store were operated only in Japanese and every entry required the kana style fields, then that would be useless, but in maintaining a database, the ideal situation is that every field in a table has a purpose and is used all of the time. If it is not, then the question arises of if it should be in a separate or linked table such that it only receives information pertinent to its intent. For example (and possibly one of the most disliked) the products_attributes table contains all of the attributes for all of the products, but not all products have attributes... So there is not an entry for every possible product, but there is an entry for every product that has attributes...

The two commands that concern me most are:

ALTER TABLE customers CHANGE customers_telephone customers_telephone VARCHAR(32);
ALTER TABLE orders CHANGE customers_telephone customers_telephone VARCHAR(32);

Which in a ZC 1.5.5 store is not necessary for the table itself (already defined that way) and I haven't done any research on the effect of executing that on an existing database... I would have thought nothing would happen, but...

[gernot]

Hello,
Many thanks for your feedback and advice. Regarding using procedures to read/write to the DB, agreed 100%, just need(ed) to spend the time to get something working reliably. I don't have any experience with auto installers and PHP so for now I'll try to cover everything with the direct SQL statement methods, but in future, for example for 1.5.5 upgrades or the 1.6 version support, I should think about that. I suppose the plugins I already used (EZPages) had an auto installer in it somewhere, but I simply followed the instructions for putting files somewhere, then ran install from the admin console. Will look at that at a later stage, once I know exactly what I want to achieve.

To your next message, I don't know the importance of the changes for Japan, so I cannot say whethere there was legal advice involved, but this is what the 1.5.1-jp language installer does. I wrote it a bit more clearly in the latest attachment, it basically forces the user to add a telephone number (not NULL) to the address book, but removed the restriction from customers and orders (altering the table to allow NULL there - the two lines you quoted). I haven't got the stage of testing users and orders, but I imagine it is something to do with what information is automatically output from the orders and customers table when doing orders/shipping actions, maybe emails sent? So it is not about collection of that data or not, only which table it is stored in.

Thanks for the advice on the implications of adding fields to tables, and how perhaps to avoid that. I am a bit stuck as I want to first get something working, and I think the hurdle is a bit much for me right now! Kana addition is not that much of a problem fortunately, and not even necesary to enter if a Japanese user does not want to.

[gernot]

Sorry if I miss something, more time tonight. I just wanted to add that in the changes to the core files required by kana support and the various address changes, I baulked at changing the order of first and last name. I don't know a good way to handle this for now, but there must be a better way than telling the user to enter last name in the first name field, and vice versa lol.

[gernot]

I've left the DB-related stuff till the weekend: there are afew more apparently convenient settings for shops in Japan, but I did not have time to write procedures for these yet (they are part of the localized zc_install SQL, I just did not add them to the previously uploaded Japan-settings-additions.sql).
To make it clear, I want to recreate the behaviour of the 1.5.1-jp localization before diving into both the PHP code and DB tables and looking at how to best separate core code from localizations with separate tables and logic. I haven't any experience of ZenCart as yet, so it makes me feel more secure to start out with a working if somewhat blighted system.

I've now put all the customized core files in place as far as I am able without further work. For admin/ and includes/classes and subfolders of includes/modules, I created dummy override folders based on my customname (<customname>-dummy) to hold both the original and the modified files, while I replaced the original with the modified file in the original location.
Files that were treated in that manner were:

1. admin/customers.php
2. includes/classes/order.php
3. includes/modules/order_total/ot_cod_fee.php
4. includes/modules/pages/account_edit/header_php.php
5. includes/modules/pages/address_book_process/header_php.php
6. includes/modules/pages/contact_us/header_php.php

Files that could be overridden with the override system were:

1. includes/modules/checkout_new_address.php
2. includes/modules/create_account.php
3. includes/templates/template_default/templates/tpl_account_edit_default.php
4. includes/templates/template_default/templates/tpl_contact_us_default.php
5. includes/templates/template_default/templates/tpl_modules_address_book_details.php
6. includes/templates/template_default/templates/tpl_modules_checkout_new_address.php
7. includes/templates/template_default/templates/tpl_modules_create_account.php
8. includes/language/english/contact_us.php

Files that I had to edit to update to 1.5.5e standard (still need to do some final reviews to be sure):

1. includes/languages/japanese.php
2. admin/includes/languages/japanese.php

I discovered that there is a way to do overrides on classes and functions using the autoload/init_includes system, worth looking into later:
https://stevenkohlmeyer.com/zen-cart...php-overrides/

As mentioned previously, application_top.php could be extended with a file in extra_configures/, the language translation was completed (files not yet attached, sorry about that - will do once I have better-reviewed the two japanese.php files above), the buttons for the Japanese language are the same as in 1.5.1-JP so they could be copied across, and the DB modifications needed for the logic in the above core files are attached in previous posts (or one can get the bare bones from the Japanese 1.5.1-jp locaization SQL).

In the next few days, providing I have not broken the system, and barring any advice to do things differently, I will try to test the behaviour of user registration and email sending as part of figuring out how best to handle the Japanese name order.

[gernot]

Furigana support is working in the admin screen at least, as shown in this screenshot.
First and last names are correctly displayed for the test user. I haven't confirmed that the code does the right thing with the fields, but at least in the database I have kept the definitions from being swapped.
Pretty happy to see this screen!
(Note: I purposesly did not translate most things in the admin section [yet])