Strip HTML and WORD formatting

Does anybody know of a reliable way to strip out redundant html tags, inline styles and WORD formatting from product descriptions?

For example:

<span style="font-family="verdana, arial, sans-serif">MY CONTENT</span>

would become:

MY CONTENT

or

<p>MY CONTENT</p>
<p></p>
<p></p>

would become

<p>MY CONTENT</p>

or

<br /><br /><br />MY CONTENT

would become

<br />MY CONTENT

or

<SPAN lang=EN-IE style="mso-ansi-language: EN-IE">
<p class="MSO Normal">
<UL style="MARGIN-TOP: 0cm" type=circle><li class=MsoNormal style='mso-list:l3 level1 lfo3;tab-stops:list 36.0pt'>
<o:p>MY CONTENT</o:p></li></p></SPAN>

would become

<p>MY CONTENT</p>


I found this but not sure how to implement it - I've tried both and neither appear to work but then it could be something I am doing wrong. I would like to target '.mainDescription' content in product descriptions.

this link which explains it http://tim.mackey.ie/CommentView,guid,2ece42de-a334-4fd0-8f94-53c6602d5718.aspx

and also this code:

function cleanHTML(input) {
// 1. remove line breaks / Mso classes
var stringStripper = /(\n|\r| class=(")?Mso[a-zA-Z]+(")?)/g;
var output = input.replace(stringStripper, ' ');
// 2. strip Word generated HTML comments
var commentSripper = new RegExp('<!--(.*?)-->','g');
var output = output.replace(commentSripper, '');
var tagStripper = new RegExp('<(/)*(meta|link|span|\\?xml:|st1:|o:|font)(.*?)>','gi');
// 3. remove tags leave content if any
output = output.replace(tagStripper, '');
// 4. Remove everything in between and including tags '<style(.)style(.)>'
var badTags = ['style', 'script','applet','embed','noframes','noscript'];

for (var i=0; i< badTags.length; i++) {
tagStripper = new RegExp('<'+badTags[i]+'.*?'+badTags[i]+'(.*?)>', 'gi');
output = output.replace(tagStripper, '');
}
// 5. remove attributes ' style="..."'
var badAttributes = ['style', 'start'];
for (var i=0; i< badAttributes.length; i++) {
var attributeStripper = new RegExp(' ' + badAttributes[i] + '="(.*?)"','gi');
output = output.replace(attributeStripper, '');
}
return output;
}

I believe the CKEditor plugin has a button to "Paste from Word" which does this cleanup for you.

Ok I realise that. However this is a database of 100's, probably a 1000 products so I would rather skip editing 1000 products if I can which is why I am looking towards JQUERY or JAVASCRIPT, or even PHP

Do you think it could be done with this?

http://php.net/manual/en/function.strip-tags.php

added to

<?php echo stripslashes($products_description); ?>

in tpl_product_info_display.php

No. Running strip_tags would remove ALL your formatting, including intentional formatting.

Sounds like the problem is in your original data, not a Zen Cart issue. So, better to go back to the source of your original data. Clean that up, then re-import.

Nick1973:

Does anybody know of a reliable way to strip out redundant html tags, inline styles and WORD formatting from product descriptions?

...

This is a real hack and probably wrong headed in many ways - but I had similar issue once, and did the following to do some cleanup:

To remove extraneous formatting from all products I exported copy of db, opened it in Notepad++ , went down to bottom of db where product descriptions are and did a ‘find’ on the offending formatting and 'replace' with nothing. I took a (big) chance and did a 'replace all'. Things like 'find <br /><br ><br />' and 'replace with <br />' also worked. When done I saved that db, dropped the original db in phpMyAdmin (after backing up), then imported the changed db. It worked for me. I found a few products that needed tweaking but this saved a huge amount of time overall.

I'm not recommending this approach. Only noting its how I muddled through a similar problem.

Ok Dr Byte, however this is also a common issue when clients try to copy and paste data direct from MS Word. Regardless of how much you try to train them not to, they will always do something different. I'm fully aware the problem exists in the original data, but it would take me an age to go through every single product and strip out the code by searching and deleting. It's not a workable option. And neither is copying and pasting the text for each product through the CKEditor plugin button "Paste from Word". I could be at it for weeks, probably very thin from malnutrition, and on the verge of insanity.

The idea is to eventually override any inline styles with CSS too.

Both javascript and jquery do the tasks I am after, however I am not sure how they should be implemented.

Nick1973:

Ok Dr Byte, however this is also a common issue when clients try to copy and paste data direct from MS Word. Regardless of how much you try to train them not to, they will always do something different.

Not denying that.

Nick1973:

The idea is to eventually override any inline styles with CSS too.

Ideal for sure.

Nick1973:

Both javascript and jquery do the tasks I am after, however I am not sure how they should be implemented.

I'm not sure how javascript and jquery can be used to clean up the data in your database. Are you planning to write a script to loop through every record, from a jquery task running in your browser?

Please expand more on what you envision here ...

The scripts already exist. I'm not after a script to rewrite/clean the entire database. I can most likely override spans with CSS, that is fine and I am aware of how that can be done.

The scripts I posted earlier on in this conversation are third party scripts which are supposed to seek out certain html/word elements and disable them, however they are supposed to keep everything in between those elements. However I could not get these to work with Zen Cart. They do not rewrite the content, just strip away/disable certain tags.

Okay. So you've got some scripts. The stuff you posted earlier is javascript.

You said you don't want to clean the entire database.

But you said you don't want to edit each product manually.
So that means you *do* want to clean all of them in some automated way.

But javascript code such as you've posted is designed to run in the browser. And the browser has no access to the database.
So you need a script to pull each record from the database individually, put it in your browser, and then perform some sort of cleaning on the data, and then send the data back to your store's admin to save the changes back to the database.

Am I correct that you're asking one of us to provide all of that for you?

DrByte:

Okay. So you've got some scripts. The stuff you posted earlier is javascript.

That is correct.

DrByte:

You said you don't want to clean the entire database.

Correct

DrByte:

But you said you don't want to edit each product manually.
So that means you *do* want to clean all of them in some automated way.

Correct

DrByte:

But javascript code such as you've posted is designed to run in the browser. And the browser has no access to the database.

Correct

DrByte:

So you need a script to pull each record from the database individually, put it in your browser, and then perform some sort of cleaning on the data, and then send the data back to your store's admin to save the changes back to the database.

Not correct

DrByte:

Am I correct that you're asking one of us to provide all of that for you?

No I am not asking you to provide that as obviously I would be asking you to clean the entire database and I'm sure you would be submitting an invoice as a result.

I think your missing the point.

I'm not asking for a script to pull each record from the database individually, put it in your browser, and then perform some sort of cleaning on the data, and then send the data back to your store's admin to save the changes back to the database.

The scripts I posted are browser side only, they will not save back to the database, I'm aware of that and I do not want them to.

However, these scripts are supposed to recognise word formatting styles, disable them and leave everything in between so what appears in the browser to the end user (the viewer) is cleaner text/content that doesn't appear to have word formatting and is overridden with a mixture of a script and CSS. Even though in fact if you looked at the source code, the word formatting styles would still be in the code.

Oh, so you don't want clean data in your database. You want to fudge the output to the browser, so that every time a customer visits your site the data is re-cleaned in front of them, while they wait for the page to display.

Sorry, I've never heard anybody say they want to leave the mess in their system and bandage it every time the customer consumes it, over and over again.

But, ya, you can do that if you like.

To add any javascript into your site, simply put the javascript in an appropriately-named file, and add an event (onload perhaps) to fire it. Most of that is outlined here: https://www.zen-cart.com/wiki/index.php/Customisation_-_Templates_-_Javascript

Why the sarcasm? I seem to get this everytime in some form, from you for some reason Dr Byte. It would be nice to get straightforward answers though. I appreciate your higher authority but it just feels like sometimes your having a go at me for asking a question? I wouldn't be asking if I knew the answer.

And really you haven't answered the original question anyway anyway as you have just told me to do what I have tried already.

Nick1973:

Why the sarcasm? I seem to get this everytime in some form, from you for some reason Dr Byte.

Sorry. It's really nothing personal. I've been battling a really challenging project today, and taking personal attacks from a crew of jerks. I'm terribly sorry it's spilled off on you. I'm "escaping" the madness by answering forum posts.
Thanks for voicing it. I'm not proud of treating you like that.

Nick1973:

And really you haven't answered the original question anyway anyway as you have just told me to do what I have tried already.

We've gone around in circles because I assumed you wanted to clean the database, cuz that's the most common kind of request I see. But, now that I think we're seeing it the same way, I have some ideas:

I haven't set up a site to test your exact situation. But this is how I'd tackle it (although I'd prefer to clean the database instead):

I'm gonna assume that the javascript you quoted for doing the cleaning is trustworthy and properly functional to do the task required.

a) get the javascript for the cleanup function into the page. This can be done several ways. Two common ways are to either paste it into the HTML of the template file, such as tpl_product_info_display.php (above where the description is output), or to put it in a jscript_xxxxx file as described in the article I linked to (so that it loads in the <head> of the page).

b) then, somewhere on the page, at a point after the cleanup function is loaded into the page's HTML, add some javascript to "fire" the cleanup. I'd probably put that in the tpl_product_info_display.php after the output of the product description field.
Perhaps this sort of code might work:

<script type="text/javascript">
// get all elements using this class name
var x = document.getElementsByClassName("mainDescription");
var i;
// loop thru all the found elements, and replace them with the cleaned version
for (i = 0; i < x.length; i++) {
    x[i].innerHTML = cleanHTML(x[i].innerHTML);
}
</script>

Nick1973:

Why the sarcasm? I seem to get this everytime in some form, from you for some reason Dr Byte. It would be nice to get straightforward answers though. I appreciate your higher authority but it just feels like sometimes your having a go at me for asking a question? I wouldn't be asking if I knew the answer.

And really you haven't answered the original question anyway anyway as you have just told me to do what I have tried already.

Can't speak for other things, but the bolded information wasn't known until this final post. Another thing is that if the source data truly remains unfixed, then those that have disabled or can't run javascript will be subject to the "Word" data regardless, also in this era of pay by data, all of that extra data will be fed to the visitor only to be reduced locally, so becomes unnecessary information. It would almost be better to put such a script on the admin side (for future use) that would essentially prevent adding all of that, or to in some way enable a feature of ckeditor to not allow such content to be added (again more of a future proofing than a correct the present situation.)

Otherwise, and haven't actually looked at the detail of the located script, but if equivalent php could be identified, EP4 (discussed briefly in a thread by the OP) could be piggy backed upon to strip the errant information while exporting the data and then the resulting file reuploaded. (There are notifiers in the export that could support such data modification.)

DrByte:

Sorry. It's really nothing personal. I've been battling a really challenging project today, and taking personal attacks from a crew of jerks. I'm terribly sorry it's spilled off on you. I'm "escaping" the madness by answering forum posts.
Thanks for voicing it. I'm not proud of treating you like that.

Apology accepted, I get exactly the same with clients who also attack me. In all honesty I get days where I want to throw my machine through the window, so I do understand. I do already understand a vast amount about Zen Cart as a developer/designer compared to most, but there are situations like this where I need help to get me through, which is why I ask questions on the forum. If I can, then I will find a way without having to ask. However sometimes, I have no choice but to ask as it is sometimes quicker to see if someone else has come up against the issues to hand on here. While I am a developer/designer, I don't know everything and don't claim to. Good, clean communication always helps to get onto the matter to hand, and for a better relationship though! And the reason I mentioned 'everytime' is because I have had other situations where you have answered in ways which have a 'feel' to them that come across as probably not really wanting to answer or could be interpreted as 'your a developer, you should know this stuff'. As I said earlier though, I don't know everything and don't claim to know everything. Right, rant over, can we be friends now and keep it that way? and also get onto the matter to hand?

DrByte:

We've gone around in circles because I assumed you wanted to clean the database, cuz that's the most common kind of request I see. But, now that I think we're seeing it the same way, I have some ideas:

It might seem silly not to clean the database and I appreciate that some people don't have Javascript enabled. However the website I am working on is most likely going to be viewed by people who do have javascript enabled and I'd like to think that as Zen Cart already utilises Javascript and Jquery, I could use it without too much hassle. I'm already aware that using Javascript or Jquery or even PHP to serve the data as it is, but to the viewer in a cleaner form, which to them looks no different to the rest of the site, may not be the correct method. And the proper method would be to meticulously clean the database. However, I have alot of jobs to do that have a higher importance, and to sit and do that would take up vast amounts of time. Also I am not being paid to clean up the DB code in such a way. So I suppose yes, I am looking for a fudge or workaround. It isn't ideal, but if it saves time and does the job without causing too many issues then it is quicker solution. Maybe not correct, but if it does the job, it does the job!

DrByte:

I haven't set up a site to test your exact situation. But this is how I'd tackle it (although I'd prefer to clean the database instead):

I'm gonna assume that the javascript you quoted for doing the cleaning is trustworthy and properly functional to do the task required.

In all honesty, its a third party script which I found through google posted from others with similar issues on other websites such as wordpress based websites. So I haven't properly tested it, but then I couldn't get them to run and I couldn't see if it was something I was missing that I needed to change. So I suppose its a stab in the dark really.

DrByte:

a) get the javascript for the cleanup function into the page. This can be done several ways. Two common ways are to either paste it into the HTML of the template file, such as tpl_product_info_display.php (above where the description is output), or to put it in a jscript_xxxxx file as described in the article I linked to (so that it loads in the head of the page).

b) then, somewhere on the page, at a point after the cleanup function is loaded into the page's HTML, add some javascript to "fire" the cleanup. I'd probably put that in the tpl_product_info_display.php after the output of the product description field.
Perhaps this sort of code might work:

<script type="text/javascript">
// get all elements using this class name
var x = document.getElementsByClassName("mainDescription");
var i;
// loop thru all the found elements, and replace them with the cleaned version
for (i = 0; i < x.length; i++) {
    x[i].innerHTML = cleanHTML(x[i].innerHTML);
}
</script>

I already tried loading this into the head, but then I probably didn't fire it properly. So I will give that a try first before I come back to this. So now we are friends, if I need your help on this can I come back to you please?

Nick1973:

So now we are friends, if I need your help on this can I come back to you please?

For sure.


Note: The JS example I posted is not based on using jQuery. I didn't want to assume it was available. And sometimes native JS is much faster/simpler anyway.

mc12345678:

Can't speak for other things, but the bolded information wasn't known until this final post.

I think I probably mentioned it, but probably wasn't clear. That is my fault so sorry if it wasn't clear.

mc12345678:

Another thing is that if the source data truly remains unfixed, then those that have disabled or can't run javascript will be subject to the "Word" data regardless, also in this era of pay by data, all of that extra data will be fed to the visitor only to be reduced locally, so becomes unnecessary information.

Appreciate that, however we are talking about people who view websites in standard browsers and who wouldn't have a clue how to turn off javascript. This is a shop selling tradesmens products and they wouldn't know what javascript is so I don't see it as a problem. Also Javascript is used in Zen Cart to run other elements in the Zen Cart scripts. So if the data is a problem with word formatting, then it is probably also a problem elsewhere too.

mc12345678:

It would almost be better to put such a script on the admin side (for future use) that would essentially prevent adding all of that, or to in some way enable a feature of ckeditor to not allow such content to be added (again more of a future proofing than a correct the present situation.)

I agree, some sort of button to clean up the data in one sweep would be very useful.

mc12345678:

Otherwise, and haven't actually looked at the detail of the located script, but if equivalent php could be identified, EP4 (discussed briefly in a thread by the OP) could be piggy backed upon to strip the errant information while exporting the data and then the resulting file reuploaded. (There are notifiers in the export that could support such data modification.)

Not entirely sure what you mean, but I think probably your talking about a cleaner exported version of the data that could be imported back into the DB.

Yes I realise there are reasons to use one or the other. Sometimes as you say Javascript can be cleaner than Jquery and vice versa. However I think mc12345678 mentioned being able to clean the code from the admin side before serving it. I agree that would be a better option. Clients will always try and do things, us developers would have a fit over. Thats reality though, and you can never expect them to know that copying and pasting out of word is wrong. I have managed to train some clients in the past, but others just don't want to know. It doesn't matter how much you drum it into them, they will still do it.

Although it would be nice to have a button that would clean out Word html in such a way. I realise its locally run software, but dreamweaver has a script that does this so perhaps something similar could be applied into Zen Cart too. Personally, while I appreciate CKEditor has a paste from word facility, I don't always find CK Editor in Zen Cart reliable anyway.

There is a plugin for wordpress that is a major leap in this area though called WP Edit, and can be found here https://en-gb.wordpress.org/plugins/wp-edit

Perhaps Zen Cart could use it too?

However an independent clean up system would be nice!

Perhaps. I do see where you're coming from. And I don't disagree.

Here's my personal perspective on it though:
I never trust any Microsoft products to create reliable content for web pages. Been burned too many times. I just never use them, at least not for that.
For better or worse, it means I have limited exposure to those problems, and have little insight on the technical mess behind it. I've never understood why people insist on using the wrong tool for the job.
While sometimes the markup by CKEditor is less than perfect, it's a million times better than Word can do. It's designed for the web. Word was not.

One of the things that influences what makes it into the Zen Cart core is "who's gonna maintain" the feature. So, given that I never use Word for web stuff, I'm certainly not the guy to become the subject-matter-expert who will maintain that feature of the software as things evolve. There's already enough things that need more maintenance that get missed either due to amount of time required ... or subject expertise required.
That's why having people contribute plugins is helpful: those who are passionate about the feature and vested in keeping it current are the best contributors to such features.