atom feed28 messages in org.xwiki.users[xwiki-users] problem with Chinese
FromSent OnAttachments
ning wangMar 16, 2007 7:32 am.html
Sergiu DumitriuMar 16, 2007 8:00 am.htm, .sql
Vincent MassolMar 16, 2007 8:24 am.htm
ning wangMar 16, 2007 8:27 am.html
Sergiu DumitriuMar 16, 2007 8:42 am.htm
Vincent MassolMar 16, 2007 8:44 am.html
ning wangMar 16, 2007 9:10 am.htm
ning wangMar 16, 2007 9:12 am.html
Gilles SerassetMar 16, 2007 9:24 am.htm
Sergiu DumitriuMar 16, 2007 9:30 am.html
Sergiu DumitriuMar 16, 2007 10:03 am.htm
jimmy at comkraft.comMar 16, 2007 10:12 am 
ning wangMar 16, 2007 10:17 am.html
Gilles SerassetMar 16, 2007 10:18 am.html
ning wangMar 16, 2007 10:38 am.htm
ning wangMar 16, 2007 11:27 am.html
ning wangMar 19, 2007 9:32 am.html
beyMar 19, 2007 12:28 pm.htm
Youcef BEYMar 19, 2007 4:23 pm.html
Jackson WangMar 20, 2007 7:55 pm.htm, .properties, .png
ning wangMar 20, 2007 8:03 pm.html
Vincent MassolMar 21, 2007 12:08 am.htm
Sergiu DumitriuMar 21, 2007 7:27 am.html
ning wangMar 21, 2007 10:15 am.htm
Sergiu DumitriuMar 21, 2007 10:25 am.htm
Jackson WangMar 22, 2007 2:36 am.html
ning wangMar 22, 2007 8:38 am.htm, .jpg
ning wangMar 22, 2007 8:51 am.htm
Subject:[xwiki-users] problem with Chinese
From:Youcef BEY (youc@imag.fr)
Date:Mar 19, 2007 4:23:49 pm
List:org.xwiki.users
Attachments:
attachment-0001.html - 11k

Hi Wang,

This problem have also took for longtime to solve it (in fact, I faced it before and hope that it correspond to yours). So I am using multilingual data in my XWiki (Arabic, Chinese, English, French and Japanese) and everything is working well with the UTF-8 encoding (MySQL, Tomcat, XWiki).

I read all your emails and responses. It seems that someone adviced you to check again the database encoding. Yes, the problem is related to the encoding of char in the database and it is simple to solve. In fact, it seems that MySql display that it's char encoding is in UTF-8 but you need also to add the following lines to my.cnf :

[client] default-character-set=utf8 [mysqld] default-character-set=utf8 character-set-server = utf8 collation-server = utf8_general_ci

Also check the following parameters for your Tomcat UTF-8 encoding (for linux, check catalina.sh):
-----------------------------------------------------------------------------------------------------

echo off rem set LANG=fr_FR.ISO8859-1 set LANG=en_US.UTF-8 set JETTY_PORT=8080 set JETTY_HOME=. java %JAVA_OPTS% -Djetty.port=%JETTY_PORT% -Djetty.home=%JETTY_HOME%
-Dfile.encoding=UTF-8 -jar %JETTY_HOME%/start.jar

Add the following variables to your system ----------------------------------------------------------- LANG = en-US_UTF8

Change the XWiki encoding : ----------------------------------------- (i) XWiki.xml encoding = UTF-8 (ii) XWiki.cfg CharSet = UTF-8

If this configuration work well, so we need after that to write (as suggested) a document for helping new users.

Please let us know whether it work?

----- Original Message ----- From: ning wang To: xwiki-users at objectweb.org Sent: Friday, March 16, 2007 11:27 AM Subject: Re: [xwiki-users] problem with Chinese

Hi Jimmy,

I tried again, and I think I found something more.

I use the xwiki without any change about encoding, just using default encoding ISO-8859-1. Use Jetty and HsqlDB. I write Chinese twice. Then roll back. The words can display correctly. But the only problem is that if I use Wiki text editor, I found that Chinese characters show like %D6%D0%CE%C4. It's can work in both view and roll-back. Therefore, I think the charaters can saved correctly in the database using ISO-8859-1. And it gose alone well with the WYSIWYG editor. But it's not convenient at all as I can't see any Chinese. So I think the ISO-8859-1 can't display Chinese characters in xwiki editor.

But if I instead ISO-8859-1 with UTF-8 int the web.xml xwiki.cfg and velocity.properties. Roll-back problem came again. But Chinese show properly int xwiki ext editor.

Hope to help some.

Thanks, Sincerely, Wang Ning

2007/3/17, jimmy at comkraft.com <jimmy at comkraft.com>: I tried to creat/modify a couple of pages in Chinese and I didn't see any problems when rolling back. I tried to use both the WYSIWYG editor and the wiki editor but both seems to work for me.

I am using 1 Beta 4 on a Windows 2000 computer Sp4 (locale is English). The database is PostgreSQL 8.2.3 I use UTF-8 for the DBs but didn't change any other settings.

I am using the following tool to input Chinese (http://www.3d114.com/tools/gb_pinyin.htm ) and then paste them into my xwiki. You may want to try putting in Chinese there and paste them into your wiki to see if there are some other encoding issues in the input phase - as there are several internal formats for Chinese Characters (Big5, GBxxxx, UTF-8 etc).

Hope this helps.

-Jimmy

> On 3/16/07, Gilles Serasset <Gilles.Serasset at imag.fr> wrote: >> >> Hi Ning, >> I also have a utf8 instance and I cannot reproduce your bug even when >> mixing French and Japanese characters. I'm using postgres though, but >> mysql >> qhould work also. >> > > However, I can reproduce this, on a fully UTF-8 system: OS, disk, > database, > XWiki settings. > > It seems that when working with the document archive the characters are > broken. In the database everything is OK, so it's something internal. > > When did you set up your wiki to use UTF8 characters ? Maybe you created > the >> database with an ISO Latin 1 encoding and then change your wiki encoding >> after the DB creation ? >> >> What happens when you ask to see the modifications between the current >> version and an older one ? Do you correctly see the chinese chars of the >> older version ? >> >> And by the way, Sergiu, I think the velocity.properties parameter are >> just >> useless (as far as I could test) as the velocity macros are called using >> Strings and not using InputStreams, in fact, it could hurt a little when >> reading the xxx.vm files that are not encoded in UTF-8 (see discussion >> on >> the encoding of the code base...). >> > > Indeed, I did not notice anything when changing this. But I don't know > what > internal stuff might break if I don't set this. > > I think that this only affect the way .vm files stored on the disk are > read, > but these files are plain ASCII. > > Regards, Gilles, >> >> On 16 mars 07, at 16:27, ning wang wrote: >> >> I do everything you said. And I make sure that all encoding is UTF-8 or >> utf8. >> But I'm sad to find that it does not work, still "?????" when roll >> back. >> >> 2007/3/16, Sergiu Dumitriu < sergiu.dumitriu at gmail.com>: >> > >> > Some ideas: >> > >> > * Check that the tables are really using UTF-8: get a mysql shell, >> open >> > the xwiki database and type "select charset(xwd_content from >> xwikidoc;". If >> > it does not say UTF-8, then you need to change the encoding. I've >> attached a >> > script that tries to do that (and creates indexes, too) >> > >> > * Check that the mysql connections are set to UTF-8. "show variables >> > like 'character_set%';". If it says something else, you can either >> globally >> > set the mysql encoding to utf-8 (in the mysql configuration file, >> search the >> > mysql documentation), or configure hibernate to request this: >> > <xwiki-dir>/WEB-INF/hibernate.cfg.xml, add >> > <property name="connection.characterEncoding">utf8</property> >> > after: >> > <property >> name="connection.driver_class">com.mysql.jdbc.Driver </property> >> > >> > * Check that you set all the encoding parameters to UTF-8: >> > ** in WEB-INF/xwiki.cfg: xwiki.encoding=UTF-8 >> > ** in WEB-INF/web.xml: >> > <param-name>encoding</param-name> >> > <param-value>UTF-8</param-value> >> > ** in WEB-INF/velocity.properties: >> > input.encoding=UTF-8 >> > output.encoding=UTF-8 >> > >> > After this, it should work just fine. >> > >> > -- > http://purl.org/net/sergiu >

-- You receive this message as a subscriber of the xwiki-users at objectweb.org mailing list. To unsubscribe: mailto:xwiki-users-unsubscribe at objectweb.org For general help: mailto: sympa at objectweb.org?subject=help ObjectWeb mailing lists service home page: http://www.objectweb.org/wws

--------------------------------------------------------------------------------