Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Are you sure this is still true? I thought that bug was fixed a little while ago?

ADDENDUM:

I remembered discussing this a while back and I just found my old comment. Someone said "fun fact, gnu grep is slow with UTF8"[1] and I said "funner fact gnu grep was slow with UTF8"[2]. I reran the same grep that I used elsewhere in this discussion with C and with UTF8:

  root@fob-xray:lk# sync ; echo 3 > /proc/sys/vm/drop_caches  
  root@fob-xray:lk# declare -x LANG=C ; /usr/bin/time grep --exclude-dir=.git -r "leap second" . > /dev/null
  0.51user 3.04system 1:28.40elapsed 4%CPU (0avgtext+0avgdata 992maxresident)k
  0inputs+0outputs (3major+340minor)pagefaults 0swaps

  root@fob-xray:lk# sync ; echo 3 > /proc/sys/vm/drop_caches  
  root@fob-xray:lk# declare -x LANG=en_US.UTF-8 ; /usr/bin/time grep --exclude-dir=.git -r "leap second" . > /dev/null
  0.41user 3.44system 1:27.01elapsed 4%CPU (0avgtext+0avgdata 1100maxresident)k
  0inputs+0outputs (2major+367minor)pagefaults 0swaps
I think you must have a version of GNU grep before 2.7.3 or 2.7.1. The UTF problem seems to have disappeared. There is a decent amount of information in the debian bug report[3].

[1] https://news.ycombinator.com/item?id=2860932

[2] https://news.ycombinator.com/item?id=2862543

[3] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=604408



Your right -- I've been stuck with older RHEL 4 systems at work, and they run grep 2.5.something (although I'm not able to reproduce the problem at home now, maybe I'm not on an old enough RHEL 4 system -- will check when I get back to work). Problem disappears with the grep in RHEL 6. However there are still glitches in the sort command. For example:

   ls -lh |sort +4 -5 -h
doesn't work (at least on my RHEL 6.4 box), yet:

    ls -lh |LANG=C sort +4 -5 -h
does work.


Wow sort is misbehaving, but I cannot verbalize the problem. I ran your two ls invocations on the kernel source directory and diffed the output.

  --- /tmp/withutf 2013-07-09 21:03:23.060064079 -0400
  +++ /tmp/withC 2013-07-09 21:03:30.069578205 -0400
  @@ -1,26 +1,26 @@
  total 544K
  -rw-r--r-- 1 dfc dfc 252 Jul 9 19:27 Kconfig
  -rw-r--r-- 1 dfc dfc 2.5K Jul 9 19:27 Kbuild
  -drwxr-xr-x 113 dfc dfc 4.0K Jul 9 19:27 drivers
  +drwxr-xr-x 2 dfc dfc 4.0K Jul 9 19:27 init
  +drwxr-xr-x 2 dfc dfc 4.0K Jul 9 19:27 ipc
  +drwxr-xr-x 2 dfc dfc 4.0K Jul 9 19:27 mm
  +drwxr-xr-x 2 dfc dfc 4.0K Jul 9 19:27 usr
  +drwxr-xr-x 3 dfc dfc 4.0K Jul 9 19:27 block
  +drwxr-xr-x 3 dfc dfc 4.0K Jul 9 19:27 virt
  +drwxr-xr-x 4 dfc dfc 4.0K Jul 9 19:27 crypto
  +drwxr-xr-x 9 dfc dfc 4.0K Jul 9 19:27 lib
  +drwxr-xr-x 9 dfc dfc 4.0K Jul 9 19:27 security
  drwxr-xr-x 11 dfc dfc 4.0K Jul 9 19:27 kernel
  drwxr-xr-x 12 dfc dfc 4.0K Jul 9 19:27 samples
  drwxr-xr-x 13 dfc dfc 4.0K Jul 9 19:27 scripts
  drwxr-xr-x 17 dfc dfc 4.0K Jul 9 19:27 tools
  drwxr-xr-x 22 dfc dfc 4.0K Jul 9 19:27 sound
  drwxr-xr-x 26 dfc dfc 4.0K Jul 9 19:27 include
  -drwxr-xr-x 2 dfc dfc 4.0K Jul 9 19:27 init
  -drwxr-xr-x 2 dfc dfc 4.0K Jul 9 19:27 ipc
  -drwxr-xr-x 2 dfc dfc 4.0K Jul 9 19:27 mm
  -drwxr-xr-x 2 dfc dfc 4.0K Jul 9 19:27 usr
  drwxr-xr-x 32 dfc dfc 4.0K Jul 9 19:27 arch
  drwxr-xr-x 36 dfc dfc 4.0K Jul 9 19:27 firmware
  -drwxr-xr-x 3 dfc dfc 4.0K Jul 9 19:27 block
  -drwxr-xr-x 3 dfc dfc 4.0K Jul 9 19:27 virt
  -drwxr-xr-x 4 dfc dfc 4.0K Jul 9 19:27 crypto
  drwxr-xr-x 55 dfc dfc 4.0K Jul 9 19:27 net
  drwxr-xr-x 73 dfc dfc 4.0K Jul 9 19:27 fs
  -drwxr-xr-x 9 dfc dfc 4.0K Jul 9 19:27 lib
  -drwxr-xr-x 9 dfc dfc 4.0K Jul 9 19:27 security
  +drwxr-xr-x 113 dfc dfc 4.0K Jul 9 19:27 drivers
  -rw-r--r-- 1 dfc dfc 7.4K Jul 9 19:27 REPORTING-BUGS
  drwxr-xr-x 101 dfc dfc 12K Jul 9 19:27 Documentation
  -rw-r--r-- 1 dfc dfc 19K Jul 9 19:27 COPYING
There were differences in the output but I am not sure if that is a bug or if it has to do with locale rules for sorting. What do you think is broken with sort?

There seems to be a bug regarding utf and sort in debian[1] but I am not sure if it is the same problem. Do you know if there is a bug in redhat's bugzilla for the issue? Up until this thread I did not realize sort behaved differently depending on the locale.[2] Are you sure its not a difference in locale expectations on how strings are sorted?

[1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=695489

[2] http://stackoverflow.com/questions/5909404/sort-not-sorting-...


The -h option is supposed to sort numerically when the numbers are in "human readable" format (i.e., 42k instead of 42137), as in the matching -h option for ls (and other commands). And the "+4 -5 -h" will sort on the 5'th column. It works great if you aren't using -h (sort +4 -5 -n, with a regular "ls -l" command). Also, works if the number is in the first column (as in: du -h |sort -h).


One is sorting "intelligently" numerically, the other is just comparing ascii values, on the first numerical field.


Thank you for indulging our rather off topic curiosity. Just so I am clear are you saying the reason for the difference in sorting has to do with locale interpretations and is not a bug in sort?




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: