"opkg upgrade" slow, duplicate entries in *.list files

[h.koenig]

root@om-gta02:~# opkg list_installed opkg
opkg - 0.1.4+svnr4640-r0.1 -

root@om-gta02:~# time opkg upgrade
Package ncurses-terminfo is already installed in root.

real 1m9.038s
user 0m56.890s
sys 0m7.390s

(this was much worse (2.5 to 3 minutes) before removing ncurses-terminfo.list -- see below!)

questions/bugs:

• why do *.list files grow with many duplicate entries ?

• why does this cause reading gtk+-fastscaling.list many times ?

• what's the problem with ncurses-terminfo ?

BTW, this was an OM2008.8 installation with all updates, now updated to testing.

with strace I noticed that one single pkg file list got opened 12294 times. once this was usr/lib/opkg/info/ncurses-terminfo.list which then I moved away and opkg upgrade was fast again. right now it's usr/lib/opkg/info/gtk+.list -- from strace output:

root@om-gta02:~# sort opkd.strace.open1 | uniq -c | sort -n | tail -2

1 syscall_983045(0x40305a70, 0x40305a70, 0x68c, 0x40306148, 0x40024060, 0, 0x60, 0xf0005, 0xffff566c, 0x608, 0x40024000, 0xbedcbaec, 0, 0xbedcb7c0, 0x40001fa8, 0x40001fc0, 0x20000010, 0x40305a70, 0, 0, 0, 0xd708, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) = 0

12294 open("usr/lib/opkg/info/gtk+.list", O_RDONLY) = 4

why does opkg (sometimes) open a pkg list so many times ? I kept this files in case someone wants to inspect them.

unfortuneately this time (re)moving /usr/lib/opkg/info/gtk+.list does not help to get opkg upgrade working fast again :-(

right now opkg upgrade runs 20 secs after opening usr/lib/opkg/info/gtk+-fastscaling.list and another 20 secs after usr/lib/opkg/info/xtscal.list (from strace -rtt -e open ...)

xtscal.list is the last list being read, so that's the "real" opkg work. (re)moving gtk+-fastscaling.list saves 20 secs of runtime!

checking gtk+-fastscaling.list I noticed that it's very long and some times are duplicates _many_ times:

sort gtk+-fastscaling.list | uniq -c | grep -v " 1 "

1024 /usr/bin/gdk-pixbuf-query-loaders

11264 /usr/lib/gtk-2.0/2.10.0/engines/libpixmap.so

11 /usr/lib/libgdk_pixbuf_xlib-2.0.so.0

so I "cleaned up" all *.list files with "sort -u" reducing from 29816 lines in all *.list files to 10371 lines!!

now ncurses-terminfo.list still got read 2424 times, and gtk+.list 8 times , running for ~2.5 minutes! without ncurses-terminfo.list it runs only 28 secs, gtk+.list does not matter in run time.

so removing the duplicate file names i *.list files reduced runtime at least from 60+ to 30- seconds.

I'd guess that the message

Package ncurses-terminfo is already installed in root.

is somehow related with the fact that ncurses-terminfo.list still is read 2k times taking ~2 minutes. the reason for reading gtk+-fastscaling.list many times seemed to be fixed by cleaning up *.list files...

[tick]

why do *.list files grow with many duplicate entries ?

• It's a bug in the hash_table.c hast_table did not check key hit after second entry. This had been solved at #4830.

why does opkg (sometimes) open a pkg list so many times ?

• It's a issue of the reference count scheme. I am planing to make it more lazy, and reduce the reading times. Still working on it.

Another issue that cause a file been written for many times is solved at #4829. (Set the flag back.)

Thank you for reporting this.

[tick]

solved at svn r4830