Ticket #2478 (new defect)

Opened 6 years ago

Last modified 2 months ago

linux 2.6.39: alsa state not correctly restored in resume

Reported by: lindi Owned by: openmoko-kernel
Priority: high Milestone:
Component: kernel Version:
Severity: blocker Keywords: kernel alsa suspend resume linux 2.6.39
Cc: Blocked By:
Blocking: Estimated Completion (week):
HasPatchForReview: no PatchReviewResult:
Reproducible: always

Description

I am using SHR 2.6.39-nodrm branch at 74c481b35be9bf9b7d74c481b35be9bf9b7d with config from http://build.shr-project.org/tests/jama/kernel-nodrm/config-2.6.39.gz

Steps to reproduce:
1) save the attached statefile as stereoout.state
2) alsactl restore -f stereoout.state
3) alsactl store -f step3.state
4) speaker-test -t sine
5) echo mem > /sys/power/state
6) alsactl store -f step6.state
7) speaker-test -t sine

Expected results:
3 & 6) alsa states look identical
4 & 7) sine wave can be heard

Actual results:
3 & 6) alsa states look identical
4) sine wave can be heard
7) nothing can be heard

More info:
1) After step7 I can fix the situation if I run

amixer -qc0 cset numid=27,name='Capture Filter Switch' off
amixer -qc0 cset numid=27,name='Capture Filter Switch' on

amixer -qc0 cset numid=24,name='Capture Switch' on,on
amixer -qc0 cset numid=24,name='Capture Switch' off,off

amixer -qc0 cset numid=67,name='ALC Mixer Mic2 Capture Switch' on
amixer -qc0 cset numid=67,name='ALC Mixer Mic2 Capture Switch' off

amixer -qc0 cset numid=59,name='Amp Spk Switch' off
amixer -qc0 cset numid=59,name='Amp Spk Switch' on

amixer -qc0 cset numid=92,name='Left Mixer Left Playback Switch' off
amixer -qc0 cset numid=92,name='Left Mixer Left Playback Switch' on

speaker-test -t sine

2) My guess is that kernel is caching some value and does not write it to hardware since it thinks it already has the right value. Toggling the controls back and forth thus can fix the issue.

Attachments

stereoout.state (16.3 KB) - added by lindi 6 years ago.

Change History

Changed 6 years ago by lindi

comment:1 Changed 6 years ago by lindi

Here are some results from systemtap during suspend/resume:

$ stap -e 'probe module("snd_soc_core").function("snd_soc_resume") { printf("%s %s\n", probefunc(), $$parms); }'

prints ""snd_soc_resume dev=0xc7af5c08"

$ stap -e 'probe module("snd_soc_wm8753").function("wm8753_resume") { printf("%s %s\n", probefunc(), $$parms); }'

does not print anything.

dmesg has "soc-audio soc-audio: resume work item may be lost"

Does this mean that the resume handling of wm8753 was completely skipped?


comment:2 Changed 6 years ago by lindi

The

"soc-audio soc-audio: resume work item may be lost"

part is probably fixed upstream in

http://mailman.alsa-project.org/pipermail/alsa-devel/2011-May/040185.html

but I don't think this explains why wm8753_resume never gets called

comment:3 Changed 6 years ago by lindi

lindi@ginger:~$ stap -v -e 'probe module("snd_soc_core").statement("snd_soc_suspend@/home/lindi/debian/debian-linux-2.6/linux-2.6-2.6.39/debian/build/source_armel_none/sound/soc/soc-core.c:1126") { printf("%s %d %x %d\n", pp(), $codec->suspended, $codec->driver->suspend, $codec->dapm->bias_level); }'
Pass 1: parsed user script and 74 library script(s) using 21340virt/12972res/2156shr kb, in 4090usr/290sys/4378real ms.
Pass 2: analyzed script: 1 probe(s), 4 function(s), 3 embed(s), 0 global(s) using 62840virt/14988res/3020shr kb, in 500usr/430sys/1482real ms.
Pass 3: translated to C into "/tmp/stapXzhuo1/stap_a0b7707bb6646bdaa8e34301146c98da_4336.c" using 63748virt/16496res/4084shr kb, in 990usr/210sys/1208real ms.
Pass 4: compiled C into "stap_a0b7707bb6646bdaa8e34301146c98da_4336.ko" in 52030usr/6460sys/58859real ms.
Pass 5: starting run.
module("snd_soc_core").statement("snd_soc_suspend@/home/lindi/debian/debian-linux-2.6/linux-2.6-2.6.39/debian/build/source_armel_none/sound/soc/soc-core.c:1127") 0 0 0
module("snd_soc_core").statement("snd_soc_suspend@/home/lindi/debian/debian-linux-2.6/linux-2.6-2.6.39/debian/build/source_armel_none/sound/soc/soc-core.c:1127") 0 bf09088c 3

shows that we have two codecs.

The first one has no ".suspend".

The second one has ".suspend" but its "dapm.bias_level" is set to 3 which is SND_SOC_BIAS_ON. This means that the codec->driver->suspend(codec, PMSG_SUSPEND); line is not executed and I guess same goes also for codec->driver->resume(codec);.

If I change bias_level to SND_SOC_BIAS_OFF using

#!/usr/bin/stap

probe module("snd_soc_core").statement("snd_soc_suspend@/home/lindi/debian/debian-linux-2.6/linux-2.6-2.6.39/debian/build/source_armel_none/sound/soc/soc-core.c:1126") {
  if ($codec->dapm->bias_level == 3) {
    $codec->dapm->bias_level = 0;
  }
}

probe module("snd_soc_wm8753").function("wm8753_suspend") {
  printf("%s\n", pp());
}

probe module("snd_soc_wm8753").function("wm8753_resume") {
  printf("%s\n", pp());
}

then wm8753_suspend and wm8753_resume are called and speaker-test works!

comment:4 Changed 6 years ago by lindi

comment:5 Changed 6 years ago by lindi

Interestingly if I load the "headset.state" then bias is 1 (STANDBY) and suspend and resume hooks do get called.

comment:6 Changed 6 years ago by lindi

It seems to be

amixer -qc0 cset numid=93,name='Left Mixer Bypass Playback Switch' on

that is enough to cause bias to rise:

wm8753_set_bias_level codec=0xc79770a0 level=0x2
wm8753_set_bias_level codec=0xc79770a0 level=0x3
}}

comment:7 Changed 6 years ago by lindi

Seems these calls to wm8753_set_bias_level come from

[ 3191.245000] [<bf17e7d8>] (wm8753_set_bias_level+0x0/0xb4 [snd_soc_wm8753]) from [<bf155424>] (dapm_pre_sequence_async+0x34/0xb4 [snd_soc_core])
[ 3191.245000] [<bf155424>] (dapm_pre_sequence_async+0x34/0xb4 [snd_soc_core]) from [<c005a1c0>] (async_run_entry_fn+0x7c/0x1b8)
[ 3191.245000] [<c005a1c0>] (async_run_entry_fn+0x7c/0x1b8) from [<c004ff68>] (process_one_work+0x11c/0x3d8)
[ 3191.245000] [<c004ff68>] (process_one_work+0x11c/0x3d8) from [<c00505b8>] (worker_thread+0x13c/0x478)
[ 3191.245000] [<c00505b8>] (worker_thread+0x13c/0x478) from [<c0054138>] (kthread+0x88/0x90)

and

[ 3191.260000] [<bf17e7d8>] (wm8753_set_bias_level+0x0/0xb4 [snd_soc_wm8753]) from [<bf155358>] (dapm_post_sequence_async+0x3c/0xd4 [snd_soc_core])
[ 3191.260000] [<bf155358>] (dapm_post_sequence_async+0x3c/0xd4 [snd_soc_core]) from [<c005a1c0>] (async_run_entry_fn+0x7c/0x1b8)
[ 3191.260000] [<c005a1c0>] (async_run_entry_fn+0x7c/0x1b8) from [<c004ff68>] (process_one_work+0x11c/0x3d8)
[ 3191.260000] [<c004ff68>] (process_one_work+0x11c/0x3d8) from [<c00505b8>] (worker_thread+0x13c/0x478)
[ 3191.260000] [<c00505b8>] (worker_thread+0x13c/0x478) from [<c0054138>] (kthread+0x88/0x90)

respectively.

comment:8 Changed 6 years ago by lindi

It seems

amixer -qc0 cset numid=93,name='Left Mixer Bypass Playback Switch' on

is actually not needed in stereoout.state at all since it is enough that only data from left dac reaches LEFTMIX.

In any case, wrong alsa state should not cause wrong data in cache.

comment:9 Changed 6 years ago by lindi

Thanks to broonie on #alsa-soc I think the solution is to remove the lines

        /* allow audio paths from the GSM modem to run during suspend */
        snd_soc_dapm_ignore_suspend(dapm, "GSM Line Out");
	snd_soc_dapm_ignore_suspend(dapm, "GSM Line In");
	snd_soc_dapm_ignore_suspend(dapm, "Headset Mic");
        snd_soc_dapm_ignore_suspend(dapm, "Handset Mic");

comment:10 Changed 6 years ago by lindi

I can confirm that removing the snd_soc_dapm_ignore_suspend calls fixes the bug.

comment:11 Changed 6 years ago by GNUtoo

The fix doesn't work for me:
On serial console(with apm -s in the middle):

root@om-gta02:~# speaker-test -t sine

speaker-test 1.0.24.2

Playback device is default
Stream parameters are 48000Hz, S16_LE, 1 channels
Sine wave rate is 440.0000Hz
Rate set to 48000Hz (requested 48000Hz)
Buffer size range from 2048 to 16384
Period size range from 1024 to 1024
Using max buffer size 16384
Periods = 4
was set period_size = 1024
was set buffer_size = 16384
 0 - Front Left
Time per period = 2.845838
 0 - Front Left
T[   96.155000] soc-audio soc-audio: resume work item may be lost
Write error: -86,Streams pipe error
[   96.260000] dma2: channel has nothing loaded

I applied that patch to meta-smartphone:

diff --git a/meta-openmoko/recipes-kernel/linux/linux-2.6.39/shr.patch b/meta-openmoko/recipes-kernel/linux/linux-2.6.39/shr.patch
index 65d087c..efa59e7 100644
--- a/meta-openmoko/recipes-kernel/linux/linux-2.6.39/shr.patch
+++ b/meta-openmoko/recipes-kernel/linux/linux-2.6.39/shr.patch
@@ -3,6 +3,7 @@ rebased on top of openmoko kernel repository (om-2.6.39-stable)
 
 https://gitorious.org/shr/linux/commits/shr-2.6.39-nodrm
 
+31bdfe1: ASOC: samsung: neo1973_wm8753.c: fix openmoko bug #2478
 b851f9f: glamo-mci: disable glamo_mci_pre_request & glamo_mci_post_request
 6e1d25a: shr-create-patch.sh: utility used to generate shr.patch for meta-smartphone
 352f236: wm8753: Workaround toolchain bug: http://gcc.gnu.org/bugzilla/show_bug.cgi?id=48783
@@ -2401,7 +2402,7 @@ index 8275a2f..8e3e56e 100644
  #endif /* __GLAMO_CORE_H */
 diff --git a/shr-create-patch.sh b/shr-create-patch.sh
 new file mode 100755
-index 0000000..b251946
+index 0000000..a09d4c2
 --- /dev/null
 +++ b/shr-create-patch.sh
 @@ -0,0 +1,12 @@
@@ -2413,9 +2414,9 @@ index 0000000..b251946
 +
 +https://gitorious.org/shr/linux/commits/shr-2.6.39-nodrm
 +
-+$(git log --pretty=format:"%h: %s" gitorious/om-2.6.39-stable..)
++$(git log --pretty=format:"%h: %s" d09e21824bacd5f9ac0285c14c6ddc2d79b96608..)
 +
-+$(git diff om-2.6.39-stable)
++$(git diff d09e21824bacd5f9ac0285c14c6ddc2d79b96608)
 +EOF
 diff --git a/sound/soc/codecs/wm8753.c b/sound/soc/codecs/wm8753.c
 index ffa2ffe..69c499a 100644
@@ -2441,7 +2442,7 @@ index ffa2ffe..69c499a 100644
  {
  	u64 Kpart;
 diff --git a/sound/soc/samsung/neo1973_wm8753.c b/sound/soc/samsung/neo1973_wm8753.c
-index 4f845fe..a3a1cc8 100644
+index 4f845fe..a7095d9 100644
 --- a/sound/soc/samsung/neo1973_wm8753.c
 +++ b/sound/soc/samsung/neo1973_wm8753.c
 @@ -19,6 +19,7 @@
@@ -2507,7 +2508,20 @@ index 4f845fe..a3a1cc8 100644
  	return 0;
  }
  
-@@ -527,6 +569,7 @@ module_init(neo1973_init);
+@@ -358,12 +400,6 @@ static int neo1973_wm8753_init(struct snd_soc_pcm_runtime *rtd)
+ 	snd_soc_dapm_disable_pin(dapm, "Headset Mic");
+ 	snd_soc_dapm_disable_pin(dapm, "Handset Mic");
+ 
+-	/* allow audio paths from the GSM modem to run during suspend */
+-	snd_soc_dapm_ignore_suspend(dapm, "GSM Line Out");
+-	snd_soc_dapm_ignore_suspend(dapm, "GSM Line In");
+-	snd_soc_dapm_ignore_suspend(dapm, "Headset Mic");
+-	snd_soc_dapm_ignore_suspend(dapm, "Handset Mic");
+-
+ 	if (machine_is_neo1973_gta02()) {
+ 		ret = neo1973_gta02_wm8753_init(codec);
+ 		if (ret)
+@@ -527,6 +563,7 @@ module_init(neo1973_init);
  
  static void __exit neo1973_exit(void)
  {
diff --git a/meta-openmoko/recipes-kernel/linux/linux_2.6.39.bb b/meta-openmoko/recipes-kernel/linux/linux_2.6.39.bb
index beebcec..0ff4433 100644
--- a/meta-openmoko/recipes-kernel/linux/linux_2.6.39.bb
+++ b/meta-openmoko/recipes-kernel/linux/linux_2.6.39.bb
@@ -1,6 +1,6 @@
 require linux.inc
 
-PR = "r6"
+PR = "r7"
 
 # Mark archs/machines that this kernel supports
 DEFAULT_PREFERENCE = "-1"

And build fso2-console-image and did the test mentioned above.

comment:12 Changed 6 years ago by GNUtoo

with echo mem > /sys/power/state I've:

root@om-gta02:~# speaker-test -t sine

speaker-test 1.0.24.2

Playback device is default
Stream parameters are 48000Hz, S16_LE, 1 channels
Sine wave rate is 440.0000Hz
Rate set to 48000Hz (requested 48000Hz)
Buffer size range from 2048 to 16384
Period size range from 1024 to 1024
Using max buffer size 16384
Periods = 4
was set period_size = 1024
was set buffer_size = 16384
 0 - Front Left
Time per period = 2.965543
 0 - Front Left
Time per period = 2.990143
 0 - Front Left
T[   70.315000] soc-audio soc-audio: resume work item may be lost
Write error: -86,Streams pipe error
[   70.430000] dma2: channel has nothing loaded
Write error: -32,Broken pipe

doesn't work means that at resume I've no more sound.

Denis.

comment:13 Changed 6 years ago by lindi

So you start speaker-test, suspend and resume -- you never stop speaker-test at all?

comment:14 Changed 6 years ago by GNUtoo

indeed that's what I did.
use cases:

  • a call arrives right when you're suspending => you start playing the ringtone and resume(manually for instance)
  • you forget to disable suspend while playing music.

however stopping the sound/music after resuming and retrying to play it doesn't make the play work again.

Denis.

comment:15 Changed 6 years ago by lindi

Ok, so this is possibly a new and different bug then.

I assume it still works if you follow the steps that I listed in the bug report?

comment:16 Changed 6 years ago by GNUtoo

I also tried that:

  • playback sound
  • suspend+resume no sound
  • stop playback no sound
  • play again and stop again no sound
  • suspend+resume again
  • play again sound

comment:17 follow-up: ↓ 19 Changed 6 years ago by lindi

My theory: wm8753_suspend does not get called since the audio path is busy. This means alsa state is not restored on resume either.

comment:18 Changed 6 years ago by GNUtoo

with the test procedure at the beginning, speaker test still can't be ear after suspend,
and here's the diff between step3 and step6:

root@om-gta02:~# diff -u step3.state step6.state 
--- step3.state
+++ step6.state
@@ -1117,4 +1117,16 @@
 			count 1
 		}
 	}
+	control.94 {
+		iface MIXER
+		name 'Media Volume'
+		value.0 255
+		value.1 255
+		comment {
+			access 'read write locked user'
+			type INTEGER
+			count 2
+			range '0 - 255'
+		}
+	}
 }

which is something normal under fso2 images(that's because of a software volume in /etc/asound.conf, which only appear after playing something).

Denis.

comment:19 in reply to: ↑ 17 Changed 6 years ago by GNUtoo

Replying to lindi:

My theory: wm8753_suspend does not get called since the audio path is busy. This means alsa state is not restored on resume either.

what should I do?
1)check if it's not called
2)how would I fix it then? still call it?

comment:20 Changed 6 years ago by GNUtoo

I did step 1 to 4 in a console
and step 5 to 7 in a new console,
which is not what the bug reporter intended....
I'll open a new bug...

Denis

comment:21 Changed 6 years ago by GNUtoo

I've verified that the original bug reported here is fixed by the fix reported here.

Thanks a lot.

Denis.

comment:22 Changed 3 months ago by Kennescoma

Daily Levitra <a href=http://zol1.xyz/low-cost-zoloft-online.php>Low Cost Zoloft Online</a> Order Cipro Online Prospecto Cialis <a href=http://cial1.xyz/mail-order-cialis.php>Mail Order Cialis</a> Crestor 10mg Viagra Cialis Impotenza <a href=http://viag1.xyz/cheap-viagra.php>Cheap Viagra</a> Propecia Calculo Find Generic Dutasteride In Canada Cash On Delivery <a href=http://kamagra.ccrpdc.com/purchase-generic-kamagra.php>Purchase Generic Kamagra</a> Buy Canine Prednisone Propecia 120 Day <a href=http://kama1.xyz/order-kamagra.php>Order Kamagra</a> Beer With Amoxicillin Levitra Posologie <a href=http://accutane.ccrpdc.com/accutane-buy-online.php>Accutane Buy Online</a> Ordering Alli Online Actos From India <a href=http://viag1.xyz/cheap-viagra-generic.php>Cheap Viagra Generic</a> Avorter Premier Mois Cafergot <a href=http://cial1.xyz/buy-cialis-online.php>Buy Cialis Online</a> Coupon For Free Combivent Inhaler Suhagra Pills <a href=http://kama1.xyz/buy-kamagra-online-cheap.php>Buy Kamagra Online Cheap</a> Cephalexin 750mg Propecia Hacienda <a href=http://kama1.xyz/buy-cheap-kamagra.php>Buy Cheap Kamagra</a> Siemens Levitra 20mg Lloyds Pharmacy Propecia <a href=http://zol1.xyz/order-generic-zoloft.php>Order Generic Zoloft</a> buy accutane online in canada Comprar Viagra Sin Receta En Valencia <a href=http://viag1.xyz/viagra-online-online.php>Viagra Online Online</a> Viagra Online Gunstig Viagra Overnight Best Price <a href=http://priligy.ccrpdc.com/dapoxetine-online-purchase.php>Dapoxetine Online Purchase</a> Viagra Frau Wirkung buy generic accutane uk <a href=http://zol1.xyz/order-zoloft.php>Order Zoloft</a> 100mg Kamagra A Buon Mercato Vente Viagra Livraison Gratuite <a href=http://kama1.xyz/order-cheap-kamagra.php>Order Cheap Kamagra</a> Aromasine Clomid Wirkung Von Viagra Forum <a href=http://kama1.xyz/best-generic-kamagra.php>Best Generic Kamagra</a> How To Buy Viagra In Germany Keflex And Pregnant Women <a href=http://cial1.xyz/mail-order-cialis.php>Mail Order Cialis</a> Buying Levitra In Mexico Buy Cheap Cialis Discount Online <a href=http://cial5mg.xyz/ordina-cialis-online.php>Ordina Cialis Online</a> Comparativa Viagra Cialis Amoxicilina Internet Saturday Delivery Visa Accepted <a href=http://cial5mg.xyz/buy-cialis-cheap.php>Buy Cialis Cheap</a> Medecine Misoprostol Achat Lioresal <a href=http://zol1.xyz/generic-zoloft.php>Generic Zoloft</a> Amoxicillin Clavulanate Acid Nolvadex Vente <a href=http://kama1.xyz/cheap-generic-kamagra.php>Cheap Generic Kamagra</a> Levitra Sans Ordonnance Keflex Antimicrobial Spectrum <a href=http://viag1.xyz/cheap-viagra-25mg.php>Cheap Viagra 25mg</a> Buy Elavil No Prescription Online Rx Canada <a href=http://cial5mg.xyz/cheap-cialis-tablets.php>Cheap Cialis Tablets</a> Viagra Homme Forum Cialis Y Cocaina <a href=http://viag1.xyz/viagra-samples.php>Viagra Samples</a> Amoxicillin Treatment Bronchitis Citalopram Without Perscription <a href=http://zol1.xyz/generic-zoloft-usa.php>Generic Zoloft Usa</a> Baclofene Commander Propecia Topical <a href=http://cial1.xyz/how-to-buy-cialis.php>How To Buy Cialis</a> Ampicillin Cvs Over The Counter Viagra Repas <a href=http://cial1.xyz/cialis-viagra.php>Cialis Viagra</a> Levitra Kopa Cialis Filmtabletten 5mg <a href=http://zithromax.ccrpdc.com/azithromycin-generic.php>Azithromycin Generic</a> Viagra Cialis Oder Levitra Gut Cheap Rx Pharmacy <a href=http://levitra.ccrpdc.com/generic-levitra-pill.php>Generic Levitra Pill</a> On Sale Online Direct Dutasteride Website Internet Fedex Celecoxib Online <a href=http://viag1.xyz/fast-delivery-viagra.php>Fast Delivery Viagra</a> Generic Propecia Propecia Nebenwirkungen Cialis 5mg <a href=http://viag1.xyz/where-to-order-viagra.php>Where To Order Viagra</a> Cialis One A Day Review Indian Pharmacy Online <a href=http://zol1.xyz/order-zoloft-online-usa.php>Order Zoloft Online Usa</a> Amoxicillin False Positive Drug Tests Retin <a href=http://zol1.xyz/zoloft-cost.php>Zoloft Cost</a> Cefaclor 1447 Erectile Dysfunction And Pe <a href=http://cial5mg.xyz/cialis-generic.php>Cialis Generic</a> Donde Se Puede Comprar Viagra Sin Receta Ceftin <a href=http://zol1.xyz/map.php>Zoloft Online Usa</a> Cialis 5 Mg Vs Viagra Comment Utiliser Le Tadalis Sx <a href=http://zol1.xyz/zoloft-online-no.php>Zoloft Online No</a> Finasteride Online Cheap Buy 10mg Levitra Online <a href=http://zol1.xyz/zoloft-prices.php>Zoloft Prices</a> Viagra Efectos Prolongados Propecia Proscar Finasteride 5 Mg <a href=http://viag1.xyz/order-viagra.php>Order Viagra</a> Best Uk Online Pharnacy Allergic Itching Due To Amoxicillin <a href=http://viag1.xyz/best-viagra-online.php>Best Viagra Online</a> Cialis ... Cheap.... 40mg Amoxicillin And Wikipedia <a href=http://cial1.xyz/buy-online-cialis.php>Buy Online Cialis</a> Ivermectin For Sale From Great Britain Zithromax Syphilis Treatment <a href=http://cial1.xyz/cialis-generic.php>Cialis Generic</a> Comment Pouvez Achat Cialis Fr <a href=http://cial5mg.xyz/mail-order-cialis.php>Mail Order Cialis</a> Viagra. Zithromax Effectiveness Chlamydia <a href=http://zol1.xyz/zoloft-generic.php>Zoloft Generic</a> Kamagra Mal De Altura

comment:23 Changed 3 months ago by ylq

http://www.michael-korsoutlet.cc/, http://bulls.nba-jersey.com/, http://pacers.nba-jersey.com/, http://nets.nba-jersey.com/, http://www.burberrys-outlet.in.net/, http://www.soccershoes.us.com/, http://www.puma-shoesoutlet.com/, http://www.retro-jordans.net/, http://chiefs.nfljersey.us.com/, http://www.polos-outlets.com/, http://www.chrome-hearts.com.co/, http://www.omegawatches.in.net/, http://www.adidas-shoes.ca/, http://www.new-balanceshoes.es/, http://www.basketballshoes.com.co/, http://www.nikestoreoutlet.us/, http://www.marcjacobs-outlet.com/, http://colts.nfljersey.us.com/, http://www.michael-kors.net.co/, http://www.jordan.com.de/, http://www.christianlouboutin-shoes.ca/, http://www.ray-bans.net.co/, http://www.hermesoutlet.shop/, http://www.oakleys-store.com.co/, http://www.bcbg-dresses.net/, http://www.rayban-sunglasses.co/, http://eagles.nfljersey.us.com/, http://www.oakleys-sunglasses.com.co/, http://raiders.nfljersey.us.com/, http://www.jimmy-chooshoes.com/, http://www.tnf-jackets.us/, http://www.polos-outletstore.net/, http://www.p90xs-workout.com/, http://www.nikeshoes.de/, http://www.barbour-jackets.com/, http://www.instylers.us/, http://www.north-face.com.co/, http://www.bcbg-maxazria.ca/, http://www.michaelkorsoutlet.se/, http://www.coachfactory.shop/, http://rayban.blackofriday.com/, http://www.oakley-sunglassessale.com.co/, http://titans.nfljersey.us.com/, http://www.mcms-bags.com/, http://www.tommy-hilfigers.in.net/, http://www.hollisters-canada.ca/, http://www.rolexwatchesforsale.us.com/, http://www.giuseppes-zanotti.com/, http://www.ralphlauren-polos.com.co/, http://www.kate-spades.com.co/, http://www.nikeshoes.org.es/, http://www.burberry-store.com/, http://thunder.nba-jersey.com/, http://www.michael-kors-outlet.us.org/, http://www.vans-shoes.co.uk/, http://grizzlies.nba-jersey.com/, http://www.longchampoutlet.com.co/, http://www.swarovski-online-shop.de/, http://www.toryburchsale.com.co/, http://www.toms-shoes-outlet.us/, http://www.nike-airmax.com.de/, http://www.newoutletonlinemall.com/, http://www.chiflatiron.net.co/, http://cavaliers.nba-jersey.com/, http://www.barbour-factory.net/, http://www.nikeairmax.com.de/, http://www.nike-schoenen.co.nl/, http://airmax.misblackfriday.com/, http://www.oakley-sunglasses2017.com/, http://www.polo-ralphlaurens.us.com/, http://www.oakleys-frame.com.co/, http://www.nikeshoesoutlet.org.uk/, http://heat.nba-jersey.com/, http://www.oakleys-outlets.net/, http://www.michael-kors.cc/, http://coach.euro-us.net/, http://www.nike-shoesoutlet.com/, http://www.michaels-kors.us/, http://www.hollister.com.se/, http://www.true-religion.com.co/, http://celtics.nba-jersey.com/, http://www.versace-outlet.com/, http://www.michael-kors.in.net/, http://www.tomsshoes-outlet.us.org/, http://www.ok-em.com/, http://dolphins.nfljersey.us.com/, http://www.michaelkors-canadaoutlet.ca/, http://www.adidas-schuheonline.de/, http://www.hermes-bags.net.co/, http://www.horlogesrolexs.nl/, http://www.vans-shoesoutlet.com/, http://www.beatsheadphone.in.net/, http://bucks.nba-jersey.com/, http://www.michaelkors-outletonline.cc/, http://www.oakley-outletonline.com.co/, http://www.nike-outlet.us.org/, http://www.christian-louboutins.in.net/, http://rockets.nba-jersey.com/, http://coach.blackofriday.com/, http://www.oakleys-outlet.net.co/, http://www.prada-shoes.com.co/, http://www.oakley-sbocco.it/, http://www.coach-factoryoutlet.net.co/, http://www.michaelskors-outlet.co.uk/, http://www.coachoutlet-online.com.co/, http://lions.nfljersey.us.com/, http://www.juicycouture.com.co/, http://azcardinals.nfljersey.us.com/, http://www.truereligion-outlet.us.org/, http://www.ralph-laurenoutletonline.com/, http://www.eyeglasses-outlet.com/, http://steelers.nfljersey.us.com/, http://www.swarovski-canada.ca/, http://www.zcoachoutlet.com/, http://www.swarovski-australia.com.au/, http://www.nfljersey.us.com/, http://www.jimmy-choosshoes.com/, http://www.cheap-pandoracharms.co.uk/, http://www.ray-bans.co.uk/, http://www.woolrich-clearance.com/, http://www.cheapthomas-sabos.org.uk/, http://broncos.nfljersey.us.com/, http://bears.nfljersey.us.com/, http://www.nike-huaraches.nl/, http://www.salvatore-ferragamos.com/, http://www.scarpehogans.it/, http://www.wedding-dresses.cc/, http://www.replica-handbags.com.co/, http://www.nike-store.com.de/, http://www.true-religions.com/, http://www.timberland-boots.com.co/, http://www.nikeairmax2015.org/, http://www.tommy-hilfiger.net.co/, http://www.swarovskissale.co.uk/, http://www.supra-shoes.org/, http://www.ferragamo.net.co/, http://www.nike-store.in.net/, http://www.new-balancecanada.ca/, http://coach-outlet.tumblr.com/, http://www.burberry-outlets.org.uk/, http://www.niketnrequinpaschers.fr/, http://www.salomonshoes.com.de/, http://www.the-northfacejackets.net.co/, http://www.toms-shoe.us.org/, http://www.co-aol.com/, http://lakers.nba-jersey.com/, http://www.cheapoakleyss.com.co/, http://www.timberlandshoes.net.co/, http://www.raybans-outlet.cc/, http://www.ralph-laurenpolosoutlet.com/, http://www.nike-skors.com.se/, http://www.nike-mercurial.com/, http://www.kates-spade.com/, http://www.airmax.com.se/, http://www.ed-hardy.us.com/, http://www.hoganshoes.org.uk/, http://www.ralphlaurens-outlet.co.uk/, http://www.northfacejackets.fr/, http://kings.nba-jersey.com/, http://www.the-northfaces.us.com/, http://www.mk-com.com/, http://www.ralphlaurenonlineshop.de/, http://www.cheap-raybansoutlet.com.co/, http://www.beatsbydrdrephone.com/, http://www.tommys-hilfigers.com/, http://www.pandorajewellery.com.au/, http://www.swarovski-crystals.com.co/, http://www.cheap-baseballbats.us/, http://www.the-northfaces.org.uk/, http://hornets.nba-jersey.com/, http://www.nba-jersey.com/, http://jets.nfljersey.us.com/, http://www.giuseppezanotti.com.co/, http://www.ralphlauren-au.com/, http://www.cheap-rayban.com.co/, http://www.toms-outletstore.com/, http://www.raybans.com.de/, http://www.outlet-burberry.net/, http://redskins.nfljersey.us.com/, http://www.jerseys-store.com/, http://www.adidasshoes.com.se/, http://www.adidas-store.net/, http://www.reeboks.com.de/, http://www.toms-shoesstore.com.co/, http://www.cheap-jerseys.mex.com/, http://www.coachoutlet-inc.net/, http://celine.blackofriday.com/, http://mavericks.nba-jersey.com/, http://www.rolexwatches-canada.ca/, http://www.nike-air-force.de/, http://www.raybans-glasses.com.co/, http://www.thenorth-face.com.co/, http://ravens.nfljersey.us.com/, http://www.raybans-outlet.com.co/, http://www.ferragamoshoes.in.net/, http://www.christianlouboutinshoesoutlet.org/, http://www.timberlandbootsoutlet.us.com/, http://www.michael-korsbags.com.co/, http://www.ralphlaurens.ca/, http://www.beats-by-dre.com.co/, http://www.michaelkorsoutlet.ar.com/, http://www.polos-ralphlauren.com.co/, http://www.polo-ralph-lauren.de/, http://www.armani-exchange.in.net/, http://www.calvin-kleins.in.net/, http://www.cheap-jordans.net/, http://www.soccers-shoes.net/, http://www.oakley-outlet.fr/, http://ralphlauren.blackofriday.com/, http://www.thomas-sabos.com.de/, http://www.cheap-watches.in.net/, http://www.toms--outlet.com.co/, http://raptors.nba-jersey.com/, http://www.philipp-pleins.com/, http://bills.nfljersey.us.com/, http://www.raybans-sunglasses.net.co/, http://www.nike-shoescanada.ca/, http://www.handbagsoutlet.in.net/, http://www.burberrys-outletonline.com/, http://www.michaelkorsoutlet-online.ar.com/, http://www.levisjeans.com.co/, http://www.nike-airmaxnc.co.uk/, http://www.mbt-outlet.com/, http://www.omegas-relojes.es/, http://www.newbalance-shoes.org/, http://coachoutlet.euro-us.net/, http://www.nike-free-runs.de/, http://www.nike-roshe-run.de/, http://www.asics-outlet.com.co/, http://www.poloralphlaurenoutlet.net.co/, http://timberwolves.nba-jersey.com/, http://www.long-champ.com.de/, http://chargers.nfljersey.us.com/, http://www.burberry-outletcanada.ca/, http://www.oakleys.mex.com/, http://www.nike-rosheruns.nl/, http://www.babyliss-pros.com/, http://www.katespades.com.co/, http://trailblazers.nba-jersey.com/, http://www.philipp-plein.us.com/, http://www.nikeair-max.ca/, http://www.nikerosherun.us/, http://www.swarovski-jewelrys.com/, http://nuggets.nba-jersey.com/, http://www.toms-outletonline.com.co/, http://www.adidas-shoes.es/, http://www.the-northfaces.net.co/, http://www.oakleys-outlet.com.co/, http://www.raybansbocco.it/, http://www.marc-jacobs.us.com/, http://www.coachblackfriday.com/, http://www.christian-louboutinshoes.in.net/, http://www.montrespaschers.fr/, http://www.michael-korsoutlet.co.uk/, http://www.longchamps.com.co/, http://clippers.nba-jersey.com/, http://knicks.nba-jersey.com/, http://www.oakleys.org.es/, http://www.nike-air-max.com.au/, http://www.designer-handbags.vip/, http://www.raybans-outlet.net.co/, http://www.cheap-mlbjerseys.us.com/, http://www.toms-outlets.us.com/, http://www.christian-louboutins.co.uk/, http://www.burberrybags.com.co/, http://www.michael-korsoutlet.top/, http://www.burberryonlineshop.de/, http://www.valentino-shoesoutlet.us.com/, http://www.roshe-runs.co.uk/, http://www.hugo-boss.com.co/, http://www.rosheruns.us/, http://www.coach-outletonline.ca/, http://rams.nfljersey.us.com/, http://www.ralphslaurenoutlet.us.com/, http://www.burberryoutlet-sale.com.co/, http://www.rolex-watches.us.com/, http://www.michael-kors-australia.com.au/, http://www.vansschuhe.com.de/, http://www.cheapjerseys.us.org/, http://jazz.nba-jersey.com/, http://www.nikefree-runs.org.uk/, http://www.fendi-outlet.in.net/, http://www.women-clothes.com/, http://panthers.nfljersey.us.com/, http://www.hollister-store.com/, http://www.cheap-nike-shoes.net/, http://browns.nfljersey.us.com/, http://www.lacoste-outlet.com.co/, http://seahawks.nfljersey.us.com/, http://www.pulseraspandora.com.es/, http://www.rolex-watchoutlet.com/, http://www.nike-max.fr/, http://www.nike-skos.com/, http://www.true-religion-jeans.com.co/, http://www.lauren-ralphsoutlet.co.uk/, http://www.tommyhilfigers.de/, http://www.hermes-birkinoutlet.net/, http://www.bottega-bags.com/, http://www.louboutin.jp.net/, http://www.raybans.org.es/, http://www.converse.net.co/, http://www.mlb-jerseys.us.com/, http://49ers.nfljersey.us.com/, http://falcons.nfljersey.us.com/, http://www.the-northface.ca/, http://www.michaelkors.co.nl/, http://www.beatsbydre.com.co/, http://www.barbours.us.com/, http://www.cheapjerseys.net.co/, http://www.hogans.com.de/, http://www.pandoracharms-canada.ca/, http://www.long-champbags.com/, http://www.mk-outletonline.us/, http://www.barbour.in.net/, http://www.michaelkorsoutletonline.net.co/, http://www.hollister-clothing.com.co/, http://www.chiflatirons.in.net/, http://www.raybans-outlet.org.uk/, http://www.raybans-cher.fr/, http://www.ralphs-laurenpolo.co.uk/, http://www.longchamp-bags.us.com/, http://www.ferragamos.us.com/, http://www.nikefree5.net/, http://www.oakley--sunglasses.com.au/, http://www.tory-burch.us.org/, http://www.ralph-laurenoutletonline.in.net/, http://www.prada-bagsoutlet.com/, http://www.cheap-michaelkors.com.co/, http://www.air-yeezys.com/, http://www.nikefree-run.net/, http://www.converse-shoes.net/, http://www.hollisterclothings-store.com/, http://www.jordanshoes.net.co/, http://www.toryburch-shoesoutlet.com/, http://jaguars.nfljersey.us.com/, http://www.pandora-jewelry.com.de/, http://www.michaelkors-store.us.org/, http://www.hollisters.us.com/, http://www.timberlandspaschere.fr/, http://www.oakley-sunglass.net.co/, http://www.the-northface.com.de/, http://www.coachsoutletonline.in.net/, http://www.dsquared2-outlet.com/, http://www.truereligion-outlet.com.co/, http://www.michael-kors.com.es/, http://www.bottega-venetasoutlet.com/, http://www.truereligions.net/, http://michaelkors.blackofriday.com/, http://www.adidas-shoes.nl/, http://spurs.nba-jersey.com/, http://buccaneers.nfljersey.us.com/, http://patriots.nfljersey.us.com/, http://www.nike-airmaxs.fr/, http://www.coachfactory.cc/, http://www.raybanoutlet.ca/, http://pistons.nba-jersey.com/, http://www.michaelkors-ins.com/, http://www.adidassuper-star.de/, http://www.ralphs-laurenpolos.net/, http://www.iphonecases.net.co/, http://www.longchampbags.com.co/, http://www.nikeair-max.es/, http://www.michaelkors.com.de/, http://www.asics-outlets.com/, http://texans.nfljersey.us.com/, http://www.tommy-hilfigers.de/, http://www.ralph-laurenpolos.co.uk/, http://www.pradas.com.de/, http://www.oakleys.com.de/, http://www.cheapshoes.net.co/, http://www.nike-shoesfactory.us/, http://www.christian-louboutin.jp.net/, http://www.cheap-rolex-watches.org.uk/, http://www.nikefreeshoes-inc.co.uk/, http://www.nike-rosherun.com.es/, http://www.under-armouroutlet.com/, http://www.givenchy.com.co/, http://www.mcmbackpacks.net/, http://warriors.nba-jersey.com/, http://www.supra-footwear.net/, http://www.nike-airmax.us.com/, http://www.burberry-handbags.net.co/, http://www.katespadeoutlet.gb.net/, http://www.montblanc-pens-sale.com/, http://www.celine-bags.com/, http://www.air-maxschoenen.nl/, http://www.airmax-shoes.com/, http://www.soccer-shoesoutlet.com/, http://www.toryburch-outlet.com.co/, http://www.rayban-sunglasses.fr/, http://www.ralph-laurens.org.uk/, http://www.ecco-shoes.us.com/, http://www.oakleys-2017.in.net/, http://www.air-huaracheshoes.co.uk/, http://pelicans.nba-jersey.com/, http://www.designer-handbagsoutlet.us.com/, http://www.pandorajewelry.top/, http://hawks.nba-jersey.com/, http://www.juicycoutureoutlet.net.co/, http://76ers.nba-jersey.com/, http://suns.nba-jersey.com/, http://www.new-balanceoutlet.org/, http://www.nike-shoes.com.co/, http://www.air-jordanshoes.us/, http://saints.nfljersey.us.com/, http://www.vibram-fivefingers.in.net/, http://www.michaelkors-bags.us.org/, http://www.hollisteronlineshop.com.de/, http://www.burberry-handbagssale.net.co/, http://www.nhl-jerseys.net/, http://www.burberry-bags.net.co/, http://www.adidas-shoes.in.net/, http://www.converseschuhe.com.de/, http://www.omega-watches.com.co/, http://www.christianlouboutinoutlet.net.co/, http://www.mcm-handbags.org/, http://giants.nfljersey.us.com/, http://magic.nba-jersey.com/, http://www.prada-outlet.com.co/, http://www.nba-shoes.com/, http://www.jordan-retro.org/, http://www.airhuaraches.co.uk/, http://www.nikeairmax-90.net/, http://www.uhren-stores.de/, http://bengals.nfljersey.us.com/, http://www.new-balance-schuhe.de/, http://www.coach-outlet.store/, http://www.cheapnhljerseys.us.com/, http://www.iphone-cases.net/, http://www.soft-ballbats.com/, http://cowboys.nfljersey.us.com/, http://vikings.nfljersey.us.com/, http://www.oakleysunglasses-canada.ca/, http://michaelkors.euro-us.net/, http://www.nikeshoes-outlet.com/, http://www.coach-factory.com.co/, http://longchamp.blackofriday.com/, http://www.long-champoutlet.com/, http://www.ray-bansoutlet.com.co/, http://packers.nfljersey.us.com/, http://www.barbour-jackets.us.com/, http://www.tracksuits-store.com/, http://www.michaelkors.so/, http://wizards.nba-jersey.com/, http://www.burberry-outlets.net.co/, http://www.raybans-outlet.nl/, http://www.mizuno-running.net/, http://www.mcmbackpacks.com.co/, http://www.tommyhilfigerca.ca/, http://www.hoodies-store.com/, http://www.burberry-outlett.co.uk/, http://www.nfl-jersey.us.org/, http://oakley.blackofriday.com/, http://www.tommyhilfigers.nl/, http://www.dsquared2s.com/, http://www.puma-shoes.de/, http://www.pradahandbags.net.co/, http://www.coach-outlets.net.co/, http://www.mcmhandbags.com.co/, http://www.converses-outlet.com/

comment:24 Changed 2 months ago by Kennethger

wh0cd721504 <a href=http://buyazithromycin.us.org/>where can i buy azithromycin online</a> <a href=http://cialis20mg.us.org/>cialis tabs</a> <a href=http://buycitalopram.us.org/>Buy Citalopram</a>

comment:25 Changed 2 months ago by Tracygaw

wh0cd8864 <a href=http://sildenafilcitrate24.us.org/>order sildenafil</a>

Note: See TracTickets for help on using tickets.