Ticket #2238 (new defect)

Opened 10 years ago

Last modified 19 months ago

CONFIG_FUNCTION_TRACER panics very early

Reported by: lindi Owned by: openmoko-kernel
Priority: normal Milestone:
Component: kernel Version: unspecified
Severity: normal Keywords:
Cc: Blocked By:
Blocking: Estimated Completion (week):
HasPatchForReview: no PatchReviewResult:
Reproducible:

Description

Enabling kernel function tracer (CONFIG_FUNCTION_TRACER) panics andy-tracking b8b36e5ec3db71d very early (before framebuffer init). Could anybody with a debug board reproduce this and attach the panic message?

The ramconsole patch in http://docs.openmoko.org/trac/ticket//2135 does not help since it works only after vm_init has beel called.

Attachments

ftrace.diff (2.6 KB) - added by fweisbec 10 years ago.
ftrace_patch
ftrace2.patch (3.2 KB) - added by lindi 10 years ago.
improved version that also adds no_instrument_function to mach-s3c2440/fiq_c_isr.c
ftrace3.patch (5.0 KB) - added by lindi 10 years ago.
add the final missing no_instrument_function attributes, now ftrace works!

Change History

comment:1 Changed 10 years ago by andy

CONFIG_FUNCTION_TRACER is incompatible with attribute ((naked)) used in the FIQ stuff... it comes in and adds a preamble where no preamble can be accepted.

It's a bug in CONFIG_FUNCTION_TRACER I think...

Changed 10 years ago by fweisbec

ftrace_patch

comment:2 Changed 10 years ago by fweisbec

Hi,

Could you please test the above patch, it will disable tracing on those naked functions.
Note that it applies on 2.6.28 (original) so tell me if it applies not well on the openmoko kernel.

It's not a smart one, just to test. If it works, it would be better you redefine the naked macro with "no_instrument_function" attribute and "naked" and replace all attribute((naked)) by naked (should perhaps even be done in mainline).

Changed 10 years ago by lindi

improved version that also adds no_instrument_function to mach-s3c2440/fiq_c_isr.c

comment:3 Changed 10 years ago by lindi

arch/arm/mach-s3c2440/fiq_c_isr.c also had some naked functions so I added no_instrument_function there too. With this ftrace2.patch the phone boots for a while but then stops. I tried this two times and both times the last message I see on the screen is

pcf50633-rtc pcf50633-rtc: setting system clock to 2009-02-22 12:55:43 UTC (1235307343)

Changed 10 years ago by lindi

add the final missing no_instrument_function attributes, now ftrace works!

comment:4 Changed 10 years ago by lindi

With ftrace3.patch the phone boots successfully and ftrace works for a while. After a few minutes of playing around with ftrace the phone stops responding and watchdog apparently reboots it. After reboot I can recover the following kernel message:

[  247.810000] ------------[ cut here ]------------
[  247.810000] WARNING: at kernel/trace/ring_buffer.c:1038 trace+0x14/0x18()
[  247.815000] Modules linked in: s3c2410_wdt g_ether
[  247.820000] [<c03bd434>] (dump_stack+0x0/0x18) from [<c0056edc>] (warn_on_slowpath+0x50/0x6c)
[  247.825000] [<c0056e8c>] (warn_on_slowpath+0x0/0x6c) from [<c008dbec>] (rb_reserve_next_event+0x6c/0x3c8)
[  247.835000]  r6:c09cce48 r5:c7810ba0 r4:00000000
[  247.840000] [<c008db80>] (rb_reserve_next_event+0x0/0x3c8) from [<c008e0f4>] (ring_buffer_lock_reserve+0x8c/0x114)
[  247.850000] [<c008e068>] (ring_buffer_lock_reserve+0x0/0x114) from [<c00917e8>] (trace_function+0x3c/0x80)
[  247.860000]  r4:c09cce68
[  247.865000] [<c00917ac>] (trace_function+0x0/0x80) from [<c00918ac>] (function_trace_call+0x80/0xe4)
[  247.870000]  r8:c00350ac r7:00000000 r6:c03beeb4 r5:20000013 r4:c09cce68
[  247.880000] [<c009182c>] (function_trace_call+0x0/0xe4) from [<c0034f9c>] (trace+0x14/0x18)
[  247.885000]  r7:c7315680 r6:00000000 r5:00000002 r4:c09cce48
[  247.890000] [<c03beea8>] (mutex_unlock+0x0/0x18) from [<c008e8b4>] (tracing_ctrl_write+0x114/0x144)
[  247.900000] [<c008e7a0>] (tracing_ctrl_write+0x0/0x144) from [<c00bd9a0>] (vfs_write+0xbc/0x14c)
[  247.910000]  r7:00000002 r6:c71cdf78 r5:000c3008 r4:c7315680
[  247.915000] [<c00bd8e4>] (vfs_write+0x0/0x14c) from [<c00bdb00>] (sys_write+0x50/0x80)
[  247.925000]  r7:00000004 r6:c7315680 r5:00000000 r4:00000000
[  247.930000] [<c00bdab0>] (sys_write+0x0/0x80) from [<c0034ec0>] (ret_fast_syscall+0x0/0x2c)
[  247.935000]  r6:401ab5e0 r5:000c3008 r4:00000002
[  247.940000] ---[ end trace 00b32d9100000001 ]---
[  457.120000] Unable to handle kernel paging request at virtual address 736e6f63
[  457.120000] pgd = c0004000
[  457.120000] [736e6f63] *pgd=00000000
[  457.120000] Internal error: Oops: 5 [#1] PREEMPT
[  457.120000] Modules linked in: s3c2410_wdt g_ether
[  457.120000] CPU: 0    Tainted: G        W   (2.6.28-GTA02_lindi-andy-tracking_b8b36e5ec3db71d5-mokodev #53)
[  457.120000] PC is at strnlen+0x1c/0x34
[  457.120000] LR is at string+0x34/0xbc
[  457.120000] pc : [<c01baf64>]    lr : [<c01bbfbc>]    psr: a0000093
[  457.120000] sp : c7bafac0  ip : 736e6f63  fp : c7bafacc
[  457.120000] r10: c055e950  r9 : c055e550  r8 : ffffffff
[  457.120000] r7 : 736e6f63  r6 : c055e950  r5 : ffffffff  r4 : c055e55f
[  457.120000] r3 : ffffffff  r2 : 00000000  r1 : ffffffff  r0 : 736e6f63
[  457.120000] Flags: NzCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
[  457.120000] Control: c000717f  Table: 37168000  DAC: 00000017
[  457.120000] Process ksdioirqd/mmc0 (pid: 1781, stack limit = 0xc7bae268)
[  457.120000] Stack: (0xc7bafac0 to 0xc7bb0000)
[  457.120000] fac0: c7bafaec c7bafad0 c01bbfbc c01baf58 c7bafbec c0464b3d 00000000 ffffffff 
[  457.120000] fae0: c7bafb34 c7bafaf0 c01bc48c c01bbf98 ffffffff 00000000 ffffffff c7bafb4c 
[  457.120000] fb00: c7bafb0c 00000400 c04e1940 00000400 00000000 60000092 ffffffff c0464b2d 
[  457.120000] fb20: 00000001 c7bafbec c7bafb4c c7bafb38 c01bc730 c01bc1ac ffff003c c7bafc10 
[  457.120000] fb40: c7bafbd4 c7bafb50 c0057bb8 c01bc728 c7bafb5c c00917e8 c09cce68 c7bafb98 
[  457.120000] fb60: 00000000 c00917e8 00000004 80000093 c03c0264 00000000 c09cce68 20000093 
[  457.120000] fb80: c03bd594 00000000 00000000 c7bafbc0 c7bafb9c c00918ac c00917bc 20000093 
[  457.120000] fba0: 00000001 c7bafc10 ffff003c c7bafc10 ffff003c 60000092 ffffffff 00000000 
[  457.120000] fbc0: 00000001 c7bafdec c7bafbe4 c7bafbd8 c03bd5a4 c0057a44 c7bafc0c c7bafbf8 
[  457.120000] fbe0: c0039808 c03bd598 c0464b2d 736e6f63 c04e1910 c03c1ddc c09cce68 c04e59ac 
[  457.120000] fc00: 00000000 c7bafc10 c03c0288 c00397dc c04e59ac 00000001 ffffffff 60000013 
[  457.120000] fc20: c04e59ac 60000013 c04e5978 00000025 00000000 00000001 c7bafdec c7bafc6c 
[  457.120000] fc40: c7bafc58 c7bafc10 c03c0288 ffff003c 60000092 ffffffff c04e59ac 60000013 
[  457.120000] fc60: c7bafc8c c7bafc70 c0086428 c03c0268 c70dc320 c7bafdac c70dc000 c7bafdec 
[  457.120000] fc80: c7bafcb4 c7bafc90 c02a1994 c00863d8 00000000 c70dc000 c70dc320 c7bafdec 
[  457.120000] fca0: c7b82800 00000000 c7bafcd4 c7bafcb8 c02a1a84 c02a1700 00000000 00000000 
[  457.120000] fcc0: c7bafdec c70dc000 c7bafd04 c7bafcd8 c0296bb8 c02a19c0 c7bafd34 c7bafdec 
[  457.120000] fce0: c7bafd04 c7bafcf0 c7bafd54 c7bafd34 c7bafdec c70dc000 c7bafd7c c7bafd08 
[  457.120000] fd00: c0296c68 c0296960 00000000 00000001 dead4ead ffffffff ffffffff c7bafd6c 
[  457.120000] fd20: c02958b4 c7bafd7c c7bafdec c7bafd58 c7bafd58 00000000 00000001 dead4ead 
[  457.120000] fd40: ffffffff ffffffff c7bafd6c c02958b4 c7bafd7c c7bafdec c7bafd58 c7bafd58 
[  457.120000] fd60: c7bafdec c7bafd80 00000010 00000001 c7bafe24 c7bafd80 c029aaa0 c0296bf0 
[  457.120000] fd80: 3b9aca00 00000000 00000010 00000001 00000000 00000200 00000000 00000000 
[  457.120000] fda0: c7bafdec 00000001 c7bafdd8 00000035 14080010 00000000 00000000 00000000 
[  457.120000] fdc0: 00000000 000001b5 00000000 00000000 c7bafd80 c7bafdec 87654321 c0b55a02 
[  457.120000] fde0: 00000e54 00000000 00000010 c7bafdac c7bafd80 00000000 c7bafd34 c0296fb4 
[  457.120000] fe00: 00000800 00000010 00000010 c7b8c400 c72d0e54 00000400 c7bafe6c c7bafe28 
[  457.120000] fe20: c029bc30 c029a938 00000001 c72d0e54 00000001 00000010 00000000 00000000 
[  457.120000] fe40: c72c51a0 00000000 c72c51a0 c72c51a0 00000400 00000259 c72d0e54 00000010 
[  457.120000] fe60: c7bafe84 c7bafe70 c029bd98 c029bae4 c72d0e54 00000010 c7bafe9c c7bafe88 
[  457.120000] fe80: c02a5038 c029bd84 00000000 c7078600 c7bafee4 c7bafea0 c02a5b38 c02a4f9c 
[  457.120000] fea0: c7bafeac c00918ac c00917bc 20000013 00000000 c72d0e34 c7baff20 c72d0e34 
[  457.120000] fec0: c7baff20 00000000 c7baff24 c7baff20 00000002 c7bae000 c7baff1c c7bafee8 
[  457.120000] fee0: c02a2d28 c02a59a4 00000259 00000000 c7b8c400 c7078600 c70dc000 c72d0e34 
[  457.120000] ff00: c7baff20 c7baff24 c70dc000 c7b82800 c7baff44 c7baff20 c02a2ec0 c02a2ca4 
[  457.120000] ff20: 00000000 00000000 c7baff34 00000000 c7b8c400 c7078600 c7baff64 c7baff48 
[  457.120000] ff40: c02a5370 c02a2e9c c7baff64 00000000 c7b8c400 c0a0354c c7baff84 c7baff68 
[  457.120000] ff60: c02a546c c02a52e8 c70dc000 00000000 00000000 00000001 c7baffd4 c7baff88 
[  457.120000] ff80: c029c950 c02a53b0 00000000 c7baffa7 c7baffd4 c7b82914 c70dc308 00000002 
[  457.120000] ffa0: 00000001 029c4d24 c004ebbc c7bae000 c70dc000 c029c7f0 00000000 00000000 
[  457.120000] ffc0: 00000000 00000000 c7bafff4 c7baffd8 c006d20c c029c800 00000000 00000000 
[  457.120000] ffe0: 00000000 00000000 00000000 c7bafff8 c005a914 c006d1b8 ffffffff ffffffff 
[  457.120000] Backtrace: 
[  457.120000] [<c01baf48>] (strnlen+0x0/0x34) from [<c01bbfbc>] (string+0x34/0xbc)
[  457.120000] [<c01bbf88>] (string+0x0/0xbc) from [<c01bc48c>] (vsnprintf+0x2f0/0x4d8)
[  457.120000]  r7:ffffffff r6:00000000 r5:c0464b3d r4:c7bafbec
[  457.120000] [<c01bc19c>] (vsnprintf+0x0/0x4d8) from [<c01bc730>] (vscnprintf+0x18/0x24)
[  457.120000] [<c01bc718>] (vscnprintf+0x0/0x24) from [<c0057bb8>] (vprintk+0x184/0x4f8)
[  457.120000]  r4:c7bafc10
[  457.120000] [<c0057a34>] (vprintk+0x0/0x4f8) from [<c03bd5a4>] (printk+0x20/0x28)
[  457.120000] [<c03bd584>] (printk+0x0/0x28) from [<c0039808>] (bad_mode+0x3c/0x5c)
[  457.120000]  r3:c03c1ddc r2:c04e1910 r1:736e6f63 r0:c0464b2d
[  457.120000] [<c00397cc>] (bad_mode+0x0/0x5c) from [<c03c0288>] (_spin_unlock_irqrestore+0x30/0x58)
[  457.120000]  r4:c04e59a
 c
[  457.120000] Code: e3a02000 e1510002 e080c002 0a000003 (e7d03002) 
[21474541.065000] ramconsole: old data ends here

comment:5 Changed 10 years ago by lindi

On second try I got a different backtrace:

[ 1017.980000] gadgetfs: USB Gadget filesystem, version 24 Aug 2004
[ 1261.730000] Bad mode in data abort handler detected
[ 1261.730000] Internal error: Oops - bad mode: 0 [#1] PREEMPT
[ 1261.730000] Modules linked in: gadgetfs s3c2410_wdt [last unloaded: g_ether]
[ 1261.730000] CPU: 0    Not tainted  (2.6.28-GTA02_lindi-andy-tracking_b8b36e5ec3db71d5-mokodev #53)
[ 1261.730000] PC is at rb_reserve_next_event+0x4/0x3c8
[ 1261.730000] LR is at 0x20000013
[ 1261.730000] pc : [<c008db84>]    lr : [<20000013>]    psr: 800000d1
[ 1261.730000] sp : c7997e60  ip : 00590000  fp : c7997ebc
[ 1261.730000] r10: 00000014  r9 : 00000001  r8 : 00000015
[ 1261.730000] r7 : c003f9b4  r6 : c09cce48  r5 : 200000d1  r4 : 00000000
[ 1261.730000] r3 : 00000017  r2 : 00000014  r1 : 00000003  r0 : c7810ba0
[ 1261.730000] Flags: Nzcv  IRQs off  FIQs off  Mode FIQ_32  ISA ARM  Segment kernel
[ 1261.730000] Control: c000717f  Table: 37b88000  DAC: 00000017
[ 1261.730000] Process events/0 (pid: 5, stack limit = 0xc7996268)
[ 1261.730000] Stack: (0xc7997e60 to 0xc7998000)
[ 1261.730000] 7e60: c7810ba0 00000003 00000014 00000017 00000000 200000d1 c09cce48 c003f9b4 
[ 1261.730000] 7e80: 00000015 00000001 00000014 c7997ebc 00590000 c7997e60 20000013 c008db84 
[ 1261.730000] 7ea0: 800000d1 ffffffff 00000001 c051cb10 c7997ed4 c7997ec0 c02928c8 c003f108 
[ 1261.730000] 7ec0: c7b58d10 00000002 c7997efc c7997ed8 c0291f0c c0292884 c7b58d10 c780bc80 
[ 1261.730000] 7ee0: c7b58ca0 c7b58d0c 00000000 c7997f58 c7997f3c c7997f00 c0292484 c0291eec 
[ 1261.730000] 7f00: c7997f3c c7997f10 c0078860 c0078078 00000001 00000000 00000000 c0068bac 
[ 1261.730000] 7f20: c7b58d10 c780bc80 c7996000 c7b58d0c c7997f94 c7997f40 c0068c24 c0292458 
[ 1261.730000] 7f40: 00000001 00000000 c0068bac 20000013 c780bca0 c0292448 c0a03354 00000000 
[ 1261.730000] 7f60: c049fa01 00000000 c7997f74 c780bca8 c7996000 c780bc80 c780bca0 c7997f9c 
[ 1261.730000] 7f80: 00000000 00000000 c7997fd4 c7997f98 c0069134 c0068b08 c03bde68 00000000 
[ 1261.730000] 7fa0: c79940a0 c006d54c c7997fa8 c7997fa8 c780bc80 c7996000 c780bc80 c0069030 
[ 1261.730000] 7fc0: 00000000 00000000 c7997ff4 c7997fd8 c006d20c c0069040 00000000 00000000 
[ 1261.730000] 7fe0: 00000000 00000000 00000000 c7997ff8 c005a914 c006d1b8 ffffffff ffffffff 
[ 1261.730000] Backtrace: 
[ 1261.730000] [<c003f0f8>] (fiq_kick+0x0/0xbc) from [<c02928c8>] (gta02hdq_read+0x54/0xb0)
[ 1261.730000]  r4:c051cb10
[ 1261.730000] [<c0292874>] (gta02hdq_read+0x0/0xb0) from [<c0291f0c>] (hdq_read16+0x30/0x88)
[ 1261.730000]  r5:00000002 r4:c7b58d10
[ 1261.730000] [<c0291edc>] (hdq_read16+0x0/0x88) from [<c0292484>] (bq27000_battery_work+0x3c/0x13c)
[ 1261.730000] [<c0292448>] (bq27000_battery_work+0x0/0x13c) from [<c0068c24>] (run_workqueue+0x12c/0x228)
[ 1261.730000]  r7:c7b58d0c r6:c7996000 r5:c780bc80 r4:c7b58d10
[ 1261.730000] [<c0068af8>] (run_workqueue+0x0/0x228) from [<c0069134>] (worker_thread+0x104/0x118)
[ 1261.730000] [<c0069030>] (worker_thread+0x0/0x118) from [<c006d20c>] (kthread+0x64/0x98)
[ 1261.730000]  r8:00000000 r7:00000000 r6:c0069030 r5:c780bc80 r4:c7996000
[ 1261.730000] [<c006d1a8>] (kthread+0x0/0x98) from [<c005a914>] (do_exit+0x0/0x308)
[ 1261.730000]  r6:00000000 r5:00000000 r4:00000000
[ 1261.730000] Code: c09cb9c0 00000367 0000028e e1a0c00d (e92ddff0) 
[ 1261.730000] Bad mode in data abort handler detected
[ 1261.730000] BUG: spinlock lockup on CPU#0, events/0/5, c04d751c
[ 1261.730000] [<c03bd434>] (dump_stack+0x0/0x18) from [<c01bfd38>] (_raw_spin_lock+0x114/0x150)
[ 1261.730000] [<c01bfc24>] (_raw_spin_lock+0x0/0x150) from [<c03c061c>] (_spin_lock_irq+0x7c/0x8c)
[ 1261.730000] [<c03c05a0>] (_spin_lock_irq+0x0/0x8c) from [<c003973c>] (die+0x3c/0xcc)
[ 1261.730000]  r7:c7997de0 r6:c0464b51 r5:00000000 r4:c7997de0
[ 1261.730000] [<c0039700>] (die+0x0/0xcc) from [<c0039818>] (bad_mode+0x4c/0x5c)
[ 1261.730000] [<c00397cc>] (bad_mode+0x0/0x5c) from [<c0056e04>] (add_taint+0x24/0x30)
[ 1261.730000]  r4:00000007
[21474541.070000] ramconsole: old data ends here

comment:6 Changed 10 years ago by lindi

http://marc.info/?l=linux-arm-kernel&m=123334871704646&w=2

"Re: [PATCH] [ARM] ftrace: fix crash due to tracing of naked"

has a patch that modifies the ringbuffer code however it can not be applied since it depends on at least

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=abc9b56d66fbd4d93302ef4bf6fa726e1b8255f9

if I understand this correctly.

comment:7 Changed 19 months ago by RoberMync

Discount Generic Cialis <a href=http://cial5mg.xyz/cialis-online-pharmacy.php>Cialis Online Pharmacy</a> Levitra Italia Amitriptyline No Perscption <a href=http://cial1.xyz/cialis-online-prices.php>Cialis Online Prices</a> Online Pharmacy Cialis Reviews Combivent Inhalers Without An Rx <a href=http://viag1.xyz/buy-generic-viagra.php>Buy Generic Viagra</a> Medicine Alternativa Al Viagra Cheapeast secure ordering isotretinoin where to order no prescription <a href=http://cial5mg.xyz/low-cost-cialis-online.php>Low Cost Cialis Online</a> Cialis E Simili Propecia Prevencion <a href=http://kama1.xyz/kamagra-buy-online.php>Kamagra Buy Online</a> Is Amoxicillin A Monoamine Oxidase Inhibitor Azithromycin Cost 500mg China <a href=http://cial1.xyz/cialis-prices.php>Cialis Prices</a> Lasix Canada Amoxicillin Toxicity <a href=http://cial5mg.xyz/cialis-online-no.php>Cialis Online No</a> Elocon Want To Buy Tadacip Online Kaufen <a href=http://zol1.xyz/zoloft-online-usa.php>Zoloft Online Usa</a> Cialis Precio Oficial Espana Flagyl Without Insurance <a href=http://cial5mg.xyz/best-cialis-online.php>Best Cialis Online</a> Cunto Cuesta La Viagra En Wallgreens Viagra Cialis Nebenwirkungen <a href=http://viag1.xyz/sildenafil-citrate.php>Sildenafil Citrate</a> Viagra Kaufen Ohne Rezept Paypal Purchase Isotretinoin Pills Next Day Pharmacy Amex <a href=http://viag1.xyz/cheap-viagra-pill.php>Cheap Viagra Pill</a> Significado De Propecia Doxycycline Rodoxycycline How To Buy <a href=http://cial1.xyz/buy-online-cialis.php>Buy Online Cialis</a> Viagra Brand At Low Price Ticarcillin Versus Amoxicillin <a href=http://viag1.xyz/fast-delivery-viagra.php>Fast Delivery Viagra</a> Zithromax Prescription Cialis Impuissance Blog <a href=http://zol1.xyz/purchase-cheap-zoloft.php>Purchase Cheap Zoloft</a> Cialis Et Effets Secondaires Come Comprare Priligy <a href=http://zol1.xyz/buy-generic-zoloft.php>Buy Generic Zoloft</a> Pyridium 200mg By Money Order Canada Pharmacy Mebendazole <a href=http://zol1.xyz/zoloft-50mg.php>Zoloft 50mg</a> Cheap Kamagra From India Drug Interaction Amoxicillin And Benzodiazepines <a href=http://cial5mg.xyz/buy-online-cialis.php>Buy Online Cialis</a> Cheapeast direct isotretinoin order in internet Comprar Cialis Y Viagra <a href=http://xenical.ccrpdc.com/xenical-price-1.php>Xenical Price</a> Kamagra Beipackzettel Levitra 5 Mg <a href=http://propecia.ccrpdc.com/buying-propecia.php>Buying Propecia</a> Diabetes Y Kamagra 72 Buy Gonorrhea Treatment <a href=http://cial1.xyz/cheapest-cialis-online.php>Cheapest Cialis Online</a> Amoxicillin Effects On Birth Control Viagra Acquisto Forum <a href=http://cial5mg.xyz/tadalafil-online.php>Tadalafil Online</a> Buy Amoxicillin 500mg Cap Cipro Cheap Online <a href=http://cial5mg.xyz/cialis-online-cs.php>Cialis Online Cs</a> No Perscription Viagra Prezzo Viagra 100 <a href=http://viag1.xyz/order-cheap-viagra.php>Order Cheap Viagra</a> Viagra Pour Femme Le Prix Mail Order Tab Amoxicilina In Internet <a href=http://cial1.xyz/generic-cialis-online.php>Generic Cialis Online</a> Cialis 10 Mg Precio Farmacia Espana Keflex Y Ketoconazol Para Westies <a href=http://kama1.xyz/buy-cheap-kamagra-pills.php>Buy Cheap Kamagra Pills</a> Doxycycline Pills Black Market Is Kamagra Safe <a href=http://kama1.xyz/kamagra.php>Kamagra</a> Generics4us Acquisto Viagra Legale <a href=http://cial5mg.xyz/buy-tadalafil-online.php>Buy Tadalafil Online</a> Kamagra 100mg Side Effects Zentel Quick Shipping <a href=http://clomid.ccrpdc.com/buy-clomiphene-online.php>Buy Clomiphene Online</a> Generic Cialis Cheapest Comprar Priligy Generico En Espana <a href=http://kama1.xyz/kamagra-oral-jelly.php>Kamagra Oral Jelly</a> Kamagra Thailand Priligy Precio En Argentina <a href=http://cial5mg.xyz/cialis-buy-online.php>Cialis Buy Online</a> Amoxil Pharmacie Mexicain Commander Kamagra Original <a href=http://doxycycline.ccrpdc.com/compra-vibramycin-online.php>Compra Vibramycin Online</a> Amoxicillin For Infants Cephalexin 6 Week Old <a href=http://viag1.xyz/order-viagra-online.php>Order Viagra Online</a> Cialis Effet Secondaire Levitra Alldaychemist Com <a href=http://cial1.xyz/brand-cialis-online.php>Brand Cialis Online</a> Cialis Generika Bestellen Forum Antibiotics Keflex <a href=http://cial1.xyz/how-to-order-cialis.php>How To Order Cialis</a> Transaminasas Propecia Generico Medicamento Priligy 30 Mg <a href=http://zol1.xyz/purchase-generic-zoloft.php>Purchase Generic Zoloft</a> Viagra Zoll Viagra Generic Prices <a href=http://lasix.ccrpdc.com/buy-lasix-online.php>Buy Lasix Online</a> Where To Buy Canadian Drugs Online Propecia Side Effect Statistics <a href=http://viag1.xyz/buying-viagra-online.php>Buying Viagra Online</a> Effets Cialis 20 Priligy Cos'E <a href=http://cial1.xyz/how-to-order-cialis.php>How To Order Cialis</a> Venta Kamagra

Note: See TracTickets for help on using tickets.