love wife love life —Roger的Oracle/MySQL数据恢复博客

Phone:18180207355 提供专业Oracle/MySQL数据恢复、性能优化、迁移升级、紧急救援等服务

About Oracle Hanganalyze

本站文章除注明转载外,均为本站原创: 转载自love wife love life —Roger的Oracle/MySQL数据恢复博客

本文链接地址: About Oracle Hanganalyze

1. 数据库hang的几种可能性

oracle死锁或者系统负载非常高比如cpu使用或其他一些锁等待很高都可能导致系统hang住,比如大量的DX锁。

通常来说,我们所指的系统hang住,是指应用无响应,普通的sqlplus几乎无法操作等等。
2. 如何进行hang分析?hang分析有哪些level?如何选择level?

hanganalyze有如下几种level:

从上面的信息看,在进行hanganalyze dump时有多种级别的level可以选择,那么如何选择level?
一般来讲,不建议使用level 3以上的操作,因为产生的trace可能会很大,尤其是大型的OLTP系统;
另外一般数据库hang住时可能系统压力都巨大,所以再产生很大的trace可能导致问题更加严重。

从oracle 9i开始hangalanyze 操作提供了针对Oracle RAC的支持,有如下2种方式:

1) ALTER SESSION SET EVENTS ‘immediate trace name HANGANALYZE level <level>’;

2) 使用oradebug 命令

ORADEBUG setmypid
ORADEBUG setinst all
ORADEBUG -g def hanganalyze <level>       —针对rac的用法
对于单实例,我们通常进行如下操作即可:
oradebug setmypid
oradebug hanganalyze 3
其次在做hang分析的时候,建议同时做一个systemstate dump或针对个别的process进行processstate dump,如下:

我们知道当一个数据库hang住时,最头痛的问题是无法登陆数据,也就无法进行故障的处理,因此很多人只能通过重启
操作系统来讲解决问题,其实从Oracle 10g开始,Oracle提供了prelim的登陆方式,如下:

退一万步讲,即使我们无法通过SQLPLUS登陆数据库,仍然还是可以从操作系统层面入手的,即通过操作系统的命令来
对进程进行dump,例如aix环境中可以使用dbx命令来dump processstate,如下:

dbx -a PID (where PID = any oracle shadow process)       —通过ps -ef|grep xxx查看
dbx() print ksudss(10)
…return value printed here
dbx() detach
3. 如何解读hang分析的trace文件,获取有用信息?

 
*** ACTION NAME:() 2010-03-12 00:04:01.497
*** MODULE NAME:(sqlplus@S7_C_YZ_YZSJK (TNS V1-V3)) 2010-03-12 00:04:01.497    —模块名 跟v$session.module_name一样
*** SERVICE NAME:(SYS$USERS) 2010-03-12 00:04:01.497
*** SESSION ID:(5184.45287) 2010-03-12 00:04:01.497         —-sid (5184)   serial# (35287)
*** 2010-03-12 00:04:01.497
==============
HANG ANALYSIS:
==============
Found 54 objects waiting for <cnode/sid/sess_srno/proc_ptr/ospid/wait_event>
<0/5210/10419/0x99d0a88/11215038/No Wait>                  —从这里看 session 5210 阻塞了54个对象
Open chains found:
Chain 1 : <cnode/sid/sess_srno/proc_ptr/ospid/wait_event> :    —从这里开始以下的session都是被前面的5210阻塞
<0/5210/10419/0x99d0a88/11215038/No Wait>
— <0/3994/15494/0xd9ac1b0/6574102/enq: TM – contention>
— <0/4962/58962/0xca03618/5710044/enq: DX – contention>
Other chains found:                                            —下面的session也是被前面所阻塞,被间接阻塞
Chain 2 : <cnode/sid/sess_srno/proc_ptr/ospid/wait_event> :
<0/4001/31548/0xf9f3ab0/4980956/enq: DX – contention>
Chain 3 : <cnode/sid/sess_srno/proc_ptr/ospid/wait_event> :
<0/4014/30717/0xaa27b48/7446746/gc buffer busy>
Chain 4 : <cnode/sid/sess_srno/proc_ptr/ospid/wait_event> :
<0/4039/42115/0xd9f5710/5595180/PX Deq: Table Q Normal>

 

Cycle 1 : <sid/sess_srno/proc_ptr/ospid/wait_event> :        —cycle 通常是死锁 一般来说很有可能就是hang的根源
<980/3887/0xe4214964/24065/latch free>
— <2518/352/0xe4216560/24574/latch free>
— <55/10/0xe41236a8/13751/latch free>

如果trace中出现了Cycle,基本上Oracle就hang了。
4. 不同版本hang分析的差异?trace有何异同?

如下是oracle8~10g的 hanganalyze trace信息格式:

Oracle 8.x : [nodenum]/sid/sess_srno/session/state/start/finish/[adjlist]/predecessor
Oracle9i:  [nodenum]/cnode/sid/sess_srno/session/ospid/state/start/finish/[adjlist]/predecessor
Oracle10g:[nodenum]/cnode/sid/sess_srno/session/ospid/state/start/finish/[adjlist]/predecessor

 

Nodenum     –》 每个session做hanganalyze生成的一个序列号
sid         –》 Session ID
sess_srno   –》 Serial#
ospid       –》 OS Process Id  (v$process spid)
state       –》 State of the node
adjlist     –》 adjacent node   (Usually represents a  blocker node) –通常是阻塞者
predecessor –》 predecessor node (Usually represents a waiter node)  –通常是被阻塞者
cnode       –》 节点号 从9i开始才有

关于state 有如下几种值:

IN_HANG      –》 该状态是一个非常危险的状态,通常表现为一个节点陷入了死循环或是hung。
一般来说出现这种情况,该节点的临辟节点也是一样的状态 即adjlist

例如:
[16]/0/17/154/0x24617be0/26800/IN_HANG/29/32/[185]/19   —从IN_HANG 我们可以看出185被16阻塞
[185]/1/16/4966/0x24617270//IN_HANG/30/31/[16]/16       —
LEAF         –》通常是被认为blockers的重点对象。那么如何去确定呢?
一般根据后面的predecesor来判断该session是不是blocker或者是waiter。
例子:
[ nodenum]/cnode/sid/sess_srno/session/ospid/state/start/finish/[adjlist]/predecessor
[16]/0/17/154/0x24617be0/26800/LEAF/29/30//19         –从这里看19是waiter 因此我们认为17阻塞了20
[19]/0/20/13/0x24619830/26791/NLEAF/33/34/[16]/186

 

LEAF_NW     –》跟leaf类似 不过可能会占用cpu
NLEAF       –》该状态的session通常被认为 “stuck” session。即其他session所需要的资源正被其holding。
IGN         –》该状态的session通常是处理IDLE状态,除非其adjlist存在,如果是,那么该session正在等待其他session。
IGN_DMP     –》跟IGN类似。

例子:

[nodenum]/cnode/sid/sess_srno/session/ospid/state/start/finish/[adjlist]/predecessor
[16]/0/17/154/0x24617be0/26800/LEAF/29/30//19
[19]/0/20/13/0x24619830/26791/NLEAF/33/34/[16]/186
[189]/1/20/36/0x24619830//IGN/95/96/[19]/none
[176]/1/7/1/0x24611d80//IGN/75/76//none

从上面看,189在等待19,19在等待16,而176是一个idle session。

SINGLE_NODE,SINGLE_NODE_NW 可以认为跟LEAF,LEAF_NW一样,除非没有依赖对象。

 

5. 11g中hanganalyze trace格式的变化
我们知道从11g(应该是从11.1)开始,Oracle的trace格式相比10g而已,有很大的改变.hanganalyze trace格式也不例外.
如下是今天一个学生提到的问题,drop user xxxx cascade 删除用户时hang住. 通过oradebug hanganalyze 3我们来
寻找问题的根本原因,首先来看一下trace的内容 :

大家可以看到trace的内容跟10g版本相比,变化比较大,其实这看起来更容易一些了,只是我们还不太习惯而已。
当然,重点是这一段内容:
State of ALL nodes
([nodenum]/cnode/sid/sess_srno/session/ospid/state/[adjlist]):
[190]/1/191/3/0x1eb8ac1b0/14027/IN_HANG/[190]

我们将上述内容进行分解:
nodenum:     190 表示sid
cnode:        1   表示数据库节点编号
sid:          191 表示sid
0x1eb8ac1b0:  3   表示serial#号
session:      0x1eb8ac1b0 表示该Session的saddr地址
ospid:        14027       表示该session的操作系统进程编号
state:        IN_HANG     表示该session的状态
adjlist:      190         表示blocker的sid值

当我们将上述内容分解之后,你会发现,之所以drop user没反应,是因为被sid 190所阻塞了,那么我们最后来看一下
sid=190 是什么进程:

很显然,mmon进程为非核心进程,我们可以直接kill掉。当然最后kill mmon进程之后drop user很快完成.

199 Responses to “About Oracle Hanganalyze”

  1. kimi Says:

    ([nodenum]/cnode/sid/sess_srno/session/ospid/state/[adjlist]):

    [190]/1/191/3/0x1eb8ac1b0/14027/IN_HANG/[190]

    我们将上述内容进行分解:
    State of LOCAL nodes
    ([nodenum]/cnode/sid/sess_srno/session/ospid/state/[adjlist]):
    [45]/1/46/49/0x8d6d5174/3966/NLEAF/[51]
    [51]/1/52/35/0x8d6c453c/3928/LEAF/

    我做出来的51并不是sid号 而是52号sid是blocker

  2. Roger Says:

    兄弟,你看错地方了,我那里是in_hang的状态,是ok的。如果是nleaf,leaf的状态,应该看我上面写的内容,如下:

    例子:
    [ nodenum]/cnode/sid/sess_srno/session/ospid/state/start/finish/[adjlist]/predecessor
    [16]/0/17/154/0x24617be0/26800/LEAF/29/30//19 –从这里看19是waiter 因此我们认为17阻塞了20
    [19]/0/20/13/0×24619830/26791/NLEAF/33/34/[16]/186

  3. kimi Says:

    您的意思就是不用看其他的 leaf状态的就是阻塞源头 nleaf的就是被阻塞的 是这样理解吗

  4. kimi Says:

    [16]/0/17/154/0x24617be0/26800/IN_HANG/29/32/[185]/19 —从IN_HANG 我们可以看出185被16阻塞

    [185]/1/16/4966/0×24617270//IN_HANG/30/31/[16]/16 —

    这段看不懂 2个都是in_hang 怎么判断哪个是堵塞者 哪个被堵塞?

  5. drop用户hang住解决思路 | SEARU.ORG Says:

    […] 一个insert也能死锁吗?这里不是太清楚,知道的朋友可以指点一下,因为drop的时候hang住了,所以这里想到了做hanganalyze分析一下,这里采用是的3级的hanganalyze分析,当是也是roger老师提示的,具体的分析hanganalyze的知识点和分析过程可以参考http://www.killdb.com/2014/01/23/about-oracle-hanganalyze.html当然我的这个案例也被他引入了,但是MMON 是可管理性监视器(Manageability Monitor),MMON主要用于AWR,ADDM,MMON会从SGA将统计结果写到系统表中。这是这个进程的作用,但是为什么跟删除一个用户有关系,原因还是不清楚希望知道的朋友指点下。 喜欢 0 抢沙发 分享 标签: […]

  6. levitra generic Says:

    levitra 20mg

    levitra for women results usuario – herbal ed – levitra order on line

  7. generic cialis canada pharmacy Says:

    cialis coupon

    retalis cialis – how to use cialis – cialis vs viagra drug guestbook.cgi?start=

  8. buy tadalafil online Says:

    tadalafil 5 mg

    recomended pharmacy sites for cialis – alternative alternative apcalis buy buy cialis jel jelly – cialis dose size you may not post replies

  9. tadalafil 20mg lowest price Says:

    tadalafil

    cialis no prescription – cialis 10 mg effectiveness total posts – cialis 10mg tablets announcements

  10. Oracle数据库hang诊断-徐万新之路 Says:

    […] repeatedly waiting*/ SPOOL OFF;   获取了数据之后 就是分析了!! Reference: http://www.killdb.com/2014/01/23/about-oracle-hanganalyze/ http://blog.itpub.net/205377/viewspace-2124541/ 赞(0) 打赏 […]

  11. viagra pills for sale Says:

    viagra pills for sale

    viagra pills for sale

  12. cialis cost per pill Says:

    cialis cost per pill

    cialis cost per pill

  13. cialis lowest price Says:

    cialis lowest price

    cialis lowest price

  14. buy cialis online Says:

    buy cialis online

    buy cialis online

  15. alternatives to viagra Says:

    alternatives to viagra

    alternatives to viagra

  16. canada pharmacy cialis Says:

    canada pharmacy cialis

    canada pharmacy cialis

  17. tadalafil generic Says:

    tadalafil generic

    tadalafil generic

  18. mail order viagra Says:

    mail order viagra

    mail order viagra

  19. mandy flores viagra Says:

    mandy flores viagra

    mandy flores viagra

  20. cialis information Says:

    cialis information

    cialis information

  21. is viagra safe Says:

    is viagra safe

    is viagra safe

  22. female viagra pills Says:

    female viagra pills

    female viagra pills

  23. female viagra reviews Says:

    female viagra reviews

    female viagra reviews

  24. generic cialis Says:

    generic cialis

    generic cialis

  25. cialis free trial Says:

    cialis free trial

    cialis free trial

  26. viagra over counter Says:

    viagra over counter

    viagra over counter

  27. vietnamese viagra Says:

    vietnamese viagra

    vietnamese viagra

  28. womens viagra pill Says:

    womens viagra pill

    womens viagra pill

  29. no prescription viagra Says:

    no prescription viagra

    no prescription viagra

  30. sildenafil coupons Says:

    sildenafil coupons

    sildenafil coupons

  31. online consult cialis usa Says:

    online consult cialis usa

    online consult cialis usa

  32. viagra single packs actress Says:

    viagra single packs actress

    viagra single packs actress

  33. generic names for cialis and viagra Says:

    generic names for cialis and viagra

    generic names for cialis and viagra

  34. cheapest 20 mg cialis and dapoxetine Says:

    cheapest 20 mg cialis and dapoxetine

    cheapest 20 mg cialis and dapoxetine

  35. ingredients viagra Says:

    ingredients viagra

    ingredients viagra

  36. side effects viagra men Says:

    side effects viagra men

    side effects viagra men

  37. consumer reports generic viagra Says:

    consumer reports generic viagra

    consumer reports generic viagra

  38. generic viagra 100mg cvs Says:

    generic viagra 100mg cvs

    generic viagra 100mg cvs

  39. cialis 10 mg side effects Says:

    cialis 10 mg side effects

    cialis 10 mg side effects

  40. sildenafil prices at walgreens Says:

    sildenafil prices at walgreens

    sildenafil prices at walgreens

  41. best price for sildenafil Says:

    best price for sildenafil

    best price for sildenafil

  42. comprar sildenafil Says:

    comprar sildenafil

    comprar sildenafil

  43. comprar cialis online amazon Says:

    comprar cialis online amazon

    comprar cialis online amazon

  44. sildenafil 100mg mexico Says:

    sildenafil 100mg mexico

    sildenafil 100mg mexico

  45. cialis commercial 2018 Says:

    cialis commercial 2018

    cialis commercial 2018

  46. cialis video Says:

    cialis video

    cialis video

  47. how long before amoxicillin works Says:

    how long before amoxicillin works

    how long before amoxicillin works

  48. buy fish amoxicillin Says:

    buy fish amoxicillin

    buy fish amoxicillin

  49. medication azithromycin Says:

    medication azithromycin

    medication azithromycin

  50. celebrex for muscle pain Says:

    celebrex for muscle pain

    celebrex for muscle pain

  51. celecoxib id Says:

    celecoxib id

    celecoxib id

  52. keflex dosage uti Says:

    keflex dosage uti

    keflex dosage uti

  53. cefpodoxime vs cephalexin for dogs Says:

    cefpodoxime vs cephalexin for dogs

    cefpodoxime vs cephalexin for dogs

  54. duloxetine generic Says:

    duloxetine generic

    duloxetine generic

  55. duloxetine generic coupon Says:

    duloxetine generic coupon

    duloxetine generic coupon

  56. alternative medication to sildenafil Says:

    aurobindo pharma limited sildenafil

    tantra sildenafil

  57. ivermectin for cattle Says:

    stromectol ivermectin

    ivermectin coronavirus

  58. non-prescription generic viagra and cialis Says:

    cialis without prescription in peru

    tadalafil 10mg without doctor prescription

  59. non-prescription viagra Says:

    viagra without a doctor prescription cvs stores

    non-prescription viagra

  60. tadalafil 20 mg Says:

    tadalafil 20mg lowest price

    generic tadalafil

  61. fildena extra power 150 Says:

    fildena 150

    fildena.com

  62. cost stromectol Says:

    stromectol “eye worm”

    que es el ivermectin stromectol)

  63. cialis average price Says:

    cialis average price

    cialis average price

  64. viagra doses Says:

    viagra doses

    viagra doses

  65. cialis 100mg uk Says:

    cialis 100mg uk

    cialis 100mg uk

  66. online real viagra Says:

    online real viagra

    online real viagra

  67. cialis reliable sites Says:

    cialis reliable sites

    cialis reliable sites

  68. ivermectin for covid 19 Says:

    ivermectin buy

    stromectol 3 mg

  69. sildenafil dosage recommendations Says:

    sildenafil dosage recommendations

    sildenafil dosage recommendations

  70. levitra cialis viagra Says:

    levitra cialis viagra

    levitra cialis viagra

  71. canadian pharmacy viagra Says:

    canadian pharmacy viagra

    canadian pharmacy viagra

  72. cialis 20mg review Says:

    cialis 20mg review

    cialis 20mg review

  73. natural viagra gnc Says:

    natural viagra gnc

    natural viagra gnc

  74. neurontin for anxiety Says:

    neurontin for anxiety

    neurontin for anxiety

  75. is cialis safe Says:

    is cialis safe

    is cialis safe

  76. levitra price usa Says:

    levitra price usa

    levitra price usa

  77. sildenafil cost Says:

    sildenafil cost

    sildenafil cost

  78. what is amlodipine Says:

    what is amlodipine

    what is amlodipine

  79. simvastatin vs atorvastatin Says:

    simvastatin vs atorvastatin

    simvastatin vs atorvastatin

  80. meloxicam dose Says:

    meloxicam dose

    meloxicam dose

  81. metoprolol tartrate medication Says:

    metoprolol tartrate medication

    metoprolol tartrate medication

  82. losartan dosage Says:

    losartan dosage

    losartan dosage

  83. viagra professional Says:

    viagra professional

    viagra professional

  84. cialis generico Says:

    cialis generico

    cialis generico

  85. vardenafil brand name Says:

    vardenafil brand name

    vardenafil brand name

  86. duloxetine dose Says:

    duloxetine dose

    duloxetine dose

  87. prednisone half life Says:

    prednisone half life

    prednisone half life

  88. amitriptyline for dogs Says:

    amitriptyline for dogs

    amitriptyline for dogs

  89. duloxetine drug class Says:

    duloxetine drug class

    duloxetine drug class

  90. valsartan hydrochlorothiazide Says:

    valsartan hydrochlorothiazide

    valsartan hydrochlorothiazide

  91. berberine vs metformin Says:

    berberine vs metformin

    berberine vs metformin

  92. bupropion er Says:

    bupropion er

    bupropion er

  93. buspirone Says:

    buspirone

    buspirone

  94. citalopram dosage Says:

    citalopram dosage

    citalopram dosage

  95. zanaflex vs flexeril Says:

    zanaflex vs flexeril

    zanaflex vs flexeril

  96. what is wellbutrin Says:

    what is wellbutrin

    what is wellbutrin

  97. diclofenac dose Says:

    diclofenac dose

    diclofenac dose

  98. clonidine price Says:

    clonidine price

    clonidine price

  99. finasteride for women Says:

    finasteride for women

    finasteride for women

  100. carvedilol usage Says:

    carvedilol usage

    carvedilol usage

  101. what does metronidazole treat Says:

    what does metronidazole treat

    what does metronidazole treat

  102. tadalafil price comparison Says:

    tadalafil price comparison

    tadalafil price comparison

  103. sildenafil 50 mg Says:

    sildenafil 50 mg

    sildenafil 50 mg

  104. cialis black reviews Says:

    cialis black reviews

    cialis black reviews

  105. viagra alternative gnc Says:

    viagra alternative gnc

    viagra alternative gnc

  106. levitra drug prices Says:

    levitra drug prices

    levitra drug prices

  107. http://ketodietione.com Says:

    http://ketodietione.com

    love wife love life —Roger的Oracle/MySQL数据恢复博客 » Blog Archive » About Oracle Hanganalyze

  108. the best keto diet Says:

    the best keto diet

    love wife love life —Roger的Oracle/MySQL数据恢复博客 » Blog Archive » About Oracle Hanganalyze

  109. keto diet good for diabetics Says:

    keto diet good for diabetics

    love wife love life —Roger的Oracle/MySQL数据恢复博客 » Blog Archive » About Oracle Hanganalyze

  110. acyclovir cost uk Says:

    acyclovir cost uk

    acyclovir cost uk

  111. amoxicillin 500mg cost Says:

    amoxicillin 500mg cost

    amoxicillin 500mg cost

  112. donepezil hcl warnings Says:

    donepezil hcl warnings

    donepezil hcl warnings

  113. amoxicillin clavulanate Says:

    amoxicillin clavulanate

    amoxicillin clavulanate

  114. azithromycin dose Says:

    azithromycin dose

    azithromycin dose

  115. the best free dating websites uk Says:

    complete free dating websites

  116. side effects of cefdinir Says:

    side effects of cefdinir

    side effects of cefdinir

  117. generic keflex Says:

    generic keflex

    generic keflex

  118. cleocin Says:

    cleocin

    cleocin

  119. erythromycin topical gel Says:

    erythromycin topical gel

    erythromycin topical gel

  120. zithromax antibiotic zithromax Says:

    zithromax antibiotic zithromax

    zithromax antibiotic zithromax

  121. viagra vs cialis Says:

    viagra vs cialis

    viagra vs cialis

  122. cialis expiration Says:

    cialis expiration

    cialis expiration

  123. generic viagra sildenafil Says:

    generic viagra sildenafil

    generic viagra sildenafil

  124. sildenafil from mexico Says:

    sildenafil from mexico

    sildenafil from mexico

  125. cialis and depression Says:

    cialis and depression

    cialis and depression

  126. coffee on keto diet Says:

    keto breakfast no eggs

  127. cialis 20mg no prescription Says:

    cialis 20mg no prescription

    cialis 20mg no prescription

  128. vardenafil 40 mg generic Says:

    vardenafil 40 mg generic

    vardenafil 40 mg generic

  129. generic sildenafil 100mg Says:

    generic sildenafil 100mg

    generic sildenafil 100mg

  130. cost of tadalafil 5mg Says:

    cost of tadalafil 5mg

    cost of tadalafil 5mg

  131. cheapest cialis prices Says:

    cheapest cialis prices

    cheapest cialis prices

  132. 200mg viagra Says:

    200mg viagra

    200mg viagra

  133. buy viagra without prescription Says:

    buy viagra without prescription

    buy viagra without prescription

  134. sildenafil 25 mg Says:

    sildenafil 25 mg

    sildenafil 25 mg

  135. stendra vs viagra Says:

    stendra vs viagra

    stendra vs viagra

  136. viagra cheap prices Says:

    viagra cheap prices

    viagra cheap prices

  137. viagra in canada Says:

    viagra in canada

    viagra in canada

  138. hydroxychloroquine mexican pharmacy Says:

    hydroxychloroquine treatment for coronavirus

    according diastolic heart failure depending

  139. metoprolol vs amlodipine Says:

    metoprolol vs amlodipine

    metoprolol vs amlodipine

  140. buy cialis canada Says:

    buy cialis canada

    buy cialis canada

  141. levitra india price Says:

    levitra india price

    levitra india price

  142. metformin hcl er Says:

    metformin hcl er

    metformin hcl er

  143. canada cialis paypal Says:

    canada cialis paypal

    canada cialis paypal

  144. Homepage Says:

    … [Trackback]

    […] Read More: killdb.com/2014/01/23/about-oracle-hanganalyze/ […]

  145. amoxicillin over the counter Says:

    amoxicillin over the counter

    amoxicillin over the counter

  146. doxycycline side effects sun Says:

    doxycycline side effects sun

    doxycycline side effects sun

  147. furosemide Says:

    furosemide

    furosemide

  148. orlistat medication Says:

    orlistat medication

    orlistat medication

  149. dapoxetine 30mg tab Says:

    dapoxetine 30mg tab

    dapoxetine 30mg tab

  150. warnings for finasteride Says:

    warnings for finasteride

    warnings for finasteride

  151. bimatoprost lash Says:

    bimatoprost lash

    bimatoprost lash

  152. clomid cm Says:

    clomid cm

    clomid cm

  153. generic form of diflucan Says:

    generic form of diflucan

    generic form of diflucan

  154. domperidone msds Says:

    domperidone msds

    domperidone msds

  155. hydroxychloroquinea usa Says:

    hydroxychloroquinea usa

    love wife love life —Roger的Oracle/MySQL数据恢复博客 » Blog Archive » About Oracle Hanganalyze

  156. tamoxifen side effects gyn Says:

    tamoxifen side effects gyn

    tamoxifen side effects gyn

  157. prednisolone acetate ophthalmic solution Says:

    prednisolone acetate ophthalmic solution

    prednisolone acetate ophthalmic solution

  158. naltrexone hcl tab 50mg Says:

    naltrexone hcl tab 50mg

    naltrexone hcl tab 50mg

  159. valacyclovir side effects urinary Says:

    valacyclovir side effects urinary

    valacyclovir side effects urinary

  160. tizanidine manufacturer information Says:

    tizanidine manufacturer information

    tizanidine manufacturer information

  161. cialis 5mg tablets Says:

    cialis 5mg tablets

    cialis 5mg tablets

  162. viagra cialis combo Says:

    viagra cialis combo

    viagra cialis combo

  163. ciprofloxacin medline Says:

    ciprofloxacin medline

    ciprofloxacin medline

  164. tadalafil generic 10mg Says:

    tadalafil generic 10mg

    tadalafil generic 10mg

  165. tadalafil research chemical Says:

    tadalafil research chemical

    tadalafil research chemical

  166. cost of cialis Says:

    cost of cialis

    cost of cialis

  167. viagra prescription Says:

    viagra prescription

    viagra prescription

  168. viagra effects Says:

    viagra effects

    viagra effects

  169. tadalafil Says:

    best price for tadalafil 20 mg

    tadalafil 5mg

  170. gay black men dating site Says:
  171. crossword gay dating site Says:
  172. online gay dating sim Says:
  173. gay dating los angeles Says:
  174. gay dating utica Says:
  175. is bumble good for gay dating? Says:
  176. top gay free dating websites Says:
  177. gay dating mature Says:
  178. gay slave and master dating Says:
  179. viagra online cheap Says:

    viagra online cheap

    WALCOME

  180. viagra buy Says:

    viagra buy

    WALCOME

  181. viagra Says:
  182. viagra online prescription free Says:

    viagra online prescription free

    WALCOME

  183. generic viagra 100mg Says:

    generic viagra 100mg

    WALCOME

  184. generic cialis no prescription Says:

    generic cialis no prescription

    WALCOME

  185. where to purchase cialis Says:

    where to purchase cialis

    WALCOME

  186. how much for cialis pills Says:

    how much for cialis pills

    WALCOME

  187. brand cialis Says:

    […] cialis tadalafil 10mg […]

  188. cialis soft Says:

    […] cialis by mail […]

  189. buy cialis Says:

    buy cialis

    WALCOME

  190. buy viagra euro Says:

    […] herbal viagra for sale in ireland […]

  191. viagra connect Says:

    viagra connect

    WALCOME

  192. sildenafil viagra Says:

    sildenafil viagra

    WALCOME

  193. horse ivermectin for humans Says:

    is ivermectin poison

    arrive aerophagia frame

  194. buy viagra online cheap Says:

    buy viagra online cheap

    WALCOME

  195. cialis germany Says:

    cialis germany

    WALCOME

  196. buy cialis brand Says:

    […] buy viagra for cheap […]

  197. viagra sample Says:

    viagra sample

    WALCOME

  198. buy cheap viagra generic online Says:

    […] how to order viagra cheap […]

  199. vigra Says:

Leave a Reply

You must be logged in to post a comment.