如何执行一条命令

干货概览

 

今天我们不聊上层建筑,不聊单机房自愈,也不聊智能运维的暗夜与黎明,今天我们聊一个很基础的话题:如何执行一条命令。许多人看到这,可能觉得这是一个简单至极的问题,事实果真如此吗,别急,下面我们来抽丝剥茧,一探究竟。

 

什么是命令

 

首先回顾一下“命令”的具体含义,发令以使之,谓发令而使其做某事,这是“命令”一词的基础释义。从这里我们可以看命令的三个最基本的要素:命令内容(令)、命令传递(发)、命令执行(使)。如果将这三要素限定在服务器上,它们又是如何运作的呢?

 

图1 windows与linux下的命令

 

命令内容(令)

 

无论是windows还是linux操作系统,都会提供相应的CLI(不要吐槽windows的CLI难用),供使用者交互执行命令或运行批处理脚本。仔细观察,所有命令行都有一个相同的特点:固定词+选项+参数,无出其右。因CLI伴随着操作系统的诞生,且命令行处理又是一个复杂但相似的过程,因此各种语言也都提供了相应的库支持,如C语言提供了getopt/getopt_long等函数,C++的boost提供了Options库,shell中处理此事的是getopts和getopt。

 

命令传递(发)

 

命令传递有两种方式,一种是文件形式,将bat/shell脚本上传到服务器然后执行。另外一种就是交互式,通过telnet/ssh等方式远程连接服务器后,直接在命令行界面执行。虽然从形式上我们将命令传递分为了两种方式,但从本质上来说,服务器上的命令传递,都没有逃脱网络传输这个过程。

 

命令执行(使)

 

对于操作系统来说,命令的执行,其实就是启动一个进程并传递相应的参数,运行完成后得到相应的结果。这里我们并不关心进程如何创建,PBC的结构如何等细节,我们只关心命令进程的启动方式以及结果的获取方式。

 

为什么要执行命令

 

在分布式产品的开发维护过程中,有三个主题是无法绕过的,分别是配置管理、部署升级和监控采集。

 

配置管理

 

配置管理的目标是为了标识变更、控制变更、确保变更正确实现并向其他有关人员报告变更。从某种角度讲,配置管理是一种标识、组织和控制修改的技术。通常情况下,配置管理都会统一部署配置服务器来同步所有节点的配置。但是在开发测试过程中,总会出现临时修改某个或某一批节点的配置的情况,这时通过人工逐个登录来完成修改显然是不太可能的。

 

部署升级

 

DevOps的概念如今日趋流行,部署升级越发成为开发运维过程中重要的一环,频繁的交互意味着频繁的部署。部署过程可以拆解为两个小的步骤,一是新软件包的上传,二是服务进程的重新启动。服务进程的重新启动不必多说,软件包的上传可能有多种方式,如sftp的集中式,p2p的点对点式等。

 

监控采集

 

软件运维过程需要时刻监控系统及业务软件的运行状态,各种运维决策都是以这些数据为依据进行的。随着自动化运维的发展,很多运维动作都从人工执行变为了自动执行,自动执行的决策过程更是需要采集大量的实时信息(前期文章《百度大规模时序数据存储》中介绍的TSDB就是为了解决这些数据的存储问题而研发的)。监控数据的来源主要分两种,一种是通过业务软件提供的接口直接读取状态数据,另一种是通过日志/进程状态/系统状态等(如使用grep提取日志,通过ps查询进程状态,通过df查询磁盘使用等)方式间接查询。

 

无论是配置管理、部署变更还是监控采集,都有一个共同的目的:控制服务器。在现阶段,要想对服务器进行控制,离不开“在大量服务器上执行命令并收集结果”这一基础能力,这也是今天我们的主题“如何执行一条命令”的意义所在。

 

面临的困难

 

命令行的三要素,也是如何执行一条命令行面对的三个问题,如前文所述,对于单机环境来说,这三个问题在前人的努力下已经被很好的解决。可是如果要在几十万台机器上每天执行几十亿条命令,同时保证时效性,保证执行成功率,保证结果正确收集,保证7*24小时稳定运行,就不是一件简单的事情了。所谓远行无轻担,量大易也难,在构建这样的执行系统的过程中要面临诸多困难,此处举几个突出的例子如下:

 

信息存储问题:为了支持水平扩展,需要高效的内存数据库作为缓存。为了做到执行命令的可追溯、可统计,需要对执行过的命令信息持久化。日均几十亿的热数据,年均上万亿的冷数据,需要仔细选择存储方案。

 

任务调度问题:为了达到在任意多台服务器上执行命令的要求,需要确定何时分发命令、何时回收结果以及怎么样的并发度批量下发。

 

消息传输问题:为了保证命令高效正确送达目标服务器,需要构建一个可靠的命令传输网络,使命令信息在准确送达的前提下保障传输的可靠与高效,毕竟百度的几十万台服务器分布在世界各地。

 

代理执行问题:为了更好的处理权限、单机并发等单机执行问题,需要在目标机构建执行代理,以应对单机的复杂执行环境。

 

图2简单问题放大后也变得困难

 

百度目前拥有分布在世界各地的几十万台服务器,并且随着业务的不断扩张,这个数字还在持续增长,构建一个高效稳定通用可扩展的命令描述、传递、执行系统在这样的环境中有着重要的现实意义。对百度各产品线的用户来说,这样的一个系统,最基础的要求是:执行高效,控制灵活,扩展方便。

 

1.执行高效:

 

单机执行,要求能够达到秒级命令下发/执行/结果收集。

 

集群执行,要求支持同时在10万台服务器上并行执行,同时保证集群中每个机器达到单机执行的性能。

 

2.控制灵活:

 

单机控制,要求支持暂停、取消、重做功能。

 

集群控制,要求支持暂停点功能,也即可以在执行到某台服务器时暂停,等待人工检查确认无问题后可继续执行。

 

3.扩展方便:

 

支持插件,要求支持自定义执行插件,用户可编写自己的插件执行相应操作。

 

支持回调,要求支持自定义用户回调,如任务执行失败调用相应回调接口。

 

除了以上的业务需求外,一个分布式系统的搭建,还要考虑可用性、可扩展性、性能、一致性等方面的硬性要求。

 

如何解决

 

为了解决这个简单的难题,我们设计了如图3所示的百度集群控制系统(Cluster Control System,简称CCS系统),通过分离控制信息与执行信息建立了两级数据模型,结合命令执行及机房部署特点建立了四级传输模型,通过三级守护方式建立了稳定的执行代理,在大规模服务器集群上解决了“命令三要素”问题。

 

图3百度集群控制系统架构

 

截至目前,CCS系统已经部署在全百度的所有机房中,用户可以方便的在任意一台机器上进行秒级命令下发和结果收集,日均承载数亿次来自各产品的接口调用。关于数据模型、传输模型、执行代理这“分布式命令三要素”的设计及应用,我们将在下一篇文章中详细介绍。

 

收藏 评论(1)
分享到:
共1条回复 最后由Q1058204131 回复于2019-08-05 15:33
#2 Q1058204131 回复于2019-08-05

不错的转发下,有啥东东,抱歉
http://gbsgjd.wikidot.com/,
http://mfghdg.wikidot.com/,
http://jsdfghd.wikidot.com/,
http://hfsgf.wikidot.com/,
http://fcnfgsg.wikidot.com/,
http://jfjdsgf.wikidot.com/,
http://hkdtrj.wikidot.com/,
http://fsdgshk.wikidot.com/,
http://fhsdgj.wikidot.com/,
http://fgsdfgs.wikidot.com/,
http://mczfjkm.wikidot.com/,
http://nzgsgj.wikidot.com/,
http://fhsahh.wikidot.com/,
http://ghsdhd.wikidot.com/,
http://nfdgsdd.wikidot.com/,
http://mfgxjch.wikidot.com/,
http://gfhfsa.wikidot.com/,
http://goyfhglk.wikidot.com/,
http://fkgkjjk.wikidot.com/,
http://hgdhds.wikidot.com/,
http://mhdfhs.wikidot.com/,
http://yursdff.wikidot.com/,
http://hdfgdgs.wikidot.com/,
http://shanghhfg.wikidot.com/,
http://fhsshbx.wikidot.com/,
http://mngbsd.wikidot.com/,
http://jsrebx.wikidot.com/,
http://kdgszg.wikidot.com/,
http://hdfhsa.wikidot.com/,
http://jfdsfhd.wikidot.com/,
http://nfgssnt.wikidot.com/,
http://fdgsdas.wikidot.com/,
http://hjrszd.wikidot.com/,
http://hgjskk.wikidot.com/,
http://hedjszm.wikidot.com/,
http://hjuskf.wikidot.com/,
http://kdsrjsg.wikidot.com/,
http://nfdgsar.wikidot.com/,
http://ghkddsj.wikidot.com/,
http://fgjcbxg.wikidot.com/,
http://tgsahd.wikidot.com/,
http://zgsgsrk.wikidot.com/,
http://qrysxk.wikidot.com/,
http://gstrho.wikidot.com/,
http://hckdsk.wikidot.com/,
http://shafhf.wikidot.com/,
http://zsfxmnh.wikidot.com/,
http://mzjtia.wikidot.com/,
http://nfdhssg.wikidot.com/,
http://hkffbd.wikidot.com/,
http://zdfwjt.wikidot.com/,
http://xcrghs.wikidot.com/,
http://nsgzfg.wikidot.com/,
http://hgsdsa.wikidot.com/,
http://aezfgh.wikidot.com/,
http://bxbned.wikidot.com/,
http://gabzjjg.wikidot.com/,
http://hsgyfj.wikidot.com/,
http://hsfdhf.wikidot.com/,
http://zvshetg.wikidot.com/,
http://zzyurej.wikidot.com/,
http://lchdhjw.wikidot.com/,
http://hzjstgrs.wikidot.com/,
http://dzjhkfy.wikidot.com/,
http://jndhas.wikidot.com/,
http://zbhfher.wikidot.com/,
http://ytdghs.wikidot.com/,
http://whhsvs.wikidot.com/,
http://wfkfdsz.wikidot.com/,
http://qdjsgt.wikidot.com/,
http://mgssgtg.wikidot.com/,
http://jdgsjah.wikidot.com/,
http://fhkfak.wikidot.com/,
http://hfhfdds.wikidot.com/,
http://hskefx.wikidot.com/,
http://xykjdg.wikidot.com/,
http://qankfhd.wikidot.com/,
http://xakfhd.wikidot.com/,
http://bjsgsw.wikidot.com/,
http://hzgdstg.wikidot.com/,
http://hfgfhn.wikidot.com/,
http://ashdfyh.wikidot.com/,
http://dldhds.wikidot.com/,
http://wsjfh.wikidot.com/,
http://bxkdhkf.wikidot.com/,
http://ykkfpgh.wikidot.com/,
http://jzkfphj.wikidot.com/,
http://pjkfpof.wikidot.com/,
http://hdkdjf.wikidot.com/,
http://ddkfpgj.wikidot.com/,
http://cykfpjf.wikidot.com/,
http://hldkfpt.wikidot.com/,
http://jygkfphj.wikidot.com/,
http://lzkfphj.wikidot.com/,
http://tlkfphk.wikidot.com/,
http://jqkfpjh.wikidot.com/,
http://tskfpjg.wikidot.com/,
http://xzkfojh.wikidot.com/,
http://wxkfpjh.wikidot.com/,
http://zykfpjh.wikidot.com/,
http://yckfpkjn.wikidot.com/,
http://ntkfpjh.wikidot.com/,
http://szkfpjh.wikidot.com/,
http://lygkfphj.wikidot.com/,
http://chgeesd.wikidot.com/,
http://cckfplk.wikidot.com/,
http://jlkfplk.wikidot.com/,
http://thkfplk.wikidot.com/,
http://spkfplk.wikidot.com/,
http://yzkfplk.wikidot.com/,
http://bckfplk.wikidot.com/,
http://ddjkfplk.wikidot.com/,
http://jjhekfplk.wikidot.com/,
http://henkfplk.wikidot.com/,
http://jnskfplk.wikidot.com/,
http://dqkfplk.wikidot.com/,
http://gykfplk.wikidot.com/,
http://qthkfplk.wikidot.com/,
http://lpskfplk.wikidot.com/,
http://zykfplk.wikidot.com/,
http://kmkfplk.wikidot.com/,
http://ljkfplk.wikidot.com/,
http://qjkfplk.wikidot.com/,
http://yxkfplk.wikidot.com/,
http://zygkfplk.wikidot.com/,
http://zzkfplk.wikidot.com/,
http://qzkfplk.wikidot.com/,
http://xmkfplk.wikidot.com/,
http://ptkfplk.wikidot.com/,
http://fzkfplk.wikidot.com/,
http://tjkfplk.wikidot.com/,
http://cskfplk.wikidot.com/,
http://sjzkfplk.wikidot.com/,
http://cqkfplk.wikidot.com/,
http://bjkfplk.wikidot.com/,
http://dfsgfda.wikidot.com/,
http://dsfgdsf.wikidot.com/,
http://dsfgdsg.wikidot.com/,
http://sdgdsfyh.wikidot.com/,
http://dsgdhg.wikidot.com/,
http://ljmdsgdfg.wikidot.com/,
http://fdhggfkj.wikidot.com/,
http://fhfdsf.wikidot.com/,
http://dsfgsdhy.wikidot.com/,
http://dsfgdfftg.wikidot.com/,
http://sdtgdfd.wikidot.com/,
http://dsgdszfg.wikidot.com/,
http://jhtgjh.wikidot.com/,
http://zdfgds.wikidot.com/,
http://dsgdfdcx.wikidot.com/,
http://vdsghg.wikidot.com/,
http://aghfghj.wikidot.com/,
http://dsgdsdfg.wikidot.com/,
http://dfgsfg.wikidot.com/,
http://dfggd.wikidot.com/,
http://dfjhklfg.wikidot.com/,
http://gfvjhfg.wikidot.com/,
http://dsfgfdf.wikidot.com/,
http://fdstdy.wikidot.com/,
http://dsftdrcg.wikidot.com/,
http://bgfcgh.wikidot.com/,
http://ghfhj.wikidot.com/,
http://hgfghv.wikidot.com/,
http://sdgf.wikidot.com/,
http://gjmhfgh.wikidot.com/,
http://vhgvh.wikidot.com/,
http://hfghg.wikidot.com/,
http://fdgddf.wikidot.com/,
http://sdgd.wikidot.com/,
http://dsfdf.wikidot.com/,
http://sdgfd.wikidot.com/,
http://dsgdf.wikidot.com/,
http://fgdfsh.wikidot.com/,
http://fgsdf.wikidot.com/,
http://dsfgdf.wikidot.com/,
http://sdgfd.wikidot.com/main:layout,
http://sdgfd.wikidot.com/system:join,
http://sdgfd.wikidot.com/help:_home,
http://sdgfd.wikidot.com/main:contact,
http://sdgfd.wikidot.com/help:first-time-user,
http://sdgfd.wikidot.com/help:creating-pages,
http://sdgfd.wikidot.com/help:editing-pages,
http://sdgfd.wikidot.com/help:navigation-bars,
http://sdgfd.wikidot.com/help:using-modules,
http://sdgfd.wikidot.com/help:templates,
http://dsgdf.wikidot.com/main:layout,
http://dsgdf.wikidot.com/system:join,
http://dsgdf.wikidot.com/help:_home,
http://dsgdf.wikidot.com/main:contact,
http://dsgdf.wikidot.com/help:first-time-user,
http://dsgdf.wikidot.com/help:quick-reference,
http://dsgdf.wikidot.com/help:creating-pages,
http://dsgdf.wikidot.com/help:editing-pages,
http://dsgdf.wikidot.com/help:navigation-bars,
http://dsgdf.wikidot.com/help:templates,
http://dsgdf.wikidot.com/help:css-themes,
http://fgdfsh.wikidot.com/main:layout,
http://fgdfsh.wikidot.com/system:join,
http://fgdfsh.wikidot.com/main:about,
http://dsgdf.wikidot.com/main:about,
http://sdgfd.wikidot.com/main:about,
http://fgsdf.wikidot.com/main:about,
http://fgsdf.wikidot.com/main:layout,
http://fgsdf.wikidot.com/system:join,
http://fgsdf.wikidot.com/help:_home,
http://fgsdf.wikidot.com/main:contact,
http://fgsdf.wikidot.com/help:first-time-user,
http://fgsdf.wikidot.com/help:quick-reference,
http://fgsdf.wikidot.com/help:creating-pages,
http://fgsdf.wikidot.com/help:editing-pages,
http://fgsdf.wikidot.com/help:navigation-bars,
http://fgsdf.wikidot.com/help:templates,
http://fgsdf.wikidot.com/help:css-themes,
http://dsfgdf.wikidot.com/main:about,
http://dsfgdf.wikidot.com/main:layout,
http://dsfgdf.wikidot.com/system:join,
http://dsfgdf.wikidot.com/help:_home,
http://dsfgdf.wikidot.com/main:contact,
http://dsfgdf.wikidot.com/help:first-time-user,
http://dsfgdf.wikidot.com/help:quick-reference,
http://dsfgdf.wikidot.com/help:creating-pages,
http://dsfgdf.wikidot.com/help:editing-pages,
http://dsfgdf.wikidot.com/help:navigation-bars,
http://dsfgdf.wikidot.com/help:templates,
http://dsfgdf.wikidot.com/help:css-themes,
http://fgsdf.wikidot.com/main:home,
http://bcvfdf.wikidot.com/,
http://fdsgfxd.wikidot.com/,
http://lmhjvb.wikidot.com/,
http://dsfsdfg.wikidot.com/,
http://mfcgdsfg.wikidot.com/,
http://mhhdf.wikidot.com/,
http://ljkhjc.wikidot.com/,
http://wefdslo.wikidot.com/,
http://iuydfh.wikidot.com/,
http://pfyfdf.wikidot.com/,
http://ertryrt.wikidot.com/,
http://werdsfsx.wikidot.com/,
http://puhjigh.wikidot.com/,
http://iyufh.wikidot.com/,
http://iytrc.wikidot.com/,
http://uysedhb.wikidot.com/,
http://opirfgh.wikidot.com/,
http://uyrthf.wikidot.com/,
http://ytydfgc.wikidot.com/,
http://retgvcx.wikidot.com/,
http://yuifdg.wikidot.com/,
http://qwreersd.wikidot.com/,
http://tyutgf.wikidot.com/,
http://resfdsf.wikidot.com/,
http://rteddf.wikidot.com/,
http://aqddcz.wikidot.com/,
http://ohjvf.wikidot.com/,
http://yursdf.wikidot.com/,
http://wzsdasd.wikidot.com/,
http://phgfdc.wikidot.com/,
http://sdfwew.wikidot.com/,
http://dtfgrerf.wikidot.com/,
http://reweas.wikidot.com/,
http://qwerad.wikidot.com/,
http://qasdfd.wikidot.com/,
http://dfsgdf.wikidot.com/,
http://fddsfgdf.wikidot.com/,
http://jhiujyu.wikidot.com/,
http://sgtyrt.wikidot.com/,
http://asdsz.wikidot.com/,
http://srfsdf.wikidot.com/,
http://desrs.wikidot.com/,
http://sdfser.wikidot.com/,
http://sddfgs.wikidot.com/,
http://sdfsdg.wikidot.com/,
http://dsfsgfe.wikidot.com/,
http://gwserg.wikidot.com/,
http://xcfdg.wikidot.com/,
http://klfghgf.wikidot.com/,
http://gfdaa.wikidot.com/,
http://yedrcg.wikidot.com/,
http://swefzf.wikidot.com/,
http://dsfsza.wikidot.com/,
http://swezd.wikidot.com/,
http://hghgshg.wikidot.com/,
http://rtsdf.wikidot.com/,
http://dsaew.wikidot.com/,
http://dsfrwlo.wikidot.com/,
http://srdtg.wikidot.com/,
http://dfgwews.wikidot.com/,
http://dsggrw.wikidot.com/,
http://aserdtg.wikidot.com/,
http://dsgadtg.wikidot.com/,
http://eytder.wikidot.com/,
http://dsghsfh.wikidot.com/,
http://dsrww.wikidot.com/,
http://dsfzgds.wikidot.com/,
http://sdtgdf.wikidot.com/,
http://dszfas.wikidot.com/,
http://dfgfxd.wikidot.com/,
http://dsfdse.wikidot.com/,
http://sdrse.wikidot.com/,
http://eawfxf.wikidot.com/,
http://dssdfg.wikidot.com/,
http://dsfwef.wikidot.com/,
http://yyhfc.wikidot.com/,
http://fdsgd.wikidot.com/,
http://scxdfgdf.wikidot.com/,
http://dfsdraa.wikidot.com/,
http://fghdftgsd.wikidot.com/,
http://fsfplk.wikidot.com/,
http://gzfolk.wikidot.com/,
http://stfplk.wikidot.com/,
http://szfplk.wikidot.com/,
http://zhfplk.wikidot.com/,
http://zqfplk.wikidot.com/,
http://zjfplk.wikidot.com/,
http://sgfplk.wikidot.com/,
http://mmfplk.wikidot.com/,
http://jmfplk.wikidot.com/,
http://hyfplk.wikidot.com/,
http://hzfplk.wikidot.com/,
http://swfplk.wikidot.com/,
http://mzfplk.wikidot.com/,
http://yjfplk.wikidot.com/,
http://zsfplk.wikidot.com/,
http://nnfplk.wikidot.com/,
http://qyfplk.wikidot.com/,
http://glfplk.wikidot.com/,
http://dgfplk.wikidot.com/,
http://bhfplk.wikidot.com/,
http://lzfplk.wikidot.com/,
http://qzfplk.wikidot.com/,
http://rlfplk.wikidot.com/,
http://wzfplk.wikidot.com/,
http://ncfplk.wikidot.com/,
http://hezfplk.wikidot.com/,
http://bsfplk.wikidot.com/,
http://fcgfplk.wikidot.com/,
http://ggfplk.wikidot.com/,
http://ganzfolk.wikidot.com/,
http://ycfplk.wikidot.com/,
http://xxfplk.wikidot.com/,
http://pxfplk.wikidot.com/,
http://jafplk.wikidot.com/,
http://srfplk.wikidot.com/,
http://jjfplk.wikidot.com/,
http://fzfplk.wikidot.com/,
http://fuzfplk.wikidot.com/,
http://jdzfplk.wikidot.com/,
http://zzfplk.wikidot.com/,
http://xmfplk.wikidot.com/,
http://quzfplk.wikidot.com/,
http://ptfplk.wikidot.com/,
http://lyfplk.wikidot.com/,
http://czfplk.wikidot.com/,
http://tyfplk.wikidot.com/,
http://shzfplk.wikidot.com/,
http://yqfplk.wikidot.com/,
http://dtfplk.wikidot.com/,
http://csfplk.wikidot.com/,
http://zjjfplk.wikidot.com/,
http://huyfplk.wikidot.com/,
http://xtfplk.wikidot.com/,
http://zzhfplk.wikidot.com/,
http://xyfplk.wikidot.com/,
http://ximfplk.wikidot.com/,
http://whfplk.wikidot.com/,
http://syfplk.wikidot.com/,
http://hsfplk.wikidot.com/,
http://hgfplk.wikidot.com/,
http://shzhfplk.wikidot.com/,
http://xgfplk.wikidot.com/,
http://xnfplk.wikidot.com/,
http://xzfplk.wikidot.com/,
http://zgfplk.wikidot.com/,
http://guyfplk.wikidot.com/,
http://luzfplk.wikidot.com/,
http://deyfplk.wikidot.com/,
http://cdfplk.wikidot.com/,
http://suzfplk.wikidot.com/,
http://qrfplk.wikidot.com/,
http://hffplk.wikidot.com/,
http://fgfplk.wikidot.com/,
http://fdgfplk.wikidot.com/,
http://aqfplk.wikidot.com/,
http://fhfplk.wikidot.com/,
http://masfplk.wikidot.com/,
http://xbfplk.wikidot.com/,
http://xcfplk.wikidot.com/,
http://yuntfplk.wikidot.com/,
http://weihfplk.wikidot.com/,
http://qdfplk.wikidot.com/,
http://jnfplk.wikidot.com/,
http://dyfplk.wikidot.com/,
http://lwfplk.wikidot.com/,
http://hezefplk.wikidot.com/,
http://wffplk.wikidot.com/,
http://zbfplk.wikidot.com/,
http://zgxfplk.wikidot.com/,
http://dezfplk.wikidot.com/,
http://lcfplk.wikidot.com/,
http://lffplk.wikidot.com/,
http://tafplk.wikidot.com/,
http://zhzhfplk.wikidot.com/,
http://zhengzfplk.wikidot.com/,
http://pdsfplk.wikidot.com/,
http://xinxfplk.wikidot.com/,
http://luoyfplk.wikidot.com/,
http://kffplk.wikidot.com/,
http://yyfplk.wikidot.com/,
http://xucfplk.wikidot.com/,
http://jzuofplk.wikidot.com/,
http://hbfplk.wikidot.com/,
http://anyfplk.wikidot.com/,
http://zkfplk.wikidot.com/,
http://xinyfplk.wikidot.com/,
http://sxmfplk.wikidot.com/,
http://sqfplk.wikidot.com/,
http://lhfplk.wikidot.com/,
http://thfplk.wikidot.com/,
http://spfplk.wikidot.com/,
http://liaoyfplk.wikidot.com/,
http://jlfplk.wikidot.com/,
http://ccfplk.wikidot.com/,
http://qqhefplk.wikidot.com/,
http://hegfplk.wikidot.com/,
http://jmsfplk.wikidot.com/,
http://hebfplk.wikidot.com/,
http://dqfplk.wikidot.com/,
http://yucfplk.wikidot.com/,
http://qthfplk.wikidot.com/,
http://mdfplk.wikidot.com/,
http://jxfplk.wikidot.com/,
http://hehfplk.wikidot.com/,
http://wxfplk.wikidot.com/,
http://taizfplk.wikidot.com/,
http://suzhfplk.wikidot.com/,
http://ntfplk.wikidot.com/,
http://njfplk.wikidot.com/,
http://hzhfplk.wikidot.com/,
http://lygfplk.wikidot.com/,
http://xuzfplk.wikidot.com/,
http://yancfplk.wikidot.com/,
http://yzhfplk.wikidot.com/,
http://wzhfplk.wikidot.com/,
http://jiaxfplk.wikidot.com/,
http://zxfplk.wikidot.com/,
http://nbfplk.wikidot.com/,
http://huzfplk.wikidot.com/,
http://hjfplk.wikidot.com/,
http://trfplk.wikidot.com/,
http://tzfplk.wikidot.com/,
http://weizfplk.wikidot.com/,
http://zhsfplk.wikidot.com/,
http://lpsfplk.wikidot.com/,
http://guizfplk.wikidot.com/,
http://bjfplk.wikidot.com/,
http://ansfplk.wikidot.com/,
http://zyfplk.wikidot.com/,
http://ztfplk.wikidot.com/,
http://rxfplk.wikidot.com/,
http://qjfplk.wikidot.com/,
http://ljfplk.wikidot.com/,
http://kmfplk.wikidot.com/,
http://xiyfplk.wikidot.com/,
http://wnfplk.wikidot.com/,
http://xafplk.wikidot.com/,
http://bjifplk.wikidot.com/,
http://tcfplk.wikidot.com/,
http://tsfplk.wikidot.com/,
http://lzhfplk.wikidot.com/,
http://jygfplk.wikidot.com/,
http://jcfplk.wikidot.com/,
http://byfplk.wikidot.com/,
http://zhyfplk.wikidot.com/,
http://plfplk.wikidot.com/,
http://wwfplk.wikidot.com/,
http://jqfplk.wikidot.com/,
http://hmfplk.wikidot.com/,
http://shyfplk.wikidot.com/,
http://fusfplk.wikidot.com/,
http://dlfplk.wikidot.com/,
http://bxfplk.wikidot.com/,
http://asfplk.wikidot.com/,
http://liayfplk.wikidot.com/,
http://fxfplk.wikidot.com/,
http://ykfplk.wikidot.com/,
http://jzfplk.wikidot.com/,
http://ddfplk.wikidot.com/,
http://cyfplk.wikidot.com/,
http://hhhtfplk.wikidot.com/,
http://pjfplk.wikidot.com/,
http://hldfplk.wikidot.com/,
http://tlfplk.wikidot.com/,
http://wuhfplk.wikidot.com/,
http://tonglfplk.wikidot.com/,
http://tjfplk.wikidot.com/,
http://btfplk.wikidot.com/,
http://cffplk.wikidot.com/,
http://wlmqfplk.wikidot.com/,
http://sjzfplk.wikidot.com/,
http://lsfplk.wikidot.com/,
http://hdfplk.wikidot.com/,
http://chdfplk.wikidot.com/,
http://mxcsa.wikidot.com/,
http://wxzaz.wikidot.com/,
http://zxsa.wikidot.com/,
http://zsdc.wikidot.com/,
http://zxcvh.wikidot.com/,
http://sddgh.wikidot.com/,
http://zdsw.wikidot.com/,
http://dfsw.wikidot.com/,
http://dsgf1.wikidot.com/,
http://s44as.wikidot.com/,
http://gzxd.wikidot.com/,
http://fgse.wikidot.com/,
http://rdcs.wikidot.com/,
http://fsec.wikidot.com/,
http://mkhy.wikidot.com/,
http://nbvc.wikidot.com/,
http://cvfd.wikidot.com/,
http://bcvf.wikidot.com/,
http://fdty.wikidot.com/,
http://ufgh.wikidot.com/,
http://hifv.wikidot.com/,
http://ytre.wikidot.com/,
http://tyug.wikidot.com/,
http://oytf.wikidot.com/,
http://sdeo.wikidot.com/,
http://hjsa.wikidot.com/,
http://fdgd.wikidot.com/,
http://fgrw.wikidot.com/,
http://ghfo.wikidot.com/,
http://ngiw.wikidot.com/,
http://jybf.wikidot.com/,
http://fvup.wikidot.com/,
http://hjfy.wikidot.com/,
http://pdun.wikidot.com/,
http://cvbs.wikidot.com/,
http://fdgdfsgdf.wikidot.com/,
http://xzczs.wikidot.com/,
http://dcvxc.wikidot.com/,
http://sdd4ds.wikidot.com/,
http://jkyv.wikidot.com/,
http://fadsgwsea.wikidot.com/,
http://sadas.wikidot.com/,
http://ffdg1df.wikidot.com/,
http://migr.wikidot.com/,
http://ffdgdf.wikidot.com/,
http://czsa.wikidot.com/,
http://hqyv.wikidot.com/,
http://lfdsds.wikidot.com/,
http://fghfh.wikidot.com/,
http://hghfg.wikidot.com/,
http://fdstgf.wikidot.com/,
http://jgsw.wikidot.com/,
http://piyr.wikidot.com/,
http://ghfs.wikidot.com/,
http://fjtv.wikidot.com/,
http://taobao.wikidot.com/,
http://chengbang.ccedpw.com/,
http://nanzhuo3.shop.bxgtd.com/,
http://qiaojiao778.shop.bxgtd.com/,
http://doushi4566.shop.bxgtd.com/,
http://taodao.shop.bxgtd.com/,
http://jidun84.shop.bxgtd.com/,
http://tianquang67.shop.bxgtd.com/,
http://bojuenu1639.shop.bxgtd.com/,
http://cengtied15.shop.bxgtd.com/,
http://yeniangzh49.shop.bxgtd.com/,
http://www.andu729.yhtx.tv/,
http://www.tangyan83.yhtx.tv/,
http://www.ditang2.yhtx.tv/,
http://www.kaonai43043.yhtx.tv/,
http://www.duishuo.yhtx.tv/,
http://www.pabenye4186.yhtx.tv/,
http://www.waxuanqi61.yhtx.tv/,
http://www.haoganson86.yhtx.tv/,
http://www.piaoqinc13.yhtx.tv/,
http://www.qianghuai78.yhtx.tv/,
http://www.shenxiao34.yhtx.tv/,
http://www.fengniuge59.yhtx.tv/,
http://www.daojiongr71.yhtx.tv/,
http://www.sunkaizu34.yhtx.tv/,
http://www.cengshes84.yhtx.tv/,
http://www.51xsq.com/qy/yunqiang208/,
http://www.51xsq.com/qy/youshi66/,
http://www.51xsq.com/qy/kongwei55447/,
http://www.51xsq.com/qy/wendou/,
http://www.51xsq.com/qy/bibei8414689/,
http://www.51xsq.com/qy/hudiecu7642/,
http://www.51xsq.com/qy/tuncangt95/,
http://www.51xsq.com/qy/jianniuda12/,
http://www.51xsq.com/qy/yanpaijio77/,
http://www.51xsq.com/qy/feixiong68/,
http://www.51xsq.com/qy/tuofengj14/,
http://www.51xsq.com/qy/mengsong30/,

0