Nova Resource:Deployment-prep/Search

From Wikitech
Jump to: navigation, search

Testing out the search box:

curl -i 'http://10.4.1.81:8123/prefix/enwiki/DOO3H?namespaces=0&offset=0&limit=10&version=2.1&iwlimit=10&searchall=0'

Indexer:

From the indexer-cron command of lucene.jobs.sh:

java -Xmx8000m -cp LuceneSearch.jar org.wikimedia.lsearch.spell.SuggestBuilder -l >> $base/log/log-spell 2>&1
java -Xmx4000m -cp LuceneSearch.jar org.wikimedia.lsearch.related.RelatedBuilder -l >> $base/log/log-related 2>&1 &


Enabling debug mode:

In /a/search/conf/lsearch.log4j :

log4j.appender.A1.Threshold=DEBUG



Initial import of enwiki.

root@deployment-searchidx01:/a/search/conf# sudo -u lsearch /a/search/lucene.jobs.sh import-db enwiki
Dumping enwiki...
2013-03-06 00:45:43: enwiki (ID 23993) 99 pages (162.6|162.6/sec all|curr), 100 revs (164.3|164.3/sec all|curr), ETA 2013-03-06 00:46:21 [max 6301]
2013-03-06 00:45:43: enwiki (ID 23993) 199 pages (301.8|3921.6/sec all|curr), 200 revs (303.3|1970.6/sec all|curr), ETA 2013-03-06 00:46:03 [max 6301]
2013-03-06 00:45:43: enwiki (ID 23993) 299 pages (419.7|5637.7/sec all|curr), 300 revs (421.1|1885.5/sec all|curr), ETA 2013-03-06 00:45:57 [max 6301]
2013-03-06 00:45:43: enwiki (ID 23993) 399 pages (517.5|6825.3/sec all|curr), 400 revs (518.8|1710.6/sec all|curr), ETA 2013-03-06 00:45:54 [max 6301]
2013-03-06 00:45:43: enwiki (ID 23993) 499 pages (607.1|9786.4/sec all|curr), 500 revs (608.3|1961.2/sec all|curr), ETA 2013-03-06 00:45:53 [max 6301]
2013-03-06 00:45:43: enwiki (ID 23993) 599 pages (684.7|11316.2/sec all|curr), 600 revs (685.8|1889.2/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:43: enwiki (ID 23993) 699 pages (666.2|4007.7/sec all|curr), 700 revs (667.1|573.3/sec all|curr), ETA 2013-03-06 00:45:52 [max 6301]
2013-03-06 00:45:43: enwiki (ID 23993) 799 pages (660.2|4962.5/sec all|curr), 800 revs (661.0|621.1/sec all|curr), ETA 2013-03-06 00:45:52 [max 6301]
2013-03-06 00:45:43: enwiki (ID 23993) 899 pages (689.8|9661.5/sec all|curr), 900 revs (690.5|1074.7/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:44: enwiki (ID 23993) 999 pages (737.1|19236.7/sec all|curr), 1000 revs (737.9|1925.6/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:44: enwiki (ID 23993) 1099 pages (647.9|3223.7/sec all|curr), 1100 revs (648.5|293.3/sec all|curr), ETA 2013-03-06 00:45:52 [max 6301]
2013-03-06 00:45:44: enwiki (ID 23993) 1199 pages (625.3|5418.3/sec all|curr), 1200 revs (625.8|451.9/sec all|curr), ETA 2013-03-06 00:45:52 [max 6301]
2013-03-06 00:45:44: enwiki (ID 23993) 1299 pages (638.2|11005.8/sec all|curr), 1300 revs (638.7|847.3/sec all|curr), ETA 2013-03-06 00:45:52 [max 6301]
2013-03-06 00:45:44: enwiki (ID 23993) 1399 pages (664.0|19593.8/sec all|curr), 1400 revs (664.5|1400.6/sec all|curr), ETA 2013-03-06 00:45:52 [max 6301]
2013-03-06 00:45:44: enwiki (ID 23993) 1499 pages (652.7|7904.2/sec all|curr), 1500 revs (653.2|527.3/sec all|curr), ETA 2013-03-06 00:45:52 [max 6301]
2013-03-06 00:45:45: enwiki (ID 23993) 1599 pages (624.1|6023.0/sec all|curr), 1600 revs (624.5|376.7/sec all|curr), ETA 2013-03-06 00:45:52 [max 6301]
2013-03-06 00:45:45: enwiki (ID 23993) 1699 pages (633.1|13974.8/sec all|curr), 1700 revs (633.5|822.5/sec all|curr), ETA 2013-03-06 00:45:52 [max 6301]
2013-03-06 00:45:45: enwiki (ID 23993) 1799 pages (652.0|23828.7/sec all|curr), 1800 revs (652.4|1324.6/sec all|curr), ETA 2013-03-06 00:45:52 [max 6301]
2013-03-06 00:45:45: enwiki (ID 23993) 1899 pages (675.2|35522.6/sec all|curr), 1900 revs (675.5|1870.6/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:45: enwiki (ID 23993) 1999 pages (694.4|30260.3/sec all|curr), 2000 revs (694.8|1513.8/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:45: enwiki (ID 23993) 2099 pages (704.6|20931.0/sec all|curr), 2100 revs (705.0|997.2/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:45: enwiki (ID 23993) 2199 pages (706.1|16219.2/sec all|curr), 2200 revs (706.4|737.6/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:46: enwiki (ID 23993) 2299 pages (679.0|8476.1/sec all|curr), 2300 revs (679.3|368.7/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301] 2013-03-06 00:45:46: enwiki (ID 23993) 2399 pages (675.5|14470.3/sec all|curr), 2400 revs (675.8|603.2/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:46: enwiki (ID 23993) 2499 pages (671.3|14610.3/sec all|curr), 2500 revs (671.6|584.6/sec all|curr), ETA 2013-03-06 00:45:52 [max 6301]
2013-03-06 00:45:46: enwiki (ID 23993) 2599 pages (676.1|21341.6/sec all|curr), 2600 revs (676.3|821.1/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:46: enwiki (ID 23993) 2699 pages (671.6|15464.3/sec all|curr), 2700 revs (671.8|573.0/sec all|curr), ETA 2013-03-06 00:45:52 [max 6301]
2013-03-06 00:45:46: enwiki (ID 23993) 2799 pages (673.7|20566.1/sec all|curr), 2800 revs (673.9|734.8/sec all|curr), ETA 2013-03-06 00:45:52 [max 6301]
2013-03-06 00:45:46: enwiki (ID 23993) 2899 pages (678.6|24746.6/sec all|curr), 2900 revs (678.8|853.6/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:47: enwiki (ID 23993) 2999 pages (680.3|21962.5/sec all|curr), 3000 revs (680.5|732.3/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:47: enwiki (ID 23993) 3099 pages (683.3|24425.2/sec all|curr), 3100 revs (683.5|788.2/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:47: enwiki (ID 23993) 3199 pages (688.6|29073.1/sec all|curr), 3200 revs (688.8|908.8/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:47: enwiki (ID 23993) 3299 pages (696.2|35347.4/sec all|curr), 3300 revs (696.4|1071.5/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:47: enwiki (ID 23993) 3399 pages (694.2|21561.8/sec all|curr), 3400 revs (694.4|634.4/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:47: enwiki (ID 23993) 3499 pages (698.4|30896.8/sec all|curr), 3500 revs (698.6|883.0/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:47: enwiki (ID 23993) 3599 pages (701.1|29194.4/sec all|curr), 3600 revs (701.3|811.2/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:47: enwiki (ID 23993) 3699 pages (704.3|31143.7/sec all|curr), 3700 revs (704.5|841.9/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:48: enwiki (ID 23993) 3799 pages (706.7|30645.7/sec all|curr), 3800 revs (706.9|806.7/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:48: enwiki (ID 23993) 3899 pages (716.4|58792.5/sec all|curr), 3900 revs (716.6|1507.9/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:48: enwiki (ID 23993) 3999 pages (728.2|80563.6/sec all|curr), 4000 revs (728.4|2014.6/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:48: enwiki (ID 23993) 4099 pages (726.3|27019.9/sec all|curr), 4100 revs (726.5|659.2/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:48: enwiki (ID 23993) 4199 pages (737.7|85804.1/sec all|curr), 4200 revs (737.8|2043.4/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:48: enwiki (ID 23993) 4299 pages (743.6|48439.5/sec all|curr), 4300 revs (743.8|1126.8/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:48: enwiki (ID 23993) 4399 pages (744.2|33933.2/sec all|curr), 4400 revs (744.4|771.4/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:48: enwiki (ID 23993) 4499 pages (740.6|27487.4/sec all|curr), 4500 revs (740.8|611.0/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:48: enwiki (ID 23993) 4599 pages (736.6|27137.9/sec all|curr), 4600 revs (736.7|590.1/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:49: enwiki (ID 23993) 4699 pages (737.6|37030.6/sec all|curr), 4700 revs (737.7|788.1/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:49: enwiki (ID 23993) 4799 pages (739.0|39045.1/sec all|curr), 4800 revs (739.2|813.6/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:49: enwiki (ID 23993) 4899 pages (737.7|33343.1/sec all|curr), 4900 revs (737.9|680.6/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:49: enwiki (ID 23993) 4999 pages (742.4|54043.1/sec all|curr), 5000 revs (742.6|1081.1/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:49: enwiki (ID 23993) 5099 pages (747.0|54704.4/sec all|curr), 5100 revs (747.1|1072.8/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:49: enwiki (ID 23993) 5199 pages (753.1|67729.7/sec all|curr), 5200 revs (753.3|1302.7/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:49: enwiki (ID 23993) 5299 pages (754.1|42816.1/sec all|curr), 5300 revs (754.2|808.0/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:49: enwiki (ID 23993) 5399 pages (752.8|37188.6/sec all|curr), 5400 revs (752.9|688.8/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:49: enwiki (ID 23993) 5499 pages (753.9|44922.4/sec all|curr), 5500 revs (754.0|816.9/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:50: enwiki (ID 23993) 5599 pages (752.6|38686.1/sec all|curr), 5600 revs (752.8|690.9/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:50: enwiki (ID 23993) 5699 pages (745.8|28212.6/sec all|curr), 5700 revs (746.0|495.0/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:50: enwiki (ID 23993) 5799 pages (737.4|26038.0/sec all|curr), 5800 revs (737.5|449.0/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
2013-03-06 00:45:50: enwiki (ID 23993) 5853 pages (725.2|28295.9/sec all|curr), 5853 revs (725.2|256.2/sec all|curr), ETA 2013-03-06 00:45:51 [max 6301]
Importing enwiki ...
MediaWiki lucene-search indexer - rebuild all indexes associated with a database.
Trying config file at path /var/lib/lsearch/.lsearch.conf
Trying config file at path /a/search/lucene-search/lsearch.conf
Trying config file at path /etc/lsearch.conf
log4j:WARN No such property [maxBackupIndex] in org.apache.log4j.DailyRollingFileAppender.
WARNING: in Global Configuration: defined new architecture single for eewiki
WARNING: in Global Configuration: role "prefix" already defined for database "eewiki". Overwriting.
WARNING: in Global Configuration: role "spell" already defined for database "eewiki". Overwriting.
Error reading InitialiseSettings.php from url file:///a/search/conf/InitialiseSettings.php : For input string: "NS_MAIN"
MediaWiki lucene-search indexer - index builder from xml database dumps.

5,000 pages (1,907.669/sec), 5,000 revs (1,907.669/sec)
5,853 pages (2,047.936/sec), 5,853 revs (2,047.936/sec)
Processed 1,000 titles (2,409.639/sec)
Processed 2,000 titles (3,795.066/sec)
Processed 3,000 titles (4,761.905/sec)
Processed 4,000 titles (5,277.045/sec)
Processed 5,000 titles (5,274.262/sec)
1,000 pages (160.694/sec), 1,000 revs (160.694/sec)
2,000 pages (193.461/sec), 2,000 revs (193.461/sec)
3,000 pages (214.669/sec), 3,000 revs (214.669/sec)
4,000 pages (197.863/sec), 4,000 revs (197.863/sec)
5,000 pages (225.337/sec), 5,000 revs (225.337/sec)
5,853 pages (231.335/sec), 5,853 revs (231.335/sec)
Finished indexing in 30s, with final index optimization in 0s
Total time: 30s
MediaWiki lucene-search indexer - rebuild prefix index used for ajax suggestions.
Processed 1,000 titles (8,130.081/sec)
Processed 2,000 titles (7,843.137/sec)
Processed 3,000 titles (8,000/sec)
Processed 4,000 titles (3,289.474/sec)
Processed 5,000 titles (3,709.199/sec)
Processed 10,000 prefixes (161,290.323/sec)
Processed 20,000 prefixes (88,888.889/sec)
Processed 30,000 prefixes (102,389.078/sec)
Processed 40,000 prefixes (117,302.053/sec)
Processed 50,000 prefixes (135,135.135/sec)
Processed 60,000 prefixes (139,534.884/sec)
Processed 70,000 prefixes (139,720.559/sec)
Processed 1,000 title keys (43,478.261/sec)
Processed 2,000 title keys (46,511.628/sec)
Processed 3,000 title keys (48,387.097/sec)
Processed 4,000 title keys (49,382.716/sec)
Processed 5,000 title keys (46,296.296/sec)
Finished in 2s
MediaWiki lucene-search indexer - build spelling suggestion index.

1,000 pages (4,032.258/sec), 1,000 revs (4,032.258/sec)
2,000 pages (2,400.96/sec), 2,000 revs (2,400.96/sec)
3,000 pages (2,446.982/sec), 3,000 revs (2,446.982/sec)
4,000 pages (990.344/sec), 4,000 revs (990.344/sec)
5,000 pages (1,045.369/sec), 5,000 revs (1,045.369/sec)
5,853 pages (1,095.452/sec), 5,853 revs (1,095.452/sec)
Processed 10,000 terms (303,030.303/sec)
Processed 20,000 terms (246,913.58/sec)
Processed 30,000 terms (193,548.387/sec)
Processed 40,000 terms (192,307.692/sec)
Processed 50,000 terms (219,298.246/sec)
Processed 60,000 terms (218,181.818/sec)
Processed 70,000 terms (202,898.551/sec)
Processed 80,000 terms (205,655.527/sec)
Processed 90,000 terms (190,677.966/sec)
Processed 100,000 terms (165,562.914/sec)
Processed 110,000 terms (166,414.523/sec)
Processed 120,000 terms (160,427.807/sec)
Processed 130,000 terms (160,295.931/sec)
Processed 140,000 terms (154,355.017/sec)
Processed 150,000 terms (151,057.402/sec)
Processed 160,000 terms (151,228.733/sec)
Processed 170,000 terms (151,921.358/sec)
Processed 180,000 terms (154,373.928/sec)
Processed 190,000 terms (152,365.678/sec)
Processed 200,000 terms (151,975.684/sec)
Processed 210,000 terms (154,185.022/sec)
Processed 220,000 terms (153,096.729/sec)
Processed 230,000 terms (153,027.279/sec)
Processed 240,000 terms (153,944.836/sec)
Processed 250,000 terms (110,277.9/sec)
Processed 260,000 terms (111,635.895/sec)
Processed 270,000 terms (113,160.101/sec)
Processed 280,000 terms (114,472.608/sec)
Processed 290,000 terms (114,533.965/sec)
Processed 300,000 terms (116,144.019/sec)
Processed 310,000 terms (117,202.268/sec)
Processed 320,000 terms (116,618.076/sec)
Processed 330,000 terms (117,815.066/sec)
Processed 340,000 terms (118,508.191/sec)
Processed 350,000 terms (120,274.914/sec)
Processed 360,000 terms (120,361.083/sec)
Processed 370,000 terms (120,717.781/sec)
Processed 380,000 terms (121,561.1/sec)
Processed 390,000 terms (121,193.288/sec)
Processed 400,000 terms (121,175.401/sec)
Processed 410,000 terms (121,878.716/sec)
Processed 420,000 terms (119,487.909/sec)
Processed 430,000 terms (120,684.816/sec)
Processed 440,000 terms (121,715.076/sec)
Processed 450,000 terms (121,951.22/sec)
Processed 460,000 terms (122,896.073/sec)
Processed 470,000 terms (123,554.154/sec)
Processed 480,000 terms (124,191.462/sec)
Processed 490,000 terms (125,866.941/sec)
Processed 500,000 terms (126,582.278/sec)
Processed 510,000 terms (127,851.592/sec)
Processed 520,000 terms (129,385.419/sec)
Processed 530,000 terms (129,933.807/sec)
Processed 540,000 terms (130,750.605/sec)
Processed 550,000 terms (131,641.934/sec)
Processed 560,000 terms (132,168.987/sec)
Processed 570,000 terms (131,944.444/sec)
Processed 580,000 terms (133,058.041/sec)
Processed 590,000 terms (133,877.921/sec)
Processed 600,000 terms (129,254.632/sec)
Processed 610,000 terms (129,209.913/sec)
Processed 620,000 terms (130,006.291/sec)
Processed 630,000 terms (129,789.864/sec)
Processed 640,000 terms (130,852.586/sec)
Processed 650,000 terms (131,552.317/sec)
Processed 660,000 terms (131,789.137/sec)
Processed 670,000 terms (132,228.143/sec)
Processed 680,000 terms (133,411.811/sec)
Finished making spell-check index in 12s
Finished build in 46s
Imported enwiki
root@deployment-searchidx01:/a/search/conf#