Obsolete:Data Dump Redesign/Throughput

This page is kept for historical purposes only.

Stubs

• 200K Samples
  • ESWIKI
    • real 20m19.417s / 25M
    • user 13m40.800s
    • sys 0m31.240s
    • ESWIKI has just under 1.2million entries
    • 1,170,721 as of 2009 06 15 Pages
    • Currently takes a little more then an hour to dump stubs
  • ENWIKI
    • 17,194,115 Pages
    • Split a the stubs from ENWIKI 20090604 into 200k chunks and process
    • 100Stubs @ 8core = 13 Machines

Splitting Stubs

Wed Jun 24 00:50:25 2009 Processed 10000 pages out of 200000 for segment 0
Wed Jun 24 01:18:41 2009 Closing Segment 0 after 200000 total pages processed. Opening next segment

• 30Mins per stub