Data

Basic Statistics

The following table shows some statistics about the crawled data in the GAW project:

Date Snapshot partitions Data in TB Start Time (UTC) End Time (UTC) Unique URLs
2012-10 2922 2.67 2012-10-09 10:50:36 2012-10-29 11:03:21 40,077,499
2013-02 6477 5.90 2013-02-11 16:27:43 2013-03-05 12:16:15 170,625,168
2013-12 7089 6.46 2013-12-04 15:01:14 2014-01-16 06:59:35 107,608,630
2014-05 6339 5.77 2014-04-30 06:40:49 2014-05-13 14:51:21 107,969,414
2014-12 6154 5.60 2014-12-01 08:28:56 2014-12-08 06:01:53 100,145,401
2015-05 6449 5.87 2015-05-27 08:25:50 2015-06-09 06:04:34 103,575,494
2015-12 5928 5.40 2015-12-15 09:42:56 2016-01-05 07:40:02 101,279,483
2016-06 7289 6.64 2016-06-01 07:34:12 2016-06-13 07:01:58 112,824,260
2016-12 7088 6.46 2016-12-08 08:13:23 2016-12-17 12:15:04 99,952,250
2017-06 7417 6.76 2017-06-12 07:44:06 2017-06-21 07:09:56 100,847,532
2017-12 7374 6.72 2017-12-05 12:42:27 2017-12-17 10:17:47 102,133,168
2018-06 7106 6.47 2018-06-18 07:08:26 2018-06-29 21:31:07 101,022,822
2018-12 7297 6.65 2018-12-03 09:37:41 2018-12-14 22:32:14 100,768,903
2019-06 7012 6.39 2019-06-13 06:48:30 2019-06-27 11:07:40 99,797,675
2019-12 7035 6.41 2019-12-03 10:08:00 2019-12-10 11:32:52 100,976,809
2020-06 6953 6.41 2020-06-03 07:44:24 2020-06-11 11:44:55 99,999,403
2020-12 6998 6.37 2020-12-01 09:19:54 2020-12-10 07:58:47 100,269,671
2021-06 7207 6.57 2021-06-01 11:21:10 2021-06-14 06:33:25 105,563,149
2021-12 7202 6.56 2021-12-07 07:20:59 2021-12-17 07:07:08 101,919,655
2022-06 7855 7.20 2022-06-09 12:06:53 2022-06-20 10:25:27