[2024-06-21 11:33:31,149] INFO: Will use torch.nn.parallel.DistributedDataParallel() and 4 gpus [2024-06-21 11:33:31,238] INFO: NVIDIA GeForce GTX 1080 Ti [2024-06-21 11:33:31,238] INFO: NVIDIA GeForce GTX 1080 Ti [2024-06-21 11:33:31,238] INFO: NVIDIA GeForce GTX 1080 Ti [2024-06-21 11:33:31,238] INFO: NVIDIA GeForce GTX 1080 Ti [2024-06-21 11:33:37,910] INFO: using dtype=torch.float32 [2024-06-21 11:33:38,971] INFO: using attention_type=math [2024-06-21 11:33:38,988] INFO: using attention_type=math [2024-06-21 11:33:39,004] INFO: using attention_type=math [2024-06-21 11:33:39,021] INFO: using attention_type=math [2024-06-21 11:33:39,038] INFO: using attention_type=math [2024-06-21 11:33:39,054] INFO: using attention_type=math [2024-06-21 11:33:43,580] INFO: mlpf_kwargs: {'input_dim': 17, 'num_classes': 6, 'input_encoding': 'joint', 'pt_mode': 'linear', 'eta_mode': 'linear', 'sin_phi_mode': 'linear', 'cos_phi_mode': 'linear', 'energy_mode': 'linear', 'elemtypes_nonzero': [1, 2], 'learned_representation_mode': 'last', 'conv_type': 'attention', 'num_convs': 3, 'dropout_ff': 0.0, 'dropout_conv_id_mha': 0.0, 'dropout_conv_id_ff': 0.0, 'dropout_conv_reg_mha': 0.0, 'dropout_conv_reg_ff': 0.0, 'activation': 'relu', 'head_dim': 16, 'num_heads': 32, 'attention_type': 'math'} [2024-06-21 11:33:43,580] INFO: Loaded model weights from /pfvol/experiments/MLPF_clic_backbone_pyg-clic_20240429_101112_971749/best_weights.pth [2024-06-21 11:33:44,798] INFO: DistributedDataParallel( (module): DeepMET( (nn): Sequential( (0): Linear(in_features=6, out_features=256, bias=True) (1): ELU(alpha=1.0) (2): LayerNorm((256,), eps=1e-05, elementwise_affine=True) (3): Dropout(p=0, inplace=False) (4): Linear(in_features=256, out_features=2, bias=True) ) ) ) [2024-06-21 11:33:44,799] INFO: DeepMET Trainable parameters: 2818 [2024-06-21 11:33:44,799] INFO: DeepMET Non-trainable parameters: 0 [2024-06-21 11:33:44,799] INFO: DeepMET Total parameters: 2818 [2024-06-21 11:33:44,801] INFO: Modules Trainable parameters Non-tranable parameters module.nn.0.weight 1536 0 module.nn.0.bias 256 0 module.nn.2.weight 256 0 module.nn.2.bias 256 0 module.nn.4.weight 512 0 module.nn.4.bias 2 0 [2024-06-21 11:33:44,802] INFO: Creating experiment dir /pfvol/experiments/MLPF_clic_backbone_pyg-clic_20240429_101112_971749/Finetuning_PFCands_20240621_113331_058383 [2024-06-21 11:33:44,802] INFO: Model directory /pfvol/experiments/MLPF_clic_backbone_pyg-clic_20240429_101112_971749/Finetuning_PFCands_20240621_113331_058383 [2024-06-21 11:33:44,838] INFO: train_dataset: clic_edm_ttbar_pf, 800800 [2024-06-21 11:33:44,948] INFO: valid_dataset: clic_edm_ttbar_pf, 200200 [2024-06-21 11:33:44,970] INFO: Initiating epoch #1 train run on device rank=0 [2024-06-21 11:36:54,015] INFO: Initiating epoch #1 valid run on device rank=0 [2024-06-21 11:37:42,175] INFO: Rank 0: epoch=1 / 400 train_loss=11.1421 valid_loss=10.7747 stale=0 time=3.95m eta=1577.4m [2024-06-21 11:37:42,176] INFO: Initiating epoch #2 train run on device rank=0 [2024-06-21 11:40:42,212] INFO: Initiating epoch #2 valid run on device rank=0 [2024-06-21 11:41:30,365] INFO: Rank 0: epoch=2 / 400 train_loss=10.7067 valid_loss=10.7508 stale=0 time=3.8m eta=1543.6m [2024-06-21 11:41:30,423] INFO: Initiating epoch #3 train run on device rank=0 [2024-06-21 11:44:33,730] INFO: Initiating epoch #3 valid run on device rank=0 [2024-06-21 11:45:21,093] INFO: Rank 0: epoch=3 / 400 train_loss=10.6894 valid_loss=10.7377 stale=0 time=3.84m eta=1535.3m [2024-06-21 11:45:21,153] INFO: Initiating epoch #4 train run on device rank=0 [2024-06-21 11:48:25,572] INFO: Initiating epoch #4 valid run on device rank=0 [2024-06-21 11:49:12,793] INFO: Rank 0: epoch=4 / 400 train_loss=10.6784 valid_loss=10.7284 stale=0 time=3.86m eta=1530.9m [2024-06-21 11:49:12,801] INFO: Initiating epoch #5 train run on device rank=0 [2024-06-21 11:52:15,467] INFO: Initiating epoch #5 valid run on device rank=0 [2024-06-21 11:53:04,354] INFO: Rank 0: epoch=5 / 400 train_loss=10.6700 valid_loss=10.7208 stale=0 time=3.86m eta=1526.5m [2024-06-21 11:53:04,377] INFO: Initiating epoch #6 train run on device rank=0 [2024-06-21 11:56:05,297] INFO: Initiating epoch #6 valid run on device rank=0 [2024-06-21 11:56:53,264] INFO: Rank 0: epoch=6 / 400 train_loss=10.6630 valid_loss=10.7141 stale=0 time=3.81m eta=1519.4m [2024-06-21 11:56:53,291] INFO: Initiating epoch #7 train run on device rank=0 [2024-06-21 11:59:55,208] INFO: Initiating epoch #7 valid run on device rank=0 [2024-06-21 12:00:42,486] INFO: Rank 0: epoch=7 / 400 train_loss=10.6565 valid_loss=10.7066 stale=0 time=3.82m eta=1513.5m [2024-06-21 12:00:42,529] INFO: Initiating epoch #8 train run on device rank=0 [2024-06-21 12:03:44,532] INFO: Initiating epoch #8 valid run on device rank=0 [2024-06-21 12:04:36,741] INFO: Rank 0: epoch=8 / 400 train_loss=10.6480 valid_loss=10.6980 stale=0 time=3.9m eta=1512.3m [2024-06-21 12:04:37,473] INFO: Initiating epoch #9 train run on device rank=0 [2024-06-21 12:07:36,073] INFO: Initiating epoch #9 valid run on device rank=0 [2024-06-21 12:08:22,924] INFO: Rank 0: epoch=9 / 400 train_loss=10.6416 valid_loss=10.6925 stale=0 time=3.76m eta=1504.6m [2024-06-21 12:08:22,940] INFO: Initiating epoch #10 train run on device rank=0 [2024-06-21 12:11:26,223] INFO: Initiating epoch #10 valid run on device rank=0 [2024-06-21 12:12:13,699] INFO: Rank 0: epoch=10 / 400 train_loss=10.6365 valid_loss=10.6875 stale=0 time=3.85m eta=1500.7m [2024-06-21 12:12:13,748] INFO: Initiating epoch #11 train run on device rank=0 [2024-06-21 12:15:14,063] INFO: Initiating epoch #11 valid run on device rank=0 [2024-06-21 12:16:01,707] INFO: Rank 0: epoch=11 / 400 train_loss=10.6317 valid_loss=10.6827 stale=0 time=3.8m eta=1495.1m [2024-06-21 12:16:01,739] INFO: Initiating epoch #12 train run on device rank=0 [2024-06-21 12:19:03,506] INFO: Initiating epoch #12 valid run on device rank=0 [2024-06-21 12:19:51,171] INFO: Rank 0: epoch=12 / 400 train_loss=10.6271 valid_loss=10.6780 stale=0 time=3.82m eta=1490.7m [2024-06-21 12:19:51,196] INFO: Initiating epoch #13 train run on device rank=0 [2024-06-21 12:22:54,393] INFO: Initiating epoch #13 valid run on device rank=0 [2024-06-21 12:23:41,745] INFO: Rank 0: epoch=13 / 400 train_loss=10.6225 valid_loss=10.6733 stale=0 time=3.84m eta=1486.9m [2024-06-21 12:23:41,824] INFO: Initiating epoch #14 train run on device rank=0 [2024-06-21 12:26:45,333] INFO: Initiating epoch #14 valid run on device rank=0 [2024-06-21 12:27:33,832] INFO: Rank 0: epoch=14 / 400 train_loss=10.6179 valid_loss=10.6687 stale=0 time=3.87m eta=1483.7m [2024-06-21 12:27:33,868] INFO: Initiating epoch #15 train run on device rank=0 [2024-06-21 12:30:37,425] INFO: Initiating epoch #15 valid run on device rank=0 [2024-06-21 12:31:25,728] INFO: Rank 0: epoch=15 / 400 train_loss=10.6134 valid_loss=10.6640 stale=0 time=3.86m eta=1480.4m [2024-06-21 12:31:25,803] INFO: Initiating epoch #16 train run on device rank=0 [2024-06-21 12:34:32,280] INFO: Initiating epoch #16 valid run on device rank=0 [2024-06-21 12:35:24,089] INFO: Rank 0: epoch=16 / 400 train_loss=10.6089 valid_loss=10.6596 stale=0 time=3.97m eta=1479.6m [2024-06-21 12:35:24,594] INFO: Initiating epoch #17 train run on device rank=0 [2024-06-21 12:38:25,503] INFO: Initiating epoch #17 valid run on device rank=0 [2024-06-21 12:39:15,870] INFO: Rank 0: epoch=17 / 400 train_loss=10.6045 valid_loss=10.6551 stale=0 time=3.85m eta=1476.0m [2024-06-21 12:39:16,418] INFO: Initiating epoch #18 train run on device rank=0 [2024-06-21 12:42:16,429] INFO: Initiating epoch #18 valid run on device rank=0 [2024-06-21 12:43:07,417] INFO: Rank 0: epoch=18 / 400 train_loss=10.6001 valid_loss=10.6505 stale=0 time=3.85m eta=1472.3m [2024-06-21 12:43:08,024] INFO: Initiating epoch #19 train run on device rank=0 [2024-06-21 12:46:08,202] INFO: Initiating epoch #19 valid run on device rank=0 [2024-06-21 12:46:58,837] INFO: Rank 0: epoch=19 / 400 train_loss=10.5956 valid_loss=10.6460 stale=0 time=3.85m eta=1468.5m [2024-06-21 12:46:59,365] INFO: Initiating epoch #20 train run on device rank=0 [2024-06-21 12:50:01,556] INFO: Initiating epoch #20 valid run on device rank=0 [2024-06-21 12:50:52,553] INFO: Rank 0: epoch=20 / 400 train_loss=10.5910 valid_loss=10.6413 stale=0 time=3.89m eta=1465.4m [2024-06-21 12:50:53,248] INFO: Initiating epoch #21 train run on device rank=0 [2024-06-21 12:53:53,201] INFO: Initiating epoch #21 valid run on device rank=0 [2024-06-21 12:54:44,551] INFO: Rank 0: epoch=21 / 400 train_loss=10.5864 valid_loss=10.6365 stale=0 time=3.86m eta=1461.7m [2024-06-21 12:54:45,268] INFO: Initiating epoch #22 train run on device rank=0 [2024-06-21 12:57:47,211] INFO: Initiating epoch #22 valid run on device rank=0 [2024-06-21 12:58:38,161] INFO: Rank 0: epoch=22 / 400 train_loss=10.5816 valid_loss=10.6315 stale=0 time=3.88m eta=1458.5m [2024-06-21 12:58:38,777] INFO: Initiating epoch #23 train run on device rank=0 [2024-06-21 13:01:41,541] INFO: Initiating epoch #23 valid run on device rank=0 [2024-06-21 13:02:32,237] INFO: Rank 0: epoch=23 / 400 train_loss=10.5763 valid_loss=10.6255 stale=0 time=3.89m eta=1455.3m [2024-06-21 13:02:32,963] INFO: Initiating epoch #24 train run on device rank=0 [2024-06-21 13:05:30,771] INFO: Initiating epoch #24 valid run on device rank=0 [2024-06-21 13:06:20,830] INFO: Rank 0: epoch=24 / 400 train_loss=10.5698 valid_loss=10.6190 stale=0 time=3.8m eta=1450.7m [2024-06-21 13:06:21,432] INFO: Initiating epoch #25 train run on device rank=0 [2024-06-21 13:09:21,449] INFO: Initiating epoch #25 valid run on device rank=0 [2024-06-21 13:10:13,087] INFO: Rank 0: epoch=25 / 400 train_loss=10.5640 valid_loss=10.6135 stale=0 time=3.86m eta=1447.0m [2024-06-21 13:10:13,550] INFO: Initiating epoch #26 train run on device rank=0 [2024-06-21 13:13:14,421] INFO: Initiating epoch #26 valid run on device rank=0 [2024-06-21 13:14:01,981] INFO: Rank 0: epoch=26 / 400 train_loss=10.5587 valid_loss=10.6080 stale=0 time=3.81m eta=1442.5m [2024-06-21 13:14:02,007] INFO: Initiating epoch #27 train run on device rank=0 [2024-06-21 13:17:05,234] INFO: Initiating epoch #27 valid run on device rank=0 [2024-06-21 13:17:51,270] INFO: Rank 0: epoch=27 / 400 train_loss=10.5533 valid_loss=10.6025 stale=0 time=3.82m eta=1438.2m [2024-06-21 13:17:51,282] INFO: Initiating epoch #28 train run on device rank=0 [2024-06-21 13:20:54,991] INFO: Initiating epoch #28 valid run on device rank=0 [2024-06-21 13:21:42,212] INFO: Rank 0: epoch=28 / 400 train_loss=10.5479 valid_loss=10.5968 stale=0 time=3.85m eta=1434.2m [2024-06-21 13:21:42,300] INFO: Initiating epoch #29 train run on device rank=0 [2024-06-21 13:24:45,529] INFO: Initiating epoch #29 valid run on device rank=0 [2024-06-21 13:25:33,723] INFO: Rank 0: epoch=29 / 400 train_loss=10.5424 valid_loss=10.5912 stale=0 time=3.86m eta=1430.4m [2024-06-21 13:25:33,771] INFO: Initiating epoch #30 train run on device rank=0 [2024-06-21 13:29:03,904] INFO: Initiating epoch #30 valid run on device rank=0 [2024-06-21 13:29:50,827] INFO: Rank 0: epoch=30 / 400 train_loss=10.5370 valid_loss=10.5859 stale=0 time=4.28m eta=1431.9m [2024-06-21 13:29:50,851] INFO: Initiating epoch #31 train run on device rank=0 [2024-06-21 13:32:52,555] INFO: Initiating epoch #31 valid run on device rank=0 [2024-06-21 13:33:40,185] INFO: Rank 0: epoch=31 / 400 train_loss=10.5318 valid_loss=10.5807 stale=0 time=3.82m eta=1427.4m [2024-06-21 13:33:40,210] INFO: Initiating epoch #32 train run on device rank=0 [2024-06-21 13:36:43,539] INFO: Initiating epoch #32 valid run on device rank=0 [2024-06-21 13:37:31,451] INFO: Rank 0: epoch=32 / 400 train_loss=10.5265 valid_loss=10.5754 stale=0 time=3.85m eta=1423.4m [2024-06-21 13:37:31,518] INFO: Initiating epoch #33 train run on device rank=0 [2024-06-21 13:40:34,278] INFO: Initiating epoch #33 valid run on device rank=0 [2024-06-21 13:41:22,102] INFO: Rank 0: epoch=33 / 400 train_loss=10.5213 valid_loss=10.5701 stale=0 time=3.84m eta=1419.3m [2024-06-21 13:41:22,123] INFO: Initiating epoch #34 train run on device rank=0 [2024-06-21 13:44:23,487] INFO: Initiating epoch #34 valid run on device rank=0 [2024-06-21 13:45:15,579] INFO: Rank 0: epoch=34 / 400 train_loss=10.5160 valid_loss=10.5649 stale=0 time=3.89m eta=1415.7m [2024-06-21 13:45:16,140] INFO: Initiating epoch #35 train run on device rank=0 [2024-06-21 13:48:16,199] INFO: Initiating epoch #35 valid run on device rank=0 [2024-06-21 13:49:08,918] INFO: Rank 0: epoch=35 / 400 train_loss=10.5107 valid_loss=10.5596 stale=0 time=3.88m eta=1412.0m [2024-06-21 13:49:09,633] INFO: Initiating epoch #36 train run on device rank=0 [2024-06-21 13:52:08,974] INFO: Initiating epoch #36 valid run on device rank=0 [2024-06-21 13:53:00,265] INFO: Rank 0: epoch=36 / 400 train_loss=10.5054 valid_loss=10.5543 stale=0 time=3.84m eta=1408.0m [2024-06-21 13:53:00,765] INFO: Initiating epoch #37 train run on device rank=0 [2024-06-21 13:56:01,125] INFO: Initiating epoch #37 valid run on device rank=0 [2024-06-21 13:56:52,589] INFO: Rank 0: epoch=37 / 400 train_loss=10.5002 valid_loss=10.5491 stale=0 time=3.86m eta=1404.2m [2024-06-21 13:56:53,287] INFO: Initiating epoch #38 train run on device rank=0 [2024-06-21 13:59:53,755] INFO: Initiating epoch #38 valid run on device rank=0 [2024-06-21 14:00:44,676] INFO: Rank 0: epoch=38 / 400 train_loss=10.4949 valid_loss=10.5439 stale=0 time=3.86m eta=1400.3m [2024-06-21 14:00:45,285] INFO: Initiating epoch #39 train run on device rank=0 [2024-06-21 14:03:45,195] INFO: Initiating epoch #39 valid run on device rank=0 [2024-06-21 14:04:36,270] INFO: Rank 0: epoch=39 / 400 train_loss=10.4897 valid_loss=10.5387 stale=0 time=3.85m eta=1396.4m [2024-06-21 14:04:36,986] INFO: Initiating epoch #40 train run on device rank=0 [2024-06-21 14:07:38,257] INFO: Initiating epoch #40 valid run on device rank=0 [2024-06-21 14:08:31,186] INFO: Rank 0: epoch=40 / 400 train_loss=10.4845 valid_loss=10.5335 stale=0 time=3.9m eta=1392.9m [2024-06-21 14:08:31,827] INFO: Initiating epoch #41 train run on device rank=0 [2024-06-21 14:11:32,903] INFO: Initiating epoch #41 valid run on device rank=0 [2024-06-21 14:12:23,360] INFO: Rank 0: epoch=41 / 400 train_loss=10.4794 valid_loss=10.5285 stale=0 time=3.86m eta=1389.1m [2024-06-21 14:12:24,218] INFO: Initiating epoch #42 train run on device rank=0 [2024-06-21 14:15:26,463] INFO: Initiating epoch #42 valid run on device rank=0 [2024-06-21 14:16:17,125] INFO: Rank 0: epoch=42 / 400 train_loss=10.4743 valid_loss=10.5234 stale=0 time=3.88m eta=1385.4m [2024-06-21 14:16:17,635] INFO: Initiating epoch #43 train run on device rank=0 [2024-06-21 14:19:17,089] INFO: Initiating epoch #43 valid run on device rank=0 [2024-06-21 14:20:04,749] INFO: Rank 0: epoch=43 / 400 train_loss=10.4693 valid_loss=10.5185 stale=0 time=3.79m eta=1380.9m [2024-06-21 14:20:04,828] INFO: Initiating epoch #44 train run on device rank=0 [2024-06-21 14:23:11,246] INFO: Initiating epoch #44 valid run on device rank=0 [2024-06-21 14:23:58,709] INFO: Rank 0: epoch=44 / 400 train_loss=10.4643 valid_loss=10.5136 stale=0 time=3.9m eta=1377.3m [2024-06-21 14:23:58,773] INFO: Initiating epoch #45 train run on device rank=0 [2024-06-21 14:27:04,086] INFO: Initiating epoch #45 valid run on device rank=0 [2024-06-21 14:27:51,713] INFO: Rank 0: epoch=45 / 400 train_loss=10.4595 valid_loss=10.5089 stale=0 time=3.88m eta=1373.6m [2024-06-21 14:27:51,721] INFO: Initiating epoch #46 train run on device rank=0 [2024-06-21 14:30:55,121] INFO: Initiating epoch #46 valid run on device rank=0 [2024-06-21 14:31:43,085] INFO: Rank 0: epoch=46 / 400 train_loss=10.4547 valid_loss=10.5042 stale=0 time=3.86m eta=1369.6m [2024-06-21 14:31:43,170] INFO: Initiating epoch #47 train run on device rank=0 [2024-06-21 14:34:46,579] INFO: Initiating epoch #47 valid run on device rank=0 [2024-06-21 14:35:39,011] INFO: Rank 0: epoch=47 / 400 train_loss=10.4500 valid_loss=10.4997 stale=0 time=3.93m eta=1366.2m [2024-06-21 14:35:39,569] INFO: Initiating epoch #48 train run on device rank=0 [2024-06-21 14:38:37,633] INFO: Initiating epoch #48 valid run on device rank=0 [2024-06-21 14:39:25,809] INFO: Rank 0: epoch=48 / 400 train_loss=10.4454 valid_loss=10.4952 stale=0 time=3.77m eta=1361.7m [2024-06-21 14:39:25,850] INFO: Initiating epoch #49 train run on device rank=0 [2024-06-21 14:42:30,186] INFO: Initiating epoch #49 valid run on device rank=0 [2024-06-21 14:43:18,933] INFO: Rank 0: epoch=49 / 400 train_loss=10.4410 valid_loss=10.4909 stale=0 time=3.88m eta=1357.9m [2024-06-21 14:43:18,994] INFO: Initiating epoch #50 train run on device rank=0 [2024-06-21 14:46:19,048] INFO: Initiating epoch #50 valid run on device rank=0 [2024-06-21 14:47:09,508] INFO: Rank 0: epoch=50 / 400 train_loss=10.4366 valid_loss=10.4867 stale=0 time=3.84m eta=1353.9m [2024-06-21 14:47:10,140] INFO: Initiating epoch #51 train run on device rank=0 [2024-06-21 14:50:07,688] INFO: Initiating epoch #51 valid run on device rank=0 [2024-06-21 14:50:57,464] INFO: Rank 0: epoch=51 / 400 train_loss=10.4323 valid_loss=10.4826 stale=0 time=3.79m eta=1349.5m [2024-06-21 14:50:58,128] INFO: Initiating epoch #52 train run on device rank=0 [2024-06-21 14:54:00,175] INFO: Initiating epoch #52 valid run on device rank=0 [2024-06-21 14:54:50,891] INFO: Rank 0: epoch=52 / 400 train_loss=10.4282 valid_loss=10.4786 stale=0 time=3.88m eta=1345.8m [2024-06-21 14:54:51,430] INFO: Initiating epoch #53 train run on device rank=0 [2024-06-21 14:57:52,161] INFO: Initiating epoch #53 valid run on device rank=0 [2024-06-21 14:58:39,479] INFO: Rank 0: epoch=53 / 400 train_loss=10.4242 valid_loss=10.4747 stale=0 time=3.8m eta=1341.6m [2024-06-21 14:58:39,522] INFO: Initiating epoch #54 train run on device rank=0 [2024-06-21 15:01:41,407] INFO: Initiating epoch #54 valid run on device rank=0 [2024-06-21 15:02:30,033] INFO: Rank 0: epoch=54 / 400 train_loss=10.4203 valid_loss=10.4709 stale=0 time=3.84m eta=1337.6m [2024-06-21 15:02:30,075] INFO: Initiating epoch #55 train run on device rank=0 [2024-06-21 15:05:37,599] INFO: Initiating epoch #55 valid run on device rank=0 [2024-06-21 15:06:26,242] INFO: Rank 0: epoch=55 / 400 train_loss=10.4164 valid_loss=10.4672 stale=0 time=3.94m eta=1334.1m [2024-06-21 15:06:26,338] INFO: Initiating epoch #56 train run on device rank=0 [2024-06-21 15:09:28,234] INFO: Initiating epoch #56 valid run on device rank=0 [2024-06-21 15:10:15,661] INFO: Rank 0: epoch=56 / 400 train_loss=10.4127 valid_loss=10.4636 stale=0 time=3.82m eta=1330.0m [2024-06-21 15:10:15,728] INFO: Initiating epoch #57 train run on device rank=0 [2024-06-21 15:13:18,927] INFO: Initiating epoch #57 valid run on device rank=0 [2024-06-21 15:14:07,111] INFO: Rank 0: epoch=57 / 400 train_loss=10.4091 valid_loss=10.4601 stale=0 time=3.86m eta=1326.1m [2024-06-21 15:14:07,166] INFO: Initiating epoch #58 train run on device rank=0 [2024-06-21 15:17:07,488] INFO: Initiating epoch #58 valid run on device rank=0 [2024-06-21 15:17:55,781] INFO: Rank 0: epoch=58 / 400 train_loss=10.4055 valid_loss=10.4566 stale=0 time=3.81m eta=1321.9m [2024-06-21 15:17:55,835] INFO: Initiating epoch #59 train run on device rank=0 [2024-06-21 15:20:56,698] INFO: Initiating epoch #59 valid run on device rank=0 [2024-06-21 15:21:48,426] INFO: Rank 0: epoch=59 / 400 train_loss=10.4020 valid_loss=10.4533 stale=0 time=3.88m eta=1318.1m [2024-06-21 15:21:49,028] INFO: Initiating epoch #60 train run on device rank=0 [2024-06-21 15:24:48,845] INFO: Initiating epoch #60 valid run on device rank=0 [2024-06-21 15:25:39,148] INFO: Rank 0: epoch=60 / 400 train_loss=10.3987 valid_loss=10.4501 stale=0 time=3.84m eta=1314.1m [2024-06-21 15:25:39,545] INFO: Initiating epoch #61 train run on device rank=0 [2024-06-21 15:28:39,625] INFO: Initiating epoch #61 valid run on device rank=0 [2024-06-21 15:29:31,809] INFO: Rank 0: epoch=61 / 400 train_loss=10.3956 valid_loss=10.4471 stale=0 time=3.87m eta=1310.3m [2024-06-21 15:29:32,470] INFO: Initiating epoch #62 train run on device rank=0 [2024-06-21 15:32:31,361] INFO: Initiating epoch #62 valid run on device rank=0 [2024-06-21 15:33:23,684] INFO: Rank 0: epoch=62 / 400 train_loss=10.3926 valid_loss=10.4443 stale=0 time=3.85m eta=1306.5m [2024-06-21 15:33:24,238] INFO: Initiating epoch #63 train run on device rank=0 [2024-06-21 15:36:21,949] INFO: Initiating epoch #63 valid run on device rank=0 [2024-06-21 15:37:10,482] INFO: Rank 0: epoch=63 / 400 train_loss=10.3897 valid_loss=10.4415 stale=0 time=3.77m eta=1302.1m [2024-06-21 15:37:10,699] INFO: Initiating epoch #64 train run on device rank=0 [2024-06-21 15:40:12,581] INFO: Initiating epoch #64 valid run on device rank=0 [2024-06-21 15:40:59,858] INFO: Rank 0: epoch=64 / 400 train_loss=10.3870 valid_loss=10.4389 stale=0 time=3.82m eta=1298.1m [2024-06-21 15:40:59,906] INFO: Initiating epoch #65 train run on device rank=0 [2024-06-21 15:44:02,437] INFO: Initiating epoch #65 valid run on device rank=0 [2024-06-21 15:44:51,698] INFO: Rank 0: epoch=65 / 400 train_loss=10.3844 valid_loss=10.4364 stale=0 time=3.86m eta=1294.2m [2024-06-21 15:44:51,757] INFO: Initiating epoch #66 train run on device rank=0 [2024-06-21 15:47:52,552] INFO: Initiating epoch #66 valid run on device rank=0 [2024-06-21 15:48:41,574] INFO: Rank 0: epoch=66 / 400 train_loss=10.3819 valid_loss=10.4340 stale=0 time=3.83m eta=1290.2m [2024-06-21 15:48:42,246] INFO: Initiating epoch #67 train run on device rank=0 [2024-06-21 15:51:47,427] INFO: Initiating epoch #67 valid run on device rank=0 [2024-06-21 15:52:36,657] INFO: Rank 0: epoch=67 / 400 train_loss=10.3795 valid_loss=10.4316 stale=0 time=3.91m eta=1286.6m [2024-06-21 15:52:36,983] INFO: Initiating epoch #68 train run on device rank=0 [2024-06-21 15:55:42,203] INFO: Initiating epoch #68 valid run on device rank=0 [2024-06-21 15:56:30,911] INFO: Rank 0: epoch=68 / 400 train_loss=10.3772 valid_loss=10.4294 stale=0 time=3.9m eta=1282.9m [2024-06-21 15:56:30,931] INFO: Initiating epoch #69 train run on device rank=0 [2024-06-21 15:59:30,664] INFO: Initiating epoch #69 valid run on device rank=0 [2024-06-21 16:00:17,747] INFO: Rank 0: epoch=69 / 400 train_loss=10.3749 valid_loss=10.4272 stale=0 time=3.78m eta=1278.6m [2024-06-21 16:00:17,773] INFO: Initiating epoch #70 train run on device rank=0 [2024-06-21 16:03:21,471] INFO: Initiating epoch #70 valid run on device rank=0 [2024-06-21 16:04:10,106] INFO: Rank 0: epoch=70 / 400 train_loss=10.3727 valid_loss=10.4250 stale=0 time=3.87m eta=1274.8m [2024-06-21 16:04:10,172] INFO: Initiating epoch #71 train run on device rank=0 [2024-06-21 16:07:13,226] INFO: Initiating epoch #71 valid run on device rank=0 [2024-06-21 16:08:00,406] INFO: Rank 0: epoch=71 / 400 train_loss=10.3705 valid_loss=10.4228 stale=0 time=3.84m eta=1270.9m [2024-06-21 16:08:00,451] INFO: Initiating epoch #72 train run on device rank=0 [2024-06-21 16:11:02,715] INFO: Initiating epoch #72 valid run on device rank=0 [2024-06-21 16:11:56,229] INFO: Rank 0: epoch=72 / 400 train_loss=10.3682 valid_loss=10.4206 stale=0 time=3.93m eta=1267.3m [2024-06-21 16:11:56,799] INFO: Initiating epoch #73 train run on device rank=0 [2024-06-21 16:14:55,156] INFO: Initiating epoch #73 valid run on device rank=0 [2024-06-21 16:15:45,688] INFO: Rank 0: epoch=73 / 400 train_loss=10.3660 valid_loss=10.4187 stale=0 time=3.81m eta=1263.3m [2024-06-21 16:15:46,203] INFO: Initiating epoch #74 train run on device rank=0 [2024-06-21 16:18:47,033] INFO: Initiating epoch #74 valid run on device rank=0 [2024-06-21 16:19:34,881] INFO: Rank 0: epoch=74 / 400 train_loss=10.3641 valid_loss=10.4169 stale=0 time=3.81m eta=1259.2m [2024-06-21 16:19:34,923] INFO: Initiating epoch #75 train run on device rank=0 [2024-06-21 16:22:36,776] INFO: Initiating epoch #75 valid run on device rank=0 [2024-06-21 16:23:28,190] INFO: Rank 0: epoch=75 / 400 train_loss=10.3623 valid_loss=10.4153 stale=0 time=3.89m eta=1255.5m [2024-06-21 16:23:28,829] INFO: Initiating epoch #76 train run on device rank=0 [2024-06-21 16:26:27,447] INFO: Initiating epoch #76 valid run on device rank=0 [2024-06-21 16:27:27,072] INFO: Rank 0: epoch=76 / 400 train_loss=10.3607 valid_loss=10.4138 stale=0 time=3.97m eta=1252.1m [2024-06-21 16:27:29,303] INFO: Initiating epoch #77 train run on device rank=0 [2024-06-21 16:30:21,516] INFO: Initiating epoch #77 valid run on device rank=0 [2024-06-21 16:31:12,358] INFO: Rank 0: epoch=77 / 400 train_loss=10.3591 valid_loss=10.4123 stale=0 time=3.72m eta=1247.8m [2024-06-21 16:31:13,103] INFO: Initiating epoch #78 train run on device rank=0 [2024-06-21 16:34:08,248] INFO: Initiating epoch #78 valid run on device rank=0 [2024-06-21 16:34:58,295] INFO: Rank 0: epoch=78 / 400 train_loss=10.3577 valid_loss=10.4109 stale=0 time=3.75m eta=1243.5m [2024-06-21 16:34:58,920] INFO: Initiating epoch #79 train run on device rank=0 [2024-06-21 16:37:50,034] INFO: Initiating epoch #79 valid run on device rank=0 [2024-06-21 16:38:37,275] INFO: Rank 0: epoch=79 / 400 train_loss=10.3562 valid_loss=10.4096 stale=0 time=3.64m eta=1238.8m [2024-06-21 16:38:37,294] INFO: Initiating epoch #80 train run on device rank=0 [2024-06-21 16:41:29,536] INFO: Initiating epoch #80 valid run on device rank=0 [2024-06-21 16:42:16,407] INFO: Rank 0: epoch=80 / 400 train_loss=10.3549 valid_loss=10.4083 stale=0 time=3.65m eta=1234.1m [2024-06-21 16:42:16,457] INFO: Initiating epoch #81 train run on device rank=0 [2024-06-21 16:45:08,553] INFO: Initiating epoch #81 valid run on device rank=0 [2024-06-21 16:45:54,161] INFO: Rank 0: epoch=81 / 400 train_loss=10.3536 valid_loss=10.4071 stale=0 time=3.63m eta=1229.3m [2024-06-21 16:45:54,172] INFO: Initiating epoch #82 train run on device rank=0 [2024-06-21 16:48:47,421] INFO: Initiating epoch #82 valid run on device rank=0 [2024-06-21 16:49:34,056] INFO: Rank 0: epoch=82 / 400 train_loss=10.3523 valid_loss=10.4059 stale=0 time=3.66m eta=1224.8m [2024-06-21 16:49:34,082] INFO: Initiating epoch #83 train run on device rank=0 [2024-06-21 16:52:27,131] INFO: Initiating epoch #83 valid run on device rank=0 [2024-06-21 16:53:14,324] INFO: Rank 0: epoch=83 / 400 train_loss=10.3511 valid_loss=10.4047 stale=0 time=3.67m eta=1220.2m [2024-06-21 16:53:14,423] INFO: Initiating epoch #84 train run on device rank=0 [2024-06-21 16:56:11,441] INFO: Initiating epoch #84 valid run on device rank=0 [2024-06-21 16:56:57,043] INFO: Rank 0: epoch=84 / 400 train_loss=10.3499 valid_loss=10.4036 stale=0 time=3.71m eta=1215.9m [2024-06-21 16:56:57,093] INFO: Initiating epoch #85 train run on device rank=0 [2024-06-21 16:59:54,074] INFO: Initiating epoch #85 valid run on device rank=0 [2024-06-21 17:00:39,880] INFO: Rank 0: epoch=85 / 400 train_loss=10.3487 valid_loss=10.4025 stale=0 time=3.71m eta=1211.5m [2024-06-21 17:00:39,949] INFO: Initiating epoch #86 train run on device rank=0 [2024-06-21 17:03:36,524] INFO: Initiating epoch #86 valid run on device rank=0 [2024-06-21 17:04:22,675] INFO: Rank 0: epoch=86 / 400 train_loss=10.3477 valid_loss=10.4015 stale=0 time=3.71m eta=1207.2m [2024-06-21 17:04:22,702] INFO: Initiating epoch #87 train run on device rank=0 [2024-06-21 17:07:21,881] INFO: Initiating epoch #87 valid run on device rank=0 [2024-06-21 17:08:10,599] INFO: Rank 0: epoch=87 / 400 train_loss=10.3466 valid_loss=10.4005 stale=0 time=3.8m eta=1203.2m [2024-06-21 17:08:10,615] INFO: Initiating epoch #88 train run on device rank=0 [2024-06-21 17:11:14,275] INFO: Initiating epoch #88 valid run on device rank=0 [2024-06-21 17:12:01,526] INFO: Rank 0: epoch=88 / 400 train_loss=10.3456 valid_loss=10.3995 stale=0 time=3.85m eta=1199.3m [2024-06-21 17:12:01,544] INFO: Initiating epoch #89 train run on device rank=0 [2024-06-21 17:15:02,740] INFO: Initiating epoch #89 valid run on device rank=0 [2024-06-21 17:15:50,253] INFO: Rank 0: epoch=89 / 400 train_loss=10.3446 valid_loss=10.3986 stale=0 time=3.81m eta=1195.4m [2024-06-21 17:15:50,297] INFO: Initiating epoch #90 train run on device rank=0 [2024-06-21 17:18:51,182] INFO: Initiating epoch #90 valid run on device rank=0 [2024-06-21 17:19:38,181] INFO: Rank 0: epoch=90 / 400 train_loss=10.3436 valid_loss=10.3976 stale=0 time=3.8m eta=1191.4m [2024-06-21 17:19:38,195] INFO: Initiating epoch #91 train run on device rank=0 [2024-06-21 17:22:38,980] INFO: Initiating epoch #91 valid run on device rank=0 [2024-06-21 17:23:24,843] INFO: Rank 0: epoch=91 / 400 train_loss=10.3427 valid_loss=10.3968 stale=0 time=3.78m eta=1187.3m [2024-06-21 17:23:24,904] INFO: Initiating epoch #92 train run on device rank=0 [2024-06-21 17:26:29,651] INFO: Initiating epoch #92 valid run on device rank=0 [2024-06-21 17:27:19,010] INFO: Rank 0: epoch=92 / 400 train_loss=10.3418 valid_loss=10.3959 stale=0 time=3.9m eta=1183.7m [2024-06-21 17:27:19,041] INFO: Initiating epoch #93 train run on device rank=0 [2024-06-21 17:30:22,703] INFO: Initiating epoch #93 valid run on device rank=0 [2024-06-21 17:31:10,653] INFO: Rank 0: epoch=93 / 400 train_loss=10.3409 valid_loss=10.3950 stale=0 time=3.86m eta=1179.9m [2024-06-21 17:31:10,679] INFO: Initiating epoch #94 train run on device rank=0 [2024-06-21 17:34:18,970] INFO: Initiating epoch #94 valid run on device rank=0 [2024-06-21 17:35:06,405] INFO: Rank 0: epoch=94 / 400 train_loss=10.3400 valid_loss=10.3942 stale=0 time=3.93m eta=1176.3m [2024-06-21 17:35:06,419] INFO: Initiating epoch #95 train run on device rank=0 [2024-06-21 17:38:12,587] INFO: Initiating epoch #95 valid run on device rank=0 [2024-06-21 17:39:00,542] INFO: Rank 0: epoch=95 / 400 train_loss=10.3392 valid_loss=10.3934 stale=0 time=3.9m eta=1172.7m [2024-06-21 17:39:00,555] INFO: Initiating epoch #96 train run on device rank=0 [2024-06-21 17:42:06,209] INFO: Initiating epoch #96 valid run on device rank=0 [2024-06-21 17:42:57,664] INFO: Rank 0: epoch=96 / 400 train_loss=10.3383 valid_loss=10.3926 stale=0 time=3.95m eta=1169.2m [2024-06-21 17:42:58,230] INFO: Initiating epoch #97 train run on device rank=0 [2024-06-21 17:45:59,785] INFO: Initiating epoch #97 valid run on device rank=0 [2024-06-21 17:46:50,335] INFO: Rank 0: epoch=97 / 400 train_loss=10.3375 valid_loss=10.3918 stale=0 time=3.87m eta=1165.4m [2024-06-21 17:46:51,120] INFO: Initiating epoch #98 train run on device rank=0 [2024-06-21 17:49:53,334] INFO: Initiating epoch #98 valid run on device rank=0 [2024-06-21 17:50:42,517] INFO: Rank 0: epoch=98 / 400 train_loss=10.3367 valid_loss=10.3910 stale=0 time=3.86m eta=1161.6m [2024-06-21 17:50:42,531] INFO: Initiating epoch #99 train run on device rank=0 [2024-06-21 17:53:46,237] INFO: Initiating epoch #99 valid run on device rank=0 [2024-06-21 17:54:33,979] INFO: Rank 0: epoch=99 / 400 train_loss=10.3359 valid_loss=10.3902 stale=0 time=3.86m eta=1157.8m [2024-06-21 17:54:34,004] INFO: Initiating epoch #100 train run on device rank=0 [2024-06-21 17:57:36,488] INFO: Initiating epoch #100 valid run on device rank=0 [2024-06-21 17:58:26,124] INFO: Rank 0: epoch=100 / 400 train_loss=10.3351 valid_loss=10.3895 stale=0 time=3.87m eta=1154.1m [2024-06-21 17:58:26,159] INFO: Initiating epoch #101 train run on device rank=0 [2024-06-21 18:01:27,493] INFO: Initiating epoch #101 valid run on device rank=0 [2024-06-21 18:02:14,947] INFO: Rank 0: epoch=101 / 400 train_loss=10.3344 valid_loss=10.3888 stale=0 time=3.81m eta=1150.1m [2024-06-21 18:02:15,309] INFO: Initiating epoch #102 train run on device rank=0 [2024-06-21 18:05:18,964] INFO: Initiating epoch #102 valid run on device rank=0 [2024-06-21 18:06:06,932] INFO: Rank 0: epoch=102 / 400 train_loss=10.3336 valid_loss=10.3880 stale=0 time=3.86m eta=1146.3m [2024-06-21 18:06:07,075] INFO: Initiating epoch #103 train run on device rank=0 [2024-06-21 18:09:08,026] INFO: Initiating epoch #103 valid run on device rank=0 [2024-06-21 18:09:56,664] INFO: Rank 0: epoch=103 / 400 train_loss=10.3329 valid_loss=10.3873 stale=0 time=3.83m eta=1142.4m [2024-06-21 18:09:56,717] INFO: Initiating epoch #104 train run on device rank=0 [2024-06-21 18:13:00,271] INFO: Initiating epoch #104 valid run on device rank=0 [2024-06-21 18:13:48,354] INFO: Rank 0: epoch=104 / 400 train_loss=10.3322 valid_loss=10.3866 stale=0 time=3.86m eta=1138.6m [2024-06-21 18:13:48,401] INFO: Initiating epoch #105 train run on device rank=0 [2024-06-21 18:16:49,468] INFO: Initiating epoch #105 valid run on device rank=0 [2024-06-21 18:17:40,447] INFO: Rank 0: epoch=105 / 400 train_loss=10.3315 valid_loss=10.3859 stale=0 time=3.87m eta=1134.8m [2024-06-21 18:17:40,456] INFO: Initiating epoch #106 train run on device rank=0 [2024-06-21 18:20:42,224] INFO: Initiating epoch #106 valid run on device rank=0 [2024-06-21 18:21:30,536] INFO: Rank 0: epoch=106 / 400 train_loss=10.3307 valid_loss=10.3852 stale=0 time=3.83m eta=1131.0m [2024-06-21 18:21:30,580] INFO: Initiating epoch #107 train run on device rank=0 [2024-06-21 18:24:33,723] INFO: Initiating epoch #107 valid run on device rank=0 [2024-06-21 18:25:21,357] INFO: Rank 0: epoch=107 / 400 train_loss=10.3300 valid_loss=10.3846 stale=0 time=3.85m eta=1127.1m [2024-06-21 18:25:21,382] INFO: Initiating epoch #108 train run on device rank=0 [2024-06-21 18:28:22,871] INFO: Initiating epoch #108 valid run on device rank=0 [2024-06-21 18:29:10,477] INFO: Rank 0: epoch=108 / 400 train_loss=10.3294 valid_loss=10.3839 stale=0 time=3.82m eta=1123.2m [2024-06-21 18:29:10,508] INFO: Initiating epoch #109 train run on device rank=0 [2024-06-21 18:32:13,228] INFO: Initiating epoch #109 valid run on device rank=0 [2024-06-21 18:33:02,119] INFO: Rank 0: epoch=109 / 400 train_loss=10.3287 valid_loss=10.3832 stale=0 time=3.86m eta=1119.4m [2024-06-21 18:33:02,236] INFO: Initiating epoch #110 train run on device rank=0 [2024-06-21 18:36:07,273] INFO: Initiating epoch #110 valid run on device rank=0 [2024-06-21 18:36:56,687] INFO: Rank 0: epoch=110 / 400 train_loss=10.3280 valid_loss=10.3826 stale=0 time=3.91m eta=1115.7m [2024-06-21 18:36:56,748] INFO: Initiating epoch #111 train run on device rank=0 [2024-06-21 18:39:57,556] INFO: Initiating epoch #111 valid run on device rank=0 [2024-06-21 18:40:46,366] INFO: Rank 0: epoch=111 / 400 train_loss=10.3273 valid_loss=10.3819 stale=0 time=3.83m eta=1111.8m [2024-06-21 18:40:46,391] INFO: Initiating epoch #112 train run on device rank=0 [2024-06-21 18:43:48,137] INFO: Initiating epoch #112 valid run on device rank=0 [2024-06-21 18:44:36,280] INFO: Rank 0: epoch=112 / 400 train_loss=10.3267 valid_loss=10.3813 stale=0 time=3.83m eta=1107.9m [2024-06-21 18:44:36,321] INFO: Initiating epoch #113 train run on device rank=0 [2024-06-21 18:47:41,912] INFO: Initiating epoch #113 valid run on device rank=0 [2024-06-21 18:48:30,263] INFO: Rank 0: epoch=113 / 400 train_loss=10.3260 valid_loss=10.3806 stale=0 time=3.9m eta=1104.2m [2024-06-21 18:48:30,287] INFO: Initiating epoch #114 train run on device rank=0 [2024-06-21 18:51:31,297] INFO: Initiating epoch #114 valid run on device rank=0 [2024-06-21 18:52:21,007] INFO: Rank 0: epoch=114 / 400 train_loss=10.3254 valid_loss=10.3800 stale=0 time=3.85m eta=1100.3m [2024-06-21 18:52:21,107] INFO: Initiating epoch #115 train run on device rank=0 [2024-06-21 18:55:22,099] INFO: Initiating epoch #115 valid run on device rank=0 [2024-06-21 18:56:08,472] INFO: Rank 0: epoch=115 / 400 train_loss=10.3247 valid_loss=10.3793 stale=0 time=3.79m eta=1096.4m [2024-06-21 18:56:08,489] INFO: Initiating epoch #116 train run on device rank=0 [2024-06-21 18:59:10,375] INFO: Initiating epoch #116 valid run on device rank=0 [2024-06-21 18:59:58,075] INFO: Rank 0: epoch=116 / 400 train_loss=10.3241 valid_loss=10.3787 stale=0 time=3.83m eta=1092.5m [2024-06-21 18:59:58,206] INFO: Initiating epoch #117 train run on device rank=0 [2024-06-21 19:03:02,329] INFO: Initiating epoch #117 valid run on device rank=0 [2024-06-21 19:03:51,480] INFO: Rank 0: epoch=117 / 400 train_loss=10.3234 valid_loss=10.3781 stale=0 time=3.89m eta=1088.7m [2024-06-21 19:03:51,527] INFO: Initiating epoch #118 train run on device rank=0 [2024-06-21 19:06:51,138] INFO: Initiating epoch #118 valid run on device rank=0 [2024-06-21 19:07:42,591] INFO: Rank 0: epoch=118 / 400 train_loss=10.3228 valid_loss=10.3775 stale=0 time=3.85m eta=1084.9m [2024-06-21 19:07:42,613] INFO: Initiating epoch #119 train run on device rank=0 [2024-06-21 19:10:43,083] INFO: Initiating epoch #119 valid run on device rank=0 [2024-06-21 19:11:32,149] INFO: Rank 0: epoch=119 / 400 train_loss=10.3222 valid_loss=10.3768 stale=0 time=3.83m eta=1081.0m [2024-06-21 19:11:32,226] INFO: Initiating epoch #120 train run on device rank=0 [2024-06-21 19:14:35,053] INFO: Initiating epoch #120 valid run on device rank=0 [2024-06-21 19:15:25,716] INFO: Rank 0: epoch=120 / 400 train_loss=10.3216 valid_loss=10.3762 stale=0 time=3.89m eta=1077.3m [2024-06-21 19:15:25,751] INFO: Initiating epoch #121 train run on device rank=0 [2024-06-21 19:18:28,078] INFO: Initiating epoch #121 valid run on device rank=0 [2024-06-21 19:19:15,979] INFO: Rank 0: epoch=121 / 400 train_loss=10.3209 valid_loss=10.3756 stale=0 time=3.84m eta=1073.4m [2024-06-21 19:19:15,987] INFO: Initiating epoch #122 train run on device rank=0 [2024-06-21 19:22:18,363] INFO: Initiating epoch #122 valid run on device rank=0 [2024-06-21 19:23:07,670] INFO: Rank 0: epoch=122 / 400 train_loss=10.3203 valid_loss=10.3750 stale=0 time=3.86m eta=1069.6m [2024-06-21 19:23:07,749] INFO: Initiating epoch #123 train run on device rank=0 [2024-06-21 19:26:12,430] INFO: Initiating epoch #123 valid run on device rank=0 [2024-06-21 19:27:00,065] INFO: Rank 0: epoch=123 / 400 train_loss=10.3197 valid_loss=10.3744 stale=0 time=3.87m eta=1065.8m [2024-06-21 19:27:00,125] INFO: Initiating epoch #124 train run on device rank=0 [2024-06-21 19:30:02,560] INFO: Initiating epoch #124 valid run on device rank=0 [2024-06-21 19:30:50,783] INFO: Rank 0: epoch=124 / 400 train_loss=10.3191 valid_loss=10.3737 stale=0 time=3.84m eta=1061.9m [2024-06-21 19:30:50,859] INFO: Initiating epoch #125 train run on device rank=0 [2024-06-21 19:33:53,280] INFO: Initiating epoch #125 valid run on device rank=0 [2024-06-21 19:34:41,824] INFO: Rank 0: epoch=125 / 400 train_loss=10.3185 valid_loss=10.3731 stale=0 time=3.85m eta=1058.1m [2024-06-21 19:34:41,841] INFO: Initiating epoch #126 train run on device rank=0 [2024-06-21 19:37:46,272] INFO: Initiating epoch #126 valid run on device rank=0 [2024-06-21 19:38:38,225] INFO: Rank 0: epoch=126 / 400 train_loss=10.3179 valid_loss=10.3725 stale=0 time=3.94m eta=1054.4m [2024-06-21 19:38:39,385] INFO: Initiating epoch #127 train run on device rank=0 [2024-06-21 19:41:43,199] INFO: Initiating epoch #127 valid run on device rank=0 [2024-06-21 19:42:34,276] INFO: Rank 0: epoch=127 / 400 train_loss=10.3173 valid_loss=10.3719 stale=0 time=3.91m eta=1050.8m [2024-06-21 19:42:34,786] INFO: Initiating epoch #128 train run on device rank=0 [2024-06-21 19:45:36,782] INFO: Initiating epoch #128 valid run on device rank=0 [2024-06-21 19:46:28,042] INFO: Rank 0: epoch=128 / 400 train_loss=10.3167 valid_loss=10.3713 stale=0 time=3.89m eta=1047.0m [2024-06-21 19:46:28,774] INFO: Initiating epoch #129 train run on device rank=0 [2024-06-21 19:49:28,452] INFO: Initiating epoch #129 valid run on device rank=0 [2024-06-21 19:50:18,442] INFO: Rank 0: epoch=129 / 400 train_loss=10.3161 valid_loss=10.3707 stale=0 time=3.83m eta=1043.2m [2024-06-21 19:50:19,035] INFO: Initiating epoch #130 train run on device rank=0 [2024-06-21 19:53:22,483] INFO: Initiating epoch #130 valid run on device rank=0 [2024-06-21 19:54:09,763] INFO: Rank 0: epoch=130 / 400 train_loss=10.3155 valid_loss=10.3701 stale=0 time=3.85m eta=1039.3m [2024-06-21 19:54:09,785] INFO: Initiating epoch #131 train run on device rank=0 [2024-06-21 19:57:14,076] INFO: Initiating epoch #131 valid run on device rank=0 [2024-06-21 19:58:02,327] INFO: Rank 0: epoch=131 / 400 train_loss=10.3149 valid_loss=10.3695 stale=0 time=3.88m eta=1035.5m [2024-06-21 19:58:02,377] INFO: Initiating epoch #132 train run on device rank=0 [2024-06-21 20:01:04,994] INFO: Initiating epoch #132 valid run on device rank=0 [2024-06-21 20:01:52,500] INFO: Rank 0: epoch=132 / 400 train_loss=10.3143 valid_loss=10.3689 stale=0 time=3.84m eta=1031.6m [2024-06-21 20:01:52,543] INFO: Initiating epoch #133 train run on device rank=0 [2024-06-21 20:04:59,151] INFO: Initiating epoch #133 valid run on device rank=0 [2024-06-21 20:05:46,687] INFO: Rank 0: epoch=133 / 400 train_loss=10.3137 valid_loss=10.3683 stale=0 time=3.9m eta=1027.9m [2024-06-21 20:05:46,749] INFO: Initiating epoch #134 train run on device rank=0 [2024-06-21 20:08:51,808] INFO: Initiating epoch #134 valid run on device rank=0 [2024-06-21 20:09:40,076] INFO: Rank 0: epoch=134 / 400 train_loss=10.3131 valid_loss=10.3677 stale=0 time=3.89m eta=1024.1m [2024-06-21 20:09:40,132] INFO: Initiating epoch #135 train run on device rank=0 [2024-06-21 20:12:44,141] INFO: Initiating epoch #135 valid run on device rank=0 [2024-06-21 20:13:31,471] INFO: Rank 0: epoch=135 / 400 train_loss=10.3125 valid_loss=10.3671 stale=0 time=3.86m eta=1020.3m [2024-06-21 20:13:31,524] INFO: Initiating epoch #136 train run on device rank=0 [2024-06-21 20:16:37,809] INFO: Initiating epoch #136 valid run on device rank=0 [2024-06-21 20:17:29,417] INFO: Rank 0: epoch=136 / 400 train_loss=10.3119 valid_loss=10.3665 stale=0 time=3.96m eta=1016.7m [2024-06-21 20:17:29,960] INFO: Initiating epoch #137 train run on device rank=0 [2024-06-21 20:20:35,174] INFO: Initiating epoch #137 valid run on device rank=0 [2024-06-21 20:21:27,642] INFO: Rank 0: epoch=137 / 400 train_loss=10.3113 valid_loss=10.3659 stale=0 time=3.96m eta=1013.1m [2024-06-21 20:21:28,407] INFO: Initiating epoch #138 train run on device rank=0 [2024-06-21 20:24:30,031] INFO: Initiating epoch #138 valid run on device rank=0 [2024-06-21 20:25:17,811] INFO: Rank 0: epoch=138 / 400 train_loss=10.3108 valid_loss=10.3653 stale=0 time=3.82m eta=1009.2m [2024-06-21 20:25:17,846] INFO: Initiating epoch #139 train run on device rank=0 [2024-06-21 20:28:17,508] INFO: Initiating epoch #139 valid run on device rank=0 [2024-06-21 20:29:05,777] INFO: Rank 0: epoch=139 / 400 train_loss=10.3102 valid_loss=10.3648 stale=0 time=3.8m eta=1005.2m [2024-06-21 20:29:05,848] INFO: Initiating epoch #140 train run on device rank=0 [2024-06-21 20:32:09,734] INFO: Initiating epoch #140 valid run on device rank=0 [2024-06-21 20:32:58,250] INFO: Rank 0: epoch=140 / 400 train_loss=10.3096 valid_loss=10.3642 stale=0 time=3.87m eta=1001.4m [2024-06-21 20:32:58,297] INFO: Initiating epoch #141 train run on device rank=0 [2024-06-21 20:36:01,771] INFO: Initiating epoch #141 valid run on device rank=0 [2024-06-21 20:36:50,379] INFO: Rank 0: epoch=141 / 400 train_loss=10.3090 valid_loss=10.3636 stale=0 time=3.87m eta=997.6m [2024-06-21 20:36:50,677] INFO: Initiating epoch #142 train run on device rank=0 [2024-06-21 20:39:53,969] INFO: Initiating epoch #142 valid run on device rank=0 [2024-06-21 20:40:42,695] INFO: Rank 0: epoch=142 / 400 train_loss=10.3084 valid_loss=10.3630 stale=0 time=3.87m eta=993.8m [2024-06-21 20:40:42,756] INFO: Initiating epoch #143 train run on device rank=0 [2024-06-21 20:43:54,378] INFO: Initiating epoch #143 valid run on device rank=0 [2024-06-21 20:44:42,414] INFO: Rank 0: epoch=143 / 400 train_loss=10.3078 valid_loss=10.3624 stale=0 time=3.99m eta=990.2m [2024-06-21 20:44:42,563] INFO: Initiating epoch #144 train run on device rank=0 [2024-06-21 20:47:45,166] INFO: Initiating epoch #144 valid run on device rank=0 [2024-06-21 20:48:34,031] INFO: Rank 0: epoch=144 / 400 train_loss=10.3073 valid_loss=10.3618 stale=0 time=3.86m eta=986.3m [2024-06-21 20:48:34,093] INFO: Initiating epoch #145 train run on device rank=0 [2024-06-21 20:51:37,520] INFO: Initiating epoch #145 valid run on device rank=0 [2024-06-21 20:52:28,531] INFO: Rank 0: epoch=145 / 400 train_loss=10.3067 valid_loss=10.3612 stale=0 time=3.91m eta=982.6m [2024-06-21 20:52:28,642] INFO: Initiating epoch #146 train run on device rank=0 [2024-06-21 20:55:31,232] INFO: Initiating epoch #146 valid run on device rank=0 [2024-06-21 20:56:18,364] INFO: Rank 0: epoch=146 / 400 train_loss=10.3061 valid_loss=10.3607 stale=0 time=3.83m eta=978.7m [2024-06-21 20:56:18,386] INFO: Initiating epoch #147 train run on device rank=0 [2024-06-21 20:59:20,711] INFO: Initiating epoch #147 valid run on device rank=0 [2024-06-21 21:00:08,668] INFO: Rank 0: epoch=147 / 400 train_loss=10.3055 valid_loss=10.3601 stale=0 time=3.84m eta=974.8m [2024-06-21 21:00:08,723] INFO: Initiating epoch #148 train run on device rank=0 [2024-06-21 21:03:13,004] INFO: Initiating epoch #148 valid run on device rank=0 [2024-06-21 21:04:00,722] INFO: Rank 0: epoch=148 / 400 train_loss=10.3050 valid_loss=10.3595 stale=0 time=3.87m eta=971.0m [2024-06-21 21:04:00,769] INFO: Initiating epoch #149 train run on device rank=0 [2024-06-21 21:07:04,617] INFO: Initiating epoch #149 valid run on device rank=0 [2024-06-21 21:07:53,285] INFO: Rank 0: epoch=149 / 400 train_loss=10.3044 valid_loss=10.3589 stale=0 time=3.88m eta=967.2m [2024-06-21 21:07:53,293] INFO: Initiating epoch #150 train run on device rank=0 [2024-06-21 21:10:58,022] INFO: Initiating epoch #150 valid run on device rank=0 [2024-06-21 21:11:46,636] INFO: Rank 0: epoch=150 / 400 train_loss=10.3038 valid_loss=10.3583 stale=0 time=3.89m eta=963.4m [2024-06-21 21:11:46,669] INFO: Initiating epoch #151 train run on device rank=0 [2024-06-21 21:14:52,186] INFO: Initiating epoch #151 valid run on device rank=0 [2024-06-21 21:15:39,785] INFO: Rank 0: epoch=151 / 400 train_loss=10.3033 valid_loss=10.3578 stale=0 time=3.89m eta=959.6m [2024-06-21 21:15:39,809] INFO: Initiating epoch #152 train run on device rank=0 [2024-06-21 21:18:45,276] INFO: Initiating epoch #152 valid run on device rank=0 [2024-06-21 21:19:32,474] INFO: Rank 0: epoch=152 / 400 train_loss=10.3027 valid_loss=10.3572 stale=0 time=3.88m eta=955.8m [2024-06-21 21:19:32,588] INFO: Initiating epoch #153 train run on device rank=0 [2024-06-21 21:22:36,969] INFO: Initiating epoch #153 valid run on device rank=0 [2024-06-21 21:23:25,459] INFO: Rank 0: epoch=153 / 400 train_loss=10.3021 valid_loss=10.3566 stale=0 time=3.88m eta=952.0m [2024-06-21 21:23:25,527] INFO: Initiating epoch #154 train run on device rank=0 [2024-06-21 21:26:25,345] INFO: Initiating epoch #154 valid run on device rank=0 [2024-06-21 21:27:13,127] INFO: Rank 0: epoch=154 / 400 train_loss=10.3016 valid_loss=10.3561 stale=0 time=3.79m eta=948.0m [2024-06-21 21:27:13,206] INFO: Initiating epoch #155 train run on device rank=0 [2024-06-21 21:30:13,975] INFO: Initiating epoch #155 valid run on device rank=0 [2024-06-21 21:31:03,353] INFO: Rank 0: epoch=155 / 400 train_loss=10.3010 valid_loss=10.3555 stale=0 time=3.84m eta=944.1m [2024-06-21 21:31:03,391] INFO: Initiating epoch #156 train run on device rank=0 [2024-06-21 21:34:05,554] INFO: Initiating epoch #156 valid run on device rank=0 [2024-06-21 21:34:53,390] INFO: Rank 0: epoch=156 / 400 train_loss=10.3005 valid_loss=10.3549 stale=0 time=3.83m eta=940.2m [2024-06-21 21:34:53,403] INFO: Initiating epoch #157 train run on device rank=0 [2024-06-21 21:37:50,540] INFO: Initiating epoch #157 valid run on device rank=0 [2024-06-21 21:38:37,653] INFO: Rank 0: epoch=157 / 400 train_loss=10.2999 valid_loss=10.3544 stale=0 time=3.74m eta=936.2m [2024-06-21 21:38:37,699] INFO: Initiating epoch #158 train run on device rank=0 [2024-06-21 21:41:43,001] INFO: Initiating epoch #158 valid run on device rank=0 [2024-06-21 21:42:32,032] INFO: Rank 0: epoch=158 / 400 train_loss=10.2993 valid_loss=10.3538 stale=0 time=3.91m eta=932.4m [2024-06-21 21:42:32,035] INFO: Initiating epoch #159 train run on device rank=0 [2024-06-21 21:45:35,211] INFO: Initiating epoch #159 valid run on device rank=0 [2024-06-21 21:46:24,244] INFO: Rank 0: epoch=159 / 400 train_loss=10.2988 valid_loss=10.3532 stale=0 time=3.87m eta=928.6m [2024-06-21 21:46:24,268] INFO: Initiating epoch #160 train run on device rank=0 [2024-06-21 21:49:34,569] INFO: Initiating epoch #160 valid run on device rank=0 [2024-06-21 21:50:22,796] INFO: Rank 0: epoch=160 / 400 train_loss=10.2982 valid_loss=10.3527 stale=0 time=3.98m eta=924.9m [2024-06-21 21:50:22,925] INFO: Initiating epoch #161 train run on device rank=0 [2024-06-21 21:53:27,595] INFO: Initiating epoch #161 valid run on device rank=0 [2024-06-21 21:54:14,455] INFO: Rank 0: epoch=161 / 400 train_loss=10.2977 valid_loss=10.3521 stale=0 time=3.86m eta=921.1m [2024-06-21 21:54:14,484] INFO: Initiating epoch #162 train run on device rank=0 [2024-06-21 21:57:17,633] INFO: Initiating epoch #162 valid run on device rank=0 [2024-06-21 21:58:06,835] INFO: Rank 0: epoch=162 / 400 train_loss=10.2971 valid_loss=10.3516 stale=0 time=3.87m eta=917.3m [2024-06-21 21:58:06,860] INFO: Initiating epoch #163 train run on device rank=0 [2024-06-21 22:01:10,542] INFO: Initiating epoch #163 valid run on device rank=0 [2024-06-21 22:01:58,484] INFO: Rank 0: epoch=163 / 400 train_loss=10.2966 valid_loss=10.3510 stale=0 time=3.86m eta=913.4m [2024-06-21 22:01:58,498] INFO: Initiating epoch #164 train run on device rank=0 [2024-06-21 22:05:00,678] INFO: Initiating epoch #164 valid run on device rank=0 [2024-06-21 22:05:49,475] INFO: Rank 0: epoch=164 / 400 train_loss=10.2960 valid_loss=10.3504 stale=0 time=3.85m eta=909.6m [2024-06-21 22:05:49,572] INFO: Initiating epoch #165 train run on device rank=0 [2024-06-21 22:08:52,800] INFO: Initiating epoch #165 valid run on device rank=0 [2024-06-21 22:09:47,157] INFO: Rank 0: epoch=165 / 400 train_loss=10.2955 valid_loss=10.3499 stale=0 time=3.96m eta=905.9m [2024-06-21 22:09:47,924] INFO: Initiating epoch #166 train run on device rank=0 [2024-06-21 22:12:47,554] INFO: Initiating epoch #166 valid run on device rank=0 [2024-06-21 22:13:39,297] INFO: Rank 0: epoch=166 / 400 train_loss=10.2949 valid_loss=10.3493 stale=0 time=3.86m eta=902.0m [2024-06-21 22:13:39,861] INFO: Initiating epoch #167 train run on device rank=0 [2024-06-21 22:16:42,170] INFO: Initiating epoch #167 valid run on device rank=0 [2024-06-21 22:17:32,720] INFO: Rank 0: epoch=167 / 400 train_loss=10.2944 valid_loss=10.3488 stale=0 time=3.88m eta=898.2m [2024-06-21 22:17:33,253] INFO: Initiating epoch #168 train run on device rank=0 [2024-06-21 22:20:36,087] INFO: Initiating epoch #168 valid run on device rank=0 [2024-06-21 22:21:26,931] INFO: Rank 0: epoch=168 / 400 train_loss=10.2938 valid_loss=10.3482 stale=0 time=3.89m eta=894.4m [2024-06-21 22:21:27,521] INFO: Initiating epoch #169 train run on device rank=0 [2024-06-21 22:24:26,699] INFO: Initiating epoch #169 valid run on device rank=0 [2024-06-21 22:25:17,916] INFO: Rank 0: epoch=169 / 400 train_loss=10.2933 valid_loss=10.3477 stale=0 time=3.84m eta=890.6m [2024-06-21 22:25:18,919] INFO: Initiating epoch #170 train run on device rank=0 [2024-06-21 22:28:19,538] INFO: Initiating epoch #170 valid run on device rank=0 [2024-06-21 22:29:11,407] INFO: Rank 0: epoch=170 / 400 train_loss=10.2928 valid_loss=10.3471 stale=0 time=3.87m eta=886.8m [2024-06-21 22:29:11,918] INFO: Initiating epoch #171 train run on device rank=0 [2024-06-21 22:32:10,631] INFO: Initiating epoch #171 valid run on device rank=0 [2024-06-21 22:33:01,598] INFO: Rank 0: epoch=171 / 400 train_loss=10.2922 valid_loss=10.3466 stale=0 time=3.83m eta=882.9m [2024-06-21 22:33:02,216] INFO: Initiating epoch #172 train run on device rank=0 [2024-06-21 22:36:02,822] INFO: Initiating epoch #172 valid run on device rank=0 [2024-06-21 22:36:53,615] INFO: Rank 0: epoch=172 / 400 train_loss=10.2917 valid_loss=10.3460 stale=0 time=3.86m eta=879.1m [2024-06-21 22:36:54,272] INFO: Initiating epoch #173 train run on device rank=0 [2024-06-21 22:39:57,365] INFO: Initiating epoch #173 valid run on device rank=0 [2024-06-21 22:40:45,112] INFO: Rank 0: epoch=173 / 400 train_loss=10.2911 valid_loss=10.3455 stale=0 time=3.85m eta=875.2m [2024-06-21 22:40:45,190] INFO: Initiating epoch #174 train run on device rank=0 [2024-06-21 22:43:46,813] INFO: Initiating epoch #174 valid run on device rank=0 [2024-06-21 22:44:34,897] INFO: Rank 0: epoch=174 / 400 train_loss=10.2906 valid_loss=10.3450 stale=0 time=3.83m eta=871.3m [2024-06-21 22:44:34,944] INFO: Initiating epoch #175 train run on device rank=0 [2024-06-21 22:47:38,410] INFO: Initiating epoch #175 valid run on device rank=0 [2024-06-21 22:48:25,511] INFO: Rank 0: epoch=175 / 400 train_loss=10.2901 valid_loss=10.3444 stale=0 time=3.84m eta=867.4m [2024-06-21 22:48:25,555] INFO: Initiating epoch #176 train run on device rank=0 [2024-06-21 22:51:31,059] INFO: Initiating epoch #176 valid run on device rank=0 [2024-06-21 22:52:23,351] INFO: Rank 0: epoch=176 / 400 train_loss=10.2895 valid_loss=10.3439 stale=0 time=3.96m eta=863.7m [2024-06-21 22:52:24,121] INFO: Initiating epoch #177 train run on device rank=0 [2024-06-21 22:55:27,088] INFO: Initiating epoch #177 valid run on device rank=0 [2024-06-21 22:56:18,563] INFO: Rank 0: epoch=177 / 400 train_loss=10.2890 valid_loss=10.3433 stale=0 time=3.91m eta=859.9m [2024-06-21 22:56:19,208] INFO: Initiating epoch #178 train run on device rank=0 [2024-06-21 22:59:20,787] INFO: Initiating epoch #178 valid run on device rank=0 [2024-06-21 23:00:13,127] INFO: Rank 0: epoch=178 / 400 train_loss=10.2885 valid_loss=10.3428 stale=0 time=3.9m eta=856.2m [2024-06-21 23:00:13,867] INFO: Initiating epoch #179 train run on device rank=0 [2024-06-21 23:03:12,676] INFO: Initiating epoch #179 valid run on device rank=0 [2024-06-21 23:04:03,566] INFO: Rank 0: epoch=179 / 400 train_loss=10.2879 valid_loss=10.3423 stale=0 time=3.83m eta=852.3m [2024-06-21 23:04:04,146] INFO: Initiating epoch #180 train run on device rank=0 [2024-06-21 23:07:08,727] INFO: Initiating epoch #180 valid run on device rank=0 [2024-06-21 23:07:59,596] INFO: Rank 0: epoch=180 / 400 train_loss=10.2874 valid_loss=10.3418 stale=0 time=3.92m eta=848.5m [2024-06-21 23:08:00,204] INFO: Initiating epoch #181 train run on device rank=0 [2024-06-21 23:11:02,261] INFO: Initiating epoch #181 valid run on device rank=0 [2024-06-21 23:11:53,700] INFO: Rank 0: epoch=181 / 400 train_loss=10.2869 valid_loss=10.3412 stale=0 time=3.89m eta=844.7m [2024-06-21 23:11:54,578] INFO: Initiating epoch #182 train run on device rank=0 [2024-06-21 23:14:55,074] INFO: Initiating epoch #182 valid run on device rank=0 [2024-06-21 23:15:43,681] INFO: Rank 0: epoch=182 / 400 train_loss=10.2864 valid_loss=10.3407 stale=0 time=3.82m eta=840.8m [2024-06-21 23:15:43,731] INFO: Initiating epoch #183 train run on device rank=0 [2024-06-21 23:18:45,972] INFO: Initiating epoch #183 valid run on device rank=0 [2024-06-21 23:19:34,075] INFO: Rank 0: epoch=183 / 400 train_loss=10.2859 valid_loss=10.3402 stale=0 time=3.84m eta=837.0m [2024-06-21 23:19:34,109] INFO: Initiating epoch #184 train run on device rank=0 [2024-06-21 23:22:37,652] INFO: Initiating epoch #184 valid run on device rank=0 [2024-06-21 23:23:26,929] INFO: Rank 0: epoch=184 / 400 train_loss=10.2853 valid_loss=10.3397 stale=0 time=3.88m eta=833.1m [2024-06-21 23:23:26,983] INFO: Initiating epoch #185 train run on device rank=0 [2024-06-21 23:26:27,463] INFO: Initiating epoch #185 valid run on device rank=0 [2024-06-21 23:27:16,285] INFO: Rank 0: epoch=185 / 400 train_loss=10.2848 valid_loss=10.3391 stale=0 time=3.82m eta=829.2m [2024-06-21 23:27:16,338] INFO: Initiating epoch #186 train run on device rank=0 [2024-06-21 23:30:20,074] INFO: Initiating epoch #186 valid run on device rank=0 [2024-06-21 23:31:07,923] INFO: Rank 0: epoch=186 / 400 train_loss=10.2843 valid_loss=10.3386 stale=0 time=3.86m eta=825.4m [2024-06-21 23:31:08,200] INFO: Initiating epoch #187 train run on device rank=0 [2024-06-21 23:34:11,147] INFO: Initiating epoch #187 valid run on device rank=0 [2024-06-21 23:34:58,490] INFO: Rank 0: epoch=187 / 400 train_loss=10.2838 valid_loss=10.3381 stale=0 time=3.84m eta=821.5m [2024-06-21 23:34:58,514] INFO: Initiating epoch #188 train run on device rank=0 [2024-06-21 23:38:03,406] INFO: Initiating epoch #188 valid run on device rank=0 [2024-06-21 23:38:52,219] INFO: Rank 0: epoch=188 / 400 train_loss=10.2833 valid_loss=10.3376 stale=0 time=3.9m eta=817.7m [2024-06-21 23:38:52,227] INFO: Initiating epoch #189 train run on device rank=0 [2024-06-21 23:41:56,579] INFO: Initiating epoch #189 valid run on device rank=0 [2024-06-21 23:42:46,276] INFO: Rank 0: epoch=189 / 400 train_loss=10.2828 valid_loss=10.3371 stale=0 time=3.9m eta=813.9m [2024-06-21 23:42:46,312] INFO: Initiating epoch #190 train run on device rank=0 [2024-06-21 23:45:48,164] INFO: Initiating epoch #190 valid run on device rank=0 [2024-06-21 23:46:35,195] INFO: Rank 0: epoch=190 / 400 train_loss=10.2823 valid_loss=10.3366 stale=0 time=3.81m eta=810.0m [2024-06-21 23:46:35,224] INFO: Initiating epoch #191 train run on device rank=0 [2024-06-21 23:49:39,436] INFO: Initiating epoch #191 valid run on device rank=0 [2024-06-21 23:50:28,234] INFO: Rank 0: epoch=191 / 400 train_loss=10.2818 valid_loss=10.3361 stale=0 time=3.88m eta=806.2m [2024-06-21 23:50:28,334] INFO: Initiating epoch #192 train run on device rank=0 [2024-06-21 23:53:29,081] INFO: Initiating epoch #192 valid run on device rank=0 [2024-06-21 23:54:17,882] INFO: Rank 0: epoch=192 / 400 train_loss=10.2813 valid_loss=10.3356 stale=0 time=3.83m eta=802.3m [2024-06-21 23:54:17,916] INFO: Initiating epoch #193 train run on device rank=0 [2024-06-21 23:57:22,151] INFO: Initiating epoch #193 valid run on device rank=0 [2024-06-21 23:58:10,645] INFO: Rank 0: epoch=193 / 400 train_loss=10.2808 valid_loss=10.3351 stale=0 time=3.88m eta=798.4m [2024-06-21 23:58:10,703] INFO: Initiating epoch #194 train run on device rank=0 [2024-06-22 00:01:11,385] INFO: Initiating epoch #194 valid run on device rank=0 [2024-06-22 00:01:59,704] INFO: Rank 0: epoch=194 / 400 train_loss=10.2803 valid_loss=10.3346 stale=0 time=3.82m eta=794.5m [2024-06-22 00:01:59,754] INFO: Initiating epoch #195 train run on device rank=0 [2024-06-22 00:05:01,548] INFO: Initiating epoch #195 valid run on device rank=0 [2024-06-22 00:05:51,000] INFO: Rank 0: epoch=195 / 400 train_loss=10.2798 valid_loss=10.3340 stale=0 time=3.85m eta=790.7m [2024-06-22 00:05:51,016] INFO: Initiating epoch #196 train run on device rank=0 [2024-06-22 00:08:55,463] INFO: Initiating epoch #196 valid run on device rank=0 [2024-06-22 00:09:43,326] INFO: Rank 0: epoch=196 / 400 train_loss=10.2793 valid_loss=10.3336 stale=0 time=3.87m eta=786.8m [2024-06-22 00:09:43,376] INFO: Initiating epoch #197 train run on device rank=0 [2024-06-22 00:12:46,497] INFO: Initiating epoch #197 valid run on device rank=0 [2024-06-22 00:13:35,728] INFO: Rank 0: epoch=197 / 400 train_loss=10.2788 valid_loss=10.3330 stale=0 time=3.87m eta=783.0m [2024-06-22 00:13:36,007] INFO: Initiating epoch #198 train run on device rank=0 [2024-06-22 00:16:35,948] INFO: Initiating epoch #198 valid run on device rank=0 [2024-06-22 00:17:25,243] INFO: Rank 0: epoch=198 / 400 train_loss=10.2783 valid_loss=10.3326 stale=0 time=3.82m eta=779.1m [2024-06-22 00:17:25,272] INFO: Initiating epoch #199 train run on device rank=0 [2024-06-22 00:20:32,063] INFO: Initiating epoch #199 valid run on device rank=0 [2024-06-22 00:21:20,122] INFO: Rank 0: epoch=199 / 400 train_loss=10.2778 valid_loss=10.3321 stale=0 time=3.91m eta=775.3m [2024-06-22 00:21:20,167] INFO: Initiating epoch #200 train run on device rank=0 [2024-06-22 00:24:23,885] INFO: Initiating epoch #200 valid run on device rank=0 [2024-06-22 00:25:11,532] INFO: Rank 0: epoch=200 / 400 train_loss=10.2773 valid_loss=10.3316 stale=0 time=3.86m eta=771.4m [2024-06-22 00:25:11,544] INFO: Initiating epoch #201 train run on device rank=0 [2024-06-22 00:28:12,687] INFO: Initiating epoch #201 valid run on device rank=0 [2024-06-22 00:29:02,128] INFO: Rank 0: epoch=201 / 400 train_loss=10.2768 valid_loss=10.3311 stale=0 time=3.84m eta=767.6m [2024-06-22 00:29:02,161] INFO: Initiating epoch #202 train run on device rank=0 [2024-06-22 00:32:04,588] INFO: Initiating epoch #202 valid run on device rank=0 [2024-06-22 00:32:53,708] INFO: Rank 0: epoch=202 / 400 train_loss=10.2763 valid_loss=10.3306 stale=0 time=3.86m eta=763.7m [2024-06-22 00:32:53,736] INFO: Initiating epoch #203 train run on device rank=0 [2024-06-22 00:35:55,904] INFO: Initiating epoch #203 valid run on device rank=0 [2024-06-22 00:36:43,511] INFO: Rank 0: epoch=203 / 400 train_loss=10.2758 valid_loss=10.3301 stale=0 time=3.83m eta=759.8m [2024-06-22 00:36:43,560] INFO: Initiating epoch #204 train run on device rank=0 [2024-06-22 00:39:53,203] INFO: Initiating epoch #204 valid run on device rank=0 [2024-06-22 00:40:41,463] INFO: Rank 0: epoch=204 / 400 train_loss=10.2753 valid_loss=10.3296 stale=0 time=3.97m eta=756.1m [2024-06-22 00:40:41,483] INFO: Initiating epoch #205 train run on device rank=0 [2024-06-22 00:43:41,566] INFO: Initiating epoch #205 valid run on device rank=0 [2024-06-22 00:44:29,831] INFO: Rank 0: epoch=205 / 400 train_loss=10.2749 valid_loss=10.3291 stale=0 time=3.81m eta=752.2m [2024-06-22 00:44:29,867] INFO: Initiating epoch #206 train run on device rank=0 [2024-06-22 00:47:34,176] INFO: Initiating epoch #206 valid run on device rank=0 [2024-06-22 00:48:23,637] INFO: Rank 0: epoch=206 / 400 train_loss=10.2744 valid_loss=10.3286 stale=0 time=3.9m eta=748.4m [2024-06-22 00:48:23,727] INFO: Initiating epoch #207 train run on device rank=0 [2024-06-22 00:51:24,781] INFO: Initiating epoch #207 valid run on device rank=0 [2024-06-22 00:52:13,084] INFO: Rank 0: epoch=207 / 400 train_loss=10.2739 valid_loss=10.3282 stale=0 time=3.82m eta=744.5m [2024-06-22 00:52:13,120] INFO: Initiating epoch #208 train run on device rank=0 [2024-06-22 00:55:14,708] INFO: Initiating epoch #208 valid run on device rank=0 [2024-06-22 00:56:02,397] INFO: Rank 0: epoch=208 / 400 train_loss=10.2734 valid_loss=10.3277 stale=0 time=3.82m eta=740.6m [2024-06-22 00:56:02,504] INFO: Initiating epoch #209 train run on device rank=0 [2024-06-22 00:59:06,389] INFO: Initiating epoch #209 valid run on device rank=0 [2024-06-22 00:59:53,811] INFO: Rank 0: epoch=209 / 400 train_loss=10.2729 valid_loss=10.3272 stale=0 time=3.86m eta=736.7m [2024-06-22 00:59:53,957] INFO: Initiating epoch #210 train run on device rank=0 [2024-06-22 01:02:56,776] INFO: Initiating epoch #210 valid run on device rank=0 [2024-06-22 01:03:44,391] INFO: Rank 0: epoch=210 / 400 train_loss=10.2725 valid_loss=10.3267 stale=0 time=3.84m eta=732.8m [2024-06-22 01:03:44,405] INFO: Initiating epoch #211 train run on device rank=0 [2024-06-22 01:06:45,686] INFO: Initiating epoch #211 valid run on device rank=0 [2024-06-22 01:07:36,346] INFO: Rank 0: epoch=211 / 400 train_loss=10.2720 valid_loss=10.3262 stale=0 time=3.87m eta=729.0m [2024-06-22 01:07:36,378] INFO: Initiating epoch #212 train run on device rank=0 [2024-06-22 01:10:40,343] INFO: Initiating epoch #212 valid run on device rank=0 [2024-06-22 01:11:28,884] INFO: Rank 0: epoch=212 / 400 train_loss=10.2715 valid_loss=10.3258 stale=0 time=3.88m eta=725.2m [2024-06-22 01:11:28,965] INFO: Initiating epoch #213 train run on device rank=0 [2024-06-22 01:14:31,751] INFO: Initiating epoch #213 valid run on device rank=0 [2024-06-22 01:15:20,024] INFO: Rank 0: epoch=213 / 400 train_loss=10.2710 valid_loss=10.3253 stale=0 time=3.85m eta=721.3m [2024-06-22 01:15:20,157] INFO: Initiating epoch #214 train run on device rank=0 [2024-06-22 01:18:22,571] INFO: Initiating epoch #214 valid run on device rank=0 [2024-06-22 01:19:10,321] INFO: Rank 0: epoch=214 / 400 train_loss=10.2706 valid_loss=10.3248 stale=0 time=3.84m eta=717.4m [2024-06-22 01:19:10,339] INFO: Initiating epoch #215 train run on device rank=0 [2024-06-22 01:22:16,470] INFO: Initiating epoch #215 valid run on device rank=0 [2024-06-22 01:23:08,092] INFO: Rank 0: epoch=215 / 400 train_loss=10.2701 valid_loss=10.3243 stale=0 time=3.96m eta=713.7m [2024-06-22 01:23:08,450] INFO: Initiating epoch #216 train run on device rank=0 [2024-06-22 01:26:08,893] INFO: Initiating epoch #216 valid run on device rank=0 [2024-06-22 01:26:56,981] INFO: Rank 0: epoch=216 / 400 train_loss=10.2696 valid_loss=10.3239 stale=0 time=3.81m eta=709.8m [2024-06-22 01:26:57,006] INFO: Initiating epoch #217 train run on device rank=0 [2024-06-22 01:29:58,890] INFO: Initiating epoch #217 valid run on device rank=0 [2024-06-22 01:30:47,935] INFO: Rank 0: epoch=217 / 400 train_loss=10.2692 valid_loss=10.3234 stale=0 time=3.85m eta=705.9m [2024-06-22 01:30:47,960] INFO: Initiating epoch #218 train run on device rank=0 [2024-06-22 01:33:54,320] INFO: Initiating epoch #218 valid run on device rank=0 [2024-06-22 01:34:42,637] INFO: Rank 0: epoch=218 / 400 train_loss=10.2687 valid_loss=10.3229 stale=0 time=3.91m eta=702.1m [2024-06-22 01:34:42,666] INFO: Initiating epoch #219 train run on device rank=0 [2024-06-22 01:37:46,014] INFO: Initiating epoch #219 valid run on device rank=0 [2024-06-22 01:38:33,537] INFO: Rank 0: epoch=219 / 400 train_loss=10.2682 valid_loss=10.3225 stale=0 time=3.85m eta=698.2m [2024-06-22 01:38:33,545] INFO: Initiating epoch #220 train run on device rank=0 [2024-06-22 01:41:35,302] INFO: Initiating epoch #220 valid run on device rank=0 [2024-06-22 01:42:22,805] INFO: Rank 0: epoch=220 / 400 train_loss=10.2678 valid_loss=10.3220 stale=0 time=3.82m eta=694.3m [2024-06-22 01:42:22,819] INFO: Initiating epoch #221 train run on device rank=0 [2024-06-22 01:45:23,640] INFO: Initiating epoch #221 valid run on device rank=0 [2024-06-22 01:46:12,555] INFO: Rank 0: epoch=221 / 400 train_loss=10.2673 valid_loss=10.3215 stale=0 time=3.83m eta=690.5m [2024-06-22 01:46:12,616] INFO: Initiating epoch #222 train run on device rank=0 [2024-06-22 01:49:15,079] INFO: Initiating epoch #222 valid run on device rank=0 [2024-06-22 01:50:03,382] INFO: Rank 0: epoch=222 / 400 train_loss=10.2668 valid_loss=10.3211 stale=0 time=3.85m eta=686.6m [2024-06-22 01:50:03,410] INFO: Initiating epoch #223 train run on device rank=0 [2024-06-22 01:53:02,998] INFO: Initiating epoch #223 valid run on device rank=0 [2024-06-22 01:53:49,909] INFO: Rank 0: epoch=223 / 400 train_loss=10.2664 valid_loss=10.3206 stale=0 time=3.77m eta=682.7m [2024-06-22 01:53:49,981] INFO: Initiating epoch #224 train run on device rank=0 [2024-06-22 01:56:52,579] INFO: Initiating epoch #224 valid run on device rank=0 [2024-06-22 01:57:40,554] INFO: Rank 0: epoch=224 / 400 train_loss=10.2659 valid_loss=10.3202 stale=0 time=3.84m eta=678.8m [2024-06-22 01:57:40,633] INFO: Initiating epoch #225 train run on device rank=0 [2024-06-22 02:00:44,005] INFO: Initiating epoch #225 valid run on device rank=0 [2024-06-22 02:01:32,438] INFO: Rank 0: epoch=225 / 400 train_loss=10.2655 valid_loss=10.3197 stale=0 time=3.86m eta=674.9m [2024-06-22 02:01:32,495] INFO: Initiating epoch #226 train run on device rank=0 [2024-06-22 02:04:42,889] INFO: Initiating epoch #226 valid run on device rank=0 [2024-06-22 02:05:30,694] INFO: Rank 0: epoch=226 / 400 train_loss=10.2651 valid_loss=10.3193 stale=0 time=3.97m eta=671.2m [2024-06-22 02:05:30,835] INFO: Initiating epoch #227 train run on device rank=0 [2024-06-22 02:08:42,049] INFO: Initiating epoch #227 valid run on device rank=0 [2024-06-22 02:09:30,695] INFO: Rank 0: epoch=227 / 400 train_loss=10.2646 valid_loss=10.3188 stale=0 time=4.0m eta=667.4m [2024-06-22 02:09:30,865] INFO: Initiating epoch #228 train run on device rank=0 [2024-06-22 02:12:31,417] INFO: Initiating epoch #228 valid run on device rank=0 [2024-06-22 02:13:18,936] INFO: Rank 0: epoch=228 / 400 train_loss=10.2642 valid_loss=10.3184 stale=0 time=3.8m eta=663.5m [2024-06-22 02:13:19,068] INFO: Initiating epoch #229 train run on device rank=0 [2024-06-22 02:16:21,655] INFO: Initiating epoch #229 valid run on device rank=0 [2024-06-22 02:17:09,677] INFO: Rank 0: epoch=229 / 400 train_loss=10.2637 valid_loss=10.3179 stale=0 time=3.84m eta=659.7m [2024-06-22 02:17:09,702] INFO: Initiating epoch #230 train run on device rank=0 [2024-06-22 02:20:10,731] INFO: Initiating epoch #230 valid run on device rank=0 [2024-06-22 02:20:59,119] INFO: Rank 0: epoch=230 / 400 train_loss=10.2633 valid_loss=10.3175 stale=0 time=3.82m eta=655.8m [2024-06-22 02:20:59,172] INFO: Initiating epoch #231 train run on device rank=0 [2024-06-22 02:24:05,577] INFO: Initiating epoch #231 valid run on device rank=0 [2024-06-22 02:24:52,743] INFO: Rank 0: epoch=231 / 400 train_loss=10.2628 valid_loss=10.3170 stale=0 time=3.89m eta=652.0m [2024-06-22 02:24:52,752] INFO: Initiating epoch #232 train run on device rank=0 [2024-06-22 02:27:56,559] INFO: Initiating epoch #232 valid run on device rank=0 [2024-06-22 02:28:44,363] INFO: Rank 0: epoch=232 / 400 train_loss=10.2624 valid_loss=10.3166 stale=0 time=3.86m eta=648.1m [2024-06-22 02:28:44,387] INFO: Initiating epoch #233 train run on device rank=0 [2024-06-22 02:31:44,494] INFO: Initiating epoch #233 valid run on device rank=0 [2024-06-22 02:32:32,804] INFO: Rank 0: epoch=233 / 400 train_loss=10.2620 valid_loss=10.3161 stale=0 time=3.81m eta=644.2m [2024-06-22 02:32:32,829] INFO: Initiating epoch #234 train run on device rank=0 [2024-06-22 02:35:35,979] INFO: Initiating epoch #234 valid run on device rank=0 [2024-06-22 02:36:23,403] INFO: Rank 0: epoch=234 / 400 train_loss=10.2615 valid_loss=10.3157 stale=0 time=3.84m eta=640.3m [2024-06-22 02:36:23,466] INFO: Initiating epoch #235 train run on device rank=0 [2024-06-22 02:39:28,160] INFO: Initiating epoch #235 valid run on device rank=0 [2024-06-22 02:40:16,787] INFO: Rank 0: epoch=235 / 400 train_loss=10.2611 valid_loss=10.3153 stale=0 time=3.89m eta=636.5m [2024-06-22 02:40:16,841] INFO: Initiating epoch #236 train run on device rank=0 [2024-06-22 02:43:18,329] INFO: Initiating epoch #236 valid run on device rank=0 [2024-06-22 02:44:08,191] INFO: Rank 0: epoch=236 / 400 train_loss=10.2607 valid_loss=10.3148 stale=0 time=3.86m eta=632.6m [2024-06-22 02:44:08,238] INFO: Initiating epoch #237 train run on device rank=0 [2024-06-22 02:47:10,164] INFO: Initiating epoch #237 valid run on device rank=0 [2024-06-22 02:47:57,936] INFO: Rank 0: epoch=237 / 400 train_loss=10.2603 valid_loss=10.3144 stale=0 time=3.83m eta=628.8m [2024-06-22 02:47:57,996] INFO: Initiating epoch #238 train run on device rank=0 [2024-06-22 02:51:00,603] INFO: Initiating epoch #238 valid run on device rank=0 [2024-06-22 02:51:49,983] INFO: Rank 0: epoch=238 / 400 train_loss=10.2598 valid_loss=10.3140 stale=0 time=3.87m eta=624.9m [2024-06-22 02:51:50,052] INFO: Initiating epoch #239 train run on device rank=0 [2024-06-22 02:54:51,962] INFO: Initiating epoch #239 valid run on device rank=0 [2024-06-22 02:55:39,370] INFO: Rank 0: epoch=239 / 400 train_loss=10.2594 valid_loss=10.3135 stale=0 time=3.82m eta=621.0m [2024-06-22 02:55:39,422] INFO: Initiating epoch #240 train run on device rank=0 [2024-06-22 02:58:40,101] INFO: Initiating epoch #240 valid run on device rank=0 [2024-06-22 02:59:30,546] INFO: Rank 0: epoch=240 / 400 train_loss=10.2590 valid_loss=10.3131 stale=0 time=3.85m eta=617.2m [2024-06-22 02:59:30,595] INFO: Initiating epoch #241 train run on device rank=0 [2024-06-22 03:02:33,233] INFO: Initiating epoch #241 valid run on device rank=0 [2024-06-22 03:03:20,724] INFO: Rank 0: epoch=241 / 400 train_loss=10.2586 valid_loss=10.3127 stale=0 time=3.84m eta=613.3m [2024-06-22 03:03:20,736] INFO: Initiating epoch #242 train run on device rank=0 [2024-06-22 03:06:19,991] INFO: Initiating epoch #242 valid run on device rank=0 [2024-06-22 03:07:09,471] INFO: Rank 0: epoch=242 / 400 train_loss=10.2581 valid_loss=10.3123 stale=0 time=3.81m eta=609.4m [2024-06-22 03:07:09,514] INFO: Initiating epoch #243 train run on device rank=0 [2024-06-22 03:10:11,637] INFO: Initiating epoch #243 valid run on device rank=0 [2024-06-22 03:10:59,339] INFO: Rank 0: epoch=243 / 400 train_loss=10.2577 valid_loss=10.3118 stale=0 time=3.83m eta=605.5m [2024-06-22 03:10:59,400] INFO: Initiating epoch #244 train run on device rank=0 [2024-06-22 03:14:03,356] INFO: Initiating epoch #244 valid run on device rank=0 [2024-06-22 03:14:51,566] INFO: Rank 0: epoch=244 / 400 train_loss=10.2573 valid_loss=10.3114 stale=0 time=3.87m eta=601.7m [2024-06-22 03:14:51,670] INFO: Initiating epoch #245 train run on device rank=0 [2024-06-22 03:17:53,362] INFO: Initiating epoch #245 valid run on device rank=0 [2024-06-22 03:18:40,688] INFO: Rank 0: epoch=245 / 400 train_loss=10.2569 valid_loss=10.3110 stale=0 time=3.82m eta=597.8m [2024-06-22 03:18:40,730] INFO: Initiating epoch #246 train run on device rank=0 [2024-06-22 03:21:41,566] INFO: Initiating epoch #246 valid run on device rank=0 [2024-06-22 03:22:29,287] INFO: Rank 0: epoch=246 / 400 train_loss=10.2565 valid_loss=10.3106 stale=0 time=3.81m eta=593.9m [2024-06-22 03:22:29,318] INFO: Initiating epoch #247 train run on device rank=0 [2024-06-22 03:25:30,278] INFO: Initiating epoch #247 valid run on device rank=0 [2024-06-22 03:26:18,099] INFO: Rank 0: epoch=247 / 400 train_loss=10.2561 valid_loss=10.3102 stale=0 time=3.81m eta=590.0m [2024-06-22 03:26:18,114] INFO: Initiating epoch #248 train run on device rank=0 [2024-06-22 03:29:20,027] INFO: Initiating epoch #248 valid run on device rank=0 [2024-06-22 03:30:08,289] INFO: Rank 0: epoch=248 / 400 train_loss=10.2557 valid_loss=10.3098 stale=0 time=3.84m eta=586.2m [2024-06-22 03:30:08,302] INFO: Initiating epoch #249 train run on device rank=0 [2024-06-22 03:33:13,014] INFO: Initiating epoch #249 valid run on device rank=0 [2024-06-22 03:34:02,125] INFO: Rank 0: epoch=249 / 400 train_loss=10.2553 valid_loss=10.3094 stale=0 time=3.9m eta=582.3m [2024-06-22 03:34:02,148] INFO: Initiating epoch #250 train run on device rank=0 [2024-06-22 03:37:05,193] INFO: Initiating epoch #250 valid run on device rank=0 [2024-06-22 03:37:52,511] INFO: Rank 0: epoch=250 / 400 train_loss=10.2549 valid_loss=10.3090 stale=0 time=3.84m eta=578.5m [2024-06-22 03:37:52,553] INFO: Initiating epoch #251 train run on device rank=0 [2024-06-22 03:40:56,506] INFO: Initiating epoch #251 valid run on device rank=0 [2024-06-22 03:41:45,572] INFO: Rank 0: epoch=251 / 400 train_loss=10.2545 valid_loss=10.3086 stale=0 time=3.88m eta=574.6m [2024-06-22 03:41:45,659] INFO: Initiating epoch #252 train run on device rank=0 [2024-06-22 03:44:48,957] INFO: Initiating epoch #252 valid run on device rank=0 [2024-06-22 03:45:38,696] INFO: Rank 0: epoch=252 / 400 train_loss=10.2541 valid_loss=10.3082 stale=0 time=3.88m eta=570.8m [2024-06-22 03:45:38,985] INFO: Initiating epoch #253 train run on device rank=0 [2024-06-22 03:48:39,137] INFO: Initiating epoch #253 valid run on device rank=0 [2024-06-22 03:49:27,940] INFO: Rank 0: epoch=253 / 400 train_loss=10.2537 valid_loss=10.3078 stale=0 time=3.82m eta=566.9m [2024-06-22 03:49:27,998] INFO: Initiating epoch #254 train run on device rank=0 [2024-06-22 03:52:28,083] INFO: Initiating epoch #254 valid run on device rank=0 [2024-06-22 03:53:16,511] INFO: Rank 0: epoch=254 / 400 train_loss=10.2533 valid_loss=10.3074 stale=0 time=3.81m eta=563.0m [2024-06-22 03:53:16,598] INFO: Initiating epoch #255 train run on device rank=0 [2024-06-22 03:56:17,419] INFO: Initiating epoch #255 valid run on device rank=0 [2024-06-22 03:57:04,563] INFO: Rank 0: epoch=255 / 400 train_loss=10.2529 valid_loss=10.3070 stale=0 time=3.8m eta=559.1m [2024-06-22 03:57:04,696] INFO: Initiating epoch #256 train run on device rank=0 [2024-06-22 04:00:08,981] INFO: Initiating epoch #256 valid run on device rank=0 [2024-06-22 04:00:57,305] INFO: Rank 0: epoch=256 / 400 train_loss=10.2525 valid_loss=10.3066 stale=0 time=3.88m eta=555.3m [2024-06-22 04:00:57,373] INFO: Initiating epoch #257 train run on device rank=0 [2024-06-22 04:03:59,956] INFO: Initiating epoch #257 valid run on device rank=0 [2024-06-22 04:04:49,202] INFO: Rank 0: epoch=257 / 400 train_loss=10.2522 valid_loss=10.3062 stale=0 time=3.86m eta=551.5m [2024-06-22 04:04:49,228] INFO: Initiating epoch #258 train run on device rank=0 [2024-06-22 04:07:54,033] INFO: Initiating epoch #258 valid run on device rank=0 [2024-06-22 04:08:42,348] INFO: Rank 0: epoch=258 / 400 train_loss=10.2518 valid_loss=10.3059 stale=0 time=3.89m eta=547.6m [2024-06-22 04:08:42,455] INFO: Initiating epoch #259 train run on device rank=0 [2024-06-22 04:11:47,305] INFO: Initiating epoch #259 valid run on device rank=0 [2024-06-22 04:12:35,107] INFO: Rank 0: epoch=259 / 400 train_loss=10.2514 valid_loss=10.3055 stale=0 time=3.88m eta=543.8m [2024-06-22 04:12:35,120] INFO: Initiating epoch #260 train run on device rank=0 [2024-06-22 04:15:36,468] INFO: Initiating epoch #260 valid run on device rank=0 [2024-06-22 04:16:24,346] INFO: Rank 0: epoch=260 / 400 train_loss=10.2510 valid_loss=10.3051 stale=0 time=3.82m eta=539.9m [2024-06-22 04:16:24,431] INFO: Initiating epoch #261 train run on device rank=0 [2024-06-22 04:19:26,734] INFO: Initiating epoch #261 valid run on device rank=0 [2024-06-22 04:20:13,782] INFO: Rank 0: epoch=261 / 400 train_loss=10.2506 valid_loss=10.3047 stale=0 time=3.82m eta=536.0m [2024-06-22 04:20:13,812] INFO: Initiating epoch #262 train run on device rank=0 [2024-06-22 04:23:13,852] INFO: Initiating epoch #262 valid run on device rank=0 [2024-06-22 04:24:02,221] INFO: Rank 0: epoch=262 / 400 train_loss=10.2503 valid_loss=10.3043 stale=0 time=3.81m eta=532.1m [2024-06-22 04:24:02,236] INFO: Initiating epoch #263 train run on device rank=0 [2024-06-22 04:27:06,360] INFO: Initiating epoch #263 valid run on device rank=0 [2024-06-22 04:27:55,550] INFO: Rank 0: epoch=263 / 400 train_loss=10.2499 valid_loss=10.3040 stale=0 time=3.89m eta=528.3m [2024-06-22 04:27:55,573] INFO: Initiating epoch #264 train run on device rank=0 [2024-06-22 04:30:59,105] INFO: Initiating epoch #264 valid run on device rank=0 [2024-06-22 04:31:46,752] INFO: Rank 0: epoch=264 / 400 train_loss=10.2495 valid_loss=10.3036 stale=0 time=3.85m eta=524.4m [2024-06-22 04:31:46,781] INFO: Initiating epoch #265 train run on device rank=0 [2024-06-22 04:34:50,477] INFO: Initiating epoch #265 valid run on device rank=0 [2024-06-22 04:35:39,441] INFO: Rank 0: epoch=265 / 400 train_loss=10.2492 valid_loss=10.3032 stale=0 time=3.88m eta=520.6m [2024-06-22 04:35:39,487] INFO: Initiating epoch #266 train run on device rank=0 [2024-06-22 04:38:43,376] INFO: Initiating epoch #266 valid run on device rank=0 [2024-06-22 04:39:30,875] INFO: Rank 0: epoch=266 / 400 train_loss=10.2488 valid_loss=10.3029 stale=0 time=3.86m eta=516.7m [2024-06-22 04:39:30,879] INFO: Initiating epoch #267 train run on device rank=0 [2024-06-22 04:42:33,003] INFO: Initiating epoch #267 valid run on device rank=0 [2024-06-22 04:43:20,922] INFO: Rank 0: epoch=267 / 400 train_loss=10.2485 valid_loss=10.3025 stale=0 time=3.83m eta=512.9m [2024-06-22 04:43:20,957] INFO: Initiating epoch #268 train run on device rank=0 [2024-06-22 04:46:24,234] INFO: Initiating epoch #268 valid run on device rank=0 [2024-06-22 04:47:12,812] INFO: Rank 0: epoch=268 / 400 train_loss=10.2481 valid_loss=10.3022 stale=0 time=3.86m eta=509.0m [2024-06-22 04:47:12,855] INFO: Initiating epoch #269 train run on device rank=0 [2024-06-22 04:50:16,171] INFO: Initiating epoch #269 valid run on device rank=0 [2024-06-22 04:51:04,553] INFO: Rank 0: epoch=269 / 400 train_loss=10.2478 valid_loss=10.3018 stale=0 time=3.86m eta=505.2m [2024-06-22 04:51:04,621] INFO: Initiating epoch #270 train run on device rank=0 [2024-06-22 04:54:06,977] INFO: Initiating epoch #270 valid run on device rank=0 [2024-06-22 04:54:56,268] INFO: Rank 0: epoch=270 / 400 train_loss=10.2474 valid_loss=10.3015 stale=0 time=3.86m eta=501.3m [2024-06-22 04:54:56,377] INFO: Initiating epoch #271 train run on device rank=0 [2024-06-22 04:58:00,378] INFO: Initiating epoch #271 valid run on device rank=0 [2024-06-22 04:58:48,044] INFO: Rank 0: epoch=271 / 400 train_loss=10.2471 valid_loss=10.3011 stale=0 time=3.86m eta=497.5m [2024-06-22 04:58:48,090] INFO: Initiating epoch #272 train run on device rank=0 [2024-06-22 05:01:50,230] INFO: Initiating epoch #272 valid run on device rank=0 [2024-06-22 05:02:37,928] INFO: Rank 0: epoch=272 / 400 train_loss=10.2467 valid_loss=10.3007 stale=0 time=3.83m eta=493.6m [2024-06-22 05:02:37,994] INFO: Initiating epoch #273 train run on device rank=0 [2024-06-22 05:05:40,717] INFO: Initiating epoch #273 valid run on device rank=0 [2024-06-22 05:06:28,226] INFO: Rank 0: epoch=273 / 400 train_loss=10.2464 valid_loss=10.3004 stale=0 time=3.84m eta=489.7m [2024-06-22 05:06:28,298] INFO: Initiating epoch #274 train run on device rank=0 [2024-06-22 05:09:29,982] INFO: Initiating epoch #274 valid run on device rank=0 [2024-06-22 05:10:20,590] INFO: Rank 0: epoch=274 / 400 train_loss=10.2460 valid_loss=10.3001 stale=0 time=3.87m eta=485.9m [2024-06-22 05:10:20,691] INFO: Initiating epoch #275 train run on device rank=0 [2024-06-22 05:13:29,698] INFO: Initiating epoch #275 valid run on device rank=0 [2024-06-22 05:14:18,504] INFO: Rank 0: epoch=275 / 400 train_loss=10.2457 valid_loss=10.2997 stale=0 time=3.96m eta=482.1m [2024-06-22 05:14:18,548] INFO: Initiating epoch #276 train run on device rank=0 [2024-06-22 05:17:21,084] INFO: Initiating epoch #276 valid run on device rank=0 [2024-06-22 05:18:08,685] INFO: Rank 0: epoch=276 / 400 train_loss=10.2453 valid_loss=10.2994 stale=0 time=3.84m eta=478.2m [2024-06-22 05:18:08,773] INFO: Initiating epoch #277 train run on device rank=0 [2024-06-22 05:21:13,954] INFO: Initiating epoch #277 valid run on device rank=0 [2024-06-22 05:22:02,241] INFO: Rank 0: epoch=277 / 400 train_loss=10.2450 valid_loss=10.2990 stale=0 time=3.89m eta=474.4m [2024-06-22 05:22:02,244] INFO: Initiating epoch #278 train run on device rank=0 [2024-06-22 05:25:02,279] INFO: Initiating epoch #278 valid run on device rank=0 [2024-06-22 05:25:51,285] INFO: Rank 0: epoch=278 / 400 train_loss=10.2447 valid_loss=10.2987 stale=0 time=3.82m eta=470.5m [2024-06-22 05:25:51,385] INFO: Initiating epoch #279 train run on device rank=0 [2024-06-22 05:28:53,928] INFO: Initiating epoch #279 valid run on device rank=0 [2024-06-22 05:29:42,180] INFO: Rank 0: epoch=279 / 400 train_loss=10.2444 valid_loss=10.2984 stale=0 time=3.85m eta=466.6m [2024-06-22 05:29:42,375] INFO: Initiating epoch #280 train run on device rank=0 [2024-06-22 05:32:46,764] INFO: Initiating epoch #280 valid run on device rank=0 [2024-06-22 05:33:34,569] INFO: Rank 0: epoch=280 / 400 train_loss=10.2440 valid_loss=10.2980 stale=0 time=3.87m eta=462.8m [2024-06-22 05:33:34,592] INFO: Initiating epoch #281 train run on device rank=0 [2024-06-22 05:36:39,740] INFO: Initiating epoch #281 valid run on device rank=0 [2024-06-22 05:37:27,365] INFO: Rank 0: epoch=281 / 400 train_loss=10.2437 valid_loss=10.2977 stale=0 time=3.88m eta=458.9m [2024-06-22 05:37:27,399] INFO: Initiating epoch #282 train run on device rank=0 [2024-06-22 05:40:29,209] INFO: Initiating epoch #282 valid run on device rank=0 [2024-06-22 05:41:17,048] INFO: Rank 0: epoch=282 / 400 train_loss=10.2434 valid_loss=10.2974 stale=0 time=3.83m eta=455.1m [2024-06-22 05:41:17,104] INFO: Initiating epoch #283 train run on device rank=0 [2024-06-22 05:44:19,846] INFO: Initiating epoch #283 valid run on device rank=0 [2024-06-22 05:45:07,462] INFO: Rank 0: epoch=283 / 400 train_loss=10.2431 valid_loss=10.2970 stale=0 time=3.84m eta=451.2m [2024-06-22 05:45:07,478] INFO: Initiating epoch #284 train run on device rank=0 [2024-06-22 05:48:12,936] INFO: Initiating epoch #284 valid run on device rank=0 [2024-06-22 05:49:00,007] INFO: Rank 0: epoch=284 / 400 train_loss=10.2427 valid_loss=10.2967 stale=0 time=3.88m eta=447.4m [2024-06-22 05:49:00,015] INFO: Initiating epoch #285 train run on device rank=0 [2024-06-22 05:52:03,233] INFO: Initiating epoch #285 valid run on device rank=0 [2024-06-22 05:52:51,899] INFO: Rank 0: epoch=285 / 400 train_loss=10.2424 valid_loss=10.2964 stale=0 time=3.86m eta=443.5m [2024-06-22 05:52:51,947] INFO: Initiating epoch #286 train run on device rank=0 [2024-06-22 05:55:55,431] INFO: Initiating epoch #286 valid run on device rank=0 [2024-06-22 05:56:43,405] INFO: Rank 0: epoch=286 / 400 train_loss=10.2421 valid_loss=10.2961 stale=0 time=3.86m eta=439.6m [2024-06-22 05:56:43,458] INFO: Initiating epoch #287 train run on device rank=0 [2024-06-22 05:59:43,892] INFO: Initiating epoch #287 valid run on device rank=0 [2024-06-22 06:00:32,607] INFO: Rank 0: epoch=287 / 400 train_loss=10.2418 valid_loss=10.2958 stale=0 time=3.82m eta=435.8m [2024-06-22 06:00:32,656] INFO: Initiating epoch #288 train run on device rank=0 [2024-06-22 06:03:34,970] INFO: Initiating epoch #288 valid run on device rank=0 [2024-06-22 06:04:23,372] INFO: Rank 0: epoch=288 / 400 train_loss=10.2415 valid_loss=10.2954 stale=0 time=3.85m eta=431.9m [2024-06-22 06:04:23,493] INFO: Initiating epoch #289 train run on device rank=0 [2024-06-22 06:07:31,197] INFO: Initiating epoch #289 valid run on device rank=0 [2024-06-22 06:08:18,642] INFO: Rank 0: epoch=289 / 400 train_loss=10.2412 valid_loss=10.2951 stale=0 time=3.92m eta=428.1m [2024-06-22 06:08:18,690] INFO: Initiating epoch #290 train run on device rank=0 [2024-06-22 06:11:21,120] INFO: Initiating epoch #290 valid run on device rank=0 [2024-06-22 06:12:08,904] INFO: Rank 0: epoch=290 / 400 train_loss=10.2409 valid_loss=10.2948 stale=0 time=3.84m eta=424.2m [2024-06-22 06:12:08,925] INFO: Initiating epoch #291 train run on device rank=0 [2024-06-22 06:15:16,802] INFO: Initiating epoch #291 valid run on device rank=0 [2024-06-22 06:16:04,400] INFO: Rank 0: epoch=291 / 400 train_loss=10.2405 valid_loss=10.2945 stale=0 time=3.92m eta=420.4m [2024-06-22 06:16:04,504] INFO: Initiating epoch #292 train run on device rank=0 [2024-06-22 06:19:08,518] INFO: Initiating epoch #292 valid run on device rank=0 [2024-06-22 06:19:58,738] INFO: Rank 0: epoch=292 / 400 train_loss=10.2402 valid_loss=10.2942 stale=0 time=3.9m eta=416.6m [2024-06-22 06:19:58,788] INFO: Initiating epoch #293 train run on device rank=0 [2024-06-22 06:23:00,374] INFO: Initiating epoch #293 valid run on device rank=0 [2024-06-22 06:23:47,719] INFO: Rank 0: epoch=293 / 400 train_loss=10.2399 valid_loss=10.2939 stale=0 time=3.82m eta=412.7m [2024-06-22 06:23:48,461] INFO: Initiating epoch #294 train run on device rank=0 [2024-06-22 06:26:55,194] INFO: Initiating epoch #294 valid run on device rank=0 [2024-06-22 06:27:43,881] INFO: Rank 0: epoch=294 / 400 train_loss=10.2396 valid_loss=10.2936 stale=0 time=3.92m eta=408.9m [2024-06-22 06:27:43,919] INFO: Initiating epoch #295 train run on device rank=0 [2024-06-22 06:30:47,421] INFO: Initiating epoch #295 valid run on device rank=0 [2024-06-22 06:31:37,430] INFO: Rank 0: epoch=295 / 400 train_loss=10.2393 valid_loss=10.2933 stale=0 time=3.89m eta=405.0m [2024-06-22 06:31:37,445] INFO: Initiating epoch #296 train run on device rank=0 [2024-06-22 06:34:39,148] INFO: Initiating epoch #296 valid run on device rank=0 [2024-06-22 06:35:26,749] INFO: Rank 0: epoch=296 / 400 train_loss=10.2390 valid_loss=10.2930 stale=0 time=3.82m eta=401.1m [2024-06-22 06:35:26,772] INFO: Initiating epoch #297 train run on device rank=0 [2024-06-22 06:38:30,521] INFO: Initiating epoch #297 valid run on device rank=0 [2024-06-22 06:39:19,232] INFO: Rank 0: epoch=297 / 400 train_loss=10.2388 valid_loss=10.2927 stale=0 time=3.87m eta=397.3m [2024-06-22 06:39:19,278] INFO: Initiating epoch #298 train run on device rank=0 [2024-06-22 06:42:22,969] INFO: Initiating epoch #298 valid run on device rank=0 [2024-06-22 06:43:09,965] INFO: Rank 0: epoch=298 / 400 train_loss=10.2385 valid_loss=10.2924 stale=0 time=3.84m eta=393.4m [2024-06-22 06:43:09,981] INFO: Initiating epoch #299 train run on device rank=0 [2024-06-22 06:46:13,247] INFO: Initiating epoch #299 valid run on device rank=0 [2024-06-22 06:47:00,618] INFO: Rank 0: epoch=299 / 400 train_loss=10.2382 valid_loss=10.2921 stale=0 time=3.84m eta=389.6m [2024-06-22 06:47:00,633] INFO: Initiating epoch #300 train run on device rank=0 [2024-06-22 06:50:08,215] INFO: Initiating epoch #300 valid run on device rank=0 [2024-06-22 06:50:57,155] INFO: Rank 0: epoch=300 / 400 train_loss=10.2379 valid_loss=10.2918 stale=0 time=3.94m eta=385.7m [2024-06-22 06:50:57,255] INFO: Initiating epoch #301 train run on device rank=0 [2024-06-22 06:53:59,445] INFO: Initiating epoch #301 valid run on device rank=0 [2024-06-22 06:54:47,776] INFO: Rank 0: epoch=301 / 400 train_loss=10.2376 valid_loss=10.2915 stale=0 time=3.84m eta=381.9m [2024-06-22 06:54:48,028] INFO: Initiating epoch #302 train run on device rank=0 [2024-06-22 06:57:49,179] INFO: Initiating epoch #302 valid run on device rank=0 [2024-06-22 06:58:37,402] INFO: Rank 0: epoch=302 / 400 train_loss=10.2373 valid_loss=10.2912 stale=0 time=3.82m eta=378.0m [2024-06-22 06:58:37,444] INFO: Initiating epoch #303 train run on device rank=0 [2024-06-22 07:01:40,456] INFO: Initiating epoch #303 valid run on device rank=0 [2024-06-22 07:02:29,267] INFO: Rank 0: epoch=303 / 400 train_loss=10.2370 valid_loss=10.2909 stale=0 time=3.86m eta=374.2m [2024-06-22 07:02:29,360] INFO: Initiating epoch #304 train run on device rank=0 [2024-06-22 07:05:31,406] INFO: Initiating epoch #304 valid run on device rank=0 [2024-06-22 07:06:19,097] INFO: Rank 0: epoch=304 / 400 train_loss=10.2367 valid_loss=10.2907 stale=0 time=3.83m eta=370.3m [2024-06-22 07:06:19,161] INFO: Initiating epoch #305 train run on device rank=0 [2024-06-22 07:09:21,519] INFO: Initiating epoch #305 valid run on device rank=0 [2024-06-22 07:10:09,986] INFO: Rank 0: epoch=305 / 400 train_loss=10.2365 valid_loss=10.2904 stale=0 time=3.85m eta=366.4m [2024-06-22 07:10:10,093] INFO: Initiating epoch #306 train run on device rank=0 [2024-06-22 07:13:10,984] INFO: Initiating epoch #306 valid run on device rank=0 [2024-06-22 07:13:58,845] INFO: Rank 0: epoch=306 / 400 train_loss=10.2362 valid_loss=10.2901 stale=0 time=3.81m eta=362.6m [2024-06-22 07:13:58,887] INFO: Initiating epoch #307 train run on device rank=0 [2024-06-22 07:17:03,675] INFO: Initiating epoch #307 valid run on device rank=0 [2024-06-22 07:17:51,339] INFO: Rank 0: epoch=307 / 400 train_loss=10.2359 valid_loss=10.2898 stale=0 time=3.87m eta=358.7m [2024-06-22 07:17:51,393] INFO: Initiating epoch #308 train run on device rank=0 [2024-06-22 07:20:53,871] INFO: Initiating epoch #308 valid run on device rank=0 [2024-06-22 07:21:42,294] INFO: Rank 0: epoch=308 / 400 train_loss=10.2356 valid_loss=10.2895 stale=0 time=3.85m eta=354.8m [2024-06-22 07:21:42,360] INFO: Initiating epoch #309 train run on device rank=0 [2024-06-22 07:24:42,872] INFO: Initiating epoch #309 valid run on device rank=0 [2024-06-22 07:25:30,431] INFO: Rank 0: epoch=309 / 400 train_loss=10.2353 valid_loss=10.2892 stale=0 time=3.8m eta=351.0m [2024-06-22 07:25:30,711] INFO: Initiating epoch #310 train run on device rank=0 [2024-06-22 07:28:32,515] INFO: Initiating epoch #310 valid run on device rank=0 [2024-06-22 07:29:20,062] INFO: Rank 0: epoch=310 / 400 train_loss=10.2351 valid_loss=10.2890 stale=0 time=3.82m eta=347.1m [2024-06-22 07:29:20,084] INFO: Initiating epoch #311 train run on device rank=0 [2024-06-22 07:32:23,828] INFO: Initiating epoch #311 valid run on device rank=0 [2024-06-22 07:33:11,887] INFO: Rank 0: epoch=311 / 400 train_loss=10.2348 valid_loss=10.2887 stale=0 time=3.86m eta=343.3m [2024-06-22 07:33:11,914] INFO: Initiating epoch #312 train run on device rank=0 [2024-06-22 07:36:12,914] INFO: Initiating epoch #312 valid run on device rank=0 [2024-06-22 07:37:00,798] INFO: Rank 0: epoch=312 / 400 train_loss=10.2345 valid_loss=10.2884 stale=0 time=3.81m eta=339.4m [2024-06-22 07:37:00,818] INFO: Initiating epoch #313 train run on device rank=0 [2024-06-22 07:40:06,515] INFO: Initiating epoch #313 valid run on device rank=0 [2024-06-22 07:40:56,818] INFO: Rank 0: epoch=313 / 400 train_loss=10.2343 valid_loss=10.2881 stale=0 time=3.93m eta=335.5m [2024-06-22 07:40:57,054] INFO: Initiating epoch #314 train run on device rank=0 [2024-06-22 07:43:56,451] INFO: Initiating epoch #314 valid run on device rank=0 [2024-06-22 07:44:43,868] INFO: Rank 0: epoch=314 / 400 train_loss=10.2340 valid_loss=10.2879 stale=0 time=3.78m eta=331.7m [2024-06-22 07:44:43,913] INFO: Initiating epoch #315 train run on device rank=0 [2024-06-22 07:47:46,110] INFO: Initiating epoch #315 valid run on device rank=0 [2024-06-22 07:48:36,218] INFO: Rank 0: epoch=315 / 400 train_loss=10.2337 valid_loss=10.2876 stale=0 time=3.87m eta=327.8m [2024-06-22 07:48:36,771] INFO: Initiating epoch #316 train run on device rank=0 [2024-06-22 07:51:35,256] INFO: Initiating epoch #316 valid run on device rank=0 [2024-06-22 07:52:29,386] INFO: Rank 0: epoch=316 / 400 train_loss=10.2334 valid_loss=10.2873 stale=0 time=3.88m eta=324.0m [2024-06-22 07:52:30,155] INFO: Initiating epoch #317 train run on device rank=0 [2024-06-22 07:55:29,792] INFO: Initiating epoch #317 valid run on device rank=0 [2024-06-22 07:56:19,329] INFO: Rank 0: epoch=317 / 400 train_loss=10.2332 valid_loss=10.2870 stale=0 time=3.82m eta=320.1m [2024-06-22 07:56:19,363] INFO: Initiating epoch #318 train run on device rank=0 [2024-06-22 07:59:23,635] INFO: Initiating epoch #318 valid run on device rank=0 [2024-06-22 08:00:11,064] INFO: Rank 0: epoch=318 / 400 train_loss=10.2329 valid_loss=10.2868 stale=0 time=3.86m eta=316.3m [2024-06-22 08:00:11,085] INFO: Initiating epoch #319 train run on device rank=0 [2024-06-22 08:03:15,260] INFO: Initiating epoch #319 valid run on device rank=0 [2024-06-22 08:04:04,266] INFO: Rank 0: epoch=319 / 400 train_loss=10.2327 valid_loss=10.2865 stale=0 time=3.89m eta=312.4m [2024-06-22 08:04:04,290] INFO: Initiating epoch #320 train run on device rank=0 [2024-06-22 08:07:02,834] INFO: Initiating epoch #320 valid run on device rank=0 [2024-06-22 08:07:52,127] INFO: Rank 0: epoch=320 / 400 train_loss=10.2324 valid_loss=10.2862 stale=0 time=3.8m eta=308.5m [2024-06-22 08:07:52,373] INFO: Initiating epoch #321 train run on device rank=0 [2024-06-22 08:10:54,424] INFO: Initiating epoch #321 valid run on device rank=0 [2024-06-22 08:11:40,877] INFO: Rank 0: epoch=321 / 400 train_loss=10.2321 valid_loss=10.2860 stale=0 time=3.81m eta=304.7m [2024-06-22 08:11:40,906] INFO: Initiating epoch #322 train run on device rank=0 [2024-06-22 08:14:41,828] INFO: Initiating epoch #322 valid run on device rank=0 [2024-06-22 08:15:31,346] INFO: Rank 0: epoch=322 / 400 train_loss=10.2319 valid_loss=10.2857 stale=0 time=3.84m eta=300.8m [2024-06-22 08:15:31,432] INFO: Initiating epoch #323 train run on device rank=0 [2024-06-22 08:18:39,903] INFO: Initiating epoch #323 valid run on device rank=0 [2024-06-22 08:19:27,042] INFO: Rank 0: epoch=323 / 400 train_loss=10.2316 valid_loss=10.2855 stale=0 time=3.93m eta=297.0m [2024-06-22 08:19:27,070] INFO: Initiating epoch #324 train run on device rank=0 [2024-06-22 08:22:30,190] INFO: Initiating epoch #324 valid run on device rank=0 [2024-06-22 08:23:17,214] INFO: Rank 0: epoch=324 / 400 train_loss=10.2314 valid_loss=10.2852 stale=0 time=3.84m eta=293.1m [2024-06-22 08:23:17,246] INFO: Initiating epoch #325 train run on device rank=0 [2024-06-22 08:26:20,578] INFO: Initiating epoch #325 valid run on device rank=0 [2024-06-22 08:27:09,561] INFO: Rank 0: epoch=325 / 400 train_loss=10.2311 valid_loss=10.2850 stale=0 time=3.87m eta=289.2m [2024-06-22 08:27:09,612] INFO: Initiating epoch #326 train run on device rank=0 [2024-06-22 08:30:12,418] INFO: Initiating epoch #326 valid run on device rank=0 [2024-06-22 08:31:00,403] INFO: Rank 0: epoch=326 / 400 train_loss=10.2309 valid_loss=10.2847 stale=0 time=3.85m eta=285.4m [2024-06-22 08:31:00,449] INFO: Initiating epoch #327 train run on device rank=0 [2024-06-22 08:34:03,325] INFO: Initiating epoch #327 valid run on device rank=0 [2024-06-22 08:34:52,450] INFO: Rank 0: epoch=327 / 400 train_loss=10.2306 valid_loss=10.2845 stale=0 time=3.87m eta=281.5m [2024-06-22 08:34:52,473] INFO: Initiating epoch #328 train run on device rank=0 [2024-06-22 08:37:57,869] INFO: Initiating epoch #328 valid run on device rank=0 [2024-06-22 08:38:46,711] INFO: Rank 0: epoch=328 / 400 train_loss=10.2304 valid_loss=10.2842 stale=0 time=3.9m eta=277.7m [2024-06-22 08:38:46,735] INFO: Initiating epoch #329 train run on device rank=0 [2024-06-22 08:41:49,768] INFO: Initiating epoch #329 valid run on device rank=0 [2024-06-22 08:42:37,981] INFO: Rank 0: epoch=329 / 400 train_loss=10.2301 valid_loss=10.2840 stale=0 time=3.85m eta=273.8m [2024-06-22 08:42:38,071] INFO: Initiating epoch #330 train run on device rank=0 [2024-06-22 08:45:38,701] INFO: Initiating epoch #330 valid run on device rank=0 [2024-06-22 08:46:26,463] INFO: Rank 0: epoch=330 / 400 train_loss=10.2299 valid_loss=10.2837 stale=0 time=3.81m eta=270.0m [2024-06-22 08:46:26,493] INFO: Initiating epoch #331 train run on device rank=0 [2024-06-22 08:49:29,036] INFO: Initiating epoch #331 valid run on device rank=0 [2024-06-22 08:50:17,105] INFO: Rank 0: epoch=331 / 400 train_loss=10.2296 valid_loss=10.2835 stale=0 time=3.84m eta=266.1m [2024-06-22 08:50:17,183] INFO: Initiating epoch #332 train run on device rank=0 [2024-06-22 08:53:21,340] INFO: Initiating epoch #332 valid run on device rank=0 [2024-06-22 08:54:09,720] INFO: Rank 0: epoch=332 / 400 train_loss=10.2294 valid_loss=10.2832 stale=0 time=3.88m eta=262.3m [2024-06-22 08:54:09,790] INFO: Initiating epoch #333 train run on device rank=0 [2024-06-22 08:57:16,825] INFO: Initiating epoch #333 valid run on device rank=0 [2024-06-22 08:58:05,747] INFO: Rank 0: epoch=333 / 400 train_loss=10.2292 valid_loss=10.2830 stale=0 time=3.93m eta=258.4m [2024-06-22 08:58:05,989] INFO: Initiating epoch #334 train run on device rank=0 [2024-06-22 09:01:08,798] INFO: Initiating epoch #334 valid run on device rank=0 [2024-06-22 09:01:58,855] INFO: Rank 0: epoch=334 / 400 train_loss=10.2289 valid_loss=10.2827 stale=0 time=3.88m eta=254.6m [2024-06-22 09:01:59,487] INFO: Initiating epoch #335 train run on device rank=0 [2024-06-22 09:04:59,589] INFO: Initiating epoch #335 valid run on device rank=0 [2024-06-22 09:05:45,997] INFO: Rank 0: epoch=335 / 400 train_loss=10.2287 valid_loss=10.2825 stale=0 time=3.78m eta=250.7m [2024-06-22 09:05:46,039] INFO: Initiating epoch #336 train run on device rank=0 [2024-06-22 09:08:47,243] INFO: Initiating epoch #336 valid run on device rank=0 [2024-06-22 09:09:35,671] INFO: Rank 0: epoch=336 / 400 train_loss=10.2284 valid_loss=10.2822 stale=0 time=3.83m eta=246.8m [2024-06-22 09:09:35,696] INFO: Initiating epoch #337 train run on device rank=0 [2024-06-22 09:12:41,020] INFO: Initiating epoch #337 valid run on device rank=0 [2024-06-22 09:13:28,825] INFO: Rank 0: epoch=337 / 400 train_loss=10.2282 valid_loss=10.2820 stale=0 time=3.89m eta=243.0m [2024-06-22 09:13:28,871] INFO: Initiating epoch #338 train run on device rank=0 [2024-06-22 09:16:35,447] INFO: Initiating epoch #338 valid run on device rank=0 [2024-06-22 09:17:22,605] INFO: Rank 0: epoch=338 / 400 train_loss=10.2280 valid_loss=10.2818 stale=0 time=3.9m eta=239.1m [2024-06-22 09:17:23,124] INFO: Initiating epoch #339 train run on device rank=0 [2024-06-22 09:20:27,128] INFO: Initiating epoch #339 valid run on device rank=0 [2024-06-22 09:21:16,770] INFO: Rank 0: epoch=339 / 400 train_loss=10.2277 valid_loss=10.2815 stale=0 time=3.89m eta=235.3m [2024-06-22 09:21:16,819] INFO: Initiating epoch #340 train run on device rank=0 [2024-06-22 09:24:20,939] INFO: Initiating epoch #340 valid run on device rank=0 [2024-06-22 09:25:09,794] INFO: Rank 0: epoch=340 / 400 train_loss=10.2275 valid_loss=10.2813 stale=0 time=3.88m eta=231.4m [2024-06-22 09:25:09,878] INFO: Initiating epoch #341 train run on device rank=0 [2024-06-22 09:28:12,831] INFO: Initiating epoch #341 valid run on device rank=0 [2024-06-22 09:29:00,365] INFO: Rank 0: epoch=341 / 400 train_loss=10.2272 valid_loss=10.2810 stale=0 time=3.84m eta=227.6m [2024-06-22 09:29:00,400] INFO: Initiating epoch #342 train run on device rank=0 [2024-06-22 09:32:05,325] INFO: Initiating epoch #342 valid run on device rank=0 [2024-06-22 09:32:52,972] INFO: Rank 0: epoch=342 / 400 train_loss=10.2270 valid_loss=10.2808 stale=0 time=3.88m eta=223.7m [2024-06-22 09:32:53,003] INFO: Initiating epoch #343 train run on device rank=0 [2024-06-22 09:35:57,058] INFO: Initiating epoch #343 valid run on device rank=0 [2024-06-22 09:36:46,449] INFO: Rank 0: epoch=343 / 400 train_loss=10.2267 valid_loss=10.2805 stale=0 time=3.89m eta=219.9m [2024-06-22 09:36:46,540] INFO: Initiating epoch #344 train run on device rank=0 [2024-06-22 09:39:57,125] INFO: Initiating epoch #344 valid run on device rank=0 [2024-06-22 09:40:48,283] INFO: Rank 0: epoch=344 / 400 train_loss=10.2265 valid_loss=10.2803 stale=0 time=4.03m eta=216.0m [2024-06-22 09:40:48,744] INFO: Initiating epoch #345 train run on device rank=0 [2024-06-22 09:43:51,944] INFO: Initiating epoch #345 valid run on device rank=0 [2024-06-22 09:44:40,123] INFO: Rank 0: epoch=345 / 400 train_loss=10.2262 valid_loss=10.2800 stale=0 time=3.86m eta=212.2m [2024-06-22 09:44:40,176] INFO: Initiating epoch #346 train run on device rank=0 [2024-06-22 09:47:43,788] INFO: Initiating epoch #346 valid run on device rank=0 [2024-06-22 09:48:31,987] INFO: Rank 0: epoch=346 / 400 train_loss=10.2260 valid_loss=10.2798 stale=0 time=3.86m eta=208.3m [2024-06-22 09:48:32,011] INFO: Initiating epoch #347 train run on device rank=0 [2024-06-22 09:51:36,608] INFO: Initiating epoch #347 valid run on device rank=0 [2024-06-22 09:52:25,174] INFO: Rank 0: epoch=347 / 400 train_loss=10.2258 valid_loss=10.2795 stale=0 time=3.89m eta=204.5m [2024-06-22 09:52:25,239] INFO: Initiating epoch #348 train run on device rank=0 [2024-06-22 09:55:27,744] INFO: Initiating epoch #348 valid run on device rank=0 [2024-06-22 09:56:16,298] INFO: Rank 0: epoch=348 / 400 train_loss=10.2256 valid_loss=10.2793 stale=0 time=3.85m eta=200.6m [2024-06-22 09:56:16,333] INFO: Initiating epoch #349 train run on device rank=0 [2024-06-22 09:59:21,115] INFO: Initiating epoch #349 valid run on device rank=0 [2024-06-22 10:00:09,902] INFO: Rank 0: epoch=349 / 400 train_loss=10.2253 valid_loss=10.2791 stale=0 time=3.89m eta=196.8m [2024-06-22 10:00:09,952] INFO: Initiating epoch #350 train run on device rank=0 [2024-06-22 10:03:14,898] INFO: Initiating epoch #350 valid run on device rank=0 [2024-06-22 10:04:01,930] INFO: Rank 0: epoch=350 / 400 train_loss=10.2251 valid_loss=10.2788 stale=0 time=3.87m eta=192.9m [2024-06-22 10:04:01,977] INFO: Initiating epoch #351 train run on device rank=0 [2024-06-22 10:07:05,931] INFO: Initiating epoch #351 valid run on device rank=0 [2024-06-22 10:07:54,406] INFO: Rank 0: epoch=351 / 400 train_loss=10.2249 valid_loss=10.2786 stale=0 time=3.87m eta=189.0m [2024-06-22 10:07:54,450] INFO: Initiating epoch #352 train run on device rank=0 [2024-06-22 10:10:57,249] INFO: Initiating epoch #352 valid run on device rank=0 [2024-06-22 10:11:44,745] INFO: Rank 0: epoch=352 / 400 train_loss=10.2247 valid_loss=10.2784 stale=0 time=3.84m eta=185.2m [2024-06-22 10:11:44,805] INFO: Initiating epoch #353 train run on device rank=0 [2024-06-22 10:14:47,155] INFO: Initiating epoch #353 valid run on device rank=0 [2024-06-22 10:15:34,553] INFO: Rank 0: epoch=353 / 400 train_loss=10.2244 valid_loss=10.2782 stale=0 time=3.83m eta=181.3m [2024-06-22 10:15:34,584] INFO: Initiating epoch #354 train run on device rank=0 [2024-06-22 10:18:35,448] INFO: Initiating epoch #354 valid run on device rank=0 [2024-06-22 10:19:22,971] INFO: Rank 0: epoch=354 / 400 train_loss=10.2242 valid_loss=10.2779 stale=0 time=3.81m eta=177.5m [2024-06-22 10:19:23,003] INFO: Initiating epoch #355 train run on device rank=0 [2024-06-22 10:22:31,060] INFO: Initiating epoch #355 valid run on device rank=0 [2024-06-22 10:23:19,509] INFO: Rank 0: epoch=355 / 400 train_loss=10.2240 valid_loss=10.2777 stale=0 time=3.94m eta=173.6m [2024-06-22 10:23:19,613] INFO: Initiating epoch #356 train run on device rank=0 [2024-06-22 10:26:22,844] INFO: Initiating epoch #356 valid run on device rank=0 [2024-06-22 10:27:10,937] INFO: Rank 0: epoch=356 / 400 train_loss=10.2238 valid_loss=10.2775 stale=0 time=3.86m eta=169.8m [2024-06-22 10:27:10,982] INFO: Initiating epoch #357 train run on device rank=0 [2024-06-22 10:30:11,774] INFO: Initiating epoch #357 valid run on device rank=0 [2024-06-22 10:30:58,658] INFO: Rank 0: epoch=357 / 400 train_loss=10.2236 valid_loss=10.2773 stale=0 time=3.79m eta=165.9m [2024-06-22 10:30:58,690] INFO: Initiating epoch #358 train run on device rank=0 [2024-06-22 10:34:02,846] INFO: Initiating epoch #358 valid run on device rank=0 [2024-06-22 10:34:52,047] INFO: Rank 0: epoch=358 / 400 train_loss=10.2234 valid_loss=10.2770 stale=0 time=3.89m eta=162.0m [2024-06-22 10:34:52,094] INFO: Initiating epoch #359 train run on device rank=0 [2024-06-22 10:37:58,197] INFO: Initiating epoch #359 valid run on device rank=0 [2024-06-22 10:38:45,860] INFO: Rank 0: epoch=359 / 400 train_loss=10.2232 valid_loss=10.2768 stale=0 time=3.9m eta=158.2m [2024-06-22 10:38:45,873] INFO: Initiating epoch #360 train run on device rank=0 [2024-06-22 10:41:49,011] INFO: Initiating epoch #360 valid run on device rank=0 [2024-06-22 10:42:38,235] INFO: Rank 0: epoch=360 / 400 train_loss=10.2230 valid_loss=10.2766 stale=0 time=3.87m eta=154.3m [2024-06-22 10:42:38,265] INFO: Initiating epoch #361 train run on device rank=0 [2024-06-22 10:45:40,955] INFO: Initiating epoch #361 valid run on device rank=0 [2024-06-22 10:46:28,868] INFO: Rank 0: epoch=361 / 400 train_loss=10.2228 valid_loss=10.2764 stale=0 time=3.84m eta=150.5m [2024-06-22 10:46:28,897] INFO: Initiating epoch #362 train run on device rank=0 [2024-06-22 10:49:34,731] INFO: Initiating epoch #362 valid run on device rank=0 [2024-06-22 10:50:22,666] INFO: Rank 0: epoch=362 / 400 train_loss=10.2225 valid_loss=10.2762 stale=0 time=3.9m eta=146.6m [2024-06-22 10:50:22,737] INFO: Initiating epoch #363 train run on device rank=0 [2024-06-22 10:53:23,953] INFO: Initiating epoch #363 valid run on device rank=0 [2024-06-22 10:54:11,990] INFO: Rank 0: epoch=363 / 400 train_loss=10.2223 valid_loss=10.2760 stale=0 time=3.82m eta=142.7m [2024-06-22 10:54:12,025] INFO: Initiating epoch #364 train run on device rank=0 [2024-06-22 10:57:14,806] INFO: Initiating epoch #364 valid run on device rank=0 [2024-06-22 10:58:03,250] INFO: Rank 0: epoch=364 / 400 train_loss=10.2221 valid_loss=10.2757 stale=0 time=3.85m eta=138.9m [2024-06-22 10:58:03,301] INFO: Initiating epoch #365 train run on device rank=0 [2024-06-22 11:01:06,798] INFO: Initiating epoch #365 valid run on device rank=0 [2024-06-22 11:01:55,626] INFO: Rank 0: epoch=365 / 400 train_loss=10.2219 valid_loss=10.2755 stale=0 time=3.87m eta=135.0m [2024-06-22 11:01:55,734] INFO: Initiating epoch #366 train run on device rank=0 [2024-06-22 11:05:00,080] INFO: Initiating epoch #366 valid run on device rank=0 [2024-06-22 11:05:47,704] INFO: Rank 0: epoch=366 / 400 train_loss=10.2217 valid_loss=10.2753 stale=0 time=3.87m eta=131.2m [2024-06-22 11:05:47,744] INFO: Initiating epoch #367 train run on device rank=0 [2024-06-22 11:08:49,459] INFO: Initiating epoch #367 valid run on device rank=0 [2024-06-22 11:09:38,733] INFO: Rank 0: epoch=367 / 400 train_loss=10.2215 valid_loss=10.2751 stale=0 time=3.85m eta=127.3m [2024-06-22 11:09:38,776] INFO: Initiating epoch #368 train run on device rank=0 [2024-06-22 11:12:41,784] INFO: Initiating epoch #368 valid run on device rank=0 [2024-06-22 11:13:30,543] INFO: Rank 0: epoch=368 / 400 train_loss=10.2213 valid_loss=10.2749 stale=0 time=3.86m eta=123.5m [2024-06-22 11:13:30,567] INFO: Initiating epoch #369 train run on device rank=0 [2024-06-22 11:16:33,204] INFO: Initiating epoch #369 valid run on device rank=0 [2024-06-22 11:17:22,127] INFO: Rank 0: epoch=369 / 400 train_loss=10.2211 valid_loss=10.2747 stale=0 time=3.86m eta=119.6m [2024-06-22 11:17:22,154] INFO: Initiating epoch #370 train run on device rank=0 [2024-06-22 11:20:24,496] INFO: Initiating epoch #370 valid run on device rank=0 [2024-06-22 11:21:12,512] INFO: Rank 0: epoch=370 / 400 train_loss=10.2209 valid_loss=10.2744 stale=0 time=3.84m eta=115.7m [2024-06-22 11:21:12,577] INFO: Initiating epoch #371 train run on device rank=0 [2024-06-22 11:24:13,599] INFO: Initiating epoch #371 valid run on device rank=0 [2024-06-22 11:25:03,056] INFO: Rank 0: epoch=371 / 400 train_loss=10.2207 valid_loss=10.2742 stale=0 time=3.84m eta=111.9m [2024-06-22 11:25:03,125] INFO: Initiating epoch #372 train run on device rank=0 [2024-06-22 11:28:06,033] INFO: Initiating epoch #372 valid run on device rank=0 [2024-06-22 11:29:00,346] INFO: Rank 0: epoch=372 / 400 train_loss=10.2205 valid_loss=10.2740 stale=0 time=3.95m eta=108.0m [2024-06-22 11:29:00,378] INFO: Initiating epoch #373 train run on device rank=0 [2024-06-22 11:32:03,647] INFO: Initiating epoch #373 valid run on device rank=0 [2024-06-22 11:32:51,202] INFO: Rank 0: epoch=373 / 400 train_loss=10.2203 valid_loss=10.2738 stale=0 time=3.85m eta=104.2m [2024-06-22 11:32:51,261] INFO: Initiating epoch #374 train run on device rank=0 [2024-06-22 11:35:53,111] INFO: Initiating epoch #374 valid run on device rank=0 [2024-06-22 11:36:40,747] INFO: Rank 0: epoch=374 / 400 train_loss=10.2200 valid_loss=10.2736 stale=0 time=3.82m eta=100.3m [2024-06-22 11:36:40,796] INFO: Initiating epoch #375 train run on device rank=0 [2024-06-22 11:39:42,654] INFO: Initiating epoch #375 valid run on device rank=0 [2024-06-22 11:40:30,863] INFO: Rank 0: epoch=375 / 400 train_loss=10.2198 valid_loss=10.2733 stale=0 time=3.83m eta=96.5m [2024-06-22 11:40:30,964] INFO: Initiating epoch #376 train run on device rank=0 [2024-06-22 11:43:31,163] INFO: Initiating epoch #376 valid run on device rank=0 [2024-06-22 11:44:19,091] INFO: Rank 0: epoch=376 / 400 train_loss=10.2196 valid_loss=10.2731 stale=0 time=3.8m eta=92.6m [2024-06-22 11:44:19,196] INFO: Initiating epoch #377 train run on device rank=0 [2024-06-22 11:47:21,387] INFO: Initiating epoch #377 valid run on device rank=0 [2024-06-22 11:48:09,956] INFO: Rank 0: epoch=377 / 400 train_loss=10.2194 valid_loss=10.2729 stale=0 time=3.85m eta=88.7m [2024-06-22 11:48:09,972] INFO: Initiating epoch #378 train run on device rank=0 [2024-06-22 11:51:16,868] INFO: Initiating epoch #378 valid run on device rank=0 [2024-06-22 11:52:04,385] INFO: Rank 0: epoch=378 / 400 train_loss=10.2192 valid_loss=10.2727 stale=0 time=3.91m eta=84.9m [2024-06-22 11:52:04,554] INFO: Initiating epoch #379 train run on device rank=0 [2024-06-22 11:55:06,268] INFO: Initiating epoch #379 valid run on device rank=0 [2024-06-22 11:55:55,143] INFO: Rank 0: epoch=379 / 400 train_loss=10.2190 valid_loss=10.2725 stale=0 time=3.84m eta=81.0m [2024-06-22 11:55:55,191] INFO: Initiating epoch #380 train run on device rank=0 [2024-06-22 11:58:57,333] INFO: Initiating epoch #380 valid run on device rank=0 [2024-06-22 11:59:45,759] INFO: Rank 0: epoch=380 / 400 train_loss=10.2188 valid_loss=10.2722 stale=0 time=3.84m eta=77.2m [2024-06-22 11:59:45,784] INFO: Initiating epoch #381 train run on device rank=0 [2024-06-22 12:02:47,504] INFO: Initiating epoch #381 valid run on device rank=0 [2024-06-22 12:03:35,523] INFO: Rank 0: epoch=381 / 400 train_loss=10.2186 valid_loss=10.2720 stale=0 time=3.83m eta=73.3m [2024-06-22 12:03:35,571] INFO: Initiating epoch #382 train run on device rank=0 [2024-06-22 12:06:41,151] INFO: Initiating epoch #382 valid run on device rank=0 [2024-06-22 12:07:28,451] INFO: Rank 0: epoch=382 / 400 train_loss=10.2184 valid_loss=10.2718 stale=0 time=3.88m eta=69.4m [2024-06-22 12:07:28,604] INFO: Initiating epoch #383 train run on device rank=0 [2024-06-22 12:10:30,360] INFO: Initiating epoch #383 valid run on device rank=0 [2024-06-22 12:11:19,284] INFO: Rank 0: epoch=383 / 400 train_loss=10.2182 valid_loss=10.2716 stale=0 time=3.84m eta=65.6m [2024-06-22 12:11:19,355] INFO: Initiating epoch #384 train run on device rank=0 [2024-06-22 12:14:21,108] INFO: Initiating epoch #384 valid run on device rank=0 [2024-06-22 12:15:08,300] INFO: Rank 0: epoch=384 / 400 train_loss=10.2180 valid_loss=10.2714 stale=0 time=3.82m eta=61.7m [2024-06-22 12:15:08,326] INFO: Initiating epoch #385 train run on device rank=0 [2024-06-22 12:18:08,949] INFO: Initiating epoch #385 valid run on device rank=0 [2024-06-22 12:18:56,628] INFO: Rank 0: epoch=385 / 400 train_loss=10.2178 valid_loss=10.2712 stale=0 time=3.81m eta=57.9m [2024-06-22 12:18:56,652] INFO: Initiating epoch #386 train run on device rank=0 [2024-06-22 12:21:58,986] INFO: Initiating epoch #386 valid run on device rank=0 [2024-06-22 12:22:46,868] INFO: Rank 0: epoch=386 / 400 train_loss=10.2176 valid_loss=10.2710 stale=0 time=3.84m eta=54.0m [2024-06-22 12:22:46,891] INFO: Initiating epoch #387 train run on device rank=0 [2024-06-22 12:25:50,886] INFO: Initiating epoch #387 valid run on device rank=0 [2024-06-22 12:26:38,497] INFO: Rank 0: epoch=387 / 400 train_loss=10.2174 valid_loss=10.2708 stale=0 time=3.86m eta=50.1m [2024-06-22 12:26:38,542] INFO: Initiating epoch #388 train run on device rank=0 [2024-06-22 12:29:40,543] INFO: Initiating epoch #388 valid run on device rank=0 [2024-06-22 12:30:28,135] INFO: Rank 0: epoch=388 / 400 train_loss=10.2172 valid_loss=10.2706 stale=0 time=3.83m eta=46.3m [2024-06-22 12:30:28,231] INFO: Initiating epoch #389 train run on device rank=0 [2024-06-22 12:33:30,143] INFO: Initiating epoch #389 valid run on device rank=0 [2024-06-22 12:34:18,785] INFO: Rank 0: epoch=389 / 400 train_loss=10.2170 valid_loss=10.2704 stale=0 time=3.84m eta=42.4m [2024-06-22 12:34:18,809] INFO: Initiating epoch #390 train run on device rank=0 [2024-06-22 12:37:25,039] INFO: Initiating epoch #390 valid run on device rank=0 [2024-06-22 12:38:13,000] INFO: Rank 0: epoch=390 / 400 train_loss=10.2168 valid_loss=10.2702 stale=0 time=3.9m eta=38.6m [2024-06-22 12:38:13,037] INFO: Initiating epoch #391 train run on device rank=0 [2024-06-22 12:41:16,479] INFO: Initiating epoch #391 valid run on device rank=0 [2024-06-22 12:42:03,776] INFO: Rank 0: epoch=391 / 400 train_loss=10.2167 valid_loss=10.2700 stale=0 time=3.85m eta=34.7m [2024-06-22 12:42:03,828] INFO: Initiating epoch #392 train run on device rank=0 [2024-06-22 12:45:07,421] INFO: Initiating epoch #392 valid run on device rank=0 [2024-06-22 12:45:57,338] INFO: Rank 0: epoch=392 / 400 train_loss=10.2165 valid_loss=10.2698 stale=0 time=3.89m eta=30.9m [2024-06-22 12:45:57,365] INFO: Initiating epoch #393 train run on device rank=0 [2024-06-22 12:48:59,729] INFO: Initiating epoch #393 valid run on device rank=0 [2024-06-22 12:49:48,138] INFO: Rank 0: epoch=393 / 400 train_loss=10.2163 valid_loss=10.2696 stale=0 time=3.85m eta=27.0m [2024-06-22 12:49:48,182] INFO: Initiating epoch #394 train run on device rank=0 [2024-06-22 12:52:52,032] INFO: Initiating epoch #394 valid run on device rank=0 [2024-06-22 12:53:40,221] INFO: Rank 0: epoch=394 / 400 train_loss=10.2161 valid_loss=10.2694 stale=0 time=3.87m eta=23.1m [2024-06-22 12:53:40,325] INFO: Initiating epoch #395 train run on device rank=0 [2024-06-22 12:56:42,270] INFO: Initiating epoch #395 valid run on device rank=0 [2024-06-22 12:57:31,050] INFO: Rank 0: epoch=395 / 400 train_loss=10.2159 valid_loss=10.2692 stale=0 time=3.85m eta=19.3m [2024-06-22 12:57:31,118] INFO: Initiating epoch #396 train run on device rank=0 [2024-06-22 13:00:34,247] INFO: Initiating epoch #396 valid run on device rank=0 [2024-06-22 13:01:22,573] INFO: Rank 0: epoch=396 / 400 train_loss=10.2157 valid_loss=10.2690 stale=0 time=3.86m eta=15.4m [2024-06-22 13:01:22,597] INFO: Initiating epoch #397 train run on device rank=0 [2024-06-22 13:04:28,656] INFO: Initiating epoch #397 valid run on device rank=0 [2024-06-22 13:05:16,154] INFO: Rank 0: epoch=397 / 400 train_loss=10.2155 valid_loss=10.2688 stale=0 time=3.89m eta=11.6m [2024-06-22 13:05:16,203] INFO: Initiating epoch #398 train run on device rank=0 [2024-06-22 13:08:24,261] INFO: Initiating epoch #398 valid run on device rank=0 [2024-06-22 13:09:12,016] INFO: Rank 0: epoch=398 / 400 train_loss=10.2153 valid_loss=10.2686 stale=0 time=3.93m eta=7.7m [2024-06-22 13:09:12,064] INFO: Initiating epoch #399 train run on device rank=0 [2024-06-22 13:12:17,387] INFO: Initiating epoch #399 valid run on device rank=0 [2024-06-22 13:13:04,098] INFO: Rank 0: epoch=399 / 400 train_loss=10.2151 valid_loss=10.2684 stale=0 time=3.87m eta=3.9m [2024-06-22 13:13:04,123] INFO: Initiating epoch #400 train run on device rank=0 [2024-06-22 13:16:08,383] INFO: Initiating epoch #400 valid run on device rank=0 [2024-06-22 13:16:56,208] INFO: Rank 0: epoch=400 / 400 train_loss=10.2149 valid_loss=10.2682 stale=0 time=3.87m eta=0.0m [2024-06-22 13:16:56,211] INFO: Done with training. Total training time on device 0 is 1543.187min