[2024-06-24 14:25:55,030] INFO: Will use torch.nn.parallel.DistributedDataParallel() and 4 gpus [2024-06-24 14:25:55,109] INFO: NVIDIA GeForce GTX 1080 Ti [2024-06-24 14:25:55,109] INFO: NVIDIA GeForce GTX 1080 Ti [2024-06-24 14:25:55,109] INFO: NVIDIA GeForce GTX 1080 Ti [2024-06-24 14:25:55,109] INFO: NVIDIA GeForce GTX 1080 Ti [2024-06-24 14:26:01,804] INFO: using dtype=torch.float32 [2024-06-24 14:26:02,785] INFO: using attention_type=math [2024-06-24 14:26:02,802] INFO: using attention_type=math [2024-06-24 14:26:02,819] INFO: using attention_type=math [2024-06-24 14:26:02,836] INFO: using attention_type=math [2024-06-24 14:26:02,853] INFO: using attention_type=math [2024-06-24 14:26:02,870] INFO: using attention_type=math [2024-06-24 14:26:07,432] INFO: mlpf_kwargs: {'input_dim': 17, 'num_classes': 6, 'input_encoding': 'joint', 'pt_mode': 'linear', 'eta_mode': 'linear', 'sin_phi_mode': 'linear', 'cos_phi_mode': 'linear', 'energy_mode': 'linear', 'elemtypes_nonzero': [1, 2], 'learned_representation_mode': 'last', 'conv_type': 'attention', 'num_convs': 3, 'dropout_ff': 0.0, 'dropout_conv_id_mha': 0.0, 'dropout_conv_id_ff': 0.0, 'dropout_conv_reg_mha': 0.0, 'dropout_conv_reg_ff': 0.0, 'activation': 'relu', 'head_dim': 16, 'num_heads': 32, 'attention_type': 'math'} [2024-06-24 14:26:07,432] INFO: Loaded model weights from /pfvol/experiments/MLPF_clic_backbone_pyg-clic_20240429_101112_971749/best_weights.pth [2024-06-24 14:26:08,494] INFO: DistributedDataParallel( (module): DeepMET( (nn): Sequential( (0): Linear(in_features=6, out_features=256, bias=True) (1): ELU(alpha=1.0) (2): LayerNorm((256,), eps=1e-05, elementwise_affine=True) (3): Dropout(p=0, inplace=False) (4): Linear(in_features=256, out_features=2, bias=True) ) ) ) [2024-06-24 14:26:08,495] INFO: DeepMET Trainable parameters: 2818 [2024-06-24 14:26:08,495] INFO: DeepMET Non-trainable parameters: 0 [2024-06-24 14:26:08,495] INFO: DeepMET Total parameters: 2818 [2024-06-24 14:26:08,497] INFO: Modules Trainable parameters Non-tranable parameters module.nn.0.weight 1536 0 module.nn.0.bias 256 0 module.nn.2.weight 256 0 module.nn.2.bias 256 0 module.nn.4.weight 512 0 module.nn.4.bias 2 0 [2024-06-24 14:26:08,507] INFO: Creating experiment dir /pfvol/experiments/MLPF_clic_backbone_pyg-clic_20240429_101112_971749/June24_PFCands_20240624_142554_905524 [2024-06-24 14:26:08,507] INFO: Model directory /pfvol/experiments/MLPF_clic_backbone_pyg-clic_20240429_101112_971749/June24_PFCands_20240624_142554_905524 [2024-06-24 14:26:08,545] INFO: train_dataset: clic_edm_ttbar_pf, 800800 [2024-06-24 14:26:08,651] INFO: valid_dataset: clic_edm_ttbar_pf, 200200 [2024-06-24 14:26:08,708] INFO: Initiating epoch #1 train run on device rank=0 [2024-06-24 14:29:07,604] INFO: Initiating epoch #1 valid run on device rank=0 [2024-06-24 14:29:54,209] INFO: Rank 0: epoch=1 / 400 train_loss=10.8913 valid_loss=10.7524 stale=0 time=3.76m eta=1499.6m [2024-06-24 14:29:54,214] INFO: Initiating epoch #2 train run on device rank=0 [2024-06-24 14:32:54,064] INFO: Initiating epoch #2 valid run on device rank=0 [2024-06-24 14:33:41,330] INFO: Rank 0: epoch=2 / 400 train_loss=10.6676 valid_loss=10.6995 stale=0 time=3.79m eta=1501.2m [2024-06-24 14:33:41,365] INFO: Initiating epoch #3 train run on device rank=0 [2024-06-24 14:36:42,622] INFO: Initiating epoch #3 valid run on device rank=0 [2024-06-24 14:37:28,355] INFO: Rank 0: epoch=3 / 400 train_loss=10.6196 valid_loss=10.6509 stale=0 time=3.78m eta=1499.0m [2024-06-24 14:37:28,409] INFO: Initiating epoch #4 train run on device rank=0 [2024-06-24 14:40:26,124] INFO: Initiating epoch #4 valid run on device rank=0 [2024-06-24 14:41:13,017] INFO: Rank 0: epoch=4 / 400 train_loss=10.5676 valid_loss=10.5951 stale=0 time=3.74m eta=1492.1m [2024-06-24 14:41:13,111] INFO: Initiating epoch #5 train run on device rank=0 [2024-06-24 14:44:12,464] INFO: Initiating epoch #5 valid run on device rank=0 [2024-06-24 14:44:58,001] INFO: Rank 0: epoch=5 / 400 train_loss=10.5156 valid_loss=10.5461 stale=0 time=3.75m eta=1486.9m [2024-06-24 14:44:58,034] INFO: Initiating epoch #6 train run on device rank=0 [2024-06-24 14:47:55,114] INFO: Initiating epoch #6 valid run on device rank=0 [2024-06-24 14:48:44,188] INFO: Rank 0: epoch=6 / 400 train_loss=10.4738 valid_loss=10.5069 stale=0 time=3.77m eta=1483.5m [2024-06-24 14:48:44,217] INFO: Initiating epoch #7 train run on device rank=0 [2024-06-24 14:51:40,729] INFO: Initiating epoch #7 valid run on device rank=0 [2024-06-24 14:52:28,101] INFO: Rank 0: epoch=7 / 400 train_loss=10.4423 valid_loss=10.4799 stale=0 time=3.73m eta=1477.9m [2024-06-24 14:52:28,127] INFO: Initiating epoch #8 train run on device rank=0 [2024-06-24 14:55:27,843] INFO: Initiating epoch #8 valid run on device rank=0 [2024-06-24 14:56:14,877] INFO: Rank 0: epoch=8 / 400 train_loss=10.4207 valid_loss=10.4607 stale=0 time=3.78m eta=1475.0m [2024-06-24 14:56:14,954] INFO: Initiating epoch #9 train run on device rank=0 [2024-06-24 14:59:06,874] INFO: Initiating epoch #9 valid run on device rank=0 [2024-06-24 14:59:54,007] INFO: Rank 0: epoch=9 / 400 train_loss=10.4061 valid_loss=10.4457 stale=0 time=3.65m eta=1466.5m [2024-06-24 14:59:54,048] INFO: Initiating epoch #10 train run on device rank=0 [2024-06-24 15:02:51,760] INFO: Initiating epoch #10 valid run on device rank=0 [2024-06-24 15:03:36,906] INFO: Rank 0: epoch=10 / 400 train_loss=10.3955 valid_loss=10.4346 stale=0 time=3.71m eta=1461.3m [2024-06-24 15:03:36,928] INFO: Initiating epoch #11 train run on device rank=0 [2024-06-24 15:06:32,392] INFO: Initiating epoch #11 valid run on device rank=0 [2024-06-24 15:07:20,099] INFO: Rank 0: epoch=11 / 400 train_loss=10.3871 valid_loss=10.4258 stale=0 time=3.72m eta=1456.6m [2024-06-24 15:07:20,142] INFO: Initiating epoch #12 train run on device rank=0 [2024-06-24 15:10:20,853] INFO: Initiating epoch #12 valid run on device rank=0 [2024-06-24 15:11:09,071] INFO: Rank 0: epoch=12 / 400 train_loss=10.3797 valid_loss=10.4178 stale=0 time=3.82m eta=1455.2m [2024-06-24 15:11:09,151] INFO: Initiating epoch #13 train run on device rank=0 [2024-06-24 15:14:07,090] INFO: Initiating epoch #13 valid run on device rank=0 [2024-06-24 15:14:54,026] INFO: Rank 0: epoch=13 / 400 train_loss=10.3729 valid_loss=10.4103 stale=0 time=3.75m eta=1451.4m [2024-06-24 15:14:54,047] INFO: Initiating epoch #14 train run on device rank=0 [2024-06-24 15:17:55,072] INFO: Initiating epoch #14 valid run on device rank=0 [2024-06-24 15:18:41,180] INFO: Rank 0: epoch=14 / 400 train_loss=10.3663 valid_loss=10.4031 stale=0 time=3.79m eta=1448.6m [2024-06-24 15:18:41,258] INFO: Initiating epoch #15 train run on device rank=0 [2024-06-24 15:21:37,029] INFO: Initiating epoch #15 valid run on device rank=0 [2024-06-24 15:22:24,084] INFO: Rank 0: epoch=15 / 400 train_loss=10.3599 valid_loss=10.3961 stale=0 time=3.71m eta=1443.9m [2024-06-24 15:22:24,268] INFO: Initiating epoch #16 train run on device rank=0 [2024-06-24 15:25:20,924] INFO: Initiating epoch #16 valid run on device rank=0 [2024-06-24 15:26:07,646] INFO: Rank 0: epoch=16 / 400 train_loss=10.3536 valid_loss=10.3892 stale=0 time=3.72m eta=1439.6m [2024-06-24 15:26:07,660] INFO: Initiating epoch #17 train run on device rank=0 [2024-06-24 15:28:59,133] INFO: Initiating epoch #17 valid run on device rank=0 [2024-06-24 15:29:46,809] INFO: Rank 0: epoch=17 / 400 train_loss=10.3474 valid_loss=10.3825 stale=0 time=3.65m eta=1433.7m [2024-06-24 15:29:46,873] INFO: Initiating epoch #18 train run on device rank=0 [2024-06-24 15:32:40,522] INFO: Initiating epoch #18 valid run on device rank=0 [2024-06-24 15:33:28,127] INFO: Rank 0: epoch=18 / 400 train_loss=10.3415 valid_loss=10.3757 stale=0 time=3.69m eta=1428.8m [2024-06-24 15:33:28,178] INFO: Initiating epoch #19 train run on device rank=0 [2024-06-24 15:36:24,253] INFO: Initiating epoch #19 valid run on device rank=0 [2024-06-24 15:37:11,469] INFO: Rank 0: epoch=19 / 400 train_loss=10.3357 valid_loss=10.3688 stale=0 time=3.72m eta=1424.7m [2024-06-24 15:37:11,524] INFO: Initiating epoch #20 train run on device rank=0 [2024-06-24 15:40:10,403] INFO: Initiating epoch #20 valid run on device rank=0 [2024-06-24 15:40:55,336] INFO: Rank 0: epoch=20 / 400 train_loss=10.3299 valid_loss=10.3620 stale=0 time=3.73m eta=1420.8m [2024-06-24 15:40:55,365] INFO: Initiating epoch #21 train run on device rank=0 [2024-06-24 15:43:52,034] INFO: Initiating epoch #21 valid run on device rank=0 [2024-06-24 15:44:38,896] INFO: Rank 0: epoch=21 / 400 train_loss=10.3243 valid_loss=10.3551 stale=0 time=3.73m eta=1416.8m [2024-06-24 15:44:38,909] INFO: Initiating epoch #22 train run on device rank=0 [2024-06-24 15:47:34,998] INFO: Initiating epoch #22 valid run on device rank=0 [2024-06-24 15:48:21,906] INFO: Rank 0: epoch=22 / 400 train_loss=10.3187 valid_loss=10.3484 stale=0 time=3.72m eta=1412.7m [2024-06-24 15:48:21,989] INFO: Initiating epoch #23 train run on device rank=0 [2024-06-24 15:51:21,348] INFO: Initiating epoch #23 valid run on device rank=0 [2024-06-24 15:52:08,089] INFO: Rank 0: epoch=23 / 400 train_loss=10.3131 valid_loss=10.3417 stale=0 time=3.77m eta=1409.5m [2024-06-24 15:52:08,159] INFO: Initiating epoch #24 train run on device rank=0 [2024-06-24 15:55:00,123] INFO: Initiating epoch #24 valid run on device rank=0 [2024-06-24 15:55:47,488] INFO: Rank 0: epoch=24 / 400 train_loss=10.3077 valid_loss=10.3356 stale=0 time=3.66m eta=1404.5m [2024-06-24 15:55:47,524] INFO: Initiating epoch #25 train run on device rank=0 [2024-06-24 15:58:43,411] INFO: Initiating epoch #25 valid run on device rank=0 [2024-06-24 15:59:30,564] INFO: Rank 0: epoch=25 / 400 train_loss=10.3026 valid_loss=10.3299 stale=0 time=3.72m eta=1400.5m [2024-06-24 15:59:30,601] INFO: Initiating epoch #26 train run on device rank=0 [2024-06-24 16:02:26,309] INFO: Initiating epoch #26 valid run on device rank=0 [2024-06-24 16:03:12,253] INFO: Rank 0: epoch=26 / 400 train_loss=10.2975 valid_loss=10.3244 stale=0 time=3.69m eta=1396.2m [2024-06-24 16:03:12,379] INFO: Initiating epoch #27 train run on device rank=0 [2024-06-24 16:06:13,177] INFO: Initiating epoch #27 valid run on device rank=0 [2024-06-24 16:07:00,250] INFO: Rank 0: epoch=27 / 400 train_loss=10.2927 valid_loss=10.3192 stale=0 time=3.8m eta=1393.3m [2024-06-24 16:07:00,310] INFO: Initiating epoch #28 train run on device rank=0 [2024-06-24 16:09:54,063] INFO: Initiating epoch #28 valid run on device rank=0 [2024-06-24 16:10:39,935] INFO: Rank 0: epoch=28 / 400 train_loss=10.2880 valid_loss=10.3143 stale=0 time=3.66m eta=1388.6m [2024-06-24 16:10:39,967] INFO: Initiating epoch #29 train run on device rank=0 [2024-06-24 16:13:35,020] INFO: Initiating epoch #29 valid run on device rank=0 [2024-06-24 16:14:22,978] INFO: Rank 0: epoch=29 / 400 train_loss=10.2836 valid_loss=10.3098 stale=0 time=3.72m eta=1384.7m [2024-06-24 16:14:23,199] INFO: Initiating epoch #30 train run on device rank=0 [2024-06-24 16:17:20,920] INFO: Initiating epoch #30 valid run on device rank=0 [2024-06-24 16:18:08,849] INFO: Rank 0: epoch=30 / 400 train_loss=10.2793 valid_loss=10.3056 stale=0 time=3.76m eta=1381.4m [2024-06-24 16:18:08,893] INFO: Initiating epoch #31 train run on device rank=0 [2024-06-24 16:21:02,593] INFO: Initiating epoch #31 valid run on device rank=0 [2024-06-24 16:21:48,859] INFO: Rank 0: epoch=31 / 400 train_loss=10.2753 valid_loss=10.3017 stale=0 time=3.67m eta=1376.8m [2024-06-24 16:21:48,912] INFO: Initiating epoch #32 train run on device rank=0 [2024-06-24 16:24:48,619] INFO: Initiating epoch #32 valid run on device rank=0 [2024-06-24 16:25:37,458] INFO: Rank 0: epoch=32 / 400 train_loss=10.2714 valid_loss=10.2981 stale=0 time=3.81m eta=1374.0m [2024-06-24 16:25:37,542] INFO: Initiating epoch #33 train run on device rank=0 [2024-06-24 16:28:35,466] INFO: Initiating epoch #33 valid run on device rank=0 [2024-06-24 16:29:21,537] INFO: Rank 0: epoch=33 / 400 train_loss=10.2677 valid_loss=10.2948 stale=0 time=3.73m eta=1370.3m [2024-06-24 16:29:21,564] INFO: Initiating epoch #34 train run on device rank=0 [2024-06-24 16:32:22,893] INFO: Initiating epoch #34 valid run on device rank=0 [2024-06-24 16:33:10,831] INFO: Rank 0: epoch=34 / 400 train_loss=10.2641 valid_loss=10.2915 stale=0 time=3.82m eta=1367.5m [2024-06-24 16:33:10,883] INFO: Initiating epoch #35 train run on device rank=0 [2024-06-24 16:36:07,131] INFO: Initiating epoch #35 valid run on device rank=0 [2024-06-24 16:36:54,024] INFO: Rank 0: epoch=35 / 400 train_loss=10.2607 valid_loss=10.2885 stale=0 time=3.72m eta=1363.6m [2024-06-24 16:36:54,077] INFO: Initiating epoch #36 train run on device rank=0 [2024-06-24 16:39:55,163] INFO: Initiating epoch #36 valid run on device rank=0 [2024-06-24 16:40:43,409] INFO: Rank 0: epoch=36 / 400 train_loss=10.2574 valid_loss=10.2858 stale=0 time=3.82m eta=1360.7m [2024-06-24 16:40:43,441] INFO: Initiating epoch #37 train run on device rank=0 [2024-06-24 16:43:42,554] INFO: Initiating epoch #37 valid run on device rank=0 [2024-06-24 16:44:29,285] INFO: Rank 0: epoch=37 / 400 train_loss=10.2542 valid_loss=10.2832 stale=0 time=3.76m eta=1357.3m [2024-06-24 16:44:29,315] INFO: Initiating epoch #38 train run on device rank=0 [2024-06-24 16:47:33,555] INFO: Initiating epoch #38 valid run on device rank=0 [2024-06-24 16:48:21,118] INFO: Rank 0: epoch=38 / 400 train_loss=10.2511 valid_loss=10.2805 stale=0 time=3.86m eta=1354.7m [2024-06-24 16:48:21,172] INFO: Initiating epoch #39 train run on device rank=0 [2024-06-24 16:51:17,920] INFO: Initiating epoch #39 valid run on device rank=0 [2024-06-24 16:52:04,856] INFO: Rank 0: epoch=39 / 400 train_loss=10.2480 valid_loss=10.2782 stale=0 time=3.73m eta=1350.8m [2024-06-24 16:52:04,909] INFO: Initiating epoch #40 train run on device rank=0 [2024-06-24 16:55:02,049] INFO: Initiating epoch #40 valid run on device rank=0 [2024-06-24 16:55:50,083] INFO: Rank 0: epoch=40 / 400 train_loss=10.2452 valid_loss=10.2760 stale=0 time=3.75m eta=1347.2m [2024-06-24 16:55:50,544] INFO: Initiating epoch #41 train run on device rank=0 [2024-06-24 16:58:50,881] INFO: Initiating epoch #41 valid run on device rank=0 [2024-06-24 16:59:38,428] INFO: Rank 0: epoch=41 / 400 train_loss=10.2424 valid_loss=10.2739 stale=0 time=3.8m eta=1344.0m [2024-06-24 16:59:38,684] INFO: Initiating epoch #42 train run on device rank=0 [2024-06-24 17:02:37,535] INFO: Initiating epoch #42 valid run on device rank=0 [2024-06-24 17:03:23,991] INFO: Rank 0: epoch=42 / 400 train_loss=10.2396 valid_loss=10.2715 stale=0 time=3.76m eta=1340.4m [2024-06-24 17:03:24,110] INFO: Initiating epoch #43 train run on device rank=0 [2024-06-24 17:06:19,755] INFO: Initiating epoch #43 valid run on device rank=0 [2024-06-24 17:07:07,341] INFO: Rank 0: epoch=43 / 400 train_loss=10.2368 valid_loss=10.2694 stale=0 time=3.72m eta=1336.5m [2024-06-24 17:07:07,370] INFO: Initiating epoch #44 train run on device rank=0 [2024-06-24 17:10:04,804] INFO: Initiating epoch #44 valid run on device rank=0 [2024-06-24 17:10:51,875] INFO: Rank 0: epoch=44 / 400 train_loss=10.2342 valid_loss=10.2674 stale=0 time=3.74m eta=1332.7m [2024-06-24 17:10:51,946] INFO: Initiating epoch #45 train run on device rank=0 [2024-06-24 17:13:47,984] INFO: Initiating epoch #45 valid run on device rank=0 [2024-06-24 17:14:34,184] INFO: Rank 0: epoch=45 / 400 train_loss=10.2318 valid_loss=10.2655 stale=0 time=3.7m eta=1328.7m [2024-06-24 17:14:34,240] INFO: Initiating epoch #46 train run on device rank=0 [2024-06-24 17:17:33,369] INFO: Initiating epoch #46 valid run on device rank=0 [2024-06-24 17:18:20,794] INFO: Rank 0: epoch=46 / 400 train_loss=10.2296 valid_loss=10.2636 stale=0 time=3.78m eta=1325.2m [2024-06-24 17:18:20,846] INFO: Initiating epoch #47 train run on device rank=0 [2024-06-24 17:21:16,151] INFO: Initiating epoch #47 valid run on device rank=0 [2024-06-24 17:22:03,798] INFO: Rank 0: epoch=47 / 400 train_loss=10.2274 valid_loss=10.2618 stale=0 time=3.72m eta=1321.3m [2024-06-24 17:22:03,824] INFO: Initiating epoch #48 train run on device rank=0 [2024-06-24 17:25:06,160] INFO: Initiating epoch #48 valid run on device rank=0 [2024-06-24 17:25:54,140] INFO: Rank 0: epoch=48 / 400 train_loss=10.2253 valid_loss=10.2600 stale=0 time=3.84m eta=1318.2m [2024-06-24 17:25:54,276] INFO: Initiating epoch #49 train run on device rank=0 [2024-06-24 17:28:50,190] INFO: Initiating epoch #49 valid run on device rank=0 [2024-06-24 17:29:36,205] INFO: Rank 0: epoch=49 / 400 train_loss=10.2231 valid_loss=10.2582 stale=0 time=3.7m eta=1314.2m [2024-06-24 17:29:36,219] INFO: Initiating epoch #50 train run on device rank=0 [2024-06-24 17:32:35,148] INFO: Initiating epoch #50 valid run on device rank=0 [2024-06-24 17:33:22,946] INFO: Rank 0: epoch=50 / 400 train_loss=10.2208 valid_loss=10.2562 stale=0 time=3.78m eta=1310.7m [2024-06-24 17:33:22,973] INFO: Initiating epoch #51 train run on device rank=0 [2024-06-24 17:36:16,877] INFO: Initiating epoch #51 valid run on device rank=0 [2024-06-24 17:37:04,398] INFO: Rank 0: epoch=51 / 400 train_loss=10.2186 valid_loss=10.2543 stale=0 time=3.69m eta=1306.5m [2024-06-24 17:37:04,493] INFO: Initiating epoch #52 train run on device rank=0 [2024-06-24 17:40:04,583] INFO: Initiating epoch #52 valid run on device rank=0 [2024-06-24 17:40:52,264] INFO: Rank 0: epoch=52 / 400 train_loss=10.2165 valid_loss=10.2524 stale=0 time=3.8m eta=1303.2m [2024-06-24 17:40:52,358] INFO: Initiating epoch #53 train run on device rank=0 [2024-06-24 17:43:49,681] INFO: Initiating epoch #53 valid run on device rank=0 [2024-06-24 17:44:37,100] INFO: Rank 0: epoch=53 / 400 train_loss=10.2145 valid_loss=10.2501 stale=0 time=3.75m eta=1299.4m [2024-06-24 17:44:37,130] INFO: Initiating epoch #54 train run on device rank=0 [2024-06-24 17:47:34,042] INFO: Initiating epoch #54 valid run on device rank=0 [2024-06-24 17:48:21,649] INFO: Rank 0: epoch=54 / 400 train_loss=10.2120 valid_loss=10.2475 stale=0 time=3.74m eta=1295.7m [2024-06-24 17:48:21,722] INFO: Initiating epoch #55 train run on device rank=0 [2024-06-24 17:51:15,703] INFO: Initiating epoch #55 valid run on device rank=0 [2024-06-24 17:52:02,373] INFO: Rank 0: epoch=55 / 400 train_loss=10.2098 valid_loss=10.2453 stale=0 time=3.68m eta=1291.5m [2024-06-24 17:52:02,395] INFO: Initiating epoch #56 train run on device rank=0 [2024-06-24 17:55:01,829] INFO: Initiating epoch #56 valid run on device rank=0 [2024-06-24 17:55:49,133] INFO: Rank 0: epoch=56 / 400 train_loss=10.2076 valid_loss=10.2429 stale=0 time=3.78m eta=1288.0m [2024-06-24 17:55:49,156] INFO: Initiating epoch #57 train run on device rank=0 [2024-06-24 17:58:50,814] INFO: Initiating epoch #57 valid run on device rank=0 [2024-06-24 17:59:35,831] INFO: Rank 0: epoch=57 / 400 train_loss=10.2054 valid_loss=10.2403 stale=0 time=3.78m eta=1284.5m [2024-06-24 17:59:35,855] INFO: Initiating epoch #58 train run on device rank=0 [2024-06-24 18:02:29,401] INFO: Initiating epoch #58 valid run on device rank=0 [2024-06-24 18:03:17,583] INFO: Rank 0: epoch=58 / 400 train_loss=10.2032 valid_loss=10.2377 stale=0 time=3.7m eta=1280.4m [2024-06-24 18:03:17,661] INFO: Initiating epoch #59 train run on device rank=0 [2024-06-24 18:06:11,993] INFO: Initiating epoch #59 valid run on device rank=0 [2024-06-24 18:06:59,086] INFO: Rank 0: epoch=59 / 400 train_loss=10.2009 valid_loss=10.2353 stale=0 time=3.69m eta=1276.4m [2024-06-24 18:06:59,111] INFO: Initiating epoch #60 train run on device rank=0 [2024-06-24 18:09:53,184] INFO: Initiating epoch #60 valid run on device rank=0 [2024-06-24 18:10:40,569] INFO: Rank 0: epoch=60 / 400 train_loss=10.1986 valid_loss=10.2326 stale=0 time=3.69m eta=1272.3m [2024-06-24 18:10:40,599] INFO: Initiating epoch #61 train run on device rank=0 [2024-06-24 18:13:40,569] INFO: Initiating epoch #61 valid run on device rank=0 [2024-06-24 18:14:28,466] INFO: Rank 0: epoch=61 / 400 train_loss=10.1957 valid_loss=10.2287 stale=0 time=3.8m eta=1268.9m [2024-06-24 18:14:28,761] INFO: Initiating epoch #62 train run on device rank=0 [2024-06-24 18:17:23,964] INFO: Initiating epoch #62 valid run on device rank=0 [2024-06-24 18:18:11,344] INFO: Rank 0: epoch=62 / 400 train_loss=10.1918 valid_loss=10.2260 stale=0 time=3.71m eta=1265.0m [2024-06-24 18:18:11,381] INFO: Initiating epoch #63 train run on device rank=0 [2024-06-24 18:21:06,932] INFO: Initiating epoch #63 valid run on device rank=0 [2024-06-24 18:21:53,145] INFO: Rank 0: epoch=63 / 400 train_loss=10.1882 valid_loss=10.2285 stale=1 time=3.7m eta=1261.0m [2024-06-24 18:21:53,198] INFO: Initiating epoch #64 train run on device rank=0 [2024-06-24 18:24:47,844] INFO: Initiating epoch #64 valid run on device rank=0 [2024-06-24 18:25:32,953] INFO: Rank 0: epoch=64 / 400 train_loss=10.1855 valid_loss=10.2273 stale=2 time=3.66m eta=1256.9m [2024-06-24 18:25:32,982] INFO: Initiating epoch #65 train run on device rank=0 [2024-06-24 18:28:29,188] INFO: Initiating epoch #65 valid run on device rank=0 [2024-06-24 18:29:17,021] INFO: Rank 0: epoch=65 / 400 train_loss=10.1832 valid_loss=10.2177 stale=0 time=3.73m eta=1253.1m [2024-06-24 18:29:17,050] INFO: Initiating epoch #66 train run on device rank=0 [2024-06-24 18:32:15,613] INFO: Initiating epoch #66 valid run on device rank=0 [2024-06-24 18:33:04,362] INFO: Rank 0: epoch=66 / 400 train_loss=10.1813 valid_loss=10.2177 stale=0 time=3.79m eta=1249.6m [2024-06-24 18:33:04,508] INFO: Initiating epoch #67 train run on device rank=0 [2024-06-24 18:36:02,731] INFO: Initiating epoch #67 valid run on device rank=0 [2024-06-24 18:36:50,344] INFO: Rank 0: epoch=67 / 400 train_loss=10.1805 valid_loss=10.2173 stale=0 time=3.76m eta=1246.0m [2024-06-24 18:36:50,384] INFO: Initiating epoch #68 train run on device rank=0 [2024-06-24 18:39:48,310] INFO: Initiating epoch #68 valid run on device rank=0 [2024-06-24 18:40:36,243] INFO: Rank 0: epoch=68 / 400 train_loss=10.1797 valid_loss=10.2166 stale=0 time=3.76m eta=1242.4m [2024-06-24 18:40:36,477] INFO: Initiating epoch #69 train run on device rank=0 [2024-06-24 18:43:31,392] INFO: Initiating epoch #69 valid run on device rank=0 [2024-06-24 18:44:18,831] INFO: Rank 0: epoch=69 / 400 train_loss=10.1789 valid_loss=10.2161 stale=0 time=3.71m eta=1238.5m [2024-06-24 18:44:19,022] INFO: Initiating epoch #70 train run on device rank=0 [2024-06-24 18:47:23,277] INFO: Initiating epoch #70 valid run on device rank=0 [2024-06-24 18:48:08,810] INFO: Rank 0: epoch=70 / 400 train_loss=10.1783 valid_loss=10.2159 stale=0 time=3.83m eta=1235.2m [2024-06-24 18:48:08,851] INFO: Initiating epoch #71 train run on device rank=0 [2024-06-24 18:51:01,708] INFO: Initiating epoch #71 valid run on device rank=0 [2024-06-24 18:51:48,584] INFO: Rank 0: epoch=71 / 400 train_loss=10.1776 valid_loss=10.2157 stale=0 time=3.66m eta=1231.0m [2024-06-24 18:51:48,629] INFO: Initiating epoch #72 train run on device rank=0 [2024-06-24 18:54:42,398] INFO: Initiating epoch #72 valid run on device rank=0 [2024-06-24 18:55:28,030] INFO: Rank 0: epoch=72 / 400 train_loss=10.1769 valid_loss=10.2156 stale=0 time=3.66m eta=1226.9m [2024-06-24 18:55:28,272] INFO: Initiating epoch #73 train run on device rank=0 [2024-06-24 18:58:26,724] INFO: Initiating epoch #73 valid run on device rank=0 [2024-06-24 18:59:12,469] INFO: Rank 0: epoch=73 / 400 train_loss=10.1759 valid_loss=10.2174 stale=1 time=3.74m eta=1223.2m [2024-06-24 18:59:12,518] INFO: Initiating epoch #74 train run on device rank=0 [2024-06-24 19:02:08,200] INFO: Initiating epoch #74 valid run on device rank=0 [2024-06-24 19:02:53,482] INFO: Rank 0: epoch=74 / 400 train_loss=10.1751 valid_loss=10.2147 stale=0 time=3.68m eta=1219.2m [2024-06-24 19:02:53,516] INFO: Initiating epoch #75 train run on device rank=0 [2024-06-24 19:05:48,443] INFO: Initiating epoch #75 valid run on device rank=0 [2024-06-24 19:06:34,936] INFO: Rank 0: epoch=75 / 400 train_loss=10.1743 valid_loss=10.2137 stale=0 time=3.69m eta=1215.2m [2024-06-24 19:06:35,006] INFO: Initiating epoch #76 train run on device rank=0 [2024-06-24 19:09:36,178] INFO: Initiating epoch #76 valid run on device rank=0 [2024-06-24 19:10:23,608] INFO: Rank 0: epoch=76 / 400 train_loss=10.1734 valid_loss=10.2079 stale=0 time=3.81m eta=1211.8m [2024-06-24 19:10:23,656] INFO: Initiating epoch #77 train run on device rank=0 [2024-06-24 19:13:17,482] INFO: Initiating epoch #77 valid run on device rank=0 [2024-06-24 19:14:05,996] INFO: Rank 0: epoch=77 / 400 train_loss=10.1728 valid_loss=10.2074 stale=0 time=3.71m eta=1207.9m [2024-06-24 19:14:06,020] INFO: Initiating epoch #78 train run on device rank=0 [2024-06-24 19:17:03,128] INFO: Initiating epoch #78 valid run on device rank=0 [2024-06-24 19:17:48,795] INFO: Rank 0: epoch=78 / 400 train_loss=10.1720 valid_loss=10.2066 stale=0 time=3.71m eta=1204.1m [2024-06-24 19:17:48,825] INFO: Initiating epoch #79 train run on device rank=0 [2024-06-24 19:20:46,205] INFO: Initiating epoch #79 valid run on device rank=0 [2024-06-24 19:21:31,621] INFO: Rank 0: epoch=79 / 400 train_loss=10.1712 valid_loss=10.2037 stale=0 time=3.71m eta=1200.2m [2024-06-24 19:21:31,660] INFO: Initiating epoch #80 train run on device rank=0 [2024-06-24 19:24:26,761] INFO: Initiating epoch #80 valid run on device rank=0 [2024-06-24 19:25:13,861] INFO: Rank 0: epoch=80 / 400 train_loss=10.1704 valid_loss=10.2056 stale=1 time=3.7m eta=1196.3m [2024-06-24 19:25:13,898] INFO: Initiating epoch #81 train run on device rank=0 [2024-06-24 19:28:08,632] INFO: Initiating epoch #81 valid run on device rank=0 [2024-06-24 19:28:56,550] INFO: Rank 0: epoch=81 / 400 train_loss=10.1692 valid_loss=10.2035 stale=0 time=3.71m eta=1192.5m [2024-06-24 19:28:56,594] INFO: Initiating epoch #82 train run on device rank=0 [2024-06-24 19:31:52,700] INFO: Initiating epoch #82 valid run on device rank=0 [2024-06-24 19:32:40,385] INFO: Rank 0: epoch=82 / 400 train_loss=10.1678 valid_loss=10.2033 stale=0 time=3.73m eta=1188.7m [2024-06-24 19:32:40,405] INFO: Initiating epoch #83 train run on device rank=0 [2024-06-24 19:35:32,399] INFO: Initiating epoch #83 valid run on device rank=0 [2024-06-24 19:36:19,706] INFO: Rank 0: epoch=83 / 400 train_loss=10.1668 valid_loss=10.2066 stale=1 time=3.66m eta=1184.7m [2024-06-24 19:36:19,713] INFO: Initiating epoch #84 train run on device rank=0 [2024-06-24 19:39:20,423] INFO: Initiating epoch #84 valid run on device rank=0 [2024-06-24 19:40:05,586] INFO: Rank 0: epoch=84 / 400 train_loss=10.1656 valid_loss=10.2011 stale=0 time=3.76m eta=1181.0m [2024-06-24 19:40:05,632] INFO: Initiating epoch #85 train run on device rank=0 [2024-06-24 19:43:01,280] INFO: Initiating epoch #85 valid run on device rank=0 [2024-06-24 19:43:48,179] INFO: Rank 0: epoch=85 / 400 train_loss=10.1642 valid_loss=10.2090 stale=1 time=3.71m eta=1177.2m [2024-06-24 19:43:48,201] INFO: Initiating epoch #86 train run on device rank=0 [2024-06-24 19:46:49,599] INFO: Initiating epoch #86 valid run on device rank=0 [2024-06-24 19:47:36,689] INFO: Rank 0: epoch=86 / 400 train_loss=10.1628 valid_loss=10.2056 stale=2 time=3.81m eta=1173.7m [2024-06-24 19:47:36,785] INFO: Initiating epoch #87 train run on device rank=0 [2024-06-24 19:50:32,159] INFO: Initiating epoch #87 valid run on device rank=0 [2024-06-24 19:51:19,352] INFO: Rank 0: epoch=87 / 400 train_loss=10.1617 valid_loss=10.2004 stale=0 time=3.71m eta=1169.9m [2024-06-24 19:51:19,382] INFO: Initiating epoch #88 train run on device rank=0 [2024-06-24 19:54:19,110] INFO: Initiating epoch #88 valid run on device rank=0 [2024-06-24 19:55:05,251] INFO: Rank 0: epoch=88 / 400 train_loss=10.1604 valid_loss=10.1972 stale=0 time=3.76m eta=1166.3m [2024-06-24 19:55:05,310] INFO: Initiating epoch #89 train run on device rank=0 [2024-06-24 19:58:01,273] INFO: Initiating epoch #89 valid run on device rank=0 [2024-06-24 19:58:47,931] INFO: Rank 0: epoch=89 / 400 train_loss=10.1592 valid_loss=10.1954 stale=0 time=3.71m eta=1162.4m [2024-06-24 19:58:47,961] INFO: Initiating epoch #90 train run on device rank=0 [2024-06-24 20:01:45,050] INFO: Initiating epoch #90 valid run on device rank=0 [2024-06-24 20:02:31,891] INFO: Rank 0: epoch=90 / 400 train_loss=10.1577 valid_loss=10.1916 stale=0 time=3.73m eta=1158.7m [2024-06-24 20:02:31,929] INFO: Initiating epoch #91 train run on device rank=0 [2024-06-24 20:05:33,363] INFO: Initiating epoch #91 valid run on device rank=0 [2024-06-24 20:06:19,940] INFO: Rank 0: epoch=91 / 400 train_loss=10.1562 valid_loss=10.1891 stale=0 time=3.8m eta=1155.1m [2024-06-24 20:06:19,987] INFO: Initiating epoch #92 train run on device rank=0 [2024-06-24 20:09:21,352] INFO: Initiating epoch #92 valid run on device rank=0 [2024-06-24 20:10:08,828] INFO: Rank 0: epoch=92 / 400 train_loss=10.1544 valid_loss=10.1851 stale=0 time=3.81m eta=1151.7m [2024-06-24 20:10:08,843] INFO: Initiating epoch #93 train run on device rank=0 [2024-06-24 20:13:07,851] INFO: Initiating epoch #93 valid run on device rank=0 [2024-06-24 20:13:55,929] INFO: Rank 0: epoch=93 / 400 train_loss=10.1522 valid_loss=10.1827 stale=0 time=3.78m eta=1148.1m [2024-06-24 20:13:55,978] INFO: Initiating epoch #94 train run on device rank=0 [2024-06-24 20:16:53,017] INFO: Initiating epoch #94 valid run on device rank=0 [2024-06-24 20:17:38,977] INFO: Rank 0: epoch=94 / 400 train_loss=10.1495 valid_loss=10.1813 stale=0 time=3.72m eta=1144.3m [2024-06-24 20:17:39,115] INFO: Initiating epoch #95 train run on device rank=0 [2024-06-24 20:20:33,731] INFO: Initiating epoch #95 valid run on device rank=0 [2024-06-24 20:21:21,066] INFO: Rank 0: epoch=95 / 400 train_loss=10.1476 valid_loss=10.1798 stale=0 time=3.7m eta=1140.4m [2024-06-24 20:21:21,120] INFO: Initiating epoch #96 train run on device rank=0 [2024-06-24 20:24:20,235] INFO: Initiating epoch #96 valid run on device rank=0 [2024-06-24 20:25:07,663] INFO: Rank 0: epoch=96 / 400 train_loss=10.1459 valid_loss=10.1788 stale=0 time=3.78m eta=1136.8m [2024-06-24 20:25:07,811] INFO: Initiating epoch #97 train run on device rank=0 [2024-06-24 20:28:04,496] INFO: Initiating epoch #97 valid run on device rank=0 [2024-06-24 20:28:51,009] INFO: Rank 0: epoch=97 / 400 train_loss=10.1447 valid_loss=10.1781 stale=0 time=3.72m eta=1133.0m [2024-06-24 20:28:51,061] INFO: Initiating epoch #98 train run on device rank=0 [2024-06-24 20:31:49,465] INFO: Initiating epoch #98 valid run on device rank=0 [2024-06-24 20:32:37,002] INFO: Rank 0: epoch=98 / 400 train_loss=10.1433 valid_loss=10.1778 stale=0 time=3.77m eta=1129.3m [2024-06-24 20:32:37,019] INFO: Initiating epoch #99 train run on device rank=0 [2024-06-24 20:35:33,091] INFO: Initiating epoch #99 valid run on device rank=0 [2024-06-24 20:36:20,489] INFO: Rank 0: epoch=99 / 400 train_loss=10.1421 valid_loss=10.1773 stale=0 time=3.72m eta=1125.5m [2024-06-24 20:36:20,578] INFO: Initiating epoch #100 train run on device rank=0 [2024-06-24 20:39:17,366] INFO: Initiating epoch #100 valid run on device rank=0 [2024-06-24 20:40:04,315] INFO: Rank 0: epoch=100 / 400 train_loss=10.1413 valid_loss=10.1761 stale=0 time=3.73m eta=1121.8m [2024-06-24 20:40:04,371] INFO: Initiating epoch #101 train run on device rank=0 [2024-06-24 20:43:03,617] INFO: Initiating epoch #101 valid run on device rank=0 [2024-06-24 20:43:51,600] INFO: Rank 0: epoch=101 / 400 train_loss=10.1404 valid_loss=10.1753 stale=0 time=3.79m eta=1118.2m [2024-06-24 20:43:51,617] INFO: Initiating epoch #102 train run on device rank=0 [2024-06-24 20:46:51,784] INFO: Initiating epoch #102 valid run on device rank=0 [2024-06-24 20:47:38,321] INFO: Rank 0: epoch=102 / 400 train_loss=10.1396 valid_loss=10.1745 stale=0 time=3.78m eta=1114.6m [2024-06-24 20:47:38,352] INFO: Initiating epoch #103 train run on device rank=0 [2024-06-24 20:50:41,327] INFO: Initiating epoch #103 valid run on device rank=0 [2024-06-24 20:51:28,050] INFO: Rank 0: epoch=103 / 400 train_loss=10.1388 valid_loss=10.1739 stale=0 time=3.83m eta=1111.1m [2024-06-24 20:51:28,085] INFO: Initiating epoch #104 train run on device rank=0 [2024-06-24 20:54:30,291] INFO: Initiating epoch #104 valid run on device rank=0 [2024-06-24 20:55:17,229] INFO: Rank 0: epoch=104 / 400 train_loss=10.1380 valid_loss=10.1734 stale=0 time=3.82m eta=1107.6m [2024-06-24 20:55:17,317] INFO: Initiating epoch #105 train run on device rank=0 [2024-06-24 20:58:13,101] INFO: Initiating epoch #105 valid run on device rank=0 [2024-06-24 20:59:02,369] INFO: Rank 0: epoch=105 / 400 train_loss=10.1372 valid_loss=10.1730 stale=0 time=3.75m eta=1103.8m [2024-06-24 20:59:03,409] INFO: Initiating epoch #106 train run on device rank=0 [2024-06-24 21:01:56,642] INFO: Initiating epoch #106 valid run on device rank=0 [2024-06-24 21:02:42,566] INFO: Rank 0: epoch=106 / 400 train_loss=10.1365 valid_loss=10.1725 stale=0 time=3.65m eta=1099.9m [2024-06-24 21:02:42,597] INFO: Initiating epoch #107 train run on device rank=0 [2024-06-24 21:05:37,930] INFO: Initiating epoch #107 valid run on device rank=0 [2024-06-24 21:06:25,512] INFO: Rank 0: epoch=107 / 400 train_loss=10.1359 valid_loss=10.1706 stale=0 time=3.72m eta=1096.1m [2024-06-24 21:06:25,542] INFO: Initiating epoch #108 train run on device rank=0 [2024-06-24 21:09:22,003] INFO: Initiating epoch #108 valid run on device rank=0 [2024-06-24 21:10:08,709] INFO: Rank 0: epoch=108 / 400 train_loss=10.1354 valid_loss=10.1703 stale=0 time=3.72m eta=1092.3m [2024-06-24 21:10:08,729] INFO: Initiating epoch #109 train run on device rank=0 [2024-06-24 21:13:05,315] INFO: Initiating epoch #109 valid run on device rank=0 [2024-06-24 21:13:52,521] INFO: Rank 0: epoch=109 / 400 train_loss=10.1349 valid_loss=10.1692 stale=0 time=3.73m eta=1088.5m [2024-06-24 21:13:52,743] INFO: Initiating epoch #110 train run on device rank=0 [2024-06-24 21:16:54,068] INFO: Initiating epoch #110 valid run on device rank=0 [2024-06-24 21:17:41,671] INFO: Rank 0: epoch=110 / 400 train_loss=10.1345 valid_loss=10.1689 stale=0 time=3.82m eta=1085.0m [2024-06-24 21:17:41,713] INFO: Initiating epoch #111 train run on device rank=0 [2024-06-24 21:20:36,107] INFO: Initiating epoch #111 valid run on device rank=0 [2024-06-24 21:21:23,562] INFO: Rank 0: epoch=111 / 400 train_loss=10.1341 valid_loss=10.1686 stale=0 time=3.7m eta=1081.1m [2024-06-24 21:21:23,762] INFO: Initiating epoch #112 train run on device rank=0 [2024-06-24 21:24:16,272] INFO: Initiating epoch #112 valid run on device rank=0 [2024-06-24 21:25:04,514] INFO: Rank 0: epoch=112 / 400 train_loss=10.1337 valid_loss=10.1682 stale=0 time=3.68m eta=1077.2m [2024-06-24 21:25:04,690] INFO: Initiating epoch #113 train run on device rank=0 [2024-06-24 21:28:02,909] INFO: Initiating epoch #113 valid run on device rank=0 [2024-06-24 21:28:48,492] INFO: Rank 0: epoch=113 / 400 train_loss=10.1333 valid_loss=10.1678 stale=0 time=3.73m eta=1073.5m [2024-06-24 21:28:48,544] INFO: Initiating epoch #114 train run on device rank=0 [2024-06-24 21:31:43,902] INFO: Initiating epoch #114 valid run on device rank=0 [2024-06-24 21:32:32,509] INFO: Rank 0: epoch=114 / 400 train_loss=10.1329 valid_loss=10.1675 stale=0 time=3.73m eta=1069.7m [2024-06-24 21:32:32,556] INFO: Initiating epoch #115 train run on device rank=0 [2024-06-24 21:35:27,120] INFO: Initiating epoch #115 valid run on device rank=0 [2024-06-24 21:36:12,948] INFO: Rank 0: epoch=115 / 400 train_loss=10.1326 valid_loss=10.1675 stale=0 time=3.67m eta=1065.8m [2024-06-24 21:36:13,044] INFO: Initiating epoch #116 train run on device rank=0 [2024-06-24 21:39:07,133] INFO: Initiating epoch #116 valid run on device rank=0 [2024-06-24 21:39:53,550] INFO: Rank 0: epoch=116 / 400 train_loss=10.1322 valid_loss=10.1675 stale=1 time=3.68m eta=1061.9m [2024-06-24 21:39:53,577] INFO: Initiating epoch #117 train run on device rank=0 [2024-06-24 21:42:47,835] INFO: Initiating epoch #117 valid run on device rank=0 [2024-06-24 21:43:33,201] INFO: Rank 0: epoch=117 / 400 train_loss=10.1318 valid_loss=10.1675 stale=2 time=3.66m eta=1058.0m [2024-06-24 21:43:33,266] INFO: Initiating epoch #118 train run on device rank=0 [2024-06-24 21:46:31,263] INFO: Initiating epoch #118 valid run on device rank=0 [2024-06-24 21:47:17,284] INFO: Rank 0: epoch=118 / 400 train_loss=10.1316 valid_loss=10.1675 stale=0 time=3.73m eta=1054.3m [2024-06-24 21:47:17,335] INFO: Initiating epoch #119 train run on device rank=0 [2024-06-24 21:50:12,749] INFO: Initiating epoch #119 valid run on device rank=0 [2024-06-24 21:51:01,285] INFO: Rank 0: epoch=119 / 400 train_loss=10.1313 valid_loss=10.1674 stale=0 time=3.73m eta=1050.5m [2024-06-24 21:51:01,550] INFO: Initiating epoch #120 train run on device rank=0 [2024-06-24 21:53:54,655] INFO: Initiating epoch #120 valid run on device rank=0 [2024-06-24 21:54:42,729] INFO: Rank 0: epoch=120 / 400 train_loss=10.1311 valid_loss=10.1673 stale=0 time=3.69m eta=1046.7m [2024-06-24 21:54:42,814] INFO: Initiating epoch #121 train run on device rank=0 [2024-06-24 21:57:40,330] INFO: Initiating epoch #121 valid run on device rank=0 [2024-06-24 21:58:26,062] INFO: Rank 0: epoch=121 / 400 train_loss=10.1309 valid_loss=10.1671 stale=0 time=3.72m eta=1042.9m [2024-06-24 21:58:26,076] INFO: Initiating epoch #122 train run on device rank=0 [2024-06-24 22:01:22,790] INFO: Initiating epoch #122 valid run on device rank=0 [2024-06-24 22:02:09,634] INFO: Rank 0: epoch=122 / 400 train_loss=10.1307 valid_loss=10.1670 stale=0 time=3.73m eta=1039.1m [2024-06-24 22:02:09,830] INFO: Initiating epoch #123 train run on device rank=0 [2024-06-24 22:05:04,258] INFO: Initiating epoch #123 valid run on device rank=0 [2024-06-24 22:05:51,019] INFO: Rank 0: epoch=123 / 400 train_loss=10.1305 valid_loss=10.1669 stale=0 time=3.69m eta=1035.3m [2024-06-24 22:05:51,077] INFO: Initiating epoch #124 train run on device rank=0 [2024-06-24 22:08:47,315] INFO: Initiating epoch #124 valid run on device rank=0 [2024-06-24 22:09:34,140] INFO: Rank 0: epoch=124 / 400 train_loss=10.1304 valid_loss=10.1667 stale=0 time=3.72m eta=1031.5m [2024-06-24 22:09:34,167] INFO: Initiating epoch #125 train run on device rank=0 [2024-06-24 22:12:29,265] INFO: Initiating epoch #125 valid run on device rank=0 [2024-06-24 22:13:16,380] INFO: Rank 0: epoch=125 / 400 train_loss=10.1302 valid_loss=10.1666 stale=0 time=3.7m eta=1027.7m [2024-06-24 22:13:16,412] INFO: Initiating epoch #126 train run on device rank=0 [2024-06-24 22:16:16,443] INFO: Initiating epoch #126 valid run on device rank=0 [2024-06-24 22:17:01,832] INFO: Rank 0: epoch=126 / 400 train_loss=10.1301 valid_loss=10.1665 stale=0 time=3.76m eta=1024.0m [2024-06-24 22:17:01,899] INFO: Initiating epoch #127 train run on device rank=0 [2024-06-24 22:19:59,285] INFO: Initiating epoch #127 valid run on device rank=0 [2024-06-24 22:20:47,087] INFO: Rank 0: epoch=127 / 400 train_loss=10.1299 valid_loss=10.1664 stale=0 time=3.75m eta=1020.3m [2024-06-24 22:20:47,117] INFO: Initiating epoch #128 train run on device rank=0 [2024-06-24 22:23:43,692] INFO: Initiating epoch #128 valid run on device rank=0 [2024-06-24 22:24:32,199] INFO: Rank 0: epoch=128 / 400 train_loss=10.1298 valid_loss=10.1662 stale=0 time=3.75m eta=1016.6m [2024-06-24 22:24:32,710] INFO: Initiating epoch #129 train run on device rank=0 [2024-06-24 22:27:32,266] INFO: Initiating epoch #129 valid run on device rank=0 [2024-06-24 22:28:20,255] INFO: Rank 0: epoch=129 / 400 train_loss=10.1297 valid_loss=10.1661 stale=0 time=3.79m eta=1013.0m [2024-06-24 22:28:20,290] INFO: Initiating epoch #130 train run on device rank=0 [2024-06-24 22:31:15,513] INFO: Initiating epoch #130 valid run on device rank=0 [2024-06-24 22:32:02,777] INFO: Rank 0: epoch=130 / 400 train_loss=10.1296 valid_loss=10.1660 stale=0 time=3.71m eta=1009.2m [2024-06-24 22:32:02,841] INFO: Initiating epoch #131 train run on device rank=0 [2024-06-24 22:35:02,832] INFO: Initiating epoch #131 valid run on device rank=0 [2024-06-24 22:35:50,581] INFO: Rank 0: epoch=131 / 400 train_loss=10.1295 valid_loss=10.1659 stale=0 time=3.8m eta=1005.6m [2024-06-24 22:35:50,630] INFO: Initiating epoch #132 train run on device rank=0 [2024-06-24 22:38:46,382] INFO: Initiating epoch #132 valid run on device rank=0 [2024-06-24 22:39:35,101] INFO: Rank 0: epoch=132 / 400 train_loss=10.1294 valid_loss=10.1658 stale=0 time=3.74m eta=1001.8m [2024-06-24 22:39:35,347] INFO: Initiating epoch #133 train run on device rank=0 [2024-06-24 22:42:40,066] INFO: Initiating epoch #133 valid run on device rank=0 [2024-06-24 22:43:27,551] INFO: Rank 0: epoch=133 / 400 train_loss=10.1293 valid_loss=10.1657 stale=0 time=3.87m eta=998.4m [2024-06-24 22:43:27,688] INFO: Initiating epoch #134 train run on device rank=0 [2024-06-24 22:46:28,438] INFO: Initiating epoch #134 valid run on device rank=0 [2024-06-24 22:47:15,761] INFO: Rank 0: epoch=134 / 400 train_loss=10.1293 valid_loss=10.1656 stale=0 time=3.8m eta=994.8m [2024-06-24 22:47:15,904] INFO: Initiating epoch #135 train run on device rank=0 [2024-06-24 22:50:37,842] INFO: Initiating epoch #135 valid run on device rank=0 [2024-06-24 22:51:30,815] INFO: Rank 0: epoch=135 / 400 train_loss=10.1292 valid_loss=10.1655 stale=0 time=4.25m eta=992.0m [2024-06-24 22:51:30,847] INFO: Initiating epoch #136 train run on device rank=0 [2024-06-24 22:54:48,397] INFO: Initiating epoch #136 valid run on device rank=0 [2024-06-24 22:55:40,024] INFO: Rank 0: epoch=136 / 400 train_loss=10.1291 valid_loss=10.1655 stale=0 time=4.15m eta=989.1m [2024-06-24 22:55:40,072] INFO: Initiating epoch #137 train run on device rank=0 [2024-06-24 22:59:02,403] INFO: Initiating epoch #137 valid run on device rank=0 [2024-06-24 22:59:53,881] INFO: Rank 0: epoch=137 / 400 train_loss=10.1291 valid_loss=10.1654 stale=0 time=4.23m eta=986.3m [2024-06-24 22:59:53,951] INFO: Initiating epoch #138 train run on device rank=0 [2024-06-24 23:03:05,878] INFO: Initiating epoch #138 valid run on device rank=0 [2024-06-24 23:04:00,617] INFO: Rank 0: epoch=138 / 400 train_loss=10.1290 valid_loss=10.1653 stale=0 time=4.11m eta=983.2m [2024-06-24 23:04:00,680] INFO: Initiating epoch #139 train run on device rank=0 [2024-06-24 23:07:19,783] INFO: Initiating epoch #139 valid run on device rank=0 [2024-06-24 23:08:13,465] INFO: Rank 0: epoch=139 / 400 train_loss=10.1289 valid_loss=10.1652 stale=0 time=4.21m eta=980.3m [2024-06-24 23:08:13,518] INFO: Initiating epoch #140 train run on device rank=0 [2024-06-24 23:11:32,680] INFO: Initiating epoch #140 valid run on device rank=0 [2024-06-24 23:12:27,157] INFO: Rank 0: epoch=140 / 400 train_loss=10.1289 valid_loss=10.1652 stale=0 time=4.23m eta=977.4m [2024-06-24 23:12:27,265] INFO: Initiating epoch #141 train run on device rank=0 [2024-06-24 23:15:48,557] INFO: Initiating epoch #141 valid run on device rank=0 [2024-06-24 23:16:41,506] INFO: Rank 0: epoch=141 / 400 train_loss=10.1288 valid_loss=10.1651 stale=0 time=4.24m eta=974.6m [2024-06-24 23:16:41,558] INFO: Initiating epoch #142 train run on device rank=0 [2024-06-24 23:19:59,356] INFO: Initiating epoch #142 valid run on device rank=0 [2024-06-24 23:20:53,809] INFO: Rank 0: epoch=142 / 400 train_loss=10.1288 valid_loss=10.1650 stale=0 time=4.2m eta=971.6m [2024-06-24 23:20:54,087] INFO: Initiating epoch #143 train run on device rank=0 [2024-06-24 23:24:14,970] INFO: Initiating epoch #143 valid run on device rank=0 [2024-06-24 23:25:07,827] INFO: Rank 0: epoch=143 / 400 train_loss=10.1287 valid_loss=10.1650 stale=0 time=4.23m eta=968.7m [2024-06-24 23:25:07,879] INFO: Initiating epoch #144 train run on device rank=0 [2024-06-24 23:28:28,185] INFO: Initiating epoch #144 valid run on device rank=0 [2024-06-24 23:29:22,025] INFO: Rank 0: epoch=144 / 400 train_loss=10.1287 valid_loss=10.1649 stale=0 time=4.24m eta=965.7m [2024-06-24 23:29:22,063] INFO: Initiating epoch #145 train run on device rank=0 [2024-06-24 23:32:45,179] INFO: Initiating epoch #145 valid run on device rank=0 [2024-06-24 23:33:37,546] INFO: Rank 0: epoch=145 / 400 train_loss=10.1286 valid_loss=10.1648 stale=0 time=4.26m eta=962.8m [2024-06-24 23:33:37,582] INFO: Initiating epoch #146 train run on device rank=0 [2024-06-24 23:36:56,300] INFO: Initiating epoch #146 valid run on device rank=0 [2024-06-24 23:37:49,813] INFO: Rank 0: epoch=146 / 400 train_loss=10.1286 valid_loss=10.1648 stale=0 time=4.2m eta=959.8m [2024-06-24 23:37:49,843] INFO: Initiating epoch #147 train run on device rank=0 [2024-06-24 23:41:11,964] INFO: Initiating epoch #147 valid run on device rank=0 [2024-06-24 23:42:03,328] INFO: Rank 0: epoch=147 / 400 train_loss=10.1285 valid_loss=10.1647 stale=0 time=4.22m eta=956.8m [2024-06-24 23:42:03,420] INFO: Initiating epoch #148 train run on device rank=0 [2024-06-24 23:45:25,998] INFO: Initiating epoch #148 valid run on device rank=0 [2024-06-24 23:46:17,132] INFO: Rank 0: epoch=148 / 400 train_loss=10.1285 valid_loss=10.1646 stale=0 time=4.23m eta=953.8m [2024-06-24 23:46:17,183] INFO: Initiating epoch #149 train run on device rank=0 [2024-06-24 23:49:38,771] INFO: Initiating epoch #149 valid run on device rank=0 [2024-06-24 23:50:30,843] INFO: Rank 0: epoch=149 / 400 train_loss=10.1284 valid_loss=10.1645 stale=0 time=4.23m eta=950.7m [2024-06-24 23:50:30,866] INFO: Initiating epoch #150 train run on device rank=0 [2024-06-24 23:53:47,624] INFO: Initiating epoch #150 valid run on device rank=0 [2024-06-24 23:54:38,640] INFO: Rank 0: epoch=150 / 400 train_loss=10.1284 valid_loss=10.1645 stale=0 time=4.13m eta=947.5m [2024-06-24 23:54:38,754] INFO: Initiating epoch #151 train run on device rank=0 [2024-06-24 23:57:58,650] INFO: Initiating epoch #151 valid run on device rank=0 [2024-06-24 23:58:49,224] INFO: Rank 0: epoch=151 / 400 train_loss=10.1283 valid_loss=10.1644 stale=0 time=4.17m eta=944.3m [2024-06-24 23:58:49,251] INFO: Initiating epoch #152 train run on device rank=0 [2024-06-25 00:02:06,766] INFO: Initiating epoch #152 valid run on device rank=0 [2024-06-25 00:02:58,848] INFO: Rank 0: epoch=152 / 400 train_loss=10.1282 valid_loss=10.1643 stale=0 time=4.16m eta=941.2m [2024-06-25 00:02:59,254] INFO: Initiating epoch #153 train run on device rank=0 [2024-06-25 00:06:16,208] INFO: Initiating epoch #153 valid run on device rank=0 [2024-06-25 00:07:10,728] INFO: Rank 0: epoch=153 / 400 train_loss=10.1282 valid_loss=10.1643 stale=0 time=4.19m eta=938.0m [2024-06-25 00:07:10,779] INFO: Initiating epoch #154 train run on device rank=0 [2024-06-25 00:10:26,491] INFO: Initiating epoch #154 valid run on device rank=0 [2024-06-25 00:11:17,993] INFO: Rank 0: epoch=154 / 400 train_loss=10.1281 valid_loss=10.1642 stale=0 time=4.12m eta=934.7m [2024-06-25 00:11:18,019] INFO: Initiating epoch #155 train run on device rank=0 [2024-06-25 00:14:37,603] INFO: Initiating epoch #155 valid run on device rank=0 [2024-06-25 00:15:28,718] INFO: Rank 0: epoch=155 / 400 train_loss=10.1280 valid_loss=10.1641 stale=0 time=4.18m eta=931.5m [2024-06-25 00:15:28,769] INFO: Initiating epoch #156 train run on device rank=0 [2024-06-25 00:18:46,775] INFO: Initiating epoch #156 valid run on device rank=0 [2024-06-25 00:19:38,727] INFO: Rank 0: epoch=156 / 400 train_loss=10.1280 valid_loss=10.1640 stale=0 time=4.17m eta=928.3m [2024-06-25 00:19:38,773] INFO: Initiating epoch #157 train run on device rank=0 [2024-06-25 00:22:56,632] INFO: Initiating epoch #157 valid run on device rank=0 [2024-06-25 00:23:49,184] INFO: Rank 0: epoch=157 / 400 train_loss=10.1279 valid_loss=10.1639 stale=0 time=4.17m eta=925.1m [2024-06-25 00:23:49,422] INFO: Initiating epoch #158 train run on device rank=0 [2024-06-25 00:27:01,851] INFO: Initiating epoch #158 valid run on device rank=0 [2024-06-25 00:27:52,381] INFO: Rank 0: epoch=158 / 400 train_loss=10.1279 valid_loss=10.1639 stale=0 time=4.05m eta=921.6m [2024-06-25 00:27:52,459] INFO: Initiating epoch #159 train run on device rank=0 [2024-06-25 00:31:18,355] INFO: Initiating epoch #159 valid run on device rank=0 [2024-06-25 00:32:11,086] INFO: Rank 0: epoch=159 / 400 train_loss=10.1278 valid_loss=10.1638 stale=0 time=4.31m eta=918.6m [2024-06-25 00:32:11,116] INFO: Initiating epoch #160 train run on device rank=0 [2024-06-25 00:35:27,019] INFO: Initiating epoch #160 valid run on device rank=0 [2024-06-25 00:36:18,613] INFO: Rank 0: epoch=160 / 400 train_loss=10.1278 valid_loss=10.1637 stale=0 time=4.12m eta=915.2m [2024-06-25 00:36:18,647] INFO: Initiating epoch #161 train run on device rank=0 [2024-06-25 00:39:34,398] INFO: Initiating epoch #161 valid run on device rank=0 [2024-06-25 00:40:27,420] INFO: Rank 0: epoch=161 / 400 train_loss=10.1277 valid_loss=10.1636 stale=0 time=4.15m eta=911.9m [2024-06-25 00:40:27,442] INFO: Initiating epoch #162 train run on device rank=0 [2024-06-25 00:43:47,054] INFO: Initiating epoch #162 valid run on device rank=0 [2024-06-25 00:44:38,969] INFO: Rank 0: epoch=162 / 400 train_loss=10.1276 valid_loss=10.1635 stale=0 time=4.19m eta=908.7m [2024-06-25 00:44:39,025] INFO: Initiating epoch #163 train run on device rank=0 [2024-06-25 00:47:54,937] INFO: Initiating epoch #163 valid run on device rank=0 [2024-06-25 00:48:48,288] INFO: Rank 0: epoch=163 / 400 train_loss=10.1276 valid_loss=10.1634 stale=0 time=4.15m eta=905.3m [2024-06-25 00:48:48,478] INFO: Initiating epoch #164 train run on device rank=0 [2024-06-25 00:52:10,146] INFO: Initiating epoch #164 valid run on device rank=0 [2024-06-25 00:53:00,999] INFO: Rank 0: epoch=164 / 400 train_loss=10.1275 valid_loss=10.1634 stale=0 time=4.21m eta=902.1m [2024-06-25 00:53:01,022] INFO: Initiating epoch #165 train run on device rank=0 [2024-06-25 00:56:21,618] INFO: Initiating epoch #165 valid run on device rank=0 [2024-06-25 00:57:12,938] INFO: Rank 0: epoch=165 / 400 train_loss=10.1274 valid_loss=10.1633 stale=0 time=4.2m eta=898.8m [2024-06-25 00:57:12,957] INFO: Initiating epoch #166 train run on device rank=0 [2024-06-25 01:00:28,512] INFO: Initiating epoch #166 valid run on device rank=0 [2024-06-25 01:01:19,809] INFO: Rank 0: epoch=166 / 400 train_loss=10.1274 valid_loss=10.1632 stale=0 time=4.11m eta=895.4m [2024-06-25 01:01:20,007] INFO: Initiating epoch #167 train run on device rank=0 [2024-06-25 01:04:33,483] INFO: Initiating epoch #167 valid run on device rank=0 [2024-06-25 01:05:27,390] INFO: Rank 0: epoch=167 / 400 train_loss=10.1273 valid_loss=10.1631 stale=0 time=4.12m eta=892.0m [2024-06-25 01:05:27,595] INFO: Initiating epoch #168 train run on device rank=0 [2024-06-25 01:08:42,795] INFO: Initiating epoch #168 valid run on device rank=0 [2024-06-25 01:09:33,577] INFO: Rank 0: epoch=168 / 400 train_loss=10.1273 valid_loss=10.1630 stale=0 time=4.1m eta=888.5m [2024-06-25 01:09:33,606] INFO: Initiating epoch #169 train run on device rank=0 [2024-06-25 01:12:46,563] INFO: Initiating epoch #169 valid run on device rank=0 [2024-06-25 01:13:40,230] INFO: Rank 0: epoch=169 / 400 train_loss=10.1272 valid_loss=10.1629 stale=0 time=4.11m eta=885.1m [2024-06-25 01:13:40,276] INFO: Initiating epoch #170 train run on device rank=0 [2024-06-25 01:16:56,685] INFO: Initiating epoch #170 valid run on device rank=0 [2024-06-25 01:17:50,031] INFO: Rank 0: epoch=170 / 400 train_loss=10.1271 valid_loss=10.1628 stale=0 time=4.16m eta=881.7m [2024-06-25 01:17:50,109] INFO: Initiating epoch #171 train run on device rank=0 [2024-06-25 01:21:05,689] INFO: Initiating epoch #171 valid run on device rank=0 [2024-06-25 01:21:59,528] INFO: Rank 0: epoch=171 / 400 train_loss=10.1271 valid_loss=10.1627 stale=0 time=4.16m eta=878.3m [2024-06-25 01:21:59,597] INFO: Initiating epoch #172 train run on device rank=0 [2024-06-25 01:25:14,321] INFO: Initiating epoch #172 valid run on device rank=0 [2024-06-25 01:26:05,112] INFO: Rank 0: epoch=172 / 400 train_loss=10.1270 valid_loss=10.1626 stale=0 time=4.09m eta=874.8m [2024-06-25 01:26:05,158] INFO: Initiating epoch #173 train run on device rank=0 [2024-06-25 01:29:18,887] INFO: Initiating epoch #173 valid run on device rank=0 [2024-06-25 01:30:09,471] INFO: Rank 0: epoch=173 / 400 train_loss=10.1269 valid_loss=10.1625 stale=0 time=4.07m eta=871.3m [2024-06-25 01:30:09,564] INFO: Initiating epoch #174 train run on device rank=0 [2024-06-25 01:33:29,788] INFO: Initiating epoch #174 valid run on device rank=0 [2024-06-25 01:34:21,533] INFO: Rank 0: epoch=174 / 400 train_loss=10.1269 valid_loss=10.1624 stale=0 time=4.2m eta=867.9m [2024-06-25 01:34:21,560] INFO: Initiating epoch #175 train run on device rank=0 [2024-06-25 01:37:40,212] INFO: Initiating epoch #175 valid run on device rank=0 [2024-06-25 01:38:31,555] INFO: Rank 0: epoch=175 / 400 train_loss=10.1268 valid_loss=10.1623 stale=0 time=4.17m eta=864.5m [2024-06-25 01:38:31,576] INFO: Initiating epoch #176 train run on device rank=0 [2024-06-25 01:41:54,800] INFO: Initiating epoch #176 valid run on device rank=0 [2024-06-25 01:42:46,016] INFO: Rank 0: epoch=176 / 400 train_loss=10.1268 valid_loss=10.1623 stale=0 time=4.24m eta=861.2m [2024-06-25 01:42:46,141] INFO: Initiating epoch #177 train run on device rank=0 [2024-06-25 01:46:00,728] INFO: Initiating epoch #177 valid run on device rank=0 [2024-06-25 01:46:53,854] INFO: Rank 0: epoch=177 / 400 train_loss=10.1267 valid_loss=10.1622 stale=0 time=4.13m eta=857.7m [2024-06-25 01:46:53,893] INFO: Initiating epoch #178 train run on device rank=0 [2024-06-25 01:50:11,533] INFO: Initiating epoch #178 valid run on device rank=0 [2024-06-25 01:51:04,478] INFO: Rank 0: epoch=178 / 400 train_loss=10.1267 valid_loss=10.1621 stale=0 time=4.18m eta=854.2m [2024-06-25 01:51:04,576] INFO: Initiating epoch #179 train run on device rank=0 [2024-06-25 01:54:19,782] INFO: Initiating epoch #179 valid run on device rank=0 [2024-06-25 01:55:12,842] INFO: Rank 0: epoch=179 / 400 train_loss=10.1266 valid_loss=10.1620 stale=0 time=4.14m eta=850.7m [2024-06-25 01:55:12,888] INFO: Initiating epoch #180 train run on device rank=0 [2024-06-25 01:58:29,543] INFO: Initiating epoch #180 valid run on device rank=0 [2024-06-25 01:59:22,136] INFO: Rank 0: epoch=180 / 400 train_loss=10.1266 valid_loss=10.1620 stale=0 time=4.15m eta=847.3m [2024-06-25 01:59:22,179] INFO: Initiating epoch #181 train run on device rank=0 [2024-06-25 02:02:40,000] INFO: Initiating epoch #181 valid run on device rank=0 [2024-06-25 02:03:31,743] INFO: Rank 0: epoch=181 / 400 train_loss=10.1265 valid_loss=10.1619 stale=0 time=4.16m eta=843.8m [2024-06-25 02:03:31,803] INFO: Initiating epoch #182 train run on device rank=0 [2024-06-25 02:06:50,044] INFO: Initiating epoch #182 valid run on device rank=0 [2024-06-25 02:07:41,484] INFO: Rank 0: epoch=182 / 400 train_loss=10.1265 valid_loss=10.1619 stale=0 time=4.16m eta=840.3m [2024-06-25 02:07:41,523] INFO: Initiating epoch #183 train run on device rank=0 [2024-06-25 02:10:58,100] INFO: Initiating epoch #183 valid run on device rank=0 [2024-06-25 02:11:49,590] INFO: Rank 0: epoch=183 / 400 train_loss=10.1264 valid_loss=10.1618 stale=0 time=4.13m eta=836.8m [2024-06-25 02:11:49,636] INFO: Initiating epoch #184 train run on device rank=0 [2024-06-25 02:15:08,700] INFO: Initiating epoch #184 valid run on device rank=0 [2024-06-25 02:16:01,399] INFO: Rank 0: epoch=184 / 400 train_loss=10.1264 valid_loss=10.1618 stale=0 time=4.2m eta=833.3m [2024-06-25 02:16:01,548] INFO: Initiating epoch #185 train run on device rank=0 [2024-06-25 02:19:17,633] INFO: Initiating epoch #185 valid run on device rank=0 [2024-06-25 02:20:08,912] INFO: Rank 0: epoch=185 / 400 train_loss=10.1264 valid_loss=10.1617 stale=0 time=4.12m eta=829.8m [2024-06-25 02:20:08,939] INFO: Initiating epoch #186 train run on device rank=0 [2024-06-25 02:23:30,317] INFO: Initiating epoch #186 valid run on device rank=0 [2024-06-25 02:24:22,612] INFO: Rank 0: epoch=186 / 400 train_loss=10.1263 valid_loss=10.1617 stale=0 time=4.23m eta=826.4m [2024-06-25 02:24:22,831] INFO: Initiating epoch #187 train run on device rank=0 [2024-06-25 02:27:34,953] INFO: Initiating epoch #187 valid run on device rank=0 [2024-06-25 02:28:25,967] INFO: Rank 0: epoch=187 / 400 train_loss=10.1263 valid_loss=10.1616 stale=0 time=4.05m eta=822.7m [2024-06-25 02:28:26,018] INFO: Initiating epoch #188 train run on device rank=0 [2024-06-25 02:31:44,425] INFO: Initiating epoch #188 valid run on device rank=0 [2024-06-25 02:32:35,928] INFO: Rank 0: epoch=188 / 400 train_loss=10.1263 valid_loss=10.1616 stale=0 time=4.17m eta=819.2m [2024-06-25 02:32:36,037] INFO: Initiating epoch #189 train run on device rank=0 [2024-06-25 02:35:53,590] INFO: Initiating epoch #189 valid run on device rank=0 [2024-06-25 02:36:45,656] INFO: Rank 0: epoch=189 / 400 train_loss=10.1262 valid_loss=10.1616 stale=0 time=4.16m eta=815.7m [2024-06-25 02:36:45,675] INFO: Initiating epoch #190 train run on device rank=0 [2024-06-25 02:39:58,196] INFO: Initiating epoch #190 valid run on device rank=0 [2024-06-25 02:40:49,153] INFO: Rank 0: epoch=190 / 400 train_loss=10.1262 valid_loss=10.1615 stale=0 time=4.06m eta=812.0m [2024-06-25 02:40:49,269] INFO: Initiating epoch #191 train run on device rank=0 [2024-06-25 02:44:08,403] INFO: Initiating epoch #191 valid run on device rank=0 [2024-06-25 02:45:01,571] INFO: Rank 0: epoch=191 / 400 train_loss=10.1262 valid_loss=10.1615 stale=0 time=4.21m eta=808.5m [2024-06-25 02:45:01,621] INFO: Initiating epoch #192 train run on device rank=0 [2024-06-25 02:48:24,888] INFO: Initiating epoch #192 valid run on device rank=0 [2024-06-25 02:49:17,579] INFO: Rank 0: epoch=192 / 400 train_loss=10.1262 valid_loss=10.1614 stale=0 time=4.27m eta=805.1m [2024-06-25 02:49:17,641] INFO: Initiating epoch #193 train run on device rank=0 [2024-06-25 02:52:35,267] INFO: Initiating epoch #193 valid run on device rank=0 [2024-06-25 02:53:28,721] INFO: Rank 0: epoch=193 / 400 train_loss=10.1261 valid_loss=10.1614 stale=0 time=4.18m eta=801.5m [2024-06-25 02:53:28,779] INFO: Initiating epoch #194 train run on device rank=0 [2024-06-25 02:56:39,432] INFO: Initiating epoch #194 valid run on device rank=0 [2024-06-25 02:57:32,917] INFO: Rank 0: epoch=194 / 400 train_loss=10.1261 valid_loss=10.1613 stale=0 time=4.07m eta=797.9m [2024-06-25 02:57:32,955] INFO: Initiating epoch #195 train run on device rank=0 [2024-06-25 03:00:46,601] INFO: Initiating epoch #195 valid run on device rank=0 [2024-06-25 03:01:40,345] INFO: Rank 0: epoch=195 / 400 train_loss=10.1261 valid_loss=10.1613 stale=0 time=4.12m eta=794.3m [2024-06-25 03:01:40,534] INFO: Initiating epoch #196 train run on device rank=0 [2024-06-25 03:04:57,324] INFO: Initiating epoch #196 valid run on device rank=0 [2024-06-25 03:05:51,875] INFO: Rank 0: epoch=196 / 400 train_loss=10.1261 valid_loss=10.1612 stale=0 time=4.19m eta=790.7m [2024-06-25 03:05:51,921] INFO: Initiating epoch #197 train run on device rank=0 [2024-06-25 03:09:07,433] INFO: Initiating epoch #197 valid run on device rank=0 [2024-06-25 03:10:01,441] INFO: Rank 0: epoch=197 / 400 train_loss=10.1260 valid_loss=10.1612 stale=0 time=4.16m eta=787.1m [2024-06-25 03:10:01,479] INFO: Initiating epoch #198 train run on device rank=0 [2024-06-25 03:13:14,644] INFO: Initiating epoch #198 valid run on device rank=0 [2024-06-25 03:14:07,325] INFO: Rank 0: epoch=198 / 400 train_loss=10.1260 valid_loss=10.1611 stale=0 time=4.1m eta=783.5m [2024-06-25 03:14:07,359] INFO: Initiating epoch #199 train run on device rank=0 [2024-06-25 03:17:30,005] INFO: Initiating epoch #199 valid run on device rank=0 [2024-06-25 03:18:23,328] INFO: Rank 0: epoch=199 / 400 train_loss=10.1260 valid_loss=10.1611 stale=1 time=4.27m eta=780.0m [2024-06-25 03:18:23,345] INFO: Initiating epoch #200 train run on device rank=0 [2024-06-25 03:21:38,131] INFO: Initiating epoch #200 valid run on device rank=0 [2024-06-25 03:22:33,248] INFO: Rank 0: epoch=200 / 400 train_loss=10.1260 valid_loss=10.1611 stale=0 time=4.17m eta=776.4m [2024-06-25 03:22:33,612] INFO: Initiating epoch #201 train run on device rank=0 [2024-06-25 03:25:53,467] INFO: Initiating epoch #201 valid run on device rank=0 [2024-06-25 03:26:44,864] INFO: Rank 0: epoch=201 / 400 train_loss=10.1260 valid_loss=10.1611 stale=1 time=4.19m eta=772.8m [2024-06-25 03:26:44,968] INFO: Initiating epoch #202 train run on device rank=0 [2024-06-25 03:30:04,832] INFO: Initiating epoch #202 valid run on device rank=0 [2024-06-25 03:30:57,349] INFO: Rank 0: epoch=202 / 400 train_loss=10.1260 valid_loss=10.1611 stale=2 time=4.21m eta=769.3m [2024-06-25 03:30:57,683] INFO: Initiating epoch #203 train run on device rank=0 [2024-06-25 03:34:13,382] INFO: Initiating epoch #203 valid run on device rank=0 [2024-06-25 03:35:06,424] INFO: Rank 0: epoch=203 / 400 train_loss=10.1259 valid_loss=10.1611 stale=3 time=4.15m eta=765.6m [2024-06-25 03:35:06,460] INFO: Initiating epoch #204 train run on device rank=0 [2024-06-25 03:38:26,825] INFO: Initiating epoch #204 valid run on device rank=0 [2024-06-25 03:39:20,790] INFO: Rank 0: epoch=204 / 400 train_loss=10.1259 valid_loss=10.1611 stale=4 time=4.24m eta=762.1m [2024-06-25 03:39:20,842] INFO: Initiating epoch #205 train run on device rank=0 [2024-06-25 03:42:44,035] INFO: Initiating epoch #205 valid run on device rank=0 [2024-06-25 03:43:36,414] INFO: Rank 0: epoch=205 / 400 train_loss=10.1259 valid_loss=10.1611 stale=5 time=4.26m eta=758.6m [2024-06-25 03:43:36,531] INFO: Initiating epoch #206 train run on device rank=0 [2024-06-25 03:46:48,904] INFO: Initiating epoch #206 valid run on device rank=0 [2024-06-25 03:47:42,525] INFO: Rank 0: epoch=206 / 400 train_loss=10.1259 valid_loss=10.1610 stale=0 time=4.1m eta=754.9m [2024-06-25 03:47:42,557] INFO: Initiating epoch #207 train run on device rank=0 [2024-06-25 03:50:57,730] INFO: Initiating epoch #207 valid run on device rank=0 [2024-06-25 03:51:51,974] INFO: Rank 0: epoch=207 / 400 train_loss=10.1259 valid_loss=10.1610 stale=0 time=4.16m eta=751.2m [2024-06-25 03:51:52,088] INFO: Initiating epoch #208 train run on device rank=0 [2024-06-25 03:55:07,314] INFO: Initiating epoch #208 valid run on device rank=0 [2024-06-25 03:56:01,286] INFO: Rank 0: epoch=208 / 400 train_loss=10.1259 valid_loss=10.1609 stale=0 time=4.15m eta=747.6m [2024-06-25 03:56:01,323] INFO: Initiating epoch #209 train run on device rank=0 [2024-06-25 03:59:19,958] INFO: Initiating epoch #209 valid run on device rank=0 [2024-06-25 04:00:14,693] INFO: Rank 0: epoch=209 / 400 train_loss=10.1258 valid_loss=10.1608 stale=0 time=4.22m eta=744.0m [2024-06-25 04:00:14,721] INFO: Initiating epoch #210 train run on device rank=0 [2024-06-25 04:03:33,745] INFO: Initiating epoch #210 valid run on device rank=0 [2024-06-25 04:04:27,115] INFO: Rank 0: epoch=210 / 400 train_loss=10.1258 valid_loss=10.1607 stale=0 time=4.21m eta=740.4m [2024-06-25 04:04:27,177] INFO: Initiating epoch #211 train run on device rank=0 [2024-06-25 04:07:42,044] INFO: Initiating epoch #211 valid run on device rank=0 [2024-06-25 04:08:35,509] INFO: Rank 0: epoch=211 / 400 train_loss=10.1258 valid_loss=10.1606 stale=0 time=4.14m eta=736.7m [2024-06-25 04:08:35,525] INFO: Initiating epoch #212 train run on device rank=0 [2024-06-25 04:11:49,925] INFO: Initiating epoch #212 valid run on device rank=0 [2024-06-25 04:12:44,043] INFO: Rank 0: epoch=212 / 400 train_loss=10.1258 valid_loss=10.1606 stale=0 time=4.14m eta=733.0m [2024-06-25 04:12:44,129] INFO: Initiating epoch #213 train run on device rank=0 [2024-06-25 04:15:59,028] INFO: Initiating epoch #213 valid run on device rank=0 [2024-06-25 04:16:53,745] INFO: Rank 0: epoch=213 / 400 train_loss=10.1257 valid_loss=10.1606 stale=1 time=4.16m eta=729.3m [2024-06-25 04:16:53,800] INFO: Initiating epoch #214 train run on device rank=0 [2024-06-25 04:20:09,418] INFO: Initiating epoch #214 valid run on device rank=0 [2024-06-25 04:21:02,780] INFO: Rank 0: epoch=214 / 400 train_loss=10.1257 valid_loss=10.1608 stale=2 time=4.15m eta=725.7m [2024-06-25 04:21:02,809] INFO: Initiating epoch #215 train run on device rank=0 [2024-06-25 04:24:22,216] INFO: Initiating epoch #215 valid run on device rank=0 [2024-06-25 04:25:15,781] INFO: Rank 0: epoch=215 / 400 train_loss=10.1255 valid_loss=10.1609 stale=3 time=4.22m eta=722.0m [2024-06-25 04:25:15,890] INFO: Initiating epoch #216 train run on device rank=0 [2024-06-25 04:28:34,194] INFO: Initiating epoch #216 valid run on device rank=0 [2024-06-25 04:29:27,268] INFO: Rank 0: epoch=216 / 400 train_loss=10.1254 valid_loss=10.1609 stale=4 time=4.19m eta=718.4m [2024-06-25 04:29:27,299] INFO: Initiating epoch #217 train run on device rank=0 [2024-06-25 04:32:41,034] INFO: Initiating epoch #217 valid run on device rank=0 [2024-06-25 04:33:33,457] INFO: Rank 0: epoch=217 / 400 train_loss=10.1253 valid_loss=10.1608 stale=5 time=4.1m eta=714.6m [2024-06-25 04:33:33,496] INFO: Initiating epoch #218 train run on device rank=0 [2024-06-25 04:36:51,306] INFO: Initiating epoch #218 valid run on device rank=0 [2024-06-25 04:37:43,897] INFO: Rank 0: epoch=218 / 400 train_loss=10.1253 valid_loss=10.1608 stale=6 time=4.17m eta=711.0m [2024-06-25 04:37:43,986] INFO: Initiating epoch #219 train run on device rank=0 [2024-06-25 04:41:09,080] INFO: Initiating epoch #219 valid run on device rank=0 [2024-06-25 04:42:00,707] INFO: Rank 0: epoch=219 / 400 train_loss=10.1252 valid_loss=10.1608 stale=7 time=4.28m eta=707.4m [2024-06-25 04:42:00,822] INFO: Initiating epoch #220 train run on device rank=0 [2024-06-25 04:45:20,625] INFO: Initiating epoch #220 valid run on device rank=0 [2024-06-25 04:46:12,459] INFO: Rank 0: epoch=220 / 400 train_loss=10.1252 valid_loss=10.1607 stale=8 time=4.19m eta=703.7m [2024-06-25 04:46:12,563] INFO: Initiating epoch #221 train run on device rank=0 [2024-06-25 04:49:28,208] INFO: Initiating epoch #221 valid run on device rank=0 [2024-06-25 04:50:20,999] INFO: Rank 0: epoch=221 / 400 train_loss=10.1252 valid_loss=10.1607 stale=9 time=4.14m eta=700.0m [2024-06-25 04:50:21,034] INFO: Initiating epoch #222 train run on device rank=0 [2024-06-25 04:53:37,664] INFO: Initiating epoch #222 valid run on device rank=0 [2024-06-25 04:54:30,936] INFO: Rank 0: epoch=222 / 400 train_loss=10.1252 valid_loss=10.1607 stale=10 time=4.17m eta=696.3m [2024-06-25 04:54:30,968] INFO: Initiating epoch #223 train run on device rank=0 [2024-06-25 04:57:44,405] INFO: Initiating epoch #223 valid run on device rank=0 [2024-06-25 04:58:36,574] INFO: Rank 0: epoch=223 / 400 train_loss=10.1251 valid_loss=10.1607 stale=11 time=4.09m eta=692.5m [2024-06-25 04:58:36,596] INFO: Initiating epoch #224 train run on device rank=0 [2024-06-25 05:01:50,746] INFO: Initiating epoch #224 valid run on device rank=0 [2024-06-25 05:02:42,835] INFO: Rank 0: epoch=224 / 400 train_loss=10.1251 valid_loss=10.1606 stale=12 time=4.1m eta=688.7m [2024-06-25 05:02:42,883] INFO: Initiating epoch #225 train run on device rank=0 [2024-06-25 05:05:59,176] INFO: Initiating epoch #225 valid run on device rank=0 [2024-06-25 05:06:52,073] INFO: Rank 0: epoch=225 / 400 train_loss=10.1251 valid_loss=10.1606 stale=13 time=4.15m eta=685.0m [2024-06-25 05:06:52,102] INFO: Initiating epoch #226 train run on device rank=0 [2024-06-25 05:10:11,045] INFO: Initiating epoch #226 valid run on device rank=0 [2024-06-25 05:11:02,082] INFO: Rank 0: epoch=226 / 400 train_loss=10.1251 valid_loss=10.1606 stale=14 time=4.17m eta=681.3m [2024-06-25 05:11:02,146] INFO: Initiating epoch #227 train run on device rank=0 [2024-06-25 05:14:16,988] INFO: Initiating epoch #227 valid run on device rank=0 [2024-06-25 05:15:09,657] INFO: Rank 0: epoch=227 / 400 train_loss=10.1251 valid_loss=10.1606 stale=15 time=4.13m eta=677.5m [2024-06-25 05:15:09,704] INFO: Initiating epoch #228 train run on device rank=0 [2024-06-25 05:18:26,987] INFO: Initiating epoch #228 valid run on device rank=0 [2024-06-25 05:19:19,747] INFO: Rank 0: epoch=228 / 400 train_loss=10.1251 valid_loss=10.1606 stale=16 time=4.17m eta=673.8m [2024-06-25 05:19:19,766] INFO: Initiating epoch #229 train run on device rank=0 [2024-06-25 05:22:35,053] INFO: Initiating epoch #229 valid run on device rank=0 [2024-06-25 05:23:28,182] INFO: Rank 0: epoch=229 / 400 train_loss=10.1250 valid_loss=10.1606 stale=17 time=4.14m eta=670.1m [2024-06-25 05:23:28,311] INFO: Initiating epoch #230 train run on device rank=0 [2024-06-25 05:26:47,307] INFO: Initiating epoch #230 valid run on device rank=0 [2024-06-25 05:27:40,869] INFO: Rank 0: epoch=230 / 400 train_loss=10.1250 valid_loss=10.1605 stale=0 time=4.21m eta=666.4m [2024-06-25 05:27:40,906] INFO: Initiating epoch #231 train run on device rank=0 [2024-06-25 05:30:59,202] INFO: Initiating epoch #231 valid run on device rank=0 [2024-06-25 05:31:52,211] INFO: Rank 0: epoch=231 / 400 train_loss=10.1250 valid_loss=10.1605 stale=0 time=4.19m eta=662.6m [2024-06-25 05:31:52,241] INFO: Initiating epoch #232 train run on device rank=0 [2024-06-25 05:35:12,479] INFO: Initiating epoch #232 valid run on device rank=0 [2024-06-25 05:36:07,171] INFO: Rank 0: epoch=232 / 400 train_loss=10.1250 valid_loss=10.1605 stale=0 time=4.25m eta=658.9m [2024-06-25 05:36:07,380] INFO: Initiating epoch #233 train run on device rank=0 [2024-06-25 05:39:24,558] INFO: Initiating epoch #233 valid run on device rank=0 [2024-06-25 05:40:18,386] INFO: Rank 0: epoch=233 / 400 train_loss=10.1250 valid_loss=10.1605 stale=0 time=4.18m eta=655.2m [2024-06-25 05:40:18,447] INFO: Initiating epoch #234 train run on device rank=0 [2024-06-25 05:43:35,934] INFO: Initiating epoch #234 valid run on device rank=0 [2024-06-25 05:44:30,653] INFO: Rank 0: epoch=234 / 400 train_loss=10.1249 valid_loss=10.1604 stale=0 time=4.2m eta=651.5m [2024-06-25 05:44:30,700] INFO: Initiating epoch #235 train run on device rank=0 [2024-06-25 05:47:51,054] INFO: Initiating epoch #235 valid run on device rank=0 [2024-06-25 05:48:44,695] INFO: Rank 0: epoch=235 / 400 train_loss=10.1249 valid_loss=10.1604 stale=0 time=4.23m eta=647.8m [2024-06-25 05:48:44,743] INFO: Initiating epoch #236 train run on device rank=0 [2024-06-25 05:52:05,457] INFO: Initiating epoch #236 valid run on device rank=0 [2024-06-25 05:52:58,294] INFO: Rank 0: epoch=236 / 400 train_loss=10.1249 valid_loss=10.1604 stale=0 time=4.23m eta=644.1m [2024-06-25 05:52:58,346] INFO: Initiating epoch #237 train run on device rank=0 [2024-06-25 05:56:17,936] INFO: Initiating epoch #237 valid run on device rank=0 [2024-06-25 05:57:13,884] INFO: Rank 0: epoch=237 / 400 train_loss=10.1249 valid_loss=10.1603 stale=0 time=4.26m eta=640.4m [2024-06-25 05:57:13,920] INFO: Initiating epoch #238 train run on device rank=0 [2024-06-25 06:00:30,638] INFO: Initiating epoch #238 valid run on device rank=0 [2024-06-25 06:01:25,356] INFO: Rank 0: epoch=238 / 400 train_loss=10.1249 valid_loss=10.1604 stale=1 time=4.19m eta=636.6m [2024-06-25 06:01:25,384] INFO: Initiating epoch #239 train run on device rank=0 [2024-06-25 06:04:45,011] INFO: Initiating epoch #239 valid run on device rank=0 [2024-06-25 06:05:38,631] INFO: Rank 0: epoch=239 / 400 train_loss=10.1249 valid_loss=10.1602 stale=0 time=4.22m eta=632.9m [2024-06-25 06:05:38,646] INFO: Initiating epoch #240 train run on device rank=0 [2024-06-25 06:08:57,278] INFO: Initiating epoch #240 valid run on device rank=0 [2024-06-25 06:09:51,092] INFO: Rank 0: epoch=240 / 400 train_loss=10.1249 valid_loss=10.1604 stale=1 time=4.21m eta=629.1m [2024-06-25 06:09:51,132] INFO: Initiating epoch #241 train run on device rank=0 [2024-06-25 06:13:04,060] INFO: Initiating epoch #241 valid run on device rank=0 [2024-06-25 06:13:57,618] INFO: Rank 0: epoch=241 / 400 train_loss=10.1249 valid_loss=10.1601 stale=0 time=4.11m eta=625.3m [2024-06-25 06:13:57,705] INFO: Initiating epoch #242 train run on device rank=0 [2024-06-25 06:17:15,326] INFO: Initiating epoch #242 valid run on device rank=0 [2024-06-25 06:18:09,818] INFO: Rank 0: epoch=242 / 400 train_loss=10.1248 valid_loss=10.1603 stale=1 time=4.2m eta=621.6m [2024-06-25 06:18:10,227] INFO: Initiating epoch #243 train run on device rank=0 [2024-06-25 06:21:25,608] INFO: Initiating epoch #243 valid run on device rank=0 [2024-06-25 06:22:19,727] INFO: Rank 0: epoch=243 / 400 train_loss=10.1248 valid_loss=10.1600 stale=0 time=4.16m eta=617.8m [2024-06-25 06:22:19,793] INFO: Initiating epoch #244 train run on device rank=0 [2024-06-25 06:25:38,426] INFO: Initiating epoch #244 valid run on device rank=0 [2024-06-25 06:26:33,259] INFO: Rank 0: epoch=244 / 400 train_loss=10.1248 valid_loss=10.1603 stale=1 time=4.22m eta=614.0m [2024-06-25 06:26:33,406] INFO: Initiating epoch #245 train run on device rank=0 [2024-06-25 06:29:49,412] INFO: Initiating epoch #245 valid run on device rank=0 [2024-06-25 06:30:41,444] INFO: Rank 0: epoch=245 / 400 train_loss=10.1248 valid_loss=10.1599 stale=0 time=4.13m eta=610.2m [2024-06-25 06:30:41,468] INFO: Initiating epoch #246 train run on device rank=0 [2024-06-25 06:33:55,288] INFO: Initiating epoch #246 valid run on device rank=0 [2024-06-25 06:34:48,117] INFO: Rank 0: epoch=246 / 400 train_loss=10.1248 valid_loss=10.1603 stale=1 time=4.11m eta=606.4m [2024-06-25 06:34:48,135] INFO: Initiating epoch #247 train run on device rank=0 [2024-06-25 06:38:05,699] INFO: Initiating epoch #247 valid run on device rank=0 [2024-06-25 06:38:58,867] INFO: Rank 0: epoch=247 / 400 train_loss=10.1248 valid_loss=10.1598 stale=0 time=4.18m eta=602.6m [2024-06-25 06:38:59,271] INFO: Initiating epoch #248 train run on device rank=0 [2024-06-25 06:42:17,261] INFO: Initiating epoch #248 valid run on device rank=0 [2024-06-25 06:43:11,859] INFO: Rank 0: epoch=248 / 400 train_loss=10.1247 valid_loss=10.1602 stale=1 time=4.21m eta=598.8m [2024-06-25 06:43:11,898] INFO: Initiating epoch #249 train run on device rank=0 [2024-06-25 06:46:31,387] INFO: Initiating epoch #249 valid run on device rank=0 [2024-06-25 06:47:24,861] INFO: Rank 0: epoch=249 / 400 train_loss=10.1247 valid_loss=10.1598 stale=0 time=4.22m eta=595.1m [2024-06-25 06:47:24,997] INFO: Initiating epoch #250 train run on device rank=0 [2024-06-25 06:50:44,973] INFO: Initiating epoch #250 valid run on device rank=0 [2024-06-25 06:51:38,790] INFO: Rank 0: epoch=250 / 400 train_loss=10.1247 valid_loss=10.1602 stale=1 time=4.23m eta=591.3m [2024-06-25 06:51:38,801] INFO: Initiating epoch #251 train run on device rank=0 [2024-06-25 06:54:55,489] INFO: Initiating epoch #251 valid run on device rank=0 [2024-06-25 06:55:48,056] INFO: Rank 0: epoch=251 / 400 train_loss=10.1247 valid_loss=10.1598 stale=0 time=4.15m eta=587.5m [2024-06-25 06:55:48,099] INFO: Initiating epoch #252 train run on device rank=0 [2024-06-25 06:59:10,139] INFO: Initiating epoch #252 valid run on device rank=0 [2024-06-25 07:00:05,938] INFO: Rank 0: epoch=252 / 400 train_loss=10.1247 valid_loss=10.1602 stale=1 time=4.3m eta=583.8m [2024-06-25 07:00:05,984] INFO: Initiating epoch #253 train run on device rank=0 [2024-06-25 07:03:19,417] INFO: Initiating epoch #253 valid run on device rank=0 [2024-06-25 07:04:14,328] INFO: Rank 0: epoch=253 / 400 train_loss=10.1246 valid_loss=10.1597 stale=0 time=4.14m eta=579.9m [2024-06-25 07:04:14,629] INFO: Initiating epoch #254 train run on device rank=0 [2024-06-25 07:07:31,841] INFO: Initiating epoch #254 valid run on device rank=0 [2024-06-25 07:08:25,857] INFO: Rank 0: epoch=254 / 400 train_loss=10.1246 valid_loss=10.1602 stale=1 time=4.19m eta=576.1m [2024-06-25 07:08:25,881] INFO: Initiating epoch #255 train run on device rank=0 [2024-06-25 07:11:45,393] INFO: Initiating epoch #255 valid run on device rank=0 [2024-06-25 07:12:38,659] INFO: Rank 0: epoch=255 / 400 train_loss=10.1246 valid_loss=10.1597 stale=0 time=4.21m eta=572.3m [2024-06-25 07:12:38,706] INFO: Initiating epoch #256 train run on device rank=0 [2024-06-25 07:15:57,263] INFO: Initiating epoch #256 valid run on device rank=0 [2024-06-25 07:16:49,713] INFO: Rank 0: epoch=256 / 400 train_loss=10.1246 valid_loss=10.1601 stale=1 time=4.18m eta=568.5m [2024-06-25 07:16:49,746] INFO: Initiating epoch #257 train run on device rank=0 [2024-06-25 07:20:06,591] INFO: Initiating epoch #257 valid run on device rank=0 [2024-06-25 07:20:59,562] INFO: Rank 0: epoch=257 / 400 train_loss=10.1246 valid_loss=10.1597 stale=0 time=4.16m eta=564.7m [2024-06-25 07:20:59,590] INFO: Initiating epoch #258 train run on device rank=0 [2024-06-25 07:24:15,733] INFO: Initiating epoch #258 valid run on device rank=0 [2024-06-25 07:25:09,813] INFO: Rank 0: epoch=258 / 400 train_loss=10.1246 valid_loss=10.1601 stale=1 time=4.17m eta=560.9m [2024-06-25 07:25:09,979] INFO: Initiating epoch #259 train run on device rank=0 [2024-06-25 07:28:39,558] INFO: Initiating epoch #259 valid run on device rank=0 [2024-06-25 07:29:34,920] INFO: Rank 0: epoch=259 / 400 train_loss=10.1245 valid_loss=10.1597 stale=0 time=4.42m eta=557.2m [2024-06-25 07:29:35,304] INFO: Initiating epoch #260 train run on device rank=0 [2024-06-25 07:32:51,697] INFO: Initiating epoch #260 valid run on device rank=0 [2024-06-25 07:33:44,220] INFO: Rank 0: epoch=260 / 400 train_loss=10.1245 valid_loss=10.1601 stale=1 time=4.15m eta=553.3m [2024-06-25 07:33:44,240] INFO: Initiating epoch #261 train run on device rank=0 [2024-06-25 07:37:05,314] INFO: Initiating epoch #261 valid run on device rank=0 [2024-06-25 07:37:58,265] INFO: Rank 0: epoch=261 / 400 train_loss=10.1245 valid_loss=10.1597 stale=0 time=4.23m eta=549.5m [2024-06-25 07:37:58,534] INFO: Initiating epoch #262 train run on device rank=0 [2024-06-25 07:41:11,125] INFO: Initiating epoch #262 valid run on device rank=0 [2024-06-25 07:42:03,979] INFO: Rank 0: epoch=262 / 400 train_loss=10.1245 valid_loss=10.1601 stale=1 time=4.09m eta=545.6m [2024-06-25 07:42:04,013] INFO: Initiating epoch #263 train run on device rank=0 [2024-06-25 07:45:25,976] INFO: Initiating epoch #263 valid run on device rank=0 [2024-06-25 07:46:19,258] INFO: Rank 0: epoch=263 / 400 train_loss=10.1245 valid_loss=10.1597 stale=0 time=4.25m eta=541.8m [2024-06-25 07:46:19,275] INFO: Initiating epoch #264 train run on device rank=0 [2024-06-25 07:49:33,212] INFO: Initiating epoch #264 valid run on device rank=0 [2024-06-25 07:50:28,767] INFO: Rank 0: epoch=264 / 400 train_loss=10.1245 valid_loss=10.1600 stale=1 time=4.16m eta=538.0m [2024-06-25 07:50:28,782] INFO: Initiating epoch #265 train run on device rank=0 [2024-06-25 07:53:50,366] INFO: Initiating epoch #265 valid run on device rank=0 [2024-06-25 07:54:43,693] INFO: Rank 0: epoch=265 / 400 train_loss=10.1245 valid_loss=10.1597 stale=0 time=4.25m eta=534.2m [2024-06-25 07:54:43,723] INFO: Initiating epoch #266 train run on device rank=0 [2024-06-25 07:58:05,245] INFO: Initiating epoch #266 valid run on device rank=0 [2024-06-25 07:58:58,386] INFO: Rank 0: epoch=266 / 400 train_loss=10.1245 valid_loss=10.1599 stale=1 time=4.24m eta=530.4m [2024-06-25 07:58:58,527] INFO: Initiating epoch #267 train run on device rank=0 [2024-06-25 08:02:24,067] INFO: Initiating epoch #267 valid run on device rank=0 [2024-06-25 08:03:16,244] INFO: Rank 0: epoch=267 / 400 train_loss=10.1245 valid_loss=10.1596 stale=0 time=4.3m eta=526.6m [2024-06-25 08:03:16,260] INFO: Initiating epoch #268 train run on device rank=0 [2024-06-25 08:06:34,427] INFO: Initiating epoch #268 valid run on device rank=0 [2024-06-25 08:07:26,960] INFO: Rank 0: epoch=268 / 400 train_loss=10.1245 valid_loss=10.1599 stale=1 time=4.18m eta=522.7m [2024-06-25 08:07:27,051] INFO: Initiating epoch #269 train run on device rank=0 [2024-06-25 08:10:45,199] INFO: Initiating epoch #269 valid run on device rank=0 [2024-06-25 08:11:38,154] INFO: Rank 0: epoch=269 / 400 train_loss=10.1244 valid_loss=10.1596 stale=0 time=4.19m eta=518.9m [2024-06-25 08:11:38,188] INFO: Initiating epoch #270 train run on device rank=0 [2024-06-25 08:14:50,072] INFO: Initiating epoch #270 valid run on device rank=0 [2024-06-25 08:15:43,534] INFO: Rank 0: epoch=270 / 400 train_loss=10.1244 valid_loss=10.1599 stale=1 time=4.09m eta=515.0m [2024-06-25 08:15:43,566] INFO: Initiating epoch #271 train run on device rank=0 [2024-06-25 08:19:02,438] INFO: Initiating epoch #271 valid run on device rank=0 [2024-06-25 08:19:55,679] INFO: Rank 0: epoch=271 / 400 train_loss=10.1244 valid_loss=10.1596 stale=0 time=4.2m eta=511.1m [2024-06-25 08:19:55,690] INFO: Initiating epoch #272 train run on device rank=0 [2024-06-25 08:23:13,858] INFO: Initiating epoch #272 valid run on device rank=0 [2024-06-25 08:24:06,941] INFO: Rank 0: epoch=272 / 400 train_loss=10.1244 valid_loss=10.1598 stale=1 time=4.19m eta=507.3m [2024-06-25 08:24:07,115] INFO: Initiating epoch #273 train run on device rank=0 [2024-06-25 08:27:26,845] INFO: Initiating epoch #273 valid run on device rank=0 [2024-06-25 08:28:20,138] INFO: Rank 0: epoch=273 / 400 train_loss=10.1244 valid_loss=10.1596 stale=0 time=4.22m eta=503.4m [2024-06-25 08:28:20,166] INFO: Initiating epoch #274 train run on device rank=0 [2024-06-25 08:31:36,794] INFO: Initiating epoch #274 valid run on device rank=0 [2024-06-25 08:32:30,784] INFO: Rank 0: epoch=274 / 400 train_loss=10.1243 valid_loss=10.1598 stale=1 time=4.18m eta=499.6m [2024-06-25 08:32:30,868] INFO: Initiating epoch #275 train run on device rank=0 [2024-06-25 08:35:58,140] INFO: Initiating epoch #275 valid run on device rank=0 [2024-06-25 08:36:52,881] INFO: Rank 0: epoch=275 / 400 train_loss=10.1243 valid_loss=10.1596 stale=0 time=4.37m eta=495.8m [2024-06-25 08:36:52,966] INFO: Initiating epoch #276 train run on device rank=0 [2024-06-25 08:40:15,560] INFO: Initiating epoch #276 valid run on device rank=0 [2024-06-25 08:41:08,510] INFO: Rank 0: epoch=276 / 400 train_loss=10.1243 valid_loss=10.1597 stale=1 time=4.26m eta=492.0m [2024-06-25 08:41:08,681] INFO: Initiating epoch #277 train run on device rank=0 [2024-06-25 08:44:26,843] INFO: Initiating epoch #277 valid run on device rank=0 [2024-06-25 08:45:21,692] INFO: Rank 0: epoch=277 / 400 train_loss=10.1243 valid_loss=10.1595 stale=0 time=4.22m eta=488.1m [2024-06-25 08:45:21,722] INFO: Initiating epoch #278 train run on device rank=0 [2024-06-25 08:48:34,704] INFO: Initiating epoch #278 valid run on device rank=0 [2024-06-25 08:49:27,378] INFO: Rank 0: epoch=278 / 400 train_loss=10.1243 valid_loss=10.1597 stale=1 time=4.09m eta=484.2m [2024-06-25 08:49:27,491] INFO: Initiating epoch #279 train run on device rank=0 [2024-06-25 08:52:44,236] INFO: Initiating epoch #279 valid run on device rank=0 [2024-06-25 08:53:37,404] INFO: Rank 0: epoch=279 / 400 train_loss=10.1242 valid_loss=10.1595 stale=0 time=4.17m eta=480.3m [2024-06-25 08:53:37,451] INFO: Initiating epoch #280 train run on device rank=0 [2024-06-25 08:56:56,292] INFO: Initiating epoch #280 valid run on device rank=0 [2024-06-25 08:57:50,601] INFO: Rank 0: epoch=280 / 400 train_loss=10.1242 valid_loss=10.1597 stale=1 time=4.22m eta=476.4m [2024-06-25 08:57:50,615] INFO: Initiating epoch #281 train run on device rank=0 [2024-06-25 09:01:09,714] INFO: Initiating epoch #281 valid run on device rank=0 [2024-06-25 09:02:06,298] INFO: Rank 0: epoch=281 / 400 train_loss=10.1242 valid_loss=10.1595 stale=0 time=4.26m eta=472.6m [2024-06-25 09:02:06,356] INFO: Initiating epoch #282 train run on device rank=0 [2024-06-25 09:05:27,658] INFO: Initiating epoch #282 valid run on device rank=0 [2024-06-25 09:06:20,414] INFO: Rank 0: epoch=282 / 400 train_loss=10.1242 valid_loss=10.1596 stale=1 time=4.23m eta=468.7m [2024-06-25 09:06:20,466] INFO: Initiating epoch #283 train run on device rank=0 [2024-06-25 09:09:39,166] INFO: Initiating epoch #283 valid run on device rank=0 [2024-06-25 09:10:33,968] INFO: Rank 0: epoch=283 / 400 train_loss=10.1242 valid_loss=10.1595 stale=0 time=4.23m eta=464.9m [2024-06-25 09:10:34,037] INFO: Initiating epoch #284 train run on device rank=0 [2024-06-25 09:13:58,733] INFO: Initiating epoch #284 valid run on device rank=0 [2024-06-25 09:14:52,802] INFO: Rank 0: epoch=284 / 400 train_loss=10.1241 valid_loss=10.1596 stale=1 time=4.31m eta=461.0m [2024-06-25 09:14:52,865] INFO: Initiating epoch #285 train run on device rank=0 [2024-06-25 09:18:13,174] INFO: Initiating epoch #285 valid run on device rank=0 [2024-06-25 09:19:06,905] INFO: Rank 0: epoch=285 / 400 train_loss=10.1241 valid_loss=10.1595 stale=0 time=4.23m eta=457.2m [2024-06-25 09:19:06,942] INFO: Initiating epoch #286 train run on device rank=0 [2024-06-25 09:22:30,512] INFO: Initiating epoch #286 valid run on device rank=0 [2024-06-25 09:23:22,474] INFO: Rank 0: epoch=286 / 400 train_loss=10.1241 valid_loss=10.1596 stale=1 time=4.26m eta=453.3m [2024-06-25 09:23:22,519] INFO: Initiating epoch #287 train run on device rank=0 [2024-06-25 09:26:36,018] INFO: Initiating epoch #287 valid run on device rank=0 [2024-06-25 09:27:29,897] INFO: Rank 0: epoch=287 / 400 train_loss=10.1241 valid_loss=10.1594 stale=0 time=4.12m eta=449.4m [2024-06-25 09:27:30,007] INFO: Initiating epoch #288 train run on device rank=0 [2024-06-25 09:30:44,989] INFO: Initiating epoch #288 valid run on device rank=0 [2024-06-25 09:31:39,210] INFO: Rank 0: epoch=288 / 400 train_loss=10.1241 valid_loss=10.1596 stale=1 time=4.15m eta=445.5m [2024-06-25 09:31:39,292] INFO: Initiating epoch #289 train run on device rank=0 [2024-06-25 09:34:58,356] INFO: Initiating epoch #289 valid run on device rank=0 [2024-06-25 09:35:52,459] INFO: Rank 0: epoch=289 / 400 train_loss=10.1241 valid_loss=10.1594 stale=0 time=4.22m eta=441.6m [2024-06-25 09:35:52,510] INFO: Initiating epoch #290 train run on device rank=0 [2024-06-25 09:39:09,333] INFO: Initiating epoch #290 valid run on device rank=0 [2024-06-25 09:40:02,589] INFO: Rank 0: epoch=290 / 400 train_loss=10.1240 valid_loss=10.1596 stale=1 time=4.17m eta=437.7m [2024-06-25 09:40:02,701] INFO: Initiating epoch #291 train run on device rank=0 [2024-06-25 09:43:21,541] INFO: Initiating epoch #291 valid run on device rank=0 [2024-06-25 09:44:16,029] INFO: Rank 0: epoch=291 / 400 train_loss=10.1240 valid_loss=10.1594 stale=0 time=4.22m eta=433.8m [2024-06-25 09:44:16,105] INFO: Initiating epoch #292 train run on device rank=0 [2024-06-25 09:47:32,928] INFO: Initiating epoch #292 valid run on device rank=0 [2024-06-25 09:48:26,281] INFO: Rank 0: epoch=292 / 400 train_loss=10.1240 valid_loss=10.1595 stale=1 time=4.17m eta=429.9m [2024-06-25 09:48:26,311] INFO: Initiating epoch #293 train run on device rank=0 [2024-06-25 09:51:44,982] INFO: Initiating epoch #293 valid run on device rank=0 [2024-06-25 09:52:38,693] INFO: Rank 0: epoch=293 / 400 train_loss=10.1240 valid_loss=10.1594 stale=0 time=4.21m eta=426.0m [2024-06-25 09:52:39,075] INFO: Initiating epoch #294 train run on device rank=0 [2024-06-25 09:55:53,855] INFO: Initiating epoch #294 valid run on device rank=0 [2024-06-25 09:56:46,012] INFO: Rank 0: epoch=294 / 400 train_loss=10.1240 valid_loss=10.1595 stale=1 time=4.12m eta=422.1m [2024-06-25 09:56:46,031] INFO: Initiating epoch #295 train run on device rank=0 [2024-06-25 10:00:02,781] INFO: Initiating epoch #295 valid run on device rank=0 [2024-06-25 10:00:56,133] INFO: Rank 0: epoch=295 / 400 train_loss=10.1239 valid_loss=10.1593 stale=0 time=4.17m eta=418.1m [2024-06-25 10:00:56,160] INFO: Initiating epoch #296 train run on device rank=0 [2024-06-25 10:04:18,646] INFO: Initiating epoch #296 valid run on device rank=0 [2024-06-25 10:05:12,611] INFO: Rank 0: epoch=296 / 400 train_loss=10.1239 valid_loss=10.1594 stale=1 time=4.27m eta=414.3m [2024-06-25 10:05:12,666] INFO: Initiating epoch #297 train run on device rank=0 [2024-06-25 10:08:31,492] INFO: Initiating epoch #297 valid run on device rank=0 [2024-06-25 10:09:24,633] INFO: Rank 0: epoch=297 / 400 train_loss=10.1239 valid_loss=10.1593 stale=0 time=4.2m eta=410.4m [2024-06-25 10:09:24,730] INFO: Initiating epoch #298 train run on device rank=0 [2024-06-25 10:12:41,502] INFO: Initiating epoch #298 valid run on device rank=0 [2024-06-25 10:13:35,561] INFO: Rank 0: epoch=298 / 400 train_loss=10.1238 valid_loss=10.1594 stale=1 time=4.18m eta=406.4m [2024-06-25 10:13:35,848] INFO: Initiating epoch #299 train run on device rank=0 [2024-06-25 10:16:53,270] INFO: Initiating epoch #299 valid run on device rank=0 [2024-06-25 10:17:46,598] INFO: Rank 0: epoch=299 / 400 train_loss=10.1238 valid_loss=10.1593 stale=0 time=4.18m eta=402.5m [2024-06-25 10:17:46,611] INFO: Initiating epoch #300 train run on device rank=0 [2024-06-25 10:21:09,440] INFO: Initiating epoch #300 valid run on device rank=0 [2024-06-25 10:22:03,481] INFO: Rank 0: epoch=300 / 400 train_loss=10.1238 valid_loss=10.1594 stale=1 time=4.28m eta=398.6m [2024-06-25 10:22:03,519] INFO: Initiating epoch #301 train run on device rank=0 [2024-06-25 10:25:21,172] INFO: Initiating epoch #301 valid run on device rank=0 [2024-06-25 10:26:14,199] INFO: Rank 0: epoch=301 / 400 train_loss=10.1238 valid_loss=10.1592 stale=0 time=4.18m eta=394.7m [2024-06-25 10:26:14,274] INFO: Initiating epoch #302 train run on device rank=0 [2024-06-25 10:29:32,303] INFO: Initiating epoch #302 valid run on device rank=0 [2024-06-25 10:30:24,800] INFO: Rank 0: epoch=302 / 400 train_loss=10.1237 valid_loss=10.1593 stale=1 time=4.18m eta=390.8m [2024-06-25 10:30:24,898] INFO: Initiating epoch #303 train run on device rank=0 [2024-06-25 10:33:42,762] INFO: Initiating epoch #303 valid run on device rank=0 [2024-06-25 10:34:36,601] INFO: Rank 0: epoch=303 / 400 train_loss=10.1237 valid_loss=10.1592 stale=0 time=4.2m eta=386.9m [2024-06-25 10:34:36,619] INFO: Initiating epoch #304 train run on device rank=0 [2024-06-25 10:37:58,683] INFO: Initiating epoch #304 valid run on device rank=0 [2024-06-25 10:38:52,542] INFO: Rank 0: epoch=304 / 400 train_loss=10.1237 valid_loss=10.1592 stale=1 time=4.27m eta=383.0m [2024-06-25 10:38:52,577] INFO: Initiating epoch #305 train run on device rank=0 [2024-06-25 10:42:07,424] INFO: Initiating epoch #305 valid run on device rank=0 [2024-06-25 10:43:01,383] INFO: Rank 0: epoch=305 / 400 train_loss=10.1237 valid_loss=10.1591 stale=0 time=4.15m eta=379.0m [2024-06-25 10:43:01,461] INFO: Initiating epoch #306 train run on device rank=0 [2024-06-25 10:46:24,264] INFO: Initiating epoch #306 valid run on device rank=0 [2024-06-25 10:47:17,448] INFO: Rank 0: epoch=306 / 400 train_loss=10.1237 valid_loss=10.1592 stale=1 time=4.27m eta=375.1m [2024-06-25 10:47:17,476] INFO: Initiating epoch #307 train run on device rank=0 [2024-06-25 10:50:38,337] INFO: Initiating epoch #307 valid run on device rank=0 [2024-06-25 10:51:32,178] INFO: Rank 0: epoch=307 / 400 train_loss=10.1236 valid_loss=10.1591 stale=0 time=4.25m eta=371.2m [2024-06-25 10:51:32,196] INFO: Initiating epoch #308 train run on device rank=0 [2024-06-25 10:54:53,879] INFO: Initiating epoch #308 valid run on device rank=0 [2024-06-25 10:55:48,882] INFO: Rank 0: epoch=308 / 400 train_loss=10.1236 valid_loss=10.1591 stale=1 time=4.28m eta=367.3m [2024-06-25 10:55:48,948] INFO: Initiating epoch #309 train run on device rank=0 [2024-06-25 10:59:14,518] INFO: Initiating epoch #309 valid run on device rank=0 [2024-06-25 11:00:08,792] INFO: Rank 0: epoch=309 / 400 train_loss=10.1236 valid_loss=10.1590 stale=0 time=4.33m eta=363.4m [2024-06-25 11:00:08,818] INFO: Initiating epoch #310 train run on device rank=0 [2024-06-25 11:03:27,048] INFO: Initiating epoch #310 valid run on device rank=0 [2024-06-25 11:04:21,203] INFO: Rank 0: epoch=310 / 400 train_loss=10.1236 valid_loss=10.1591 stale=1 time=4.21m eta=359.5m [2024-06-25 11:04:21,256] INFO: Initiating epoch #311 train run on device rank=0 [2024-06-25 11:07:44,249] INFO: Initiating epoch #311 valid run on device rank=0 [2024-06-25 11:08:38,283] INFO: Rank 0: epoch=311 / 400 train_loss=10.1236 valid_loss=10.1590 stale=0 time=4.28m eta=355.6m [2024-06-25 11:08:38,354] INFO: Initiating epoch #312 train run on device rank=0 [2024-06-25 11:12:02,457] INFO: Initiating epoch #312 valid run on device rank=0 [2024-06-25 11:12:56,133] INFO: Rank 0: epoch=312 / 400 train_loss=10.1236 valid_loss=10.1590 stale=1 time=4.3m eta=351.7m [2024-06-25 11:12:56,198] INFO: Initiating epoch #313 train run on device rank=0 [2024-06-25 11:16:13,766] INFO: Initiating epoch #313 valid run on device rank=0 [2024-06-25 11:17:07,645] INFO: Rank 0: epoch=313 / 400 train_loss=10.1235 valid_loss=10.1589 stale=0 time=4.19m eta=347.7m [2024-06-25 11:17:07,671] INFO: Initiating epoch #314 train run on device rank=0 [2024-06-25 11:20:32,519] INFO: Initiating epoch #314 valid run on device rank=0 [2024-06-25 11:21:26,988] INFO: Rank 0: epoch=314 / 400 train_loss=10.1235 valid_loss=10.1590 stale=1 time=4.32m eta=343.8m [2024-06-25 11:21:27,091] INFO: Initiating epoch #315 train run on device rank=0 [2024-06-25 11:24:50,677] INFO: Initiating epoch #315 valid run on device rank=0 [2024-06-25 11:25:43,929] INFO: Rank 0: epoch=315 / 400 train_loss=10.1235 valid_loss=10.1589 stale=0 time=4.28m eta=339.9m [2024-06-25 11:25:43,947] INFO: Initiating epoch #316 train run on device rank=0 [2024-06-25 11:29:00,941] INFO: Initiating epoch #316 valid run on device rank=0 [2024-06-25 11:29:55,093] INFO: Rank 0: epoch=316 / 400 train_loss=10.1235 valid_loss=10.1589 stale=1 time=4.19m eta=335.9m [2024-06-25 11:29:55,285] INFO: Initiating epoch #317 train run on device rank=0 [2024-06-25 11:33:15,317] INFO: Initiating epoch #317 valid run on device rank=0 [2024-06-25 11:34:09,323] INFO: Rank 0: epoch=317 / 400 train_loss=10.1235 valid_loss=10.1589 stale=0 time=4.23m eta=332.0m [2024-06-25 11:34:09,357] INFO: Initiating epoch #318 train run on device rank=0 [2024-06-25 11:37:28,740] INFO: Initiating epoch #318 valid run on device rank=0 [2024-06-25 11:38:23,417] INFO: Rank 0: epoch=318 / 400 train_loss=10.1235 valid_loss=10.1589 stale=1 time=4.23m eta=328.1m [2024-06-25 11:38:23,603] INFO: Initiating epoch #319 train run on device rank=0 [2024-06-25 11:41:43,821] INFO: Initiating epoch #319 valid run on device rank=0 [2024-06-25 11:42:37,179] INFO: Rank 0: epoch=319 / 400 train_loss=10.1235 valid_loss=10.1588 stale=0 time=4.23m eta=324.1m [2024-06-25 11:42:37,284] INFO: Initiating epoch #320 train run on device rank=0 [2024-06-25 11:45:52,995] INFO: Initiating epoch #320 valid run on device rank=0 [2024-06-25 11:46:47,655] INFO: Rank 0: epoch=320 / 400 train_loss=10.1235 valid_loss=10.1588 stale=1 time=4.17m eta=320.2m [2024-06-25 11:46:47,874] INFO: Initiating epoch #321 train run on device rank=0 [2024-06-25 11:50:08,260] INFO: Initiating epoch #321 valid run on device rank=0 [2024-06-25 11:51:02,331] INFO: Rank 0: epoch=321 / 400 train_loss=10.1235 valid_loss=10.1588 stale=0 time=4.24m eta=316.2m [2024-06-25 11:51:02,398] INFO: Initiating epoch #322 train run on device rank=0 [2024-06-25 11:54:17,325] INFO: Initiating epoch #322 valid run on device rank=0 [2024-06-25 11:55:11,451] INFO: Rank 0: epoch=322 / 400 train_loss=10.1235 valid_loss=10.1588 stale=1 time=4.15m eta=312.3m [2024-06-25 11:55:11,510] INFO: Initiating epoch #323 train run on device rank=0 [2024-06-25 11:58:32,432] INFO: Initiating epoch #323 valid run on device rank=0 [2024-06-25 11:59:27,418] INFO: Rank 0: epoch=323 / 400 train_loss=10.1235 valid_loss=10.1587 stale=0 time=4.27m eta=308.3m [2024-06-25 11:59:27,482] INFO: Initiating epoch #324 train run on device rank=0 [2024-06-25 12:02:54,204] INFO: Initiating epoch #324 valid run on device rank=0 [2024-06-25 12:03:46,678] INFO: Rank 0: epoch=324 / 400 train_loss=10.1235 valid_loss=10.1587 stale=0 time=4.32m eta=304.4m [2024-06-25 12:03:46,696] INFO: Initiating epoch #325 train run on device rank=0 [2024-06-25 12:07:06,660] INFO: Initiating epoch #325 valid run on device rank=0 [2024-06-25 12:07:59,616] INFO: Rank 0: epoch=325 / 400 train_loss=10.1234 valid_loss=10.1587 stale=0 time=4.22m eta=300.4m [2024-06-25 12:07:59,659] INFO: Initiating epoch #326 train run on device rank=0 [2024-06-25 12:11:18,830] INFO: Initiating epoch #326 valid run on device rank=0 [2024-06-25 12:12:13,851] INFO: Rank 0: epoch=326 / 400 train_loss=10.1234 valid_loss=10.1587 stale=0 time=4.24m eta=296.5m [2024-06-25 12:12:13,952] INFO: Initiating epoch #327 train run on device rank=0 [2024-06-25 12:15:31,214] INFO: Initiating epoch #327 valid run on device rank=0 [2024-06-25 12:16:24,702] INFO: Rank 0: epoch=327 / 400 train_loss=10.1234 valid_loss=10.1586 stale=0 time=4.18m eta=292.5m [2024-06-25 12:16:24,737] INFO: Initiating epoch #328 train run on device rank=0 [2024-06-25 12:19:40,202] INFO: Initiating epoch #328 valid run on device rank=0 [2024-06-25 12:20:34,821] INFO: Rank 0: epoch=328 / 400 train_loss=10.1234 valid_loss=10.1586 stale=0 time=4.17m eta=288.5m [2024-06-25 12:20:34,841] INFO: Initiating epoch #329 train run on device rank=0 [2024-06-25 12:24:02,654] INFO: Initiating epoch #329 valid run on device rank=0 [2024-06-25 12:24:57,822] INFO: Rank 0: epoch=329 / 400 train_loss=10.1234 valid_loss=10.1586 stale=0 time=4.38m eta=284.6m [2024-06-25 12:24:57,874] INFO: Initiating epoch #330 train run on device rank=0 [2024-06-25 12:28:17,930] INFO: Initiating epoch #330 valid run on device rank=0 [2024-06-25 12:29:14,264] INFO: Rank 0: epoch=330 / 400 train_loss=10.1234 valid_loss=10.1586 stale=0 time=4.27m eta=280.7m [2024-06-25 12:29:14,448] INFO: Initiating epoch #331 train run on device rank=0 [2024-06-25 12:32:33,654] INFO: Initiating epoch #331 valid run on device rank=0 [2024-06-25 12:33:28,559] INFO: Rank 0: epoch=331 / 400 train_loss=10.1234 valid_loss=10.1586 stale=0 time=4.24m eta=276.7m [2024-06-25 12:33:28,641] INFO: Initiating epoch #332 train run on device rank=0 [2024-06-25 12:36:51,401] INFO: Initiating epoch #332 valid run on device rank=0 [2024-06-25 12:37:45,004] INFO: Rank 0: epoch=332 / 400 train_loss=10.1234 valid_loss=10.1585 stale=0 time=4.27m eta=272.7m [2024-06-25 12:37:45,046] INFO: Initiating epoch #333 train run on device rank=0 [2024-06-25 12:41:05,175] INFO: Initiating epoch #333 valid run on device rank=0 [2024-06-25 12:41:57,431] INFO: Rank 0: epoch=333 / 400 train_loss=10.1234 valid_loss=10.1585 stale=0 time=4.21m eta=268.8m [2024-06-25 12:41:57,478] INFO: Initiating epoch #334 train run on device rank=0 [2024-06-25 12:45:10,949] INFO: Initiating epoch #334 valid run on device rank=0 [2024-06-25 12:46:05,639] INFO: Rank 0: epoch=334 / 400 train_loss=10.1234 valid_loss=10.1585 stale=0 time=4.14m eta=264.8m [2024-06-25 12:46:05,685] INFO: Initiating epoch #335 train run on device rank=0 [2024-06-25 12:49:22,573] INFO: Initiating epoch #335 valid run on device rank=0 [2024-06-25 12:50:16,560] INFO: Rank 0: epoch=335 / 400 train_loss=10.1234 valid_loss=10.1585 stale=0 time=4.18m eta=260.8m [2024-06-25 12:50:16,579] INFO: Initiating epoch #336 train run on device rank=0 [2024-06-25 12:53:37,998] INFO: Initiating epoch #336 valid run on device rank=0 [2024-06-25 12:54:31,365] INFO: Rank 0: epoch=336 / 400 train_loss=10.1234 valid_loss=10.1585 stale=0 time=4.25m eta=256.8m [2024-06-25 12:54:31,400] INFO: Initiating epoch #337 train run on device rank=0 [2024-06-25 12:57:50,282] INFO: Initiating epoch #337 valid run on device rank=0 [2024-06-25 12:58:44,520] INFO: Rank 0: epoch=337 / 400 train_loss=10.1234 valid_loss=10.1585 stale=0 time=4.22m eta=252.9m [2024-06-25 12:58:44,684] INFO: Initiating epoch #338 train run on device rank=0 [2024-06-25 13:02:02,208] INFO: Initiating epoch #338 valid run on device rank=0 [2024-06-25 13:02:56,050] INFO: Rank 0: epoch=338 / 400 train_loss=10.1233 valid_loss=10.1584 stale=0 time=4.19m eta=248.9m [2024-06-25 13:02:56,069] INFO: Initiating epoch #339 train run on device rank=0 [2024-06-25 13:06:18,592] INFO: Initiating epoch #339 valid run on device rank=0 [2024-06-25 13:07:14,888] INFO: Rank 0: epoch=339 / 400 train_loss=10.1233 valid_loss=10.1584 stale=0 time=4.31m eta=244.9m [2024-06-25 13:07:14,923] INFO: Initiating epoch #340 train run on device rank=0 [2024-06-25 13:10:34,311] INFO: Initiating epoch #340 valid run on device rank=0 [2024-06-25 13:11:28,808] INFO: Rank 0: epoch=340 / 400 train_loss=10.1233 valid_loss=10.1584 stale=0 time=4.23m eta=240.9m [2024-06-25 13:11:29,490] INFO: Initiating epoch #341 train run on device rank=0 [2024-06-25 13:14:47,069] INFO: Initiating epoch #341 valid run on device rank=0 [2024-06-25 13:15:40,767] INFO: Rank 0: epoch=341 / 400 train_loss=10.1233 valid_loss=10.1584 stale=0 time=4.19m eta=237.0m [2024-06-25 13:15:40,841] INFO: Initiating epoch #342 train run on device rank=0 [2024-06-25 13:19:04,039] INFO: Initiating epoch #342 valid run on device rank=0 [2024-06-25 13:19:57,456] INFO: Rank 0: epoch=342 / 400 train_loss=10.1233 valid_loss=10.1584 stale=0 time=4.28m eta=233.0m [2024-06-25 13:19:57,469] INFO: Initiating epoch #343 train run on device rank=0 [2024-06-25 13:23:22,108] INFO: Initiating epoch #343 valid run on device rank=0 [2024-06-25 13:24:16,145] INFO: Rank 0: epoch=343 / 400 train_loss=10.1233 valid_loss=10.1584 stale=0 time=4.31m eta=229.0m [2024-06-25 13:24:16,172] INFO: Initiating epoch #344 train run on device rank=0 [2024-06-25 13:27:35,103] INFO: Initiating epoch #344 valid run on device rank=0 [2024-06-25 13:28:28,894] INFO: Rank 0: epoch=344 / 400 train_loss=10.1233 valid_loss=10.1583 stale=0 time=4.21m eta=225.0m [2024-06-25 13:28:28,962] INFO: Initiating epoch #345 train run on device rank=0 [2024-06-25 13:31:51,052] INFO: Initiating epoch #345 valid run on device rank=0 [2024-06-25 13:32:44,723] INFO: Rank 0: epoch=345 / 400 train_loss=10.1233 valid_loss=10.1583 stale=0 time=4.26m eta=221.1m [2024-06-25 13:32:44,777] INFO: Initiating epoch #346 train run on device rank=0 [2024-06-25 13:36:02,276] INFO: Initiating epoch #346 valid run on device rank=0 [2024-06-25 13:36:56,358] INFO: Rank 0: epoch=346 / 400 train_loss=10.1233 valid_loss=10.1583 stale=0 time=4.19m eta=217.1m [2024-06-25 13:36:56,409] INFO: Initiating epoch #347 train run on device rank=0 [2024-06-25 13:40:11,960] INFO: Initiating epoch #347 valid run on device rank=0 [2024-06-25 13:41:03,950] INFO: Rank 0: epoch=347 / 400 train_loss=10.1232 valid_loss=10.1583 stale=0 time=4.13m eta=213.1m [2024-06-25 13:41:04,078] INFO: Initiating epoch #348 train run on device rank=0 [2024-06-25 13:44:22,731] INFO: Initiating epoch #348 valid run on device rank=0 [2024-06-25 13:45:16,167] INFO: Rank 0: epoch=348 / 400 train_loss=10.1232 valid_loss=10.1583 stale=0 time=4.2m eta=209.1m [2024-06-25 13:45:16,178] INFO: Initiating epoch #349 train run on device rank=0 [2024-06-25 13:48:39,621] INFO: Initiating epoch #349 valid run on device rank=0 [2024-06-25 13:49:34,218] INFO: Rank 0: epoch=349 / 400 train_loss=10.1232 valid_loss=10.1583 stale=0 time=4.3m eta=205.1m [2024-06-25 13:49:34,275] INFO: Initiating epoch #350 train run on device rank=0 [2024-06-25 13:52:59,631] INFO: Initiating epoch #350 valid run on device rank=0 [2024-06-25 13:53:53,117] INFO: Rank 0: epoch=350 / 400 train_loss=10.1232 valid_loss=10.1583 stale=0 time=4.31m eta=201.1m [2024-06-25 13:53:53,150] INFO: Initiating epoch #351 train run on device rank=0 [2024-06-25 13:57:17,967] INFO: Initiating epoch #351 valid run on device rank=0 [2024-06-25 13:58:11,166] INFO: Rank 0: epoch=351 / 400 train_loss=10.1232 valid_loss=10.1583 stale=0 time=4.3m eta=197.1m [2024-06-25 13:58:11,233] INFO: Initiating epoch #352 train run on device rank=0 [2024-06-25 14:01:30,491] INFO: Initiating epoch #352 valid run on device rank=0 [2024-06-25 14:02:24,507] INFO: Rank 0: epoch=352 / 400 train_loss=10.1232 valid_loss=10.1583 stale=1 time=4.22m eta=193.1m [2024-06-25 14:02:24,534] INFO: Initiating epoch #353 train run on device rank=0 [2024-06-25 14:05:48,045] INFO: Initiating epoch #353 valid run on device rank=0 [2024-06-25 14:06:41,646] INFO: Rank 0: epoch=353 / 400 train_loss=10.1232 valid_loss=10.1583 stale=2 time=4.29m eta=189.1m [2024-06-25 14:06:41,816] INFO: Initiating epoch #354 train run on device rank=0 [2024-06-25 14:09:58,266] INFO: Initiating epoch #354 valid run on device rank=0 [2024-06-25 14:10:52,499] INFO: Rank 0: epoch=354 / 400 train_loss=10.1232 valid_loss=10.1583 stale=3 time=4.18m eta=185.1m [2024-06-25 14:10:52,526] INFO: Initiating epoch #355 train run on device rank=0 [2024-06-25 14:14:10,977] INFO: Initiating epoch #355 valid run on device rank=0 [2024-06-25 14:15:05,244] INFO: Rank 0: epoch=355 / 400 train_loss=10.1232 valid_loss=10.1583 stale=4 time=4.21m eta=181.1m [2024-06-25 14:15:05,268] INFO: Initiating epoch #356 train run on device rank=0 [2024-06-25 14:18:26,157] INFO: Initiating epoch #356 valid run on device rank=0 [2024-06-25 14:19:18,627] INFO: Rank 0: epoch=356 / 400 train_loss=10.1232 valid_loss=10.1583 stale=5 time=4.22m eta=177.1m [2024-06-25 14:19:18,673] INFO: Initiating epoch #357 train run on device rank=0 [2024-06-25 14:22:35,128] INFO: Initiating epoch #357 valid run on device rank=0 [2024-06-25 14:23:30,976] INFO: Rank 0: epoch=357 / 400 train_loss=10.1232 valid_loss=10.1583 stale=6 time=4.21m eta=173.1m [2024-06-25 14:23:31,009] INFO: Initiating epoch #358 train run on device rank=0 [2024-06-25 14:26:54,667] INFO: Initiating epoch #358 valid run on device rank=0 [2024-06-25 14:27:48,127] INFO: Rank 0: epoch=358 / 400 train_loss=10.1232 valid_loss=10.1583 stale=7 time=4.29m eta=169.1m [2024-06-25 14:27:48,172] INFO: Initiating epoch #359 train run on device rank=0 [2024-06-25 14:31:03,632] INFO: Initiating epoch #359 valid run on device rank=0 [2024-06-25 14:31:58,000] INFO: Rank 0: epoch=359 / 400 train_loss=10.1232 valid_loss=10.1583 stale=8 time=4.16m eta=165.1m [2024-06-25 14:31:58,025] INFO: Initiating epoch #360 train run on device rank=0 [2024-06-25 14:35:21,565] INFO: Initiating epoch #360 valid run on device rank=0 [2024-06-25 14:36:15,065] INFO: Rank 0: epoch=360 / 400 train_loss=10.1232 valid_loss=10.1583 stale=9 time=4.28m eta=161.1m [2024-06-25 14:36:15,093] INFO: Initiating epoch #361 train run on device rank=0 [2024-06-25 14:39:36,026] INFO: Initiating epoch #361 valid run on device rank=0 [2024-06-25 14:40:30,883] INFO: Rank 0: epoch=361 / 400 train_loss=10.1232 valid_loss=10.1583 stale=10 time=4.26m eta=157.1m [2024-06-25 14:40:30,937] INFO: Initiating epoch #362 train run on device rank=0 [2024-06-25 14:43:46,305] INFO: Initiating epoch #362 valid run on device rank=0 [2024-06-25 14:44:40,148] INFO: Rank 0: epoch=362 / 400 train_loss=10.1232 valid_loss=10.1583 stale=11 time=4.15m eta=153.1m [2024-06-25 14:44:40,213] INFO: Initiating epoch #363 train run on device rank=0 [2024-06-25 14:47:58,505] INFO: Initiating epoch #363 valid run on device rank=0 [2024-06-25 14:48:52,943] INFO: Rank 0: epoch=363 / 400 train_loss=10.1232 valid_loss=10.1583 stale=12 time=4.21m eta=149.1m [2024-06-25 14:48:53,005] INFO: Initiating epoch #364 train run on device rank=0 [2024-06-25 14:52:14,251] INFO: Initiating epoch #364 valid run on device rank=0 [2024-06-25 14:53:07,765] INFO: Rank 0: epoch=364 / 400 train_loss=10.1231 valid_loss=10.1583 stale=13 time=4.25m eta=145.1m [2024-06-25 14:53:07,809] INFO: Initiating epoch #365 train run on device rank=0 [2024-06-25 14:56:22,615] INFO: Initiating epoch #365 valid run on device rank=0 [2024-06-25 14:57:15,779] INFO: Rank 0: epoch=365 / 400 train_loss=10.1231 valid_loss=10.1583 stale=0 time=4.13m eta=141.1m [2024-06-25 14:57:15,825] INFO: Initiating epoch #366 train run on device rank=0 [2024-06-25 15:00:36,729] INFO: Initiating epoch #366 valid run on device rank=0 [2024-06-25 15:01:29,269] INFO: Rank 0: epoch=366 / 400 train_loss=10.1231 valid_loss=10.1583 stale=1 time=4.22m eta=137.1m [2024-06-25 15:01:29,370] INFO: Initiating epoch #367 train run on device rank=0 [2024-06-25 15:04:49,559] INFO: Initiating epoch #367 valid run on device rank=0 [2024-06-25 15:05:43,942] INFO: Rank 0: epoch=367 / 400 train_loss=10.1231 valid_loss=10.1583 stale=0 time=4.24m eta=133.0m [2024-06-25 15:05:43,973] INFO: Initiating epoch #368 train run on device rank=0 [2024-06-25 15:09:03,549] INFO: Initiating epoch #368 valid run on device rank=0 [2024-06-25 15:09:55,592] INFO: Rank 0: epoch=368 / 400 train_loss=10.1231 valid_loss=10.1583 stale=1 time=4.19m eta=129.0m [2024-06-25 15:09:55,675] INFO: Initiating epoch #369 train run on device rank=0 [2024-06-25 15:13:19,704] INFO: Initiating epoch #369 valid run on device rank=0 [2024-06-25 15:14:12,261] INFO: Rank 0: epoch=369 / 400 train_loss=10.1231 valid_loss=10.1583 stale=2 time=4.28m eta=125.0m [2024-06-25 15:14:12,688] INFO: Initiating epoch #370 train run on device rank=0 [2024-06-25 15:17:34,950] INFO: Initiating epoch #370 valid run on device rank=0 [2024-06-25 15:18:29,091] INFO: Rank 0: epoch=370 / 400 train_loss=10.1231 valid_loss=10.1582 stale=0 time=4.27m eta=121.0m [2024-06-25 15:18:29,126] INFO: Initiating epoch #371 train run on device rank=0 [2024-06-25 15:21:42,797] INFO: Initiating epoch #371 valid run on device rank=0 [2024-06-25 15:22:37,390] INFO: Rank 0: epoch=371 / 400 train_loss=10.1231 valid_loss=10.1582 stale=1 time=4.14m eta=117.0m [2024-06-25 15:22:37,652] INFO: Initiating epoch #372 train run on device rank=0 [2024-06-25 15:25:55,899] INFO: Initiating epoch #372 valid run on device rank=0 [2024-06-25 15:26:49,642] INFO: Rank 0: epoch=372 / 400 train_loss=10.1231 valid_loss=10.1583 stale=2 time=4.2m eta=113.0m [2024-06-25 15:26:49,689] INFO: Initiating epoch #373 train run on device rank=0 [2024-06-25 15:30:03,403] INFO: Initiating epoch #373 valid run on device rank=0 [2024-06-25 15:30:57,726] INFO: Rank 0: epoch=373 / 400 train_loss=10.1230 valid_loss=10.1583 stale=3 time=4.13m eta=108.9m [2024-06-25 15:30:57,765] INFO: Initiating epoch #374 train run on device rank=0 [2024-06-25 15:34:19,726] INFO: Initiating epoch #374 valid run on device rank=0 [2024-06-25 15:35:14,184] INFO: Rank 0: epoch=374 / 400 train_loss=10.1230 valid_loss=10.1583 stale=4 time=4.27m eta=104.9m [2024-06-25 15:35:14,351] INFO: Initiating epoch #375 train run on device rank=0 [2024-06-25 15:38:35,932] INFO: Initiating epoch #375 valid run on device rank=0 [2024-06-25 15:39:27,982] INFO: Rank 0: epoch=375 / 400 train_loss=10.1230 valid_loss=10.1583 stale=5 time=4.23m eta=100.9m [2024-06-25 15:39:28,009] INFO: Initiating epoch #376 train run on device rank=0 [2024-06-25 15:42:44,440] INFO: Initiating epoch #376 valid run on device rank=0 [2024-06-25 15:43:38,206] INFO: Rank 0: epoch=376 / 400 train_loss=10.1230 valid_loss=10.1582 stale=6 time=4.17m eta=96.9m [2024-06-25 15:43:38,346] INFO: Initiating epoch #377 train run on device rank=0 [2024-06-25 15:46:58,925] INFO: Initiating epoch #377 valid run on device rank=0 [2024-06-25 15:47:53,662] INFO: Rank 0: epoch=377 / 400 train_loss=10.1230 valid_loss=10.1582 stale=0 time=4.26m eta=92.8m [2024-06-25 15:47:53,698] INFO: Initiating epoch #378 train run on device rank=0 [2024-06-25 15:51:13,752] INFO: Initiating epoch #378 valid run on device rank=0 [2024-06-25 15:52:07,382] INFO: Rank 0: epoch=378 / 400 train_loss=10.1230 valid_loss=10.1582 stale=1 time=4.23m eta=88.8m [2024-06-25 15:52:07,414] INFO: Initiating epoch #379 train run on device rank=0 [2024-06-25 15:55:24,304] INFO: Initiating epoch #379 valid run on device rank=0 [2024-06-25 15:56:18,118] INFO: Rank 0: epoch=379 / 400 train_loss=10.1230 valid_loss=10.1582 stale=0 time=4.18m eta=84.8m [2024-06-25 15:56:18,152] INFO: Initiating epoch #380 train run on device rank=0 [2024-06-25 15:59:40,427] INFO: Initiating epoch #380 valid run on device rank=0 [2024-06-25 16:00:34,899] INFO: Rank 0: epoch=380 / 400 train_loss=10.1230 valid_loss=10.1582 stale=1 time=4.28m eta=80.8m [2024-06-25 16:00:34,954] INFO: Initiating epoch #381 train run on device rank=0 [2024-06-25 16:03:55,838] INFO: Initiating epoch #381 valid run on device rank=0 [2024-06-25 16:04:48,325] INFO: Rank 0: epoch=381 / 400 train_loss=10.1230 valid_loss=10.1582 stale=2 time=4.22m eta=76.7m [2024-06-25 16:04:48,356] INFO: Initiating epoch #382 train run on device rank=0 [2024-06-25 16:08:03,925] INFO: Initiating epoch #382 valid run on device rank=0 [2024-06-25 16:08:57,422] INFO: Rank 0: epoch=382 / 400 train_loss=10.1230 valid_loss=10.1582 stale=3 time=4.15m eta=72.7m [2024-06-25 16:08:57,459] INFO: Initiating epoch #383 train run on device rank=0 [2024-06-25 16:12:15,016] INFO: Initiating epoch #383 valid run on device rank=0 [2024-06-25 16:13:08,794] INFO: Rank 0: epoch=383 / 400 train_loss=10.1230 valid_loss=10.1582 stale=4 time=4.19m eta=68.7m [2024-06-25 16:13:08,914] INFO: Initiating epoch #384 train run on device rank=0 [2024-06-25 16:16:31,111] INFO: Initiating epoch #384 valid run on device rank=0 [2024-06-25 16:17:23,029] INFO: Rank 0: epoch=384 / 400 train_loss=10.1230 valid_loss=10.1582 stale=5 time=4.24m eta=64.6m [2024-06-25 16:17:23,062] INFO: Initiating epoch #385 train run on device rank=0 [2024-06-25 16:20:39,661] INFO: Initiating epoch #385 valid run on device rank=0 [2024-06-25 16:21:32,135] INFO: Rank 0: epoch=385 / 400 train_loss=10.1230 valid_loss=10.1582 stale=6 time=4.15m eta=60.6m [2024-06-25 16:21:32,173] INFO: Initiating epoch #386 train run on device rank=0 [2024-06-25 16:24:54,335] INFO: Initiating epoch #386 valid run on device rank=0 [2024-06-25 16:25:47,276] INFO: Rank 0: epoch=386 / 400 train_loss=10.1229 valid_loss=10.1582 stale=7 time=4.25m eta=56.6m [2024-06-25 16:25:47,303] INFO: Initiating epoch #387 train run on device rank=0 [2024-06-25 16:29:02,240] INFO: Initiating epoch #387 valid run on device rank=0 [2024-06-25 16:29:56,204] INFO: Rank 0: epoch=387 / 400 train_loss=10.1229 valid_loss=10.1582 stale=8 time=4.15m eta=52.5m [2024-06-25 16:29:56,491] INFO: Initiating epoch #388 train run on device rank=0 [2024-06-25 16:33:14,024] INFO: Initiating epoch #388 valid run on device rank=0 [2024-06-25 16:34:07,542] INFO: Rank 0: epoch=388 / 400 train_loss=10.1229 valid_loss=10.1582 stale=9 time=4.18m eta=48.5m [2024-06-25 16:34:07,561] INFO: Initiating epoch #389 train run on device rank=0 [2024-06-25 16:37:21,450] INFO: Initiating epoch #389 valid run on device rank=0 [2024-06-25 16:38:14,661] INFO: Rank 0: epoch=389 / 400 train_loss=10.1229 valid_loss=10.1582 stale=10 time=4.12m eta=44.5m [2024-06-25 16:38:14,911] INFO: Initiating epoch #390 train run on device rank=0 [2024-06-25 16:41:38,584] INFO: Initiating epoch #390 valid run on device rank=0 [2024-06-25 16:42:31,507] INFO: Rank 0: epoch=390 / 400 train_loss=10.1229 valid_loss=10.1582 stale=11 time=4.28m eta=40.4m [2024-06-25 16:42:31,537] INFO: Initiating epoch #391 train run on device rank=0 [2024-06-25 16:45:46,507] INFO: Initiating epoch #391 valid run on device rank=0 [2024-06-25 16:46:39,564] INFO: Rank 0: epoch=391 / 400 train_loss=10.1229 valid_loss=10.1583 stale=12 time=4.13m eta=36.4m [2024-06-25 16:46:39,792] INFO: Initiating epoch #392 train run on device rank=0 [2024-06-25 16:49:59,517] INFO: Initiating epoch #392 valid run on device rank=0 [2024-06-25 16:50:52,275] INFO: Rank 0: epoch=392 / 400 train_loss=10.1229 valid_loss=10.1583 stale=13 time=4.21m eta=32.3m [2024-06-25 16:50:52,294] INFO: Initiating epoch #393 train run on device rank=0 [2024-06-25 16:54:08,397] INFO: Initiating epoch #393 valid run on device rank=0 [2024-06-25 16:55:01,878] INFO: Rank 0: epoch=393 / 400 train_loss=10.1229 valid_loss=10.1583 stale=14 time=4.16m eta=28.3m [2024-06-25 16:55:01,993] INFO: Initiating epoch #394 train run on device rank=0 [2024-06-25 16:58:20,509] INFO: Initiating epoch #394 valid run on device rank=0 [2024-06-25 16:59:15,056] INFO: Rank 0: epoch=394 / 400 train_loss=10.1229 valid_loss=10.1582 stale=15 time=4.22m eta=24.3m [2024-06-25 16:59:15,290] INFO: Initiating epoch #395 train run on device rank=0 [2024-06-25 17:02:32,298] INFO: Initiating epoch #395 valid run on device rank=0 [2024-06-25 17:03:26,536] INFO: Rank 0: epoch=395 / 400 train_loss=10.1229 valid_loss=10.1582 stale=16 time=4.19m eta=20.2m [2024-06-25 17:03:26,570] INFO: Initiating epoch #396 train run on device rank=0 [2024-06-25 17:06:42,566] INFO: Initiating epoch #396 valid run on device rank=0 [2024-06-25 17:07:37,456] INFO: Rank 0: epoch=396 / 400 train_loss=10.1229 valid_loss=10.1582 stale=17 time=4.18m eta=16.2m [2024-06-25 17:07:37,466] INFO: Initiating epoch #397 train run on device rank=0 [2024-06-25 17:10:50,364] INFO: Initiating epoch #397 valid run on device rank=0 [2024-06-25 17:11:44,254] INFO: Rank 0: epoch=397 / 400 train_loss=10.1229 valid_loss=10.1582 stale=18 time=4.11m eta=12.1m [2024-06-25 17:11:44,281] INFO: Initiating epoch #398 train run on device rank=0 [2024-06-25 17:15:06,441] INFO: Initiating epoch #398 valid run on device rank=0 [2024-06-25 17:16:00,824] INFO: Rank 0: epoch=398 / 400 train_loss=10.1229 valid_loss=10.1583 stale=19 time=4.28m eta=8.1m [2024-06-25 17:16:00,854] INFO: Initiating epoch #399 train run on device rank=0 [2024-06-25 17:19:15,938] INFO: Initiating epoch #399 valid run on device rank=0 [2024-06-25 17:20:08,925] INFO: Rank 0: epoch=399 / 400 train_loss=10.1229 valid_loss=10.1583 stale=20 time=4.13m eta=4.0m [2024-06-25 17:20:08,979] INFO: Initiating epoch #400 train run on device rank=0 [2024-06-25 17:23:25,287] INFO: Initiating epoch #400 valid run on device rank=0 [2024-06-25 17:24:19,332] INFO: Done with training. Total training time on device 0 is 1618.177min