Using RTX 4000 series which doesn't support faster communication speedups. Ensuring P2P and IB communications are disabled.
INFO:distributed_c10d:2024-11-05 13:17:49:Added key: store_based_barrier_key:1 to store for rank: 1
INFO:distributed_c10d:2024-11-05 13:17:49:Added key: store_based_barrier_key:1 to store for rank: 0
INFO:distributed_c10d:2024-11-05 13:17:49:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
INFO:distributed_c10d:2024-11-05 13:17:49:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
INFO:Trainer:2024-11-05 13:18:00:[cuda:0] Initialization completed, start model training.
INFO:Trainer:2024-11-05 13:18:00:[cuda:1] Initialization completed, start model training.
INFO:distributed:2024-11-05 13:18:02:Reducer buckets have been rebuilt in this iteration.
INFO:distributed:2024-11-05 13:18:02:Reducer buckets have been rebuilt in this iteration.
INFO:Trainer:2024-11-05 13:25:48:[cuda:0] Epoch 100 | The model starts evaluation on the validation set.

  0%|          | 0/196 [00:00<?, ?it/s]
  1%|          | 1/196 [00:01<04:19,  1.33s/it]
  3%|▎         | 6/196 [00:01<00:34,  5.49it/s]
  6%|▌         | 12/196 [00:01<00:15, 11.93it/s]
  9%|▉         | 18/196 [00:01<00:09, 18.49it/s]
 12%|█▏        | 24/196 [00:01<00:06, 24.94it/s]
 15%|█▌        | 30/196 [00:01<00:05, 30.89it/s]
 18%|█▊        | 36/196 [00:02<00:04, 35.96it/s]
 21%|██▏       | 42/196 [00:02<00:03, 40.45it/s]
 24%|██▍       | 48/196 [00:02<00:03, 43.96it/s]
 28%|██▊       | 54/196 [00:02<00:03, 46.61it/s]
 31%|███       | 60/196 [00:02<00:02, 48.68it/s]
 34%|███▎      | 66/196 [00:02<00:02, 49.82it/s]
 37%|███▋      | 72/196 [00:02<00:02, 51.11it/s]
 40%|███▉      | 78/196 [00:02<00:02, 49.48it/s]
 43%|████▎     | 84/196 [00:02<00:02, 47.80it/s]
 45%|████▌     | 89/196 [00:03<00:02, 46.96it/s]
 48%|████▊     | 94/196 [00:03<00:02, 46.24it/s]
 51%|█████     | 99/196 [00:03<00:02, 45.59it/s]
 53%|█████▎    | 104/196 [00:03<00:02, 45.41it/s]
 56%|█████▌    | 109/196 [00:03<00:01, 45.08it/s]
 58%|█████▊    | 114/196 [00:03<00:01, 44.85it/s]
 61%|██████    | 119/196 [00:03<00:01, 45.15it/s]
 63%|██████▎   | 124/196 [00:03<00:01, 44.80it/s]
 66%|██████▌   | 129/196 [00:03<00:01, 44.20it/s]
 68%|██████▊   | 134/196 [00:04<00:01, 44.93it/s]
 71%|███████   | 139/196 [00:04<00:01, 44.55it/s]
 73%|███████▎  | 144/196 [00:04<00:01, 43.26it/s]
 76%|███████▌  | 149/196 [00:04<00:01, 44.61it/s]
 79%|███████▊  | 154/196 [00:04<00:00, 44.43it/s]
 81%|████████  | 159/196 [00:04<00:00, 43.91it/s]
 84%|████████▎ | 164/196 [00:04<00:00, 44.64it/s]
 86%|████████▌ | 169/196 [00:04<00:00, 46.01it/s]
 89%|████████▉ | 175/196 [00:04<00:00, 47.87it/s]
 92%|█████████▏| 181/196 [00:05<00:00, 49.95it/s]
 95%|█████████▌| 187/196 [00:05<00:00, 50.23it/s]
 98%|█████████▊| 193/196 [00:05<00:00, 50.71it/s]
100%|██████████| 196/196 [00:05<00:00, 35.67it/s]
INFO:Trainer:2024-11-05 13:25:54:mr: 106.685
INFO:Trainer:2024-11-05 13:25:54:mrr: 0.435
INFO:Trainer:2024-11-05 13:25:54:hits@1: 0.297
INFO:Trainer:2024-11-05 13:25:54:hits@3: 0.518
INFO:Trainer:2024-11-05 13:25:54:hits@10: 0.69
INFO:Trainer:2024-11-05 13:25:54:mr_type: 44.65
INFO:Trainer:2024-11-05 13:25:54:mrr_type: 0.484
INFO:Trainer:2024-11-05 13:25:54:hits@1_type: 0.348
INFO:Trainer:2024-11-05 13:25:54:hits@3_type: 0.567
INFO:Trainer:2024-11-05 13:25:54:hits@10_type: 0.736
INFO:EarlyStopping:2024-11-05 13:25:54:Validation score improved (-inf --> 0.690000).  Saving model ...
INFO:Trainer:2024-11-05 13:25:54:[cuda:0] Epoch 100 | Training checkpoint saved at ../../checkpoint/transe-100.pth
INFO:Trainer:2024-11-05 13:25:54:[cuda:0] Epoch [ 100/1000] | Batchsize: 8192 | loss:  0.092149 | 4.68673 seconds/epoch
INFO:Trainer:2024-11-05 13:25:54:[cuda:1] Epoch [ 100/1000] | Batchsize: 8192 | loss:  0.092458 | 4.68671 seconds/epoch
INFO:Trainer:2024-11-05 13:33:47:[cuda:0] Epoch 200 | The model starts evaluation on the validation set.

  0%|          | 0/196 [00:00<?, ?it/s]
  1%|          | 1/196 [00:00<02:36,  1.25it/s]
  3%|▎         | 6/196 [00:00<00:22,  8.55it/s]
  6%|▌         | 12/196 [00:01<00:10, 17.26it/s]
  9%|▉         | 18/196 [00:01<00:07, 25.12it/s]
 12%|█▏        | 24/196 [00:01<00:05, 31.88it/s]
 15%|█▌        | 30/196 [00:01<00:04, 37.52it/s]
 18%|█▊        | 36/196 [00:01<00:03, 41.81it/s]
 21%|██▏       | 42/196 [00:01<00:03, 45.04it/s]
 24%|██▍       | 48/196 [00:01<00:03, 47.55it/s]
 28%|██▊       | 54/196 [00:01<00:02, 49.35it/s]
 31%|███       | 60/196 [00:01<00:02, 46.73it/s]
 33%|███▎      | 65/196 [00:02<00:02, 44.54it/s]
 36%|███▌      | 70/196 [00:02<00:02, 45.59it/s]
 38%|███▊      | 75/196 [00:02<00:02, 45.38it/s]
 41%|████      | 80/196 [00:02<00:02, 45.34it/s]
 43%|████▎     | 85/196 [00:02<00:02, 44.56it/s]
 46%|████▌     | 90/196 [00:02<00:02, 43.86it/s]
 48%|████▊     | 95/196 [00:02<00:02, 44.11it/s]
 51%|█████     | 100/196 [00:02<00:02, 43.90it/s]
 54%|█████▎    | 105/196 [00:02<00:02, 43.87it/s]
 56%|█████▌    | 110/196 [00:03<00:01, 44.56it/s]
 59%|█████▊    | 115/196 [00:03<00:01, 43.84it/s]
 61%|██████    | 120/196 [00:03<00:01, 43.48it/s]
 64%|██████▍   | 125/196 [00:03<00:01, 44.13it/s]
 66%|██████▋   | 130/196 [00:03<00:01, 43.70it/s]
 69%|██████▉   | 135/196 [00:03<00:01, 43.82it/s]
 71%|███████▏  | 140/196 [00:03<00:01, 44.28it/s]
 74%|███████▍  | 145/196 [00:03<00:01, 43.95it/s]
 77%|███████▋  | 150/196 [00:03<00:01, 43.40it/s]
 79%|███████▉  | 155/196 [00:04<00:00, 44.01it/s]
 82%|████████▏ | 160/196 [00:04<00:00, 43.97it/s]
 84%|████████▍ | 165/196 [00:04<00:00, 42.56it/s]
 87%|████████▋ | 170/196 [00:04<00:00, 42.31it/s]
 89%|████████▉ | 175/196 [00:04<00:00, 42.06it/s]
 92%|█████████▏| 180/196 [00:04<00:00, 41.20it/s]
 94%|█████████▍| 185/196 [00:04<00:00, 41.86it/s]
 97%|█████████▋| 191/196 [00:04<00:00, 44.74it/s]
100%|██████████| 196/196 [00:05<00:00, 37.91it/s]
INFO:Trainer:2024-11-05 13:33:52:mr: 97.4
INFO:Trainer:2024-11-05 13:33:52:mrr: 0.47
INFO:Trainer:2024-11-05 13:33:52:hits@1: 0.331
INFO:Trainer:2024-11-05 13:33:52:hits@3: 0.558
INFO:Trainer:2024-11-05 13:33:52:hits@10: 0.719
INFO:Trainer:2024-11-05 13:33:52:mr_type: 41.337
INFO:Trainer:2024-11-05 13:33:52:mrr_type: 0.517
INFO:Trainer:2024-11-05 13:33:52:hits@1_type: 0.38
INFO:Trainer:2024-11-05 13:33:52:hits@3_type: 0.604
INFO:Trainer:2024-11-05 13:33:52:hits@10_type: 0.761
INFO:EarlyStopping:2024-11-05 13:33:52:Validation score improved (0.690000 --> 0.719000).  Saving model ...
INFO:Trainer:2024-11-05 13:33:52:[cuda:0] Epoch 200 | Training checkpoint saved at ../../checkpoint/transe-200.pth
INFO:Trainer:2024-11-05 13:33:52:[cuda:0] Epoch [ 200/1000] | Batchsize: 8192 | loss:  0.074799 | 4.73561 seconds/epoch
INFO:Trainer:2024-11-05 13:33:52:[cuda:1] Epoch [ 200/1000] | Batchsize: 8192 | loss:  0.074608 | 4.73551 seconds/epoch
INFO:Trainer:2024-11-05 13:41:43:[cuda:0] Epoch 300 | The model starts evaluation on the validation set.

  0%|          | 0/196 [00:00<?, ?it/s]
  1%|          | 1/196 [00:00<02:58,  1.09it/s]
  3%|▎         | 6/196 [00:01<00:24,  7.62it/s]
  6%|▌         | 12/196 [00:01<00:11, 15.62it/s]
  9%|▉         | 18/196 [00:01<00:08, 21.08it/s]
 12%|█▏        | 24/196 [00:01<00:06, 27.69it/s]
 15%|█▌        | 30/196 [00:01<00:04, 33.43it/s]
 18%|█▊        | 36/196 [00:01<00:04, 38.24it/s]
 21%|██▏       | 42/196 [00:01<00:03, 41.88it/s]
 24%|██▍       | 48/196 [00:01<00:03, 44.80it/s]
 28%|██▊       | 54/196 [00:01<00:03, 46.89it/s]
 31%|███       | 60/196 [00:02<00:02, 48.34it/s]
 34%|███▎      | 66/196 [00:02<00:02, 49.59it/s]
 37%|███▋      | 72/196 [00:02<00:02, 50.24it/s]
 40%|███▉      | 78/196 [00:02<00:02, 48.94it/s]
 43%|████▎     | 84/196 [00:02<00:02, 47.09it/s]
 45%|████▌     | 89/196 [00:02<00:02, 46.54it/s]
 48%|████▊     | 94/196 [00:02<00:02, 46.10it/s]
 51%|█████     | 99/196 [00:02<00:02, 45.42it/s]
 53%|█████▎    | 104/196 [00:03<00:02, 44.97it/s]
 56%|█████▌    | 109/196 [00:03<00:01, 44.89it/s]
 58%|█████▊    | 114/196 [00:03<00:01, 44.35it/s]
 61%|██████    | 119/196 [00:03<00:01, 44.61it/s]
 63%|██████▎   | 124/196 [00:03<00:01, 44.93it/s]
 66%|██████▌   | 129/196 [00:03<00:01, 44.67it/s]
 68%|██████▊   | 134/196 [00:03<00:01, 44.26it/s]
 71%|███████   | 139/196 [00:03<00:01, 44.35it/s]
 73%|███████▎  | 144/196 [00:03<00:01, 44.33it/s]
 76%|███████▌  | 149/196 [00:04<00:01, 44.39it/s]
 79%|███████▊  | 154/196 [00:04<00:00, 44.77it/s]
 81%|████████  | 159/196 [00:04<00:00, 44.65it/s]
 84%|████████▎ | 164/196 [00:04<00:00, 44.26it/s]
 86%|████████▌ | 169/196 [00:04<00:00, 45.53it/s]
 89%|████████▉ | 175/196 [00:04<00:00, 47.10it/s]
 92%|█████████▏| 180/196 [00:04<00:00, 47.85it/s]
 95%|█████████▍| 186/196 [00:04<00:00, 48.82it/s]
 97%|█████████▋| 191/196 [00:04<00:00, 49.00it/s]
100%|██████████| 196/196 [00:05<00:00, 37.77it/s]
INFO:Trainer:2024-11-05 13:41:49:mr: 92.897
INFO:Trainer:2024-11-05 13:41:49:mrr: 0.493
INFO:Trainer:2024-11-05 13:41:49:hits@1: 0.355
INFO:Trainer:2024-11-05 13:41:49:hits@3: 0.582
INFO:Trainer:2024-11-05 13:41:49:hits@10: 0.738
INFO:Trainer:2024-11-05 13:41:49:mr_type: 39.12
INFO:Trainer:2024-11-05 13:41:49:mrr_type: 0.539
INFO:Trainer:2024-11-05 13:41:49:hits@1_type: 0.404
INFO:Trainer:2024-11-05 13:41:49:hits@3_type: 0.629
INFO:Trainer:2024-11-05 13:41:49:hits@10_type: 0.778
INFO:EarlyStopping:2024-11-05 13:41:49:Validation score improved (0.719000 --> 0.738000).  Saving model ...
INFO:Trainer:2024-11-05 13:41:49:[cuda:0] Epoch 300 | Training checkpoint saved at ../../checkpoint/transe-300.pth
INFO:Trainer:2024-11-05 13:41:49:[cuda:0] Epoch [ 300/1000] | Batchsize: 8192 | loss:  0.065236 | 4.74605 seconds/epoch
INFO:Trainer:2024-11-05 13:41:49:[cuda:1] Epoch [ 300/1000] | Batchsize: 8192 | loss:  0.064975 | 4.74603 seconds/epoch
INFO:Trainer:2024-11-05 13:49:39:[cuda:0] Epoch 400 | The model starts evaluation on the validation set.

  0%|          | 0/196 [00:00<?, ?it/s]
  1%|          | 1/196 [00:00<02:40,  1.22it/s]
  3%|▎         | 6/196 [00:00<00:22,  8.38it/s]
  6%|▌         | 12/196 [00:01<00:10, 16.99it/s]
  9%|▉         | 18/196 [00:01<00:07, 24.54it/s]
 12%|█▏        | 24/196 [00:01<00:05, 31.39it/s]
 15%|█▌        | 30/196 [00:01<00:04, 37.10it/s]
 18%|█▊        | 36/196 [00:01<00:03, 41.50it/s]
 21%|██▏       | 42/196 [00:01<00:03, 44.91it/s]
 24%|██▍       | 48/196 [00:01<00:03, 47.37it/s]
 28%|██▊       | 54/196 [00:01<00:02, 49.16it/s]
 31%|███       | 60/196 [00:01<00:02, 50.60it/s]
 34%|███▎      | 66/196 [00:02<00:02, 49.60it/s]
 37%|███▋      | 72/196 [00:02<00:02, 47.47it/s]
 39%|███▉      | 77/196 [00:02<00:02, 45.93it/s]
 42%|████▏     | 82/196 [00:02<00:02, 44.60it/s]
 44%|████▍     | 87/196 [00:02<00:02, 44.00it/s]
 47%|████▋     | 92/196 [00:02<00:02, 43.64it/s]
 49%|████▉     | 97/196 [00:02<00:02, 43.48it/s]
 52%|█████▏    | 102/196 [00:02<00:02, 43.32it/s]
 55%|█████▍    | 107/196 [00:03<00:02, 42.92it/s]
 57%|█████▋    | 112/196 [00:03<00:01, 42.43it/s]
 60%|█████▉    | 117/196 [00:03<00:01, 42.42it/s]
 62%|██████▏   | 122/196 [00:03<00:01, 42.71it/s]
 65%|██████▍   | 127/196 [00:03<00:01, 42.76it/s]
 67%|██████▋   | 132/196 [00:03<00:01, 42.84it/s]
 70%|██████▉   | 137/196 [00:03<00:01, 42.60it/s]
 72%|███████▏  | 142/196 [00:03<00:01, 42.27it/s]
 75%|███████▌  | 147/196 [00:03<00:01, 42.37it/s]
 78%|███████▊  | 152/196 [00:04<00:01, 42.58it/s]
 80%|████████  | 157/196 [00:04<00:00, 42.69it/s]
 83%|████████▎ | 162/196 [00:04<00:00, 42.79it/s]
 85%|████████▌ | 167/196 [00:04<00:00, 43.11it/s]
 88%|████████▊ | 172/196 [00:04<00:00, 44.20it/s]
 90%|█████████ | 177/196 [00:04<00:00, 45.46it/s]
 93%|█████████▎| 182/196 [00:04<00:00, 46.61it/s]
 95%|█████████▌| 187/196 [00:04<00:00, 47.55it/s]
 98%|█████████▊| 192/196 [00:04<00:00, 48.16it/s]
100%|██████████| 196/196 [00:05<00:00, 38.21it/s]
INFO:Trainer:2024-11-05 13:49:44:mr: 85.137
INFO:Trainer:2024-11-05 13:49:44:mrr: 0.527
INFO:Trainer:2024-11-05 13:49:44:hits@1: 0.392
INFO:Trainer:2024-11-05 13:49:44:hits@3: 0.619
INFO:Trainer:2024-11-05 13:49:44:hits@10: 0.761
INFO:Trainer:2024-11-05 13:49:44:mr_type: 36.351
INFO:Trainer:2024-11-05 13:49:44:mrr_type: 0.572
INFO:Trainer:2024-11-05 13:49:44:hits@1_type: 0.44
INFO:Trainer:2024-11-05 13:49:44:hits@3_type: 0.663
INFO:Trainer:2024-11-05 13:49:44:hits@10_type: 0.798
INFO:EarlyStopping:2024-11-05 13:49:44:Validation score improved (0.738000 --> 0.761000).  Saving model ...
INFO:Trainer:2024-11-05 13:49:44:[cuda:0] Epoch 400 | Training checkpoint saved at ../../checkpoint/transe-400.pth
INFO:Trainer:2024-11-05 13:49:44:[cuda:0] Epoch [ 400/1000] | Batchsize: 8192 | loss:  0.050724 | 4.74811 seconds/epoch
INFO:Trainer:2024-11-05 13:49:44:[cuda:1] Epoch [ 400/1000] | Batchsize: 8192 | loss:  0.050384 | 4.74808 seconds/epoch
INFO:Trainer:2024-11-05 13:57:32:[cuda:0] Epoch 500 | The model starts evaluation on the validation set.

  0%|          | 0/196 [00:00<?, ?it/s]
  1%|          | 1/196 [00:00<02:44,  1.18it/s]
  4%|▎         | 7/196 [00:00<00:19,  9.46it/s]
  7%|▋         | 13/196 [00:01<00:10, 17.52it/s]
  9%|▉         | 18/196 [00:01<00:07, 22.55it/s]
 12%|█▏        | 24/196 [00:01<00:05, 29.61it/s]
 15%|█▌        | 30/196 [00:01<00:04, 35.29it/s]
 18%|█▊        | 36/196 [00:01<00:04, 39.41it/s]
 21%|██▏       | 42/196 [00:01<00:03, 43.05it/s]
 24%|██▍       | 48/196 [00:01<00:03, 45.89it/s]
 28%|██▊       | 54/196 [00:01<00:03, 47.32it/s]
 31%|███       | 60/196 [00:02<00:02, 48.77it/s]
 34%|███▎      | 66/196 [00:02<00:02, 49.52it/s]
 37%|███▋      | 72/196 [00:02<00:02, 50.15it/s]
 40%|███▉      | 78/196 [00:02<00:02, 50.02it/s]
 43%|████▎     | 84/196 [00:02<00:02, 48.12it/s]
 45%|████▌     | 89/196 [00:02<00:02, 46.69it/s]
 48%|████▊     | 94/196 [00:02<00:02, 45.74it/s]
 51%|█████     | 99/196 [00:02<00:02, 45.08it/s]
 53%|█████▎    | 104/196 [00:02<00:02, 44.25it/s]
 56%|█████▌    | 109/196 [00:03<00:01, 44.40it/s]
 58%|█████▊    | 114/196 [00:03<00:01, 43.71it/s]
 61%|██████    | 119/196 [00:03<00:01, 44.37it/s]
 63%|██████▎   | 124/196 [00:03<00:01, 44.07it/s]
 66%|██████▌   | 129/196 [00:03<00:01, 43.93it/s]
 68%|██████▊   | 134/196 [00:03<00:01, 42.02it/s]
 71%|███████   | 139/196 [00:03<00:01, 40.35it/s]
 73%|███████▎  | 144/196 [00:03<00:01, 39.19it/s]
 76%|███████▌  | 148/196 [00:04<00:01, 39.12it/s]
 78%|███████▊  | 152/196 [00:04<00:01, 38.64it/s]
 80%|███████▉  | 156/196 [00:04<00:01, 38.24it/s]
 82%|████████▏ | 160/196 [00:04<00:00, 37.65it/s]
 84%|████████▎ | 164/196 [00:04<00:00, 37.82it/s]
 86%|████████▌ | 169/196 [00:04<00:00, 38.69it/s]
 89%|████████▉ | 174/196 [00:04<00:00, 39.64it/s]
 91%|█████████▏| 179/196 [00:04<00:00, 40.32it/s]
 94%|█████████▍| 184/196 [00:04<00:00, 40.68it/s]
 96%|█████████▋| 189/196 [00:05<00:00, 40.77it/s]
 99%|█████████▉| 194/196 [00:05<00:00, 39.78it/s]
100%|██████████| 196/196 [00:05<00:00, 36.49it/s]
INFO:Trainer:2024-11-05 13:57:38:mr: 84.444
INFO:Trainer:2024-11-05 13:57:38:mrr: 0.533
INFO:Trainer:2024-11-05 13:57:38:hits@1: 0.399
INFO:Trainer:2024-11-05 13:57:38:hits@3: 0.626
INFO:Trainer:2024-11-05 13:57:38:hits@10: 0.764
INFO:Trainer:2024-11-05 13:57:38:mr_type: 36.09
INFO:Trainer:2024-11-05 13:57:38:mrr_type: 0.578
INFO:Trainer:2024-11-05 13:57:38:hits@1_type: 0.447
INFO:Trainer:2024-11-05 13:57:38:hits@3_type: 0.669
INFO:Trainer:2024-11-05 13:57:38:hits@10_type: 0.802
INFO:EarlyStopping:2024-11-05 13:57:38:EarlyStopping counter: 1 / 2
INFO:Trainer:2024-11-05 13:57:38:[cuda:0] Epoch 500 | Training checkpoint saved at ../../checkpoint/transe-500.pth
INFO:Trainer:2024-11-05 13:57:38:[cuda:1] Epoch [ 500/1000] | Batchsize: 8192 | loss:  0.048685 | 4.74528 seconds/epoch
INFO:Trainer:2024-11-05 13:57:38:[cuda:0] Epoch [ 500/1000] | Batchsize: 8192 | loss:  0.048885 | 4.74529 seconds/epoch
INFO:Trainer:2024-11-05 14:05:26:[cuda:0] Epoch 600 | The model starts evaluation on the validation set.

  0%|          | 0/196 [00:00<?, ?it/s]
  1%|          | 1/196 [00:00<02:54,  1.12it/s]
  3%|▎         | 6/196 [00:01<00:24,  7.79it/s]
  6%|▌         | 12/196 [00:01<00:11, 15.93it/s]
  9%|▉         | 18/196 [00:01<00:07, 22.33it/s]
 12%|█▏        | 24/196 [00:01<00:05, 29.02it/s]
 15%|█▌        | 30/196 [00:01<00:04, 34.58it/s]
 18%|█▊        | 36/196 [00:01<00:04, 39.21it/s]
 21%|██▏       | 42/196 [00:01<00:03, 42.76it/s]
 24%|██▍       | 48/196 [00:01<00:03, 45.30it/s]
 28%|██▊       | 54/196 [00:01<00:02, 47.35it/s]
 31%|███       | 60/196 [00:02<00:02, 48.67it/s]
 34%|███▎      | 66/196 [00:02<00:02, 49.28it/s]
 37%|███▋      | 72/196 [00:02<00:02, 45.98it/s]
 39%|███▉      | 77/196 [00:02<00:02, 42.67it/s]
 42%|████▏     | 82/196 [00:02<00:02, 40.99it/s]
 44%|████▍     | 87/196 [00:02<00:02, 41.50it/s]
 47%|████▋     | 92/196 [00:02<00:02, 42.09it/s]
 49%|████▉     | 97/196 [00:02<00:02, 42.26it/s]
 52%|█████▏    | 102/196 [00:03<00:02, 42.98it/s]
 55%|█████▍    | 107/196 [00:03<00:02, 42.49it/s]
 57%|█████▋    | 112/196 [00:03<00:01, 42.43it/s]
 60%|█████▉    | 117/196 [00:03<00:01, 42.45it/s]
 62%|██████▏   | 122/196 [00:03<00:01, 42.99it/s]
 65%|██████▍   | 127/196 [00:03<00:01, 42.94it/s]
 67%|██████▋   | 132/196 [00:03<00:01, 42.94it/s]
 70%|██████▉   | 137/196 [00:03<00:01, 43.08it/s]
 72%|███████▏  | 142/196 [00:04<00:01, 42.51it/s]
 75%|███████▌  | 147/196 [00:04<00:01, 42.56it/s]
 78%|███████▊  | 152/196 [00:04<00:01, 43.27it/s]
 80%|████████  | 157/196 [00:04<00:00, 42.97it/s]
 83%|████████▎ | 162/196 [00:04<00:00, 43.02it/s]
 85%|████████▌ | 167/196 [00:04<00:00, 43.72it/s]
 88%|████████▊ | 172/196 [00:04<00:00, 44.82it/s]
 90%|█████████ | 177/196 [00:04<00:00, 45.94it/s]
 93%|█████████▎| 183/196 [00:04<00:00, 47.40it/s]
 96%|█████████▋| 189/196 [00:05<00:00, 48.18it/s]
 99%|█████████▉| 194/196 [00:05<00:00, 48.35it/s]
100%|██████████| 196/196 [00:05<00:00, 37.16it/s]
INFO:Trainer:2024-11-05 14:05:32:mr: 84.059
INFO:Trainer:2024-11-05 14:05:32:mrr: 0.536
INFO:Trainer:2024-11-05 14:05:32:hits@1: 0.402
INFO:Trainer:2024-11-05 14:05:32:hits@3: 0.627
INFO:Trainer:2024-11-05 14:05:32:hits@10: 0.766
INFO:Trainer:2024-11-05 14:05:32:mr_type: 35.784
INFO:Trainer:2024-11-05 14:05:32:mrr_type: 0.58
INFO:Trainer:2024-11-05 14:05:32:hits@1_type: 0.45
INFO:Trainer:2024-11-05 14:05:32:hits@3_type: 0.671
INFO:Trainer:2024-11-05 14:05:32:hits@10_type: 0.803
INFO:EarlyStopping:2024-11-05 14:05:32:EarlyStopping counter: 2 / 2
INFO:Trainer:2024-11-05 14:05:32:[cuda:0] Send an early stopping signal
INFO:Trainer:2024-11-05 14:05:32:[cuda:0] Epoch 600 | Training checkpoint saved at ../../checkpoint/transe-600.pth
INFO:Trainer:2024-11-05 14:05:32:[cuda:0] Early stopping
INFO:Trainer:2024-11-05 14:05:32:[cuda:1] Early stopping
INFO:Trainer:2024-11-05 14:05:32:[cuda:1] The model training is completed, taking a total of 2846.64839 seconds.
INFO:Trainer:2024-11-05 14:05:32:[cuda:0] The model training is completed, taking a total of 2846.64717 seconds.
INFO:Trainer:2024-11-05 14:05:32:[cuda:0] Model saved at ../../checkpoint/transe.pth.
INFO:Trainer:2024-11-05 14:05:32:[cuda:0] The model starts evaluating in the test set.

  0%|          | 0/231 [00:00<?, ?it/s]
  0%|          | 1/231 [00:00<03:31,  1.09it/s]
  3%|▎         | 6/231 [00:01<00:29,  7.62it/s]
  5%|▌         | 12/231 [00:01<00:13, 15.70it/s]
  7%|▋         | 17/231 [00:01<00:10, 21.12it/s]
 10%|▉         | 22/231 [00:01<00:08, 23.52it/s]
 12%|█▏        | 28/231 [00:01<00:06, 29.99it/s]
 15%|█▍        | 34/231 [00:01<00:05, 35.65it/s]
 17%|█▋        | 40/231 [00:01<00:04, 39.94it/s]
 20%|█▉        | 46/231 [00:01<00:04, 43.31it/s]
 23%|██▎       | 52/231 [00:01<00:03, 45.96it/s]
 25%|██▌       | 58/231 [00:02<00:03, 47.79it/s]
 28%|██▊       | 64/231 [00:02<00:03, 49.06it/s]
 30%|███       | 70/231 [00:02<00:03, 49.10it/s]
 33%|███▎      | 76/231 [00:02<00:03, 46.09it/s]
 35%|███▌      | 81/231 [00:02<00:03, 44.58it/s]
 37%|███▋      | 86/231 [00:02<00:03, 44.03it/s]
 39%|███▉      | 91/231 [00:02<00:03, 44.03it/s]
 42%|████▏     | 96/231 [00:02<00:03, 43.87it/s]
 44%|████▎     | 101/231 [00:03<00:02, 43.63it/s]
 46%|████▌     | 106/231 [00:03<00:02, 42.22it/s]
 48%|████▊     | 111/231 [00:03<00:02, 41.68it/s]
 50%|█████     | 116/231 [00:03<00:02, 42.09it/s]
 52%|█████▏    | 121/231 [00:03<00:02, 42.36it/s]
 55%|█████▍    | 126/231 [00:03<00:02, 42.75it/s]
 57%|█████▋    | 131/231 [00:03<00:02, 42.70it/s]
 59%|█████▉    | 136/231 [00:03<00:02, 41.67it/s]
 61%|██████    | 141/231 [00:04<00:02, 41.34it/s]
 63%|██████▎   | 146/231 [00:04<00:02, 41.57it/s]
 65%|██████▌   | 151/231 [00:04<00:01, 42.09it/s]
 68%|██████▊   | 156/231 [00:04<00:01, 42.44it/s]
 70%|██████▉   | 161/231 [00:04<00:01, 42.17it/s]
 72%|███████▏  | 166/231 [00:04<00:01, 41.60it/s]
 74%|███████▍  | 171/231 [00:04<00:01, 41.13it/s]
 76%|███████▌  | 176/231 [00:04<00:01, 41.22it/s]
 78%|███████▊  | 181/231 [00:05<00:01, 41.92it/s]
 81%|████████  | 186/231 [00:05<00:01, 42.15it/s]
 83%|████████▎ | 191/231 [00:05<00:00, 41.90it/s]
 85%|████████▍ | 196/231 [00:05<00:00, 41.48it/s]
 87%|████████▋ | 201/231 [00:05<00:00, 41.25it/s]
 89%|████████▉ | 206/231 [00:05<00:00, 43.05it/s]
 91%|█████████▏| 211/231 [00:05<00:00, 44.72it/s]
 94%|█████████▎| 216/231 [00:05<00:00, 45.90it/s]
 96%|█████████▌| 221/231 [00:05<00:00, 46.22it/s]
 98%|█████████▊| 226/231 [00:06<00:00, 45.72it/s]
100%|██████████| 231/231 [00:06<00:00, 46.64it/s]
100%|██████████| 231/231 [00:06<00:00, 37.03it/s]
INFO:Trainer:2024-11-05 14:05:38:mr: 83.782
INFO:Trainer:2024-11-05 14:05:38:mrr: 0.535
INFO:Trainer:2024-11-05 14:05:38:hits@1: 0.4
INFO:Trainer:2024-11-05 14:05:38:hits@3: 0.628
INFO:Trainer:2024-11-05 14:05:38:hits@10: 0.767
INFO:Trainer:2024-11-05 14:05:38:mr_type: 35.354
INFO:Trainer:2024-11-05 14:05:38:mrr_type: 0.58
INFO:Trainer:2024-11-05 14:05:38:hits@1_type: 0.449
INFO:Trainer:2024-11-05 14:05:38:hits@3_type: 0.673
INFO:Trainer:2024-11-05 14:05:38:hits@10_type: 0.803