Using RTX 4000 series which doesn't support faster communication speedups. Ensuring P2P and IB communications are disabled.
INFO:distributed_c10d:2024-12-05 16:58:22:Added key: store_based_barrier_key:1 to store for rank: 1
INFO:distributed_c10d:2024-12-05 16:58:23:Added key: store_based_barrier_key:1 to store for rank: 0
INFO:distributed_c10d:2024-12-05 16:58:23:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
INFO:distributed_c10d:2024-12-05 16:58:23:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
INFO:Trainer:2024-12-05 16:58:24:[cuda:1] Initialization completed, start model training.
INFO:Trainer:2024-12-05 16:58:24:[cuda:0] Initialization completed, start model training.
INFO:distributed:2024-12-05 16:58:26:Reducer buckets have been rebuilt in this iteration.
INFO:distributed:2024-12-05 16:58:26:Reducer buckets have been rebuilt in this iteration.
INFO:Trainer:2024-12-05 17:04:38:[cuda:0] Epoch 100 | The model starts evaluation on the validation set.

  0%|          | 0/69 [00:00<?, ?it/s]
  1%|▏         | 1/69 [00:01<01:34,  1.38s/it]
  3%|▎         | 2/69 [00:01<00:42,  1.57it/s]
  4%|▍         | 3/69 [00:01<00:25,  2.55it/s]
  6%|▌         | 4/69 [00:01<00:18,  3.56it/s]
  7%|▋         | 5/69 [00:01<00:14,  4.55it/s]
  9%|▊         | 6/69 [00:01<00:11,  5.53it/s]
 10%|█         | 7/69 [00:02<00:09,  6.42it/s]
 13%|█▎        | 9/69 [00:02<00:07,  8.04it/s]
 16%|█▌        | 11/69 [00:02<00:06,  9.04it/s]
 19%|█▉        | 13/69 [00:02<00:05,  9.69it/s]
 22%|██▏       | 15/69 [00:02<00:05, 10.08it/s]
 25%|██▍       | 17/69 [00:02<00:05, 10.36it/s]
 28%|██▊       | 19/69 [00:03<00:04, 10.53it/s]
 30%|███       | 21/69 [00:03<00:04, 10.69it/s]
 33%|███▎      | 23/69 [00:03<00:04, 10.78it/s]
 36%|███▌      | 25/69 [00:03<00:04, 10.85it/s]
 39%|███▉      | 27/69 [00:03<00:03, 10.91it/s]
 42%|████▏     | 29/69 [00:04<00:03, 10.95it/s]
 45%|████▍     | 31/69 [00:04<00:03, 10.98it/s]
 48%|████▊     | 33/69 [00:04<00:03, 11.02it/s]
 51%|█████     | 35/69 [00:04<00:03, 11.05it/s]
 54%|█████▎    | 37/69 [00:04<00:02, 11.07it/s]
 57%|█████▋    | 39/69 [00:04<00:02, 11.09it/s]
 59%|█████▉    | 41/69 [00:05<00:02, 11.10it/s]
 62%|██████▏   | 43/69 [00:05<00:02, 11.12it/s]
 65%|██████▌   | 45/69 [00:05<00:02, 11.12it/s]
 68%|██████▊   | 47/69 [00:05<00:01, 11.13it/s]
 71%|███████   | 49/69 [00:05<00:01, 11.13it/s]
 74%|███████▍  | 51/69 [00:06<00:01, 11.13it/s]
 77%|███████▋  | 53/69 [00:06<00:01, 11.14it/s]
 80%|███████▉  | 55/69 [00:06<00:01, 11.14it/s]
 83%|████████▎ | 57/69 [00:06<00:01, 11.14it/s]
 86%|████████▌ | 59/69 [00:06<00:00, 11.14it/s]
 88%|████████▊ | 61/69 [00:06<00:00, 11.13it/s]
 91%|█████████▏| 63/69 [00:07<00:00, 11.14it/s]
 94%|█████████▍| 65/69 [00:07<00:00, 11.13it/s]
 97%|█████████▋| 67/69 [00:07<00:00, 11.13it/s]
100%|██████████| 69/69 [00:07<00:00, 12.05it/s]
100%|██████████| 69/69 [00:07<00:00,  8.99it/s]
INFO:Trainer:2024-12-05 17:04:45:mr: 244.334
INFO:Trainer:2024-12-05 17:04:45:mrr: 0.275
INFO:Trainer:2024-12-05 17:04:45:hits@1: 0.18
INFO:Trainer:2024-12-05 17:04:45:hits@3: 0.311
INFO:Trainer:2024-12-05 17:04:45:hits@10: 0.46
INFO:Trainer:2024-12-05 17:04:45:mr_type: 130.063
INFO:Trainer:2024-12-05 17:04:45:mrr_type: 0.321
INFO:Trainer:2024-12-05 17:04:45:hits@1_type: 0.23
INFO:Trainer:2024-12-05 17:04:45:hits@3_type: 0.352
INFO:Trainer:2024-12-05 17:04:45:hits@10_type: 0.502
INFO:EarlyStopping:2024-12-05 17:04:45:Validation score improved (-inf --> 0.460000).  Saving model ...
INFO:Trainer:2024-12-05 17:04:45:[cuda:0] Epoch 100 | Training checkpoint saved at ../../checkpoint/transh-100.pth
INFO:Trainer:2024-12-05 17:04:45:[cuda:0] Epoch [ 100/1000] | Batchsize: 8192 | loss:  0.499170 | 3.74025 seconds/epoch
INFO:Trainer:2024-12-05 17:04:45:[cuda:1] Epoch [ 100/1000] | Batchsize: 8192 | loss:  0.498411 | 3.74026 seconds/epoch
INFO:Trainer:2024-12-05 17:10:11:[cuda:0] Epoch 200 | The model starts evaluation on the validation set.

  0%|          | 0/69 [00:00<?, ?it/s]
  1%|▏         | 1/69 [00:00<00:58,  1.16it/s]
  4%|▍         | 3/69 [00:01<00:19,  3.43it/s]
  7%|▋         | 5/69 [00:01<00:12,  5.32it/s]
 10%|█         | 7/69 [00:01<00:09,  6.81it/s]
 13%|█▎        | 9/69 [00:01<00:07,  7.96it/s]
 16%|█▌        | 11/69 [00:01<00:06,  8.82it/s]
 19%|█▉        | 13/69 [00:01<00:05,  9.46it/s]
 22%|██▏       | 15/69 [00:02<00:05,  9.92it/s]
 25%|██▍       | 17/69 [00:02<00:05, 10.24it/s]
 28%|██▊       | 19/69 [00:02<00:04, 10.48it/s]
 30%|███       | 21/69 [00:02<00:04, 10.65it/s]
 33%|███▎      | 23/69 [00:02<00:04, 10.76it/s]
 36%|███▌      | 25/69 [00:03<00:04, 10.85it/s]
 39%|███▉      | 27/69 [00:03<00:03, 10.91it/s]
 42%|████▏     | 29/69 [00:03<00:03, 10.95it/s]
 45%|████▍     | 31/69 [00:03<00:03, 10.98it/s]
 48%|████▊     | 33/69 [00:03<00:03, 10.99it/s]
 51%|█████     | 35/69 [00:03<00:03, 11.00it/s]
 54%|█████▎    | 37/69 [00:04<00:02, 11.01it/s]
 57%|█████▋    | 39/69 [00:04<00:02, 11.03it/s]
 59%|█████▉    | 41/69 [00:04<00:02, 11.06it/s]
 62%|██████▏   | 43/69 [00:04<00:02, 11.09it/s]
 65%|██████▌   | 45/69 [00:04<00:02, 11.10it/s]
 68%|██████▊   | 47/69 [00:05<00:01, 11.11it/s]
 71%|███████   | 49/69 [00:05<00:01, 11.12it/s]
 74%|███████▍  | 51/69 [00:05<00:01, 11.12it/s]
 77%|███████▋  | 53/69 [00:05<00:01, 11.13it/s]
 80%|███████▉  | 55/69 [00:05<00:01, 11.10it/s]
 83%|████████▎ | 57/69 [00:05<00:01, 11.11it/s]
 86%|████████▌ | 59/69 [00:06<00:00, 11.12it/s]
 88%|████████▊ | 61/69 [00:06<00:00, 11.13it/s]
 91%|█████████▏| 63/69 [00:06<00:00, 11.13it/s]
 94%|█████████▍| 65/69 [00:06<00:00, 11.13it/s]
 97%|█████████▋| 67/69 [00:06<00:00, 11.14it/s]
100%|██████████| 69/69 [00:06<00:00, 12.06it/s]
100%|██████████| 69/69 [00:07<00:00,  9.74it/s]
INFO:Trainer:2024-12-05 17:10:18:mr: 225.999
INFO:Trainer:2024-12-05 17:10:18:mrr: 0.292
INFO:Trainer:2024-12-05 17:10:18:hits@1: 0.195
INFO:Trainer:2024-12-05 17:10:18:hits@3: 0.33
INFO:Trainer:2024-12-05 17:10:18:hits@10: 0.484
INFO:Trainer:2024-12-05 17:10:18:mr_type: 113.408
INFO:Trainer:2024-12-05 17:10:18:mrr_type: 0.339
INFO:Trainer:2024-12-05 17:10:18:hits@1_type: 0.246
INFO:Trainer:2024-12-05 17:10:18:hits@3_type: 0.373
INFO:Trainer:2024-12-05 17:10:18:hits@10_type: 0.527
INFO:EarlyStopping:2024-12-05 17:10:18:Validation score improved (0.460000 --> 0.484000).  Saving model ...
INFO:Trainer:2024-12-05 17:10:18:[cuda:0] Epoch 200 | Training checkpoint saved at ../../checkpoint/transh-200.pth
INFO:Trainer:2024-12-05 17:10:18:[cuda:1] Epoch [ 200/1000] | Batchsize: 8192 | loss:  0.373016 | 3.53623 seconds/epoch
INFO:Trainer:2024-12-05 17:10:18:[cuda:0] Epoch [ 200/1000] | Batchsize: 8192 | loss:  0.374192 | 3.53624 seconds/epoch
INFO:Trainer:2024-12-05 17:15:53:[cuda:0] Epoch 300 | The model starts evaluation on the validation set.

  0%|          | 0/69 [00:00<?, ?it/s]
  1%|▏         | 1/69 [00:00<00:57,  1.18it/s]
  4%|▍         | 3/69 [00:01<00:18,  3.49it/s]
  7%|▋         | 5/69 [00:01<00:11,  5.39it/s]
 10%|█         | 7/69 [00:01<00:08,  6.89it/s]
 13%|█▎        | 9/69 [00:01<00:07,  8.04it/s]
 16%|█▌        | 11/69 [00:01<00:06,  8.90it/s]
 19%|█▉        | 13/69 [00:01<00:05,  9.53it/s]
 22%|██▏       | 15/69 [00:02<00:05,  9.99it/s]
 25%|██▍       | 17/69 [00:02<00:05, 10.31it/s]
 28%|██▊       | 19/69 [00:02<00:04, 10.54it/s]
 30%|███       | 21/69 [00:02<00:04, 10.71it/s]
 33%|███▎      | 23/69 [00:02<00:04, 10.83it/s]
 36%|███▌      | 25/69 [00:03<00:04, 10.91it/s]
 39%|███▉      | 27/69 [00:03<00:03, 10.96it/s]
 42%|████▏     | 29/69 [00:03<00:03, 11.00it/s]
 45%|████▍     | 31/69 [00:03<00:03, 11.03it/s]
 48%|████▊     | 33/69 [00:03<00:03, 11.05it/s]
 51%|█████     | 35/69 [00:03<00:03, 11.07it/s]
 54%|█████▎    | 37/69 [00:04<00:02, 11.08it/s]
 57%|█████▋    | 39/69 [00:04<00:02, 11.10it/s]
 59%|█████▉    | 41/69 [00:04<00:02, 11.11it/s]
 62%|██████▏   | 43/69 [00:04<00:02, 11.11it/s]
 65%|██████▌   | 45/69 [00:04<00:02, 11.12it/s]
 68%|██████▊   | 47/69 [00:04<00:01, 11.12it/s]
 71%|███████   | 49/69 [00:05<00:01, 11.12it/s]
 74%|███████▍  | 51/69 [00:05<00:01, 11.13it/s]
 77%|███████▋  | 53/69 [00:05<00:01, 11.12it/s]
 80%|███████▉  | 55/69 [00:05<00:01, 11.12it/s]
 83%|████████▎ | 57/69 [00:05<00:01, 11.13it/s]
 86%|████████▌ | 59/69 [00:06<00:00, 11.13it/s]
 88%|████████▊ | 61/69 [00:06<00:00, 11.13it/s]
 91%|█████████▏| 63/69 [00:06<00:00, 11.13it/s]
 94%|█████████▍| 65/69 [00:06<00:00, 11.13it/s]
 97%|█████████▋| 67/69 [00:06<00:00, 11.13it/s]
100%|██████████| 69/69 [00:06<00:00, 12.06it/s]
100%|██████████| 69/69 [00:07<00:00,  9.81it/s]
INFO:Trainer:2024-12-05 17:16:00:mr: 237.583
INFO:Trainer:2024-12-05 17:16:00:mrr: 0.296
INFO:Trainer:2024-12-05 17:16:00:hits@1: 0.197
INFO:Trainer:2024-12-05 17:16:00:hits@3: 0.336
INFO:Trainer:2024-12-05 17:16:00:hits@10: 0.492
INFO:Trainer:2024-12-05 17:16:00:mr_type: 111.196
INFO:Trainer:2024-12-05 17:16:00:mrr_type: 0.344
INFO:Trainer:2024-12-05 17:16:00:hits@1_type: 0.248
INFO:Trainer:2024-12-05 17:16:00:hits@3_type: 0.381
INFO:Trainer:2024-12-05 17:16:00:hits@10_type: 0.536
INFO:EarlyStopping:2024-12-05 17:16:00:EarlyStopping counter: 1 / 2
INFO:Trainer:2024-12-05 17:16:00:[cuda:0] Epoch 300 | Training checkpoint saved at ../../checkpoint/transh-300.pth
INFO:Trainer:2024-12-05 17:16:00:[cuda:1] Epoch [ 300/1000] | Batchsize: 8192 | loss:  0.318348 | 3.49771 seconds/epoch
INFO:Trainer:2024-12-05 17:16:00:[cuda:0] Epoch [ 300/1000] | Batchsize: 8192 | loss:  0.320309 | 3.49767 seconds/epoch
INFO:Trainer:2024-12-05 17:21:33:[cuda:0] Epoch 400 | The model starts evaluation on the validation set.

  0%|          | 0/69 [00:00<?, ?it/s]
  1%|▏         | 1/69 [00:00<01:00,  1.12it/s]
  4%|▍         | 3/69 [00:01<00:19,  3.34it/s]
  7%|▋         | 5/69 [00:01<00:12,  5.21it/s]
 10%|█         | 7/69 [00:01<00:09,  6.71it/s]
 13%|█▎        | 9/69 [00:01<00:07,  7.88it/s]
 16%|█▌        | 11/69 [00:01<00:06,  8.76it/s]
 19%|█▉        | 13/69 [00:01<00:05,  9.41it/s]
 22%|██▏       | 15/69 [00:02<00:05,  9.88it/s]
 25%|██▍       | 17/69 [00:02<00:05, 10.21it/s]
 28%|██▊       | 19/69 [00:02<00:04, 10.45it/s]
 30%|███       | 21/69 [00:02<00:04, 10.63it/s]
 33%|███▎      | 23/69 [00:02<00:04, 10.75it/s]
 36%|███▌      | 25/69 [00:03<00:04, 10.84it/s]
 39%|███▉      | 27/69 [00:03<00:03, 10.90it/s]
 42%|████▏     | 29/69 [00:03<00:03, 10.94it/s]
 45%|████▍     | 31/69 [00:03<00:03, 10.97it/s]
 48%|████▊     | 33/69 [00:03<00:03, 10.98it/s]
 51%|█████     | 35/69 [00:03<00:03, 11.00it/s]
 54%|█████▎    | 37/69 [00:04<00:02, 11.01it/s]
 57%|█████▋    | 39/69 [00:04<00:02, 11.04it/s]
 59%|█████▉    | 41/69 [00:04<00:02, 11.07it/s]
 62%|██████▏   | 43/69 [00:04<00:02, 11.09it/s]
 65%|██████▌   | 45/69 [00:04<00:02, 11.10it/s]
 68%|██████▊   | 47/69 [00:05<00:01, 11.08it/s]
 71%|███████   | 49/69 [00:05<00:01, 11.10it/s]
 74%|███████▍  | 51/69 [00:05<00:01, 11.11it/s]
 77%|███████▋  | 53/69 [00:05<00:01, 11.12it/s]
 80%|███████▉  | 55/69 [00:05<00:01, 11.12it/s]
 83%|████████▎ | 57/69 [00:05<00:01, 11.13it/s]
 86%|████████▌ | 59/69 [00:06<00:00, 11.13it/s]
 88%|████████▊ | 61/69 [00:06<00:00, 11.13it/s]
 91%|█████████▏| 63/69 [00:06<00:00, 11.13it/s]
 94%|█████████▍| 65/69 [00:06<00:00, 11.13it/s]
 97%|█████████▋| 67/69 [00:06<00:00, 11.13it/s]
100%|██████████| 69/69 [00:06<00:00, 12.05it/s]
100%|██████████| 69/69 [00:07<00:00,  9.72it/s]
INFO:Trainer:2024-12-05 17:21:41:mr: 235.165
INFO:Trainer:2024-12-05 17:21:41:mrr: 0.302
INFO:Trainer:2024-12-05 17:21:41:hits@1: 0.204
INFO:Trainer:2024-12-05 17:21:41:hits@3: 0.342
INFO:Trainer:2024-12-05 17:21:41:hits@10: 0.497
INFO:Trainer:2024-12-05 17:21:41:mr_type: 109.444
INFO:Trainer:2024-12-05 17:21:41:mrr_type: 0.352
INFO:Trainer:2024-12-05 17:21:41:hits@1_type: 0.258
INFO:Trainer:2024-12-05 17:21:41:hits@3_type: 0.386
INFO:Trainer:2024-12-05 17:21:41:hits@10_type: 0.542
INFO:EarlyStopping:2024-12-05 17:21:41:Validation score improved (0.484000 --> 0.497000).  Saving model ...
INFO:Trainer:2024-12-05 17:21:41:[cuda:0] Epoch 400 | Training checkpoint saved at ../../checkpoint/transh-400.pth
INFO:Trainer:2024-12-05 17:21:41:[cuda:1] Epoch [ 400/1000] | Batchsize: 8192 | loss:  0.243966 | 3.47454 seconds/epoch
INFO:Trainer:2024-12-05 17:21:41:[cuda:0] Epoch [ 400/1000] | Batchsize: 8192 | loss:  0.242802 | 3.47454 seconds/epoch
INFO:Trainer:2024-12-05 17:27:06:[cuda:0] Epoch 500 | The model starts evaluation on the validation set.

  0%|          | 0/69 [00:00<?, ?it/s]
  1%|▏         | 1/69 [00:00<00:57,  1.19it/s]
  4%|▍         | 3/69 [00:01<00:18,  3.50it/s]
  7%|▋         | 5/69 [00:01<00:11,  5.39it/s]
 10%|█         | 7/69 [00:01<00:08,  6.89it/s]
 13%|█▎        | 9/69 [00:01<00:07,  8.04it/s]
 16%|█▌        | 11/69 [00:01<00:06,  8.91it/s]
 19%|█▉        | 13/69 [00:01<00:05,  9.54it/s]
 22%|██▏       | 15/69 [00:02<00:05,  9.99it/s]
 25%|██▍       | 17/69 [00:02<00:05, 10.31it/s]
 28%|██▊       | 19/69 [00:02<00:04, 10.55it/s]
 30%|███       | 21/69 [00:02<00:04, 10.71it/s]
 33%|███▎      | 23/69 [00:02<00:04, 10.83it/s]
 36%|███▌      | 25/69 [00:03<00:04, 10.92it/s]
 39%|███▉      | 27/69 [00:03<00:03, 10.98it/s]
 42%|████▏     | 29/69 [00:03<00:03, 11.02it/s]
 45%|████▍     | 31/69 [00:03<00:03, 11.04it/s]
 48%|████▊     | 33/69 [00:03<00:03, 11.05it/s]
 51%|█████     | 35/69 [00:03<00:03, 11.07it/s]
 54%|█████▎    | 37/69 [00:04<00:02, 11.08it/s]
 57%|█████▋    | 39/69 [00:04<00:02, 11.10it/s]
 59%|█████▉    | 41/69 [00:04<00:02, 11.11it/s]
 62%|██████▏   | 43/69 [00:04<00:02, 11.11it/s]
 65%|██████▌   | 45/69 [00:04<00:02, 11.12it/s]
 68%|██████▊   | 47/69 [00:04<00:01, 11.13it/s]
 71%|███████   | 49/69 [00:05<00:01, 11.13it/s]
 74%|███████▍  | 51/69 [00:05<00:01, 11.13it/s]
 77%|███████▋  | 53/69 [00:05<00:01, 11.13it/s]
 80%|███████▉  | 55/69 [00:05<00:01, 11.13it/s]
 83%|████████▎ | 57/69 [00:05<00:01, 11.13it/s]
 86%|████████▌ | 59/69 [00:06<00:00, 11.13it/s]
 88%|████████▊ | 61/69 [00:06<00:00, 11.13it/s]
 91%|█████████▏| 63/69 [00:06<00:00, 11.11it/s]
 94%|█████████▍| 65/69 [00:06<00:00, 11.12it/s]
 97%|█████████▋| 67/69 [00:06<00:00, 11.12it/s]
100%|██████████| 69/69 [00:06<00:00, 12.05it/s]
100%|██████████| 69/69 [00:07<00:00,  9.81it/s]
INFO:Trainer:2024-12-05 17:27:13:mr: 239.586
INFO:Trainer:2024-12-05 17:27:13:mrr: 0.301
INFO:Trainer:2024-12-05 17:27:13:hits@1: 0.202
INFO:Trainer:2024-12-05 17:27:13:hits@3: 0.341
INFO:Trainer:2024-12-05 17:27:13:hits@10: 0.497
INFO:Trainer:2024-12-05 17:27:13:mr_type: 110.079
INFO:Trainer:2024-12-05 17:27:13:mrr_type: 0.352
INFO:Trainer:2024-12-05 17:27:13:hits@1_type: 0.258
INFO:Trainer:2024-12-05 17:27:13:hits@3_type: 0.387
INFO:Trainer:2024-12-05 17:27:13:hits@10_type: 0.543
INFO:EarlyStopping:2024-12-05 17:27:13:EarlyStopping counter: 1 / 2
INFO:Trainer:2024-12-05 17:27:13:[cuda:0] Epoch 500 | Training checkpoint saved at ../../checkpoint/transh-500.pth
INFO:Trainer:2024-12-05 17:27:13:[cuda:1] Epoch [ 500/1000] | Batchsize: 8192 | loss:  0.237786 | 3.44533 seconds/epoch
INFO:Trainer:2024-12-05 17:27:13:[cuda:0] Epoch [ 500/1000] | Batchsize: 8192 | loss:  0.238102 | 3.44533 seconds/epoch
INFO:Trainer:2024-12-05 17:32:40:[cuda:0] Epoch 600 | The model starts evaluation on the validation set.

  0%|          | 0/69 [00:00<?, ?it/s]
  1%|▏         | 1/69 [00:00<00:58,  1.15it/s]
  4%|▍         | 3/69 [00:01<00:19,  3.43it/s]
  7%|▋         | 5/69 [00:01<00:12,  5.32it/s]
 10%|█         | 7/69 [00:01<00:09,  6.82it/s]
 13%|█▎        | 9/69 [00:01<00:07,  7.98it/s]
 16%|█▌        | 11/69 [00:01<00:06,  8.85it/s]
 19%|█▉        | 13/69 [00:01<00:05,  9.49it/s]
 22%|██▏       | 15/69 [00:02<00:05,  9.95it/s]
 25%|██▍       | 17/69 [00:02<00:05, 10.29it/s]
 28%|██▊       | 19/69 [00:02<00:04, 10.53it/s]
 30%|███       | 21/69 [00:02<00:04, 10.70it/s]
 33%|███▎      | 23/69 [00:02<00:04, 10.82it/s]
 36%|███▌      | 25/69 [00:03<00:04, 10.90it/s]
 39%|███▉      | 27/69 [00:03<00:03, 10.95it/s]
 42%|████▏     | 29/69 [00:03<00:03, 11.00it/s]
 45%|████▍     | 31/69 [00:03<00:03, 11.03it/s]
 48%|████▊     | 33/69 [00:03<00:03, 11.05it/s]
 51%|█████     | 35/69 [00:03<00:03, 11.07it/s]
 54%|█████▎    | 37/69 [00:04<00:02, 11.08it/s]
 57%|█████▋    | 39/69 [00:04<00:02, 11.10it/s]
 59%|█████▉    | 41/69 [00:04<00:02, 11.11it/s]
 62%|██████▏   | 43/69 [00:04<00:02, 11.12it/s]
 65%|██████▌   | 45/69 [00:04<00:02, 11.12it/s]
 68%|██████▊   | 47/69 [00:05<00:01, 11.12it/s]
 71%|███████   | 49/69 [00:05<00:01, 11.13it/s]
 74%|███████▍  | 51/69 [00:05<00:01, 11.13it/s]
 77%|███████▋  | 53/69 [00:05<00:01, 11.13it/s]
 80%|███████▉  | 55/69 [00:05<00:01, 11.13it/s]
 83%|████████▎ | 57/69 [00:05<00:01, 11.13it/s]
 86%|████████▌ | 59/69 [00:06<00:00, 11.13it/s]
 88%|████████▊ | 61/69 [00:06<00:00, 11.13it/s]
 91%|█████████▏| 63/69 [00:06<00:00, 11.14it/s]
 94%|█████████▍| 65/69 [00:06<00:00, 11.14it/s]
 97%|█████████▋| 67/69 [00:06<00:00, 11.14it/s]
100%|██████████| 69/69 [00:06<00:00, 12.06it/s]
100%|██████████| 69/69 [00:07<00:00,  9.78it/s]
INFO:Trainer:2024-12-05 17:32:47:mr: 242.557
INFO:Trainer:2024-12-05 17:32:47:mrr: 0.299
INFO:Trainer:2024-12-05 17:32:47:hits@1: 0.201
INFO:Trainer:2024-12-05 17:32:47:hits@3: 0.34
INFO:Trainer:2024-12-05 17:32:47:hits@10: 0.496
INFO:Trainer:2024-12-05 17:32:47:mr_type: 110.379
INFO:Trainer:2024-12-05 17:32:47:mrr_type: 0.351
INFO:Trainer:2024-12-05 17:32:47:hits@1_type: 0.257
INFO:Trainer:2024-12-05 17:32:47:hits@3_type: 0.386
INFO:Trainer:2024-12-05 17:32:47:hits@10_type: 0.542
INFO:EarlyStopping:2024-12-05 17:32:47:EarlyStopping counter: 2 / 2
INFO:Trainer:2024-12-05 17:32:47:[cuda:0] Send an early stopping signal
INFO:Trainer:2024-12-05 17:32:47:[cuda:0] Epoch 600 | Training checkpoint saved at ../../checkpoint/transh-600.pth
INFO:Trainer:2024-12-05 17:32:47:[cuda:0] Early stopping
INFO:Trainer:2024-12-05 17:32:47:[cuda:1] Early stopping
INFO:Trainer:2024-12-05 17:32:47:[cuda:1] The model training is completed, taking a total of 2055.94791 seconds.
INFO:Trainer:2024-12-05 17:32:47:[cuda:0] The model training is completed, taking a total of 2055.95156 seconds.
INFO:Trainer:2024-12-05 17:32:47:[cuda:0] Model saved at ../../checkpoint/transh.pth.
INFO:Trainer:2024-12-05 17:32:47:[cuda:0] The model starts evaluating in the test set.

  0%|          | 0/80 [00:00<?, ?it/s]
  1%|▏         | 1/80 [00:00<01:11,  1.11it/s]
  4%|▍         | 3/80 [00:01<00:23,  3.31it/s]
  6%|▋         | 5/80 [00:01<00:14,  5.17it/s]
  9%|▉         | 7/80 [00:01<00:10,  6.66it/s]
 11%|█▏        | 9/80 [00:01<00:09,  7.84it/s]
 14%|█▍        | 11/80 [00:01<00:07,  8.72it/s]
 16%|█▋        | 13/80 [00:01<00:07,  9.38it/s]
 19%|█▉        | 15/80 [00:02<00:06,  9.86it/s]
 21%|██▏       | 17/80 [00:02<00:06, 10.20it/s]
 24%|██▍       | 19/80 [00:02<00:05, 10.45it/s]
 26%|██▋       | 21/80 [00:02<00:05, 10.62it/s]
 29%|██▉       | 23/80 [00:02<00:05, 10.74it/s]
 31%|███▏      | 25/80 [00:03<00:05, 10.83it/s]
 34%|███▍      | 27/80 [00:03<00:04, 10.89it/s]
 36%|███▋      | 29/80 [00:03<00:04, 10.93it/s]
 39%|███▉      | 31/80 [00:03<00:04, 10.97it/s]
 41%|████▏     | 33/80 [00:03<00:04, 11.01it/s]
 44%|████▍     | 35/80 [00:03<00:04, 11.04it/s]
 46%|████▋     | 37/80 [00:04<00:03, 11.06it/s]
 49%|████▉     | 39/80 [00:04<00:03, 11.08it/s]
 51%|█████▏    | 41/80 [00:04<00:03, 11.09it/s]
 54%|█████▍    | 43/80 [00:04<00:03, 11.09it/s]
 56%|█████▋    | 45/80 [00:04<00:03, 11.10it/s]
 59%|█████▉    | 47/80 [00:05<00:02, 11.10it/s]
 61%|██████▏   | 49/80 [00:05<00:02, 11.10it/s]
 64%|██████▍   | 51/80 [00:05<00:02, 11.11it/s]
 66%|██████▋   | 53/80 [00:05<00:02, 11.12it/s]
 69%|██████▉   | 55/80 [00:05<00:02, 11.13it/s]
 71%|███████▏  | 57/80 [00:05<00:02, 11.13it/s]
 74%|███████▍  | 59/80 [00:06<00:01, 11.13it/s]
 76%|███████▋  | 61/80 [00:06<00:01, 11.13it/s]
 79%|███████▉  | 63/80 [00:06<00:01, 11.13it/s]
 81%|████████▏ | 65/80 [00:06<00:01, 11.13it/s]
 84%|████████▍ | 67/80 [00:06<00:01, 11.14it/s]
 86%|████████▋ | 69/80 [00:07<00:00, 11.14it/s]
 89%|████████▉ | 71/80 [00:07<00:00, 11.13it/s]
 91%|█████████▏| 73/80 [00:07<00:00, 11.14it/s]
 94%|█████████▍| 75/80 [00:07<00:00, 11.14it/s]
 96%|█████████▋| 77/80 [00:07<00:00, 11.14it/s]
 99%|█████████▉| 79/80 [00:07<00:00, 11.13it/s]
100%|██████████| 80/80 [00:08<00:00,  9.83it/s]
INFO:Trainer:2024-12-05 17:32:55:mr: 256.942
INFO:Trainer:2024-12-05 17:32:55:mrr: 0.293
INFO:Trainer:2024-12-05 17:32:55:hits@1: 0.194
INFO:Trainer:2024-12-05 17:32:55:hits@3: 0.332
INFO:Trainer:2024-12-05 17:32:55:hits@10: 0.493
INFO:Trainer:2024-12-05 17:32:55:mr_type: 112.139
INFO:Trainer:2024-12-05 17:32:55:mrr_type: 0.344
INFO:Trainer:2024-12-05 17:32:55:hits@1_type: 0.248
INFO:Trainer:2024-12-05 17:32:55:hits@3_type: 0.38
INFO:Trainer:2024-12-05 17:32:55:hits@10_type: 0.537