Using RTX 4000 series which doesn't support faster communication speedups. Ensuring P2P and IB communications are disabled.
DEBUG:cmd:2024-12-05 18:50:19:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False)
DEBUG:cmd:2024-12-05 18:50:19:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False)
DEBUG:cmd:2024-12-05 18:50:19:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False)
DEBUG:cmd:2024-12-05 18:50:19:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False)
DEBUG:connectionpool:2024-12-05 18:50:19:Starting new HTTPS connection (1): api.wandb.ai:443
DEBUG:connectionpool:2024-12-05 18:50:20:Starting new HTTPS connection (1): api.wandb.ai:443
DEBUG:connectionpool:2024-12-05 18:50:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
DEBUG:connectionpool:2024-12-05 18:50:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
DEBUG:connectionpool:2024-12-05 18:50:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
wandb: Currently logged in as: 3555028709. Use `wandb login --relogin` to force relogin
DEBUG:connectionpool:2024-12-05 18:50:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
wandb: Currently logged in as: 3555028709. Use `wandb login --relogin` to force relogin
DEBUG:cmd:2024-12-05 18:50:20:Popen(['git', 'cat-file', '--batch-check'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=<valid stream>, shell=False, universal_newlines=False)
DEBUG:cmd:2024-12-05 18:50:20:Popen(['git', 'cat-file', '--batch-check'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=<valid stream>, shell=False, universal_newlines=False)
wandb: Tracking run with wandb version 0.16.6
wandb: Run data is saved locally in /home/luyanfeng/my_code/github/pybind11-OpenKE/examples/TransH/wandb/run-20240512_185020-q2t7lym7
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run TransH-FB15K237-multi
wandb: ⭐️ View project at https://wandb.ai/3555028709/pybind11-ke
wandb: 🚀 View run at https://wandb.ai/3555028709/pybind11-ke/runs/q2t7lym7
wandb: Tracking run with wandb version 0.16.6
wandb: Run data is saved locally in /home/luyanfeng/my_code/github/pybind11-OpenKE/examples/TransH/wandb/run-20240512_185020-x9qosnku
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run TransH-FB15K237-multi
wandb: ⭐️ View project at https://wandb.ai/3555028709/pybind11-ke
wandb: 🚀 View run at https://wandb.ai/3555028709/pybind11-ke/runs/x9qosnku
INFO:distributed_c10d:2024-12-05 18:50:27:Added key: store_based_barrier_key:1 to store for rank: 1
INFO:distributed_c10d:2024-12-05 18:50:28:Added key: store_based_barrier_key:1 to store for rank: 0
INFO:distributed_c10d:2024-12-05 18:50:28:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
INFO:distributed_c10d:2024-12-05 18:50:28:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
INFO:Trainer:2024-12-05 18:50:29:[cuda:0] Initialization completed, start model training.
INFO:Trainer:2024-12-05 18:50:29:[cuda:1] Initialization completed, start model training.
INFO:distributed:2024-12-05 18:50:31:Reducer buckets have been rebuilt in this iteration.
INFO:distributed:2024-12-05 18:50:31:Reducer buckets have been rebuilt in this iteration.
INFO:Trainer:2024-12-05 18:56:00:[cuda:0] Epoch 100 | The model starts evaluation on the validation set.

  0%|          | 0/69 [00:00<?, ?it/s]
  1%|▏         | 1/69 [00:01<01:28,  1.30s/it]
  3%|▎         | 2/69 [00:01<00:40,  1.67it/s]
  6%|▌         | 4/69 [00:01<00:18,  3.50it/s]
  9%|▊         | 6/69 [00:01<00:12,  5.10it/s]
 10%|█         | 7/69 [00:01<00:10,  5.78it/s]
 12%|█▏        | 8/69 [00:02<00:09,  6.50it/s]
 14%|█▍        | 10/69 [00:02<00:07,  7.83it/s]
 17%|█▋        | 12/69 [00:02<00:06,  8.82it/s]
 20%|██        | 14/69 [00:02<00:05,  9.51it/s]
 23%|██▎       | 16/69 [00:02<00:05,  9.99it/s]
 26%|██▌       | 18/69 [00:02<00:04, 10.31it/s]
 29%|██▉       | 20/69 [00:03<00:04, 10.54it/s]
 32%|███▏      | 22/69 [00:03<00:04, 10.71it/s]
 35%|███▍      | 24/69 [00:03<00:04, 10.83it/s]
 38%|███▊      | 26/69 [00:03<00:03, 10.92it/s]
 41%|████      | 28/69 [00:03<00:03, 10.98it/s]
 43%|████▎     | 30/69 [00:03<00:03, 11.02it/s]
 46%|████▋     | 32/69 [00:04<00:03, 11.03it/s]
 49%|████▉     | 34/69 [00:04<00:03, 11.06it/s]
 52%|█████▏    | 36/69 [00:04<00:02, 11.08it/s]
 55%|█████▌    | 38/69 [00:04<00:02, 11.08it/s]
 58%|█████▊    | 40/69 [00:04<00:02, 11.10it/s]
 61%|██████    | 42/69 [00:05<00:02, 11.11it/s]
 64%|██████▍   | 44/69 [00:05<00:02, 11.12it/s]
 67%|██████▋   | 46/69 [00:05<00:02, 11.13it/s]
 70%|██████▉   | 48/69 [00:05<00:01, 11.13it/s]
 72%|███████▏  | 50/69 [00:05<00:01, 11.14it/s]
 75%|███████▌  | 52/69 [00:05<00:01, 11.14it/s]
 78%|███████▊  | 54/69 [00:06<00:01, 11.14it/s]
 81%|████████  | 56/69 [00:06<00:01, 11.14it/s]
 84%|████████▍ | 58/69 [00:06<00:00, 11.14it/s]
 87%|████████▋ | 60/69 [00:06<00:00, 11.14it/s]
 90%|████████▉ | 62/69 [00:06<00:00, 11.14it/s]
 93%|█████████▎| 64/69 [00:07<00:00, 11.14it/s]
 96%|█████████▌| 66/69 [00:07<00:00, 11.14it/s]
 99%|█████████▊| 68/69 [00:07<00:00, 11.14it/s]
100%|██████████| 69/69 [00:07<00:00,  9.14it/s]
INFO:Trainer:2024-12-05 18:56:07:mr: 238.165
INFO:Trainer:2024-12-05 18:56:07:mrr: 0.272
INFO:Trainer:2024-12-05 18:56:07:hits@1: 0.177
INFO:Trainer:2024-12-05 18:56:07:hits@3: 0.31
INFO:Trainer:2024-12-05 18:56:07:hits@10: 0.46
INFO:Trainer:2024-12-05 18:56:07:mr_type: 127.422
INFO:Trainer:2024-12-05 18:56:07:mrr_type: 0.319
INFO:Trainer:2024-12-05 18:56:07:hits@1_type: 0.229
INFO:Trainer:2024-12-05 18:56:07:hits@3_type: 0.349
INFO:Trainer:2024-12-05 18:56:07:hits@10_type: 0.501
INFO:EarlyStopping:2024-12-05 18:56:07:Validation score improved (-inf --> 0.460000).  Saving model ...
INFO:Trainer:2024-12-05 18:56:08:[cuda:0] Epoch 100 | Training checkpoint saved at ../../checkpoint/transh-100.pth
INFO:Trainer:2024-12-05 18:56:08:[cuda:0] Epoch [ 100/1000] | Batchsize: 8192 | loss:  0.498154 | 3.30994 seconds/epoch
INFO:Trainer:2024-12-05 18:56:08:[cuda:1] Epoch [ 100/1000] | Batchsize: 8192 | loss:  0.504389 | 3.30982 seconds/epoch
INFO:Trainer:2024-12-05 19:01:45:[cuda:0] Epoch 200 | The model starts evaluation on the validation set.

  0%|          | 0/69 [00:00<?, ?it/s]
  1%|▏         | 1/69 [00:00<00:55,  1.23it/s]
  4%|▍         | 3/69 [00:00<00:18,  3.60it/s]
  7%|▋         | 5/69 [00:01<00:11,  5.51it/s]
 10%|█         | 7/69 [00:01<00:08,  6.99it/s]
 13%|█▎        | 9/69 [00:01<00:07,  8.12it/s]
 16%|█▌        | 11/69 [00:01<00:06,  8.95it/s]
 19%|█▉        | 13/69 [00:01<00:05,  9.55it/s]
 22%|██▏       | 15/69 [00:02<00:05,  9.98it/s]
 25%|██▍       | 17/69 [00:02<00:05, 10.29it/s]
 28%|██▊       | 19/69 [00:02<00:04, 10.51it/s]
 30%|███       | 21/69 [00:02<00:04, 10.67it/s]
 33%|███▎      | 23/69 [00:02<00:04, 10.78it/s]
 36%|███▌      | 25/69 [00:02<00:04, 10.86it/s]
 39%|███▉      | 27/69 [00:03<00:03, 10.92it/s]
 42%|████▏     | 29/69 [00:03<00:03, 10.95it/s]
 45%|████▍     | 31/69 [00:03<00:03, 10.98it/s]
 48%|████▊     | 33/69 [00:03<00:03, 11.00it/s]
 51%|█████     | 35/69 [00:03<00:03, 11.01it/s]
 54%|█████▎    | 37/69 [00:04<00:02, 11.02it/s]
 57%|█████▋    | 39/69 [00:04<00:02, 11.04it/s]
 59%|█████▉    | 41/69 [00:04<00:02, 11.05it/s]
 62%|██████▏   | 43/69 [00:04<00:02, 11.06it/s]
 65%|██████▌   | 45/69 [00:04<00:02, 11.06it/s]
 68%|██████▊   | 47/69 [00:04<00:01, 11.05it/s]
 71%|███████   | 49/69 [00:05<00:01, 11.06it/s]
 74%|███████▍  | 51/69 [00:05<00:01, 11.07it/s]
 77%|███████▋  | 53/69 [00:05<00:01, 11.09it/s]
 80%|███████▉  | 55/69 [00:05<00:01, 11.10it/s]
 83%|████████▎ | 57/69 [00:05<00:01, 11.11it/s]
 86%|████████▌ | 59/69 [00:06<00:00, 11.12it/s]
 88%|████████▊ | 61/69 [00:06<00:00, 11.13it/s]
 91%|█████████▏| 63/69 [00:06<00:00, 11.13it/s]
 94%|█████████▍| 65/69 [00:06<00:00, 11.13it/s]
 97%|█████████▋| 67/69 [00:06<00:00, 11.14it/s]
100%|██████████| 69/69 [00:06<00:00, 12.07it/s]
100%|██████████| 69/69 [00:07<00:00,  9.84it/s]
INFO:Trainer:2024-12-05 19:01:52:mr: 224.531
INFO:Trainer:2024-12-05 19:01:52:mrr: 0.292
INFO:Trainer:2024-12-05 19:01:52:hits@1: 0.194
INFO:Trainer:2024-12-05 19:01:52:hits@3: 0.331
INFO:Trainer:2024-12-05 19:01:52:hits@10: 0.485
INFO:Trainer:2024-12-05 19:01:52:mr_type: 113.473
INFO:Trainer:2024-12-05 19:01:52:mrr_type: 0.339
INFO:Trainer:2024-12-05 19:01:52:hits@1_type: 0.246
INFO:Trainer:2024-12-05 19:01:52:hits@3_type: 0.373
INFO:Trainer:2024-12-05 19:01:52:hits@10_type: 0.529
INFO:EarlyStopping:2024-12-05 19:01:52:Validation score improved (0.460000 --> 0.485000).  Saving model ...
INFO:Trainer:2024-12-05 19:01:52:[cuda:0] Epoch 200 | Training checkpoint saved at ../../checkpoint/transh-200.pth
INFO:Trainer:2024-12-05 19:01:52:[cuda:1] Epoch [ 200/1000] | Batchsize: 8192 | loss:  0.370109 | 3.38214 seconds/epoch
INFO:Trainer:2024-12-05 19:01:52:[cuda:0] Epoch [ 200/1000] | Batchsize: 8192 | loss:  0.371922 | 3.38220 seconds/epoch
INFO:Trainer:2024-12-05 19:07:22:[cuda:0] Epoch 300 | The model starts evaluation on the validation set.

  0%|          | 0/69 [00:00<?, ?it/s]
  1%|▏         | 1/69 [00:00<00:59,  1.15it/s]
  4%|▍         | 3/69 [00:01<00:19,  3.41it/s]
  7%|▋         | 5/69 [00:01<00:12,  5.29it/s]
 10%|█         | 7/69 [00:01<00:09,  6.79it/s]
 13%|█▎        | 9/69 [00:01<00:07,  7.94it/s]
 16%|█▌        | 11/69 [00:01<00:06,  8.81it/s]
 19%|█▉        | 13/69 [00:01<00:05,  9.44it/s]
 22%|██▏       | 15/69 [00:02<00:05,  9.90it/s]
 25%|██▍       | 17/69 [00:02<00:05, 10.24it/s]
 28%|██▊       | 19/69 [00:02<00:04, 10.48it/s]
 30%|███       | 21/69 [00:02<00:04, 10.65it/s]
 33%|███▎      | 23/69 [00:02<00:04, 10.77it/s]
 36%|███▌      | 25/69 [00:03<00:04, 10.85it/s]
 39%|███▉      | 27/69 [00:03<00:03, 10.91it/s]
 42%|████▏     | 29/69 [00:03<00:03, 10.95it/s]
 45%|████▍     | 31/69 [00:03<00:03, 10.97it/s]
 48%|████▊     | 33/69 [00:03<00:03, 11.00it/s]
 51%|█████     | 35/69 [00:03<00:03, 11.01it/s]
 54%|█████▎    | 37/69 [00:04<00:02, 11.03it/s]
 57%|█████▋    | 39/69 [00:04<00:02, 11.04it/s]
 59%|█████▉    | 41/69 [00:04<00:02, 11.07it/s]
 62%|██████▏   | 43/69 [00:04<00:02, 11.09it/s]
 65%|██████▌   | 45/69 [00:04<00:02, 11.10it/s]
 68%|██████▊   | 47/69 [00:05<00:01, 11.11it/s]
 71%|███████   | 49/69 [00:05<00:01, 11.12it/s]
 74%|███████▍  | 51/69 [00:05<00:01, 11.13it/s]
 77%|███████▋  | 53/69 [00:05<00:01, 11.13it/s]
 80%|███████▉  | 55/69 [00:05<00:01, 11.13it/s]
 83%|████████▎ | 57/69 [00:05<00:01, 11.13it/s]
 86%|████████▌ | 59/69 [00:06<00:00, 11.13it/s]
 88%|████████▊ | 61/69 [00:06<00:00, 11.13it/s]
 91%|█████████▏| 63/69 [00:06<00:00, 11.13it/s]
 94%|█████████▍| 65/69 [00:06<00:00, 11.12it/s]
 97%|█████████▋| 67/69 [00:06<00:00, 11.12it/s]
100%|██████████| 69/69 [00:06<00:00, 12.05it/s]
100%|██████████| 69/69 [00:07<00:00,  9.76it/s]
INFO:Trainer:2024-12-05 19:07:29:mr: 227.772
INFO:Trainer:2024-12-05 19:07:29:mrr: 0.295
INFO:Trainer:2024-12-05 19:07:29:hits@1: 0.196
INFO:Trainer:2024-12-05 19:07:29:hits@3: 0.336
INFO:Trainer:2024-12-05 19:07:29:hits@10: 0.488
INFO:Trainer:2024-12-05 19:07:29:mr_type: 110.917
INFO:Trainer:2024-12-05 19:07:29:mrr_type: 0.344
INFO:Trainer:2024-12-05 19:07:29:hits@1_type: 0.25
INFO:Trainer:2024-12-05 19:07:29:hits@3_type: 0.38
INFO:Trainer:2024-12-05 19:07:29:hits@10_type: 0.535
INFO:EarlyStopping:2024-12-05 19:07:29:EarlyStopping counter: 1 / 2
INFO:Trainer:2024-12-05 19:07:29:[cuda:0] Epoch 300 | Training checkpoint saved at ../../checkpoint/transh-300.pth
INFO:Trainer:2024-12-05 19:07:29:[cuda:1] Epoch [ 300/1000] | Batchsize: 8192 | loss:  0.321621 | 3.37550 seconds/epoch
INFO:Trainer:2024-12-05 19:07:29:[cuda:0] Epoch [ 300/1000] | Batchsize: 8192 | loss:  0.322573 | 3.37551 seconds/epoch
INFO:Trainer:2024-12-05 19:13:04:[cuda:0] Epoch 400 | The model starts evaluation on the validation set.

  0%|          | 0/69 [00:00<?, ?it/s]
  1%|▏         | 1/69 [00:00<01:00,  1.13it/s]
  4%|▍         | 3/69 [00:01<00:19,  3.36it/s]
  7%|▋         | 5/69 [00:01<00:12,  5.23it/s]
 10%|█         | 7/69 [00:01<00:09,  6.73it/s]
 13%|█▎        | 9/69 [00:01<00:07,  7.89it/s]
 16%|█▌        | 11/69 [00:01<00:06,  8.76it/s]
 19%|█▉        | 13/69 [00:01<00:05,  9.40it/s]
 22%|██▏       | 15/69 [00:02<00:05,  9.87it/s]
 25%|██▍       | 17/69 [00:02<00:05, 10.20it/s]
 28%|██▊       | 19/69 [00:02<00:04, 10.45it/s]
 30%|███       | 21/69 [00:02<00:04, 10.62it/s]
 33%|███▎      | 23/69 [00:02<00:04, 10.74it/s]
 36%|███▌      | 25/69 [00:03<00:04, 10.83it/s]
 39%|███▉      | 27/69 [00:03<00:03, 10.89it/s]
 42%|████▏     | 29/69 [00:03<00:03, 10.93it/s]
 45%|████▍     | 31/69 [00:03<00:03, 10.96it/s]
 48%|████▊     | 33/69 [00:03<00:03, 10.98it/s]
 51%|█████     | 35/69 [00:03<00:03, 11.00it/s]
 54%|█████▎    | 37/69 [00:04<00:02, 11.02it/s]
 57%|█████▋    | 39/69 [00:04<00:02, 11.05it/s]
 59%|█████▉    | 41/69 [00:04<00:02, 11.08it/s]
 62%|██████▏   | 43/69 [00:04<00:02, 11.09it/s]
 65%|██████▌   | 45/69 [00:04<00:02, 11.10it/s]
 68%|██████▊   | 47/69 [00:05<00:01, 11.11it/s]
 71%|███████   | 49/69 [00:05<00:01, 11.12it/s]
 74%|███████▍  | 51/69 [00:05<00:01, 11.12it/s]
 77%|███████▋  | 53/69 [00:05<00:01, 11.13it/s]
 80%|███████▉  | 55/69 [00:05<00:01, 11.13it/s]
 83%|████████▎ | 57/69 [00:05<00:01, 11.13it/s]
 86%|████████▌ | 59/69 [00:06<00:00, 11.13it/s]
 88%|████████▊ | 61/69 [00:06<00:00, 11.13it/s]
 91%|█████████▏| 63/69 [00:06<00:00, 11.14it/s]
 94%|█████████▍| 65/69 [00:06<00:00, 11.13it/s]
 97%|█████████▋| 67/69 [00:06<00:00, 11.13it/s]
100%|██████████| 69/69 [00:06<00:00, 12.06it/s]
100%|██████████| 69/69 [00:07<00:00,  9.74it/s]
INFO:Trainer:2024-12-05 19:13:11:mr: 231.937
INFO:Trainer:2024-12-05 19:13:11:mrr: 0.301
INFO:Trainer:2024-12-05 19:13:11:hits@1: 0.202
INFO:Trainer:2024-12-05 19:13:11:hits@3: 0.342
INFO:Trainer:2024-12-05 19:13:11:hits@10: 0.496
INFO:Trainer:2024-12-05 19:13:11:mr_type: 108.675
INFO:Trainer:2024-12-05 19:13:11:mrr_type: 0.352
INFO:Trainer:2024-12-05 19:13:11:hits@1_type: 0.257
INFO:Trainer:2024-12-05 19:13:11:hits@3_type: 0.386
INFO:Trainer:2024-12-05 19:13:11:hits@10_type: 0.542
INFO:EarlyStopping:2024-12-05 19:13:11:Validation score improved (0.485000 --> 0.496000).  Saving model ...
INFO:Trainer:2024-12-05 19:13:11:[cuda:0] Epoch 400 | Training checkpoint saved at ../../checkpoint/transh-400.pth
INFO:Trainer:2024-12-05 19:13:11:[cuda:1] Epoch [ 400/1000] | Batchsize: 8192 | loss:  0.245308 | 3.38706 seconds/epoch
INFO:Trainer:2024-12-05 19:13:11:[cuda:0] Epoch [ 400/1000] | Batchsize: 8192 | loss:  0.245154 | 3.38708 seconds/epoch
INFO:Trainer:2024-12-05 19:18:45:[cuda:0] Epoch 500 | The model starts evaluation on the validation set.

  0%|          | 0/69 [00:00<?, ?it/s]
  1%|▏         | 1/69 [00:01<01:30,  1.33s/it]
  4%|▍         | 3/69 [00:01<00:27,  2.43it/s]
  7%|▋         | 5/69 [00:01<00:15,  4.06it/s]
 10%|█         | 7/69 [00:01<00:11,  5.54it/s]
 13%|█▎        | 9/69 [00:02<00:08,  6.82it/s]
 16%|█▌        | 11/69 [00:02<00:07,  7.88it/s]
 19%|█▉        | 13/69 [00:02<00:06,  8.71it/s]
 22%|██▏       | 15/69 [00:02<00:05,  9.34it/s]
 25%|██▍       | 17/69 [00:02<00:05,  9.82it/s]
 28%|██▊       | 19/69 [00:02<00:04, 10.17it/s]
 30%|███       | 21/69 [00:03<00:04, 10.43it/s]
 33%|███▎      | 23/69 [00:03<00:04, 10.63it/s]
 36%|███▌      | 25/69 [00:03<00:04, 10.77it/s]
 39%|███▉      | 27/69 [00:03<00:03, 10.86it/s]
 42%|████▏     | 29/69 [00:03<00:03, 10.94it/s]
 45%|████▍     | 31/69 [00:04<00:03, 10.98it/s]
 48%|████▊     | 33/69 [00:04<00:03, 11.02it/s]
 51%|█████     | 35/69 [00:04<00:03, 11.05it/s]
 54%|█████▎    | 37/69 [00:04<00:02, 11.06it/s]
 57%|█████▋    | 39/69 [00:04<00:02, 11.08it/s]
 59%|█████▉    | 41/69 [00:04<00:02, 11.10it/s]
 62%|██████▏   | 43/69 [00:05<00:02, 11.11it/s]
 65%|██████▌   | 45/69 [00:05<00:02, 11.12it/s]
 68%|██████▊   | 47/69 [00:05<00:01, 11.12it/s]
 71%|███████   | 49/69 [00:05<00:01, 11.13it/s]
 74%|███████▍  | 51/69 [00:05<00:01, 11.13it/s]
 77%|███████▋  | 53/69 [00:06<00:01, 11.13it/s]
 80%|███████▉  | 55/69 [00:06<00:01, 11.13it/s]
 83%|████████▎ | 57/69 [00:06<00:01, 11.13it/s]
 86%|████████▌ | 59/69 [00:06<00:00, 11.14it/s]
 88%|████████▊ | 61/69 [00:06<00:00, 11.14it/s]
 91%|█████████▏| 63/69 [00:06<00:00, 11.14it/s]
 94%|█████████▍| 65/69 [00:07<00:00, 11.14it/s]
 97%|█████████▋| 67/69 [00:07<00:00, 11.14it/s]
100%|██████████| 69/69 [00:07<00:00, 12.06it/s]
100%|██████████| 69/69 [00:07<00:00,  9.19it/s]
INFO:Trainer:2024-12-05 19:18:53:mr: 235.678
INFO:Trainer:2024-12-05 19:18:53:mrr: 0.3
INFO:Trainer:2024-12-05 19:18:53:hits@1: 0.2
INFO:Trainer:2024-12-05 19:18:53:hits@3: 0.342
INFO:Trainer:2024-12-05 19:18:53:hits@10: 0.495
INFO:Trainer:2024-12-05 19:18:53:mr_type: 108.77
INFO:Trainer:2024-12-05 19:18:53:mrr_type: 0.352
INFO:Trainer:2024-12-05 19:18:53:hits@1_type: 0.257
INFO:Trainer:2024-12-05 19:18:53:hits@3_type: 0.387
INFO:Trainer:2024-12-05 19:18:53:hits@10_type: 0.542
INFO:EarlyStopping:2024-12-05 19:18:53:EarlyStopping counter: 1 / 2
INFO:Trainer:2024-12-05 19:18:53:[cuda:0] Epoch 500 | Training checkpoint saved at ../../checkpoint/transh-500.pth
INFO:Trainer:2024-12-05 19:18:53:[cuda:1] Epoch [ 500/1000] | Batchsize: 8192 | loss:  0.240402 | 3.39273 seconds/epoch
INFO:Trainer:2024-12-05 19:18:53:[cuda:0] Epoch [ 500/1000] | Batchsize: 8192 | loss:  0.239735 | 3.39274 seconds/epoch
INFO:Trainer:2024-12-05 19:24:27:[cuda:0] Epoch 600 | The model starts evaluation on the validation set.

  0%|          | 0/69 [00:00<?, ?it/s]
  1%|▏         | 1/69 [00:00<00:57,  1.18it/s]
  4%|▍         | 3/69 [00:01<00:18,  3.49it/s]
  7%|▋         | 5/69 [00:01<00:11,  5.39it/s]
 10%|█         | 7/69 [00:01<00:08,  6.89it/s]
 13%|█▎        | 9/69 [00:01<00:07,  8.04it/s]
 16%|█▌        | 11/69 [00:01<00:06,  8.89it/s]
 19%|█▉        | 13/69 [00:01<00:05,  9.52it/s]
 22%|██▏       | 15/69 [00:02<00:05,  9.97it/s]
 25%|██▍       | 17/69 [00:02<00:05, 10.29it/s]
 28%|██▊       | 19/69 [00:02<00:04, 10.53it/s]
 30%|███       | 21/69 [00:02<00:04, 10.70it/s]
 33%|███▎      | 23/69 [00:02<00:04, 10.82it/s]
 36%|███▌      | 25/69 [00:03<00:04, 10.91it/s]
 39%|███▉      | 27/69 [00:03<00:03, 10.96it/s]
 42%|████▏     | 29/69 [00:03<00:03, 11.00it/s]
 45%|████▍     | 31/69 [00:03<00:03, 11.02it/s]
 48%|████▊     | 33/69 [00:03<00:03, 11.04it/s]
 51%|█████     | 35/69 [00:03<00:03, 11.06it/s]
 54%|█████▎    | 37/69 [00:04<00:02, 11.07it/s]
 57%|█████▋    | 39/69 [00:04<00:02, 11.08it/s]
 59%|█████▉    | 41/69 [00:04<00:02, 11.10it/s]
 62%|██████▏   | 43/69 [00:04<00:02, 11.11it/s]
 65%|██████▌   | 45/69 [00:04<00:02, 11.11it/s]
 68%|██████▊   | 47/69 [00:04<00:01, 11.12it/s]
 71%|███████   | 49/69 [00:05<00:01, 11.12it/s]
 74%|███████▍  | 51/69 [00:05<00:01, 11.12it/s]
 77%|███████▋  | 53/69 [00:05<00:01, 11.12it/s]
 80%|███████▉  | 55/69 [00:05<00:01, 11.12it/s]
 83%|████████▎ | 57/69 [00:05<00:01, 11.12it/s]
 86%|████████▌ | 59/69 [00:06<00:00, 11.12it/s]
 88%|████████▊ | 61/69 [00:06<00:00, 11.12it/s]
 91%|█████████▏| 63/69 [00:06<00:00, 11.13it/s]
 94%|█████████▍| 65/69 [00:06<00:00, 11.13it/s]
 97%|█████████▋| 67/69 [00:06<00:00, 11.13it/s]
100%|██████████| 69/69 [00:06<00:00, 12.06it/s]
100%|██████████| 69/69 [00:07<00:00,  9.82it/s]
INFO:Trainer:2024-12-05 19:24:34:mr: 236.028
INFO:Trainer:2024-12-05 19:24:34:mrr: 0.3
INFO:Trainer:2024-12-05 19:24:34:hits@1: 0.199
INFO:Trainer:2024-12-05 19:24:34:hits@3: 0.343
INFO:Trainer:2024-12-05 19:24:34:hits@10: 0.496
INFO:Trainer:2024-12-05 19:24:34:mr_type: 109.356
INFO:Trainer:2024-12-05 19:24:34:mrr_type: 0.351
INFO:Trainer:2024-12-05 19:24:34:hits@1_type: 0.257
INFO:Trainer:2024-12-05 19:24:34:hits@3_type: 0.388
INFO:Trainer:2024-12-05 19:24:34:hits@10_type: 0.54
INFO:EarlyStopping:2024-12-05 19:24:34:EarlyStopping counter: 2 / 2
INFO:Trainer:2024-12-05 19:24:34:[cuda:0] Send an early stopping signal
INFO:Trainer:2024-12-05 19:24:34:[cuda:0] Epoch 600 | Training checkpoint saved at ../../checkpoint/transh-600.pth
INFO:Trainer:2024-12-05 19:24:34:[cuda:0] Early stopping
INFO:Trainer:2024-12-05 19:24:34:[cuda:1] Early stopping
INFO:Trainer:2024-12-05 19:24:34:[cuda:0] The model training is completed, taking a total of 2038.45749 seconds.
INFO:Trainer:2024-12-05 19:24:34:[cuda:1] The model training is completed, taking a total of 2038.45216 seconds.
INFO:Trainer:2024-12-05 19:24:34:[cuda:0] Model saved at ../../checkpoint/transh.pth.
INFO:Trainer:2024-12-05 19:24:34:[cuda:0] The model starts evaluating in the test set.

  0%|          | 0/80 [00:00<?, ?it/s]
  1%|▏         | 1/80 [00:00<01:14,  1.07it/s]
  4%|▍         | 3/80 [00:01<00:23,  3.22it/s]
  6%|▋         | 5/80 [00:01<00:14,  5.06it/s]
  9%|▉         | 7/80 [00:01<00:11,  6.56it/s]
 11%|█▏        | 9/80 [00:01<00:09,  7.75it/s]
 14%|█▍        | 11/80 [00:01<00:07,  8.65it/s]
 16%|█▋        | 13/80 [00:02<00:07,  9.32it/s]
 19%|█▉        | 15/80 [00:02<00:06,  9.81it/s]
 21%|██▏       | 17/80 [00:02<00:06, 10.17it/s]
 24%|██▍       | 19/80 [00:02<00:05, 10.42it/s]
 26%|██▋       | 21/80 [00:02<00:05, 10.61it/s]
 29%|██▉       | 23/80 [00:02<00:05, 10.73it/s]
 31%|███▏      | 25/80 [00:03<00:05, 10.82it/s]
 34%|███▍      | 27/80 [00:03<00:04, 10.88it/s]
 36%|███▋      | 29/80 [00:03<00:04, 10.91it/s]
 39%|███▉      | 31/80 [00:03<00:04, 10.95it/s]
 41%|████▏     | 33/80 [00:03<00:04, 11.00it/s]
 44%|████▍     | 35/80 [00:04<00:04, 11.03it/s]
 46%|████▋     | 37/80 [00:04<00:03, 11.05it/s]
 49%|████▉     | 39/80 [00:04<00:03, 11.07it/s]
 51%|█████▏    | 41/80 [00:04<00:03, 11.08it/s]
 54%|█████▍    | 43/80 [00:04<00:03, 11.09it/s]
 56%|█████▋    | 45/80 [00:04<00:03, 11.09it/s]
 59%|█████▉    | 47/80 [00:05<00:02, 11.09it/s]
 61%|██████▏   | 49/80 [00:05<00:02, 11.10it/s]
 64%|██████▍   | 51/80 [00:05<00:02, 11.11it/s]
 66%|██████▋   | 53/80 [00:05<00:02, 11.11it/s]
 69%|██████▉   | 55/80 [00:05<00:02, 11.12it/s]wandb: - 0.022 MB of 0.022 MB uploaded

 71%|███████▏  | 57/80 [00:05<00:02, 11.12it/s]
 74%|███████▍  | 59/80 [00:06<00:01, 11.13it/s]
 76%|███████▋  | 61/80 [00:06<00:01, 11.13it/s]
 79%|███████▉  | 63/80 [00:06<00:01, 11.13it/s]
 81%|████████▏ | 65/80 [00:06<00:01, 11.13it/s]
 84%|████████▍ | 67/80 [00:06<00:01, 11.13it/s]wandb: \ 0.023 MB of 0.032 MB uploaded

 86%|████████▋ | 69/80 [00:07<00:00, 11.13it/s]
 89%|████████▉ | 71/80 [00:07<00:00, 11.13it/s]
 91%|█████████▏| 73/80 [00:07<00:00, 11.13it/s]
 94%|█████████▍| 75/80 [00:07<00:00, 11.13it/s]
 96%|█████████▋| 77/80 [00:07<00:00, 11.13it/s]wandb: | 0.031 MB of 0.032 MB uploaded

 99%|█████████▉| 79/80 [00:07<00:00, 11.13it/s]
100%|██████████| 80/80 [00:08<00:00,  9.79it/s]
INFO:Trainer:2024-12-05 19:24:43:mr: 250.415
INFO:Trainer:2024-12-05 19:24:43:mrr: 0.293
INFO:Trainer:2024-12-05 19:24:43:hits@1: 0.194
INFO:Trainer:2024-12-05 19:24:43:hits@3: 0.332
INFO:Trainer:2024-12-05 19:24:43:hits@10: 0.489
INFO:Trainer:2024-12-05 19:24:43:mr_type: 112.433
INFO:Trainer:2024-12-05 19:24:43:mrr_type: 0.343
INFO:Trainer:2024-12-05 19:24:43:hits@1_type: 0.248
INFO:Trainer:2024-12-05 19:24:43:hits@3_type: 0.378
INFO:Trainer:2024-12-05 19:24:43:hits@10_type: 0.532
wandb: / 0.031 MB of 0.032 MB uploaded
wandb: - 0.032 MB of 0.032 MB uploaded
wandb:                                                                                
wandb: 🚀 View run TransH-FB15K237-multi at: https://wandb.ai/3555028709/pybind11-ke/runs/q2t7lym7
wandb: ⭐️ View project at: https://wandb.ai/3555028709/pybind11-ke
wandb: Synced 5 W&B file(s), 0 media file(s), 2 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./wandb/run-20240512_185020-q2t7lym7/logs
DEBUG:connectionpool:2024-12-05 19:24:45:Starting new HTTPS connection (1): o151352.ingest.sentry.io:443
wandb: - 0.014 MB of 0.014 MB uploaded
DEBUG:connectionpool:2024-12-05 19:24:45:https://o151352.ingest.sentry.io:443 "POST /api/4504800232407040/envelope/ HTTP/1.1" 200 0
wandb: \ 0.014 MB of 0.014 MB uploaded
wandb: | 0.014 MB of 0.014 MB uploaded
wandb: / 0.014 MB of 0.014 MB uploaded
wandb: - 0.014 MB of 0.014 MB uploaded
wandb: \ 0.024 MB of 0.024 MB uploaded (0.002 MB deduped)
wandb: | 0.032 MB of 0.040 MB uploaded (0.002 MB deduped)
wandb: / 0.038 MB of 0.040 MB uploaded (0.002 MB deduped)
wandb: - 0.038 MB of 0.040 MB uploaded (0.002 MB deduped)
wandb: \ 0.040 MB of 0.040 MB uploaded (0.002 MB deduped)
wandb:                                                                                
wandb: W&B sync reduced upload amount by 5.0%             
wandb: 
wandb: Run history:
wandb:          duration ▁
wandb:       test/hits@1 ▁
wandb:      test/hits@10 ▁
wandb: test/hits@10_type ▁
wandb:  test/hits@1_type ▁
wandb:       test/hits@3 ▁
wandb:  test/hits@3_type ▁
wandb:           test/mr ▁
wandb:      test/mr_type ▁
wandb:          test/mrr ▁
wandb:     test/mrr_type ▁
wandb:       train/epoch ▁▃▅▆█
wandb:  train/train_loss █▅▃▁▁
wandb:         val/epoch ▁▂▄▅▇█
wandb:        val/hits@1 ▁▆▆█▇▇
wandb:       val/hits@10 ▁▆▆███
wandb:  val/hits@10_type ▁▆▇███
wandb:   val/hits@1_type ▁▅▆███
wandb:        val/hits@3 ▁▅▇███
wandb:   val/hits@3_type ▁▅▇███
wandb:            val/mr █▁▃▅▇▇
wandb:       val/mr_type █▃▂▁▁▁
wandb:           val/mrr ▁▆▇███
wandb:      val/mrr_type ▁▅▆███
wandb: 
wandb: Run summary:
wandb:          duration 2038.45749
wandb:       test/hits@1 0.194
wandb:      test/hits@10 0.489
wandb: test/hits@10_type 0.532
wandb:  test/hits@1_type 0.248
wandb:       test/hits@3 0.332
wandb:  test/hits@3_type 0.378
wandb:           test/mr 250.415
wandb:      test/mr_type 112.433
wandb:          test/mrr 0.293
wandb:     test/mrr_type 0.343
wandb:       train/epoch 500
wandb:  train/train_loss 0.23973
wandb:         val/epoch 599
wandb:        val/hits@1 0.199
wandb:       val/hits@10 0.496
wandb:  val/hits@10_type 0.54
wandb:   val/hits@1_type 0.257
wandb:        val/hits@3 0.343
wandb:   val/hits@3_type 0.388
wandb:            val/mr 236.028
wandb:       val/mr_type 109.356
wandb:           val/mrr 0.3
wandb:      val/mrr_type 0.351
wandb: 
wandb: 🚀 View run TransH-FB15K237-multi at: https://wandb.ai/3555028709/pybind11-ke/runs/x9qosnku
wandb: ⭐️ View project at: https://wandb.ai/3555028709/pybind11-ke
wandb: Synced 5 W&B file(s), 0 media file(s), 2 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./wandb/run-20240512_185020-x9qosnku/logs
DEBUG:connectionpool:2024-12-05 19:24:54:Starting new HTTPS connection (1): o151352.ingest.sentry.io:443
DEBUG:connectionpool:2024-12-05 19:24:55:https://o151352.ingest.sentry.io:443 "POST /api/4504800232407040/envelope/ HTTP/1.1" 200 0