Using RTX 4000 series which doesn't support faster communication speedups. Ensuring P2P and IB communications are disabled.
DEBUG:cmd:2024-11-05 14:39:22:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False)
DEBUG:cmd:2024-11-05 14:39:22:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False)
DEBUG:cmd:2024-11-05 14:39:22:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False)
DEBUG:cmd:2024-11-05 14:39:22:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False)
DEBUG:connectionpool:2024-11-05 14:39:23:Starting new HTTPS connection (1): api.wandb.ai:443
DEBUG:connectionpool:2024-11-05 14:39:23:Starting new HTTPS connection (1): api.wandb.ai:443
DEBUG:connectionpool:2024-11-05 14:39:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
DEBUG:connectionpool:2024-11-05 14:39:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
DEBUG:connectionpool:2024-11-05 14:39:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
wandb: Currently logged in as: 3555028709. Use `wandb login --relogin` to force relogin
DEBUG:cmd:2024-11-05 14:39:24:Popen(['git', 'cat-file', '--batch-check'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=<valid stream>, shell=False, universal_newlines=False)
DEBUG:connectionpool:2024-11-05 14:39:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
wandb: Currently logged in as: 3555028709. Use `wandb login --relogin` to force relogin
DEBUG:cmd:2024-11-05 14:39:25:Popen(['git', 'cat-file', '--batch-check'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=<valid stream>, shell=False, universal_newlines=False)
wandb: - Waiting for wandb.init()...
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: | Waiting for wandb.init()...
wandb: wandb version 0.17.0 is available!  To upgrade, please run:
wandb:  $ pip install wandb --upgrade
wandb: Tracking run with wandb version 0.16.6
wandb: Run data is saved locally in /home/luyanfeng/my_code/github/pybind11-OpenKE/examples/TransE/wandb/run-20240511_143925-89550puv
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run TransE-FB15K-multi
wandb: ⭐️ View project at https://wandb.ai/3555028709/pybind11-ke
wandb: 🚀 View run at https://wandb.ai/3555028709/pybind11-ke/runs/89550puv
wandb: wandb version 0.17.0 is available!  To upgrade, please run:
wandb:  $ pip install wandb --upgrade
wandb: Tracking run with wandb version 0.16.6
wandb: Run data is saved locally in /home/luyanfeng/my_code/github/pybind11-OpenKE/examples/TransE/wandb/run-20240511_143924-7dqahmj4
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run TransE-FB15K-multi
wandb: ⭐️ View project at https://wandb.ai/3555028709/pybind11-ke
wandb: 🚀 View run at https://wandb.ai/3555028709/pybind11-ke/runs/7dqahmj4
INFO:distributed_c10d:2024-11-05 14:39:32:Added key: store_based_barrier_key:1 to store for rank: 1
wandb: Network error (TransientError), entering retry loop.
INFO:distributed_c10d:2024-11-05 14:39:35:Added key: store_based_barrier_key:1 to store for rank: 0
INFO:distributed_c10d:2024-11-05 14:39:35:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
INFO:distributed_c10d:2024-11-05 14:39:35:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
INFO:Trainer:2024-11-05 14:39:36:[cuda:0] Initialization completed, start model training.
INFO:Trainer:2024-11-05 14:39:36:[cuda:1] Initialization completed, start model training.
INFO:distributed:2024-11-05 14:39:38:Reducer buckets have been rebuilt in this iteration.
INFO:distributed:2024-11-05 14:39:38:Reducer buckets have been rebuilt in this iteration.
INFO:Trainer:2024-11-05 14:47:39:[cuda:0] Epoch 100 | The model starts evaluation on the validation set.

  0%|          | 0/196 [00:00<?, ?it/s]
  1%|          | 1/196 [00:01<04:32,  1.40s/it]
  3%|▎         | 6/196 [00:01<00:36,  5.26it/s]
  6%|▌         | 12/196 [00:01<00:16, 11.43it/s]
  9%|▉         | 18/196 [00:01<00:09, 17.92it/s]
 12%|█▏        | 24/196 [00:01<00:07, 24.25it/s]
 15%|█▌        | 30/196 [00:01<00:05, 30.13it/s]
 18%|█▊        | 36/196 [00:02<00:04, 35.30it/s]
 21%|██▏       | 42/196 [00:02<00:03, 39.52it/s]
 24%|██▍       | 48/196 [00:02<00:03, 43.07it/s]
 28%|██▊       | 54/196 [00:02<00:03, 45.60it/s]
 31%|███       | 60/196 [00:02<00:02, 47.64it/s]
 34%|███▎      | 66/196 [00:02<00:02, 48.94it/s]
 37%|███▋      | 72/196 [00:02<00:02, 48.46it/s]
 40%|███▉      | 78/196 [00:02<00:02, 46.06it/s]
 42%|████▏     | 83/196 [00:03<00:02, 45.32it/s]
 45%|████▍     | 88/196 [00:03<00:02, 43.33it/s]
 47%|████▋     | 93/196 [00:03<00:02, 44.56it/s]
 50%|█████     | 98/196 [00:03<00:02, 44.53it/s]
 53%|█████▎    | 103/196 [00:03<00:02, 44.32it/s]
 55%|█████▌    | 108/196 [00:03<00:02, 43.43it/s]
 58%|█████▊    | 113/196 [00:03<00:01, 43.56it/s]
 60%|██████    | 118/196 [00:03<00:01, 43.23it/s]
 63%|██████▎   | 123/196 [00:03<00:01, 43.58it/s]
 65%|██████▌   | 128/196 [00:04<00:01, 43.27it/s]
 68%|██████▊   | 133/196 [00:04<00:01, 43.49it/s]
 70%|███████   | 138/196 [00:04<00:01, 42.96it/s]
 73%|███████▎  | 143/196 [00:04<00:01, 43.12it/s]
 76%|███████▌  | 148/196 [00:04<00:01, 43.30it/s]
 78%|███████▊  | 153/196 [00:04<00:00, 43.61it/s]
 81%|████████  | 158/196 [00:04<00:00, 43.98it/s]
 83%|████████▎ | 163/196 [00:04<00:00, 42.00it/s]
 86%|████████▌ | 168/196 [00:05<00:00, 40.90it/s]
 88%|████████▊ | 173/196 [00:05<00:00, 41.13it/s]
 91%|█████████ | 178/196 [00:05<00:00, 41.29it/s]
 93%|█████████▎| 183/196 [00:05<00:00, 41.42it/s]
 96%|█████████▌| 188/196 [00:05<00:00, 41.72it/s]
 98%|█████████▊| 193/196 [00:05<00:00, 41.84it/s]
100%|██████████| 196/196 [00:05<00:00, 33.65it/s]
INFO:Trainer:2024-11-05 14:47:45:mr: 102.84
INFO:Trainer:2024-11-05 14:47:45:mrr: 0.442
INFO:Trainer:2024-11-05 14:47:45:hits@1: 0.304
INFO:Trainer:2024-11-05 14:47:45:hits@3: 0.524
INFO:Trainer:2024-11-05 14:47:45:hits@10: 0.693
INFO:Trainer:2024-11-05 14:47:45:mr_type: 43.708
INFO:Trainer:2024-11-05 14:47:45:mrr_type: 0.489
INFO:Trainer:2024-11-05 14:47:45:hits@1_type: 0.354
INFO:Trainer:2024-11-05 14:47:45:hits@3_type: 0.571
INFO:Trainer:2024-11-05 14:47:45:hits@10_type: 0.738
INFO:EarlyStopping:2024-11-05 14:47:45:Validation score improved (-inf --> 0.693000).  Saving model ...
INFO:Trainer:2024-11-05 14:47:45:[cuda:0] Epoch 100 | Training checkpoint saved at ../../checkpoint/transe-100.pth
INFO:Trainer:2024-11-05 14:47:45:[cuda:1] Epoch [ 100/1000] | Batchsize: 8192 | loss:  0.091870 | 4.83229 seconds/epoch
INFO:Trainer:2024-11-05 14:47:45:[cuda:0] Epoch [ 100/1000] | Batchsize: 8192 | loss:  0.092685 | 4.83236 seconds/epoch
INFO:Trainer:2024-11-05 14:55:41:[cuda:0] Epoch 200 | The model starts evaluation on the validation set.

  0%|          | 0/196 [00:00<?, ?it/s]
  1%|          | 1/196 [00:00<02:57,  1.10it/s]
  3%|▎         | 6/196 [00:01<00:24,  7.66it/s]
  6%|▌         | 12/196 [00:01<00:11, 15.74it/s]
  9%|▊         | 17/196 [00:01<00:08, 21.82it/s]
 12%|█▏        | 23/196 [00:01<00:05, 29.05it/s]
 15%|█▍        | 29/196 [00:01<00:04, 34.97it/s]
 18%|█▊        | 35/196 [00:01<00:04, 39.72it/s]
 21%|██        | 41/196 [00:01<00:03, 43.46it/s]
 24%|██▍       | 47/196 [00:01<00:03, 44.62it/s]
 27%|██▋       | 53/196 [00:01<00:03, 39.97it/s]
 30%|███       | 59/196 [00:02<00:03, 42.88it/s]
 33%|███▎      | 64/196 [00:02<00:03, 42.61it/s]
 35%|███▌      | 69/196 [00:02<00:03, 39.18it/s]
 38%|███▊      | 74/196 [00:02<00:02, 41.31it/s]
 40%|████      | 79/196 [00:02<00:02, 40.07it/s]
 43%|████▎     | 84/196 [00:02<00:02, 39.37it/s]
 45%|████▌     | 89/196 [00:02<00:02, 38.27it/s]
 47%|████▋     | 93/196 [00:03<00:02, 38.09it/s]
 49%|████▉     | 97/196 [00:03<00:02, 37.79it/s]
 52%|█████▏    | 101/196 [00:03<00:02, 37.78it/s]
 54%|█████▎    | 105/196 [00:03<00:02, 38.00it/s]
 56%|█████▌    | 109/196 [00:03<00:02, 38.08it/s]
 58%|█████▊    | 113/196 [00:03<00:02, 37.84it/s]
 60%|█████▉    | 117/196 [00:03<00:02, 37.10it/s]
 62%|██████▏   | 121/196 [00:03<00:02, 37.13it/s]
 64%|██████▍   | 125/196 [00:03<00:01, 37.29it/s]
 66%|██████▌   | 129/196 [00:03<00:01, 37.25it/s]
 68%|██████▊   | 133/196 [00:04<00:01, 37.49it/s]
 70%|██████▉   | 137/196 [00:04<00:01, 37.74it/s]
 72%|███████▏  | 141/196 [00:04<00:01, 37.92it/s]
 74%|███████▍  | 145/196 [00:04<00:01, 37.58it/s]
 76%|███████▌  | 149/196 [00:04<00:01, 37.14it/s]
 78%|███████▊  | 153/196 [00:04<00:01, 37.30it/s]
 80%|████████  | 157/196 [00:04<00:01, 37.47it/s]
 82%|████████▏ | 161/196 [00:04<00:00, 37.38it/s]
 84%|████████▍ | 165/196 [00:04<00:00, 37.76it/s]
 87%|████████▋ | 170/196 [00:05<00:00, 40.29it/s]
 89%|████████▉ | 175/196 [00:05<00:00, 42.93it/s]
 92%|█████████▏| 180/196 [00:05<00:00, 44.37it/s]
 95%|█████████▍| 186/196 [00:05<00:00, 46.41it/s]
 98%|█████████▊| 192/196 [00:05<00:00, 47.84it/s]
100%|██████████| 196/196 [00:05<00:00, 34.56it/s]
INFO:Trainer:2024-11-05 14:55:46:mr: 95.422
INFO:Trainer:2024-11-05 14:55:46:mrr: 0.474
INFO:Trainer:2024-11-05 14:55:46:hits@1: 0.335
INFO:Trainer:2024-11-05 14:55:46:hits@3: 0.561
INFO:Trainer:2024-11-05 14:55:46:hits@10: 0.722
INFO:Trainer:2024-11-05 14:55:46:mr_type: 40.483
INFO:Trainer:2024-11-05 14:55:46:mrr_type: 0.52
INFO:Trainer:2024-11-05 14:55:46:hits@1_type: 0.384
INFO:Trainer:2024-11-05 14:55:46:hits@3_type: 0.608
INFO:Trainer:2024-11-05 14:55:46:hits@10_type: 0.764
INFO:EarlyStopping:2024-11-05 14:55:46:Validation score improved (0.693000 --> 0.722000).  Saving model ...
INFO:Trainer:2024-11-05 14:55:46:[cuda:0] Epoch 200 | Training checkpoint saved at ../../checkpoint/transe-200.pth
INFO:Trainer:2024-11-05 14:55:46:[cuda:1] Epoch [ 200/1000] | Batchsize: 8192 | loss:  0.074659 | 4.82544 seconds/epoch
INFO:Trainer:2024-11-05 14:55:46:[cuda:0] Epoch [ 200/1000] | Batchsize: 8192 | loss:  0.073929 | 4.82542 seconds/epoch
INFO:Trainer:2024-11-05 15:03:43:[cuda:0] Epoch 300 | The model starts evaluation on the validation set.

  0%|          | 0/196 [00:00<?, ?it/s]
  1%|          | 1/196 [00:00<03:10,  1.03it/s]
  3%|▎         | 6/196 [00:01<00:26,  7.24it/s]
  6%|▌         | 12/196 [00:01<00:12, 14.99it/s]
  9%|▉         | 18/196 [00:01<00:07, 22.33it/s]
 12%|█▏        | 24/196 [00:01<00:05, 28.99it/s]
 15%|█▌        | 30/196 [00:01<00:04, 34.55it/s]
 18%|█▊        | 36/196 [00:01<00:04, 38.98it/s]
 21%|██▏       | 42/196 [00:01<00:03, 42.69it/s]
 24%|██▍       | 48/196 [00:01<00:03, 45.42it/s]
 28%|██▊       | 54/196 [00:02<00:03, 46.16it/s]
 31%|███       | 60/196 [00:02<00:02, 45.55it/s]
 33%|███▎      | 65/196 [00:02<00:02, 45.04it/s]
 36%|███▌      | 70/196 [00:02<00:02, 44.78it/s]
 38%|███▊      | 75/196 [00:02<00:02, 45.34it/s]
 41%|████      | 80/196 [00:02<00:02, 45.22it/s]
 43%|████▎     | 85/196 [00:02<00:02, 45.43it/s]
 46%|████▌     | 90/196 [00:02<00:02, 44.83it/s]
 48%|████▊     | 95/196 [00:02<00:02, 44.71it/s]
 51%|█████     | 100/196 [00:03<00:02, 44.66it/s]
 54%|█████▎    | 105/196 [00:03<00:02, 45.38it/s]
 56%|█████▌    | 110/196 [00:03<00:01, 45.73it/s]
 59%|█████▊    | 115/196 [00:03<00:01, 45.36it/s]
 61%|██████    | 120/196 [00:03<00:01, 44.80it/s]
 64%|██████▍   | 125/196 [00:03<00:01, 45.38it/s]
 66%|██████▋   | 130/196 [00:03<00:01, 44.89it/s]
 69%|██████▉   | 135/196 [00:03<00:01, 45.32it/s]
 71%|███████▏  | 140/196 [00:03<00:01, 45.74it/s]
 74%|███████▍  | 145/196 [00:04<00:01, 45.48it/s]
 77%|███████▋  | 150/196 [00:04<00:01, 44.79it/s]
 79%|███████▉  | 155/196 [00:04<00:00, 45.21it/s]
 82%|████████▏ | 160/196 [00:04<00:00, 44.78it/s]
 84%|████████▍ | 165/196 [00:04<00:00, 44.99it/s]
 87%|████████▋ | 171/196 [00:04<00:00, 46.73it/s]
 90%|████████▉ | 176/196 [00:04<00:00, 47.28it/s]
 92%|█████████▏| 181/196 [00:04<00:00, 47.65it/s]
 95%|█████████▌| 187/196 [00:04<00:00, 48.62it/s]
 98%|█████████▊| 192/196 [00:05<00:00, 47.59it/s]
100%|██████████| 196/196 [00:05<00:00, 37.36it/s]
INFO:Trainer:2024-11-05 15:03:48:mr: 91.915
INFO:Trainer:2024-11-05 15:03:48:mrr: 0.495
INFO:Trainer:2024-11-05 15:03:48:hits@1: 0.358
INFO:Trainer:2024-11-05 15:03:48:hits@3: 0.584
INFO:Trainer:2024-11-05 15:03:48:hits@10: 0.739
INFO:Trainer:2024-11-05 15:03:48:mr_type: 38.307
INFO:Trainer:2024-11-05 15:03:48:mrr_type: 0.541
INFO:Trainer:2024-11-05 15:03:48:hits@1_type: 0.406
INFO:Trainer:2024-11-05 15:03:48:hits@3_type: 0.63
INFO:Trainer:2024-11-05 15:03:48:hits@10_type: 0.779
INFO:EarlyStopping:2024-11-05 15:03:48:Validation score improved (0.722000 --> 0.739000).  Saving model ...
INFO:Trainer:2024-11-05 15:03:48:[cuda:0] Epoch 300 | Training checkpoint saved at ../../checkpoint/transe-300.pth
INFO:Trainer:2024-11-05 15:03:48:[cuda:1] Epoch [ 300/1000] | Batchsize: 8192 | loss:  0.063407 | 4.82434 seconds/epoch
INFO:Trainer:2024-11-05 15:03:48:[cuda:0] Epoch [ 300/1000] | Batchsize: 8192 | loss:  0.064902 | 4.82432 seconds/epoch
INFO:Trainer:2024-11-05 15:11:45:[cuda:0] Epoch 400 | The model starts evaluation on the validation set.

  0%|          | 0/196 [00:00<?, ?it/s]
  1%|          | 1/196 [00:00<03:08,  1.03it/s]
  3%|▎         | 6/196 [00:01<00:26,  7.29it/s]
  6%|▌         | 12/196 [00:01<00:12, 15.16it/s]
  9%|▊         | 17/196 [00:01<00:08, 20.39it/s]
 12%|█▏        | 23/196 [00:01<00:06, 27.63it/s]
 15%|█▍        | 29/196 [00:01<00:04, 33.82it/s]
 18%|█▊        | 35/196 [00:01<00:04, 38.62it/s]
 21%|██        | 41/196 [00:01<00:03, 42.64it/s]
 24%|██▍       | 47/196 [00:01<00:03, 45.70it/s]
 27%|██▋       | 53/196 [00:01<00:02, 47.80it/s]
 30%|███       | 59/196 [00:02<00:02, 49.40it/s]
 33%|███▎      | 65/196 [00:02<00:02, 50.62it/s]
 36%|███▌      | 71/196 [00:02<00:02, 51.44it/s]
 39%|███▉      | 77/196 [00:02<00:02, 49.02it/s]
 42%|████▏     | 83/196 [00:02<00:02, 47.74it/s]
 45%|████▍     | 88/196 [00:02<00:02, 47.22it/s]
 47%|████▋     | 93/196 [00:02<00:02, 46.45it/s]
 50%|█████     | 98/196 [00:02<00:02, 44.19it/s]
 53%|█████▎    | 104/196 [00:03<00:02, 45.86it/s]
 56%|█████▌    | 109/196 [00:03<00:01, 45.25it/s]
 58%|█████▊    | 114/196 [00:03<00:01, 45.16it/s]
 61%|██████    | 119/196 [00:03<00:01, 45.64it/s]
 63%|██████▎   | 124/196 [00:03<00:01, 45.33it/s]
 66%|██████▌   | 129/196 [00:03<00:01, 45.13it/s]
 68%|██████▊   | 134/196 [00:03<00:01, 45.08it/s]
 71%|███████   | 139/196 [00:03<00:01, 45.14it/s]
 73%|███████▎  | 144/196 [00:03<00:01, 44.38it/s]
 76%|███████▌  | 149/196 [00:04<00:01, 43.07it/s]
 79%|███████▊  | 154/196 [00:04<00:00, 43.32it/s]
 81%|████████  | 159/196 [00:04<00:00, 43.09it/s]
 84%|████████▎ | 164/196 [00:04<00:00, 41.83it/s]
 86%|████████▌ | 169/196 [00:04<00:00, 43.27it/s]
 89%|████████▉ | 174/196 [00:04<00:00, 45.06it/s]
 91%|█████████▏| 179/196 [00:04<00:00, 46.00it/s]
 94%|█████████▍| 184/196 [00:04<00:00, 47.01it/s]
 96%|█████████▋| 189/196 [00:04<00:00, 47.54it/s]
 99%|█████████▉| 194/196 [00:05<00:00, 47.48it/s]
100%|██████████| 196/196 [00:05<00:00, 37.59it/s]
INFO:Trainer:2024-11-05 15:11:50:mr: 83.502
INFO:Trainer:2024-11-05 15:11:50:mrr: 0.531
INFO:Trainer:2024-11-05 15:11:50:hits@1: 0.396
INFO:Trainer:2024-11-05 15:11:50:hits@3: 0.624
INFO:Trainer:2024-11-05 15:11:50:hits@10: 0.764
INFO:Trainer:2024-11-05 15:11:50:mr_type: 35.525
INFO:Trainer:2024-11-05 15:11:50:mrr_type: 0.575
INFO:Trainer:2024-11-05 15:11:50:hits@1_type: 0.444
INFO:Trainer:2024-11-05 15:11:50:hits@3_type: 0.667
INFO:Trainer:2024-11-05 15:11:50:hits@10_type: 0.8
INFO:EarlyStopping:2024-11-05 15:11:50:Validation score improved (0.739000 --> 0.764000).  Saving model ...
INFO:Trainer:2024-11-05 15:11:50:[cuda:0] Epoch 400 | Training checkpoint saved at ../../checkpoint/transe-400.pth
INFO:Trainer:2024-11-05 15:11:50:[cuda:1] Epoch [ 400/1000] | Batchsize: 8192 | loss:  0.050420 | 4.82323 seconds/epoch
INFO:Trainer:2024-11-05 15:11:50:[cuda:0] Epoch [ 400/1000] | Batchsize: 8192 | loss:  0.050849 | 4.82319 seconds/epoch
INFO:Trainer:2024-11-05 15:19:46:[cuda:0] Epoch 500 | The model starts evaluation on the validation set.

  0%|          | 0/196 [00:00<?, ?it/s]
  1%|          | 1/196 [00:00<03:04,  1.06it/s]
  3%|▎         | 6/196 [00:01<00:25,  7.42it/s]
  6%|▌         | 12/196 [00:01<00:12, 15.33it/s]
  9%|▊         | 17/196 [00:01<00:08, 20.08it/s]
 12%|█▏        | 23/196 [00:01<00:06, 27.09it/s]
 15%|█▍        | 29/196 [00:01<00:05, 33.06it/s]
 18%|█▊        | 35/196 [00:01<00:04, 37.88it/s]
 21%|██        | 41/196 [00:01<00:03, 41.49it/s]
 24%|██▍       | 47/196 [00:01<00:03, 44.31it/s]
 27%|██▋       | 53/196 [00:01<00:03, 46.49it/s]
 30%|███       | 59/196 [00:02<00:02, 47.74it/s]
 33%|███▎      | 65/196 [00:02<00:02, 49.04it/s]
 36%|███▌      | 71/196 [00:02<00:02, 48.60it/s]
 39%|███▉      | 77/196 [00:02<00:02, 43.37it/s]
 42%|████▏     | 82/196 [00:02<00:02, 41.41it/s]
 44%|████▍     | 87/196 [00:02<00:02, 40.10it/s]
 47%|████▋     | 92/196 [00:02<00:02, 39.27it/s]
 49%|████▉     | 96/196 [00:03<00:02, 38.89it/s]
 51%|█████     | 100/196 [00:03<00:02, 38.46it/s]
 53%|█████▎    | 104/196 [00:03<00:02, 37.52it/s]
 55%|█████▌    | 108/196 [00:03<00:02, 36.91it/s]
 57%|█████▋    | 112/196 [00:03<00:02, 37.06it/s]
 59%|█████▉    | 116/196 [00:03<00:02, 37.63it/s]
 62%|██████▏   | 121/196 [00:03<00:01, 39.45it/s]
 64%|██████▍   | 126/196 [00:03<00:01, 40.77it/s]
 67%|██████▋   | 131/196 [00:03<00:01, 41.61it/s]
 69%|██████▉   | 136/196 [00:04<00:01, 41.41it/s]
 72%|███████▏  | 141/196 [00:04<00:01, 41.86it/s]
 74%|███████▍  | 146/196 [00:04<00:01, 42.34it/s]
 77%|███████▋  | 151/196 [00:04<00:01, 42.56it/s]
 80%|███████▉  | 156/196 [00:04<00:00, 42.95it/s]
 82%|████████▏ | 161/196 [00:04<00:00, 43.02it/s]
 85%|████████▍ | 166/196 [00:04<00:00, 42.76it/s]
 87%|████████▋ | 171/196 [00:04<00:00, 44.30it/s]
 90%|████████▉ | 176/196 [00:04<00:00, 45.85it/s]
 92%|█████████▏| 181/196 [00:05<00:00, 46.54it/s]
 95%|█████████▍| 186/196 [00:05<00:00, 47.46it/s]
 97%|█████████▋| 191/196 [00:05<00:00, 48.07it/s]
100%|██████████| 196/196 [00:05<00:00, 48.62it/s]
100%|██████████| 196/196 [00:05<00:00, 35.69it/s]
INFO:Trainer:2024-11-05 15:19:52:mr: 82.593
INFO:Trainer:2024-11-05 15:19:52:mrr: 0.539
INFO:Trainer:2024-11-05 15:19:52:hits@1: 0.405
INFO:Trainer:2024-11-05 15:19:52:hits@3: 0.631
INFO:Trainer:2024-11-05 15:19:52:hits@10: 0.768
INFO:Trainer:2024-11-05 15:19:52:mr_type: 35.092
INFO:Trainer:2024-11-05 15:19:52:mrr_type: 0.583
INFO:Trainer:2024-11-05 15:19:52:hits@1_type: 0.453
INFO:Trainer:2024-11-05 15:19:52:hits@3_type: 0.674
INFO:Trainer:2024-11-05 15:19:52:hits@10_type: 0.804
INFO:EarlyStopping:2024-11-05 15:19:52:EarlyStopping counter: 1 / 2
INFO:Trainer:2024-11-05 15:19:52:[cuda:0] Epoch 500 | Training checkpoint saved at ../../checkpoint/transe-500.pth
INFO:Trainer:2024-11-05 15:19:52:[cuda:1] Epoch [ 500/1000] | Batchsize: 8192 | loss:  0.048449 | 4.82109 seconds/epoch
INFO:Trainer:2024-11-05 15:19:52:[cuda:0] Epoch [ 500/1000] | Batchsize: 8192 | loss:  0.049114 | 4.82115 seconds/epoch
INFO:Trainer:2024-11-05 15:27:55:[cuda:0] Epoch 600 | The model starts evaluation on the validation set.

  0%|          | 0/196 [00:00<?, ?it/s]
  1%|          | 1/196 [00:00<02:51,  1.13it/s]
  3%|▎         | 6/196 [00:00<00:24,  7.88it/s]
  6%|▌         | 12/196 [00:01<00:11, 16.14it/s]
  9%|▊         | 17/196 [00:01<00:08, 20.34it/s]
 12%|█▏        | 23/196 [00:01<00:06, 27.43it/s]
 15%|█▍        | 29/196 [00:01<00:04, 33.49it/s]
 18%|█▊        | 35/196 [00:01<00:04, 38.33it/s]
 21%|██        | 41/196 [00:01<00:03, 42.26it/s]
 24%|██▍       | 47/196 [00:01<00:03, 45.30it/s]
 27%|██▋       | 53/196 [00:01<00:03, 47.55it/s]
 30%|███       | 59/196 [00:02<00:02, 49.25it/s]
 33%|███▎      | 65/196 [00:02<00:02, 50.48it/s]
 36%|███▌      | 71/196 [00:02<00:02, 51.30it/s]
 39%|███▉      | 77/196 [00:02<00:02, 48.68it/s]
 42%|████▏     | 83/196 [00:02<00:02, 46.74it/s]
 45%|████▍     | 88/196 [00:02<00:02, 45.64it/s]
 47%|████▋     | 93/196 [00:02<00:02, 45.02it/s]
 50%|█████     | 98/196 [00:02<00:02, 44.47it/s]
 53%|█████▎    | 103/196 [00:03<00:02, 43.79it/s]
 55%|█████▌    | 108/196 [00:03<00:02, 43.29it/s]
 58%|█████▊    | 113/196 [00:03<00:01, 43.14it/s]
 60%|██████    | 118/196 [00:03<00:01, 43.16it/s]
 63%|██████▎   | 123/196 [00:03<00:01, 43.47it/s]
 65%|██████▌   | 128/196 [00:03<00:01, 43.18it/s]
 68%|██████▊   | 133/196 [00:03<00:01, 42.86it/s]
 70%|███████   | 138/196 [00:03<00:01, 42.80it/s]
 73%|███████▎  | 143/196 [00:03<00:01, 42.55it/s]
 76%|███████▌  | 148/196 [00:04<00:01, 42.69it/s]
 78%|███████▊  | 153/196 [00:04<00:01, 42.86it/s]
 81%|████████  | 158/196 [00:04<00:00, 42.98it/s]
 83%|████████▎ | 163/196 [00:04<00:00, 42.76it/s]
 86%|████████▌ | 168/196 [00:04<00:00, 43.47it/s]
 88%|████████▊ | 173/196 [00:04<00:00, 44.89it/s]
 91%|█████████ | 178/196 [00:04<00:00, 45.88it/s]
 93%|█████████▎| 183/196 [00:04<00:00, 46.81it/s]
 96%|█████████▌| 188/196 [00:04<00:00, 47.45it/s]
 98%|█████████▊| 193/196 [00:05<00:00, 47.41it/s]
100%|██████████| 196/196 [00:05<00:00, 37.45it/s]
INFO:Trainer:2024-11-05 15:28:00:mr: 82.113
INFO:Trainer:2024-11-05 15:28:00:mrr: 0.541
INFO:Trainer:2024-11-05 15:28:00:hits@1: 0.408
INFO:Trainer:2024-11-05 15:28:00:hits@3: 0.633
INFO:Trainer:2024-11-05 15:28:00:hits@10: 0.769
INFO:Trainer:2024-11-05 15:28:00:mr_type: 34.967
INFO:Trainer:2024-11-05 15:28:00:mrr_type: 0.585
INFO:Trainer:2024-11-05 15:28:00:hits@1_type: 0.456
INFO:Trainer:2024-11-05 15:28:00:hits@3_type: 0.676
INFO:Trainer:2024-11-05 15:28:00:hits@10_type: 0.805
INFO:EarlyStopping:2024-11-05 15:28:00:EarlyStopping counter: 2 / 2
INFO:Trainer:2024-11-05 15:28:00:[cuda:0] Send an early stopping signal
INFO:Trainer:2024-11-05 15:28:00:[cuda:0] Epoch 600 | Training checkpoint saved at ../../checkpoint/transe-600.pth
INFO:Trainer:2024-11-05 15:28:00:[cuda:0] Early stopping
INFO:Trainer:2024-11-05 15:28:00:[cuda:1] Early stopping
INFO:Trainer:2024-11-05 15:28:00:[cuda:0] The model training is completed, taking a total of 2898.86857 seconds.
INFO:Trainer:2024-11-05 15:28:00:[cuda:1] The model training is completed, taking a total of 2898.86804 seconds.
INFO:Trainer:2024-11-05 15:28:00:[cuda:0] Model saved at ../../checkpoint/transe.pth.
INFO:Trainer:2024-11-05 15:28:00:[cuda:0] The model starts evaluating in the test set.

  0%|          | 0/231 [00:00<?, ?it/s]
  0%|          | 1/231 [00:01<04:08,  1.08s/it]
  3%|▎         | 6/231 [00:01<00:34,  6.62it/s]
  5%|▌         | 12/231 [00:01<00:15, 13.89it/s]
  7%|▋         | 17/231 [00:01<00:10, 19.75it/s]
 10%|▉         | 22/231 [00:01<00:08, 23.28it/s]
 12%|█▏        | 28/231 [00:01<00:06, 29.73it/s]
 15%|█▍        | 34/231 [00:01<00:05, 35.16it/s]
 17%|█▋        | 40/231 [00:01<00:04, 39.43it/s]wandb: - 0.014 MB of 0.014 MB uploaded

 20%|█▉        | 46/231 [00:02<00:04, 42.77it/s]
 22%|██▏       | 51/231 [00:02<00:04, 44.55it/s]
 24%|██▍       | 56/231 [00:02<00:04, 42.24it/s]
 26%|██▋       | 61/231 [00:02<00:04, 40.82it/s]
 29%|██▊       | 66/231 [00:02<00:04, 39.71it/s]
 31%|███       | 71/231 [00:02<00:04, 39.42it/s]
 33%|███▎      | 76/231 [00:02<00:03, 39.46it/s]
 35%|███▌      | 81/231 [00:02<00:03, 39.20it/s]wandb: \ 0.014 MB of 0.014 MB uploaded

 37%|███▋      | 85/231 [00:03<00:03, 38.82it/s]
 39%|███▊      | 89/231 [00:03<00:03, 38.43it/s]
 40%|████      | 93/231 [00:03<00:03, 38.66it/s]
 42%|████▏     | 97/231 [00:03<00:03, 38.59it/s]
 44%|████▎     | 101/231 [00:03<00:03, 38.72it/s]
 46%|████▌     | 106/231 [00:03<00:03, 40.22it/s]
 48%|████▊     | 111/231 [00:03<00:02, 41.84it/s]
 50%|█████     | 116/231 [00:03<00:02, 42.89it/s]
 52%|█████▏    | 121/231 [00:03<00:02, 43.39it/s]
 55%|█████▍    | 126/231 [00:03<00:02, 44.16it/s]wandb: | 0.014 MB of 0.014 MB uploaded

 57%|█████▋    | 131/231 [00:04<00:02, 44.69it/s]
 59%|█████▉    | 136/231 [00:04<00:02, 45.23it/s]
 61%|██████    | 141/231 [00:04<00:01, 45.26it/s]
 63%|██████▎   | 146/231 [00:04<00:01, 45.14it/s]
 65%|██████▌   | 151/231 [00:04<00:01, 45.11it/s]
 68%|██████▊   | 156/231 [00:04<00:01, 45.42it/s]
 70%|██████▉   | 161/231 [00:04<00:01, 45.69it/s]
 72%|███████▏  | 166/231 [00:04<00:01, 45.96it/s]
 74%|███████▍  | 171/231 [00:04<00:01, 45.97it/s]wandb: / 0.014 MB of 0.014 MB uploaded

 76%|███████▌  | 176/231 [00:05<00:01, 45.55it/s]
 78%|███████▊  | 181/231 [00:05<00:01, 45.39it/s]
 81%|████████  | 186/231 [00:05<00:00, 45.49it/s]
 83%|████████▎ | 191/231 [00:05<00:00, 45.60it/s]
 85%|████████▍ | 196/231 [00:05<00:00, 45.74it/s]
 87%|████████▋ | 201/231 [00:05<00:00, 45.95it/s]
 89%|████████▉ | 206/231 [00:05<00:00, 46.85it/s]
 91%|█████████▏| 211/231 [00:05<00:00, 47.71it/s]
 94%|█████████▍| 217/231 [00:05<00:00, 48.73it/s]wandb: - 0.014 MB of 0.014 MB uploaded

 97%|█████████▋| 223/231 [00:06<00:00, 49.54it/s]
 99%|█████████▉| 229/231 [00:06<00:00, 50.08it/s]
100%|██████████| 231/231 [00:06<00:00, 36.38it/s]
INFO:Trainer:2024-11-05 15:28:06:mr: 84.434
INFO:Trainer:2024-11-05 15:28:06:mrr: 0.54
INFO:Trainer:2024-11-05 15:28:06:hits@1: 0.407
INFO:Trainer:2024-11-05 15:28:06:hits@3: 0.632
INFO:Trainer:2024-11-05 15:28:06:hits@10: 0.769
INFO:Trainer:2024-11-05 15:28:06:mr_type: 35.518
INFO:Trainer:2024-11-05 15:28:06:mrr_type: 0.584
INFO:Trainer:2024-11-05 15:28:06:hits@1_type: 0.454
INFO:Trainer:2024-11-05 15:28:06:hits@3_type: 0.675
INFO:Trainer:2024-11-05 15:28:06:hits@10_type: 0.804
wandb: \ 0.014 MB of 0.014 MB uploaded
wandb: | 0.014 MB of 0.014 MB uploaded
wandb: - 0.014 MB of 0.014 MB uploaded
wandb: / 0.014 MB of 0.014 MB uploaded
wandb: \ 0.014 MB of 0.014 MB uploaded
wandb: - 0.021 MB of 0.021 MB uploaded (0.002 MB deduped)
wandb: | 0.014 MB of 0.014 MB uploaded
wandb: \ 0.021 MB of 0.029 MB uploaded (0.002 MB deduped)
wandb: / 0.014 MB of 0.014 MB uploaded
wandb: | 0.029 MB of 0.029 MB uploaded (0.002 MB deduped)
wandb: - 0.023 MB of 0.023 MB uploaded (0.002 MB deduped)
wandb: / 0.029 MB of 0.029 MB uploaded (0.002 MB deduped)
wandb: - 0.029 MB of 0.029 MB uploaded (0.002 MB deduped)
wandb:                                                                                
wandb: W&B sync reduced upload amount by 7.0%             
wandb: \ 0.023 MB of 0.037 MB uploaded (0.002 MB deduped)
wandb: 🚀 View run TransE-FB15K-multi at: https://wandb.ai/3555028709/pybind11-ke/runs/89550puv
wandb: ⭐️ View project at: https://wandb.ai/3555028709/pybind11-ke
wandb: Synced 5 W&B file(s), 0 media file(s), 2 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./wandb/run-20240511_143925-89550puv/logs
DEBUG:connectionpool:2024-11-05 15:28:14:Starting new HTTPS connection (1): o151352.ingest.sentry.io:443
wandb: | 0.033 MB of 0.037 MB uploaded (0.002 MB deduped)
DEBUG:connectionpool:2024-11-05 15:28:15:https://o151352.ingest.sentry.io:443 "POST /api/4504800232407040/envelope/ HTTP/1.1" 200 0
wandb: / 0.037 MB of 0.037 MB uploaded (0.002 MB deduped)
wandb: - 0.037 MB of 0.037 MB uploaded (0.002 MB deduped)
wandb: \ 0.037 MB of 0.037 MB uploaded (0.002 MB deduped)
wandb:                                                                                
wandb: W&B sync reduced upload amount by 5.4%             
wandb: 
wandb: Run history:
wandb:          duration ▁
wandb:       test/hits@1 ▁
wandb:      test/hits@10 ▁
wandb: test/hits@10_type ▁
wandb:  test/hits@1_type ▁
wandb:       test/hits@3 ▁
wandb:  test/hits@3_type ▁
wandb:           test/mr ▁
wandb:      test/mr_type ▁
wandb:          test/mrr ▁
wandb:     test/mrr_type ▁
wandb:       train/epoch ▁▃▅▆█
wandb:  train/train_loss █▅▄▁▁
wandb:         val/epoch ▁▂▄▅▇█
wandb:        val/hits@1 ▁▃▅▇██
wandb:       val/hits@10 ▁▄▅███
wandb:  val/hits@10_type ▁▄▅▇██
wandb:   val/hits@1_type ▁▃▅▇██
wandb:        val/hits@3 ▁▃▅▇██
wandb:   val/hits@3_type ▁▃▅▇██
wandb:            val/mr █▅▄▁▁▁
wandb:       val/mr_type █▅▄▁▁▁
wandb:           val/mrr ▁▃▅▇██
wandb:      val/mrr_type ▁▃▅▇██
wandb: 
wandb: Run summary:
wandb:          duration 2898.86857
wandb:       test/hits@1 0.407
wandb:      test/hits@10 0.769
wandb: test/hits@10_type 0.804
wandb:  test/hits@1_type 0.454
wandb:       test/hits@3 0.632
wandb:  test/hits@3_type 0.675
wandb:           test/mr 84.434
wandb:      test/mr_type 35.518
wandb:          test/mrr 0.54
wandb:     test/mrr_type 0.584
wandb:       train/epoch 500
wandb:  train/train_loss 0.04911
wandb:         val/epoch 599
wandb:        val/hits@1 0.408
wandb:       val/hits@10 0.769
wandb:  val/hits@10_type 0.805
wandb:   val/hits@1_type 0.456
wandb:        val/hits@3 0.633
wandb:   val/hits@3_type 0.676
wandb:            val/mr 82.113
wandb:       val/mr_type 34.967
wandb:           val/mrr 0.541
wandb:      val/mrr_type 0.585
wandb: 
wandb: 🚀 View run TransE-FB15K-multi at: https://wandb.ai/3555028709/pybind11-ke/runs/7dqahmj4
wandb: ⭐️ View project at: https://wandb.ai/3555028709/pybind11-ke
wandb: Synced 5 W&B file(s), 0 media file(s), 2 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./wandb/run-20240511_143924-7dqahmj4/logs
DEBUG:connectionpool:2024-11-05 15:28:17:Starting new HTTPS connection (1): o151352.ingest.sentry.io:443
DEBUG:connectionpool:2024-11-05 15:28:18:https://o151352.ingest.sentry.io:443 "POST /api/4504800232407040/envelope/ HTTP/1.1" 200 0